Sagar Patil
7f7d5416dd
- test.md: 25 endpoint tests (13 GET + 12 POST) all PASS, including Kafka flow - troubleshoot.md: 9 real problems faced during deployment with fixes - Updated mkdocs.yml nav to include new pages Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
193 lines
6.1 KiB
Markdown
193 lines
6.1 KiB
Markdown
# Troubleshoot Guide — Scheduler Service
|
|
|
|
All real problems faced during deployment and testing at namespace `sagarpatil-4844` on `os-tf-qa.onlinesales.ai`.
|
|
|
|
---
|
|
|
|
## Problem 1: Hades Context Not Needed
|
|
|
|
**Symptom:** Init container `setup-hades-context` was failing/hanging because Hades service was not available or not needed.
|
|
|
|
**Root Cause:** Scheduler-service does NOT use Hades for DB credentials. It connects to MySQL directly via Hibernate XML config (`dbConfig.hibernateXml`). No code in `BaseCommunicator.java` calls `HadesClient.getContext()` — it only calls `HibernateDatabase.initHibernateDB(filePath)`.
|
|
|
|
**Fix:**
|
|
- Set `hadesContext.enabled: false` in both `values.yaml` and `values-local.yaml`
|
|
- Removed Hades from `catalog-info.yaml` dependency-urls and `spec.dependsOn`
|
|
- Removed hades-service from `externalServices` in values files
|
|
|
|
---
|
|
|
|
## Problem 2: GET /job and GET /jobStat Return "Internal Error"
|
|
|
|
**Symptom:**
|
|
```
|
|
curl .../schedulerService/job?jsonQuery={"id":1}
|
|
→ HTTP 500 "Internal Error"
|
|
```
|
|
|
|
**Root Cause:** These are POST-only endpoints. The servlet initialization uses `null` for GET class:
|
|
```java
|
|
// Job.java
|
|
initServletObjectMap(null, JobPostRequest.class, null, JobPostResponse.class);
|
|
// JobStat.java
|
|
initServletObjectMap(null, JobStatPostRequest.class, ...);
|
|
```
|
|
|
|
**Fix:** No fix needed — this is by design. Use POST for `/job` and `/jobStat`.
|
|
|
|
---
|
|
|
|
## Problem 3: GET /jobType Fails Without Both Parameters
|
|
|
|
**Symptom:**
|
|
```
|
|
curl .../schedulerService/jobType?groupName=os-group-1
|
|
→ empty or error response
|
|
```
|
|
|
|
**Root Cause:** `/jobType` GET requires BOTH `groupName` AND `name` (or `id`). A single param is not enough.
|
|
|
|
**Fix:**
|
|
```
|
|
GET /schedulerService/jobType?groupName=os-group-1&name=Default
|
|
```
|
|
|
|
---
|
|
|
|
## Problem 4: GET /tags Fails Without Required Parameters
|
|
|
|
**Symptom:**
|
|
```
|
|
curl .../schedulerService/tags?groupName=os-group-1
|
|
→ error response
|
|
```
|
|
|
|
**Root Cause:** `/tags` GET requires `id` (or `name`) AND `groupId`. Just `groupName` alone won't work.
|
|
|
|
**Fix:**
|
|
```
|
|
GET /schedulerService/tags?id=1&groupId=1
|
|
```
|
|
|
|
---
|
|
|
|
## Problem 5: POST /job with useArgoWorkflows:true Fails
|
|
|
|
**Symptom:**
|
|
```json
|
|
{
|
|
"status": "ERROR",
|
|
"message": "Failed to publish event to EventManager for job 2"
|
|
}
|
|
```
|
|
|
|
**Root Cause:** Event-manager-v2 servlet initialization was crashing at `TopicSettingServlet.refreshTopicSettings()`. The flow:
|
|
1. Event-manager fetches Kafka creds from Hades — `{"brokerList":"kafka:9092","isAuthEnabled":false}` — OK
|
|
2. Event-manager calls property-settings for `PROD_KAFKA_TOPIC_VERSION_SETTING` with `entityType=AGENCY, entityValue=ALL`
|
|
3. Property-settings returns `{"properties":[null]}` — no matching property existed in DB
|
|
|
|
**Fix:** Insert the seed data into `property_settings_db_v2.properties` table:
|
|
```sql
|
|
INSERT IGNORE INTO properties (entity_type, entity_value, property_type, property_value, is_active)
|
|
VALUES ('AGENCY', 'ALL', 'PROD_KAFKA_TOPIC_VERSION_SETTING', '{"events":["GCP_KAFKA"]}', 1);
|
|
```
|
|
|
|
**Permanent Fix:** Updated `002_populate_properties_table.sql` in the `propertysettings-v2-chart` from `'{}'` to `'{"events":["GCP_KAFKA"]}'`.
|
|
|
|
---
|
|
|
|
## Problem 6: Wrong PROD_KAFKA_TOPIC_VERSION_SETTING Format
|
|
|
|
**Symptom:**
|
|
```
|
|
MismatchedInputException: Cannot deserialize value of type HashSet<KafkaVersion> from Object value
|
|
```
|
|
|
|
**Root Cause:** First attempt used wrong JSON format:
|
|
```json
|
|
{"topicVersionSettings":{"default":{"topic":"events","version":"v1"}}}
|
|
```
|
|
|
|
The `PropertySetttingUtil.java` expects `Map<String, Set<KafkaVersion>>`:
|
|
```java
|
|
TypeReference<Map<String, Set<KafkaVersion>>> typeRefence = new TypeReference<>(){};
|
|
```
|
|
|
|
**Fix:** Correct format is:
|
|
```json
|
|
{"events":["GCP_KAFKA"]}
|
|
```
|
|
|
|
Where `KafkaVersion` enum values are: `AWS_KAFKA`, `GCP_KAFKA`. Only `GCP_KAFKA` is active in `DbEvent.java` (AWS_KAFKA case is commented out).
|
|
|
|
---
|
|
|
|
## Problem 7: POST /jobType monitoringTags Format
|
|
|
|
**Symptom:**
|
|
```
|
|
Marshalling Error when sending monitoringTags as JSON array
|
|
```
|
|
|
|
**Root Cause:** `monitoringTags` field expects a JSON **string** (stringified JSON), not a raw JSON object/array.
|
|
|
|
**Fix:**
|
|
```json
|
|
{
|
|
"jobType": {
|
|
"monitoringTags": "{\"FAILED_JOBS\": \"sev4\"}"
|
|
}
|
|
}
|
|
```
|
|
Note the escaped quotes — the value is a string containing JSON, not a nested JSON object.
|
|
|
|
---
|
|
|
|
## Problem 8: POST /job with useArgoWorkflows:false Works But true Fails
|
|
|
|
**Symptom:** Job creation itself works fine (creates `job_details`, `jobs`, `job_triggers` in DB) but the Kafka event publishing step fails.
|
|
|
|
**Root Cause:** When `useArgoWorkflows: true`, scheduler calls `ArgoEventSender` which POSTs to event-manager-v2. If event-manager's Kafka producer is not initialized (due to Problem 5), the event publish fails.
|
|
|
|
**Fix:** Fix Problem 5 first. Once event-manager has the correct `PROD_KAFKA_TOPIC_VERSION_SETTING`, the full flow works:
|
|
```
|
|
POST /job → ArgoEventSender → EventManager → Kafka (GCP_KAFKA)
|
|
```
|
|
|
|
Scheduler logs on success:
|
|
```
|
|
ArgoEventSender: Sending event for job 6 | eventType: argo_ondemand_cluster_1 | eventName: ARGO_JOB_SUBMISSION
|
|
ArgoEventSender: Successfully posted event for job: 6
|
|
```
|
|
|
|
---
|
|
|
|
## Problem 9: MySQL Connection — Finding Root Password
|
|
|
|
**Symptom:** Need to connect to `mysql-8-0` pod for debugging but don't know the root password.
|
|
|
|
**Fix:** Password is stored in Kubernetes secret:
|
|
```bash
|
|
kubectl get secret mysql-8-0 -n sagarpatil-4844 -o jsonpath='{.data.mysql-root-password}' | base64 -d
|
|
```
|
|
Result: `local-root-pass`
|
|
|
|
Connect:
|
|
```bash
|
|
kubectl exec -it mysql-8-0-0 -n sagarpatil-4844 -- mysql -uroot -plocal-root-pass
|
|
```
|
|
|
|
---
|
|
|
|
## Quick Reference
|
|
|
|
| Problem | Root Cause | Fix |
|
|
|---------|-----------|-----|
|
|
| Hades init hanging | Scheduler doesn't use Hades | Disable `hadesContext.enabled` |
|
|
| GET /job returns 500 | POST-only endpoint | Use POST instead |
|
|
| GET /jobType empty | Needs both params | Add `groupName` AND `name` |
|
|
| GET /tags fails | Needs `id` + `groupId` | Add both params |
|
|
| useArgoWorkflows fails | Missing Kafka topic setting | Seed `PROD_KAFKA_TOPIC_VERSION_SETTING` |
|
|
| Wrong topic format | Needs `Map<String, Set<KafkaVersion>>` | Use `{"events":["GCP_KAFKA"]}` |
|
|
| monitoringTags error | Must be stringified JSON | Escape quotes in string |
|