schedulerService-chart/docs/troubleshoot.md

Troubleshoot Guide — Scheduler Service

All real problems faced during deployment and testing at namespace sagarpatil-4844 on os-tf-qa.onlinesales.ai.

---

Problem 1: Hades Context Not Needed

Symptom: Init container setup-hades-context was failing/hanging because Hades service was not available or not needed.

Root Cause: Scheduler-service does NOT use Hades for DB credentials. It connects to MySQL directly via Hibernate XML config (dbConfig.hibernateXml). No code in BaseCommunicator.java calls HadesClient.getContext() — it only calls HibernateDatabase.initHibernateDB(filePath).

Fix:

• Set hadesContext.enabled: false in both values.yaml and values-local.yaml
• Removed Hades from catalog-info.yaml dependency-urls and spec.dependsOn
• Removed hades-service from externalServices in values files

---

Problem 2: GET /job and GET /jobStat Return "Internal Error"

Symptom:

curl .../schedulerService/job?jsonQuery={"id":1}
→ HTTP 500 "Internal Error"

Root Cause: These are POST-only endpoints. The servlet initialization uses null for GET class:

// Job.java
initServletObjectMap(null, JobPostRequest.class, null, JobPostResponse.class);
// JobStat.java
initServletObjectMap(null, JobStatPostRequest.class, ...);

Fix: No fix needed — this is by design. Use POST for /job and /jobStat.

---

Problem 3: GET /jobType Fails Without Both Parameters

Symptom:

curl .../schedulerService/jobType?groupName=os-group-1
→ empty or error response

Root Cause: /jobType GET requires BOTH groupName AND name (or id). A single param is not enough.

Fix:

GET /schedulerService/jobType?groupName=os-group-1&name=Default

---

Problem 4: GET /tags Fails Without Required Parameters

Symptom:

curl .../schedulerService/tags?groupName=os-group-1
→ error response

Root Cause: /tags GET requires id (or name) AND groupId. Just groupName alone won't work.

Fix:

GET /schedulerService/tags?id=1&groupId=1

---

Problem 5: POST /job with useArgoWorkflows:true Fails

Symptom:

{
  "status": "ERROR",
  "message": "Failed to publish event to EventManager for job 2"
}

Root Cause: Event-manager-v2 servlet initialization was crashing at TopicSettingServlet.refreshTopicSettings(). The flow:

1. Event-manager fetches Kafka creds from Hades — {"brokerList":"kafka:9092","isAuthEnabled":false} — OK
2. Event-manager calls property-settings for PROD_KAFKA_TOPIC_VERSION_SETTING with entityType=AGENCY, entityValue=ALL
3. Property-settings returns {"properties":[null]} — no matching property existed in DB

Fix: Insert the seed data into property_settings_db_v2.properties table:

INSERT IGNORE INTO properties (entity_type, entity_value, property_type, property_value, is_active)
VALUES ('AGENCY', 'ALL', 'PROD_KAFKA_TOPIC_VERSION_SETTING', '{"events":["GCP_KAFKA"]}', 1);

Permanent Fix: Updated 002_populate_properties_table.sql in the propertysettings-v2-chart from '{}' to '{"events":["GCP_KAFKA"]}'.

---

Problem 6: Wrong PROD_KAFKA_TOPIC_VERSION_SETTING Format

Symptom:

MismatchedInputException: Cannot deserialize value of type HashSet<KafkaVersion> from Object value

Root Cause: First attempt used wrong JSON format:

{"topicVersionSettings":{"default":{"topic":"events","version":"v1"}}}

The PropertySetttingUtil.java expects Map<String, Set<KafkaVersion>>:

TypeReference<Map<String, Set<KafkaVersion>>> typeRefence = new TypeReference<>(){};

Fix: Correct format is:

{"events":["GCP_KAFKA"]}

Where KafkaVersion enum values are: AWS_KAFKA, GCP_KAFKA. Only GCP_KAFKA is active in DbEvent.java (AWS_KAFKA case is commented out).

---

Problem 7: POST /jobType monitoringTags Format

Symptom:

Marshalling Error when sending monitoringTags as JSON array

Root Cause: monitoringTags field expects a JSON string (stringified JSON), not a raw JSON object/array.

Fix:

{
  "jobType": {
    "monitoringTags": "{\"FAILED_JOBS\": \"sev4\"}"
  }
}

Note the escaped quotes — the value is a string containing JSON, not a nested JSON object.

---

Problem 8: POST /job with useArgoWorkflows:false Works But true Fails

Symptom: Job creation itself works fine (creates job_details, jobs, job_triggers in DB) but the Kafka event publishing step fails.

Root Cause: When useArgoWorkflows: true, scheduler calls ArgoEventSender which POSTs to event-manager-v2. If event-manager's Kafka producer is not initialized (due to Problem 5), the event publish fails.

Fix: Fix Problem 5 first. Once event-manager has the correct PROD_KAFKA_TOPIC_VERSION_SETTING, the full flow works:

POST /job → ArgoEventSender → EventManager → Kafka (GCP_KAFKA)

Scheduler logs on success:

ArgoEventSender: Sending event for job 6 | eventType: argo_ondemand_cluster_1 | eventName: ARGO_JOB_SUBMISSION
ArgoEventSender: Successfully posted event for job: 6

---

Problem 9: MySQL Connection — Finding Root Password

Symptom: Need to connect to mysql-8-0 pod for debugging but don't know the root password.

Fix: Password is stored in Kubernetes secret:

kubectl get secret mysql-8-0 -n sagarpatil-4844 -o jsonpath='{.data.mysql-root-password}' | base64 -d

Result: local-root-pass

Connect:

kubectl exec -it mysql-8-0-0 -n sagarpatil-4844 -- mysql -uroot -plocal-root-pass

---

Quick Reference

Problem	Root Cause	Fix
Hades init hanging	Scheduler doesn't use Hades	Disable hadesContext.enabled
GET /job returns 500	POST-only endpoint	Use POST instead
GET /jobType empty	Needs both params	Add groupName AND name
GET /tags fails	Needs id + groupId	Add both params
useArgoWorkflows fails	Missing Kafka topic setting	Seed PROD_KAFKA_TOPIC_VERSION_SETTING
Wrong topic format	Needs Map<String, Set<KafkaVersion>>	Use {"events":["GCP_KAFKA"]}
monitoringTags error	Must be stringified JSON	Escape quotes in string