# Troubleshoot Guide — Scheduler Service

All real problems faced during deployment and testing at namespace `sagarpatil-4844` on `os-tf-qa.onlinesales.ai`.

---

## Problem 1: Hades Context Not Needed

**Symptom:** Init container `setup-hades-context` was failing/hanging because Hades service was not available or not needed.

**Root Cause:** Scheduler-service does NOT use Hades for DB credentials. It connects to MySQL directly via Hibernate XML config (`dbConfig.hibernateXml`). No code in `BaseCommunicator.java` calls `HadesClient.getContext()` — it only calls `HibernateDatabase.initHibernateDB(filePath)`.

**Fix:**
- Set `hadesContext.enabled: false` in both `values.yaml` and `values-local.yaml`
- Removed Hades from `catalog-info.yaml` dependency-urls and `spec.dependsOn`
- Removed hades-service from `externalServices` in values files

---

## Problem 2: GET /job and GET /jobStat Return "Internal Error"

**Symptom:**
```
curl .../schedulerService/job?jsonQuery={"id":1}
→ HTTP 500 "Internal Error"
```

**Root Cause:** These are POST-only endpoints. The servlet initialization uses `null` for GET class:
```java
// Job.java
initServletObjectMap(null, JobPostRequest.class, null, JobPostResponse.class);
// JobStat.java
initServletObjectMap(null, JobStatPostRequest.class, ...);
```

**Fix:** No fix needed — this is by design. Use POST for `/job` and `/jobStat`.

---

## Problem 3: GET /jobType Fails Without Both Parameters

**Symptom:**
```
curl .../schedulerService/jobType?groupName=os-group-1
→ empty or error response
```

**Root Cause:** `/jobType` GET requires BOTH `groupName` AND `name` (or `id`). A single param is not enough.

**Fix:**
```
GET /schedulerService/jobType?groupName=os-group-1&name=Default
```

---

## Problem 4: GET /tags Fails Without Required Parameters

**Symptom:**
```
curl .../schedulerService/tags?groupName=os-group-1
→ error response
```

**Root Cause:** `/tags` GET requires `id` (or `name`) AND `groupId`. Just `groupName` alone won't work.

**Fix:**
```
GET /schedulerService/tags?id=1&groupId=1
```

---

## Problem 5: POST /job with useArgoWorkflows:true Fails

**Symptom:**
```json
{
  "status": "ERROR",
  "message": "Failed to publish event to EventManager for job 2"
}
```

**Root Cause:** Event-manager-v2 servlet initialization was crashing at `TopicSettingServlet.refreshTopicSettings()`. The flow:
1. Event-manager fetches Kafka creds from Hades — `{"brokerList":"kafka:9092","isAuthEnabled":false}` — OK
2. Event-manager calls property-settings for `PROD_KAFKA_TOPIC_VERSION_SETTING` with `entityType=AGENCY, entityValue=ALL`
3. Property-settings returns `{"properties":[null]}` — no matching property existed in DB

**Fix:** Insert the seed data into `property_settings_db_v2.properties` table:
```sql
INSERT IGNORE INTO properties (entity_type, entity_value, property_type, property_value, is_active)
VALUES ('AGENCY', 'ALL', 'PROD_KAFKA_TOPIC_VERSION_SETTING', '{"events":["GCP_KAFKA"]}', 1);
```

**Permanent Fix:** Updated `002_populate_properties_table.sql` in the `propertysettings-v2-chart` from `'{}'` to `'{"events":["GCP_KAFKA"]}'`.

---

## Problem 6: Wrong PROD_KAFKA_TOPIC_VERSION_SETTING Format

**Symptom:**
```
MismatchedInputException: Cannot deserialize value of type HashSet<KafkaVersion> from Object value
```

**Root Cause:** First attempt used wrong JSON format:
```json
{"topicVersionSettings":{"default":{"topic":"events","version":"v1"}}}
```

The `PropertySetttingUtil.java` expects `Map<String, Set<KafkaVersion>>`:
```java
TypeReference<Map<String, Set<KafkaVersion>>> typeRefence = new TypeReference<>(){};
```

**Fix:** Correct format is:
```json
{"events":["GCP_KAFKA"]}
```

Where `KafkaVersion` enum values are: `AWS_KAFKA`, `GCP_KAFKA`. Only `GCP_KAFKA` is active in `DbEvent.java` (AWS_KAFKA case is commented out).

---

## Problem 7: POST /jobType monitoringTags Format

**Symptom:**
```
Marshalling Error when sending monitoringTags as JSON array
```

**Root Cause:** `monitoringTags` field expects a JSON **string** (stringified JSON), not a raw JSON object/array.

**Fix:**
```json
{
  "jobType": {
    "monitoringTags": "{\"FAILED_JOBS\": \"sev4\"}"
  }
}
```
Note the escaped quotes — the value is a string containing JSON, not a nested JSON object.

---

## Problem 8: POST /job with useArgoWorkflows:false Works But true Fails

**Symptom:** Job creation itself works fine (creates `job_details`, `jobs`, `job_triggers` in DB) but the Kafka event publishing step fails.

**Root Cause:** When `useArgoWorkflows: true`, scheduler calls `ArgoEventSender` which POSTs to event-manager-v2. If event-manager's Kafka producer is not initialized (due to Problem 5), the event publish fails.

**Fix:** Fix Problem 5 first. Once event-manager has the correct `PROD_KAFKA_TOPIC_VERSION_SETTING`, the full flow works:
```
POST /job → ArgoEventSender → EventManager → Kafka (GCP_KAFKA)
```

Scheduler logs on success:
```
ArgoEventSender: Sending event for job 6 | eventType: argo_ondemand_cluster_1 | eventName: ARGO_JOB_SUBMISSION
ArgoEventSender: Successfully posted event for job: 6
```

---

## Problem 9: MySQL Connection — Finding Root Password

**Symptom:** Need to connect to `mysql-8-0` pod for debugging but don't know the root password.

**Fix:** Password is stored in Kubernetes secret:
```bash
kubectl get secret mysql-8-0 -n sagarpatil-4844 -o jsonpath='{.data.mysql-root-password}' | base64 -d
```
Result: `local-root-pass`

Connect:
```bash
kubectl exec -it mysql-8-0-0 -n sagarpatil-4844 -- mysql -uroot -plocal-root-pass
```

---

## Quick Reference

| Problem | Root Cause | Fix |
|---------|-----------|-----|
| Hades init hanging | Scheduler doesn't use Hades | Disable `hadesContext.enabled` |
| GET /job returns 500 | POST-only endpoint | Use POST instead |
| GET /jobType empty | Needs both params | Add `groupName` AND `name` |
| GET /tags fails | Needs `id` + `groupId` | Add both params |
| useArgoWorkflows fails | Missing Kafka topic setting | Seed `PROD_KAFKA_TOPIC_VERSION_SETTING` |
| Wrong topic format | Needs `Map<String, Set<KafkaVersion>>` | Use `{"events":["GCP_KAFKA"]}` |
| monitoringTags error | Must be stringified JSON | Escape quotes in string |