Skip to content

fix: create jaeger ExternalName services after app deploy in conductor - #663

Draft
somyas25 wants to merge 3 commits into
SREGym:mainfrom
somyas25:jaeger-issue-somyas3
Draft

fix: create jaeger ExternalName services after app deploy in conductor#663
somyas25 wants to merge 3 commits into
SREGym:mainfrom
somyas25:jaeger-issue-somyas3

Conversation

@somyas25

Copy link
Copy Markdown

Resolution to the hotfix (#541) for train-ticket. This PR resolves creation of Jaeger ExternalName services before deploying train-ticket.

@yimingsu01
yimingsu01 requested a review from HacksonClark March 20, 2026 00:31

@HacksonClark HacksonClark left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jaeger is not working in train-ticket:

2026-03-24 00:02:44,705 - mcp.server.lowlevel.server - INFO - Processing request of type CallToolRequest                                 │
│ 2026-03-24 00:02:44,710 - all.mcp.utils - ERROR - Request failed: 400 Client Error: Bad Request for url: http://jaeger-out.observe.svc.c │
│ luster.local:16686/api/traces?service=&start=1774310564706000&end=1774310564706034&limit=20                                              │
│ 2026-03-24 00:02:44,710 - all.mcp.jaeger_server - ERROR - [ob_mcp] Error querying get_traces: 400 Client Error: Bad Request for url: htt │
│ p://jaeger-out.observe.svc.cluster.local:16686/api/traces?service=&start=1774310564706000&end=1774310564706034&limit=20                  │
│                                                                                                                         

Make sure to test Jaeger with MCP inspector to make sure the tool works!

@somyas25

Copy link
Copy Markdown
Author

Jaeger is not working in train-ticket:

2026-03-24 00:02:44,705 - mcp.server.lowlevel.server - INFO - Processing request of type CallToolRequest                                 │
│ 2026-03-24 00:02:44,710 - all.mcp.utils - ERROR - Request failed: 400 Client Error: Bad Request for url: http://jaeger-out.observe.svc.c │
│ luster.local:16686/api/traces?service=&start=1774310564706000&end=1774310564706034&limit=20                                              │
│ 2026-03-24 00:02:44,710 - all.mcp.jaeger_server - ERROR - [ob_mcp] Error querying get_traces: 400 Client Error: Bad Request for url: htt │
│ p://jaeger-out.observe.svc.cluster.local:16686/api/traces?service=&start=1774310564706000&end=1774310564706034&limit=20                  │
│                                                                                                                         

Make sure to test Jaeger with MCP inspector to make sure the tool works!

I am trying to recreate this locally and when running the problem 17 and using the MCP inspector to get services for jaeger I only get 'jaeger-all-in-one' service. When trying to get the trace for this service, I get a success response but an empty output. Could you give me more insight on how you got this error so I can analyze it? Thanks!

@HacksonClark

Copy link
Copy Markdown
Member

Hmm, that means that there are not traces being collected! @Saadmrp1038 Didn't we delete the jaeger images from trainticket since they were causing some bugs? I think that may be the source of this issue.

@Saadmrp1038

Saadmrp1038 commented Mar 26, 2026

Copy link
Copy Markdown
Collaborator

@HacksonClark yes, the current train-ticket images are not jaeger instrumented. The best way is to instrument the services ourselves as the previous jaeger-based images had some issues.

@HacksonClark

Copy link
Copy Markdown
Member

Gotcha! I'll have to re-review this PR then.

@yimingsu01

Copy link
Copy Markdown
Collaborator

To update: I briefly chatted with @somyas25 in person. First @somyas25: Thanks for keeping pushing! This is not an easy first issue at all.

I explained what does tracing mean to @somyas25, and now she will try to instrument the services so that Jaeger can trace them.

I will thus convert this PR to a draft.

@yimingsu01
yimingsu01 marked this pull request as draft March 29, 2026 03:04
@HacksonClark

Copy link
Copy Markdown
Member

@yimingsu01 sounds great! This is a great task to learn about distributed tracing, a very in-demand skill. Great work so far @somyas25! Keep it up!

@somyas25

Copy link
Copy Markdown
Author

Hi, I’ve been looking into this issue and trying to understand the train-ticket deployment flow a little better.

From what I understand right now, the conductor triggers the train-ticket-deploy-job, which runs the image from values.yaml. That image then runs deploy.sh, and deploy.sh uses the quickstart-k8s path to deploy everything.

What I’m noticing is that quickstart-k8s does not seem to have Jaeger tracing set up, while the k8s-with-jaeger path does. From what I saw, k8s-with-jaeger uses the *-with-jaeger images and sets JAEGER_AGENT_HOST=jaeger, so it seems like that is the tracing-enabled path.

Based on what @Saadmrp1038 said I am assuming that there are some bugs in the images in k8s-with-jaeger so my plan was to fix those and switch deployment back to k8s-with-jaeger. I wanted to confirm if I should:

  • keep the original k8s-with-jaeger approach that uses the tracing-enabled images, or
  • try to move k8s-with-jaeger toward the Helm and YAML style used by quickstart-k8s

Just wanted to check that I’m understanding this correctly and that I am on the right track!

@Saadmrp1038

Saadmrp1038 commented Apr 15, 2026

Copy link
Copy Markdown
Collaborator

@somyas25 Your observations are correct. For your question, I like the quickstart-k8s style more. I'm not sure about using the existing k8s-with-jaeger images though. Those images (codewisdom/*-with-jaeger:v1) are third-party pre-built images from the original upstream project. We don't have the source code that was used to build them, so getting them to work the way we want is difficult. And since we can't see or modify the code, we're stuck if we need to make code changes.

I was thinking we should add Jaeger tracing instrumentation to our existing app code and build our own Jaeger-enabled images. That way we have full control over the tracing setup and can update services whenever we need to.

In practice that would look like:

  1. Add a Jaeger tracing dependency (like opentracing-spring-jaeger-cloud-starter). Spring Boot has pretty good auto instrumentation so it should automatically trace the request components without code changes.
  2. Build the new images
  3. Use the new images and follow the quickstart-k8s Helm/YAML structure

Feel free to ask about anything! train-ticket is quite a large app and not easy to work on.

@somyas25

Copy link
Copy Markdown
Author

@somyas25 Your observations are correct. For your question, I like the quickstart-k8s style more. I'm not sure about using the existing k8s-with-jaeger images though. Those images (codewisdom/*-with-jaeger:v1) are third-party pre-built images from the original upstream project. We don't have the source code that was used to build them, so getting them to work the way we want is difficult. And since we can't see or modify the code, we're stuck if we need to make code changes.

I was thinking we should add Jaeger tracing instrumentation to our existing app code and build our own Jaeger-enabled images. That way we have full control over the tracing setup and can update services whenever we need to.

In practice that would look like:

  1. Add a Jaeger tracing dependency (like opentracing-spring-jaeger-cloud-starter). Spring Boot has pretty good auto instrumentation so it should automatically trace the request components without code changes.
  2. Build the new images
  3. Use the new images and follow the quickstart-k8s Helm/YAML structure

Feel free to ask about anything! train-ticket is quite a large app and not easy to work on.

Understood, thanks. I will start implementing this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants