Skip to content

Decouple Kafka DSM instrumentation from APM tracing enablement - #12350

Open
ericfirth wants to merge 32 commits into
masterfrom
worktree-dsm-kafka-decoupling
Open

ericfirth wants to merge 32 commits into
masterfrom
worktree-dsm-kafka-decoupling

Conversation

@ericfirth

@ericfirth ericfirth commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Switches Kafka clients (0.11, 3.8) and kafka-streams-0.11 instrumentation modules to InstrumenterModule.DataStreams, so DSM checkpoints are created even when APM tracing/integrations are disabled.
  • Forces DSM-only produce/consume/poll spans to PrioritySampling.USER_DROP via SamplingMechanism.DATA_STREAMS when they are the local root span, so they aren't billed as APM — while never dropping an ambient customer trace.
  • Dedupes each KafkaDecorator's hardcoded integration-name list into shared constants reused by TRACING_ENABLED and every module constructor, delegates Config.isDataStreamsEnabled() to InstrumenterConfig, extracts a duplicated DSM-drop guard in KafkaStreamTaskInstrumentation into a helper, and rewrites InstrumenterModuleTest to use @WithConfig instead of raw reflection.

Motivation

Using dd-trace-rb with JRuby doesn't support DSM. This explores whether Java's DSM instrumentation can run standalone (independent of APM tracing) as a path back to DSM support when tracing is handled by a different tracer, such as dd-trace-rb

Jira ticket: [DSM2-278]

Test plan

  • ./gradlew :dd-java-agent:instrumentation:kafka-clients-0.11:test
  • ./gradlew :dd-java-agent:instrumentation:kafka-clients-3.8:test
  • ./gradlew :dd-java-agent:instrumentation:kafka-streams-0.11:test
  • ./gradlew :internal-api:test
  • ./gradlew :dd-java-agent:agent-tooling:test
  • Manual verification against a real Java Kafka client test app with DSM enabled / APM tracing disabled
  • Techdebt and perf review passes run over the branch; findings fixed

🤖 Generated with Claude Code

@ericfirth ericfirth added type: feature Enhancements and improvements inst: kafka Kafka instrumentation comp: data streams Data Streams Monitoring tag: ai generated Largely based on code generated by an AI or LLM labels Aug 31, 2026
@datadog-datadog-prod-us1

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.83 s 14.73 s [-0.5%; +1.8%] (no difference)
startup:insecure-bank:tracing:Agent 13.68 s 13.71 s [-1.1%; +0.5%] (no difference)
startup:petclinic:appsec:Agent 17.64 s 17.32 s [+0.9%; +2.7%] (maybe worse)
startup:petclinic:iast:Agent 17.57 s 17.64 s [-1.5%; +0.7%] (no difference)
startup:petclinic:profiling:Agent 16.66 s 17.42 s [-8.4%; -0.3%] (maybe better)
startup:petclinic:sca:Agent 17.59 s 17.51 s [-0.6%; +1.4%] (no difference)
startup:petclinic:tracing:Agent 16.58 s 16.29 s [-2.5%; +6.1%] (no difference)

Commit: be3a989d · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@pr-commenter

pr-commenter Bot commented Aug 31, 2026

Copy link
Copy Markdown

Kafka / consumer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master worktree-dsm-kafka-decoupling
git_commit_date 1789566771 1789673168
git_commit_sha c595628 be3a989
See matching parameters
Baseline Candidate
ci_job_date 1789674756 1789674756
ci_job_id 2055294161 2055294161
ci_pipeline_id 138296209 138296209
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion 11.0.31 11.0.31
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=15002 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=15002 -Dhttp.nonProxyHosts=localhost *.localhost
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-sinxfg73 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-sinxfg73 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results
scenario Δ mean throughput
scenario:not-instrumented/KafkaConsumerBenchmark.benchConsume same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaConsumerBenchmark.benchConsume same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaConsumerBenchmark.benchConsume unsure
[+767.743op/s; +9862.954op/s] or [+0.415%; +5.332%]

Comment thread internal-api/src/main/java/datadog/trace/api/sampling/SamplingMechanism.java Outdated
Comment thread internal-api/src/main/java/datadog/trace/api/InstrumenterConfig.java Outdated
@dougqh

dougqh commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Heads up: the TRACING_CONFIG_NONDEFAULT system-tests scenario (main / End-to-end #8 / spring-boot 8) is currently failing because of this PR's core behavior change:

FAILED tests/test_config_consistency.py::Test_Config_IntegrationEnabled_False::test_integration_enabled_false
AssertionError: kafka.produce span was found in trace: [...]

That test disables the Kafka integration and asserts no kafka.produce span should reach the agent at all — the pre-PR contract. With this PR, when DSM is enabled, a kafka.produce/kafka.consume span is still created and sent (now carrying sampling_priority=USER_DROP) even though Kafka tracing itself is disabled — that's the intended decoupling. The scenario has DSM traffic (api_v2_data_streams_messages.json present in its logs), so this is a genuine interaction, not a flake.

This will need a system-tests-side update before merge: either exempt/adjust Test_Config_IntegrationEnabled_False for the DSM-enabled case, or have the scenario disable DSM alongside the integration so the old assertion still holds where DSM isn't in play. Worth confirming with whoever owns that shared parametric test (it's cross-language) before changing its semantics unilaterally.

@pr-commenter

pr-commenter Bot commented Sep 4, 2026

Copy link
Copy Markdown

Kafka / producer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master worktree-dsm-kafka-decoupling
git_commit_date 1789566771 1789673168
git_commit_sha c595628 be3a989
See matching parameters
Baseline Candidate
ci_job_date 1789674712 1789674712
ci_job_id 2055294159 2055294159
ci_pipeline_id 138296209 138296209
cpu_model Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
jdkVersion 11.0.31 11.0.31
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=15002 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=15002 -Dhttp.nonProxyHosts=localhost *.localhost
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-nsqnza35 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-nsqnza35 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results
scenario Δ mean throughput
scenario:not-instrumented/KafkaProduceBenchmark.benchProduce same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaProduceBenchmark.benchProduce unsure
[-7302.610op/s; -268.767op/s] or [-4.171%; -0.154%]
scenario:only-tracing-dsm-enabled-benchmarks/KafkaProduceBenchmark.benchProduce unsure
[-4576.868op/s; -948.949op/s] or [-3.092%; -0.641%]

ericfirth and others added 12 commits September 9, 2026 09:49
Implements the pre-instrumentation config accessor for the data streams feature flag,
mirroring the existing usmEnabled pattern. Reads the same data.streams.enabled key
that Config.isDataStreamsEnabled() uses, enabling DSM to work independently of the
general DD_INTEGRATIONS_ENABLED APM gate.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ledOverridesFalse

Save and restore the original dataStreamsEnabled value instead of
hardcoding false in the finally block. This prevents test pollution
where an incorrect state could leak to the next test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add DATA_STREAMS sampling mechanism (byte value 14) to support
decoupling Kafka DSM from APM tracing. This mechanism enforces
USER_DROP priority for DSM-only spans and allows overriding locked
priority when DSM is enabled.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Use qualified PrioritySampling.USER_DROP form in validateWithSamplingPriority
to match the DATA_JOBS case style (line 70) which uses qualified form.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…reams base class

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
When Kafka tracing is disabled but Data Streams Monitoring is enabled,
produce-path spans created without a propagated trace context should be
marked with PrioritySampling.USER_DROP to avoid billing as APM spans.
This implements the 'local-root spans only' scoping from the spec: spans
with an extracted parent context (genuine distributed traces) are left
completely untouched.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…eams base class

Change extends InstrumenterModule.Tracing to extends InstrumenterModule.DataStreams
for all 8 instrumentation classes in the kafka-clients-3.8 module. This decouples
Data Streams Monitoring from APM tracing and prepares for billing-suppression logic.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…me paths)

Add TRACING_ENABLED constant and billing-suppression logic for produce and
consume paths to suppress unnecessary billing when APM tracing is disabled but
DSM is enabled. The suppression only applies to local-root spans (no extracted
parent context) per the DSM-Kafka decoupling plan.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ericfirth and others added 2 commits September 9, 2026 09:50
Ensures the cross-language contract holds: when APM tracing is disabled
and DSM is enabled, no span is ever written to the agent for Kafka
instrumentations. Replaces AgentTracer.startSpan with a lightweight
Utils.newPathwayOnlySpan shim across kafka-clients-0.11, kafka-clients-3.8,
and kafka-streams-0.11, and extracts shared DSM checkpoint/inject and
transaction-tracking logic into helper methods to remove duplication.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… mechanism

InstrumenterConfig.java unconditionally disabled dataStreamsEnabled under
GraalVM native-image, unlike its pre-PR behavior; DSM has no native-image
restriction, so its read is moved outside that carve-out to match
dataJobsEnabled.

SamplingMechanism.DATA_STREAMS forced a USER_DROP sampling priority to keep
DSM-only Kafka spans from being billed as APM traces. That approach is now
fully superseded by Utils.newPathwayOnlySpan, which never creates a real
span for the DSM-only case, so DATA_STREAMS had zero remaining production
callers and is removed as dead code.

Also collapses the near-identical TRACING_ENABLED computation duplicated
across the three Kafka decorators (kafka-clients-0.11, kafka-clients-3.8,
kafka-streams-0.11) into a single Utils.isTracingEnabled helper in
kafka-common, per techdebt review.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@ericfirth
ericfirth force-pushed the worktree-dsm-kafka-decoupling branch from bb5d6cf to c498c71 Compare September 9, 2026 14:37
ericfirth and others added 2 commits September 9, 2026 11:20
Several kafka-clients instrumentations referenced KafkaDecorator (whose
TRACING_ENABLED field now calls Utils.isTracingEnabled) without declaring
kafka_common.Utils as a helper class, causing muzzle to fail resolving
that dependency.

Separately, kafka-streams' StartSpanAdvice/StartSpanAdvice27 advice
classes contained their own private static helper methods, which muzzle
cannot validate since advice classes aren't part of the checked helper
set. Moved that logic into KafkaStreamsDecorator, which is already a
registered helper class.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@dougqh

dougqh commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Claude (re-review update): Checked back in on this since my 08-31 review — the three substantive findings from back then are all resolved via a genuine redesign rather than a patch:

  • Native-image regression (InstrumenterConfig.isDataStreamsEnabled()): now read unconditionally outside the Platform.isNativeImageBuilder() carve-out, with a comment explaining why DSM doesn't need that restriction.
  • Hardcoded USER_DROP guard duplicated 4x: replaced by shared TRACING_ENABLED/INTEGRATION_NAME/LEGACY_INTEGRATION_NAME constants each instrumentation's constructor now reuses.
  • TracingIterator / static-vs-dynamic isDataStreamsEnabled() mismatch: moot — the DSM-only path no longer creates a real span or forces sampling priority at all. It now uses a newPathwayOnlySpan shim that only carries the pathway context, so it doesn't route through CoreSpanBuilder.startSpan (and my original propagation-behavior concern with it) anymore.

Two other things worth noting for reviewers:

  • "Enforce Groovy migration" check is failing on the 3 new .groovy files, but all of them extend InstrumentationSpecification (genuine instrumented tests against real embedded Kafka) — the sanctioned Groovy/Spock exception per this repo's testing conventions. This looks like a false positive from the blanket new-file check rather than a real issue; the tag: override groovy enforcement label should clear it.
  • The TRACING_CONFIG_NONDEFAULT system-tests scenario I flagged on 08-31 has been green since the 09-09 14:37 system-tests run — that's fixed. A separate play 5 end-to-end failure showed up in the 09-09 15:21 run; unclear if it's related to the muzzle-validation fix pushed right before it or just transient — worth a glance if it recurs.

Scope has grown since my last pass to also cover kafka-streams-1.0 and kafka-connect-0.11.

ericfirth and others added 2 commits September 16, 2026 11:39
BC_IMPOSSIBLE_INSTANCEOF flags the Object-cast instanceof check used to
detect RecordMetadata support (only available from kafka-streams 2.7+),
which SpotBugs can't reason about statically but is always true at
runtime given this class's version constraint.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…-trace-java into worktree-dsm-kafka-decoupling
@ericfirth ericfirth added the tag: override groovy enforcement Override the "Enforce Groovy Migration" check label Sep 16, 2026
@ericfirth
ericfirth marked this pull request as ready for review September 17, 2026 17:00
@ericfirth
ericfirth requested review from a team as code owners September 17, 2026 17:00
@ericfirth
ericfirth requested review from mcculls, sarahchen6 and vandonr and removed request for a team September 17, 2026 17:00
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 17, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-17T17:10:42.536272Z 820ab35 Draft marked ready
🔒 Security Review Completed 2026-09-17T17:07:34.048320Z 820ab35 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadog-datadog-prod-us1 datadog-datadog-prod-us1 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: FAIL

The two most critical issues are: (1) AgentInstaller never installs CoreTracer when tracing is disabled but DSM is enabled, so every DSM checkpoint is silently dropped via the no-op implementation; and (2) the DSM-only span/carrier zeroes out trace and span IDs, severing continuity for any active customer trace and its producer callbacks.

Open Bits AI session

🤖 Datadog Autotest · Commit 820ab35 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 820ab35e6d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Several branch-selection checks used the dynamic, remote-config-aware
traceConfig().isDataStreamsEnabled() alongside the static TRACING_ENABLED
flag when deciding whether to create a real APM span vs. a DSM-only
pathway shim. That let toggling DSM off at runtime silently re-enable
real Kafka spans that should stay disabled. Also fixes producer
callbacks losing the active trace, and restores strict span-write
ordering between broker/queue spans and their consume-span children.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…tale span

If an exception was thrown between startTracedConsumeSpan setting
pendingQueueSpanToFinish and it being finished/nulled in
startNewRecordSpan, the field stayed set and a later, unrelated call
would finish that stale queue span at the wrong time.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

@dougqh dougqh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, I think this looks like a good start at decoupled DSM & tracing.

I have once concern around clearing pendingSpan on an exception in the TracingIterators.

@mcculls

mcculls commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

This is exploratory — tracked in DSM2-278. It may or may not be accepted upstream.

@ericfirth is this still exploratory, or is this something you'd like to integrate before the next release?

@ericfirth

ericfirth commented Sep 18, 2026

Copy link
Copy Markdown
Contributor Author

This is exploratory — tracked in DSM2-278. It may or may not be accepted upstream.

@ericfirth is this still exploratory, or is this something you'd like to integrate before the next release?

@mcculls No, not really exploratory anymore. We'd like to merge it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: data streams Data Streams Monitoring inst: kafka Kafka instrumentation tag: ai generated Largely based on code generated by an AI or LLM tag: override groovy enforcement Override the "Enforce Groovy Migration" check type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants