Skip to content

Expose OTel thread/process context without requiring profiling to be enabled - #12546

Draft
jandro996 wants to merge 5 commits into
masterfrom
otel-context-without-profiling
Draft

jandro996 wants to merge 5 commits into
masterfrom
otel-context-without-profiling

Conversation

@jandro996

@jandro996 jandro996 commented Sep 17, 2026

Copy link
Copy Markdown
Member

What Does This Do

  • Adds Config.isDatadogProfilerSafeAndConfigured(), exposing the existing raw Datadog-profiler
    env-safety/explicit-flag predicate (native-image/J9/JDK8-aarch64 exclusions included) without the
    isProfilingEnabled() AND-prefix that isDatadogProfilerEnabled() applies.
  • Adds Config.isOtelContextExposureEnabled(): enabled whenever the profiler is safe and configured
    and either profiling is enabled or AppSec's activation level is ProductActivation.FULLY_ENABLED.
    There is no dedicated override flag - the same way isProfilingEnabled() has no per-feature
    override beyond DD_PROFILING_ENABLED, disabling profiling and AppSec (DD_PROFILING_ENABLED,
    DD_APPSEC_ENABLED) is already the kill switch for this feature too.
  • Changes Agent.createProfilingContextIntegration()'s ddprof branch to gate on
    isDatadogProfilerEnabled() || isOtelContextExposureEnabled() (additive - never disables ddprof
    for a user where real profiling already enabled it) instead of solely on
    isDatadogProfilerEnabled().
  • Adds DeferredProfilingContextIntegration, which wraps the real DatadogProfilingIntegration
    behind a no-op delegate until AgentTaskScheduler runs the construction off the premain thread,
    then swaps it in. Constructing the real integration touches java.nio.file (via
    TempLocationManager) and loads the ddprof native library, which must not happen on the
    primordial premain thread. Profiling-enabled users are unaffected - they keep the exact
    synchronous construction they had before, since profiling accuracy needs every scope from the
    first one. The new AppSec-only trigger is the first path that could have reached this
    construction from premain, so it goes through the deferred wrapper instead.
  • Reflectively calls ProcessContext.register(ConfigProvider) from the same branch, so the
    process-wide OTel context descriptor is published even when ProfilingAgent.run() never executes
    (the AppSec-only, profiling-disabled case). initializeAllContext() is idempotent by design
    (verified from the ddprof sources), so no double-invocation guard was added.

Why AppSec-only users need this

Today, exposing the OTel thread-local span context and the process-wide descriptor to external
eBPF/CWS consumers is entirely collapsed into Config.isProfilingEnabled(). AppSec-only
deployments (profiling disabled, AppSec FULLY_ENABLED) never get either signal, even though
neither actually depends on the profiling recording engine running - only on the ddprof native
library being loaded. This change decouples both gates from profiling so an AppSec-only JVM can
be read by the eBPF/CWS consumer, matching cross-tracer precedent (dd-trace-py's decoupled flag,
PHP's plain DD_APPSEC_ENABLED=true trigger).

Motivation

Jira ticket APPSEC-70088.

Additional Notes

  • _dd.profiling.ctx will now appear on AppSec-only users' spans with no real profile behind it -
    this is an accepted, pre-existing side effect of instantiating DatadogProfilingIntegration, not
    a new bug introduced by this change.
  • No new public configuration flag is introduced by this PR: an earlier iteration added
    DD_TRACE_OTEL_CONTEXT_EXPOSURE_ENABLED as an explicit override, but it was dropped in favor of
    pure derivation from the existing DD_PROFILING_ENABLED/DD_APPSEC_ENABLED flags, following the
    same "no dedicated sub-flag" precedent isProfilingEnabled() itself sets.
  • The THREAD_CONTEXT_SHARING system-tests scenario (eBPF/CWS end-to-end validation) requires a
    Linux host with system-probe support and cannot be run from this darwin dev machine - it's a
    manual/CI follow-up, not covered by this PR's test suite.

Contributor Checklist

Jira ticket: APPSEC-70088

Note: Once your PR is ready to merge, add it to the merge queue by commenting /merge. /merge -c cancels the queue request. /merge -f --reason "reason" skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.

- Add Config.isDatadogProfilerSafeAndConfigured() as the raw ddprof
  env-safety/explicit-flag predicate, without the isProfilingEnabled()
  AND-prefix
- Add Config.isOtelContextExposureEnabled(), defaulting to enabled when
  the profiler is safe/configured and either profiling is enabled or
  AppSec is fully enabled, with an explicit
  DD_TRACE_OTEL_CONTEXT_EXPOSURE_ENABLED override
- Gate Agent.createProfilingContextIntegration()'s ddprof branch on the
  new flag (additive, ORed with the existing profiling gate) and
  reflectively register the process context even when profiling never
  starts
- Add TRACE_OTEL_CONTEXT_EXPOSURE_ENABLED to OtlpConfig and
  supported-configurations.json
@jandro996 jandro996 added type: feature Enhancements and improvements comp: config Configuration comp: profiling Profiling labels Sep 17, 2026
@jandro996

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 17, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-18T13:17:54.561970Z 060ff1f Manual request
🔒 Security Review Completed 2026-09-18T13:18:24.412996Z 060ff1f Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 2940f6aa6e

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2940f6aa6e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/Agent.java Outdated
Comment thread dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/Agent.java Outdated
@datadog-official

datadog-official Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
Patch Coverage: 62.16%
Overall Coverage: 59.10% (+0.00%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 060ff1f | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

🔴 Java Benchmark SLOs — Performance SLO breach detected

Suite Status
Startup 🔴 breach

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.81 s 14.59 s [+0.7%; +2.2%] (maybe worse)
startup:insecure-bank:tracing:Agent 13.66 s 13.69 s [-1.0%; +0.5%] (no difference)
startup:petclinic:appsec:Agent 17.06 s 16.93 s [+0.0%; +1.5%] (maybe worse)
startup:petclinic:iast:Agent 16.89 s 16.97 s [-1.4%; +0.4%] (no difference)
startup:petclinic:profiling:Agent 16.74 s 16.76 s [-1.0%; +0.8%] (no difference)
startup:petclinic:sca:Agent 16.96 s 16.71 s [+0.8%; +2.3%] (maybe worse)
startup:petclinic:tracing:Agent 16.13 s 15.64 s [-1.1%; +7.4%] (no difference)

Commit: 060ff1f2 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

…-only trigger

Constructing DatadogProfilingIntegration touches java.nio.file (via
TempLocationManager) and loads the ddprof native library, which must not
happen on the primordial premain thread. Users with the Datadog profiler
enabled were unaffected (they already ran this synchronously), but the
new AppSec-only trigger reached this construction from premain for the
first time.

DeferredProfilingContextIntegration wraps the real integration behind a
NoOp delegate until AgentTaskScheduler runs the deferred construction off
the premain thread, then swaps it in. The profiler-enabled path keeps the
exact synchronous behavior it had before, since profiling accuracy needs
every scope from the first one.

Addresses a P1 finding from the Codex review on this PR.
…profiling/AppSec

isOtelContextExposureEnabled() no longer has its own explicit override. It mirrors
isProfilingEnabled(), which has no dedicated sub-flag either: the kill switch is
disabling DD_PROFILING_ENABLED and DD_APPSEC_ENABLED, the same flags that already
drive the derivation. This removes the DD_TRACE_OTEL_CONTEXT_EXPOSURE_ENABLED public
config entirely, along with its metadata/supported-configurations.json entry - so
there is no new config requiring Feature Parity Dashboard registration, which was
causing the config-inversion-local-validation.py CI job to fail.
- Add OtelContextExposureSmokeTest verifying OTel process context registration follows AppSec activation, not profiling
- Extend DeferredProfilingContextIntegrationTest to cover all delegate pass-through methods (onAttach/onDetach/encodeOperationName/encodeResourceName/onRootSpanFinished), not just newScopeState/name
decisions.md is a session-local planning artifact and should not ship as part of
the PR diff; it is now added to the global gitignore alongside progress.md/task_plan.md.
@jandro996 jandro996 added the tag: override groovy enforcement Override the "Enforce Groovy Migration" check label Sep 18, 2026
@jandro996

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 060ff1f2d6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 060ff1f2d6

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@jandro996

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: 060ff1f2d6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 060ff1f2d6

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@jandro996

jandro996 commented Sep 18, 2026

Copy link
Copy Markdown
Member Author

Looked into the three "maybe worse" startup scenarios:

  • petclinic:appsec: expected. isOtelContextExposureEnabled() (Config.java) now returns true when AppSec is FULLY_ENABLED even with profiling off, so this scenario reaches the ddprof branch in Agent.createProfilingContextIntegration() for the first time. The extra synchronous work before the deferred construction hands off to AgentTaskScheduler (classloader lookups, task submission) accounts for the small cost.
  • petclinic:sca: same root cause, not something new to this scenario specifically. Per the benchmark config (apm-sdks-benchmarks/.gitlab/ci-java-startup-parallel.yml), the SCA instance launches with -Ddd.appsec.enabled=true -Ddd.appsec.sca.enabled=true, i.e. AppSec is fully enabled for this scenario too, so it hits the same new branch as petclinic:appsec.
  • insecure-bank:iast: looks like a false positive. That instance only sets -Ddd.iast.enabled=true, with no -Ddd.appsec.enabled, so getAppSecActivation() stays ENABLED_INACTIVE and isOtelContextExposureEnabled() is false: this PR's diff is an unreached branch there, behavior identical to master. This is corroborated by petclinic:iast, which has the exact same gate condition and is reported as "no difference" in the same run.

Given appsec/sca are the intended new trigger for OTel context exposure, this small startup cost is accepted as part of APPSEC-70088's scope. Happy to look further if this reproduces as a bigger regression on a re-run.

Note: If reviewers agree with this change before merging, I'll open a PR against DataDog/apm-sdks-benchmarks to bump the SLO threshold accordingly, then re-run the benchmarks for this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: config Configuration comp: profiling Profiling tag: override groovy enforcement Override the "Enforce Groovy Migration" check type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant