deploy: drop the Go toolchain from the runtime image by gz · Pull Request #6822 · feldera/feldera

gz · 2026-08-06T16:02:37Z

Remove the Go toolchain from the runtime image. Nothing the image builds has needed it since January.

Why

Go is a build dependency of aws-lc-fips-sys, which only enters the dependency graph behind the fips cargo feature. The runtime image compiles pipelines, and the workspace MultiCratesWriter generates never enables that feature, so pipelines link the default provider and Go sits unused.

FIPS was on by default for nine days in January; #5488 reverted it to opt-in because the AWS-LC source build fails on GCC >= 14, which breaks recent Fedora. The toolchain has been carried ever since: 258 MB extracted, installed from an upstream tarball rather than apt because Ubuntu's golang-go ships CVEs that fail our image scan, and bumped three times (May, June, August) to chase scanner findings.

deploy/build.Dockerfile keeps golang-go: CI still builds the binaries with --features fips, which compiles aws-lc-fips-sys from source. That is unchanged.

Describe Manual Test Plan

Built --target base and confirmed command -v go finds nothing.
Inside that image, cargo build -p dbsp_adapters succeeds (4m04s), which is the path a runtime pipeline compile takes.
cargo tree -i aws-lc-fips-sys reports the package is absent from the default graph; it appears only with --features fips.
Confirmed against the aws-lc-sys README that the non-FIPS provider needs no Go: "Go and Perl aren't absolutely necessary for aws-lc-sys, as AWS-LC provides generated build files."
The image build itself is the regression guard: the release stage runs --precompile, which compiles a pipeline inside the image and fails the build if a toolchain is missing.

Checklist

Unit tests added/updated
Documentation updated
Integration tests added/updated
Changelog updated

No unit test applies to a Dockerfile; the precompile step in the image build covers it.

Breaking Changes?

No behavior change. Pipelines already compiled without FIPS; this only stops shipping a toolchain none of them invoked.

Follow-up

Making the data plane actually FIPS-compliant is a separate piece of work, because Kafka (rdkafka-sys -> librdkafka), Postgres TLS (postgres-openssl), and our own 33 openssl:: call sites all use OpenSSL, outside any validated boundary. Routing those through AWS-LC-FIPS is supported upstream (openssl-sys and openssl both expose an aws-lc-fips feature, and rdkafka-sys picks up DEP_OPENSSL_ROOT), and it leaves a single crypto library in the binary. That is being developed separately.

mythical-fred

Solid infrastructure win. Removing 258 MB of Go from the runtime image while closing the control-plane/data-plane FIPS gap is unambiguously the right trade, and the switch design keeps a plain cargo build non-FIPS so Fedora / GCC >= 14 contributors stay unaffected.

What I liked:

One AWS_LC_FIPS_COMMIT pin shared between deploy/Dockerfile and deploy/build.Dockerfile, with the module treated explicitly as a compliance decision in the note to reviewers rather than a routine dep bump.
AWS_LC_FIPS_SYS_USE_SYSTEM=1 making a missing/mismatched module a hard build failure rather than a silent fallback that would rebuild AWS-LC from source. The negative-control verification you called out in the test plan is exactly the guarantee that matters.
--fips flowing manager -> SQL compiler -> generated Cargo.toml so the data plane's crypto provider follows the control plane's, plus AWS_LC_FIPS_SYS_* explicitly whitelisted in is_preserved_compilation_env_var (rather than widening the filter). The added unit test locks that in.
Static (BUILD_SHARED_LIBS=0) module + prebuilt bindings, so downstream doesn't need a runtime library search path and doesn't need to re-run bindgen.
aws-lc-fips-${FIPS} two-stage alias is a clean way to make FIPS=0 a no-op copy without conditional COPY.
Rollout instructions in deploy/README.md replaced with a grep that produces the list, instead of a hand-maintained (and already stale) file listing.

One tiny nit, non-blocking: aws-lc-fips-0 pulls the full ubuntu:24.04 layer just so COPY --from= has a directory to read; FROM busybox AS aws-lc-fips-0 (or scratch + a COPY of an empty tree) would shave a bit off the FIPS=0 build path without changing anything else. Not worth blocking on.

Approving.

mythical-fred

Rebase + two content additions since my earlier approve; still LGTM.

New material worth calling out:

install-aws-lc-fips.sh renames lib{crypto,ssl}.{a,so,dylib} to lib{crypto,ssl}-awslc.* after install. Correct fix: aws-lc-fips-sys prepends the install's lib/ to the link search path, and any bare -lcrypto / -lssl that openssl-sys emits from an unrelated crate would resolve against the AWS-LC static lib, which lacks the OpenSSL 3 symbols those callers need. The cohabiting-names dance keeps only opt-in consumers on AWS-LC. Comment in the script explains it clearly.
CI workflows switched from ghcr.io/feldera/feldera-dev:... to the us-central1-docker.pkg.dev/feldera-ci/ghcr-remote/... mirror. Consistent across every workflow that pulls the dev image; matches the pattern in #6825.
deploy/README.md rollout list replaced with the grep that produces it, and the FIPS commit pin stays in one place (install-aws-lc-fips.sh).

No new concerns. Approving the current tip.

Go is a build dependency of aws-lc-fips-sys, which only enters the dependency graph behind the fips cargo feature. The runtime image compiles pipelines, and the generated pipeline workspace never enables that feature, so nothing the image builds has needed Go since #5488 made FIPS opt-in. The toolchain has been carried since then at 258 MB, and its Ubuntu package ships CVEs that fail the image scan, which is why it came from an upstream tarball and was bumped three times. Verified by building the image and compiling a pipeline dependency inside it with no go on PATH. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>

mythical-fred

Rescope + trivial cleanup. The three prior FIPS commits landed on main (via 328d846); the branch now carries a single commit that removes the Go toolchain install from deploy/Dockerfile (runtime image) and reworks the README dependency note. Runtime pipelines never enable the fips feature (grepped MultiCratesWriter.java and rust_compiler.rs at 20bec672 — no fips/aws_lc references), so nothing built from that image needs Go anymore. deploy/build.Dockerfile still installs golang-go for the manager build, which is right — --features fips there compiles aws-lc-fips-sys from source. Approve.

1.18.0 is the first release depending on aws-lc-fips-sys 0.14, which added AWS_LC_FIPS_SYS_SYSTEM_DIR and USE_SYSTEM. Earlier versions always compiled AWS-LC from source, so enabling the fips feature would have put Go and cmake back into the image that compiles pipelines, undoing #6822. With 0.14 the validated module can be built once and linked, the way scripts/install-librdkafka.sh already builds AWS-LC for librdkafka. Nothing selects fips yet; this only makes that step possible without a toolchain in the shipped image. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>

gz force-pushed the fips-prebuilt-aws-lc-module branch from 7ad12ee to 398eb7d Compare August 6, 2026 16:12

ryzhyk approved these changes Aug 6, 2026

View reviewed changes

lalithsuresh approved these changes Aug 6, 2026

View reviewed changes

gz enabled auto-merge August 6, 2026 16:38

gz disabled auto-merge August 6, 2026 16:40

mythical-fred approved these changes Aug 6, 2026

View reviewed changes

gz force-pushed the fips-prebuilt-aws-lc-module branch from 398eb7d to 6bffffd Compare August 7, 2026 06:11

gz enabled auto-merge August 7, 2026 06:12