feat: eliminate runtime npm install from binary installer#13514
Conversation
…taller Bundle all dependencies into the universal archive so the Go binary installer no longer needs to run `npm install` at runtime. Changes: - Bundle pure JS deps into sf-core.js (ajv, ajv-formats, esbuild JS API, @aws-sdk/signature-v4a, @aws-sdk/client-cloudfront-keyvaluestore) - Remove @aws-sdk/signature-v4-crt side-effect imports (pure JS fallback) - Ship ajv runtime files in dist/node_modules/ for cached validators - Ship all 5 esbuild platform binaries in dist/node_modules/@esbuild/ - Set ESBUILD_BINARY_PATH in injects-shim.js before esbuild module init - Replace npm pack with tar czf (npm pack strips node_modules/) - Go binary: skip npm install when package.json has no dependencies, clean up unused esbuild platform binaries (keeps only current platform) Backward compatible: - Old Go binary + new archive: npm install is a no-op, framework works - New Go binary + old archive: detects dependencies, runs npm install
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (7)
✅ Files skipped from review due to trivial changes (1)
🚧 Files skipped from review as they are similar to previous changes (3)
📝 WalkthroughWalkthroughRefactors distribution packaging to embed AJV and platform esbuild binaries into the framework tarball, removes runtime dependencies from the published package.json, updates packaging/pack scripts to produce normalized tarballs, and changes the binary installer to skip ChangesBinary installer behavior
Framework package & bundling
Distribution tarball creation & assets
Minor cleanup
Sequence Diagram(s)sequenceDiagram
participant Build as BuildScript
participant FS as FileSystem
participant Registry as NPMRegistry
participant Installer as BinaryInstaller
participant PKG as package.json
rect rgba(100, 200, 150, 0.5)
Note over Build,Registry: Build-time bundling
Build->>Registry: Download `@esbuild/<platform>` tarballs (concurrent)
Registry-->>Build: Platform tarballs
Build->>FS: Extract esbuild binaries to framework-dist/dist/node_modules/@esbuild/<platform>/
Build->>FS: Copy AJV runtime files into framework-dist/dist/node_modules/ajv...
end
rect rgba(150, 150, 200, 0.5)
Note over Installer,FS: Install-time behavior
Installer->>FS: Extract package archive
Installer->>PKG: Read/check `dependencies` via archiveHasDependencies()
alt dependencies present
Installer->>Registry: Run `npm install`
Registry-->>Installer: Install results
else no dependencies
Installer->>FS: cleanupUnusedEsbuildBinaries() (remove other-platform dirs)
end
Installer->>FS: Finalize installation
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 golangci-lint (2.12.2)level=error msg="[linters_context] typechecking error: pattern ./...: directory prefix . does not contain main module or its selected dependencies" Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@binary-installer/src/version.go`:
- Around line 516-526: Only treat a missing package.json as the compatibility
case; when calling os.ReadFile(filepath.Join(packageDir, "package.json")) check
if the error is an os.IsNotExist error and return (false, nil) only then,
otherwise propagate the read error (return false, err) so failures bubble up;
likewise, when json.Unmarshal(data, &pkg) fails do not swallow the error—return
(false, err) instead of (false, nil). Update the error handling around
os.ReadFile and json.Unmarshal (symbols: os.ReadFile, filepath.Join, packageDir,
"package.json", json.Unmarshal, pkg) so corrupted/unreadable package.json causes
the install to fail and trigger cleanup.
In `@packages/sf-core/scripts/prepareDistributionTarballs.js`:
- Around line 12-17: The two require initializations (serverlessRequire and
sfCoreRequire) use import.meta.dirname which is inconsistent with the codebase;
replace that by deriving a directory from fileURLToPath(import.meta.url) and
using path.dirname so createRequire calls use
path.resolve(path.dirname(fileURLToPath(import.meta.url)), '<relative-path>');
import fileURLToPath from 'url', compute the current file dir with
path.dirname(fileURLToPath(import.meta.url)), then pass that dir into the
existing createRequire/path.resolve calls for serverlessRequire and
sfCoreRequire to match the project's established pattern.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 7137417c-9209-4dee-9a95-9b57a6695db4
📒 Files selected for processing (11)
binary-installer/src/version.gobinary-installer/src/version_test.gopackages/engine/src/lib/aws/cloudfront-kv.jspackages/engine/src/lib/deploymentTypes/aws/deploy-dynamic-routing.jspackages/framework-dist/npm-shrinkwrap.jsonpackages/framework-dist/package.jsonpackages/serverless/lib/classes/config-schema-handler/resolve-ajv-validate.jspackages/sf-core/esbuild.jspackages/sf-core/injects-shim.jspackages/sf-core/prepareReleaseTars.shpackages/sf-core/scripts/prepareDistributionTarballs.js
💤 Files with no reviewable changes (3)
- packages/engine/src/lib/deploymentTypes/aws/deploy-dynamic-routing.js
- packages/sf-core/esbuild.js
- packages/engine/src/lib/aws/cloudfront-kv.js
…leURLToPath for Node 18 compat - archiveHasDependencies: only treat missing package.json as "skip npm install". I/O errors and malformed JSON now return errors so the deferred cleanup removes the broken release instead of marking it as successfully installed. - prepareDistributionTarballs.js: replace import.meta.dirname (Node 21.2+) with fileURLToPath(import.meta.url) for consistency with the rest of the codebase. Consolidate two createRequire instances into one (resolveFromServerless) since both ajv and esbuild are deps of the serverless package.
There was a problem hiding this comment.
🧹 Nitpick comments (2)
binary-installer/src/version_test.go (2)
465-499: Minor inconsistency in Windows binary path structure.This test creates
bin/esbuildfor all platforms includingwin32-x64, butcreateNewArchive(line 564) correctly usesesbuild.exedirectly in the platform directory for Windows. While this doesn't affect the cleanup test (which only checks directory removal), it's inconsistent with the actual archive structure.Consider aligning the binary creation logic with
createNewArchivefor consistency:💡 Suggested fix for consistency
platforms := []string{"darwin-arm64", "darwin-x64", "linux-arm64", "linux-x64", "win32-x64"} for _, p := range platforms { - dir := filepath.Join(esbuildDir, p, "bin") - if err := os.MkdirAll(dir, 0o755); err != nil { + var binPath string + if p == "win32-x64" { + binPath = filepath.Join(esbuildDir, p, "esbuild.exe") + } else { + binPath = filepath.Join(esbuildDir, p, "bin", "esbuild") + } + if err := os.MkdirAll(filepath.Dir(binPath), 0o755); err != nil { t.Fatal(err) } - _ = os.WriteFile(filepath.Join(dir, "esbuild"), []byte("binary"), 0o755) + _ = os.WriteFile(binPath, []byte("binary"), 0o755) }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@binary-installer/src/version_test.go` around lines 465 - 499, The test TestCleanupUnusedEsbuildBinaries creates a Windows artifact under bin/esbuild but createNewArchive uses esbuild.exe directly in the platform dir; update the test so when platform == "win32-x64" it writes "esbuild.exe" at filepath.Join(esbuildDir, p, "esbuild.exe") (matching createNewArchive) instead of writing under bin/, leaving cleanupUnusedEsbuildBinaries and esbuildPlatformDir logic unchanged.
827-829: Consider adding an explicit assertion or comment for clarity.The test validates that
cleanupUnusedEsbuildBinariesdoesn't panic when the directory doesn't exist, but this intent isn't explicitly documented. Adding a brief comment would improve readability.💡 Suggested improvement
func TestCleanupUnusedEsbuildBinaries_MissingDir(t *testing.T) { + // Should not panic when esbuild directory doesn't exist cleanupUnusedEsbuildBinaries("/nonexistent/path") }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@binary-installer/src/version_test.go` around lines 827 - 829, The test TestCleanupUnusedEsbuildBinaries_MissingDir currently calls cleanupUnusedEsbuildBinaries("/nonexistent/path") without documenting intent; update the test to either add a brief comment stating "ensures cleanupUnusedEsbuildBinaries does not panic when directory is missing" or replace the direct call with an explicit non-panic assertion (e.g., using require.NotPanics or a defer/recover check) to make the intention clear; refer to the TestCleanupUnusedEsbuildBinaries_MissingDir test and the cleanupUnusedEsbuildBinaries function when making the change.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@binary-installer/src/version_test.go`:
- Around line 465-499: The test TestCleanupUnusedEsbuildBinaries creates a
Windows artifact under bin/esbuild but createNewArchive uses esbuild.exe
directly in the platform dir; update the test so when platform == "win32-x64" it
writes "esbuild.exe" at filepath.Join(esbuildDir, p, "esbuild.exe") (matching
createNewArchive) instead of writing under bin/, leaving
cleanupUnusedEsbuildBinaries and esbuildPlatformDir logic unchanged.
- Around line 827-829: The test TestCleanupUnusedEsbuildBinaries_MissingDir
currently calls cleanupUnusedEsbuildBinaries("/nonexistent/path") without
documenting intent; update the test to either add a brief comment stating
"ensures cleanupUnusedEsbuildBinaries does not panic when directory is missing"
or replace the direct call with an explicit non-panic assertion (e.g., using
require.NotPanics or a defer/recover check) to make the intention clear; refer
to the TestCleanupUnusedEsbuildBinaries_MissingDir test and the
cleanupUnusedEsbuildBinaries function when making the change.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: a79ad5eb-fad2-4652-986a-f3976ad050ac
📒 Files selected for processing (3)
binary-installer/src/version.gobinary-installer/src/version_test.gopackages/sf-core/scripts/prepareDistributionTarballs.js
🚧 Files skipped from review as they are similar to previous changes (2)
- packages/sf-core/scripts/prepareDistributionTarballs.js
- binary-installer/src/version.go
|
Short note here: This approach bundles esbuild binaries for all 5 platforms into the tarball, bringing it to 65MB. Consider following the pattern used by @biomejs/biome to ship platform-specific packages instead. A consumers would have to download only the binary they need (~14MB). For example:
|
Thanks for the suggestion! I looked into adopting the biomejs pattern and wanted to share what's blocking it. Why we can't use optionalDependencies on the Without a post-install step, the Go binary would need to locate the npm-installed esbuild binary at runtime. Discovery is complex across install scenarios (npm global, npm local, pnpm, Yarn PnP, etc.) and doesn't work at all for users who install the Go binary via the curl-based install script from install.serverless.com — they have no On archive size (the 65MB concern): compressed archive is ~37MB (5 esbuild binaries compress to ~20MB from ~50MB). Total user download is ~37MB vs ~35MB today (14MB archive + ~21MB On disk: ~110MB after cleanup (Go binary deletes 4 unused platform binaries), vs 173MB today. If download size becomes a real concern later, we can switch to platform-specific S3 archives ( |
# Conflicts: # packages/framework-dist/npm-shrinkwrap.json # packages/framework-dist/package.json
# Conflicts: # packages/framework-dist/npm-shrinkwrap.json # packages/framework-dist/package.json
The framework-dist version had drifted to 4.3.3 on main (likely a typo for 4.33.3 in the original "publish as npm package" commit). Bump to 4.35.1 to match sf-core, which is the runtime source of truth that prepareDistributionTarballs.js overwrites this field with at build time anyway.
Extract the framework-dist tarball packaging into a shared script (pack-framework-dist.sh), called from both prepareReleaseTars.sh and the sf-core test:build npm script. test:build previously ended in `npm pack`, which produced a tarball with different ownership metadata and could include stray files (like prior tarballs left in the directory). Local builds now mirror what CI/CD produces.
Pull the SRI hash recorded by npm for each @esbuild/<platform> package out of the workspace package-lock.json and verify the downloaded tarball matches before extracting. Same trust anchor as `npm install` — protects the release build from registry tampering between dependency install and tarball download.
…mment - Collapse esbuildPlatformDir and validEsbuildPlatforms onto a single source-of-truth map; adding a future platform now touches one place instead of two. - Replace a hardcoded line-number reference in resolve-ajv-validate.js with a content-based pointer that won't rot when the file changes.
Summary
Eliminates the runtime
npm installfrom the Go binary installer by bundling all dependencies into the universal archive. The binary no longer needs npm to install the framework.sf-core.js(esbuild JS API, ajv, ajv-formats, AWS SDK signature-v4a, CloudFront KVS client)@aws-sdk/signature-v4-crtnative dependency — pure JS@aws-sdk/signature-v4afallback handles SigV4A signing for CloudFront KeyValueStorenpm packwithtar czf(npm packstripsnode_modules/directories)npm installwhenpackage.jsonhas no dependencies, clean up unused esbuild platform binaries after extractionBackward compatibility
All four combinations work — phases are independently deployable:
npm installruns as before (unchanged)npm installis a no-op (empty deps), esbuild binaries already in archivepackage.json→ runsnpm installnpm install, cleans up unused platform binaries (keeps 1 of 5)Archive size
npm installdownloadChanges
JS build pipeline
esbuild.js— Remove allexternalentries; everything bundled intosf-core.jsinjects-shim.js— Detect esbuild platform binary atdist/node_modules/@esbuild/<platform>/and setESBUILD_BINARY_PATHbefore esbuild module initializationresolve-ajv-validate.js— Remove__dirnameredirect so cached standalone validators resolve ajv runtime files fromdist/node_modules/prepareDistributionTarballs.js— Copy ajv runtime files + download esbuild binaries for all 5 platforms (parallel downloads, secure temp dirs)prepareReleaseTars.sh— Replacenpm packwithtar czf(GNU + BSD compatible, normalized uid/gid ownership)framework-dist/package.json— Emptydependenciesnpm-shrinkwrap.json— DeletedEngine
cloudfront-kv.js— Removeimport '@aws-sdk/signature-v4-crt'deploy-dynamic-routing.js— Removeimport '@aws-sdk/signature-v4-crt'The AWS SDK falls back to pure JS
@aws-sdk/signature-v4afor SigV4A signing. This eliminates theaws-crtnative dependency (33MB).Go binary installer
version.go— After archive extraction, readpackage.jsondependencies:npm installas before (old archive compatibility)npm install, delete unused@esbuild/*platform directoriesversion_test.go— Unit tests for helpers + 4-scenario compatibility matrix with realistic archive structuresSummary by CodeRabbit
Release Notes
New Features
Chores