Skip to content

Decouple projects from storage API#5831

Open
beasteers wants to merge 4 commits into
node-red:mainfrom
beasteers:runtime-projects
Open

Decouple projects from storage API#5831
beasteers wants to merge 4 commits into
node-red:mainfrom
beasteers:runtime-projects

Conversation

@beasteers

@beasteers beasteers commented Jun 17, 2026

Copy link
Copy Markdown
Contributor
  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)

tl;dr - Projects can now work with custom storage providers; the important changes are in the third commit; the design choices are concentrated in storage/localfilesystem/flows.js and projects/git/Project.js - the rest is just scaffolding around that.

Proposed changes

This PR moves project orchestration out of the localfilesystem storage implementation and makes it a runtime service (runtime.projects), so project management is no longer tied to the default single-file flow storage layout. Storage APIs can now provide Project lifecycle methods that perform project storage operations such as initializing or migrating to a project, recording path settings in package.json, and listing tracked files. In addition, it is also implemented with swappable "Project Types" so that git could be substituted for something else (e.g. object storage).

Previously, storage/localfilesystem/projects mixed together three separate concerns:

  • the runtime-facing projects API
  • git-specific project behavior
  • assumptions about the default flow/credentials file layout

This change separates those responsibilities into dedicated modules:

  • runtime/lib/projects/index.js
    The top-level runtime projects service. similar API to before but has pluggable project types (e.g. git)
  • runtime/lib/projects/git/index.js
    The built-in git project type implementation. It owns project lifecycle behavior and delegates git operations to Project.js.
  • runtime/lib/storage/localfilesystem/flows.js
    The canonical single-file flow layout. It owns getFlows, saveFlows, getCredentials, saveCredentials, and the opt-in project flow-file lifecycle operations that depend on how this layout stores flows.

As part of that split, runtime/lib/projects/git/Project.js no longer needs to know how flow files are structured on disk. Instead, it delegates flow-file semantics to the active storage layout via storage.getProjectLayout().

In order to enable projects for your storage provider, you need to define:

  • storage.getProjectType() -> 'git'
  • storage.getProjectLayout() -> layout

That layout now provides the project file lifecycle hooks the built-in git project type needs:

  • flowFileExists - does a (non-project) flow file exist on disk - used for the create project form
  • flowsDeclared - are flows declared in the project settings
  • exportFilesForEditor - exports flow/credentials paths that are used in the editor
  • listAutosaveFiles - the files that will auto-commit when enabled
  • createFiles - create new project files
  • migrateFiles - migrate existing project files
  • updateFiles - update project on PUT /projects/:id
  • buildPackageSettings - populate package settings with file paths

Commit structure

This is easiest to review commit-by-commit:

  1. 9ac2fcc9b
    Pure move/path-change commit - not much to see here.

    • runtime/lib/storage/localfilesystem/projects/* -> runtime/lib/projects/*
    • runtime/lib/storage/localfilesystem/util.js -> runtime/lib/storage/util.js
  2. 1ac551c3a
    No real changes - behaviour-preserving refactor of api/projects unit tests onto shared helpers, so the runtime service move in the next commit is easier to review.

  3. 95f736cd9
    Main refactor that decouples projects from storage and introduces the runtime service / git project type / flow layout split described above.

The third commit is the one I would focus on in isolation.

Review focus

Primary review targets are:

  • runtime/lib/projects/index.js
  • runtime/lib/projects/git/index.js
  • runtime/lib/projects/git/Project.js **
  • runtime/lib/storage/localfilesystem/flows.js **

Other runtime-facing updates are mostly mechanical call-site changes from runtime.storage.projects to runtime.projects, including:

  • runtime/lib/api/projects.js
  • runtime/lib/api/settings.js
  • runtime/lib/flows/index.js
  • runtime/lib/nodes/credentials.js
  • runtime/lib/index.js

There are also small supporting changes in:

  • runtime/lib/storage/index.js - adds flows.getProjectType and flows.getFlowLayout to configure Projects
  • runtime/lib/storage/localfilesystem/index.js - inits flows.js instead of projects.js
  • runtime/lib/projects/git/defaultFileSet.js - project.files.flow assignment now in projects/git/Project.js: buildDefaultFile

And then there's just moves and updated import paths:

  • runtime/lib/storage/localfilesystem/projects/* -> runtime/lib/projects/git/*
  • runtime/lib/storage/localfilesystem/library.js
  • runtime/lib/storage/localfilesystem/sessions.js
  • runtime/lib/storage/localfilesystem/settings.js

Possible additions

  1. storage API-defined editor validation - currently the editor is hardcoded with a json regex - just passing a small regex alongside the file names in RED.settings.files so that the editor is agnostic
  2. legacy projects adapter - a compatibility shim to use runtime.storage.projects for custom storage modules that implement their own project management.
  3. generalize the rich diff editor to support multiple split flow files and generalize to non-json storage formats.

I figured 3 could be a follow-up PR, 1 and 2 are drafted if you want me to include them in this PR.

Tests

New test files:

  • test/unit/@node-red/runtime/lib/projects/index_spec.js
  • test/unit/@node-red/runtime/lib/projects/git/index_spec.js
  • test/unit/@node-red/runtime/lib/projects/git/Project_spec.js
  • test/unit/@node-red/runtime/lib/projects/git/Project_writepath_spec.js
  • test/unit/@node-red/runtime/lib/storage/localfilesystem/flows_spec.js

Updated test files (basically just doing runtime.settings.projects to runtime.projects):

  • test/unit/@node-red/runtime/lib/projects/git/defaultFileSet_spec.js
  • test/unit/@node-red/runtime/lib/api/projects_spec.js
  • test/unit/@node-red/runtime/lib/api/settings_spec.js

Moves/updated import paths:

  • test/unit/@node-red/runtime/lib/storage/util_spec.js
  • test/unit/@node-red/runtime/lib/projects/git/**/*_spec.js

Checklist

  • I have read the contribution guidelines
  • For non-bugfix PRs, I have discussed this change on the forum/slack team.
  • I have run npm run test to verify the unit tests pass
  • I have added suitable unit tests to cover the new/changed functionality
  • I have added suitable documentation to cover the new/changed functionality

@linux-foundation-easycla

linux-foundation-easycla Bot commented Jun 17, 2026

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

@beasteers beasteers force-pushed the runtime-projects branch 2 times, most recently from 565b9a4 to 2a36203 Compare June 17, 2026 22:21
@knolleary

Copy link
Copy Markdown
Member

Hey @beasteers - thanks for this.

As discussed in slack, I can't promise how soon we'll be able to properly review this given the scale of change. It's a good proposal for sure - just a lot to work through.

Pure file moves ahead of the projects/storage decoupling, updating the require
paths/references that point at the relocated modules:
  storage/localfilesystem/projects/*  -> projects/git/*
No behaviour change.
Pull the repeated per-method runtime mocking into createRuntime() /
rejectedError() / assertRejects() helpers and a data-driven table for the
pass-through methods. Behaviour-preserving: same coverage, still exercising
the storage.projects API.
Extract project versioning out of the localfilesystem storage module into a
runtime-level service driven by a pluggable storage seam:

- runtime/lib/projects: type-agnostic project service; storage declares its
  project type + flow layout via getProjectType()/getFlowLayout(); api, flows
  and nodes talk to runtime.projects instead of storage.projects.
- storage/localfilesystem/flows.js: the single-file flow layout (read/save +
  namespaced layout.project.* write path) the built-in git type drives.
- projects/git: the git project type and Project model, write path delegated
  to the layout.

api/projects tests: createRuntime() switches storage.projects -> projects and
available() moves to a delegated call (prior commit's helper refactor makes
this a small change).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants