Validate configuration file against JSON schema - #1050

Description

This change adds in-process JSON schemas validation to villas-node. This enables us to catch many more invalid configurations and present more readable errors for these. The main goals I wanted to achieve here are:

1. Reject invalid option values.
2. Reject unknown options and typos.
3. Present sensible location information.
4. Present a reasonable error message.

Implementation

Adding a Validator

The first goal was quite easy to achieve:

1. Add a schema validation library.
2. Bundle our JSON schema into the process.
3. Run the schema validator on a configuration file before we continue parsing.

We've previously added a new dependency on nlohmann::json for a more idiomatic and less error-prone JSON library written in C++. I had to choose between the two established JSON schema validator projects that support nlohmann::json values: json-schema-validator and valijson. The choice here fell on json-schema-validator due to its support for patching default values into the configuration after validation.

Reworking the Schema

This first step already applies the schema checks to all options in the configuration and thus make a lot of the checks and exception throwing in the C++ code unnecessary. But this wouldn't be able to catch unknown keys or typos. Both json-schema-validator and valijson only support JSON Schema Draft-07 schemas. The only way to achieve goal two - rejecting unknown keys and typos - in Draft-07 is to use an additionalProperties: false annotation, but that requires all properties of any object values to be declared in a single schema. This is fundamentally incompatible with the way our schemas are currently structured.

This schema rework was partially LLM assisted in schema generation and validation against the C++ parsing code. I've rewritten every single schema file from scratch, copying descriptions and titles from the existing schemas where possible. The new schemas now live in a single flat directory following a new naming convention.

The schema bundling routine uses the redocly CLI tool, which will place all subschemas into a single flat JSON object at /components/schemas. The key of the schema in the flattened object is derived from the file's basename, not its directory which would have previously caused e.g. the config/nodes/stats.yaml and config/hooks/stats.yaml in the old directory layout to collide. These collisions aren't fatal, redocly will just rename one of the schemas to prevent a collision, but this will cause the schema name to be unpredictable, which I deemed undesirable. Placing all schemas in a single directory with a proper naming convention prevents these collisions by design.

Plugin Subschemas

This refactor achieved the second and third goal - rejecting unknown keys or typos and providing location information in the form of a json pointer. The problem that remains for goal four is that discriminated unions do not have a good representation in a JSON schema. There isn't a way to specify that some property within an object serves as a discriminator. The discriminator keyword we have is only an OpenAPI annotation which isn't used for validation or error reporting. This means that if we have a typo in a property, the validator will try to do a structural match for all plugin types and report why each plugin would not pass validation, burying the actual plugin as indicated by type in a sea of unrelated errors.

I've solved this by validating the configuration in stages. The schema sitting in doc/openapi is not completely valid by itself. The schema for plugin types like Node, Hook or Format only specify a discriminator without an adjacent logical combinator. That discriminator has a x-villas-plugin annotation that is expanded to a proper logical combinator for the documentation by the doc/villas.js redocly plugin. The schema bundled into the binary will does not expand the discriminator, I can then use that annotation to discover subschemas myself and report proper, readable errors based on the actual plugin type.

Migrations

The schema rework led me to discover a bunch of unintuitive, undesirable or straight up broken behavior in our parsing code. I've added migration code that detects some of these cases, where I've deemed that retaining the current behavior would lead to worse schemas, but the option is common enough to break the experience for many users. I'm using the structural schema traversal from the subschemas above to apply these migrations depending on the plugin type specified.

Note that this migration is ephemeral; it will run on every node startup and emit warnings for all migration operations that had to be done, but it won't write those changes back to disk. You'll have to check those migrations and apply them manually.

Node migrations

I've removed six options from the Node base-class:

• enabled: This option is completely unnecessary. A Node that is not used by any path will be disabled by its SuperNode regardless of this configuration value. Setting this configuration value to false for a node that is used by a path will just throw an exception on startup. There is literally no case where specifying this option makes a useful difference so I don't provide a migration here.

• initial_sequenceno: This option is also completely unnecessary, this functionality is better served by a shift_seq hook. I've only ever seen one use of this option in the node-stats integration test so I didn't provide an automatic migration here either.

• builtin/vectorize/signals/hooks: These options can be specified at the top-level of a node. The NodeParse function then distributes these values into the in and out subobjects if they are present. This code was used in a few places, it allowed the user to e.g. specify vectorize for both the in and out direction of a node at the same time. The problem I have with this is that this code bloats our schemas, these four keys would have had to be added to every node-*.yaml schema file, where I deemed the overhead to be larger than these options' usefulness. Another problem is that this code introduced subtle bugs. I've added automatic migrations for these to keep existing configurations working.

I've also removed the object-with-count and string representations for signal lists because those needed a giant amount of work to properly support in the JSON schema and we didn't even bother to implement these any node specific schema in the past. These are also automatically migrated.

Path migrations

I've made the reverse option into a migration, this option would only work on paths that didn't have any mapping expressions and I wanted to be sure that a path that validates against our schema would also work correctly. Encoding this logic into the schema is more complicated than just removing it, especially because it's seldomly used.

Tooling

I've also worked on our tooling that supports our configuration files. We've previously had villas-conf2json, which just read in a libconfig file and printed a JSON version of that file to stdout, and villas-test-config, which tries to validate a configuration by parsing it without any starting any node.

I've merged both tools into a single villas-config tool that serves multiple purposes:

1. Load the input file from either libconfig of JSON.
2. Apply migrations to the input file. (optional)
3. Validate the input file against our schema.
4. Patch default values into the configuration. (optional)
5. Print the resulting configuration to stdout. (optional)

I've needed all these features for our integration tests and found them to be also genuinely useful for interactive introspection and debugging of a configuration file.

You can also use this tool for your manual migrations. I still consider these to be manual because all comments will be dropped, which often information that useful for documentation purposes.