Skip to content

Latest commit

Β 

History

History
436 lines (311 loc) Β· 16.1 KB

File metadata and controls

436 lines (311 loc) Β· 16.1 KB

Developer Guide β€” ProvarDX CLI

Everything you need to clone, build, run locally, and test changes to @provartesting/provardx-cli.


Table of Contents


Prerequisites

Tool Minimum version How to check
Node.js 18.0.0 node --version
npm 8+ (ships with Node 18) npm --version
Salesforce CLI (sf) any recent sf --version
Git any git --version

The Salesforce CLI is required to run the compiled plugin commands in development mode (bin/dev.js). Install it from developer.salesforce.com/tools/salesforcecli if needed.


First-time setup

# 1. Clone the repository
git clone https://github.com/ProvarTesting/provardx-cli.git
cd provardx-cli

# 2. Install dependencies
npm install

# 3. Compile TypeScript + copy JSON rule files to lib/
npm run compile

Windows note: All scripts use shx for cross-platform shell commands, so npm run compile works identically on Windows, macOS, and Linux.


Project structure

provardx-cli/
β”œβ”€β”€ src/
β”‚   β”œβ”€β”€ commands/provar/          # oclif CLI command implementations
β”‚   β”‚   β”œβ”€β”€ config/               #   provar config get|set
β”‚   β”‚   β”œβ”€β”€ automation/config/    #   provar automation config *
β”‚   β”‚   └── mcp/                  #   provar mcp start
β”‚   └── mcp/
β”‚       β”œβ”€β”€ server.ts             # MCP server factory (registers all tools)
β”‚       β”œβ”€β”€ tools/                # One file per MCP tool + engine modules
β”‚       β”‚   β”œβ”€β”€ bestPracticesEngine.ts   # Scoring engine (port of Lambda)
β”‚       β”‚   β”œβ”€β”€ hierarchyValidate.ts     # Suite/plan/project validation engine
β”‚       β”‚   β”œβ”€β”€ testCaseValidate.ts      # Schema-level TC validation
β”‚       β”‚   β”œβ”€β”€ pageObjectValidate.ts    # Java Page Object validation
β”‚       β”‚   β”œβ”€β”€ testCaseGenerate.ts      # XML generation
β”‚       β”‚   β”œβ”€β”€ pageObjectGenerate.ts    # Java PO generation
β”‚       β”‚   β”œβ”€β”€ projectInspect.ts        # Project folder inspection
β”‚       β”‚   └── {testSuite,testPlan,testProject}Validate.ts
β”‚       β”œβ”€β”€ rules/                # provar_best_practices_rules.json (runtime)
β”‚       β”œβ”€β”€ schemas/              # Shared Zod schemas + common helpers
β”‚       β”œβ”€β”€ security/             # Path policy enforcement
β”‚       └── logging/              # Structured logger
β”œβ”€β”€ test/
β”‚   β”œβ”€β”€ unit/mcp/                 # Pure function unit tests (Mocha + Node assert)
β”‚   └── commands/provar/          # NUT integration tests (require sf CLI)
β”œβ”€β”€ docs/
β”‚   β”œβ”€β”€ development.md            # This file
β”‚   └── mcp.md                    # MCP server user guide
β”œβ”€β”€ messages/                     # oclif message files (descriptions, examples)
β”œβ”€β”€ lib/                          # Compiled output (git-ignored, generated by compile)
β”œβ”€β”€ bin/
β”‚   β”œβ”€β”€ dev.js                    # Local dev entry point (uses ts-node, reads src/)
β”‚   └── run.js                    # Production entry point (reads lib/)
β”œβ”€β”€ .mocharc.json                 # Mocha config (ts-node/esm loader, 10 min timeout)
β”œβ”€β”€ .nycrc                        # Istanbul/nyc coverage thresholds
└── tsconfig.json                 # TypeScript config (strict ESM, outDir: lib)

Build system overview

This project uses Wireit as a build orchestrator on top of npm scripts. Wireit provides:

  • Incremental builds β€” only recompiles files that changed
  • Dependency ordering β€” build depends on compile + lint; test depends on test:compile + test:only + lint
  • Caching β€” subsequent npm run compile runs skip unchanged files

The compile step does two things:

  1. tsc β€” transpiles src/**/*.ts β†’ lib/
  2. shx cp src/mcp/rules/*.json lib/mcp/rules/ β€” copies the JSON rules file (TypeScript does not copy non-.ts assets automatically)

Building

# Compile TypeScript only (fastest β€” use while iterating)
npm run compile

# Compile + lint (full build gate, same as CI)
npm run build

# Remove all compiled output and caches
npm run clean

# Remove all compiled output AND node_modules
npm run clean-all

After npm run compile succeeds, the compiled plugin is available under lib/ and the development entry point (bin/dev.js) is ready to use.


Running locally (without installing)

bin/dev.js is the development entry point. It loads the plugin directly from your local lib/ directory rather than from the installed npm package, so any changes you compile are immediately available.

# General form
node bin/dev.js <command> [flags]

# Examples
node bin/dev.js provar config get environment.testEnvironment -f provardx-properties.json
node bin/dev.js provar automation config validate
node bin/dev.js provar mcp start --allowed-paths /path/to/project

Tip: On Unix you can chmod +x bin/dev.js and run it as ./bin/dev.js or add a shell alias:

alias sfdev="node $(pwd)/bin/dev.js"
sfdev provar mcp start

bin/run.js uses the installed lib/ too but targets production settings. Prefer bin/dev.js during development.


Testing

Unit tests

Unit tests live in test/unit/mcp/ and cover all pure validator functions (no filesystem, no network, no Salesforce CLI required). There are three ways to run them, depending on your workflow:

Command When to use
npm run test:dev Daily development β€” always executes, never cached, fastest for iteration
npm run test:watch TDD mode β€” re-runs automatically whenever a src/ or test/ file changes
npm run test:only CI / pre-commit β€” wireit-managed; skips if no files changed since last run
# Always runs β€” no caching, best for iterating on changes
npm run test:dev

# Re-runs on every file save β€” great for test-driven development
npm run test:watch

# Wireit-managed (may be skipped if inputs are unchanged β€” see note below)
npm run test:only

Expected output: 96 passing (Xms) β€” all tests should be green.

The test runner is Mocha with ts-node/esm as the loader, so test files are executed directly from TypeScript source without a prior compile step.

Wireit caching note: npm run test:only is managed by Wireit, which fingerprints your source and test files. If nothing has changed since the last successful run it prints Ran 0 scripts and skipped 1 and exits without executing Mocha β€” this is intentional for CI performance. If you want the tests to always run unconditionally, use npm run test:dev instead. To force a fresh run via test:only (e.g. after pulling changes), clear the cache first:

rm -rf .wireit && npm run test:only

Current test files and what they cover:

File Covers
testCaseValidate.test.ts XML schema rules TC_001–TC_035
pageObjectValidate.test.ts Java PO rules PO_001–PO_080
pathPolicy.test.ts Path security policy (allowed paths, traversal)
hierarchyValidate.test.ts Suite/plan/project structural + naming rules, buildHierarchySummary
bestPracticesEngine.test.ts Scoring formula (exact Lambda parity), runBestPractices

Coverage

Coverage is collected by nyc (Istanbul) and runs as part of test:only. Thresholds are enforced in .nycrc:

{ "lines": 80, "statements": 80, "functions": 60, "branches": 30 }

To view a coverage report:

# After running test:only, open the HTML report
npx nyc report --reporter=html
open coverage/index.html          # macOS
start coverage/index.html         # Windows

NUT integration tests

NUT (Named Unit Test) files in test/commands/provar/ perform end-to-end CLI execution and require a functioning Salesforce CLI installation. They are not run by default in the unit test suite.

# Run all NUT tests (requires sf CLI in PATH and a valid Salesforce org)
npm run test:nuts

These are primarily run in CI against a real environment. For local development, the unit tests in test/unit/mcp/ are sufficient for MCP-related changes.

Full test + lint gate

# Equivalent to what CI runs: test:compile + test:only + lint
npm test

This must pass before any PR is merged.


Linting and formatting

# Run ESLint (reports errors only, does not fix)
npm run lint

# Auto-format source and test files with Prettier
npm run format

The project uses @salesforce/dev-scripts for shared ESLint and Prettier configs. The camelcase rule is suppressed in files that deal with JSON field names from external APIs (note the /* eslint-disable camelcase */ comments in MCP tool files).


Developing the MCP server locally

To test MCP server changes interactively with an AI client:

Claude Code (simplest)

  1. Compile your changes: npm run compile
  2. Add a project-scoped MCP server entry in .claude/mcp.json at the repo root:
{
  "mcpServers": {
    "provar-dev": {
      "command": "node",
      "args": ["bin/dev.js", "provar", "mcp", "start", "--allowed-paths", "/path/to/test/project"]
    }
  }
}
  1. Open Claude Code in this repo β€” the provar-dev MCP server is loaded automatically.

Claude Desktop

Update ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "provar-dev": {
      "command": "node",
      "args": [
        "/absolute/path/to/provardx-cli/bin/dev.js",
        "provar",
        "mcp",
        "start",
        "--allowed-paths",
        "/path/to/your/provar/project"
      ]
    }
  }
}

Restart Claude Desktop after saving.

MCP Inspector (interactive tool testing)

The MCP Inspector is a browser-based UI for calling MCP tools directly β€” no AI client required. Use it to iterate on tool schemas and verify request/response shapes.

# Compile first, then launch the Inspector against the dev server
npm run compile
npx @modelcontextprotocol/inspector node bin/dev.js provar mcp start --allowed-paths /absolute/path/to/your/provar/project

MCP resources in dev mode: bin/dev.js runs the TypeScript source via ts-node. The server automatically falls back to reading bundled Markdown resources (e.g. provar://nitrox/component-catalog, provar://docs/step-reference) from the repo-root docs/ directory when the compiled lib/mcp/docs/ path is not present, so resources work correctly after npm run compile with either entry point.

Run this as a single line. Line-wrapping the command (e.g. with a newline before --allowed-paths) causes the shell to treat --allowed-paths as a separate command and fail with command not found. Use \ for explicit line continuation in bash if needed.

If the Inspector fails with Proxy Server PORT IS IN USE at port 6277, a previous Inspector process is still running. Free the ports and try again:

# macOS / Linux
kill $(lsof -ti :6277) $(lsof -ti :6274) 2>/dev/null

# Windows PowerShell
foreach ($p in 6277,6274) { $c = Get-NetTCPConnection -LocalPort $p -EA 0; if ($c) { Stop-Process -Id $c.OwningProcess -Force } }

The command starts two processes: the MCP Inspector proxy (default port 6277) and opens a browser tab at http://localhost:6274. From there you can:

  • Browse all registered tools and their input schemas
  • Invoke any tool with custom JSON input
  • Inspect the raw JSON-RPC request and response
  • View server stderr logs in the Notifications panel

Windows path note: Use forward slashes or escaped backslashes in --allowed-paths (e.g. C:/Users/you/provar-project or C:\\Users\\you\\provar-project). The path must match one of the roots configured in provardx-properties.json.

To pin a specific Inspector version or avoid repeated downloads, install it once globally:

npm install -g @modelcontextprotocol/inspector
# Then run as:
mcp-inspector node bin/dev.js provar mcp start --allowed-paths /path/to/project

Debugging MCP tool calls

The MCP server writes structured logs to stderr. To see them when running manually:

# Start the server with verbose output visible
node bin/dev.js provar mcp start 2>mcp-debug.log

In another terminal, tail the log:

tail -f mcp-debug.log

Each tool call logs a requestId, the tool name, and the key input parameters β€” helpful for tracing unexpected behaviour.

Adding a new MCP tool

  1. Create src/mcp/tools/myTool.ts and export a registerMyTool(server: McpServer): void function.
  2. Import and call it in src/mcp/server.ts.
  3. Write unit tests in test/unit/mcp/myTool.test.ts.
  4. Add the tool name to the tools list in messages/sf.provar.mcp.start.md and docs/mcp.md.
  5. Run npm run compile && npm run test:only to verify everything passes.

Troubleshooting

npm run test:only shows "Ran 0 scripts and skipped 1" β€” tests don't run

This is wireit's input-fingerprinting cache in action: it recorded a successful test run and sees no changes to src/ or test/ since then, so it skips execution entirely. This is expected behaviour in CI but can be surprising locally.

Fixes:

# Option 1 (preferred for daily dev) β€” always executes, never cached
npm run test:dev

# Option 2 β€” clear the cache, then re-run via wireit
rm -rf .wireit && npm run test:only

wireit reports "script is already running"

Wireit locks scripts to prevent concurrent runs. If a previous run was interrupted:

# Remove the wireit lock and cache files
rm -rf .wireit
npm run compile

TypeScript incremental build is stale

If you see type errors that don't match the source, force a clean rebuild:

npm run clean
npm run compile

Cannot find module '.../lib/mcp/rules/provar_best_practices_rules.json'

The JSON rules file is not copied by tsc automatically. Make sure you compiled with the full wireit task (not bare tsc):

npm run compile   # ← copies JSON rules to lib/mcp/rules/

If using bare tsc for IDE integration, run the copy manually:

mkdir -p lib/mcp/rules && cp src/mcp/rules/*.json lib/mcp/rules/

Tests fail with ExperimentalWarning: --experimental-loader

This is an informational warning from Node.js about ts-node/esm and can be safely ignored. It does not indicate a test failure.

sf command not found when using bin/dev.js

bin/dev.js delegates to the Salesforce CLI oclif runtime. Ensure sf is installed and on your PATH:

npm install -g @salesforce/cli
sf --version

ESLint camelcase errors in MCP files

MCP tool files use /* eslint-disable camelcase */ at the top because JSON field names from the Quality Hub API use snake_case. This is intentional β€” do not remove the disable comment.