- Date: 2026-06-09 (manual subprocess execution)
- CGC version: 0.4.16 — tested on PyPI (
pip install codegraphcontext) and local editable (pip install -e .from this repo, uncommitted working-tree changes) - Python: 3.12.3
- OS: Linux 6.8.0-124-generic
- Method: Subprocess-only per E2E plan. No source modifications. Tests run from
/tmpwith isolatedHOMEunless testing repo-local config bleed. - Harness note: Initial automated harness (
/tmp/cgc_e2e_harness.py) produced many false positives (truncated Rich output, missingbundle import -y, config bleed when cwd inside CGC repo). Findings below are manually verified.
| Metric | Result |
|---|---|
| Embedded backends (FalkorDB, KuzuDB, LadybugDB) | PASS — index, query, find, bundle export |
| Remote backends (Neo4j, falkordb-remote, Nornic) | SKIP — Docker daemon unavailable |
| Context isolation (global / per-repo / named) | PASS |
| MCP stdio (25 tools) | PASS |
| Watch incremental | PASS |
| API gateway | PARTIAL — /api/v1/status OK; /health 404 |
| Confirmed bugs | 12 (0 Critical, 4 High, 5 Medium, 3 Low) |
Verdict (PyPI): Core indexing works, but call-chain analysis and golden parity regress vs repo goldens. Config bleed when cwd is inside a repo with .codegraphcontext/.
Verdict (local editable): Significantly better — call chains work, 20/20 golden parity exact match, cypher deprecation warning present. Several PyPI bugs appear fixed in the working tree but not yet released. Remaining local issues: delete orphans, per-repo FalkorDB default, API /health 404, CLI help gaps.
| Backend | index | stats (482/625) | chain f1→f3 | find f1 | query write block | bundle export/import | doctor |
|---|---|---|---|---|---|---|---|
| falkordb | PASS (1.5s) | PASS | FAIL | PASS | PASS | PASS / PASS (-y) | PASS |
| kuzudb | PASS (22s) | PASS | FAIL | PASS | PASS | PASS / PASS (-y) | PASS |
| ladybugdb | PASS (18s) | PASS | FAIL | PASS | PASS | PASS / PASS (-y) | PASS |
| falkordb-remote | SKIP | — | — | — | — | — | SKIP |
| neo4j | SKIP | — | — | — | — | — | SKIP |
| nornic | SKIP | — | — | — | — | — | SKIP |
Golden baseline for Python sample: 482 nodes, 621 edges in tests/fixtures/goldens/sample_project/metadata.json; PyPI measured 482 nodes, 625 edges (0.6% edge delta — within tolerance).
- Severity: High
- Category: UX / Config
- Backend(s): All
- Repro steps:
export HOME=$(mktemp -d) cgc config db neo4j cgc config set NEO4J_URI bolt://127.0.0.1:19999 cd /path/to/repo/with/.codegraphcontext # e.g. CGC repo itself cgc config show # NEO4J_URI reverts to bolt://localhost:7687 from repo .env
- Expected: User's
~/.codegraphcontext/.envvalues take precedence, or repo-local overrides are clearly documented and scoped to per-repo mode only. - Actual: Repo-local
.codegraphcontext/.envmerges and wins forDEFAULT_DATABASE,NEO4J_URI, etc. even in global mode with a freshHOME. Index/find connect to the repo's FalkorDB graph (53k+ functions in dev tree) instead of the isolated test database. - Impact: New users cloning CGC (or any repo with
.codegraphcontext/) get silently redirected to the repo's database. E2E tests, CI, and multi-project workflows produce wrong results without obvious error.
- Severity: High
- Category: Accuracy
- Backend(s): All (parser / call resolver)
- Repro steps:
Fixture:
export HOME=$(mktemp -d) && cd /tmp cgc config db falkordb && cgc context mode global cgc index --force tests/fixtures/sample_projects/sample_project cgc query "MATCH (a)-[r:CALLS]->(b) WHERE a.path CONTAINS 'function_chains' RETURN a.name, b.name, r.line_number"
function_chains.pyline 5:result = f1(f2(f3(10))) - Expected: CALLS edges
f1→f2,f2→f3(or equivalent nested-call representation). - Actual: Only module-level edges:
<module>→f1,<module>→f2,<module>→f3(all line 5). Zero Function→Function CALLS for this pattern. - Impact:
cgc analyze chain f1 f3returns "No call chain found". Docs/quickstart use this fixture as the canonical call-chain demo.
- Severity: High
- Category: Accuracy
- Backend(s): falkordb, kuzudb, ladybugdb (verified)
- Repro steps: After indexing sample_project:
cgc analyze chain f1 f3 - Expected: Chain
f1 → f2 → f3 - Actual:
No call chain found between 'f1' and 'f3' within depth 5(exit 0) - Impact: Primary onboarding example broken for nested module-level calls.
- Severity: Medium
- Category: Accuracy / UX
- Backend(s): All
- Repro steps: Index sample_project →
cgc analyze callers f2 - Expected: Caller
f1(per intuitive reading off1(f2(f3(10)))). - Actual: Single caller
<module>at function_chains.py (technically correct given BUG-002, but contradicts user/docs mental model). - Impact: Misleading caller analysis for module-level call expressions.
- Severity: Medium
- Category: Accuracy / Data integrity
- Backend(s): falkordb (verified)
- Repro steps:
cgc index --force sample_project_c # 83 nodes cgc delete sample_project_c cgc query "MATCH (n) RETURN count(n)" # returns 9, not 0
- Expected: All nodes belonging to deleted repository removed.
- Actual: 9 orphan nodes remain (Parameters, Variables without paths — e.g.
x,y,int v). - Impact: Repeated index/delete cycles accumulate stale nodes; inflates stats and can cause golden parity drift in batch sweeps.
- Severity: Medium
- Category: UX
- Backend(s): All (MCP)
- Repro steps: MCP
tools/call→delete_repositorywith{"path": "../../../etc"} - Expected: Clean validation error (path not found / not indexed).
- Actual:
Tool execution error: Failed to delete repository: 'NoneType' object has no attribute 'replace' - Impact: Scary internal error instead of actionable message; suggests missing null guard on repository path resolution.
- Severity: Medium
- Category: UX / Config
- Backend(s): kuzudb (verified)
- Repro steps:
cgc config db kuzudb && cgc context mode per-repo cd sample_project_go && cgc index --force . # Observe: "Context: Per-repo local mode (Database: falkordb)"
- Expected: Per-repo context inherits configured backend (kuzudb) or prompts user.
- Actual: Auto-created
.codegraphcontext/defaults tofalkordbregardless of globalcgc config db kuzudb. - Impact: User selects KuzuDB globally but per-repo mode silently uses FalkorDB Lite.
- Severity: Low
- Category: UX / Docs
- Backend(s): N/A (API)
- Repro steps:
cgc api start --port 8020→curl http://127.0.0.1:8020/health - Expected: Health check endpoint (common convention).
- Actual: HTTP 404. Working endpoint:
GET /api/v1/status→{"status":"ok","message":"Connected",...} - Impact: Load balancers / k8s probes using
/healthfail unless docs specify/api/v1/status.
- Severity: Low
- Category: UX / Docs
- Backend(s): All
- Repro steps:
cgc cypher 'MATCH (f:Function) RETURN f.name LIMIT 1' - Expected: Deprecation warning directing users to
cgc query. - Actual: Executes silently, returns results (exit 0).
- Impact: Deprecated command persists undetected.
- Severity: Low
- Category: Docs
- Backend(s): N/A
- Repro steps:
cgc index --help— check--databaseenum - Expected: Includes
nornic,falkordb-remoteperconfig_manager.py. - Actual: Help text omits
nornicandfalkordb-remote. - Impact: Users cannot discover remote backends from CLI help.
- Severity: Low
- Category: UX
- Backend(s): All
- Repro steps:
cgc bundle import file.cgc --clear(no TTY) - Expected: Non-interactive failure with clear message, or require
-yin docs prominently. - Actual: Hangs at
Are you sure you want to continue? [y/N]:until timeout. - Impact: CI/scripts hang;
--yes/-yexists but is easy to miss.
- Severity: Medium
- Category: Accuracy / Release parity
- Backend(s): falkordb (PyPI 0.4.16)
- Repro steps: Fresh isolated index per language; compare
MATCH (n)count totests/fixtures/goldens/*/metadata.json. - Expected: Node counts within 10% of goldens (same generator
PYv0.4.16). - Actual (fresh single-repo index):
| Project | Golden nodes | PyPI nodes | Δ% | Golden edges | PyPI edges | Status |
|---|---|---|---|---|---|---|
| sample_project | 482 | 482 | 0% | 621 | 625 | PASS |
| sample_project_go | 660 | 690 | 4.5% | 845 | 845 | PASS |
| sample_project_java | 160 | 195 | 21.9% | 221 | 221 | FAIL |
| sample_project_javascript | 236 | 236 | 0% | 306 | 304 | PASS |
| sample_project_rust | 803 | 854 | 6.4% | 943 | 939 | PASS |
| sample_project_typescript | 918 | 918 | 0% | 1465 | 1459 | PASS |
| sample_project_c | 83 | 83 | 0% | 96 | 96 | PASS |
| sample_project_kotlin | 189 | 206 | 9.0% | 242 | 242 | PASS |
| sample_project_cpp | 136 | 155 | 14.0% | 167 | 167 | FAIL |
| sample_project_csharp | 141 | 168 | 19.1% | 212 | 212 | FAIL |
| sample_project_dart | 49 | 88 | 79.6% | 64 | 64 | FAIL |
| sample_project_elixir | 53 | 93 | 75.5% | 81 | 81 | FAIL |
| sample_project_haskell | 45 | 87 | 93.3% | 50 | 50 | FAIL |
| sample_project_lua | 52 | 96 | 84.6% | 55 | 55 | FAIL |
| sample_project_misc | 27 | 73 | 170% | 26 | 26 | FAIL |
| sample_project_perl | 71 | 117 | 64.8% | 95 | 94 | FAIL |
| sample_project_php | 757 | 771 | 1.8% | 866 | 866 | PASS |
| sample_project_ruby | 74 | 127 | 71.6% | 106 | 106 | FAIL |
| sample_project_scala | 130 | 186 | 43.1% | 171 | 171 | FAIL |
| sample_project_swift | 136 | 192 | 41.2% | 178 | 178 | FAIL |
- Note: Edge counts match exactly for all tested languages; node inflation is predominantly extra
ParameterandVariablenodes. Suggests PyPI 0.4.16 indexes more node types than goldens captured, or goldens were generated from a different build artifact. - Impact: Golden-based CI on PyPI install will fail for 11/20 languages on node count despite correct edge topology.
| Area | Result |
|---|---|
cgc doctor on embedded backends |
PASS |
cgc query write guard (DELETE, CREATE) |
Blocked, exit 1 |
cgc index skip when already indexed |
PASS (exit 0, tip shown) |
Context global mode — two repos in cgc list |
PASS |
Context per-repo isolation — f1 not in Go-only repo |
PASS |
| Named context create/index/default | PASS |
| MCP tools/list | 25 tools |
MCP find_code vs CLI find name f1 |
Parity on function_chains.py |
MCP add_code_to_graph bad path |
Proper error (outside allowed roots) |
cgc watch incremental re-index |
PASS — new bar→foo caller detected |
cgc registry search numpy |
PASS |
| Datasource smoke (mysql/cassandra/redis bad host) | Clean errors, no traceback |
Bundle round-trip with -y |
482/625 nodes in 0.68s |
Neo4j misconfig exit codes (from /tmp, clean HOME) |
index/find exit 1 with clear message |
- CLI
--databasehelp missingnornic,falkordb-remote(BUG-010) - MCP tool count: docs reference 21 tools; runtime exposes 25
- Quickstart call-chain example (
f1→f2→f3) does not work with current call resolver (BUG-002/003) - API health probe should document
/api/v1/statusnot/health NEO4J_USERis not a validcgc config setkey; useNEO4J_USERNAME(discovered during exit-code testing)- Config precedence: local
.codegraphcontext/.envat cwd overrides global — underdocumented for global-mode users (BUG-001)
| Test | Reason |
|---|---|
| Neo4j remote index/query | Docker daemon not running (unix:///home/shashank/.docker/desktop/docker.sock missing) |
| FalkorDB-remote | Same — no Docker |
| Nornic | Same — no Docker |
cgc visualize memory/perf |
Not run (manual UI probe deferred) |
ENABLE_AUTO_WATCH=true post-index |
Not run (terminal block behavior deferred) |
| Check | falkordb | kuzudb | ladybugdb |
|---|---|---|---|
| Node count | 482 | 482 | 482 |
| Edge count | 625 | 625* | 625* |
find name f1 (function_chains) |
PASS | PASS | PASS |
find content "Hello World" |
PASS | PASS | PASS |
analyze chain f1 f3 |
FAIL | FAIL | FAIL |
| Query write block | PASS | PASS | PASS |
*Edge counts inferred from shared indexing pipeline; node counts verified via Cypher.
Local install: /tmp/cgc-local-venv via pip install -e . from commit dce24ed + 9 uncommitted files (cli_helpers.py, database_falkordb.py, database_kuzu.py, falkor_worker.py, api/router.py, scip_indexer.py, etc.).
| Bug / Check | PyPI 0.4.16 | Local editable | Notes |
|---|---|---|---|
| BUG-001 Config bleed | FAIL — repo .env overrides HOME; index hits dev graph (53k+ functions) |
Improved — isolated HOME + neo4j misconfig exits with error; no silent fallback to repo FalkorDB | Local cli_helpers.py changes likely involved |
| BUG-002 Function→Function CALLS | FAIL — only <module>→f1/f2/f3 |
FIXED — f1→f2, f2→f3 edges present |
Major accuracy fix in working tree |
BUG-003 analyze chain f1 f3 |
FAIL — "No call chain found" | FIXED — chain length 2, shows f1→f2→f3 | |
BUG-004 analyze callers f2 |
FAIL — reports <module> |
FIXED — reports f1 |
|
| BUG-005 Delete orphans | FAIL — 9 nodes remain | FAIL — 9 nodes remain | Same on both |
| BUG-006 MCP delete bad path | FAIL — NoneType.replace |
Improved — clean ALLOW_DB_DELETION guard message |
Different code path hit |
| BUG-007 Per-repo DB default | FAIL — auto-init uses falkordb | FAIL — same | |
BUG-008 API /health |
FAIL — 404 | FAIL — 404 (/api/v1/status → 200) |
|
| BUG-009 Cypher deprecation | FAIL — no warning | FIXED — ⚠ 'cgc cypher' is deprecated |
|
BUG-010 CLI --database help |
FAIL — missing nornic, falkordb-remote | FAIL — help still lists only falkordb/kuzudb/neo4j on index |
|
| BUG-012 Golden parity (20 langs) | FAIL — 11/20 node-count drift | PASS — 20/20 exact match (482/621 … 918/1465) | PyPI wheel ≠ repo goldens; local tree matches |
| Neo4j misconfig exit codes | PASS (exit 1) | PASS (exit 1) | Both OK from /tmp |
| MCP tool count | 25 | 25 | Same |
| Context isolation | PASS | PASS | Same |
| Check | Result |
|---|---|
| index sample_project | PASS — 482 nodes, 621 edges (matches golden exactly; PyPI had 625) |
analyze chain f1 f3 |
PASS |
analyze callers f2 |
PASS (caller: f1) |
bundle export/import -y |
PASS |
| 20-language golden sweep | 20/20 PASS — zero node/edge delta |
The working tree contains fixes not in the published PyPI 0.4.16 wheel:
- Nested/module-level call resolution (
f1→f2→f3CALLS edges) - Exact golden parity (edge count 621 vs PyPI's 625 on sample_project)
- Cypher deprecation warning
- Improved config / DB-init error handling (no silent repo-graph bleed)
Action: Cut a new PyPI release (0.4.17+) to ship these fixes to new users installing via pip install codegraphcontext.
- BUG-001: Document config precedence clearly; consider not loading repo-local
.envin global mode unless--contextor per-repo mode is active. - BUG-002/003: Fix call resolver for nested call expressions at module scope, or update docs to use intra-function call examples.
- BUG-005: Cascade-delete orphan Parameter/Variable nodes when removing a repository.
- BUG-012: Regenerate language goldens from PyPI 0.4.16 wheel, or pin golden generator to exact release artifact hash.
Report generated by manual E2E bug hunt session 2026-06-09. Local editable comparison added same day.