Skip to content

bug(indexer): cleanup_old_jobs deletes jobs that are still RUNNING — status updates silently no-op and check_job_status reports not_found mid-index #1537

Description

@Shashankss1205

core/jobs.py:133-138, invoked from server.py:778 every 50 requests

if (job.end_time or job.start_time) < cutoff_time:

Falling back to start_time is deliberate — it reaps jobs that crashed without ever setting end_time. But there is no status check, so a job that is genuinely still RUNNING and simply older than max_age_hours is evicted too.

Failure

Index a large monorepo that takes longer than the threshold (default 24h). Mid-run, a routine cleanup deletes the job record. Afterwards:

  • update_job calls silently no-op — they are guarded by if job_id in self.jobs, so the worker keeps running with its progress going nowhere.
  • check_job_status reports not_found for a job that is actively writing to the graph.

The user is told the job does not exist while it continues to mutate the database.

Suggested fix

Skip jobs whose status is RUNNING (or PENDING) when aging by start_time, and reap only terminal-state jobs plus jobs with no recent progress update. Tracking a last_update_time would let genuinely-hung jobs still be reclaimed without evicting healthy long-running ones.


Environment
Commit c0e0bed (main)
Python 3.12.3, Linux

Found during a source audit. The logic is unambiguous on reading; not reproduced, since triggering it needs a >24h job.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area: indexerParsing, discovery, resolution, persistencebugSomething isn't workingseverity: lowCosmetic, dead code, or a narrow-trigger correctness issue

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions