Conversation
[Sync] merge to main for 3.0.2 release - bug fixes; knowledge indexing; future compatibility & dependencies
Return `NaN` mAP/mAR when no ground-truth individuals qualify for OKS evaluation. This avoids computing undefined metrics in cases like single-bodypart projects, where OKS cannot be normalized because poses lack enough visible keypoints.
Forward the dataset single-animal setting from the training API into the pose training runner so epoch metrics are computed with the same code path as standalone evaluation. This keeps logged training metrics consistent for single-animal projects.
Replace the `logging.warning` in `compute_oks` with `warnings.warn(..., stacklevel=2)` when no ground-truth individual meets the OKS visibility requirement. This keeps the warning attached to the calling code path and makes the undefined mAP/mAR case more visible without changing the metric result.
Add a PyTorch evaluation API regression test covering CombinedEvaluation result updates when a snapshot is re-evaluated. The test ensures NaN metrics such as undefined mAP replace the old row instead of being backfilled from previous results, while untouched snapshots keep their existing scores.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Degenerate multi-keypoint ground truths can still produce invalid OKS values reported as valid zero scores.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Fixes single-animal training metrics and prevents invalid or stale evaluation results.
Changes:
- Propagates single-animal mode into training metrics.
- Reports undefined OKS metrics as NaN.
- Replaces stale combined-evaluation rows and adds regressions.
File summaries
| File | Description |
|---|---|
deeplabcut/core/metrics/distance_metrics.py |
Handles unavailable OKS metrics. |
deeplabcut/pose_estimation_pytorch/apis/evaluation.py |
Replaces prior evaluation rows. |
deeplabcut/pose_estimation_pytorch/apis/training.py |
Derives single-animal mode. |
deeplabcut/pose_estimation_pytorch/runners/train.py |
Propagates mode into metric computation. |
tests/core/metrics/test_metrics_api.py |
Tests zero versus undefined metrics. |
tests/pose_estimation_pytorch/apis/test_apis_evaluate.py |
Tests stale-score replacement. |
tests/pose_estimation_pytorch/runners/test_runners_train.py |
Tests runner flag propagation. |
Review details
- Files reviewed: 7/7 changed files
- Comments generated: 1
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This change guards OKS scoring when the ground-truth pose has no spatial extent. The scale is now computed in a dedicated helper, degenerate poses are filtered before AP/AR aggregation, and the metric stays undefined (NaN) when no OKS scale exists. A configurable bbox margin can still make an otherwise degenerate pose scoreable. Regression tests cover degenerate, margin-enabled, and mixed valid/invalid pose cases.
C-Achard
marked this pull request as ready for review
September 17, 2026 10:07
Collaborator
Author
|
Note: I think we could make this a bit more user friendly by preventing using invalid metrics as "best" snapshot selector for certain project types see #3521 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3518.