New feature: TrackByDetection - real-time multiple objects tracker - #736
New feature: TrackByDetection - real-time multiple objects tracker #736ehsainit wants to merge 12 commits into
Conversation
There was a problem hiding this comment.
Hi @ehsainit an additional check point would be good to throw an error is it's not an appropriate tracker, i.e. if someone runs this on say two individual mice, this does not work, i.e.
Analyzing /Users/mwmathis/Desktop/two_white_mice_052820-SN-2020-05-28/videos/xxx_200000.h5
3811it [00:06, 624.46it/s]
Traceback (most recent call last):
File "/Users/mwmathis/Documents/DeepLabCut/deeplabcut/gui/analyze_videos.py", line 417, in convert2_tracklets
track_method=self.trackertypes.GetValue(),
File "/Users/mwmathis/Documents/DeepLabCut/deeplabcut/pose_estimation_tensorflow/predict_videos.py", line 1464, in convert_detections2tracklets
trackers = mot_tracker.track(bb)
File "/Users/mwmathis/Documents/DeepLabCut/deeplabcut/pose_estimation_tensorflow/lib/single_object_tracker.py", line 54, in track
self.tracks[i].skipped_frames = 0
IndexError: list index out of rangeI would suggest a warning immediately upon parsing the pickle file, i.e. throw an error that say "this is for single objects/animals only" i.e. 1 individual in the config.yaml (or such)
|
@MMathisLab oh yeah right ! i forgot about that thanks for the note :) |
|
will send you an email, thanks @ehsainit ! |
…causing a out an range errors
…causing a out an range errors
|
Hey @MMathisLab, It was a small bug in code that I fixed now - the tracker did actually with the videos you sent me. Could you test again and give me some feedback ? Thanks ! |
|
Hi @ehsainit so indeed it works to track multiple parts (and actually as it's code now must be multiple parts), but the "issue" is that each point is treated as a single object, i.e. the animals are not assembled, so that is why your tracker is highly similar to ours, but it does not do the "assembly" of the animal. Of course I think having a single tracker is very valuable, i.e. for instances where someone wants to track a "single point" per object, i.e. 1 point per mouse. This could be beneficial for fast tracking, etc.
Right now though, I guess the images you post above, aside for the mouse - which is with 2.2 - the single objects are not made with 2.2 (i.e. #736 (comment)), i.e it is made with pre-2.2 networks, etc (as related to this PR: #534). As currently this crashes. So, we will work on this part such that 1 point per multi-animal is accepted. |
|
Hi @MMathisLab, Indeed the flies were tracked using DLC1 and a different method to extract loc-maxima from the score map. What crashes currently ? let me know if I can help. |
|
Hi @ehsainit sorry for the long pause - right now the issue is that the PAFs assume a skeleton, but if I want to use this as a SO detector, this needs to be adapted in the code. I have some working code, but have yet to debug why the NN is crashing. in general the other trackers work on assembled animals, i.e. the general notion is that you (1) predict keypoints, (2) group together appropriate bps to assemble individual animals, and (3) track these individuals; without this you loose the animal ID. Thus, in your case - which is indeed a good case to have ;) - (1) prediction pose (2) track points (i.e. skipping assembly). This is really useful for tracking as you show above, i.e. 1 keypoint per animal/fly/mouse. here is some modified code you can try out, which allows you to create the training set efficiently, but still crashing at the NN step. i.e. use a multi-animal video, label only 1 point per def getpafgraph(cfg, printnames=True):
""" Auxiliary function that turns skeleton (list of connected bodypart pairs)
into a list of corresponding indices (with regard to the stacked multianimal/uniquebodyparts)
Convention: multianimalbodyparts go first!
"""
individuals, uniquebodyparts, multianimalbodyparts = extractindividualsandbodyparts(
cfg
)
# Attention this order has to be consistent (for training set creation, training, inference etc.)
bodypartnames = multianimalbodyparts + uniquebodyparts
lookupdict = {bodypartnames[j]: j for j in range(len(bodypartnames))}
if len(cfg["multianimalbodyparts"])==1:
partaffinityfield_graph = []
else:
if cfg["skeleton"] is None:
cfg["skeleton"] = []
# CHECKS if each bpt is connected to at least one other bpt
# TODO: check that there is a path leading from each (multi)bpt to each other (multi)bpt!
connected = set()
partaffinityfield_graph = []
for link in cfg["skeleton"]:
if link[0] in bodypartnames and link[1] in bodypartnames:
bp1 = int(lookupdict[link[0]])
bp2 = int(lookupdict[link[1]])
connected.add(bp1)
connected.add(bp2)
partaffinityfield_graph.append([bp1, bp2])
else:
print("Attention, parts do not exist!", link)
unconnected = set(range(len(multianimalbodyparts))).difference(connected)
if unconnected:
raise ValueError(
f'Unconnected {", ".join(multianimalbodyparts[i] for i in unconnected)}. '
f"For multi-animal projects, all multianimalbodyparts should be connected. "
f"Ideally there should be at least one (multinode) path from each multianimalbodyparts to each other multianimalbodyparts. "
f"Please verify the skeleton in the config.yaml."
)
if printnames:
graph2names(cfg, partaffinityfield_graph)
return partaffinityfield_graph |
|
Hi @ehsainit Hope you are well! Let me know if you have any questions, etc! |
|
Hey @MMathisLab, Regarding the NN crush upon setting to false (if The training is still in process right now (I am using my CPU to train) ... it is probably going to take few days :'), no NN crushes up until now ! |
|
Hi All! as you can see there have been a lot of code changes; but we do still want to support SO tracking, so it's slated for 2.2.1 ;). If you want to merge the new code and do so, great - but I also understand it's a big time commitment, so we are happy to do so (and that would be fast for us) in another PR. |
|
Hey Back ! Best wishes, |
Goal & workflow
The goal is to extend DLC to support real-time multiple instances-of-one-object tracking. For simplicity and efficiency, a traditional tracking by detection approach was introduced.
How does it work ?
Candidates detections will be collected from the score-maps, all of the candidates will be assigned as new births , each having a unique identity, Kalman filter instance initialized with the first detection , a constant signifying the amount of skipped frames because of no available detection and a unique color. Afterwards, the detection-track-association problem is solved using the Hungarian algorithm. Only motion-information (Mahalanobis distance) is used.
Next, the tracks will be maintained and updated using the matched detection, only if their motion metric value is reasonable (e.g. not too high), otherwise, they will be re-assigned. Detections with no existing tracks will be assigned as births.
Finally the matched tracks Kalman filter instances will be updated using their correspondent's detection. To deal with the unmatched tracks, we keep track using the a priori information that their Kalman filter instances hold. If they remain unmatched up tp certain number of frames they will be assigned as deaths.
Note: This was done in the course of my studies as a Bachelor's project in the Straw Lab at the University of Freiburg. The Straw Lab is interested in real-time flies tracking , so the cost metric & other stuff were specifically chosen with the regard to the input.
Steps:
At time step t, for a processed frame:
Evaluation Results
This feature was evaluated using 1, 3 and 10 flies offline due to lack of time (back then),
Videos proprieties :
Here are some of the results:
Trajectories plots of tracking process on one fly. The 4 subplots diagrams demonstrate the accuracy performs of Kalman filter prediction against the DLC1 detectors .while the top two subplots illustrate the detectors performance over 40 second video, the other two subplots shows how accurate the Kalman filtering was.
This figure demonstrate a trajectories comparison between the predictions (orange line) and the DLC1 detection (blue line) over 20 seconds video of 3 flies. Here we can see the x, y trajectories of one fly,
Illustration of the prediction performance of the Kalman filter upon missing measurements over a 10 frames sequence.
Here we can see the time complexity of the framework. the x-axis represent the time in second and the y-axis the number of tracked objects.
In Action