New feature: real-time multiple object tracking based DLC mode by ehsainit · Pull Request #534 · DeepLabCut/DeepLabCut

ehsainit · 2019-12-21T13:33:27Z

Goal & workflow

The goal is to extend DLC to support real-time multiple object tracking. For simplicity and efficiency, a traditional tracking by detection approach was introduced.

The DLC network detects a scale of high probabilities representing animals or look-alike objects, which allows modeling the score-map as a function with multiple local maxima.

Afterwards, candidates detections will be collected from the score-maps, all of the candidates will be assigned as new births , each having a unique identity, Kalman filter instance initialized with the first detection , a constant signifying the amount of skipped frames because of no available detection and a unique color. After that, the detection-track-association problem is solved using the Hungarian algorithm. Only motion-information (Mahalanobis distance) is used.
Next, the tracks will be maintained and updated using the matched detection, only if their motion metric value is reasonable (e.g. not too high), otherwise, they will be re-assigned. Detections with no existing tracks will be assigned as births.
Finally the matched tracks Kalman filter instances will be updated using their correspondent's detection. To deal with the unmatched tracks, we keep track using the a priori information that their Kalman filter instances hold. If they remain unmatched up tp certain number of frames they will be assigned as deaths.

Note: This was done in the course of my studies as a Bachelor's project in the Straw Lab at the University of Freiburg. The Straw Lab is interested in real-time flies tracking , so the cost metric & other stuff were specifically chosen with the regard to the input.

Glossary:

Detection : an instance of the class Detection, contains coordinates , height and width of a local maximum extracted from the score-map
Track : an instance of the class Track, contains information about an associated detection 's coordinates or predicted coordinate, associated color.
Birth : new track instance, i.e. new animal or object detected by the NN and was not associated with any existing tracks
Death : an existing track instance with a skipped frame number greater than a threshold, i.e. the animal or the object no longer detected. Therefore deleted.

Steps:

At time step t, for a processed frame:

DLC inference returns a score-map which contain probability information about the processed frame
Maximum and minimum filters are applied to extract the n objects with probability greater than a threshold (i.e. local maxima) & assigned as detections
Create tracks if the frame is the first or if no tracks exists
Data association: associate new detections with existing tracks, according to a cost metric, remove deaths, assign new births etc.
Feed relative-to-object-size bounding boxes of matches, predict coordinates for unmatches

Evaluation Results

This feature was evaluated using 1, 3 and 10 flies offline due to lack of time (back then),
Videos proprieties :

FPS : 25
Type : mp4
Length : 20s- 180s
GPU: ToDo

Here are some of the results:

Trajectories plots of tracking process on one fly. The 4 subplots diagrams demonstrate the accuracy performs of Kalman filter prediction against the DLC1 detectors .while the top two subplots illustrate the detectors performance over 40 second video, the other two subplots shows how accurate the Kalman filtering was.

This figure demonstrate a trajectories comparison between the predictions (orange line) and the DLC1 detection (blue line) over 20 seconds video of 3 flies. Here we can see the x, y trajectories of one fly,

Illustration of the prediction performance of the Kalman filter upon missing measurements over a 10 frames sequence.

Here we can see the time complexity of the framework. the x-axis represent the time in second and the y-axis the number of tracked objects.

In Action

Notes

The Kalman filter is a constant velocity model
Evaluation was done using only the initial input videos (See picture in header)
This feature merely tested on DLC2, as it was developed under DLC1

ToDos

More Testing, Evaluation on real-time cameras with different fps rates
Evaluation on different videos @AlexEMG
Better local maxima extraction - more reliable method
The Math for the transition noise covariance matrix of the Kalman filter is wrong, must be corrected

…based

MMathisLab · 2020-01-30T20:34:57Z

@ehsainit thanks for your PR. We will not accept code additions that do not conform to the headers/open-source nature of the code, ie.. files that say:

# Copyright 2019 by
 # Tabet Ehsainieh, ehsainit@informatik.uni-freiburg.de
 # All rights reserved.

need to be edited, thanks!

We also ask you provide a description, and why this is a useful feature for the DLC codebase. i.e.: https://github.com/AlexEMG/DeepLabCut/blob/master/docs/contribute.md

MMathisLab · 2020-01-30T22:45:14Z

 # [DEMO the code](/examples)

 We provide several Jupyter Notebooks: one that walks you through a demo dataset to test your installation, and another Notebook to run DeepLabCut from the beginning on your own data. We also show you how to use the code in Docker, and on Google Colab.
+# Update: DeepLabCut with new feature: Real-Time MOT tracking 


remove from readme.md -> would go into release note, and docs for use.

MMathisLab · 2020-01-30T23:08:17Z

Hi @ehsainit - happy to review. Could you please update your original post such that it states the purpose of the request, what features it adds, and some performance metrics. i.e. here is a good example PR: #409

ehsainit · 2020-01-30T23:25:18Z

Hi @ehsainit - happy to review. Could you please update your original post such that it states the purpose of the request, what features it adds, and some performance metrics. i.e. here is a good example PR: #409

Hi @MMathisLab - thanks for the reply ! i sent to you a document with detailed description of this addition a while back.

MMathisLab · 2020-01-30T23:32:10Z

Hi @ehsainit yep! But given this is in the public domain now, I think it would be good to document this here, and specifically point to the code edits, etc.

ehsainit · 2020-01-30T23:38:51Z

@MMathisLab , Got it ! i will do so in the upcoming days.
Thanks!

ehsainit · 2020-03-16T19:48:28Z

Hi @MMathisLab @AlexEMG,
This is my first pull request, so I am not sure if you received notification last time I updated the description of the PR. I would really appreciate if you could give me some feedback !

MarineChap · 2020-03-16T19:57:41Z

Hi @ehsainit
I am not part of the project but I was interesting to use your feature for our lab. I already using the actual tool.
I can test it (howevere in offline simulated online because of coronavirus) and give you a feedback within 2 days hopefully, if @MMathisLab and @AlexEMG are interested about it.
In waiting their answer, can you give me some tips on how run it, please ?

ehsainit · 2020-03-16T20:23:41Z

Hi @MarineChap

Thanks for your interest in the feature, sure feel free to use.
I would appreciate some feedback

Note: the size of the bounding boxes might be to small, you might need to make it bigger

https://github.com/ehsainit/DeepLabCut/blob/1dc088f1e5ab4164fc646d5f1129437a675038bc/deeplabcut/lmot/assets.py#L25

AlexEMG · 2020-03-16T20:37:24Z

Dear Marine, Yes, please check it out!

…

________________________________ From: Marine Chaput <notifications@github.com> Sent: Monday, March 16, 2020 3:57:58 PM To: AlexEMG/DeepLabCut <DeepLabCut@noreply.github.com> Cc: Mathis, Alexander Thomas <amathis@fas.harvard.edu>; Mention <mention@noreply.github.com> Subject: Re: [AlexEMG/DeepLabCut] New feature: real-time multiple object tracking based DLC mode (#534) Hi @ehsainit<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ehsainit&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=11wEEDBv3Ke3n3b8dICjuQC5vgZ23dfGPax018VOZ2g&m=VvKAVG0deSVnt-O3w8mDPS4sNtv8aeSF3TdNWkG5NtQ&s=k550dKGiws2LhghgysnMTHOOuUUw_C5FIXXRp9I2s1w&e=> I am not part of the project but I was interesting to use your feature for our lab. I already using the actual tool. I can test it (howevere in offline simulated online because of coronavirus) and give you a feedback within 2 days hopefully, if @MMathisLab<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_MMathisLab&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=11wEEDBv3Ke3n3b8dICjuQC5vgZ23dfGPax018VOZ2g&m=VvKAVG0deSVnt-O3w8mDPS4sNtv8aeSF3TdNWkG5NtQ&s=uDS7KWq9KUYyVsEw6LwXVTyFJEok4BI7T6_0IuXDImY&e=> and @AlexEMG<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_AlexEMG&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=11wEEDBv3Ke3n3b8dICjuQC5vgZ23dfGPax018VOZ2g&m=VvKAVG0deSVnt-O3w8mDPS4sNtv8aeSF3TdNWkG5NtQ&s=fRuAGBq_spWyR_V9D88sdUf_BItDMoYKzQx1VoJSFZg&e=> are interested about it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_AlexEMG_DeepLabCut_pull_534-23issuecomment-2D599732097&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=11wEEDBv3Ke3n3b8dICjuQC5vgZ23dfGPax018VOZ2g&m=VvKAVG0deSVnt-O3w8mDPS4sNtv8aeSF3TdNWkG5NtQ&s=sOPq2GLNmSPm5obSrPkSaPh0VgfWUX9HKchr1Xt9ZHM&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AE7CMXVN7O2YSBLXC4GE6TDRH2AENANCNFSM4J6HDCDQ&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=11wEEDBv3Ke3n3b8dICjuQC5vgZ23dfGPax018VOZ2g&m=VvKAVG0deSVnt-O3w8mDPS4sNtv8aeSF3TdNWkG5NtQ&s=WBWN6fk0Wa2Yrjsk4c6jOkyCnpwQiVrvCfEi_2dIOMA&e=>.

MarineChap · 2020-03-16T21:28:51Z

Nice ! @ehsainit, First of all, it is unclear how to run it. ;)
Can you confirm me this, please ?
I should run :
analyze_image(config, vid, shuffle=1, trainingsetindex=0, gputouse=None, save_as_csv=False)

The config file is the same as in the trained model ? No need to change anything ?
vid : filename or flux video from the camera ?
What is your advice for setting :
- shuffle
- trainingsetindex

MarineChap · 2020-03-16T21:56:55Z

                               [scmap[maxloc][joint_idx]])))
    return np.array(pose)

+


This function is defined twice --> See two methods below

The is intended as I suggested before I still don't know how this feature will be integrated into the DLC. For your testing purposes use https://github.com/ehsainit/DeepLabCut/blob/c34b849999a497f81675f9c27ef4d212e71c7183/deeplabcut/pose_estimation_tensorflow/nnet/predict.py#L78

ehsainit · 2020-03-16T22:04:03Z

Nice ! @ehsainit, First of all, it is unclear how to run it. ;)
Can you confirm me this, please ?
I should run :
analyze_image(config, vid, shuffle=1, trainingsetindex=0, gputouse=None, save_as_csv=False)

The config file is the same as in the trained model ? No need to change anything ?

vid : filename or flux video from the camera ?

What is your advice for setting :

shuffle

trainingsetindex

@MarineChap you are right that is because it is still unclear how this feature should be integrated in the DLC. But yeah for the initial purposes i ran it using the main.py class as you guessed

For the shuffle, trainingsetindex parameters it depends on your trained network. for me it was the default values

MarineChap · 2020-03-17T08:25:24Z

Hello !
I ran it this night just with default values. I had zero detection and definitely not in real-time. Seems a heavy computation. What did you use to run it in your video ? CPU/GPU ?

My video is a rodent with 2 leds on the head running in a maze. The model is trained to detect these two LEDs. I changed the threshold in extract_localmaxima for 0.1. Does it something you were expecting to change following the type of application ?
At this point, I am having some possible_objects but I having a lot of bugs because for a unknow reason my detection are in 3d [dx, dy, dz] while the whole code is set up for 2d dimension. Do you have an idea why I have this third dimension?

Some remarks :

It would be nice to implement the "fix path" feature already present in the other deeplabcut steps.
I am unsure about the output location. I would prefer if the output is automatically next to the model that I am using in a folder with a logical name rather than dump alone where I am running it. It would make sense you follow the same rule about this than the other folders naming in deeplabcut.

Have a good day !

ehsainit · 2020-03-17T17:55:27Z

Hello !
I ran it this night just with default values. I had zero detection and definitely not in real-time. Seems a heavy computation. What did you use to run it in your video ? CPU/GPU ?

My video is a rodent with 2 leds on the head running in a maze. The model is trained to detect these two LEDs. I changed the threshold in extract_localmaxima for 0.1. Does it something you were expecting to change following the type of application ?
At this point, I am having some possible_objects but I having a lot of bugs because for a unknow reason my detection are in 3d [dx, dy, dz] while the whole code is set up for 2d dimension. Do you have an idea why I have this third dimension?

Some remarks :

It would be nice to implement the "fix path" feature already present in the other deeplabcut steps.

I am unsure about the output location. I would prefer if the output is automatically next to the model that I am using in a folder with a logical name rather than dump alone where I am running it. It would make sense you follow the same rule about this than the other folders naming in deeplabcut.

Have a good day !

Hey @MarineChap ,

Thanks for your feedback. I remember testing the model on CPU and i was really slow mainly in inference of DLC1 stage. However in GPU mode it was quick fast, did you run it using the GPU ?

I still not convinced of the way I'm extracting the local maxima from the score-maps but this method achieved the best results on the tested videos, i did expect this, yet it is surprising that you change it to 0.1(too low). My guess is that you defined more than one label and the wrong label is extracted ?
Could you please provide your trained network configuration file ?

About the 3D errors could you should you error code ?

Thanks !

MarineChap · 2020-03-17T19:23:32Z

  File "/home/chaput/Documents/DeepLabCut/deeplabcut/lmot/extract_local_maxima.py", line 19, in extract_locmaxima
    for dx, dy in possible_objects:
ValueError: too many values to unpack (expected 2)

I am having this error mainly because possible_objects = [slice(...), slice(...), slice(...)] and not what I am guessing it was for you possible_objects = [slice(...), slice(...)]

Also, indeed, you are right. As I am confined at my home, I am using an old model where I have 4 labels (2 useless which are computing nothing) instead of 2. What I don't understand is why it would cause a problem?
For the threshold, I did not know how to choose so I use the mean value of my scoremap. But maybe I should do the same while removing the 2 not in use labels.

Also because I am stuck at home, I have only my laptop so no GPU. But, from my point of view, a tracking should be able to work without a huge requirement of computation. People won't use it if they need others stuffs than there usual computer. It is not like offline computation which can be send in a cluster easily. But it is only my opinion. So... @MMathisLab what do you think on this subject ? A problem or not a problem the need of GPU for real-time tracking?

MMathisLab · 2020-03-17T19:26:19Z

I think for best tracking performance utilizing GPUs is important. That being said, one can do this inexpensively, and indeed we will release information how to do this soon. I think this code is not sounding as flexible as one needs, but I’ll need to re-evaluate once our roadmap (see docs) items are out.

ehsainit · 2020-03-17T20:35:22Z

  File "/home/chaput/Documents/DeepLabCut/deeplabcut/lmot/extract_local_maxima.py", line 19, in extract_locmaxima
    for dx, dy in possible_objects:
ValueError: too many values to unpack (expected 2)
I am having this error mainly because possible_objects = [slice(...), slice(...), slice(...)] and not what I am guessing it was for you possible_objects = [slice(...), slice(...)]

Also, indeed, you are right. As I am confined at my home, I am using an old model where I have 4 labels (2 useless which are computing nothing) instead of 2. What I don't understand is why it would cause a problem?
For the threshold, I did not know how to choose so I use the mean value of my scoremap. But maybe I should do the same while removing the 2 not in use labels.

Also because I am stuck at home, I have only my laptop so no GPU. But, from my point of view, a tracking should be able to work without a huge requirement of computation. People won't use it if they need others stuffs than there usual computer. It is not like offline computation which can be send in a cluster easily. But it is only my opinion. So... @MMathisLab what do you think on this subject ? A problem or not a problem the need of GPU for real-time tracking?

@MarineChap well, this error is caused by the score-map structure. When tracking one type of object you don't need multiple labels. the score-map that is extracted from network include 4 "probability tables" one for each label, in our case you need just one. Do you still remember which color did you use to label the images ? if so just add those two lines of code
scmap= scmap[:, :, -number of label-]
locref = locref[:, :, -number of label-]
change -number of label- to the correspond label (2 for red I guess) the here :
https://github.com/ehsainit/DeepLabCut/blob/c34b849999a497f81675f9c27ef4d212e71c7183/deeplabcut/lmot/extract_local_maxima.py#L9

I agree with @MMathisLab on the GPU part. The heavy computational was mostly because of the network computation. @MMathisLab i think the code is flexible but still not integrated in DLC, I assumed that there will be an option in the GUI later on for training for the purpose of real-time tracking (using only one label).

AlexEMG · 2020-05-22T13:36:23Z

This will be integrated with DLC 2.2 with a new PR. Closing this thread here.

ehsainit and others added 4 commits December 21, 2019 14:15

new feature: adding real-time multiple object tracking DLC-inference-…

fb8f286

…based

new feature: adding real-time multiple object tracking DLC-inference-…

a0cb92d

…based

updating copy rights

704c463

Merge branch 'master' into master

a6a0fab

AlexEMG self-requested a review January 4, 2020 00:26

AlexEMG self-assigned this Jan 4, 2020

Update README.md

3a587a7

removing headers

dd3d669

MMathisLab reviewed Jan 30, 2020

View reviewed changes

README.md remove line

1dc088f

ehsainit requested a review from MMathisLab January 30, 2020 23:01

ehsainit changed the title ~~New feature: adding real-time multiple object tracking based DLC mode~~ New feature: real-time multiple object tracking based DLC mode Feb 22, 2020

general notes and information

c34b849

MarineChap reviewed Mar 16, 2020

View reviewed changes

fix space

fbdede4

MMathisLab reviewed Mar 21, 2020

View reviewed changes

Comment thread deeplabcut/lmot/main.py

MMathisLab added the enhancement New feature or request label Mar 21, 2020

AlexEMG closed this May 22, 2020

MMathisLab mentioned this pull request Jun 23, 2020

New feature: TrackByDetection - real-time multiple objects tracker #736

Closed

Uh oh!

Uh oh!

Conversation

ehsainit commented Dec 21, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Goal & workflow

Glossary:

Steps:

Evaluation Results

In Action

Notes

ToDos

Uh oh!

MMathisLab commented Jan 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MMathisLab Jan 30, 2020

Choose a reason for hiding this comment

Uh oh!

MMathisLab commented Jan 30, 2020

Uh oh!

ehsainit commented Jan 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MMathisLab commented Jan 30, 2020

Uh oh!

ehsainit commented Jan 30, 2020

Uh oh!

ehsainit commented Mar 16, 2020

Uh oh!

MarineChap commented Mar 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ehsainit commented Mar 16, 2020

Uh oh!

AlexEMG commented Mar 16, 2020 via email

Uh oh!

MarineChap commented Mar 16, 2020

Uh oh!

MarineChap Mar 16, 2020

Choose a reason for hiding this comment

Uh oh!

ehsainit Mar 16, 2020

Choose a reason for hiding this comment

Uh oh!

ehsainit commented Mar 16, 2020

Uh oh!

MarineChap commented Mar 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ehsainit commented Mar 17, 2020

Uh oh!

MarineChap commented Mar 17, 2020

Uh oh!

MMathisLab commented Mar 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ehsainit commented Mar 17, 2020

Uh oh!

Uh oh!

AlexEMG commented May 22, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ehsainit commented Dec 21, 2019 •

edited

Loading

MMathisLab commented Jan 30, 2020 •

edited

Loading

ehsainit commented Jan 30, 2020 •

edited

Loading

MarineChap commented Mar 16, 2020 •

edited

Loading

MarineChap commented Mar 17, 2020 •

edited

Loading

MMathisLab commented Mar 17, 2020 •

edited

Loading