Skip to content

dNN training speed seems to be low: low GPU clock speed; however, full VRAM utilization #491

Description

@caniko

Operating system and DeepLabCut version
Windows 10, with an Anaconda Env, & DeepLabCut 2.x.
CUDA 10, tf 1.13.1
GTX 1080 TI

The problem
When I launch the training, the training starts, but it doesn't utilize my GPU that well. 1000 iterations took about 5-11 minutes(?). Here is my log.txt

I have reinstalled my drivers, including CUDA; rebuilt my conda environment. In the meantime, I should mention that the card does in fact draw a lot of power, and that it may in fact be utilized. I just can't see it on task manager.

How to Reproduce the problem
Steps to reproduce the behavior:
Run deeplabcut.train_network(path_config_file, gputouse=0)

Over view of my GPU usage
image

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions