Describe the bug
When training with mult-GPU.
Setting sorting: random and sorting: ascending in hparams/*.yaml file show the same activity.
In case sorting setted ascending, batch dataset processed randomly in training.
Expected behaviour
When I set sorting=ascending, shorter data trained earlier.
To Reproduce
In yaml file, set as below
sorting: ascending
then ASR train!
It takes the same time when set sorting: random
Versions
v0.5.13
With v0.5.11, no problem appeared.
I can train with short data at first in v0.5.11.
Relevant log output
No response
Additional context
I tested and come to know there are some probelms in line 1000 of core.py.
1000 with tqdm(
train_set,
initial=self.step,
dynamic_ncols=True,
disable=not enable,
) as t:
for batch in t:
the train_set is sorted befor tqdm, but batch data is not sorted after tqdm().
Describe the bug
When training with mult-GPU.
Setting sorting: random and sorting: ascending in hparams/*.yaml file show the same activity.
In case sorting setted ascending, batch dataset processed randomly in training.
Expected behaviour
When I set sorting=ascending, shorter data trained earlier.
To Reproduce
In yaml file, set as below
sorting: ascending
then ASR train!
It takes the same time when set sorting: random
Versions
v0.5.13
With v0.5.11, no problem appeared.
I can train with short data at first in v0.5.11.
Relevant log output
No response
Additional context
I tested and come to know there are some probelms in line 1000 of core.py.
1000 with tqdm(
train_set,
initial=self.step,
dynamic_ncols=True,
disable=not enable,
) as t:
for batch in t:
the train_set is sorted befor tqdm, but batch data is not sorted after tqdm().