Skip to content

fix sorting bug #1730

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
TParcollet merged 6 commits into from
Dec 2, 2022
Merged

fix sorting bug #1730

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
2e6c1df
hotfix
anautsch Nov 28, 2022
5d8fe3e
lints
anautsch Nov 28, 2022
c9394a6
shuffle to false
anautsch Nov 28, 2022
ddd2f74
tests for fixing ddp sampling w/ sorting
anautsch Nov 30, 2022
3ac910c
lints & naming
anautsch Nov 30, 2022
0d71b21
adjusted test
anautsch Dec 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 25 additions & 7 deletions speechbrain/core.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -739,6 +739,14 @@ def _train_loader_specifics(self, dataset, loader_kwargs):

# Possibly make a DistributedSampler or a wrapper for some other sampler
if self.distributed_launch and not isinstance(dataset, IterableDataset):
# sort or not
if hasattr(self.hparams, "sorting"):
shuffle_ddp = (
self.hparams.sorting == "random"
) # False if 'ascending' or 'descending'
else:
shuffle_ddp = True

drop_last = loader_kwargs.get("drop_last", False)
# num_replicas arg is equal to world_size
# and retrieved automatically within
Expand All @@ -757,7 +765,10 @@ def _train_loader_specifics(self, dataset, loader_kwargs):
elif loader_kwargs.get("batch_sampler") is None:
# no sampler and batch-sampler
self.train_sampler = DistributedSampler(
dataset, rank=self.rank, shuffle=True, drop_last=drop_last
dataset,
rank=self.rank,
shuffle=shuffle_ddp,
drop_last=drop_last,
)

# with DistributedSamplerWrapper, one must disable shuffling for dataloader
Expand All @@ -767,7 +778,7 @@ def _train_loader_specifics(self, dataset, loader_kwargs):
self.train_sampler = DistributedSamplerWrapper(
loader_kwargs.get("batch_sampler", None),
rank=self.rank,
shuffle=True,
shuffle=shuffle_ddp,
)
loader_kwargs["batch_sampler"] = self.train_sampler
elif self.distributed_launch and isinstance(dataset, IterableDataset):
Expand Down Expand Up @@ -1200,11 +1211,18 @@ def _wrap_distributed(self):
for name, module in self.modules.items():
if any(p.requires_grad for p in module.parameters()):
module = SyncBatchNorm.convert_sync_batchnorm(module)
module = DDP(
module,
device_ids=[self.device],
find_unused_parameters=self.find_unused_parameters,
)
if self.distributed_backend == "gloo":
Comment thread
TParcollet marked this conversation as resolved.
module = DDP(
module,
device_ids=None,
find_unused_parameters=self.find_unused_parameters,
)
else:
module = DDP(
module,
device_ids=[self.device],
find_unused_parameters=self.find_unused_parameters,
)
self.modules[name] = module
else:
# data_parallel_backend
Expand Down
11 changes: 6 additions & 5 deletions speechbrain/utils/distributed.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,11 +126,12 @@ def ddp_init_group(run_opts):
"--distributed_backend=nccl"
)
else:
if run_opts["local_rank"] + 1 > torch.cuda.device_count():
raise ValueError(
"Killing process " + str() + "\n"
"Not enough GPUs available!"
)
if not run_opts["distributed_backend"] == "gloo":

@TParcollet TParcollet Dec 1, 2022

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, why is gloo treated like that? :p

@anautsch anautsch Dec 2, 2022

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CPU vs GPU - there is no GPU testing on github, or pay for it :p

@anautsch anautsch Dec 2, 2022 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the way it is implemented for GPU is incompatible with CPU: gloo will crash when run in cpu only.

This pytorch example for gloo is incompatible with SB, as it was before the contributed fix
https://pytorch.org/tutorials/intermediate/dist_tuto.html

@anautsch anautsch Dec 2, 2022

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for this part, if there is only CPU, ofc there are not enough GPUs ;)

if run_opts["local_rank"] + 1 > torch.cuda.device_count():
raise ValueError(
"Killing process " + str() + "\n"
"Not enough GPUs available!"
)
if "RANK" in os.environ is None or os.environ["RANK"] == "":
raise ValueError(
"To use DDP backend, start your script with:\n\t"
Expand Down
56 changes: 56 additions & 0 deletions tests/integration/sampling/asc.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# target of this test
sorting: ascending
avoid_if_longer_than: 6.283

# below from: tests/integration/VAD/hyperparams.yaml
# Seed needs to be set at top of yaml, before objects with parameters are made
seed: 1234
__set_seed: !apply:torch.manual_seed [!ref <seed>]
sample_rate: 16000


# Training params
N_epochs: 1
lr: 0.01
dataloader_options:
batch_size: 20

# Feature parameters
n_mfcc: 20

# Model parameters
rnn_layers: 2
rnn_neurons: 256
emb_size: 23
dropout: 0.1
output_neurons: 1

compute_features: !new:speechbrain.lobes.features.MFCC
n_mfcc: !ref <n_mfcc>

mean_var_norm: !new:speechbrain.processing.features.InputNormalization
norm_type: global

rnn: !new:speechbrain.nnet.RNN.LSTM
input_size: !ref <n_mfcc> * 33 # d & dd = *3, 5 left & 5 right = *11
hidden_size: !ref <rnn_neurons>
num_layers: !ref <rnn_layers>
dropout: !ref <dropout>
bidirectional: False
re_init: True

lin: !new:speechbrain.nnet.linear.Linear
input_size: !ref <rnn_neurons>
n_neurons: !ref <output_neurons>
bias: False

modules:
compute_features: !ref <compute_features>
rnn: !ref <rnn>
lin: !ref <lin>
mean_var_norm: !ref <mean_var_norm>

opt_class: !name:torch.optim.Adam
lr: !ref <lr>

compute_loss: !name:speechbrain.nnet.losses.bce_loss
56 changes: 56 additions & 0 deletions tests/integration/sampling/dsc.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# target of this test
sorting: descending
avoid_if_longer_than: 6.283

# below from: tests/integration/VAD/hyperparams.yaml
# Seed needs to be set at top of yaml, before objects with parameters are made
seed: 1234
__set_seed: !apply:torch.manual_seed [!ref <seed>]
sample_rate: 16000


# Training params
N_epochs: 1
lr: 0.01
dataloader_options:
batch_size: 20

# Feature parameters
n_mfcc: 20

# Model parameters
rnn_layers: 2
rnn_neurons: 256
emb_size: 23
dropout: 0.1
output_neurons: 1

compute_features: !new:speechbrain.lobes.features.MFCC
n_mfcc: !ref <n_mfcc>

mean_var_norm: !new:speechbrain.processing.features.InputNormalization
norm_type: global

rnn: !new:speechbrain.nnet.RNN.LSTM
input_size: !ref <n_mfcc> * 33 # d & dd = *3, 5 left & 5 right = *11
hidden_size: !ref <rnn_neurons>
num_layers: !ref <rnn_layers>
dropout: !ref <dropout>
bidirectional: False
re_init: True

lin: !new:speechbrain.nnet.linear.Linear
input_size: !ref <rnn_neurons>
n_neurons: !ref <output_neurons>
bias: False

modules:
compute_features: !ref <compute_features>
rnn: !ref <rnn>
lin: !ref <lin>
mean_var_norm: !ref <mean_var_norm>

opt_class: !name:torch.optim.Adam
lr: !ref <lr>

compute_loss: !name:speechbrain.nnet.losses.bce_loss
Loading