Skip to content

There is a problem with initializing torch.distribution.device_mesh.init_device_mesh #1

Open
Open
There is a problem with initializing torch.distribution.device_mesh.init_device_mesh#1

Description

@RealTapeL
RealTapeL
opened on May 31, 2025

This is my table-r1-sft.sh file. Since I only have one gpu, I set the nproc_per_node to 1:

#!/usr/bin/env bash
set -x


project_name='Table-R1'
exp_name='Table-R1-SFT-7B'
home_dir='Table-R1'
mkdir -p "Table-R1/log/$project_name/$exp_name"
save_path="Table-R1/checkpoints/$project_name/$exp_name"
mkdir -p $save_path

MODEL_PATH=/home/ps/Qwen3-4B
TRAIN_FILE="/home/ps/Table-R1/Table-R1-SFT-Dataset-filtered/data/table-r1-sft-dataset-filtered.parquet"
TEST_FILE="/home/ps/Table-R1/Table-R1-SFT-Dataset-filtered/data/table-r1-sft-dataset-filtered.parquet"

torchrun --standalone --nnodes=1 --nproc_per_node=1 \
     -m verl.trainer.fsdp_sft_trainer \
    data.train_batch_size=256 \
    data.max_length=20480 \
    data.train_files=$TRAIN_FILE \
    data.val_files=$TEST_FILE \
    data.prompt_key=extra_info \
    data.response_key=extra_info \
    data.prompt_dict_keys=['question'] \
    data.response_dict_keys=['answer'] \
    data.micro_batch_size_per_gpu=1 \
    model.partial_pretrain=$MODEL_PATH \
    ulysses_sequence_parallel_size=4 \
    use_remove_padding=True \
    trainer.default_local_dir=$save_path \
    trainer.project_name=$project_name \
    trainer.experiment_name=$exp_name \
    trainer.total_epochs=3 \
    trainer.logger=['console','wandb'] $@ 2>&1 | tee >(split -b 5M -d --additional-suffix=.log - "Table-R1/log/$project_name/$exp_name/")

But it reported an error:

(tabler1) ps@ps:~/Table-R1$ bash script/table-r1-sft.sh
+ export HYDRA_FULL_ERROR=1
+ HYDRA_FULL_ERROR=1
+ project_name=Table-R1
+ exp_name=Table-R1-SFT-7B
+ home_dir=Table-R1
+ mkdir -p Table-R1/log/Table-R1/Table-R1-SFT-7B
+ save_path=Table-R1/checkpoints/Table-R1/Table-R1-SFT-7B
+ mkdir -p Table-R1/checkpoints/Table-R1/Table-R1-SFT-7B
+ MODEL_PATH=/home/ps/Qwen3-4B
+ TRAIN_FILE=/home/ps/Table-R1/Table-R1-SFT-Dataset-filtered/data/table-r1-sft-dataset-filtered.parquet
+ TEST_FILE=/home/ps/Table-R1/Table-R1-SFT-Dataset-filtered/data/table-r1-sft-dataset-filtered.parquet
+ torchrun --standalone --nnodes=1 --nproc_per_node=1 -m verl.trainer.fsdp_sft_trainer data.train_batch_size=256 data.max_length=20480 data.train_files=/home/ps/Table-R1/Table-R1-SFT-Dataset-filtered/data/table-r1-sft-dataset-filtered.parquet data.val_files=/home/ps/Table-R1/Table-R1-SFT-Dataset-filtered/data/table-r1-sft-dataset-filtered.parquet data.prompt_key=extra_info data.response_key=extra_info 'data.prompt_dict_keys=[question]' 'data.response_dict_keys=[answer]' data.micro_batch_size_per_gpu=1 model.partial_pretrain=/home/ps/Qwen3-4B ulysses_sequence_parallel_size=4 use_remove_padding=True trainer.default_local_dir=Table-R1/checkpoints/Table-R1/Table-R1-SFT-7B trainer.project_name=Table-R1 trainer.experiment_name=Table-R1-SFT-7B trainer.total_epochs=3 'trainer.logger=[console,wandb]'
+ tee /dev/fd/63
++ split -b 5M -d --additional-suffix=.log - Table-R1/log/Table-R1/Table-R1-SFT-7B/
Error executing job with overrides: ['data.train_batch_size=256', 'data.max_length=20480', 'data.train_files=/home/ps/Table-R1/Table-R1-SFT-Dataset-filtered/data/table-r1-sft-dataset-filtered.parquet', 'data.val_files=/home/ps/Table-R1/Table-R1-SFT-Dataset-filtered/data/table-r1-sft-dataset-filtered.parquet', 'data.prompt_key=extra_info', 'data.response_key=extra_info', 'data.prompt_dict_keys=[question]', 'data.response_dict_keys=[answer]', 'data.micro_batch_size_per_gpu=1', 'model.partial_pretrain=/home/ps/Qwen3-4B', 'ulysses_sequence_parallel_size=4', 'use_remove_padding=True', 'trainer.default_local_dir=Table-R1/checkpoints/Table-R1/Table-R1-SFT-7B', 'trainer.project_name=Table-R1', 'trainer.experiment_name=Table-R1-SFT-7B', 'trainer.total_epochs=3', 'trainer.logger=[console,wandb]']
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/runpy.py", line 196, in _run_module_as_main
[rank0]:     return _run_code(code, main_globals, None,
[rank0]:   File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/runpy.py", line 86, in _run_code
[rank0]:     exec(code, run_globals)
[rank0]:   File "/home/ps/Table-R1/verl/verl/trainer/fsdp_sft_trainer.py", line 607, in <module>
[rank0]:     main()
[rank0]:   File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main
[rank0]:     _run_hydra(
[rank0]:   File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
[rank0]:     _run_app(
[rank0]:   File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
[rank0]:     run_and_report(
[rank0]:   File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
[rank0]:     raise ex
[rank0]:   File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
[rank0]:     return func()
[rank0]:   File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
[rank0]:     lambda: hydra.run(
[rank0]:   File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
[rank0]:     _ = ret.return_value
[rank0]:   File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
[rank0]:     raise self._return_value
[rank0]:   File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
[rank0]:     ret.return_value = task_function(task_cfg)
[rank0]:   File "/home/ps/Table-R1/verl/verl/trainer/fsdp_sft_trainer.py", line 572, in main
[rank0]:     ulysses_device_mesh = init_device_mesh(device_type=device_name, mesh_shape=(dp_size, config.ulysses_sequence_parallel_size), mesh_dim_names=("dp", "sp"))
[rank0]:   File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/torch/distributed/device_mesh.py", line 1003, in init_device_mesh
[rank0]:     device_mesh = DeviceMesh(
[rank0]:   File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/torch/distributed/device_mesh.py", line 453, in __init__
[rank0]:     self._init_process_groups()
[rank0]:   File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/torch/distributed/device_mesh.py", line 530, in _init_process_groups
[rank0]:     pg_ranks_by_dim = self.mesh.swapdims(-1, dim).reshape(
[rank0]: RuntimeError: cannot reshape tensor of 0 elements into shape [-1, 0] because the unspecified dimension size -1 can be any value and is ambiguous
[rank0]:[W601 00:14:51.175990163 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
E0601 00:14:51.756000 175130 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 0 (pid: 175188) of binary: /home/ps/miniconda3/envs/tabler1/bin/python
Traceback (most recent call last):
  File "/home/ps/miniconda3/envs/tabler1/bin/torchrun", line 33, in <module>
    sys.exit(load_entry_point('torch==2.6.0', 'console_scripts', 'torchrun')())
  File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
  File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/torch/distributed/run.py", line 918, in main
    run(args)
  File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/torch/distributed/run.py", line 909, in run
    elastic_launch(
  File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
verl.trainer.fsdp_sft_trainer FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-06-01_00:14:51
  host      : ps
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 175188)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

Later, I adjusted the parameters to fit my machine, but I encountered new problems that I couldn't solve at present:

(tabler1) ps@ps:~/Table-R1$ bash script/table-r1-sft.sh
+ project_name=Table-R1
+ exp_name=Table-R1-SFT-7B
+ home_dir=Table-R1
+ mkdir -p Table-R1/log/Table-R1/Table-R1-SFT-7B
+ save_path=Table-R1/checkpoints/Table-R1/Table-R1-SFT-7B
+ mkdir -p Table-R1/checkpoints/Table-R1/Table-R1-SFT-7B
+ MODEL_PATH=/home/ps/Qwen2.5-0.5B-Instruct
+ TRAIN_FILE=/home/ps/Table-R1/Table-R1-SFT-Dataset-filtered/data/table-r1-sft-dataset-filtered.parquet
+ TEST_FILE=/home/ps/Table-R1/Table-R1-SFT-Dataset-filtered/data/table-r1-sft-dataset-filtered.parquet
+ torchrun --standalone --nnodes=1 --nproc_per_node=1 -m verl.trainer.fsdp_sft_trainer data.train_batch_size=1 data.max_length=20480 data.train_files=/home/ps/Table-R1/Table-R1-SFT-Dataset-filtered/data/table-r1-sft-dataset-filtered.parquet data.val_files=/home/ps/Table-R1/Table-R1-SFT-Dataset-filtered/data/table-r1-sft-dataset-filtered.parquet data.prompt_key=extra_info data.response_key=extra_info 'data.prompt_dict_keys=[question]' 'data.response_dict_keys=[answer]' data.micro_batch_size_per_gpu=1 model.partial_pretrain=/home/ps/Qwen2.5-0.5B-Instruct ulysses_sequence_parallel_size=1 use_remove_padding=True trainer.default_local_dir=Table-R1/checkpoints/Table-R1/Table-R1-SFT-7B trainer.project_name=Table-R1 trainer.experiment_name=Table-R1-SFT-7B trainer.total_epochs=3 'trainer.logger=[console,wandb]'
+ tee /dev/fd/63
++ split -b 5M -d --additional-suffix=.log - Table-R1/log/Table-R1/Table-R1-SFT-7B/
/home/ps/Table-R1/verl/verl/utils/dataset/sft_dataset.py:82: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
/home/ps/Table-R1/verl/verl/utils/dataset/sft_dataset.py:82: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
/home/ps/Table-R1/verl/verl/utils/dataset/sft_dataset.py:82: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
/home/ps/Table-R1/verl/verl/utils/dataset/sft_dataset.py:82: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  ls = ls[0]
Normalize batch size by dp 1
Using sequence parallel size: 1
Using remove padding: True
Using FSDP rank 0 and size 1 for data distribution
Flash Attention 2.0 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen2ForCausalLM is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `torch_dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", torch_dtype=torch.float16)`
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
Monkey patch _flash_attention_forward in transformers.integrations.flash_attention
functools.partial(<function _or_policy at 0x7e01f524ee60>, policies=[functools.partial(<function transformer_auto_wrap_policy at 0x7e01f524ed40>, transformer_layer_cls={<class 'transformers.models.qwen2.modeling_qwen2.Qwen2DecoderLayer'>})])
NCCL version 2.21.5+cuda12.4
Number of steps/epoch 33601, number of epochs 3, total number of steps 100803
{'data': {'train_batch_size': 1, 'micro_batch_size': None, 'micro_batch_size_per_gpu': 1, 'train_files': '/home/ps/Table-R1/Table-R1-SFT-Dataset-filtered/data/table-r1-sft-dataset-filtered.parquet', 'val_files': '/home/ps/Table-R1/Table-R1-SFT-Dataset-filtered/data/table-r1-sft-dataset-filtered.parquet', 'prompt_key': 'extra_info', 'response_key': 'extra_info', 'prompt_dict_keys': ['question'], 'response_dict_keys': ['answer'], 'multiturn': {'enable': False, 'messages_key': 'messages'}, 'max_length': 20480, 'truncation': 'error', 'balance_dp_token': False, 'chat_template': None, 'custom_cls': {'path': None, 'name': None}, 'use_shm': False}, 'model': {'partial_pretrain': '/home/ps/Qwen2.5-0.5B-Instruct', 'use_shm': False, 'fsdp_config': {'wrap_policy': {'min_num_params': 0}, 'cpu_offload': False, 'offload_params': False}, 'external_lib': None, 'enable_gradient_checkpointing': False, 'trust_remote_code': False, 'lora_rank': 0, 'lora_alpha': 16, 'target_modules': 'all-linear', 'use_liger': False, 'strategy': 'fsdp2'}, 'optim': {'lr': 1e-05, 'betas': [0.9, 0.95], 'weight_decay': 0.01, 'warmup_steps_ratio': 0.1, 'clip_grad': 1.0, 'lr_scheduler': 'cosine'}, 'ulysses_sequence_parallel_size': 1, 'use_remove_padding': True, 'trainer': {'default_local_dir': 'Table-R1/checkpoints/Table-R1/Table-R1-SFT-7B', 'default_hdfs_dir': 'hdfs://tmp/experiments/gsm8k/gemma-1.1-7b-it/', 'resume_path': None, 'project_name': 'Table-R1', 'experiment_name': 'Table-R1-SFT-7B', 'total_epochs': 3, 'total_training_steps': None, 'logger': ['console', 'wandb'], 'seed': 1}}
Error executing job with overrides: ['data.train_batch_size=1', 'data.max_length=20480', 'data.train_files=/home/ps/Table-R1/Table-R1-SFT-Dataset-filtered/data/table-r1-sft-dataset-filtered.parquet', 'data.val_files=/home/ps/Table-R1/Table-R1-SFT-Dataset-filtered/data/table-r1-sft-dataset-filtered.parquet', 'data.prompt_key=extra_info', 'data.response_key=extra_info', 'data.prompt_dict_keys=[question]', 'data.response_dict_keys=[answer]', 'data.micro_batch_size_per_gpu=1', 'model.partial_pretrain=/home/ps/Qwen2.5-0.5B-Instruct', 'ulysses_sequence_parallel_size=1', 'use_remove_padding=True', 'trainer.default_local_dir=Table-R1/checkpoints/Table-R1/Table-R1-SFT-7B', 'trainer.project_name=Table-R1', 'trainer.experiment_name=Table-R1-SFT-7B', 'trainer.total_epochs=3', 'trainer.logger=[console,wandb]']
Traceback (most recent call last):
  File "/home/ps/Table-R1/verl/verl/trainer/fsdp_sft_trainer.py", line 583, in main
    trainer.fit()
  File "/home/ps/Table-R1/verl/verl/trainer/fsdp_sft_trainer.py", line 497, in fit
    tracking = Tracking(
  File "/home/ps/Table-R1/verl/verl/utils/tracking.py", line 54, in __init__
    wandb.init(project=project_name, name=experiment_name, config=config)
  File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1691, in init
    wandb._sentry.reraise(e)
  File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/wandb/analytics/sentry.py", line 156, in reraise
    raise exc.with_traceback(sys.exc_info()[2])
  File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1623, in init
    wi.maybe_login(init_settings)
  File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 208, in maybe_login
    wandb_login._login(
  File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/wandb/sdk/wandb_login.py", line 314, in _login
    key, key_status = wlogin.prompt_api_key(referrer=referrer)
  File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/wandb/sdk/wandb_login.py", line 242, in prompt_api_key
    raise UsageError("api_key not configured (no-tty). call " + directive)
wandb.errors.errors.UsageError: api_key not configured (no-tty). call wandb.login(key=[your_api_key])

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
[rank0]:[W601 01:04:33.444856305 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
E0601 01:04:34.651000 280355 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 0 (pid: 280421) of binary: /home/ps/miniconda3/envs/tabler1/bin/python
Traceback (most recent call last):
  File "/home/ps/miniconda3/envs/tabler1/bin/torchrun", line 33, in <module>
    sys.exit(load_entry_point('torch==2.6.0', 'console_scripts', 'torchrun')())
  File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
  File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/torch/distributed/run.py", line 918, in main
    run(args)
  File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/torch/distributed/run.py", line 909, in run
    elastic_launch(
  File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/ps/miniconda3/envs/tabler1/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
verl.trainer.fsdp_sft_trainer FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-06-01_01:04:34
  host      : ps
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 280421)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
(tabler1) ps@ps:~/Table-R1$

May I ask how I can solve it?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions