Segmentation fault (core dumped)

#55
by zhouxinxin - opened

I install the whole pip packages and fix all env prombles, but error in decoding , the whole log is :

[NeMo I 2025-07-10 15:58:29 mixins:181] Tokenizer SentencePieceTokenizer initialized with 1024 tokens
[NeMo W 2025-07-10 15:58:29 modelPT:176] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
Train config :
use_lhotse: true
skip_missing_manifest_entries: true
input_cfg: null
tarred_audio_filepaths: null
manifest_filepath: null
sample_rate: 16000
shuffle: true
num_workers: 2
pin_memory: true
max_duration: 40.0
min_duration: 0.1
text_field: answer
batch_duration: null
use_bucketing: true
bucket_duration_bins: null
bucket_batch_size: null
num_buckets: 30
bucket_buffer_size: 20000
shuffle_buffer_size: 10000

[NeMo W 2025-07-10 15:58:29 modelPT:183] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s).
Validation config :
use_lhotse: true
manifest_filepath: null
sample_rate: 16000
batch_size: 16
shuffle: false
max_duration: 40.0
min_duration: 0.1
num_workers: 2
pin_memory: true
text_field: answer

[NeMo I 2025-07-10 15:58:29 features:305] PADDING: 0
[NeMo I 2025-07-10 15:58:32 rnnt_models:226] Using RNNT Loss : tdt
Loss tdt_kwargs: {'fastemit_lambda': 0.0, 'clamp': -1.0, 'durations': [0, 1, 2, 3, 4], 'sigma': 0.02, 'omega': 0.1}
[NeMo I 2025-07-10 15:58:32 rnnt_models:226] Using RNNT Loss : tdt
Loss tdt_kwargs: {'fastemit_lambda': 0.0, 'clamp': -1.0, 'durations': [0, 1, 2, 3, 4], 'sigma': 0.02, 'omega': 0.1}
[NeMo W 2025-07-10 15:58:32 tdt_loop_labels_computer:281] No conditional node support for Cuda.
Cuda graphs with while loops are disabled, decoding speed will be slower
Reason: Driver supports cuda toolkit version 12.2, but the driver needs to support at least 12,3. Please update your cuda driver.
[NeMo I 2025-07-10 15:58:32 rnnt_models:226] Using RNNT Loss : tdt
Loss tdt_kwargs: {'fastemit_lambda': 0.0, 'clamp': -1.0, 'durations': [0, 1, 2, 3, 4], 'sigma': 0.02, 'omega': 0.1}
[NeMo W 2025-07-10 15:58:32 tdt_loop_labels_computer:281] No conditional node support for Cuda.
Cuda graphs with while loops are disabled, decoding speed will be slower
Reason: Driver supports cuda toolkit version 12.2, but the driver needs to support at least 12,3. Please update your cuda driver.
[NeMo I 2025-07-10 15:58:34 save_restore_connector:275] Model EncDecRNNTBPEModel was successfully restored from /root/.cache/huggingface/hub/models--nvidia--parakeet-tdt-0.6b-v2/snapshots/d97f7ac5d85e7185b7a7c4771c883c0e26d1d16f/parakeet-tdt-0.6b-v2.nemo.
Transcribing: 0%| | 0/1 [00:00<?, ?it/s]Segmentation fault (core dumped)

I don't know how to check and solve it, plz...

Hi, I also encountered a segmentation fault, but I was able to resolve it with this setting.

NeMo TDT Environment Setup

System Information

  • GPU: NVIDIA A10G (22.1 GB VRAM)
  • Driver Version: 580.95.05
  • CUDA Version: 11.8
  • OS: Ubuntu 24.04 LTS
  • Python: 3.10.19 (conda environment)

Installed Components

Core Dependencies

  • NVIDIA Driver: 580.95.05 (via driver 550 metapackage)
  • CUDA Toolkit: 11.8.89
  • Miniconda: Latest (Python 3.13 base)
  • Conda Environment: nemo_tdt (Python 3.10.19)

Python Packages

  • PyTorch: 2.5.1 with CUDA 11.8 support
  • Numba: 0.57.1 (FP16 support for TDT)
  • NeMo Toolkit: 2.5.0
  • PyTorch Lightning: 2.0.7
  • Lightning: 2.4.0
  • Transformers: 4.53.3
  • Huggingface Hub: 0.23.0 (downgraded for NeMo compatibility)
  • cuda-python: 11.8.3 (CRITICAL: Must match CUDA 11.8)

Sign up or log in to comment