MCTCT

M-CTC-T

This model is in maintenance mode only, so we won’t accept any new PRs changing its code.

If you run into any issues running this model, please reinstall the last version that supported this model: v4.30.0. You can do so by running the following command: pip install -U transformers==4.30.0.

Overview

The M-CTC-T model was proposed in Pseudo-Labeling For Massively Multilingual Speech Recognition by Loren Lugosch, Tatiana Likhomanenko, Gabriel Synnaeve, and Ronan Collobert. The model is a 1B-param transformer encoder, with a CTC head over 8065 character labels and a language identification head over 60 language ID labels. It is trained on Common Voice (version 6.1, December 2020 release) and VoxPopuli. After training on Common Voice and VoxPopuli, the model is trained on Common Voice only. The labels are unnormalized character-level transcripts (punctuation and capitalization are not removed). The model takes as input Mel filterbank features from a 16Khz audio signal.

The abstract from the paper is the following:

Semi-supervised learning through pseudo-labeling has become a staple of state-of-the-art monolingual speech recognition systems. In this work, we extend pseudo-labeling to massively multilingual speech recognition with 60 languages. We propose a simple pseudo-labeling recipe that works well even with low-resource languages: train a supervised multilingual model, fine-tune it with semi-supervised learning on a target language, generate pseudo-labels for that language, and train a final model using pseudo-labels for all languages, either from scratch or by fine-tuning. Experiments on the labeled Common Voice and unlabeled VoxPopuli datasets show that our recipe can yield a model with better performance for many languages that also transfers well to LibriSpeech.

This model was contributed by cwkeam. The original code can be found here.

Documentation resources

Tips:

  • The PyTorch version of this model is only available in torch 1.9 and higher.

MCTCTConfig

class transformers.MCTCTConfig

<source>

( vocab_size = 8065hidden_size = 1536num_hidden_layers = 36intermediate_size = 6144num_attention_heads = 4attention_head_dim = 384max_position_embeddings = 920layer_norm_eps = 1e-05layerdrop = 0.3hidden_act = 'relu'initializer_range = 0.02hidden_dropout_prob = 0.3attention_probs_dropout_prob = 0.3pad_token_id = 1bos_token_id = 0eos_token_id = 2conv_glu_dim = 1conv_dropout = 0.3num_conv_layers = 1conv_kernel = (7,)conv_stride = (3,)input_feat_per_channel = 80input_channels = 1conv_channels = Nonectc_loss_reduction = 'sum'ctc_zero_infinity = False**kwargs )

Parameters

  • vocab_size (int, optional, defaults to 8065) — Vocabulary size of the M-CTC-T model. Defines the number of different tokens that can be represented by the inputs_ids passed when calling MCTCTModel.

  • hidden_size (int, optional, defaults to 1536) — Dimension of the encoder layers and the pooler layer.

  • num_hidden_layers (int, optional, defaults to 36) — Number of hidden layers in the Transformer encoder.

  • intermediate_size (int, optional, defaults to 6144) — Dimension of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.

  • num_attention_heads (int, optional, defaults to 4) — Number of attention heads for each attention layer in the Transformer encoder.

  • attention_head_dim (int, optional, defaults to 384) — Dimensions of each attention head for each attention layer in the Transformer encoder.

  • max_position_embeddings (int, optional, defaults to 920) — The maximum sequence length that this model might ever be used with (after log-mel spectrogram extraction).

  • layer_norm_eps (float, optional, defaults to 1e-5) — The epsilon used by the layer normalization layers.

  • layerdrop (float, optional, defaults to 0.3) — The probability of dropping an encoder layer during training. The default 0.3 value is used in the original implementation.

  • hidden_act (str or function, optional, defaults to "relu") — The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu", "relu", "selu" and "gelu_new" are supported.

  • initializer_range (float, optional, defaults to 0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices.

  • hidden_dropout_prob (float, optional, defaults to 0.1) — The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.

  • attention_probs_dropout_prob (float, optional, defaults to 0.1) — The dropout ratio for the attention probabilities.

  • pad_token_id (int, optional, defaults to 1) — The tokenizer index of the pad token.

  • bos_token_id (int, optional, defaults to 0) — The tokenizer index of the bos token.

  • eos_token_id (int, optional, defaults to 2) — The tokenizer index of the eos token.

  • conv_glu_dim (int, optional, defaults to 1) — The dimension of the output of the Conv1dSubsampler layer in which GLU is applied on. Though the original Flashlight code uses the value of 2, here it’s adapted to 1 due to transposition differences.

  • conv_dropout (int, optional, defaults to 0.3) — The probability of randomly dropping the Conv1dSubsampler layer during training.

  • num_conv_layers (int, optional, defaults to 1) — Number of convolution layers before applying transformer encoder layers.

  • conv_kernel (List[int], optional, defaults to [7]) — The kernel size of the 1D convolution applied before transformer layers. len(conv_kernel) must be equal to num_conv_layers.

  • conv_stride (List[int], optional, defaults to [3]) — The stride length of the 1D convolution applied before transformer layers. len(conv_stride) must be equal to num_conv_layers.

  • input_feat_per_channel (int, optional, defaults to 80) — Feature dimensions of the channels of the input to the Conv1D layer.

  • input_channels (int, optional, defaults to 1) — Number of input channels of the input to the Conv1D layer.

  • conv_channels (List[int], optional, defaults to None) — Channel sizes of intermediate Conv1D layers.

  • ctc_loss_reduction (str, optional, defaults to "sum") — Specifies the reduction to apply to the output of torch.nn.CTCLoss. Only relevant when training an instance of MCTCTForCTC.

  • ctc_zero_infinity (bool, optional, defaults to False) — Whether to zero infinite losses and the associated gradients of torch.nn.CTCLoss. Infinite losses mainly occur when the inputs are too short to be aligned to the targets. Only relevant when training an instance of MCTCTForCTC.

This is the configuration class to store the configuration of a MCTCTModel. It is used to instantiate an M-CTC-T model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the M-CTC-T speechbrain/m-ctc-t-large architecture.

Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.

Example:

Copied

>>> from transformers import MCTCTConfig, MCTCTModel

>>> # Initializing a M-CTC-T mctct-large style configuration
>>> configuration = MCTCTConfig()

>>> # Initializing a model (with random weights) from the mctct-large style configuration
>>> model = MCTCTModel(configuration)

>>> # Accessing the model configuration
>>> configuration = model.config

MCTCTFeatureExtractor

class transformers.MCTCTFeatureExtractor

<source>

( feature_size = 80sampling_rate = 16000padding_value = 0.0hop_length = 10win_length = 25win_function = 'hamming_window'frame_signal_scale = 32768.0preemphasis_coeff = 0.97mel_floor = 1.0normalize_means = Truenormalize_vars = Truereturn_attention_mask = False**kwargs )

Parameters

  • feature_size (int, defaults to 80) — The feature dimension of the extracted features. This is the number of mel_frequency

  • sampling_rate (int, defaults to 16000) — The sampling rate at which the audio files should be digitalized expressed in hertz (Hz).

  • padding_value (float, defaults to 0.0) — The value that is used to fill the padding values.

  • hop_length (int, defaults to 10) — Number of audio samples between windows. Otherwise referred to as “shift” in many papers.

  • win_length (int, defaults to 25) — Number of ms per window

  • win_function (str, defaults to "hamming_window") — Name for the window function used for windowing, must be accessible via torch.{win_function}

  • frame_signal_scale (float, defaults to 32768.0) — Constant multiplied in creating the frames before applying DFT.

  • preemphasis_coeff (float, defaults to 0.97) — Constant multiplied in applying Pre-emphasis before DFT.

  • mel_floor (float defaults to 1.0) — Minimum value of mel frequency banks.

  • normalize_means (bool, optional, defaults to True) — Whether or not to zero-mean normalize the extracted features.

  • normalize_vars (bool, optional, defaults to True) — Whether or not to unit-variance normalize the extracted features.

Constructs a M-CTC-T feature extractor.

This feature extractor inherits from SequenceFeatureExtractor which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. This code has been adapted from Flashlight’s C++ code. For more information about the implementation, one can refer to this notebook that takes the user step-by-step in the implementation.

__call__

<source>

( raw_speech: typing.Union[numpy.ndarray, typing.List[float], typing.List[numpy.ndarray], typing.List[typing.List[float]]]padding: typing.Union[bool, str, transformers.utils.generic.PaddingStrategy] = Falsemax_length: typing.Optional[int] = Nonetruncation: bool = Falsepad_to_multiple_of: typing.Optional[int] = Nonereturn_attention_mask: typing.Optional[bool] = Nonereturn_tensors: typing.Union[str, transformers.utils.generic.TensorType, NoneType] = Nonesampling_rate: typing.Optional[int] = None**kwargs )

Parameters

  • raw_speech (torch.Tensor, np.ndarray, List[float], List[torch.Tensor], List[np.ndarray], List[List[float]]) — The sequence or batch of sequences to be padded. Each sequence can be a tensor, a numpy array, a list of float values, a list of tensors, a list of numpy arrays or a list of list of float values. Must be mono channel audio, not stereo, i.e. single float per timestep.

  • padding (bool, str or PaddingStrategy, optional, defaults to False) — Select a strategy to pad the returned sequences (according to the model’s padding side and padding index) among:

    • True or 'longest': Pad to the longest sequence in the batch (or no padding if only a single sequence if provided).

    • 'max_length': Pad to a maximum length specified with the argument max_length or to the maximum acceptable input length for the model if that argument is not provided.

    • False or 'do_not_pad' (default): No padding (i.e., can output a batch with sequences of different lengths).

  • max_length (int, optional) — Maximum length of the returned list and optionally padding length (see above).

  • truncation (bool) — Activates truncation to cut input sequences longer than max_length to max_length.

  • pad_to_multiple_of (int, optional) — If set will pad the sequence to a multiple of the provided value.

    This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128.

  • return_attention_mask (bool, optional) — Whether to return the attention mask. If left to the default, will return the attention mask according to the specific feature_extractor’s default.

    What are attention masks?

  • return_tensors (str or TensorType, optional) — If set, will return tensors instead of list of python integers. Acceptable values are:

    • 'tf': Return TensorFlow tf.constant objects.

    • 'pt': Return PyTorch torch.Tensor objects.

    • 'np': Return Numpy np.ndarray objects.

  • sampling_rate (int, optional) — The sampling rate at which the raw_speech input was sampled. It is strongly recommended to pass sampling_rate at the forward call to prevent silent errors.

  • padding_value (float, defaults to 0.0) —

Main method to featurize and prepare for the model one or several sequence(s). sequences. It returns the log-mel spectrogram of the input audio, as implemented in the original Flashlight MFSC feature extraction code.

MCTCTProcessor

class transformers.MCTCTProcessor

<source>

( feature_extractortokenizer )

Parameters

  • feature_extractor (MCTCTFeatureExtractor) — An instance of MCTCTFeatureExtractor. The feature extractor is a required input.

  • tokenizer (AutoTokenizer) — An instance of AutoTokenizer. The tokenizer is a required input.

Constructs a MCTCT processor which wraps a MCTCT feature extractor and a MCTCT tokenizer into a single processor.

MCTCTProcessor offers all the functionalities of MCTCTFeatureExtractor and AutoTokenizer. See the call() and decode() for more information.

__call__

<source>

( *args**kwargs )

When used in normal mode, this method forwards all its arguments to MCTCTFeatureExtractor’s call() and returns its output. If used in the context as_target_processor() this method forwards all its arguments to AutoTokenizer’s __call__(). Please refer to the doctsring of the above two methods for more information.

from_pretrained

<source>

( pretrained_model_name_or_path: typing.Union[str, os.PathLike]cache_dir: typing.Union[str, os.PathLike, NoneType] = Noneforce_download: bool = Falselocal_files_only: bool = Falsetoken: typing.Union[bool, str, NoneType] = Nonerevision: str = 'main'**kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — This can be either:

    • a string, the model id of a pretrained feature_extractor hosted inside a model repo on boincai.com. Valid model ids can be located at the root-level, like bert-base-uncased, or namespaced under a user or organization name, like dbmdz/bert-base-german-cased.

    • a path to a directory containing a feature extractor file saved using the save_pretrained() method, e.g., ./my_model_directory/.

    • a path or url to a saved feature extractor JSON file, e.g., ./my_model_directory/preprocessor_config.json. **kwargs — Additional keyword arguments passed along to both from_pretrained() and ~tokenization_utils_base.PreTrainedTokenizer.from_pretrained.

Instantiate a processor associated with a pretrained model.

This class method is simply calling the feature extractor from_pretrained(), image processor ImageProcessingMixin and the tokenizer ~tokenization_utils_base.PreTrainedTokenizer.from_pretrained methods. Please refer to the docstrings of the methods above for more information.

save_pretrained

<source>

( save_directorypush_to_hub: bool = False**kwargs )

Parameters

  • save_directory (str or os.PathLike) — Directory where the feature extractor JSON file and the tokenizer files will be saved (directory will be created if it does not exist).

  • push_to_hub (bool, optional, defaults to False) — Whether or not to push your model to the BOINC AI model hub after saving it. You can specify the repository you want to push to with repo_id (will default to the name of save_directory in your namespace).

  • kwargs (Dict[str, Any], optional) — Additional key word arguments passed along to the push_to_hub() method.

Saves the attributes of this processor (feature extractor, tokenizer…) in the specified directory so that it can be reloaded using the from_pretrained() method.

This class method is simply calling save_pretrained() and save_pretrained(). Please refer to the docstrings of the methods above for more information.

batch_decode

<source>

( *args**kwargs )

This method forwards all its arguments to AutoTokenizer’s batch_decode(). Please refer to the docstring of this method for more information.

decode

<source>

( *args**kwargs )

This method forwards all its arguments to AutoTokenizer’s decode(). Please refer to the docstring of this method for more information.

MCTCTModel

class transformers.MCTCTModel

<source>

( config )

Parameters

  • config (MCTCTConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The bare M-CTC-T Model transformer outputting raw hidden-states without any specific head on top. This model is a PyTorch torch.nn.Module sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

forward

<source>

( input_features: Tensorattention_mask: typing.Optional[torch.Tensor] = Nonehead_mask: typing.Optional[torch.Tensor] = Noneoutput_attentions: typing.Optional[bool] = Noneoutput_hidden_states: typing.Optional[bool] = Nonereturn_dict: typing.Optional[bool] = None ) → transformers.modeling_outputs.BaseModelOutput or tuple(torch.FloatTensor)

Parameters

  • input_features (torch.LongTensor of shape (batch_size, sequence_length)) — Indices of input sequence tokens in the vocabulary.

    Indices can be obtained using Wav2Vec2CTCTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

    What are input IDs?

  • attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:

    • 1 for tokens that are not masked,

    • 0 for tokens that are masked.

    What are attention masks?

  • head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:

    • 1 indicates the head is not masked,

    • 0 indicates the head is masked.

  • output_attentions (bool, optional) — Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail.

  • output_hidden_states (bool, optional) — Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail.

  • return_dict (bool, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

transformers.modeling_outputs.BaseModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.BaseModelOutput or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (MCTCTConfig) and inputs.

  • last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.

  • hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

    Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

  • attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

    Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The MCTCTModel forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

Copied

>>> from transformers import AutoProcessor, MCTCTModel
>>> from datasets import load_dataset
>>> import torch

>>> dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation")
>>> dataset = dataset.sort("id")
>>> sampling_rate = dataset.features["audio"].sampling_rate

>>> processor = AutoProcessor.from_pretrained("speechbrain/m-ctc-t-large")
>>> model = MCTCTModel.from_pretrained("speechbrain/m-ctc-t-large")

>>> # audio file is decoded on the fly
>>> inputs = processor(dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> predicted_ids = torch.argmax(logits, dim=-1)

>>> # transcribe speech
>>> transcription = processor.batch_decode(predicted_ids)
>>> transcription[0]
[1, 195, 1536]

>>> inputs["labels"] = processor(text=dataset[0]["text"], return_tensors="pt").input_ids

>>> # compute loss
>>> loss = model(**inputs).loss

MCTCTForCTC

class transformers.MCTCTForCTC

<source>

( config )

Parameters

  • config (MCTCTConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

MCTCT Model with a language modeling head on top for Connectionist Temporal Classification (CTC). This model is a PyTorch torch.nn.Module sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

forward

<source>

( input_features: Tensorattention_mask: typing.Optional[torch.Tensor] = Nonehead_mask: typing.Optional[torch.Tensor] = Noneoutput_attentions: typing.Optional[bool] = Noneoutput_hidden_states: typing.Optional[bool] = Nonereturn_dict: typing.Optional[bool] = Nonelabels: typing.Optional[torch.LongTensor] = None ) → transformers.modeling_outputs.CausalLMOutput or tuple(torch.FloatTensor)

Parameters

  • input_features (torch.LongTensor of shape ({0})) — Indices of input sequence tokens in the vocabulary.

    Indices can be obtained using Wav2Vec2CTCTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

    What are input IDs?

  • attention_mask (torch.FloatTensor of shape ({0}), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:

    • 1 for tokens that are not masked,

    • 0 for tokens that are masked.

    What are attention masks?

  • head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:

    • 1 indicates the head is not masked,

    • 0 indicates the head is masked.

  • output_attentions (bool, optional) — Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail.

  • output_hidden_states (bool, optional) — Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail.

  • return_dict (bool, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

  • labels (torch.LongTensor of shape (batch_size, target_length), optional) — Labels for connectionist temporal classification. Note that target_length has to be smaller or equal to the sequence length of the output logits. Indices are selected in [-100, 0, ..., config.vocab_size - 1]. All labels set to -100 are ignored (masked), the loss is only computed for labels in [0, ..., config.vocab_size - 1].

Returns

transformers.modeling_outputs.CausalLMOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.CausalLMOutput or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (MCTCTConfig) and inputs.

  • loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Language modeling loss (for next-token prediction).

  • logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).

  • hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

    Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

  • attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

    Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The MCTCTForCTC forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

Copied

>>> from transformers import AutoProcessor, MCTCTForCTC
>>> from datasets import load_dataset
>>> import torch

>>> dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation")
>>> dataset = dataset.sort("id")
>>> sampling_rate = dataset.features["audio"].sampling_rate

>>> processor = AutoProcessor.from_pretrained("speechbrain/m-ctc-t-large")
>>> model = MCTCTForCTC.from_pretrained("speechbrain/m-ctc-t-large")

>>> # audio file is decoded on the fly
>>> inputs = processor(dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> predicted_ids = torch.argmax(logits, dim=-1)

>>> # transcribe speech
>>> transcription = processor.batch_decode(predicted_ids)
>>> transcription[0]
"Mr. Quilter is the apostle of the middle classes, and we're glad to welcome his gospel."

>>> inputs["labels"] = processor(text=dataset[0]["text"], return_tensors="pt").input_ids

>>> # compute loss
>>> loss = model(**inputs).loss
>>> round(loss.item(), 2)
1885.65
PreviousHubertNextMMS

Last updated