CLAP
Last updated
Last updated
The CLAP model was proposed in by Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, Shlomo Dubnov.
CLAP (Contrastive Language-Audio Pretraining) is a neural network trained on a variety of (audio, text) pairs. It can be instructed in to predict the most relevant text snippet, given an audio, without directly optimizing for the task. The CLAP model uses a SWINTransformer to get audio features from a log-Mel spectrogram input, and a RoBERTa model to get text features. Both the text and audio features are then projected to a latent space with identical dimension. The dot product between the projected audio and text features is then used as a similar score.
The abstract from the paper is the following:
Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zeroshot setting and is able to obtain performance comparable to models’ results in the non-zero-shot setting. LAION-Audio-6
This model was contributed by and . The original code can be found .
( text_config = Noneaudio_config = Nonelogit_scale_init_value = 14.285714285714285projection_dim = 512projection_hidden_act = 'relu'initializer_factor = 1.0**kwargs )
Parameters
text_config (dict, optional) — Dictionary of configuration options used to initialize .
audio_config (dict, optional) — Dictionary of configuration options used to initialize .
projection_dim (int, optional, defaults to 512) — Dimentionality of text and audio projection layers.
logit_scale_init_value (float, optional, defaults to 2.6592) — The inital value of the logit_scale paramter. Default is used as per the original CLAP implementation.
projection_hidden_act (str, optional, defaults to "relu") — Activation function for the projection layers.
initializer_factor (float, optional, defaults to 1.0) — Factor to scale the initialization of the model weights.
kwargs (optional) — Dictionary of keyword arguments.
Example:
Copied
from_text_audio_configs
Returns
An instance of a configuration object
( vocab_size = 50265hidden_size = 768num_hidden_layers = 12num_attention_heads = 12intermediate_size = 3072hidden_act = 'gelu'hidden_dropout_prob = 0.1attention_probs_dropout_prob = 0.1max_position_embeddings = 514type_vocab_size = 1initializer_factor = 1.0layer_norm_eps = 1e-12projection_dim = 512pad_token_id = 1bos_token_id = 0eos_token_id = 2position_embedding_type = 'absolute'use_cache = Trueprojection_hidden_act = 'relu'**kwargs )
Parameters
hidden_size (int, optional, defaults to 768) — Dimensionality of the encoder layers and the pooler layer.
num_hidden_layers (int, optional, defaults to 12) — Number of hidden layers in the Transformer encoder.
num_attention_heads (int, optional, defaults to 12) — Number of attention heads for each attention layer in the Transformer encoder.
intermediate_size (int, optional, defaults to 3072) — Dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer encoder.
hidden_act (str or Callable, optional, defaults to "relu") — The non-linear activation function (function or string) in the encoder and pooler. If string, "relu", "relu", "silu" and "relu_new" are supported.
hidden_dropout_prob (float, optional, defaults to 0.1) — The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
attention_probs_dropout_prob (float, optional, defaults to 0.1) — The dropout ratio for the attention probabilities.
max_position_embeddings (int, optional, defaults to 512) — The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
layer_norm_eps (float, optional, defaults to 1e-12) — The epsilon used by the layer normalization layers.
is_decoder (bool, optional, defaults to False) — Whether the model is used as a decoder or not. If False, the model is used as an encoder.
use_cache (bool, optional, defaults to True) — Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if config.is_decoder=True.
projection_hidden_act (str, optional, defaults to "relu") — The non-linear activation function (function or string) in the projection layer. If string, "gelu", "relu", "silu" and "gelu_new" are supported.
projection_dim (int, optional, defaults to 512) — Dimension of the projection head of the ClapTextModelWithProjection.
Examples:
Copied
( window_size = 8num_mel_bins = 64spec_size = 256hidden_act = 'gelu'patch_size = 4patch_stride = [4, 4]num_classes = 527hidden_size = 768projection_dim = 512depths = [2, 2, 6, 2]num_attention_heads = [4, 8, 16, 32]enable_fusion = Falsehidden_dropout_prob = 0.1fusion_type = Nonepatch_embed_input_channels = 1flatten_patch_embeds = Truepatch_embeds_hidden_size = 96enable_patch_layer_norm = Truedrop_path_rate = 0.0attention_probs_dropout_prob = 0.0qkv_bias = Truemlp_ratio = 4.0aff_block_r = 4num_hidden_layers = 4projection_hidden_act = 'relu'layer_norm_eps = 1e-05initializer_factor = 1.0**kwargs )
Parameters
window_size (int, optional, defaults to 8) — Image size of the spectrogram
num_mel_bins (int, optional, defaults to 64) — Number of mel features used per frames. Should correspond to the value used in the ClapProcessor class.
spec_size (int, optional, defaults to 256) — Desired input size of the spectrogram that the model supports. It can be different from the output of the ClapFeatureExtractor, in which case the input features will be resized. Corresponds to the image_size of the audio models.
hidden_act (str, optional, defaults to "gelu") — The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu", "relu", "silu" and "gelu_new" are supported.
patch_size (int, optional, defaults to 4) — Patch size for the audio spectrogram
patch_stride (list, optional, defaults to [4, 4]) — Patch stride for the audio spectrogram
num_classes (int, optional, defaults to 527) — Number of classes used for the head training
hidden_size (int, optional, defaults to 768) — Hidden size of the output of the audio encoder. Correspond to the dimension of the penultimate layer’s output,which is sent to the projection MLP layer.
projection_dim (int, optional, defaults to 512) — Hidden size of the projection layer.
depths (list, optional, defaults to [2, 2, 6, 2]) — Depths used for the Swin Layers of the audio model
num_attention_heads (list, optional, defaults to [4, 8, 16, 32]) — Number of attention heads used for the Swin Layers of the audio model
enable_fusion (bool, optional, defaults to False) — Whether or not to enable patch fusion. This is the main contribution of the authors, and should give the best results.
hidden_dropout_prob (float, optional, defaults to 0.1) — The dropout probabilitiy for all fully connected layers in the encoder.
fusion_type ([type], optional) — Fusion type used for the patch fusion.
patch_embed_input_channels (int, optional, defaults to 1) — Number of channels used for the input spectrogram
flatten_patch_embeds (bool, optional, defaults to True) — Whether or not to flatten the patch embeddings
patch_embeds_hidden_size (int, optional, defaults to 96) — Hidden size of the patch embeddings. It is used as the number of output channels.
enable_patch_layer_norm (bool, optional, defaults to True) — Whether or not to enable layer normalization for the patch embeddings
drop_path_rate (float, optional, defaults to 0.0) — Drop path rate for the patch fusion
attention_probs_dropout_prob (float, optional, defaults to 0.0) — The dropout ratio for the attention probabilities.
qkv_bias (bool, optional, defaults to True) — Whether or not to add a bias to the query, key, value projections.
mlp_ratio (float, optional, defaults to 4.0) — Ratio of the mlp hidden dim to embedding dim.
aff_block_r (int, optional, defaults to 4) — downsize_ratio used in the AudioFF block
num_hidden_layers (int, optional, defaults to 4) — Number of hidden layers in the Transformer encoder.
projection_hidden_act (str, optional, defaults to "relu") — The non-linear activation function (function or string) in the projection layer. If string, "gelu", "relu", "silu" and "gelu_new" are supported.
layer_norm_eps ([type], optional, defaults to 1e-5) — The epsilon used by the layer normalization layers.
initializer_factor (float, optional, defaults to 1.0) — A factor for initializing all weight matrices (should be kept to 1, used internally for initialization testing).
Example:
Copied
( feature_size = 64sampling_rate = 48000hop_length = 480max_length_s = 10fft_window_size = 1024padding_value = 0.0return_attention_mask = Falsefrequency_min: float = 0frequency_max: float = 14000top_db: int = Nonetruncation: str = 'fusion'padding: str = 'repeatpad'**kwargs )
Parameters
feature_size (int, defaults to 64) — The feature dimension of the extracted Mel spectrograms. This corresponds to the number of mel filters (n_mels).
sampling_rate (int, defaults to 48_000) — The sampling rate at which the audio files should be digitalized expressed in hertz (Hz). This only serves to warn users if the audio fed to the feature extractor does not have the same sampling rate.
hop_length (int, defaults to 480) — Length of the overlaping windows for the STFT used to obtain the Mel Spectrogram. The audio will be split in smaller frames with a step of hop_length between each frame.
max_length_s (int, defaults to 10) — The maximum input length of the model in seconds. This is used to pad the audio.
fft_window_size (int, defaults to 1024) — Size of the window (in samples) on which the Fourier transform is applied. This controls the frequency resolution of the spectrogram. 400 means that the fourrier transform is computed on windows of 400 samples.
padding_value (float, optional, defaults to 0.0) — Padding value used to pad the audio. Should correspond to silences.
return_attention_mask (bool, optional, defaults to False) — Whether or not the model should return the attention masks coresponding to the input.
frequency_min (float, optional, default to 0) — The lowest frequency of interest. The STFT will not be computed for values below this.
frequency_max (float, optional, default to 14_000) — The highest frequency of interest. The STFT will not be computed for values above this.
top_db (float, optional) — The highest decibel value used to convert the mel spectrogram to the log scale. For more details see the audio_utils.power_to_db function
truncation (str, optional, default to "fusions") — Truncation pattern for long audio inputs. Two patterns are available:
fusion will use _random_mel_fusion, which stacks 3 random crops from the mel spectrogram and a downsampled version of the entire mel spectrogram. If config.fusion is set to True, shorter audios also need to to return 4 mels, which will just be a copy of the original mel obtained from the padded audio.
rand_trunc will select a random crop of the mel spectrogram.
padding (str, optional, defaults to "repeatpad") — Padding pattern for shorter audio inputs. Three patterns were originally implemented:
repeatpad: the audio is repeated, and then padded to fit the max_length.
repeat: the audio is repeated and then cut to fit the max_length
pad: the audio is padded.
Constructs a CLAP feature extractor.
This class extracts mel-filter bank features from raw speech using a custom numpy implementation of the Short Time Fourier Transform (STFT) which should match pytorch’s torch.stft equivalent.
to_dict
( ) → Dict[str, Any]
Returns
Dict[str, Any]
Dictionary of all the attributes that make up this configuration instance, excpet for the mel filter banks, which do not need to be saved or printed as they are too long.
Serializes this instance to a Python dictionary.
( feature_extractortokenizer )
Parameters
Constructs a CLAP processor which wraps a CLAP feature extractor and a RoBerta tokenizer into a single processor.
batch_decode
( *args**kwargs )
decode
( *args**kwargs )
( config: ClapConfig )
Parameters
forward
( input_ids: typing.Optional[torch.LongTensor] = Noneinput_features: typing.Optional[torch.FloatTensor] = Noneis_longer: typing.Optional[torch.BoolTensor] = Noneattention_mask: typing.Optional[torch.Tensor] = Noneposition_ids: typing.Optional[torch.LongTensor] = Nonereturn_loss: typing.Optional[bool] = Noneoutput_attentions: typing.Optional[bool] = Noneoutput_hidden_states: typing.Optional[bool] = Nonereturn_dict: typing.Optional[bool] = None ) → transformers.models.clap.modeling_clap.ClapOutput or tuple(torch.FloatTensor)
Parameters
input_ids (torch.LongTensor of shape (batch_size, sequence_length)) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it.
attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
return_loss (bool, optional) — Whether or not to return the contrastive loss.
output_attentions (bool, optional) — Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail.
output_hidden_states (bool, optional) — Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail.
Returns
transformers.models.clap.modeling_clap.ClapOutput or tuple(torch.FloatTensor)
A transformers.models.clap.modeling_clap.ClapOutput or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (<class 'transformers.models.clap.configuration_clap.ClapConfig'>) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when return_loss is True) — Contrastive loss for audio-text similarity.
logits_per_audio:(torch.FloatTensor of shape (audio_batch_size, text_batch_size)) — The scaled dot product scores between audio_embeds and text_embeds. This represents the audio-text similarity scores.
logits_per_text:(torch.FloatTensor of shape (text_batch_size, audio_batch_size)) — The scaled dot product scores between text_embeds and audio_embeds. This represents the text-audio similarity scores.
Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.
Examples:
Copied
get_text_features
( input_ids: typing.Optional[torch.Tensor] = Noneattention_mask: typing.Optional[torch.Tensor] = Noneposition_ids: typing.Optional[torch.Tensor] = Noneoutput_attentions: typing.Optional[bool] = Noneoutput_hidden_states: typing.Optional[bool] = Nonereturn_dict: typing.Optional[bool] = None ) → text_features (torch.FloatTensor of shape (batch_size, output_dim)
Parameters
input_ids (torch.LongTensor of shape (batch_size, sequence_length)) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it.
attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
output_attentions (bool, optional) — Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail.
output_hidden_states (bool, optional) — Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail.
Returns
text_features (torch.FloatTensor of shape (batch_size, output_dim)
Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.
Examples:
Copied
get_audio_features
( input_features: typing.Optional[torch.Tensor] = Noneis_longer: typing.Optional[torch.Tensor] = Noneattention_mask: typing.Optional[torch.Tensor] = Noneoutput_attentions: typing.Optional[bool] = Noneoutput_hidden_states: typing.Optional[bool] = Nonereturn_dict: typing.Optional[bool] = None ) → audio_features (torch.FloatTensor of shape (batch_size, output_dim)
Parameters
is_longer (torch.FloatTensor, of shape (batch_size, 1), optional) — Whether the audio clip is longer than max_length. If True, a feature fusion will be enabled to enhance the features.
output_attentions (bool, optional) — Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail.
output_hidden_states (bool, optional) — Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail.
Returns
audio_features (torch.FloatTensor of shape (batch_size, output_dim)
Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.
Examples:
Copied
( configadd_pooling_layer = True )
The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in Attention is all you need_ by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin.
To behave as an decoder the model needs to be initialized with the is_decoder argument of the configuration set to True. To be used in a Seq2Seq model, the model needs to initialized with both is_decoder argument and add_cross_attention set to True; an encoder_hidden_states is then expected as an input to the forward pass.
forward
( input_ids: typing.Optional[torch.Tensor] = Noneattention_mask: typing.Optional[torch.Tensor] = Nonetoken_type_ids: typing.Optional[torch.Tensor] = Noneposition_ids: typing.Optional[torch.Tensor] = Nonehead_mask: typing.Optional[torch.Tensor] = Noneinputs_embeds: typing.Optional[torch.Tensor] = Noneencoder_hidden_states: typing.Optional[torch.Tensor] = Noneencoder_attention_mask: typing.Optional[torch.Tensor] = Nonepast_key_values: typing.Optional[typing.List[torch.FloatTensor]] = Noneuse_cache: typing.Optional[bool] = Noneoutput_attentions: typing.Optional[bool] = Noneoutput_hidden_states: typing.Optional[bool] = Nonereturn_dict: typing.Optional[bool] = None )
encoder_hidden_states (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked. past_key_values (tuple(tuple(torch.FloatTensor)) of length config.n_layers with each tuple having 4 tensors of shape (batch_size, num_heads, sequence_length - 1, embed_size_per_head)): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding.
If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all decoder_input_ids of shape (batch_size, sequence_length). use_cache (bool, optional): If set to True, past_key_values key value states are returned and can be used to speed up decoding (see past_key_values).
( config: ClapTextConfig )
Parameters
CLAP Text Model with a projection layer on top (a linear layer on top of the pooled output).
forward
( input_ids: typing.Optional[torch.Tensor] = Noneattention_mask: typing.Optional[torch.Tensor] = Noneposition_ids: typing.Optional[torch.Tensor] = Noneoutput_attentions: typing.Optional[bool] = Noneoutput_hidden_states: typing.Optional[bool] = Nonereturn_dict: typing.Optional[bool] = None ) → transformers.models.clap.modeling_clap.ClapTextModelOutput or tuple(torch.FloatTensor)
Parameters
input_ids (torch.LongTensor of shape (batch_size, sequence_length)) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it.
attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
output_attentions (bool, optional) — Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail.
output_hidden_states (bool, optional) — Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail.
Returns
transformers.models.clap.modeling_clap.ClapTextModelOutput or tuple(torch.FloatTensor)
A transformers.models.clap.modeling_clap.ClapTextModelOutput or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (<class 'transformers.models.clap.configuration_clap.ClapTextConfig'>) and inputs.
text_embeds (torch.FloatTensor of shape (batch_size, output_dim) optional returned when model is initialized with with_projection=True) — The text embeddings obtained by applying the projection layer to the pooler_output.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.
Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.
Examples:
Copied
( config: ClapAudioConfig )
forward
Parameters
is_longer (torch.FloatTensor, of shape (batch_size, 1), optional) — Whether the audio clip is longer than max_length. If True, a feature fusion will be enabled to enhance the features.
output_attentions (bool, optional) — Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail.
output_hidden_states (bool, optional) — Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail.
Returns
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
pooler_output (torch.FloatTensor of shape (batch_size, hidden_size)) — Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.
Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.
Examples:
Copied
( config: ClapAudioConfig )
Parameters
CLAP Audio Model with a projection layer on top (a linear layer on top of the pooled output).
forward
( input_features: typing.Optional[torch.FloatTensor] = Noneis_longer: typing.Optional[torch.BoolTensor] = Noneoutput_attentions: typing.Optional[bool] = Noneoutput_hidden_states: typing.Optional[bool] = Nonereturn_dict: typing.Optional[bool] = None ) → transformers.models.clap.modeling_clap.ClapAudioModelOutput or tuple(torch.FloatTensor)
Parameters
is_longer (torch.FloatTensor, of shape (batch_size, 1), optional) — Whether the audio clip is longer than max_length. If True, a feature fusion will be enabled to enhance the features.
output_attentions (bool, optional) — Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail.
output_hidden_states (bool, optional) — Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail.
Returns
transformers.models.clap.modeling_clap.ClapAudioModelOutput or tuple(torch.FloatTensor)
A transformers.models.clap.modeling_clap.ClapAudioModelOutput or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (<class 'transformers.models.clap.configuration_clap.ClapAudioConfig'>) and inputs.
audio_embeds (torch.FloatTensor of shape (batch_size, hidden_size)) — The Audio embeddings obtained by applying the projection layer to the pooler_output.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.
Examples:
Copied
is the configuration class to store the configuration of a . It is used to instantiate a CLAP model according to the specified arguments, defining the text model and audio model configs. Instantiating a configuration with the defaults will yield a similar configuration to that of the CLAP architecture.
Configuration objects inherit from and can be used to control the model outputs. Read the documentation from for more information.
( text_config: ClapTextConfigaudio_config: ClapAudioConfig**kwargs ) →
Instantiate a (or a derived class) from clap text model configuration and clap audio model configuration.
vocab_size (int, optional, defaults to 30522) — Vocabulary size of the CLAP model. Defines the number of different tokens that can be represented by the inputs_ids passed when calling .
type_vocab_size (int, optional, defaults to 2) — The vocabulary size of the token_type_ids passed when calling .
position_embedding_type (str, optional, defaults to "absolute") — Type of position embedding. Choose one of "absolute", "relative_key", "relative_key_query". For positional embeddings use "absolute". For more information on "relative_key", please refer to . For more information on "relative_key_query", please refer to Method 4 in .
This is the configuration class to store the configuration of a . It is used to instantiate a CLAP model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the CLAP architecture.
Configuration objects inherit from and can be used to control the model outputs. Read the documentation from for more information.
This is the configuration class to store the configuration of a . It is used to instantiate a CLAP audio encoder according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the audio encoder of the CLAP architecture.
Configuration objects inherit from and can be used to control the model outputs. Read the documentation from for more information.
This feature extractor inherits from which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.
feature_extractor () — The audio processor is a required input.
tokenizer () — The tokenizer is a required input.
offers all the functionalities of and . See the __call__() and for more information.
This method forwards all its arguments to RobertaTokenizerFast’s . Please refer to the docstring of this method for more information.
This method forwards all its arguments to RobertaTokenizerFast’s . Please refer to the docstring of this method for more information.
config () — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the method to load the model weights.
This model inherits from . Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)
This model is also a PyTorch subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.
Indices can be obtained using . See and for details.
input_features (torch.FloatTensor of shape (batch_size, num_channels, height, width)) — Input audio features. This should be returnes by the class that you can also retrieve from . See ClapFeatureExtractor.__call__() for details.
return_dict (bool, optional) — Whether or not to return a instead of a plain tuple.
text_embeds(torch.FloatTensor of shape (batch_size, output_dim) — The text embeddings obtained by applying the projection layer to the pooled output of .
audio_embeds(torch.FloatTensor of shape (batch_size, output_dim) — The audio embeddings obtained by applying the projection layer to the pooled output of .
text_model_output(BaseModelOutputWithPooling): The output of the .
audio_model_output(BaseModelOutputWithPooling): The output of the .
The forward method, overrides the __call__ special method.
Indices can be obtained using . See and for details.
return_dict (bool, optional) — Whether or not to return a instead of a plain tuple.
The text embeddings obtained by applying the projection layer to the pooled output of .
The forward method, overrides the __call__ special method.
input_features (torch.FloatTensor of shape (batch_size, num_channels, height, width)) — Input audio features. This should be returnes by the class that you can also retrieve from . See ClapFeatureExtractor.__call__() for details.
return_dict (bool, optional) — Whether or not to return a instead of a plain tuple.
The audio embeddings obtained by applying the projection layer to the pooled output of .
The forward method, overrides the __call__ special method.
.. _Attention is all you need:
config () — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the method to load the model weights.
This model inherits from . Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)
This model is also a PyTorch subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.
Indices can be obtained using . See and for details.
return_dict (bool, optional) — Whether or not to return a instead of a plain tuple.
The forward method, overrides the __call__ special method.
( input_features: typing.Optional[torch.FloatTensor] = Noneis_longer: typing.Optional[torch.BoolTensor] = Noneoutput_attentions: typing.Optional[bool] = Noneoutput_hidden_states: typing.Optional[bool] = Nonereturn_dict: typing.Optional[bool] = None ) → or tuple(torch.FloatTensor)
input_features (torch.FloatTensor of shape (batch_size, num_channels, height, width)) — Input audio features. This should be returnes by the class that you can also retrieve from . See ClapFeatureExtractor.__call__() for details.
return_dict (bool, optional) — Whether or not to return a instead of a plain tuple.
or tuple(torch.FloatTensor)
A or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (<class 'transformers.models.clap.configuration_clap.ClapAudioConfig'>) and inputs.
The forward method, overrides the __call__ special method.
config () — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the method to load the model weights.
This model inherits from . Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)
This model is also a PyTorch subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.
input_features (torch.FloatTensor of shape (batch_size, num_channels, height, width)) — Input audio features. This should be returnes by the class that you can also retrieve from . See ClapFeatureExtractor.__call__() for details.
return_dict (bool, optional) — Whether or not to return a instead of a plain tuple.
The forward method, overrides the __call__ special method.