Transformers
  • 🌍GET STARTED
    • Transformers
    • Quick tour
    • Installation
  • 🌍TUTORIALS
    • Run inference with pipelines
    • Write portable code with AutoClass
    • Preprocess data
    • Fine-tune a pretrained model
    • Train with a script
    • Set up distributed training with BOINC AI Accelerate
    • Load and train adapters with BOINC AI PEFT
    • Share your model
    • Agents
    • Generation with LLMs
  • 🌍TASK GUIDES
    • 🌍NATURAL LANGUAGE PROCESSING
      • Text classification
      • Token classification
      • Question answering
      • Causal language modeling
      • Masked language modeling
      • Translation
      • Summarization
      • Multiple choice
    • 🌍AUDIO
      • Audio classification
      • Automatic speech recognition
    • 🌍COMPUTER VISION
      • Image classification
      • Semantic segmentation
      • Video classification
      • Object detection
      • Zero-shot object detection
      • Zero-shot image classification
      • Depth estimation
    • 🌍MULTIMODAL
      • Image captioning
      • Document Question Answering
      • Visual Question Answering
      • Text to speech
    • 🌍GENERATION
      • Customize the generation strategy
    • 🌍PROMPTING
      • Image tasks with IDEFICS
  • 🌍DEVELOPER GUIDES
    • Use fast tokenizers from BOINC AI Tokenizers
    • Run inference with multilingual models
    • Use model-specific APIs
    • Share a custom model
    • Templates for chat models
    • Run training on Amazon SageMaker
    • Export to ONNX
    • Export to TFLite
    • Export to TorchScript
    • Benchmarks
    • Notebooks with examples
    • Community resources
    • Custom Tools and Prompts
    • Troubleshoot
  • 🌍PERFORMANCE AND SCALABILITY
    • Overview
    • 🌍EFFICIENT TRAINING TECHNIQUES
      • Methods and tools for efficient training on a single GPU
      • Multiple GPUs and parallelism
      • Efficient training on CPU
      • Distributed CPU training
      • Training on TPUs
      • Training on TPU with TensorFlow
      • Training on Specialized Hardware
      • Custom hardware for training
      • Hyperparameter Search using Trainer API
    • 🌍OPTIMIZING INFERENCE
      • Inference on CPU
      • Inference on one GPU
      • Inference on many GPUs
      • Inference on Specialized Hardware
    • Instantiating a big model
    • Troubleshooting
    • XLA Integration for TensorFlow Models
    • Optimize inference using `torch.compile()`
  • 🌍CONTRIBUTE
    • How to contribute to transformers?
    • How to add a model to BOINC AI Transformers?
    • How to convert a BOINC AI Transformers model to TensorFlow?
    • How to add a pipeline to BOINC AI Transformers?
    • Testing
    • Checks on a Pull Request
  • 🌍CONCEPTUAL GUIDES
    • Philosophy
    • Glossary
    • What BOINC AI Transformers can do
    • How BOINC AI Transformers solve tasks
    • The Transformer model family
    • Summary of the tokenizers
    • Attention mechanisms
    • Padding and truncation
    • BERTology
    • Perplexity of fixed-length models
    • Pipelines for webserver inference
    • Model training anatomy
  • 🌍API
    • 🌍MAIN CLASSES
      • Agents and Tools
      • 🌍Auto Classes
        • Extending the Auto Classes
        • AutoConfig
        • AutoTokenizer
        • AutoFeatureExtractor
        • AutoImageProcessor
        • AutoProcessor
        • Generic model classes
          • AutoModel
          • TFAutoModel
          • FlaxAutoModel
        • Generic pretraining classes
          • AutoModelForPreTraining
          • TFAutoModelForPreTraining
          • FlaxAutoModelForPreTraining
        • Natural Language Processing
          • AutoModelForCausalLM
          • TFAutoModelForCausalLM
          • FlaxAutoModelForCausalLM
          • AutoModelForMaskedLM
          • TFAutoModelForMaskedLM
          • FlaxAutoModelForMaskedLM
          • AutoModelForMaskGenerationge
          • TFAutoModelForMaskGeneration
          • AutoModelForSeq2SeqLM
          • TFAutoModelForSeq2SeqLM
          • FlaxAutoModelForSeq2SeqLM
          • AutoModelForSequenceClassification
          • TFAutoModelForSequenceClassification
          • FlaxAutoModelForSequenceClassification
          • AutoModelForMultipleChoice
          • TFAutoModelForMultipleChoice
          • FlaxAutoModelForMultipleChoice
          • AutoModelForNextSentencePrediction
          • TFAutoModelForNextSentencePrediction
          • FlaxAutoModelForNextSentencePrediction
          • AutoModelForTokenClassification
          • TFAutoModelForTokenClassification
          • FlaxAutoModelForTokenClassification
          • AutoModelForQuestionAnswering
          • TFAutoModelForQuestionAnswering
          • FlaxAutoModelForQuestionAnswering
          • AutoModelForTextEncoding
          • TFAutoModelForTextEncoding
        • Computer vision
          • AutoModelForDepthEstimation
          • AutoModelForImageClassification
          • TFAutoModelForImageClassification
          • FlaxAutoModelForImageClassification
          • AutoModelForVideoClassification
          • AutoModelForMaskedImageModeling
          • TFAutoModelForMaskedImageModeling
          • AutoModelForObjectDetection
          • AutoModelForImageSegmentation
          • AutoModelForImageToImage
          • AutoModelForSemanticSegmentation
          • TFAutoModelForSemanticSegmentation
          • AutoModelForInstanceSegmentation
          • AutoModelForUniversalSegmentation
          • AutoModelForZeroShotImageClassification
          • TFAutoModelForZeroShotImageClassification
          • AutoModelForZeroShotObjectDetection
        • Audio
          • AutoModelForAudioClassification
          • AutoModelForAudioFrameClassification
          • TFAutoModelForAudioFrameClassification
          • AutoModelForCTC
          • AutoModelForSpeechSeq2Seq
          • TFAutoModelForSpeechSeq2Seq
          • FlaxAutoModelForSpeechSeq2Seq
          • AutoModelForAudioXVector
          • AutoModelForTextToSpectrogram
          • AutoModelForTextToWaveform
        • Multimodal
          • AutoModelForTableQuestionAnswering
          • TFAutoModelForTableQuestionAnswering
          • AutoModelForDocumentQuestionAnswering
          • TFAutoModelForDocumentQuestionAnswering
          • AutoModelForVisualQuestionAnswering
          • AutoModelForVision2Seq
          • TFAutoModelForVision2Seq
          • FlaxAutoModelForVision2Seq
      • Callbacks
      • Configuration
      • Data Collator
      • Keras callbacks
      • Logging
      • Models
      • Text Generation
      • ONNX
      • Optimization
      • Model outputs
      • Pipelines
      • Processors
      • Quantization
      • Tokenizer
      • Trainer
      • DeepSpeed Integration
      • Feature Extractor
      • Image Processor
    • 🌍MODELS
      • 🌍TEXT MODELS
        • ALBERT
        • BART
        • BARThez
        • BARTpho
        • BERT
        • BertGeneration
        • BertJapanese
        • Bertweet
        • BigBird
        • BigBirdPegasus
        • BioGpt
        • Blenderbot
        • Blenderbot Small
        • BLOOM
        • BORT
        • ByT5
        • CamemBERT
        • CANINE
        • CodeGen
        • CodeLlama
        • ConvBERT
        • CPM
        • CPMANT
        • CTRL
        • DeBERTa
        • DeBERTa-v2
        • DialoGPT
        • DistilBERT
        • DPR
        • ELECTRA
        • Encoder Decoder Models
        • ERNIE
        • ErnieM
        • ESM
        • Falcon
        • FLAN-T5
        • FLAN-UL2
        • FlauBERT
        • FNet
        • FSMT
        • Funnel Transformer
        • GPT
        • GPT Neo
        • GPT NeoX
        • GPT NeoX Japanese
        • GPT-J
        • GPT2
        • GPTBigCode
        • GPTSAN Japanese
        • GPTSw3
        • HerBERT
        • I-BERT
        • Jukebox
        • LED
        • LLaMA
        • LLama2
        • Longformer
        • LongT5
        • LUKE
        • M2M100
        • MarianMT
        • MarkupLM
        • MBart and MBart-50
        • MEGA
        • MegatronBERT
        • MegatronGPT2
        • Mistral
        • mLUKE
        • MobileBERT
        • MPNet
        • MPT
        • MRA
        • MT5
        • MVP
        • NEZHA
        • NLLB
        • NLLB-MoE
        • NystrΓΆmformer
        • Open-Llama
        • OPT
        • Pegasus
        • PEGASUS-X
        • Persimmon
        • PhoBERT
        • PLBart
        • ProphetNet
        • QDQBert
        • RAG
        • REALM
        • Reformer
        • RemBERT
        • RetriBERT
        • RoBERTa
        • RoBERTa-PreLayerNorm
        • RoCBert
        • RoFormer
        • RWKV
        • Splinter
        • SqueezeBERT
        • SwitchTransformers
        • T5
        • T5v1.1
        • TAPEX
        • Transformer XL
        • UL2
        • UMT5
        • X-MOD
        • XGLM
        • XLM
        • XLM-ProphetNet
        • XLM-RoBERTa
        • XLM-RoBERTa-XL
        • XLM-V
        • XLNet
        • YOSO
      • 🌍VISION MODELS
        • BEiT
        • BiT
        • Conditional DETR
        • ConvNeXT
        • ConvNeXTV2
        • CvT
        • Deformable DETR
        • DeiT
        • DETA
        • DETR
        • DiNAT
        • DINO V2
        • DiT
        • DPT
        • EfficientFormer
        • EfficientNet
        • FocalNet
        • GLPN
        • ImageGPT
        • LeViT
        • Mask2Former
        • MaskFormer
        • MobileNetV1
        • MobileNetV2
        • MobileViT
        • MobileViTV2
        • NAT
        • PoolFormer
        • Pyramid Vision Transformer (PVT)
        • RegNet
        • ResNet
        • SegFormer
        • SwiftFormer
        • Swin Transformer
        • Swin Transformer V2
        • Swin2SR
        • Table Transformer
        • TimeSformer
        • UperNet
        • VAN
        • VideoMAE
        • Vision Transformer (ViT)
        • ViT Hybrid
        • ViTDet
        • ViTMAE
        • ViTMatte
        • ViTMSN
        • ViViT
        • YOLOS
      • 🌍AUDIO MODELS
        • Audio Spectrogram Transformer
        • Bark
        • CLAP
        • EnCodec
        • Hubert
        • MCTCT
        • MMS
        • MusicGen
        • Pop2Piano
        • SEW
        • SEW-D
        • Speech2Text
        • Speech2Text2
        • SpeechT5
        • UniSpeech
        • UniSpeech-SAT
        • VITS
        • Wav2Vec2
        • Wav2Vec2-Conformer
        • Wav2Vec2Phoneme
        • WavLM
        • Whisper
        • XLS-R
        • XLSR-Wav2Vec2
      • 🌍MULTIMODAL MODELS
        • ALIGN
        • AltCLIP
        • BLIP
        • BLIP-2
        • BridgeTower
        • BROS
        • Chinese-CLIP
        • CLIP
        • CLIPSeg
        • Data2Vec
        • DePlot
        • Donut
        • FLAVA
        • GIT
        • GroupViT
        • IDEFICS
        • InstructBLIP
        • LayoutLM
        • LayoutLMV2
        • LayoutLMV3
        • LayoutXLM
        • LiLT
        • LXMERT
        • MatCha
        • MGP-STR
        • Nougat
        • OneFormer
        • OWL-ViT
        • Perceiver
        • Pix2Struct
        • Segment Anything
        • Speech Encoder Decoder Models
        • TAPAS
        • TrOCR
        • TVLT
        • ViLT
        • Vision Encoder Decoder Models
        • Vision Text Dual Encoder
        • VisualBERT
        • X-CLIP
      • 🌍REINFORCEMENT LEARNING MODELS
        • Decision Transformer
        • Trajectory Transformer
      • 🌍TIME SERIES MODELS
        • Autoformer
        • Informer
        • Time Series Transformer
      • 🌍GRAPH MODELS
        • Graphormer
  • 🌍INTERNAL HELPERS
    • Custom Layers and Utilities
    • Utilities for pipelines
    • Utilities for Tokenizers
    • Utilities for Trainer
    • Utilities for Generation
    • Utilities for Image Processors
    • Utilities for Audio processing
    • General Utilities
    • Utilities for Time Series
Powered by GitBook
On this page
  • Data Collator
  • Default data collator
  • DefaultDataCollator
  • DataCollatorWithPadding
  • DataCollatorForTokenClassification
  • DataCollatorForSeq2Seq
  • DataCollatorForLanguageModeling
  • DataCollatorForWholeWordMask
  • DataCollatorForPermutationLanguageModeling
  1. API
  2. MAIN CLASSES

Data Collator

PreviousConfigurationNextKeras callbacks

Last updated 1 year ago

Data Collator

Data collators are objects that will form a batch by using a list of dataset elements as input. These elements are of the same type as the elements of train_dataset or eval_dataset.

To be able to build batches, data collators may apply some processing (like padding). Some of them (like ) also apply some random data augmentation (like random masking) on the formed batch.

Examples of use can be found in the or .

Default data collator

transformers.default_data_collator

( features: typing.List[InputDataClass]return_tensors = 'pt' )

Very simple data collator that simply collates batches of dict-like objects and performs special handling for potential keys named:

  • label: handles a single value (int or float) per object

  • label_ids: handles a list of values per object

Does not do any additional preprocessing: property names of the input object will be used as corresponding inputs to the model. See glue and ner for example of how it’s useful.

DefaultDataCollator

class transformers.DefaultDataCollator

( return_tensors: str = 'pt' )

Parameters

  • return_tensors (str) β€” The type of Tensor to return. Allowable values are β€œnp”, β€œpt” and β€œtf”.

Very simple data collator that simply collates batches of dict-like objects and performs special handling for potential keys named:

  • label: handles a single value (int or float) per object

  • label_ids: handles a list of values per object

Does not do any additional preprocessing: property names of the input object will be used as corresponding inputs to the model. See glue and ner for example of how it’s useful.

This is an object (like other data collators) rather than a pure function like default_data_collator. This can be helpful if you need to set a return_tensors value at initialization.

DataCollatorWithPadding

class transformers.DataCollatorWithPadding

( tokenizer: PreTrainedTokenizerBasepadding: typing.Union[bool, str, transformers.utils.generic.PaddingStrategy] = Truemax_length: typing.Optional[int] = Nonepad_to_multiple_of: typing.Optional[int] = Nonereturn_tensors: str = 'pt' )

Parameters

    • True or 'longest' (default): Pad to the longest sequence in the batch (or no padding if only a single sequence is provided).

    • 'max_length': Pad to a maximum length specified with the argument max_length or to the maximum acceptable input length for the model if that argument is not provided.

    • False or 'do_not_pad': No padding (i.e., can output a batch with sequences of different lengths).

  • max_length (int, optional) β€” Maximum length of the returned list and optionally padding length (see above).

  • pad_to_multiple_of (int, optional) β€” If set will pad the sequence to a multiple of the provided value.

    This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta).

  • return_tensors (str) β€” The type of Tensor to return. Allowable values are β€œnp”, β€œpt” and β€œtf”.

Data collator that will dynamically pad the inputs received.

DataCollatorForTokenClassification

class transformers.DataCollatorForTokenClassification

( tokenizer: PreTrainedTokenizerBasepadding: typing.Union[bool, str, transformers.utils.generic.PaddingStrategy] = Truemax_length: typing.Optional[int] = Nonepad_to_multiple_of: typing.Optional[int] = Nonelabel_pad_token_id: int = -100return_tensors: str = 'pt' )

Parameters

    • True or 'longest' (default): Pad to the longest sequence in the batch (or no padding if only a single sequence is provided).

    • 'max_length': Pad to a maximum length specified with the argument max_length or to the maximum acceptable input length for the model if that argument is not provided.

    • False or 'do_not_pad': No padding (i.e., can output a batch with sequences of different lengths).

  • max_length (int, optional) β€” Maximum length of the returned list and optionally padding length (see above).

  • pad_to_multiple_of (int, optional) β€” If set will pad the sequence to a multiple of the provided value.

    This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta).

  • label_pad_token_id (int, optional, defaults to -100) β€” The id to use when padding the labels (-100 will be automatically ignore by PyTorch loss functions).

  • return_tensors (str) β€” The type of Tensor to return. Allowable values are β€œnp”, β€œpt” and β€œtf”.

Data collator that will dynamically pad the inputs received, as well as the labels.

DataCollatorForSeq2Seq

class transformers.DataCollatorForSeq2Seq

( tokenizer: PreTrainedTokenizerBasemodel: typing.Optional[typing.Any] = Nonepadding: typing.Union[bool, str, transformers.utils.generic.PaddingStrategy] = Truemax_length: typing.Optional[int] = Nonepad_to_multiple_of: typing.Optional[int] = Nonelabel_pad_token_id: int = -100return_tensors: str = 'pt' )

Parameters

  • This is useful when using label_smoothing to avoid calculating loss twice.

    • True or 'longest' (default): Pad to the longest sequence in the batch (or no padding if only a single sequence is provided).

    • 'max_length': Pad to a maximum length specified with the argument max_length or to the maximum acceptable input length for the model if that argument is not provided.

    • False or 'do_not_pad': No padding (i.e., can output a batch with sequences of different lengths).

  • max_length (int, optional) β€” Maximum length of the returned list and optionally padding length (see above).

  • pad_to_multiple_of (int, optional) β€” If set will pad the sequence to a multiple of the provided value.

    This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta).

  • label_pad_token_id (int, optional, defaults to -100) β€” The id to use when padding the labels (-100 will be automatically ignored by PyTorch loss functions).

  • return_tensors (str) β€” The type of Tensor to return. Allowable values are β€œnp”, β€œpt” and β€œtf”.

Data collator that will dynamically pad the inputs received, as well as the labels.

DataCollatorForLanguageModeling

class transformers.DataCollatorForLanguageModeling

( tokenizer: PreTrainedTokenizerBasemlm: bool = Truemlm_probability: float = 0.15pad_to_multiple_of: typing.Optional[int] = Nonetf_experimental_compile: bool = Falsereturn_tensors: str = 'pt' )

Parameters

  • mlm (bool, optional, defaults to True) β€” Whether or not to use masked language modeling. If set to False, the labels are the same as the inputs with the padding tokens ignored (by setting them to -100). Otherwise, the labels are -100 for non-masked tokens and the value to predict for the masked token.

  • mlm_probability (float, optional, defaults to 0.15) β€” The probability with which to (randomly) mask tokens in the input, when mlm is set to True.

  • pad_to_multiple_of (int, optional) β€” If set will pad the sequence to a multiple of the provided value.

  • return_tensors (str) β€” The type of Tensor to return. Allowable values are β€œnp”, β€œpt” and β€œtf”.

Data collator used for language modeling. Inputs are dynamically padded to the maximum length of a batch if they are not all of the same length.

numpy_mask_tokens

( inputs: typing.Anyspecial_tokens_mask: typing.Optional[typing.Any] = None )

Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original.

tf_mask_tokens

( inputs: typing.Anyvocab_sizemask_token_idspecial_tokens_mask: typing.Optional[typing.Any] = None )

Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original.

torch_mask_tokens

( inputs: typing.Anyspecial_tokens_mask: typing.Optional[typing.Any] = None )

Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original.

DataCollatorForWholeWordMask

class transformers.DataCollatorForWholeWordMask

( tokenizer: PreTrainedTokenizerBasemlm: bool = Truemlm_probability: float = 0.15pad_to_multiple_of: typing.Optional[int] = Nonetf_experimental_compile: bool = Falsereturn_tensors: str = 'pt' )

Data collator used for language modeling that masks entire words.

  • collates batches of tensors, honoring their tokenizer’s pad_token

  • preprocesses batches for masked language modeling

numpy_mask_tokens

( inputs: typing.Anymask_labels: typing.Any )

Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original. Set β€˜mask_labels’ means we use whole word mask (wwm), we directly mask idxs according to it’s ref.

tf_mask_tokens

( inputs: typing.Anymask_labels: typing.Any )

Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original. Set β€˜mask_labels’ means we use whole word mask (wwm), we directly mask idxs according to it’s ref.

torch_mask_tokens

( inputs: typing.Anymask_labels: typing.Any )

Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original. Set β€˜mask_labels’ means we use whole word mask (wwm), we directly mask idxs according to it’s ref.

DataCollatorForPermutationLanguageModeling

class transformers.DataCollatorForPermutationLanguageModeling

( tokenizer: PreTrainedTokenizerBaseplm_probability: float = 0.16666666666666666max_span_length: int = 5return_tensors: str = 'pt' )

Data collator used for permutation language modeling.

  • collates batches of tensors, honoring their tokenizer’s pad_token

  • preprocesses batches for permutation language modeling with procedures specific to XLNet

numpy_mask_tokens

( inputs: typing.Any )

The masked tokens to be predicted for a particular sequence are determined by the following algorithm:

  1. Start from the beginning of the sequence by setting cur_len = 0 (number of tokens processed so far).

  2. Sample a span_length from the interval [1, max_span_length] (length of span of tokens to be masked)

  3. Reserve a context of length context_length = span_length / plm_probability to surround span to be masked

  4. Sample a starting point start_index from the interval [cur_len, cur_len + context_length - span_length] and mask tokens start_index:start_index + span_length

  5. Set cur_len = cur_len + context_length. If cur_len < max_len (i.e. there are tokens remaining in the sequence to be processed), repeat from Step 1.

tf_mask_tokens

( inputs: typing.Any )

The masked tokens to be predicted for a particular sequence are determined by the following algorithm:

  1. Start from the beginning of the sequence by setting cur_len = 0 (number of tokens processed so far).

  2. Sample a span_length from the interval [1, max_span_length] (length of span of tokens to be masked)

  3. Reserve a context of length context_length = span_length / plm_probability to surround span to be masked

  4. Sample a starting point start_index from the interval [cur_len, cur_len + context_length - span_length] and mask tokens start_index:start_index + span_length

  5. Set cur_len = cur_len + context_length. If cur_len < max_len (i.e. there are tokens remaining in the sequence to be processed), repeat from Step 1.

torch_mask_tokens

( inputs: typing.Any )

The masked tokens to be predicted for a particular sequence are determined by the following algorithm:

  1. Start from the beginning of the sequence by setting cur_len = 0 (number of tokens processed so far).

  2. Sample a span_length from the interval [1, max_span_length] (length of span of tokens to be masked)

  3. Reserve a context of length context_length = span_length / plm_probability to surround span to be masked

  4. Sample a starting point start_index from the interval [cur_len, cur_len + context_length - span_length] and mask tokens start_index:start_index + span_length

  5. Set cur_len = cur_len + context_length. If cur_len < max_len (i.e. there are tokens remaining in the sequence to be processed), repeat from Step 1.

tokenizer ( or ) β€” The tokenizer used for encoding the data.

padding (bool, str or , optional, defaults to True) β€” Select a strategy to pad the returned sequences (according to the model’s padding side and padding index) among:

tokenizer ( or ) β€” The tokenizer used for encoding the data.

padding (bool, str or , optional, defaults to True) β€” Select a strategy to pad the returned sequences (according to the model’s padding side and padding index) among:

tokenizer ( or ) β€” The tokenizer used for encoding the data.

model () β€” The model that is being trained. If set and has the prepare_decoder_input_ids_from_labels, use it to prepare the decoder_input_ids

padding (bool, str or , optional, defaults to True) β€” Select a strategy to pad the returned sequences (according to the model’s padding side and padding index) among:

tokenizer ( or ) β€” The tokenizer used for encoding the data.

For best performance, this data collator should be used with a dataset having items that are dictionaries or BatchEncoding, with the "special_tokens_mask" key, as returned by a or a with the argument return_special_tokens_mask=True.

This collator relies on details of the implementation of subword tokenization by , specifically that subword tokens are prefixed with ##. For tokenizers that do not adhere to this scheme, this collator will produce an output that is roughly equivalent to .DataCollatorForLanguageModeling.

🌍
🌍
DataCollatorForLanguageModeling
example scripts
example notebooks
<source>
<source>
<source>
PreTrainedTokenizer
PreTrainedTokenizerFast
PaddingStrategy
<source>
PreTrainedTokenizer
PreTrainedTokenizerFast
PaddingStrategy
<source>
PreTrainedTokenizer
PreTrainedTokenizerFast
PreTrainedModel
PaddingStrategy
<source>
PreTrainedTokenizer
PreTrainedTokenizerFast
PreTrainedTokenizer
PreTrainedTokenizerFast
<source>
<source>
<source>
<source>
BertTokenizer
<source>
<source>
<source>
<source>
<source>
<source>
<source>