Kind: instance method of TextClassificationPipelineReturns: Promise.<(Array<Object>|Object)> - A promise that resolves to an array or object containing the predicted labels and scores.
Param
Type
Default
Description
texts
any
The input texts to be classified.
options
Object
An optional object containing the following properties:
[options.topk]
number
1
The number of top predictions to be returned.
pipelines.TokenClassificationPipeline
Named Entity Recognition pipeline using any ModelForTokenClassification.
Example: Perform named entity recognition with Xenova/bert-base-NER.
Copied
Example: Perform named entity recognition with Xenova/bert-base-NER (and return all labels).
Kind: instance method of TokenClassificationPipelineReturns: Promise.<(Array<Object>|Object)> - A promise that resolves to an array or object containing the predicted labels and scores.
Param
Type
Description
texts
any
The input texts to be classified.
options
Object
An optional object containing the following properties:
pipelines.QuestionAnsweringPipeline
Question Answering pipeline using any ModelForQuestionAnswering.
Example: Run question answering with Xenova/distilbert-base-uncased-distilled-squad.
Kind: instance method of QuestionAnsweringPipelineReturns: QuestionAnsweringReturnType - A promise that resolves to an array or object containing the predicted answers and scores.
Param
Type
Default
Description
question
string | Array<string>
The question(s) to be answered.
context
string | Array<string>
The context(s) where the answer(s) can be found.
options
Object
An optional object containing the following properties:
[options.topk]
number
1
The number of top answer predictions to be returned.
pipelines.FillMaskPipeline
Masked language modeling prediction pipeline using any ModelWithLMHead.
Example: Perform masked language modelling (a.k.a. “fill-mask”) with Xenova/bert-base-uncased.
Copied
Example: Perform masked language modelling (a.k.a. “fill-mask”) with Xenova/bert-base-cased (and return top result).
Fill the masked token in the text(s) given as inputs.
Kind: instance method of FillMaskPipelineReturns: Promise.<(Array<Object>|Object)> - A promise that resolves to an array or object containing the predicted tokens and scores.
Param
Type
Default
Description
texts
any
The masked input texts.
options
Object
An optional object containing the following properties:
[options.topk]
number
5
The number of top predictions to be returned.
pipelines.Text2TextGenerationPipeline
Text2TextGenerationPipeline class for generating text using a model that performs text-to-text generation tasks.
text2TextGenerationPipeline._call(texts, [options]) ⇒ <code> Promise. < any > </code>
Fill the masked token in the text(s) given as inputs.
Kind: instance method of Text2TextGenerationPipelineReturns: Promise.<any> - An array of objects containing the score, predicted token, predicted token string, and the sequence with the predicted token filled in, or an array of such arrays (one for each input text). If only one input text is given, the output will be an array of objects.
Throws:
Error When the mask token is not found in the input text.
Param
Type
Default
Description
texts
string | Array<string>
The text or array of texts to be processed.
[options]
Object
{}
Options for the fill-mask pipeline.
[options.topk]
number
5
The number of top-k predictions to return.
pipelines.SummarizationPipeline
A pipeline for summarization tasks, inheriting from Text2TextGenerationPipeline.
Language generation pipeline using any ModelWithLMHead or ModelForCausalLM. This pipeline predicts the words that will follow a specified text prompt. NOTE: For the full list of generation parameters, see GenerationConfig.
Example: Text generation with Xenova/distilgpt2 (default settings).
Copied
Example: Text generation with Xenova/distilgpt2 (custom settings).
Copied
Example: Run code generation with Xenova/codegen-350M-mono.
textGenerationPipeline._call(texts, [generate_kwargs]) ⇒ <code> Promise. < any > </code>
Generates text based on an input prompt.
Kind: instance method of TextGenerationPipelineReturns: Promise.<any> - The generated text or texts.
Param
Type
Default
Description
texts
any
The input prompt or prompts to generate text from.
[generate_kwargs]
Object
{}
Additional arguments for text generation.
pipelines.ZeroShotClassificationPipeline
NLI-based zero-shot classification pipeline using a ModelForSequenceClassification trained on NLI (natural language inference) tasks. Equivalent of text-classification pipelines, but these models don’t require a hardcoded number of potential classes, they can be chosen at runtime. It usually means it’s slower but it is much more flexible.
Example: Zero shot classification with Xenova/mobilebert-uncased-mnli.
Copied
Example: Zero shot classification with Xenova/nli-deberta-v3-xsmall (multi-label).
Kind: instance method of ZeroShotClassificationPipelineReturns: Promise.<(Object|Array<Object>)> - The prediction(s), as a map (or list of maps) from label to score.
Param
Type
Default
Description
texts
Array.<any>
candidate_labels
Array.<string>
options
Object
Additional options:
[options.hypothesis_template]
string
""This example is {}.""
The template used to turn each candidate label into an NLI-style hypothesis. The candidate label will replace the {} placeholder.
[options.multi_label]
boolean
false
Whether or not multiple candidate labels can be true. If false, the scores are normalized such that the sum of the label likelihoods for each sequence is 1. If true, the labels are considered independent and probabilities are normalized for each candidate by doing a softmax of the entailment score vs. the contradiction score.
pipelines.FeatureExtractionPipeline
Feature extraction pipeline using no model head. This pipeline extracts the hidden states from the base transformer, which can be used as features in downstream tasks.
Example: Run feature extraction with bert-base-uncased (without pooling/normalization).
Copied
Example: Run feature extraction with bert-base-uncased (with pooling/normalization).
Copied
Example: Calculating embeddings with sentence-transformers models.
Kind: instance method of AudioClassificationPipelineReturns: Promise.<(Array<Object>|Object)> - A promise that resolves to an array or object containing the predicted labels and scores.
Param
Type
Default
Description
audio
any
The input audio files to be classified.
options
Object
An optional object containing the following properties:
[options.topk]
number
5
The number of top predictions to be returned.
pipelines.AutomaticSpeechRecognitionPipeline
Pipeline that aims at extracting spoken text contained within some audio.
Example: Transcribe English.
Copied
Example: Transcribe English w/ timestamps.
Copied
Example: Transcribe English w/ word-level timestamps.
Copied
Example: Transcribe French.
Copied
Example: Translate French to English.
Copied
Example: Transcribe/translate audio longer than 30 seconds.
Asynchronously processes audio and generates text transcription using the model.
Kind: instance method of AutomaticSpeechRecognitionPipelineReturns: Promise.<Object> - A Promise that resolves to an object containing the transcription text and optionally timestamps if return_timestamps is true.
Param
Type
Default
Description
audio
Float32Array | Array<Float32Array>
The audio to be transcribed. Can be a single Float32Array or an array of Float32Arrays.
[kwargs]
Object
{}
Optional arguments.
[kwargs.return_timestamps]
boolean | 'word'
Whether to return timestamps or not. Default is false.
[kwargs.chunk_length_s]
number
The length of audio chunks to process in seconds. Default is 0 (no chunking).
[kwargs.stride_length_s]
number
The length of overlap between consecutive audio chunks in seconds. If not provided, defaults to chunk_length_s / 6.
[kwargs.chunk_callback]
ChunkCallback
Callback function to be called with each chunk processed.
[kwargs.force_full_sequences]
boolean
Whether to force outputting full sequences or not. Default is false.
[kwargs.language]
string
The source language. Default is null, meaning it should be auto-detected. Use this to potentially improve performance if the source language is known.
[kwargs.task]
string
The task to perform. Default is null, meaning it should be auto-detected.
[kwargs.forced_decoder_ids]
Array.<Array<number>>
A list of pairs of integers which indicates a mapping from generation indices to token indices that will be forced before sampling. For example, [[1, 123]] means the second generated token will always be a token of index 123.
pipelines.ImageToTextPipeline
Image To Text pipeline using a AutoModelForVision2Seq. This pipeline predicts a caption for a given image.
Example: Generate a caption for an image w/ Xenova/vit-gpt2-image-captioning.
Kind: instance method of ImageToTextPipelineReturns: Promise.<(Object|Array<Object>)> - A Promise that resolves to an object (or array of objects) containing the generated text(s).
Param
Type
Default
Description
images
Array.<any>
The images to be captioned.
[generate_kwargs]
Object
{}
Optional generation arguments.
pipelines.ImageClassificationPipeline
Image classification pipeline using any AutoModelForImageClassification. This pipeline predicts the class of an image.
Example: Classify an image.
Copied
Example: Classify an image and return top n classes.
Copied
Example: Classify an image and return all classes.
Kind: instance method of ImageSegmentationPipelineReturns: Promise.<Array> - The annotated segments.
Param
Type
Default
Description
images
Array
The input images.
options
Object
The options to use for segmentation.
[options.threshold]
number
0.5
Probability threshold to filter out predicted masks.
[options.mask_threshold]
number
0.5
Threshold to use when turning the predicted masks into binary values.
[options.overlap_mask_area_threshold]
number
0.8
Mask overlap threshold to eliminate small, disconnected segments.
[options.subtask]
null | string
Segmentation task to be performed. One of [panoptic, instance, and semantic], depending on model capabilities. If not set, the pipeline will attempt to resolve (in that order).
[options.label_ids_to_fuse]
Array
List of label ids to fuse. If not set, do not fuse any labels.
[options.target_sizes]
Array
List of target sizes for the input images. If not set, use the original image sizes.
pipelines.ZeroShotImageClassificationPipeline
Zero shot image classification pipeline. This pipeline predicts the class of an image when you provide an image and a set of candidate_labels.
Example: Zero shot image classification w/ Xenova/clip-vit-base-patch32.
The task of the pipeline. Useful for specifying subtasks.
[options.model]
PreTrainedModel
The model to use.
[options.tokenizer]
PreTrainedTokenizer
The tokenizer to use.
[options.processor]
Processor
The processor to use.
zeroShotImageClassificationPipeline._call(images, candidate_labels, options) ⇒ <code> Promise. < any > </code>
Classify the input images with candidate labels using a zero-shot approach.
Kind: instance method of ZeroShotImageClassificationPipelineReturns: Promise.<any> - An array of classifications for each input image or a single classification object if only one input image is provided.
Param
Type
Description
images
Array
The input images.
candidate_labels
Array.<string>
The candidate labels.
options
Object
The options for the classification.
[options.hypothesis_template]
string
The hypothesis template to use for zero-shot classification. Default: "This is a photo of {}".
pipelines.ObjectDetectionPipeline
Object detection pipeline using any AutoModelForObjectDetection. This pipeline predicts bounding boxes of objects and their classes.
Example: Run object-detection with facebook/detr-resnet-50.
Whether to return the boxes coordinates in percentage (true) or in pixels (false).
pipelines.DocumentQuestionAnsweringPipeline
Document Question Answering pipeline using any AutoModelForDocumentQuestionAnswering. The inputs/outputs are similar to the (extractive) question answering pipeline; however, the pipeline takes an image (and optional OCR’d words/boxes) as input instead of text context.
Example: Answer questions about a document with Xenova/donut-base-finetuned-docvqa.
Answer the question given as input by using the document.
Kind: instance method of DocumentQuestionAnsweringPipelineReturns: Promise.<(Object|Array<Object>)> - A Promise that resolves to an object (or array of objects) containing the generated text(s).
Param
Type
Default
Description
image
any
The image of the document to use.
question
string
A question to ask of the document.
[generate_kwargs]
Object
{}
Optional generation arguments.
pipelines.TextToAudioPipeline
Text-to-audio generation pipeline using any AutoModelForTextToWaveform or AutoModelForTextToSpectrogram. This pipeline generates an audio file from an input text and optional other conditional inputs.
Example: Generate audio from text with Xenova/speecht5_tts.
Copied
You can then save the audio to a .wav file with the wavefile package:
let classifier = await pipeline('token-classification', 'Xenova/bert-base-NER');
let output = await classifier('My name is Sarah and I live in London');
// [
// { entity: 'B-PER', score: 0.9980202913284302, index: 4, word: 'Sarah' },
// { entity: 'B-LOC', score: 0.9994474053382874, index: 9, word: 'London' }
// ]
let question = 'Who was Jim Henson?';
let context = 'Jim Henson was a nice puppet.';
let answerer = await pipeline('question-answering', 'Xenova/distilbert-base-uncased-distilled-squad');
let output = await answerer(question, context);
// {
// "answer": "a nice puppet",
// "score": 0.5768911502526741
// }
let unmasker = await pipeline('fill-mask', 'Xenova/bert-base-cased');
let output = await unmasker('The goal of life is [MASK].');
// [
// { token_str: 'survival', score: 0.06137419492006302, token: 8115, sequence: 'The goal of life is survival.' },
// { token_str: 'love', score: 0.03902450203895569, token: 1567, sequence: 'The goal of life is love.' },
// { token_str: 'happiness', score: 0.03253183513879776, token: 9266, sequence: 'The goal of life is happiness.' },
// { token_str: 'freedom', score: 0.018736306577920914, token: 4438, sequence: 'The goal of life is freedom.' },
// { token_str: 'life', score: 0.01859794743359089, token: 1297, sequence: 'The goal of life is life.' }
// ]
let unmasker = await pipeline('fill-mask', 'Xenova/bert-base-cased');
let output = await unmasker('The Milky Way is a [MASK] galaxy.', { topk: 1 });
// [{ token_str: 'spiral', score: 0.6299987435340881, token: 14061, sequence: 'The Milky Way is a spiral galaxy.' }]
let generator = await pipeline('text2text-generation', 'Xenova/LaMini-Flan-T5-783M');
let output = await generator('how can I become more healthy?', {
max_new_tokens: 100,
});
// [ 'To become more healthy, you can: 1. Eat a balanced diet with plenty of fruits, vegetables, whole grains, lean proteins, and healthy fats. 2. Stay hydrated by drinking plenty of water. 3. Get enough sleep and manage stress levels. 4. Avoid smoking and excessive alcohol consumption. 5. Regularly exercise and maintain a healthy weight. 6. Practice good hygiene and sanitation. 7. Seek medical attention if you experience any health issues.' ]
let text = 'The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, ' +
'and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. ' +
'During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest ' +
'man-made structure in the world, a title it held for 41 years until the Chrysler Building in New ' +
'York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to ' +
'the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the ' +
'Chrysler Building by 5.2 metres (17 ft). Excluding transmitters, the Eiffel Tower is the second ' +
'tallest free-standing structure in France after the Millau Viaduct.';
let generator = await pipeline('summarization', 'Xenova/distilbart-cnn-6-6');
let output = await generator(text, {
max_new_tokens: 100,
});
// [{ summary_text: ' The Eiffel Tower is about the same height as an 81-storey building and the tallest structure in Paris. It is the second tallest free-standing structure in France after the Millau Viaduct.' }]
let translator = await pipeline('translation', 'Xenova/nllb-200-distilled-600M');
let output = await translator('जीवन एक चॉकलेट बॉक्स की तरह है।', {
src_lang: 'hin_Deva', // Hindi
tgt_lang: 'fra_Latn', // French
});
// [{ translation_text: 'La vie est comme une boîte à chocolat.' }]
let translator = await pipeline('translation', 'Xenova/m2m100_418M');
let output = await translator('生活就像一盒巧克力。', {
src_lang: 'zh', // Chinese
tgt_lang: 'en', // English
});
// [{ translation_text: 'Life is like a box of chocolate.' }]
let translator = await pipeline('translation', 'Xenova/mbart-large-50-many-to-many-mmt');
let output = await translator('संयुक्त राष्ट्र के प्रमुख का कहना है कि सीरिया में कोई सैन्य समाधान नहीं है', {
src_lang: 'hi_IN', // Hindi
tgt_lang: 'fr_XX', // French
});
// [{ translation_text: 'Le chef des Nations affirme qu 'il n 'y a military solution in Syria.' }]
let text = 'I enjoy walking with my cute dog,';
let classifier = await pipeline('text-generation', 'Xenova/distilgpt2');
let output = await classifier(text);
// [{ generated_text: "I enjoy walking with my cute dog, and I love to play with the other dogs." }]
let text = 'Once upon a time, there was';
let classifier = await pipeline('text-generation', 'Xenova/distilgpt2');
let output = await classifier(text, {
temperature: 2,
max_new_tokens: 10,
repetition_penalty: 1.5,
no_repeat_ngram_size: 2,
num_beams: 2,
num_return_sequences: 2,
});
// [{
// "generated_text": "Once upon a time, there was an abundance of information about the history and activities that"
// }, {
// "generated_text": "Once upon a time, there was an abundance of information about the most important and influential"
// }]
let text = 'def fib(n):';
let classifier = await pipeline('text-generation', 'Xenova/codegen-350M-mono');
let output = await classifier(text, {
max_new_tokens: 44,
});
// [{
// generated_text: 'def fib(n):\n' +
// ' if n == 0:\n' +
// ' return 0\n' +
// ' elif n == 1:\n' +
// ' return 1\n' +
// ' else:\n' +
// ' return fib(n-1) + fib(n-2)\n'
// }]
let text = 'Last week I upgraded my iOS version and ever since then my phone has been overheating whenever I use your app.';
let labels = [ 'mobile', 'billing', 'website', 'account access' ];
let classifier = await pipeline('zero-shot-classification', 'Xenova/mobilebert-uncased-mnli');
let output = await classifier(text, labels);
// {
// sequence: 'Last week I upgraded my iOS version and ever since then my phone has been overheating whenever I use your app.',
// labels: [ 'mobile', 'website', 'billing', 'account access' ],
// scores: [ 0.5562091040482018, 0.1843621307860853, 0.13942646639336376, 0.12000229877234923 ]
// }
let text = 'I have a problem with my iphone that needs to be resolved asap!';
let labels = [ 'urgent', 'not urgent', 'phone', 'tablet', 'computer' ];
let classifier = await pipeline('zero-shot-classification', 'Xenova/nli-deberta-v3-xsmall');
let output = await classifier(text, labels, { multi_label: true });
// {
// sequence: 'I have a problem with my iphone that needs to be resolved asap!',
// labels: [ 'urgent', 'phone', 'computer', 'tablet', 'not urgent' ],
// scores: [ 0.9958870956360275, 0.9923963400697035, 0.002333537946160235, 0.0015134138567598765, 0.0010699384208377163 ]
// }
let extractor = await pipeline('feature-extraction', 'Xenova/bert-base-uncased', { revision: 'default' });
let output = await extractor('This is a simple test.');
// Tensor {
// type: 'float32',
// data: Float32Array [0.05939924716949463, 0.021655935794115067, ...],
// dims: [1, 8, 768]
// }
let url = 'https://boincai.com/datasets/Xenova/transformers.js-docs/resolve/main/jfk.wav';
let transcriber = await pipeline('automatic-speech-recognition', 'Xenova/whisper-tiny.en');
let output = await transcriber(url);
// { text: " And so my fellow Americans ask not what your country can do for you, ask what you can do for your country." }
let url = 'https://boincai.com/datasets/Xenova/transformers.js-docs/resolve/main/jfk.wav';
let transcriber = await pipeline('automatic-speech-recognition', 'Xenova/whisper-tiny.en');
let output = await transcriber(url, { return_timestamps: true });
// {
// text: " And so my fellow Americans ask not what your country can do for you, ask what you can do for your country."
// chunks: [
// { timestamp: [0, 8], text: " And so my fellow Americans ask not what your country can do for you" }
// { timestamp: [8, 11], text: " ask what you can do for your country." }
// ]
// }
let url = 'https://boincai.com/datasets/Xenova/transformers.js-docs/resolve/main/jfk.wav';
let transcriber = await pipeline('automatic-speech-recognition', 'Xenova/whisper-tiny.en', {
revision: 'output_attentions',
});
let output = await transcriber(url, { return_timestamps: 'word' });
// {
// "text": " And so my fellow Americans ask not what your country can do for you ask what you can do for your country.",
// "chunks": [
// { "text": " And", "timestamp": [0, 0.78] },
// { "text": " so", "timestamp": [0.78, 1.06] },
// { "text": " my", "timestamp": [1.06, 1.46] },
// ...
// { "text": " for", "timestamp": [9.72, 9.92] },
// { "text": " your", "timestamp": [9.92, 10.22] },
// { "text": " country.", "timestamp": [10.22, 13.5] }
// ]
// }
let url = 'https://boincai.com/datasets/Xenova/transformers.js-docs/resolve/main/french-audio.mp3';
let transcriber = await pipeline('automatic-speech-recognition', 'Xenova/whisper-small');
let output = await transcriber(url, { language: 'french', task: 'transcribe' });
// { text: " J'adore, j'aime, je n'aime pas, je déteste." }
let url = 'https://boincai.com/datasets/Xenova/transformers.js-docs/resolve/main/french-audio.mp3';
let transcriber = await pipeline('automatic-speech-recognition', 'Xenova/whisper-small');
let output = await transcriber(url, { language: 'french', task: 'translate' });
// { text: " I love, I like, I don't like, I hate." }
let url = 'https://boincai.com/datasets/Xenova/transformers.js-docs/resolve/main/ted_60.wav';
let transcriber = await pipeline('automatic-speech-recognition', 'Xenova/whisper-tiny.en');
let output = await transcriber(url, { chunk_length_s: 30, stride_length_s: 5 });
// { text: " So in college, I was a government major, which means [...] So I'd start off light and I'd bump it up" }
let url = 'https://boincai.com/datasets/Xenova/transformers.js-docs/resolve/main/cats.jpg';
let captioner = await pipeline('image-to-text', 'Xenova/vit-gpt2-image-captioning');
let output = await captioner(url);
// [{ generated_text: 'a cat laying on a couch with another cat' }]
let classifier = await pipeline('image-classification', 'Xenova/vit-base-patch16-224');
let url = 'https://boincai.com/datasets/Xenova/transformers.js-docs/resolve/main/tiger.jpg';
let output = await classifier(url);
// [
// {label: 'tiger, Panthera tigris', score: 0.632695734500885},
// ]
let image = 'https://boincai.com/datasets/Xenova/transformers.js-docs/resolve/main/invoice.png';
let question = 'What is the invoice number?';
let qa_pipeline = await pipeline('document-question-answering', 'Xenova/donut-base-finetuned-docvqa');
let output = await qa_pipeline(image, question);
// [{ answer: 'us-001' }]
let speaker_embeddings = 'https://boincai.com/datasets/Xenova/transformers.js-docs/resolve/main/speaker_embeddings.bin';
let synthesizer = await pipeline('text-to-speech', 'Xenova/speecht5_tts', { quantized: false });
let out = await synthesizer('Hello, my dog is cute', { speaker_embeddings });
// {
// audio: Float32Array(26112) [-0.00005657337896991521, 0.00020583874720614403, ...],
// sampling_rate: 16000
// }
import wavefile from 'wavefile';
import fs from 'fs';
let wav = new wavefile.WaveFile();
wav.fromScratch(1, out.sampling_rate, '32f', out.audio);
fs.writeFileSync('out.wav', wav.toBuffer());
