Utilities for pipelines
Last updated
Last updated
This page lists all the utility functions the library provides for pipelines.
Most of those are only useful if you are studying the code of the models in the library.
( )
Base interface for handling arguments for each .
( )
Handles arguments for zero-shot for text classification by turning each possible label into an NLI premise/hypothesis pair.
( )
QuestionAnsweringPipeline requires the user to provide multiple arguments (i.e. question & context) to be mapped to internal SquadExample.
QuestionAnsweringArgumentHandler manages all the possible to create a SquadExample from the command-line supplied arguments.
( output_path: typing.Optional[str]input_path: typing.Optional[str]column: typing.Optional[str]overwrite: bool = False )
Parameters
output_path (str, optional) — Where to save the outgoing data.
input_path (str, optional) — Where to look for the input data.
column (str, optional) — The column to read.
overwrite (bool, optional, defaults to False) — Whether or not to overwrite the output_path.
Base class for all the pipeline supported data format both for reading and writing. Supported data formats currently includes:
JSON
CSV
stdin/stdout (pipe)
PipelineDataFormat also includes some utilities to work with multi-columns like mapping from datasets columns to pipelines keyword arguments through the dataset_kwarg_1=dataset_column_1 format.
from_str
Parameters
format (str) — The format of the desired pipeline. Acceptable values are "json", "csv" or "pipe".
output_path (str, optional) — Where to save the outgoing data.
input_path (str, optional) — Where to look for the input data.
column (str, optional) — The column to read.
overwrite (bool, optional, defaults to False) — Whether or not to overwrite the output_path.
Returns
The proper data format.
save
( data: typing.Union[dict, typing.List[dict]] )
Parameters
data (dict or list of dict) — The data to store.
save_binary
( data: typing.Union[dict, typing.List[dict]] ) → str
Parameters
data (dict or list of dict) — The data to store.
Returns
str
Path where the data has been saved.
Save the provided data object as a pickle-formatted binary data on the disk.
( output_path: typing.Optional[str]input_path: typing.Optional[str]column: typing.Optional[str]overwrite = False )
Parameters
output_path (str, optional) — Where to save the outgoing data.
input_path (str, optional) — Where to look for the input data.
column (str, optional) — The column to read.
overwrite (bool, optional, defaults to False) — Whether or not to overwrite the output_path.
Support for pipelines using CSV data format.
save
( data: typing.List[dict] )
Parameters
data (List[dict]) — The data to store.
( output_path: typing.Optional[str]input_path: typing.Optional[str]column: typing.Optional[str]overwrite = False )
Parameters
output_path (str, optional) — Where to save the outgoing data.
input_path (str, optional) — Where to look for the input data.
column (str, optional) — The column to read.
overwrite (bool, optional, defaults to False) — Whether or not to overwrite the output_path.
Support for pipelines using JSON file format.
save
( data: dict )
Parameters
data (dict) — The data to store.
Save the provided data object in a json file.
( output_path: typing.Optional[str]input_path: typing.Optional[str]column: typing.Optional[str]overwrite: bool = False )
Parameters
output_path (str, optional) — Where to save the outgoing data.
input_path (str, optional) — Where to look for the input data.
column (str, optional) — The column to read.
overwrite (bool, optional, defaults to False) — Whether or not to overwrite the output_path.
Read data from piped input to the python process. For multi columns data, columns should separated by
If columns are provided, then the output will be a dictionary with {column_x: value_x}
save
( data: dict )
Parameters
data (dict) — The data to store.
Print the data.
( task: strmodel: strreason: str )
Parameters
task (str) — The task of the pipeline.
model (str) — The model used by the pipeline.
reason (str) — The error message to display.
( format: stroutput_path: typing.Optional[str]input_path: typing.Optional[str]column: typing.Optional[str]overwrite = False ) →
Creates an instance of the right subclass of depending on format.
Save the provided data object with the representation for the current .
Save the provided data object with the representation for the current .
Raised by a when handling call.