Agents and Tools
Last updated
Last updated
Transformers Agents is an experimental API which is subject to change at any time. Results returned by the agents can vary as the APIs or underlying models are prone to change.
To learn more about agents and tools make sure to read the . This page contains the API docs for the underlying classes.
We provide three types of agents: uses inference endpoints for opensource models, uses a model of your choice locally and uses OpenAI closed models.
( url_endpointtoken = Nonechat_prompt_template = Nonerun_prompt_template = Noneadditional_tools = None )
Parameters
url_endpoint (str) — The name of the url endpoint to use.
token (str, optional) — The token to use as HTTP bearer authorization for remote files. If unset, will use the token generated when running boincai-cli login (stored in ~/.boincai).
chat_prompt_template (str, optional) — Pass along your own prompt if you want to override the default template for the chat method. Can be the actual prompt template or a repo ID (on the BOINC AI Hub). The prompt should be in a file named chat_prompt_template.txt in this repo in this case.
run_prompt_template (str, optional) — Pass along your own prompt if you want to override the default template for the run method. Can be the actual prompt template or a repo ID (on the BOINC AI Hub). The prompt should be in a file named run_prompt_template.txt in this repo in this case.
additional_tools (, list of tools or dictionary with tool values, optional) — Any additional tools to include on top of the default ones. If you pass along a tool with the same name as one of the default tools, that default tool will be overridden.
Agent that uses an inference endpoint to generate code.
Example:
Copied
( modeltokenizerchat_prompt_template = Nonerun_prompt_template = Noneadditional_tools = None )
Parameters
chat_prompt_template (str, optional) — Pass along your own prompt if you want to override the default template for the chat method. Can be the actual prompt template or a repo ID (on the BOINC AI Hub). The prompt should be in a file named chat_prompt_template.txt in this repo in this case.
run_prompt_template (str, optional) — Pass along your own prompt if you want to override the default template for the run method. Can be the actual prompt template or a repo ID (on the BOINC AI Hub). The prompt should be in a file named run_prompt_template.txt in this repo in this case.
Agent that uses a local model and tokenizer to generate code.
Example:
Copied
from_pretrained
( pretrained_model_name_or_path**kwargs )
Parameters
pretrained_model_name_or_path (str or os.PathLike) — The name of a repo on the Hub or a local path to a folder containing both model and tokenizer.
Convenience method to build a LocalAgent from a pretrained checkpoint.
Example:
Copied
( model = 'text-davinci-003'api_key = Nonechat_prompt_template = Nonerun_prompt_template = Noneadditional_tools = None )
Parameters
model (str, optional, defaults to "text-davinci-003") — The name of the OpenAI model to use.
api_key (str, optional) — The API key to use. If unset, will look for the environment variable "OPENAI_API_KEY".
chat_prompt_template (str, optional) — Pass along your own prompt if you want to override the default template for the chat method. Can be the actual prompt template or a repo ID (on the BOINC AI Hub). The prompt should be in a file named chat_prompt_template.txt in this repo in this case.
run_prompt_template (str, optional) — Pass along your own prompt if you want to override the default template for the run method. Can be the actual prompt template or a repo ID (on the BOINC AI Hub). The prompt should be in a file named run_prompt_template.txt in this repo in this case.
Agent that uses the openai API to generate code.
The openAI models are used in generation mode, so even for the chat() API, it’s better to use models like "text-davinci-003" over the chat-GPT variant. Proper support for chat-GPT models will come in a next version.
Example:
Copied
( deployment_idapi_key = Noneresource_name = Noneapi_version = '2022-12-01'is_chat_model = Nonechat_prompt_template = Nonerun_prompt_template = Noneadditional_tools = None )
Expand 8 parameters
Parameters
deployment_id (str) — The name of the deployed Azure openAI model to use.
api_key (str, optional) — The API key to use. If unset, will look for the environment variable "AZURE_OPENAI_API_KEY".
resource_name (str, optional) — The name of your Azure OpenAI Resource. If unset, will look for the environment variable "AZURE_OPENAI_RESOURCE_NAME".
api_version (str, optional, default to "2022-12-01") — The API version to use for this agent.
is_chat_mode (bool, optional) — Whether you are using a completion model or a chat model (see note above, chat models won’t be as efficient). Will default to gpt being in the deployment_id or not.
chat_prompt_template (str, optional) — Pass along your own prompt if you want to override the default template for the chat method. Can be the actual prompt template or a repo ID (on the BOINC AI Hub). The prompt should be in a file named chat_prompt_template.txt in this repo in this case.
run_prompt_template (str, optional) — Pass along your own prompt if you want to override the default template for the run method. Can be the actual prompt template or a repo ID (on the BOINC AIHub). The prompt should be in a file named run_prompt_template.txt in this repo in this case.
The openAI models are used in generation mode, so even for the chat() API, it’s better to use models like "text-davinci-003" over the chat-GPT variant. Proper support for chat-GPT models will come in a next version.
Example:
Copied
( chat_prompt_template = Nonerun_prompt_template = Noneadditional_tools = None )
Parameters
chat_prompt_template (str, optional) — Pass along your own prompt if you want to override the default template for the chat method. Can be the actual prompt template or a repo ID (on the BOINC AI Hub). The prompt should be in a file named chat_prompt_template.txt in this repo in this case.
run_prompt_template (str, optional) — Pass along your own prompt if you want to override the default template for the run method. Can be the actual prompt template or a repo ID (on the BOINC AI Hub). The prompt should be in a file named run_prompt_template.txt in this repo in this case.
Base class for all agents which contains the main API methods.
chat
( taskreturn_code = Falseremote = False**kwargs )
Parameters
task (str) — The task to perform
return_code (bool, optional, defaults to False) — Whether to just return code and not evaluate it.
remote (bool, optional, defaults to False) — Whether or not to use remote tools (inference endpoints) instead of local ones.
kwargs (additional keyword arguments, optional) — Any keyword argument to send to the agent when evaluating the code.
Sends a new request to the agent in a chat. Will use the previous ones in its history.
Example:
Copied
run
( taskreturn_code = Falseremote = False**kwargs )
Parameters
task (str) — The task to perform
return_code (bool, optional, defaults to False) — Whether to just return code and not evaluate it.
remote (bool, optional, defaults to False) — Whether or not to use remote tools (inference endpoints) instead of local ones.
kwargs (additional keyword arguments, optional) — Any keyword argument to send to the agent when evaluating the code.
Sends a request to the agent.
Example:
Copied
prepare_for_new_chat
( )
transformers.load_tool
( task_or_repo_idmodel_repo_id = Noneremote = Falsetoken = None**kwargs )
Expand 5 parameters
Parameters
task_or_repo_id (str) — The task for which to load the tool or a repo ID of a tool on the Hub. Tasks implemented in Transformers are:
"document-question-answering"
"image-captioning"
"image-question-answering"
"image-segmentation"
"speech-to-text"
"summarization"
"text-classification"
"text-question-answering"
"text-to-speech"
"translation"
model_repo_id (str, optional) — Use this argument to use a different model than the default one for the tool you selected.
remote (bool, optional, defaults to False) — Whether to use your tool by downloading the model or (if it is available) with an inference endpoint.
token (str, optional) — The token to identify you on hf.co. If unset, will use the token generated when running boincai-cli login (stored in ~/.boincai).
kwargs (additional keyword arguments, optional) — Additional keyword arguments that will be split in two: all arguments relevant to the Hub (such as cache_dir, revision, subfolder) will be used when downloading the files for your tool, and the others will be passed along to its init.
Main function to quickly load a tool, be it on the Hub or in the Transformers library.
( *args**kwargs )
A base class for the functions used by the agent. Subclass this and implement the __call__ method as well as the following class attributes:
description (str) — A short description of what your tool does, the inputs it expects and the output(s) it will return. For instance ‘This is a tool that downloads a file from a url. It takes the url as input, and returns the text contained in the file’.
name (str) — A performative name that will be used for your tool in the prompt to the agent. For instance "text-classifier" or "image_generator".
inputs (List[str]) — The list of modalities expected for the inputs (in the same order as in the call). Modalitiies should be "text", "image" or "audio". This is only used by launch_gradio_demo or to make a nice space from your tool.
outputs (List[str]) — The list of modalities returned but the tool (in the same order as the return of the call method). Modalitiies should be "text", "image" or "audio". This is only used by launch_gradio_demo or to make a nice space from your tool.
from_gradio
( gradio_tool )
from_hub
( repo_id: strmodel_repo_id: typing.Optional[str] = Nonetoken: typing.Optional[str] = Noneremote: bool = False**kwargs )
Parameters
repo_id (str) — The name of the repo on the Hub where your tool is defined.
model_repo_id (str, optional) — If your tool uses a model and you want to use a different model than the default, you can pass a second repo ID or an endpoint url to this argument.
token (str, optional) — The token to identify you on hf.co. If unset, will use the token generated when running boincai-cli login (stored in ~/.boincai).
remote (bool, optional, defaults to False) — Whether to use your tool by downloading the model or (if it is available) with an inference endpoint.
kwargs (additional keyword arguments, optional) — Additional keyword arguments that will be split in two: all arguments relevant to the Hub (such as cache_dir, revision, subfolder) will be used when downloading the files for your tool, and the others will be passed along to its init.
Loads a tool defined on the Hub.
push_to_hub
( repo_id: strcommit_message: str = 'Upload tool'private: typing.Optional[bool] = Nonetoken: typing.Union[bool, str, NoneType] = Nonecreate_pr: bool = False )
Parameters
repo_id (str) — The name of the repository you want to push your tool to. It should contain your organization name when pushing to a given organization.
commit_message (str, optional, defaults to "Upload tool") — Message to commit while pushing.
private (bool, optional) — Whether or not the repository created should be private.
token (bool or str, optional) — The token to use as HTTP bearer authorization for remote files. If unset, will use the token generated when running boincai-cli login (stored in ~/.boincai).
create_pr (bool, optional, defaults to False) — Whether or not to create a PR with the uploaded files or directly commit.
Upload the tool to the Hub.
save
( output_dir )
Parameters
output_dir (str) — The folder in which you want to save your tool.
Saves the relevant code files for your tool so it can be pushed to the Hub. This will copy the code of your tool in output_dir as well as autogenerate:
a config file named tool_config.json
an app.py file so that your tool can be converted to a space
a requirements.txt containing the names of the module used by your tool (as detected when inspecting its code)
You should only use this method to save tools that are defined in a separate module (not __main__).
setup
( )
Overwrite this method here for any operation that is expensive and needs to be executed before you start using your tool. Such as loading a big model.
( model = Nonepre_processor = Nonepost_processor = Nonedevice = Nonedevice_map = Nonemodel_kwargs = Nonetoken = None**hub_kwargs )
Expand 8 parameters
Parameters
pre_processor (str or Any, optional) — The name of the checkpoint to use for the pre-processor, or the instantiated pre-processor (can be a tokenizer, an image processor, a feature extractor or a processor). Will default to the value of model if unset.
post_processor (str or Any, optional) — The name of the checkpoint to use for the post-processor, or the instantiated pre-processor (can be a tokenizer, an image processor, a feature extractor or a processor). Will default to the pre_processor if unset.
device (int, str or torch.device, optional) — The device on which to execute the model. Will default to any accelerator available (GPU, MPS etc…), the CPU otherwise.
device_map (str or dict, optional) — If passed along, will be used to instantiate the model.
model_kwargs (dict, optional) — Any keyword argument to send to the model instantiation.
token (str, optional) — The token to use as HTTP bearer authorization for remote files. If unset, will use the token generated when running boincai-cli login (stored in ~/.boincai).
hub_kwargs (additional keyword arguments, optional) — Any additional keyword argument to send to the methods that will load the data from the Hub.
model_class (type) — The class to use to load the model in this tool.
default_checkpoint (str) — The default checkpoint that should be used when the user doesn’t specify one.
decode
( outputs )
Uses the post_processor to decode the model output.
encode
( raw_inputs )
Uses the pre_processor to prepare the inputs for the model.
forward
( inputs )
Sends the inputs through the model.
setup
( )
Instantiates the pre_processor, model and post_processor if necessary.
( endpoint_url = Nonetoken = Nonetool_class = None )
Parameters
endpoint_url (str) — The url of the endpoint to use.
token (str, optional) — The token to use as HTTP bearer authorization for remote files. If unset, will use the token generated when running boincai-cli login (stored in ~/.boincai).
tool_class (type, optional) — The corresponding tool_class if this is a remote version of an existing tool. Will help determine when the output should be converted to another type (like images).
extract_outputs
( outputs )
prepare_inputs
( *args**kwargs )
Prepare the inputs received for the HTTP client sending data to the endpoint. Positional arguments will be matched with the signature of the tool_class if it was provided at instantation. Images will be encoded into bytes.
transformers.launch_gradio_demo
( tool_class: Tool )
Parameters
tool_class (type) — The class of the tool for which to launch the demo.
Launches a gradio demo for a tool. The corresponding tool class needs to properly implement the class attributes inputs and outputs.
Agents can handle any type of object in-between tools; tools, being completely multimodal, can accept and return text, image, audio, video, among other types. In order to increase compatibility between tools, as well as to correctly render these returns in ipython (jupyter, colab, ipython notebooks, …), we implement wrapper classes around these types.
The wrapped objects should continue behaving as initially; a text object should still behave as a string, an image object should still behave as a PIL.Image.
These types have three specific purposes:
Calling to_raw on the type should return the underlying object
Calling to_string on the type should return the object as a string: that can be the string in case of an AgentText but will be the path of the serialized version of the object in other instances
Displaying it in an ipython kernel should display the object correctly
( value )
Text type returned by the agent. Behaves as a string.
( value )
Image type returned by the agent. Behaves as a PIL.Image.
to_raw
( )
Returns the “raw” version of that object. In the case of an AgentImage, it is a PIL.Image.
to_string
( )
Returns the stringified version of that object. In the case of an AgentImage, it is a path to the serialized version of the image.
( valuesamplerate = 16000 )
Audio type returned by the agent.
to_raw
( )
Returns the “raw” version of that object. It is a torch.Tensor object.
to_string
( )
Returns the stringified version of that object. In the case of an AgentAudio, it is a path to the serialized version of the audio.
model () — The model to use for the agent.
tokenizer () — The tokenizer to use for the agent.
additional_tools (, list of tools or dictionary with tool values, optional) — Any additional tools to include on top of the default ones. If you pass along a tool with the same name as one of the default tools, that default tool will be overridden.
kwargs (Dict[str, Any], optional) — Keyword arguments passed along to .
additional_tools (, list of tools or dictionary with tool values, optional) — Any additional tools to include on top of the default ones. If you pass along a tool with the same name as one of the default tools, that default tool will be overridden.
additional_tools (, list of tools or dictionary with tool values, optional) — Any additional tools to include on top of the default ones. If you pass along a tool with the same name as one of the default tools, that default tool will be overridden.
Agent that uses Azure OpenAI to generate code. See the to learn how to deploy an openAI model on Azure
additional_tools (, list of tools or dictionary with tool values, optional) — Any additional tools to include on top of the default ones. If you pass along a tool with the same name as one of the default tools, that default tool will be overridden.
Clears the history of prior calls to .
You can also override the method if your tool as an expensive operation to perform before being usable (such as loading a model). will be called the first time you use your tool, but not at instantiation.
Creates a from a gradio tool.
model (str or , optional) — The name of the checkpoint to use for the model, or the instantiated model. If unset, will default to the value of the class attribute default_checkpoint.
A tailored towards Transformer models. On top of the class attributes of the base class , you will need to specify:
pre_processor_class (type, optional, defaults to ) — The class to use to load the pre-processor
post_processor_class (type, optional, defaults to ) — The class to use to load the post-processor (when different from the pre-processor).
A that will make requests to an inference endpoint.
You can override this method in your custom class of to apply some custom post-processing of the outputs of the endpoint.
You can override this method in your custom class of .