> For the complete documentation index, see [llms.txt](https://boinc-ai.gitbook.io/transformers/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://boinc-ai.gitbook.io/transformers/api/models/vision-models/bit.md).

# BiT

## Big Transfer (BiT)

### Overview

The BiT model was proposed in [Big Transfer (BiT): General Visual Representation Learning](https://arxiv.org/abs/1912.11370) by Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, Neil Houlsby. BiT is a simple recipe for scaling up pre-training of [ResNet](https://huggingface.co/docs/transformers/model_doc/resnet)-like architectures (specifically, ResNetv2). The method results in significant improvements for transfer learning.

The abstract from the paper is the following:

*Transfer of pre-trained representations improves sample efficiency and simplifies hyperparameter tuning when training deep neural networks for vision. We revisit the paradigm of pre-training on large supervised datasets and fine-tuning the model on a target task. We scale up pre-training, and propose a simple recipe that we call Big Transfer (BiT). By combining a few carefully selected components, and transferring using a simple heuristic, we achieve strong performance on over 20 datasets. BiT performs well across a surprisingly wide range of data regimes — from 1 example per class to 1M total examples. BiT achieves 87.5% top-1 accuracy on ILSVRC-2012, 99.4% on CIFAR-10, and 76.3% on the 19 task Visual Task Adaptation Benchmark (VTAB). On small datasets, BiT attains 76.8% on ILSVRC-2012 with 10 examples per class, and 97.0% on CIFAR-10 with 10 examples per class. We conduct detailed analysis of the main components that lead to high transfer performance.*

Tips:

* BiT models are equivalent to ResNetv2 in terms of architecture, except that: 1) all batch normalization layers are replaced by [group normalization](https://arxiv.org/abs/1803.08494), 2) [weight standardization](https://arxiv.org/abs/1903.10520) is used for convolutional layers. The authors show that the combination of both is useful for training with large batch sizes, and has a significant impact on transfer learning.

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/google-research/big_transfer).

### Resources

A list of official BOINC AI and community (indicated by 🌎) resources to help you get started with BiT.

Image Classification

* [BitForImageClassification](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bit#transformers.BitForImageClassification) is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
* See also: [Image classification task guide](https://huggingface.co/docs/transformers/tasks/image_classification)

If you’re interested in submitting a resource to be included here, please feel free to open a Pull Request and we’ll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

### BitConfig

#### class transformers.BitConfig

[\<source>](https://github.com/huggingface/transformers/blob/v4.34.1/src/transformers/models/bit/configuration_bit.py#L29)

( num\_channels = 3embedding\_size = 64hidden\_sizes = \[256, 512, 1024, 2048]depths = \[3, 4, 6, 3]layer\_type = 'preactivation'hidden\_act = 'relu'global\_padding = Nonenum\_groups = 32drop\_path\_rate = 0.0embedding\_dynamic\_padding = Falseoutput\_stride = 32width\_factor = 1out\_features = Noneout\_indices = None\*\*kwargs )

Parameters

* **num\_channels** (`int`, *optional*, defaults to 3) — The number of input channels.
* **embedding\_size** (`int`, *optional*, defaults to 64) — Dimensionality (hidden size) for the embedding layer.
* **hidden\_sizes** (`List[int]`, *optional*, defaults to `[256, 512, 1024, 2048]`) — Dimensionality (hidden size) at each stage.
* **depths** (`List[int]`, *optional*, defaults to `[3, 4, 6, 3]`) — Depth (number of layers) for each stage.
* **layer\_type** (`str`, *optional*, defaults to `"preactivation"`) — The layer to use, it can be either `"preactivation"` or `"bottleneck"`.
* **hidden\_act** (`str`, *optional*, defaults to `"relu"`) — The non-linear activation function in each block. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` are supported.
* **global\_padding** (`str`, *optional*) — Padding strategy to use for the convolutional layers. Can be either `"valid"`, `"same"`, or `None`.
* **num\_groups** (`int`, *optional*, defaults to `32`) — Number of groups used for the `BitGroupNormActivation` layers.
* **drop\_path\_rate** (`float`, *optional*, defaults to 0.0) — The drop path rate for the stochastic depth.
* **embedding\_dynamic\_padding** (`bool`, *optional*, defaults to `False`) — Whether or not to make use of dynamic padding for the embedding layer.
* **output\_stride** (`int`, *optional*, defaults to 32) — The output stride of the model.
* **width\_factor** (`int`, *optional*, defaults to 1) — The width factor for the model.
* **out\_features** (`List[str]`, *optional*) — If used as backbone, list of features to output. Can be any of `"stem"`, `"stage1"`, `"stage2"`, etc. (depending on how many stages the model has). If unset and `out_indices` is set, will default to the corresponding stages. If unset and `out_indices` is unset, will default to the last stage.
* **out\_indices** (`List[int]`, *optional*) — If used as backbone, list of indices of features to output. Can be any of 0, 1, 2, etc. (depending on how many stages the model has). If unset and `out_features` is set, will default to the corresponding stages. If unset and `out_features` is unset, will default to the last stage.

This is the configuration class to store the configuration of a [BitModel](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bit#transformers.BitModel). It is used to instantiate an BiT model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the BiT [google/bit-50](https://huggingface.co/google/bit-50) architecture.

Configuration objects inherit from [PretrainedConfig](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/configuration#transformers.PretrainedConfig) and can be used to control the model outputs. Read the documentation from [PretrainedConfig](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/configuration#transformers.PretrainedConfig) for more information.

Example:

Copied

```
>>> from transformers import BitConfig, BitModel

>>> # Initializing a BiT bit-50 style configuration
>>> configuration = BitConfig()

>>> # Initializing a model (with random weights) from the bit-50 style configuration
>>> model = BitModel(configuration)

>>> # Accessing the model configuration
>>> configuration = model.config
```

### BitImageProcessor

#### class transformers.BitImageProcessor

[\<source>](https://github.com/huggingface/transformers/blob/v4.34.1/src/transformers/models/bit/image_processing_bit.py#L50)

( do\_resize: bool = Truesize: typing.Dict\[str, int] = Noneresample: Resampling = \<Resampling.BICUBIC: 3>do\_center\_crop: bool = Truecrop\_size: typing.Dict\[str, int] = Nonedo\_rescale: bool = Truerescale\_factor: typing.Union\[int, float] = 0.00392156862745098do\_normalize: bool = Trueimage\_mean: typing.Union\[float, typing.List\[float], NoneType] = Noneimage\_std: typing.Union\[float, typing.List\[float], NoneType] = Nonedo\_convert\_rgb: bool = True\*\*kwargs )

Parameters

* **do\_resize** (`bool`, *optional*, defaults to `True`) — Whether to resize the image’s (height, width) dimensions to the specified `size`. Can be overridden by `do_resize` in the `preprocess` method.
* **size** (`Dict[str, int]` *optional*, defaults to `{"shortest_edge" -- 224}`): Size of the image after resizing. The shortest edge of the image is resized to size\[“shortest\_edge”], with the longest edge resized to keep the input aspect ratio. Can be overridden by `size` in the `preprocess` method.
* **resample** (`PILImageResampling`, *optional*, defaults to `PILImageResampling.BICUBIC`) — Resampling filter to use if resizing the image. Can be overridden by `resample` in the `preprocess` method.
* **do\_center\_crop** (`bool`, *optional*, defaults to `True`) — Whether to center crop the image to the specified `crop_size`. Can be overridden by `do_center_crop` in the `preprocess` method.
* **crop\_size** (`Dict[str, int]` *optional*, defaults to 224) — Size of the output image after applying `center_crop`. Can be overridden by `crop_size` in the `preprocess` method.
* **do\_rescale** (`bool`, *optional*, defaults to `True`) — Whether to rescale the image by the specified scale `rescale_factor`. Can be overridden by `do_rescale` in the `preprocess` method.
* **rescale\_factor** (`int` or `float`, *optional*, defaults to `1/255`) — Scale factor to use if rescaling the image. Can be overridden by `rescale_factor` in the `preprocess` method. do\_normalize — Whether to normalize the image. Can be overridden by `do_normalize` in the `preprocess` method.
* **image\_mean** (`float` or `List[float]`, *optional*, defaults to `OPENAI_CLIP_MEAN`) — Mean to use if normalizing the image. This is a float or list of floats the length of the number of channels in the image. Can be overridden by the `image_mean` parameter in the `preprocess` method.
* **image\_std** (`float` or `List[float]`, *optional*, defaults to `OPENAI_CLIP_MEAN`) — Standard deviation to use if normalizing the image. This is a float or list of floats the length of the number of channels in the image. Can be overridden by the `image_std` parameter in the `preprocess` method. Can be overridden by the `image_std` parameter in the `preprocess` method.
* **do\_convert\_rgb** (`bool`, *optional*, defaults to `True`) — Whether to convert the image to RGB.

Constructs a BiT image processor.

**preprocess**

[\<source>](https://github.com/huggingface/transformers/blob/v4.34.1/src/transformers/models/bit/image_processing_bit.py#L165)

( images: typing.Union\[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), typing.List\[ForwardRef('PIL.Image.Image')], typing.List\[numpy.ndarray], typing.List\[ForwardRef('torch.Tensor')]]do\_resize: bool = Nonesize: typing.Dict\[str, int] = Noneresample: Resampling = Nonedo\_center\_crop: bool = Nonecrop\_size: int = Nonedo\_rescale: bool = Nonerescale\_factor: float = Nonedo\_normalize: bool = Noneimage\_mean: typing.Union\[float, typing.List\[float], NoneType] = Noneimage\_std: typing.Union\[float, typing.List\[float], NoneType] = Nonedo\_convert\_rgb: bool = Nonereturn\_tensors: typing.Union\[str, transformers.utils.generic.TensorType, NoneType] = Nonedata\_format: typing.Optional\[transformers.image\_utils.ChannelDimension] = \<ChannelDimension.FIRST: 'channels\_first'>input\_data\_format: typing.Union\[str, transformers.image\_utils.ChannelDimension, NoneType] = None\*\*kwargs )

Parameters

* **images** (`ImageInput`) — Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If passing in images with pixel values between 0 and 1, set `do_rescale=False`.
* **do\_resize** (`bool`, *optional*, defaults to `self.do_resize`) — Whether to resize the image.
* **size** (`Dict[str, int]`, *optional*, defaults to `self.size`) — Size of the image after resizing. Shortest edge of the image is resized to size\[“shortest\_edge”], with the longest edge resized to keep the input aspect ratio.
* **resample** (`int`, *optional*, defaults to `self.resample`) — Resampling filter to use if resizing the image. This can be one of the enum `PILImageResampling`. Only has an effect if `do_resize` is set to `True`.
* **do\_center\_crop** (`bool`, *optional*, defaults to `self.do_center_crop`) — Whether to center crop the image.
* **crop\_size** (`Dict[str, int]`, *optional*, defaults to `self.crop_size`) — Size of the center crop. Only has an effect if `do_center_crop` is set to `True`.
* **do\_rescale** (`bool`, *optional*, defaults to `self.do_rescale`) — Whether to rescale the image.
* **rescale\_factor** (`float`, *optional*, defaults to `self.rescale_factor`) — Rescale factor to rescale the image by if `do_rescale` is set to `True`.
* **do\_normalize** (`bool`, *optional*, defaults to `self.do_normalize`) — Whether to normalize the image.
* **image\_mean** (`float` or `List[float]`, *optional*, defaults to `self.image_mean`) — Image mean to use for normalization. Only has an effect if `do_normalize` is set to `True`.
* **image\_std** (`float` or `List[float]`, *optional*, defaults to `self.image_std`) — Image standard deviation to use for normalization. Only has an effect if `do_normalize` is set to `True`.
* **do\_convert\_rgb** (`bool`, *optional*, defaults to `self.do_convert_rgb`) — Whether to convert the image to RGB.
* **return\_tensors** (`str` or `TensorType`, *optional*) — The type of tensors to return. Can be one of:
  * Unset: Return a list of `np.ndarray`.
  * `TensorType.TENSORFLOW` or `'tf'`: Return a batch of type `tf.Tensor`.
  * `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`.
  * `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`.
  * `TensorType.JAX` or `'jax'`: Return a batch of type `jax.numpy.ndarray`.
* **data\_format** (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`) — The channel dimension format for the output image. Can be one of:
  * `"channels_first"` or `ChannelDimension.FIRST`: image in (num\_channels, height, width) format.
  * `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num\_channels) format.
  * Unset: Use the channel dimension format of the input image.
* **input\_data\_format** (`ChannelDimension` or `str`, *optional*) — The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of:
  * `"channels_first"` or `ChannelDimension.FIRST`: image in (num\_channels, height, width) format.
  * `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num\_channels) format.
  * `"none"` or `ChannelDimension.NONE`: image in (height, width) format.

Preprocess an image or batch of images.

### BitModel

#### class transformers.BitModel

[\<source>](https://github.com/huggingface/transformers/blob/v4.34.1/src/transformers/models/bit/modeling_bit.py#L706)

( config )

Parameters

* **config** ([BitConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bit#transformers.BitConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

The bare BiT model outputting raw features without any specific head on top. This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

**forward**

[\<source>](https://github.com/huggingface/transformers/blob/v4.34.1/src/transformers/models/bit/modeling_bit.py#L724)

( pixel\_values: Tensoroutput\_hidden\_states: typing.Optional\[bool] = Nonereturn\_dict: typing.Optional\[bool] = None ) → `transformers.modeling_outputs.BaseModelOutputWithPoolingAndNoAttention` or `tuple(torch.FloatTensor)`

Parameters

* **pixel\_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`) — Pixel values. Pixel values can be obtained using [AutoImageProcessor](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/auto#transformers.AutoImageProcessor). See [BitImageProcessor.**call**()](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/deit#transformers.DeiTFeatureExtractor.__call__) for details.
* **output\_hidden\_states** (`bool`, *optional*) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
* **return\_dict** (`bool`, *optional*) — Whether or not to return a [ModelOutput](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/output#transformers.utils.ModelOutput) instead of a plain tuple.

Returns

`transformers.modeling_outputs.BaseModelOutputWithPoolingAndNoAttention` or `tuple(torch.FloatTensor)`

A `transformers.modeling_outputs.BaseModelOutputWithPoolingAndNoAttention` or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration ([BitConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bit#transformers.BitConfig)) and inputs.

* **last\_hidden\_state** (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`) — Sequence of hidden-states at the output of the last layer of the model.
* **pooler\_output** (`torch.FloatTensor` of shape `(batch_size, hidden_size)`) — Last layer hidden-state after a pooling operation on the spatial dimensions.
* **hidden\_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, num_channels, height, width)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

The [BitModel](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bit#transformers.BitModel) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

Copied

```
>>> from transformers import AutoImageProcessor, BitModel
>>> import torch
>>> from datasets import load_dataset

>>> dataset = load_dataset("boincai/cats-image")
>>> image = dataset["test"]["image"][0]

>>> image_processor = AutoImageProcessor.from_pretrained("google/bit-50")
>>> model = BitModel.from_pretrained("google/bit-50")

>>> inputs = image_processor(image, return_tensors="pt")

>>> with torch.no_grad():
...     outputs = model(**inputs)

>>> last_hidden_states = outputs.last_hidden_state
>>> list(last_hidden_states.shape)
[1, 2048, 7, 7]
```

### BitForImageClassification

#### class transformers.BitForImageClassification

[\<source>](https://github.com/huggingface/transformers/blob/v4.34.1/src/transformers/models/bit/modeling_bit.py#L769)

( config )

Parameters

* **config** ([BitConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bit#transformers.BitConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

BiT Model with an image classification head on top (a linear layer on top of the pooled features), e.g. for ImageNet.

This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

**forward**

[\<source>](https://github.com/huggingface/transformers/blob/v4.34.1/src/transformers/models/bit/modeling_bit.py#L782)

( pixel\_values: typing.Optional\[torch.FloatTensor] = Nonelabels: typing.Optional\[torch.LongTensor] = Noneoutput\_hidden\_states: typing.Optional\[bool] = Nonereturn\_dict: typing.Optional\[bool] = None ) → [transformers.modeling\_outputs.ImageClassifierOutputWithNoAttention](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/output#transformers.modeling_outputs.ImageClassifierOutputWithNoAttention) or `tuple(torch.FloatTensor)`

Parameters

* **pixel\_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`) — Pixel values. Pixel values can be obtained using [AutoImageProcessor](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/auto#transformers.AutoImageProcessor). See [BitImageProcessor.**call**()](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/deit#transformers.DeiTFeatureExtractor.__call__) for details.
* **output\_hidden\_states** (`bool`, *optional*) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
* **return\_dict** (`bool`, *optional*) — Whether or not to return a [ModelOutput](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/output#transformers.utils.ModelOutput) instead of a plain tuple.
* **labels** (`torch.LongTensor` of shape `(batch_size,)`, *optional*) — Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy).

Returns

[transformers.modeling\_outputs.ImageClassifierOutputWithNoAttention](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/output#transformers.modeling_outputs.ImageClassifierOutputWithNoAttention) or `tuple(torch.FloatTensor)`

A [transformers.modeling\_outputs.ImageClassifierOutputWithNoAttention](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/output#transformers.modeling_outputs.ImageClassifierOutputWithNoAttention) or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration ([BitConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bit#transformers.BitConfig)) and inputs.

* **loss** (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided) — Classification (or regression if config.num\_labels==1) loss.
* **logits** (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`) — Classification (or regression if config.num\_labels==1) scores (before SoftMax).
* **hidden\_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each stage) of shape `(batch_size, num_channels, height, width)`. Hidden-states (also called feature maps) of the model at the output of each stage.

The [BitForImageClassification](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bit#transformers.BitForImageClassification) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

Copied

```
>>> from transformers import AutoImageProcessor, BitForImageClassification
>>> import torch
>>> from datasets import load_dataset

>>> dataset = load_dataset("boincai/cats-image")
>>> image = dataset["test"]["image"][0]

>>> image_processor = AutoImageProcessor.from_pretrained("google/bit-50")
>>> model = BitForImageClassification.from_pretrained("google/bit-50")

>>> inputs = image_processor(image, return_tensors="pt")

>>> with torch.no_grad():
...     logits = model(**inputs).logits

>>> # model predicts one of the 1000 ImageNet classes
>>> predicted_label = logits.argmax(-1).item()
>>> print(model.config.id2label[predicted_label])
tiger cat
```