# VQ Diffusion

## VQ Diffusion

[Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://huggingface.co/papers/2111.14822) is by Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, Baining Guo.

The abstract from the paper is:

_We present the vector quantized diffusion (VQ-Diffusion) model for text-to-image generation. This method is based on a vector quantized variational autoencoder (VQ-VAE) whose latent space is modeled by a conditional variant of the recently developed Denoising Diffusion Probabilistic Model (DDPM). We find that this latent-space method is well-suited for text-to-image generation tasks because it not only eliminates the unidirectional bias with existing methods but also allows us to incorporate a mask-and-replace diffusion strategy to avoid the accumulation of errors, which is a serious problem with existing methods. Our experiments show that the VQ-Diffusion produces significantly better text-to-image generation results when compared with conventional autoregressive (AR) models with similar numbers of parameters. Compared with previous GAN-based text-to-image methods, our VQ-Diffusion can handle more complex scenes and improve the synthesized image quality by a large margin. Finally, we show that the image generation computation in our method can be made highly efficient by reparameterization. With traditional AR methods, the text-to-image generation time increases linearly with the output image resolution and hence is quite time consuming even for normal size images. The VQ-Diffusion allows us to achieve a better trade-off between quality and speed. Our experiments indicate that the VQ-Diffusion model with the reparameterization is fifteen times faster than traditional AR methods while achieving a better image quality._

The original codebase can be found at [microsoft/VQ-Diffusion](https://github.com/microsoft/VQ-Diffusion).

Make sure to check out the Schedulers [guide](https://huggingface.co/using-diffusers/schedulers) to learn how to explore the tradeoff between scheduler speed and quality, and see the [reuse components across pipelines](https://huggingface.co/using-diffusers/loading#reuse-components-across-pipelines) section to learn how to efficiently load the same components into multiple pipelines.

### VQDiffusionPipeline

#### class diffusers.VQDiffusionPipeline

[\<source>](https://github.com/huggingface/diffusers/blob/v0.21.0/src/diffusers/pipelines/vq_diffusion/pipeline_vq_diffusion.py#L52)

( vqvae: VQModeltext\_encoder: CLIPTextModeltokenizer: CLIPTokenizertransformer: Transformer2DModelscheduler: VQDiffusionSchedulerlearned\_classifier\_free\_sampling\_embeddings: LearnedClassifierFreeSamplingEmbeddings )

Parameters

* **vqvae** ([VQModel](https://huggingface.co/docs/diffusers/v0.21.0/en/api/models/vq#diffusers.VQModel)) — Vector Quantized Variational Auto-Encoder (VAE) model to encode and decode images to and from latent representations.
* **text\_encoder** (`CLIPTextModel`) — Frozen text-encoder ([clip-vit-base-patch32](https://huggingface.co/openai/clip-vit-base-patch32)).
* **tokenizer** (`CLIPTokenizer`) — A `CLIPTokenizer` to tokenize text.
* **transformer** ([Transformer2DModel](https://huggingface.co/docs/diffusers/v0.21.0/en/api/models/transformer2d#diffusers.Transformer2DModel)) — A conditional `Transformer2DModel` to denoise the encoded image latents.
* **scheduler** ([VQDiffusionScheduler](https://huggingface.co/docs/diffusers/v0.21.0/en/api/schedulers/vq_diffusion#diffusers.VQDiffusionScheduler)) — A scheduler to be used in combination with `transformer` to denoise the encoded image latents.

Pipeline for text-to-image generation using VQ Diffusion.

This model inherits from [DiffusionPipeline](https://huggingface.co/docs/diffusers/v0.21.0/en/api/pipelines/overview#diffusers.DiffusionPipeline). Check the superclass documentation for the generic methods implemented for all pipelines (downloading, saving, running on a particular device, etc.).

**\_\_call\_\_**

[\<source>](https://github.com/huggingface/diffusers/blob/v0.21.0/src/diffusers/pipelines/vq_diffusion/pipeline_vq_diffusion.py#L163)

( prompt: typing.Union\[str, typing.List\[str]]num\_inference\_steps: int = 100guidance\_scale: float = 5.0truncation\_rate: float = 1.0num\_images\_per\_prompt: int = 1generator: typing.Union\[torch.\_C.Generator, typing.List\[torch.\_C.Generator], NoneType] = Nonelatents: typing.Optional\[torch.FloatTensor] = Noneoutput\_type: typing.Optional\[str] = 'pil'return\_dict: bool = Truecallback: typing.Union\[typing.Callable\[\[int, int, torch.FloatTensor], NoneType], NoneType] = Nonecallback\_steps: int = 1 ) → [ImagePipelineOutput](https://huggingface.co/docs/diffusers/v0.21.0/en/api/pipelines/latent_diffusion_uncond#diffusers.ImagePipelineOutput) or `tuple`

Parameters

* **prompt** (`str` or `List[str]`) — The prompt or prompts to guide image generation.
* **num\_inference\_steps** (`int`, _optional_, defaults to 100) — The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference.
* **guidance\_scale** (`float`, _optional_, defaults to 7.5) — A higher guidance scale value encourages the model to generate images closely linked to the text `prompt` at the expense of lower image quality. Guidance scale is enabled when `guidance_scale > 1`.
* **truncation\_rate** (`float`, _optional_, defaults to 1.0 (equivalent to no truncation)) — Used to “truncate” the predicted classes for x\_0 such that the cumulative probability for a pixel is at most `truncation_rate`. The lowest probabilities that would increase the cumulative probability above `truncation_rate` are set to zero.
* **num\_images\_per\_prompt** (`int`, _optional_, defaults to 1) — The number of images to generate per prompt.
* **generator** (`torch.Generator`, _optional_) — A [`torch.Generator`](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic.
* **latents** (`torch.FloatTensor` of shape (batch), _optional_) — Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for image generation. Must be valid embedding indices.If not provided, a latents tensor will be generated of completely masked latent pixels.
* **output\_type** (`str`, _optional_, defaults to `"pil"`) — The output format of the generated image. Choose between `PIL.Image` or `np.array`.
* **return\_dict** (`bool`, _optional_, defaults to `True`) — Whether or not to return a [ImagePipelineOutput](https://huggingface.co/docs/diffusers/v0.21.0/en/api/pipelines/latent_diffusion_uncond#diffusers.ImagePipelineOutput) instead of a plain tuple.
* **callback** (`Callable`, _optional_) — A function that calls every `callback_steps` steps during inference. The function is called with the following arguments: `callback(step: int, timestep: int, latents: torch.FloatTensor)`.
* **callback\_steps** (`int`, _optional_, defaults to 1) — The frequency at which the `callback` function is called. If not specified, the callback is called at every step.

Returns

[ImagePipelineOutput](https://huggingface.co/docs/diffusers/v0.21.0/en/api/pipelines/latent_diffusion_uncond#diffusers.ImagePipelineOutput) or `tuple`

If `return_dict` is `True`, [ImagePipelineOutput](https://huggingface.co/docs/diffusers/v0.21.0/en/api/pipelines/latent_diffusion_uncond#diffusers.ImagePipelineOutput) is returned, otherwise a `tuple` is returned where the first element is a list with the generated images.

The call function to the pipeline for generation.

**truncate**

[\<source>](https://github.com/huggingface/diffusers/blob/v0.21.0/src/diffusers/pipelines/vq_diffusion/pipeline_vq_diffusion.py#L304)

( log\_p\_x\_0: FloatTensortruncation\_rate: float )

Truncates `log_p_x_0` such that for each column vector, the total cumulative probability is `truncation_rate` The lowest probabilities that would increase the cumulative probability above `truncation_rate` are set to zero.

### ImagePipelineOutput

#### class diffusers.ImagePipelineOutput

[\<source>](https://github.com/huggingface/diffusers/blob/v0.21.0/src/diffusers/pipelines/pipeline_utils.py#L112)

( images: typing.Union\[typing.List\[PIL.Image.Image], numpy.ndarray] )

Parameters

* **images** (`List[PIL.Image.Image]` or `np.ndarray`) — List of denoised PIL images of length `batch_size` or NumPy array of shape `(batch_size, height, width, num_channels)`.

Output class for image pipelines.
