Wuerstchen

Wurstchen

Würstchen: Efficient Pretraining of Text-to-Image Models is by Pablo Pernias, Dominic Rampas, and Marc Aubreville.

The abstract from the paper is:

We introduce Würstchen, a novel technique for text-to-image synthesis that unites competitive performance with unprecedented cost-effectiveness and ease of training on constrained hardware. Building on recent advancements in machine learning, our approach, which utilizes latent diffusion strategies at strong latent image compression rates, significantly reduces the computational burden, typically associated with state-of-the-art models, while preserving, if not enhancing, the quality of generated images. Wuerstchen achieves notable speed improvements at inference time, thereby rendering real-time applications more viable. One of the key advantages of our method lies in its modest training requirements of only 9,200 GPU hours, slashing the usual costs significantly without compromising the end performance. In a comparison against the state-of-the-art, we found the approach to yield strong competitiveness. This paper opens the door to a new line of research that prioritizes both performance and computational accessibility, hence democratizing the use of sophisticated AI technologies. Through Wuerstchen, we demonstrate a compelling stride forward in the realm of text-to-image synthesis, offering an innovative path to explore in future research.

Würstchen v2 comes to Diffusers

After the initial paper release, we have improved numerous things in the architecture, training and sampling, making Würstchen competetive to current state-of-the-art models in many ways. We are excited to release this new version together with Diffusers. Here is a list of the improvements.

  • Higher resolution (1024x1024 up to 2048x2048)

  • Faster inference

  • Multi Aspect Resolution Sampling

  • Better quality

We are releasing 3 checkpoints for the text-conditional image generation model (Stage C). Those are:

  • v2-base

  • v2-aesthetic

  • v2-interpolated (50% interpolation between v2-base and v2-aesthetic)

We recommend to use v2-interpolated, as it has a nice touch of both photorealism and aesthetic. Use v2-base for finetunings as it does not have a style bias and use v2-aesthetic for very artistic generations. A comparison can be seen here:

Text-to-Image Generation

For the sake of usability Würstchen can be used with a single pipeline. This pipeline is called WuerstchenCombinedPipeline and can be used as follows:

Copied

import torch
from diffusers import AutoPipelineForText2Image
from diffusers.pipelines.wuerstchen import DEFAULT_STAGE_C_TIMESTEPS

pipe = AutoPipelineForText2Image.from_pretrained("warp-ai/wuerstchen", torch_dtype=torch.float16).to("cuda")

caption = "Anthropomorphic cat dressed as a fire fighter"
images = pipe(
    caption, 
    width=1024,
    height=1536,
    prior_timesteps=DEFAULT_STAGE_C_TIMESTEPS,
    prior_guidance_scale=4.0,
    num_images_per_prompt=2,
).images

For explanation purposes, we can also initialize the two main pipelines of Würstchen individually. Würstchen consists of 3 stages: Stage C, Stage B, Stage A. They all have different jobs and work only together. When generating text-conditional images, Stage C will first generate the latents in a very compressed latent space. This is what happens in the prior_pipeline. Afterwards, the generated latents will be passed to Stage B, which decompresses the latents into a bigger latent space of a VQGAN. These latents can then be decoded by Stage A, which is a VQGAN, into the pixel-space. Stage B & Stage A are both encapsulated in the decoder_pipeline. For more details, take a look at the paper.

Copied

import torch
from diffusers import WuerstchenDecoderPipeline, WuerstchenPriorPipeline
from diffusers.pipelines.wuerstchen import DEFAULT_STAGE_C_TIMESTEPS

device = "cuda"
dtype = torch.float16
num_images_per_prompt = 2

prior_pipeline = WuerstchenPriorPipeline.from_pretrained(
    "warp-ai/wuerstchen-prior", torch_dtype=dtype
).to(device)
decoder_pipeline = WuerstchenDecoderPipeline.from_pretrained(
    "warp-ai/wuerstchen", torch_dtype=dtype
).to(device)

caption = "Anthropomorphic cat dressed as a fire fighter"
negative_prompt = ""

prior_output = prior_pipeline(
    prompt=caption,
    height=1024,
    width=1536,
    timesteps=DEFAULT_STAGE_C_TIMESTEPS,
    negative_prompt=negative_prompt,
    guidance_scale=4.0,
    num_images_per_prompt=num_images_per_prompt,
)
decoder_output = decoder_pipeline(
    image_embeddings=prior_output.image_embeddings,
    prompt=caption,
    negative_prompt=negative_prompt,
    num_images_per_prompt=num_images_per_prompt,
    guidance_scale=0.0,
    output_type="pil",
).images

Speed-Up Inference

You can make use of `torch.compile` function and gain a speed-up of about 2-3x:Copied

pipeline.prior = torch.compile(pipeline.prior, mode="reduce-overhead", fullgraph=True)
pipeline.decoder = torch.compile(pipeline.decoder, mode="reduce-overhead", fullgraph=True)

Limitations

  • Due to the high compression employed by Würstchen, generations can lack a good amount of detail. To our human eye, this is especially noticeable in faces, hands etc.

  • Images can only be generated in 128-pixel steps, e.g. the next higher resolution after 1024x1024 is 1152x1152

  • The model lacks the ability to render correct text in images

  • The model often does not achieve photorealism

  • Difficult compositional prompts are hard for the model

The original codebase, as well as experimental ideas, can be found at dome272/Wuerstchen.

WuerschenPipeline

class diffusers.WuerstchenCombinedPipeline

<source>

( tokenizer: CLIPTokenizertext_encoder: CLIPTextModeldecoder: WuerstchenDiffNeXtscheduler: DDPMWuerstchenSchedulervqgan: PaellaVQModelprior_tokenizer: CLIPTokenizerprior_text_encoder: CLIPTextModelprior_prior: WuerstchenPriorprior_scheduler: DDPMWuerstchenScheduler )

Parameters

  • tokenizer (CLIPTokenizer) — The decoder tokenizer to be used for text inputs.

  • text_encoder (CLIPTextModel) — The decoder text encoder to be used for text inputs.

  • decoder (WuerstchenDiffNeXt) — The decoder model to be used for decoder image generation pipeline.

  • scheduler (DDPMWuerstchenScheduler) — The scheduler to be used for decoder image generation pipeline.

  • vqgan (PaellaVQModel) — The VQGAN model to be used for decoder image generation pipeline.

  • prior_tokenizer (CLIPTokenizer) — The prior tokenizer to be used for text inputs.

  • prior_text_encoder (CLIPTextModel) — The prior text encoder to be used for text inputs.

  • prior (WuerstchenPrior) — The prior model to be used for prior pipeline.

  • prior_scheduler (DDPMWuerstchenScheduler) — The scheduler to be used for prior pipeline.

Combined Pipeline for text-to-image generation using Wuerstchen

This model inherits from DiffusionPipeline. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.)

__call__

<source>

( prompt: typing.Union[str, typing.List[str]]height: int = 512width: int = 512prior_num_inference_steps: int = 60prior_timesteps: typing.Optional[typing.List[float]] = Noneprior_guidance_scale: float = 4.0num_inference_steps: int = 12decoder_timesteps: typing.Optional[typing.List[float]] = Nonedecoder_guidance_scale: float = 0.0negative_prompt: typing.Union[str, typing.List[str], NoneType] = Noneprompt_embeds: typing.Optional[torch.FloatTensor] = Nonenegative_prompt_embeds: typing.Optional[torch.FloatTensor] = Nonenum_images_per_prompt: int = 1generator: typing.Union[torch._C.Generator, typing.List[torch._C.Generator], NoneType] = Nonelatents: typing.Optional[torch.FloatTensor] = Noneoutput_type: typing.Optional[str] = 'pil'return_dict: bool = True )

Parameters

  • prompt (str or List[str]) — The prompt or prompts to guide the image generation for the prior and decoder.

  • negative_prompt (str or List[str], optional) — The prompt or prompts not to guide the image generation. Ignored when not using guidance (i.e., ignored if guidance_scale is less than 1).

  • prompt_embeds (torch.FloatTensor, optional) — Pre-generated text embeddings for the prior. Can be used to easily tweak text inputs, e.g. prompt weighting. If not provided, text embeddings will be generated from prompt input argument.

  • negative_prompt_embeds (torch.FloatTensor, optional) — Pre-generated negative text embeddings for the prior. Can be used to easily tweak text inputs, e.g. prompt weighting. If not provided, negative_prompt_embeds will be generated from negative_prompt input argument.

  • num_images_per_prompt (int, optional, defaults to 1) — The number of images to generate per prompt.

  • height (int, optional, defaults to 512) — The height in pixels of the generated image.

  • width (int, optional, defaults to 512) — The width in pixels of the generated image.

  • prior_guidance_scale (float, optional, defaults to 4.0) — Guidance scale as defined in Classifier-Free Diffusion Guidance. prior_guidance_scale is defined as w of equation 2. of Imagen Paper. Guidance scale is enabled by setting prior_guidance_scale > 1. Higher guidance scale encourages to generate images that are closely linked to the text prompt, usually at the expense of lower image quality.

  • prior_num_inference_steps (Union[int, Dict[float, int]], optional, defaults to 30) — The number of prior denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. For more specific timestep spacing, you can pass customized prior_timesteps

  • num_inference_steps (int, optional, defaults to 12) — The number of decoder denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. For more specific timestep spacing, you can pass customized timesteps

  • prior_timesteps (List[float], optional) — Custom timesteps to use for the denoising process for the prior. If not defined, equal spaced prior_num_inference_steps timesteps are used. Must be in descending order.

  • decoder_timesteps (List[float], optional) — Custom timesteps to use for the denoising process for the decoder. If not defined, equal spaced num_inference_steps timesteps are used. Must be in descending order.

  • decoder_guidance_scale (float, optional, defaults to 0.0) — Guidance scale as defined in Classifier-Free Diffusion Guidance. guidance_scale is defined as w of equation 2. of Imagen Paper. Guidance scale is enabled by setting guidance_scale > 1. Higher guidance scale encourages to generate images that are closely linked to the text prompt, usually at the expense of lower image quality.

  • generator (torch.Generator or List[torch.Generator], optional) — One or a list of torch generator(s) to make generation deterministic.

  • latents (torch.FloatTensor, optional) — Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor will ge generated by sampling using the supplied random generator.

  • output_type (str, optional, defaults to "pil") — The output format of the generate image. Choose between: "pil" (PIL.Image.Image), "np" (np.array) or "pt" (torch.Tensor).

  • return_dict (bool, optional, defaults to True) — Whether or not to return a ImagePipelineOutput instead of a plain tuple.

Function invoked when calling the pipeline for generation.

Examples:

Copied

>>> from diffusions import WuerstchenCombinedPipeline

>>> pipe = WuerstchenCombinedPipeline.from_pretrained("warp-ai/Wuerstchen", torch_dtype=torch.float16).to(
...     "cuda"
... )
>>> prompt = "an image of a shiba inu, donning a spacesuit and helmet"
>>> images = pipe(prompt=prompt)

enable_sequential_cpu_offload

<source>

( gpu_id = 0 )

Offloads all models (unet, text_encoder, vae, and safety checker state dicts) to CPU using 🌍 Accelerate, significantly reducing memory usage. Models are moved to a torch.device('meta') and loaded on a GPU only when their specific submodule’s forward method is called. Offloading happens on a submodule basis. Memory savings are higher than using enable_model_cpu_offload, but performance is lower.

WuerstchenPriorPipeline

class diffusers.WuerstchenDecoderPipeline

<source>

( tokenizer: CLIPTokenizertext_encoder: CLIPTextModeldecoder: WuerstchenDiffNeXtscheduler: DDPMWuerstchenSchedulervqgan: PaellaVQModellatent_dim_scale: float = 10.67 )

Parameters

  • tokenizer (CLIPTokenizer) — The CLIP tokenizer.

  • text_encoder (CLIPTextModel) — The CLIP text encoder.

  • decoder (WuerstchenDiffNeXt) — The WuerstchenDiffNeXt unet decoder.

  • vqgan (PaellaVQModel) — The VQGAN model.

  • scheduler (DDPMWuerstchenScheduler) — A scheduler to be used in combination with prior to generate image embedding.

  • latent_dim_scale (float, optional, defaults to 10.67) — Multiplier to determine the VQ latent space size from the image embeddings. If the image embeddings are height=24 and width=24, the VQ latent shape needs to be height=int(2410.67)=256 and width=int(2410.67)=256 in order to match the training conditions.

Pipeline for generating images from the Wuerstchen model.

This model inherits from DiffusionPipeline. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.)

__call__

<source>

( image_embeddings: typing.Union[torch.FloatTensor, typing.List[torch.FloatTensor]]prompt: typing.Union[str, typing.List[str]] = Nonenum_inference_steps: int = 12timesteps: typing.Optional[typing.List[float]] = Noneguidance_scale: float = 0.0negative_prompt: typing.Union[str, typing.List[str], NoneType] = Nonenum_images_per_prompt: int = 1generator: typing.Union[torch._C.Generator, typing.List[torch._C.Generator], NoneType] = Nonelatents: typing.Optional[torch.FloatTensor] = Noneoutput_type: typing.Optional[str] = 'pil'return_dict: bool = True )

Parameters

  • image_embedding (torch.FloatTensor or List[torch.FloatTensor]) — Image Embeddings either extracted from an image or generated by a Prior Model.

  • prompt (str or List[str]) — The prompt or prompts to guide the image generation.

  • num_inference_steps (int, optional, defaults to 30) — The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference.

  • timesteps (List[int], optional) — Custom timesteps to use for the denoising process. If not defined, equal spaced num_inference_steps timesteps are used. Must be in descending order.

  • guidance_scale (float, optional, defaults to 4.0) — Guidance scale as defined in Classifier-Free Diffusion Guidance. decoder_guidance_scale is defined as w of equation 2. of Imagen Paper. Guidance scale is enabled by setting decoder_guidance_scale > 1. Higher guidance scale encourages to generate images that are closely linked to the text prompt, usually at the expense of lower image quality.

  • negative_prompt (str or List[str], optional) — The prompt or prompts not to guide the image generation. Ignored when not using guidance (i.e., ignored if decoder_guidance_scale is less than 1).

  • num_images_per_prompt (int, optional, defaults to 1) — The number of images to generate per prompt.

  • generator (torch.Generator or List[torch.Generator], optional) — One or a list of torch generator(s) to make generation deterministic.

  • latents (torch.FloatTensor, optional) — Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor will ge generated by sampling using the supplied random generator.

  • output_type (str, optional, defaults to "pil") — The output format of the generate image. Choose between: "pil" (PIL.Image.Image), "np" (np.array) or "pt" (torch.Tensor).

  • return_dict (bool, optional, defaults to True) — Whether or not to return a ImagePipelineOutput instead of a plain tuple.

Function invoked when calling the pipeline for generation.

Examples:

Copied

>>> import torch
>>> from diffusers import WuerstchenPriorPipeline, WuerstchenDecoderPipeline

>>> prior_pipe = WuerstchenPriorPipeline.from_pretrained(
...     "warp-ai/wuerstchen-prior", torch_dtype=torch.float16
... ).to("cuda")
>>> gen_pipe = WuerstchenDecoderPipeline.from_pretrain("warp-ai/wuerstchen", torch_dtype=torch.float16).to(
...     "cuda"
... )

>>> prompt = "an image of a shiba inu, donning a spacesuit and helmet"
>>> prior_output = pipe(prompt)
>>> images = gen_pipe(prior_output.image_embeddings, prompt=prompt)

WuerstchenPriorPipelineOutput

class diffusers.pipelines.wuerstchen.pipeline_wuerstchen_prior.WuerstchenPriorPipelineOutput

<source>

( image_embeddings: typing.Union[torch.FloatTensor, numpy.ndarray] )

Parameters

  • image_embeddings (torch.FloatTensor or np.ndarray) — Prior image embeddings for text prompt

Output class for WuerstchenPriorPipeline.

WuerstchenDecoderPipeline

class diffusers.WuerstchenDecoderPipeline

<source>

( tokenizer: CLIPTokenizertext_encoder: CLIPTextModeldecoder: WuerstchenDiffNeXtscheduler: DDPMWuerstchenSchedulervqgan: PaellaVQModellatent_dim_scale: float = 10.67 )

Parameters

  • tokenizer (CLIPTokenizer) — The CLIP tokenizer.

  • text_encoder (CLIPTextModel) — The CLIP text encoder.

  • decoder (WuerstchenDiffNeXt) — The WuerstchenDiffNeXt unet decoder.

  • vqgan (PaellaVQModel) — The VQGAN model.

  • scheduler (DDPMWuerstchenScheduler) — A scheduler to be used in combination with prior to generate image embedding.

  • latent_dim_scale (float, optional, defaults to 10.67) — Multiplier to determine the VQ latent space size from the image embeddings. If the image embeddings are height=24 and width=24, the VQ latent shape needs to be height=int(2410.67)=256 and width=int(2410.67)=256 in order to match the training conditions.

Pipeline for generating images from the Wuerstchen model.

This model inherits from DiffusionPipeline. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.)

__call__

<source>

( image_embeddings: typing.Union[torch.FloatTensor, typing.List[torch.FloatTensor]]prompt: typing.Union[str, typing.List[str]] = Nonenum_inference_steps: int = 12timesteps: typing.Optional[typing.List[float]] = Noneguidance_scale: float = 0.0negative_prompt: typing.Union[str, typing.List[str], NoneType] = Nonenum_images_per_prompt: int = 1generator: typing.Union[torch._C.Generator, typing.List[torch._C.Generator], NoneType] = Nonelatents: typing.Optional[torch.FloatTensor] = Noneoutput_type: typing.Optional[str] = 'pil'return_dict: bool = True )

Parameters

  • image_embedding (torch.FloatTensor or List[torch.FloatTensor]) — Image Embeddings either extracted from an image or generated by a Prior Model.

  • prompt (str or List[str]) — The prompt or prompts to guide the image generation.

  • num_inference_steps (int, optional, defaults to 30) — The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference.

  • timesteps (List[int], optional) — Custom timesteps to use for the denoising process. If not defined, equal spaced num_inference_steps timesteps are used. Must be in descending order.

  • guidance_scale (float, optional, defaults to 4.0) — Guidance scale as defined in Classifier-Free Diffusion Guidance. decoder_guidance_scale is defined as w of equation 2. of Imagen Paper. Guidance scale is enabled by setting decoder_guidance_scale > 1. Higher guidance scale encourages to generate images that are closely linked to the text prompt, usually at the expense of lower image quality.

  • negative_prompt (str or List[str], optional) — The prompt or prompts not to guide the image generation. Ignored when not using guidance (i.e., ignored if decoder_guidance_scale is less than 1).

  • num_images_per_prompt (int, optional, defaults to 1) — The number of images to generate per prompt.

  • generator (torch.Generator or List[torch.Generator], optional) — One or a list of torch generator(s) to make generation deterministic.

  • latents (torch.FloatTensor, optional) — Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor will ge generated by sampling using the supplied random generator.

  • output_type (str, optional, defaults to "pil") — The output format of the generate image. Choose between: "pil" (PIL.Image.Image), "np" (np.array) or "pt" (torch.Tensor).

  • return_dict (bool, optional, defaults to True) — Whether or not to return a ImagePipelineOutput instead of a plain tuple.

Function invoked when calling the pipeline for generation.

Examples:

Copied

>>> import torch
>>> from diffusers import WuerstchenPriorPipeline, WuerstchenDecoderPipeline

>>> prior_pipe = WuerstchenPriorPipeline.from_pretrained(
...     "warp-ai/wuerstchen-prior", torch_dtype=torch.float16
... ).to("cuda")
>>> gen_pipe = WuerstchenDecoderPipeline.from_pretrain("warp-ai/wuerstchen", torch_dtype=torch.float16).to(
...     "cuda"
... )

>>> prompt = "an image of a shiba inu, donning a spacesuit and helmet"
>>> prior_output = pipe(prompt)
>>> images = gen_pipe(prior_output.image_embeddings, prompt=prompt)
PreviousVQ DiffusionNextSCHEDULERS

Last updated