Image variation
Last updated
Last updated
The Stable Diffusion model can also generate variations from an input image. It uses a fine-tuned version of a Stable Diffusion model by from .
The original codebase can be found at and additional official checkpoints for image variation can be found at .
Make sure to check out the Stable Diffusion section to learn how to explore the tradeoff between scheduler speed and quality, and how to reuse pipeline components efficiently!
( vae: AutoencoderKLimage_encoder: CLIPVisionModelWithProjectionunet: UNet2DConditionModelscheduler: KarrasDiffusionSchedulerssafety_checker: StableDiffusionSafetyCheckerfeature_extractor: CLIPImageProcessorrequires_safety_checker: bool = True )
Parameters
vae () β Variational Auto-Encoder (VAE) model to encode and decode images to and from latent representations.
image_encoder (CLIPVisionModelWithProjection) β Frozen CLIP image-encoder ().
text_encoder (CLIPTextModel) β Frozen text-encoder ().
tokenizer (CLIPTokenizer) β A CLIPTokenizer to tokenize text.
unet () β A UNet2DConditionModel to denoise the encoded image latents.
scheduler () β A scheduler to be used in combination with unet to denoise the encoded image latents. Can be one of , , or .
safety_checker (StableDiffusionSafetyChecker) β Classification module that estimates whether generated images could be considered offensive or harmful. Please refer to the for more details about a modelβs potential harms.
feature_extractor (CLIPImageProcessor) β A CLIPImageProcessor to extract features from generated images; used as inputs to the safety_checker.
Pipeline to generate image variations from an input image using Stable Diffusion.
__call__
Parameters
height (int, optional, defaults to self.unet.config.sample_size * self.vae_scale_factor) β The height in pixels of the generated image.
width (int, optional, defaults to self.unet.config.sample_size * self.vae_scale_factor) β The width in pixels of the generated image.
num_inference_steps (int, optional, defaults to 50) β The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. This parameter is modulated by strength.
guidance_scale (float, optional, defaults to 7.5) β A higher guidance scale value encourages the model to generate images closely linked to the text prompt at the expense of lower image quality. Guidance scale is enabled when guidance_scale > 1.
num_images_per_prompt (int, optional, defaults to 1) β The number of images to generate per prompt.
latents (torch.FloatTensor, optional) β Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor is generated by sampling using the supplied random generator.
output_type (str, optional, defaults to "pil") β The output format of the generated image. Choose between PIL.Image or np.array.
callback (Callable, optional) β A function that calls every callback_steps steps during inference. The function is called with the following arguments: callback(step: int, timestep: int, latents: torch.FloatTensor).
callback_steps (int, optional, defaults to 1) β The frequency at which the callback function is called. If not specified, the callback is called at every step.
Returns
The call function to the pipeline for generation.
Examples:
Copied
enable_attention_slicing
( slice_size: typing.Union[str, int, NoneType] = 'auto' )
Parameters
slice_size (str or int, optional, defaults to "auto") β When "auto", halves the input to the attention heads, so attention will be computed in two steps. If "max", maximum amount of memory will be saved by running only one slice at a time. If a number is provided, uses as many slices as attention_head_dim // slice_size. In this case, attention_head_dim must be a multiple of slice_size.
Enable sliced attention computation. When this option is enabled, the attention module splits the input tensor in slices to compute attention in several steps. For more than one attention head, the computation is performed sequentially over each head. This is useful to save some memory in exchange for a small speed decrease.
β οΈ Donβt enable attention slicing if youβre already using scaled_dot_product_attention (SDPA) from PyTorch 2.0 or xFormers. These attention computations are already very memory efficient so you wonβt need to enable this function. If you enable attention slicing with SDPA or xFormers, it can lead to serious slow downs!
Examples:
Copied
disable_attention_slicing
( )
Disable sliced attention computation. If enable_attention_slicing was previously called, attention is computed in one step.
enable_xformers_memory_efficient_attention
( attention_op: typing.Optional[typing.Callable] = None )
Parameters
β οΈ When memory efficient attention and sliced attention are both enabled, memory efficient attention takes precedent.
Examples:
Copied
disable_xformers_memory_efficient_attention
( )
( images: typing.Union[typing.List[PIL.Image.Image], numpy.ndarray]nsfw_content_detected: typing.Optional[typing.List[bool]] )
Parameters
images (List[PIL.Image.Image] or np.ndarray) β List of denoised PIL images of length batch_size or NumPy array of shape (batch_size, height, width, num_channels).
nsfw_content_detected (List[bool]) β List indicating whether the corresponding generated image contains βnot-safe-for-workβ (nsfw) content or None if safety checking could not be performed.
Output class for Stable Diffusion pipelines.
This model inherits from . Check the superclass documentation for the generic methods implemented for all pipelines (downloading, saving, running on a particular device, etc.).
( image: typing.Union[PIL.Image.Image, typing.List[PIL.Image.Image], torch.FloatTensor]height: typing.Optional[int] = Nonewidth: typing.Optional[int] = Nonenum_inference_steps: int = 50guidance_scale: float = 7.5num_images_per_prompt: typing.Optional[int] = 1eta: float = 0.0generator: typing.Union[torch._C.Generator, typing.List[torch._C.Generator], NoneType] = Nonelatents: typing.Optional[torch.FloatTensor] = Noneoutput_type: typing.Optional[str] = 'pil'return_dict: bool = Truecallback: typing.Union[typing.Callable[[int, int, torch.FloatTensor], NoneType], NoneType] = Nonecallback_steps: int = 1 ) β or tuple
image (PIL.Image.Image or List[PIL.Image.Image] or torch.FloatTensor) β Image or images to guide image generation. If you provide a tensor, it needs to be compatible with .
eta (float, optional, defaults to 0.0) β Corresponds to parameter eta (Ξ·) from the paper. Only applies to the , and is ignored in other schedulers.
generator (torch.Generator or List[torch.Generator], optional) β A to make generation deterministic.
return_dict (bool, optional, defaults to True) β Whether or not to return a instead of a plain tuple.
or tuple
If return_dict is True, is returned, otherwise a tuple is returned where the first element is a list with the generated images and the second element is a list of bools indicating whether the corresponding generated image contains βnot-safe-for-workβ (nsfw) content.
attention_op (Callable, optional) β Override the default None operator for use as op argument to the function of xFormers.
Enable memory efficient attention from . When this option is enabled, you should observe lower GPU memory usage and a potential speed up during inference. Speed up during training is not guaranteed.
Disable memory efficient attention from .