PaintByExample
Last updated
Last updated
is by Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, Fang Wen.
The abstract from the paper is:
Language-guided image editing has achieved great success recently. In this paper, for the first time, we investigate exemplar-guided image editing for more precise control. We achieve this goal by leveraging self-supervised training to disentangle and re-organize the source image and the exemplar. However, the naive approach will cause obvious fusing artifacts. We carefully analyze it and propose an information bottleneck and strong augmentations to avoid the trivial solution of directly copying and pasting the exemplar image. Meanwhile, to ensure the controllability of the editing process, we design an arbitrary shape mask for the exemplar image and leverage the classifier-free guidance to increase the similarity to the exemplar image. The whole framework involves a single forward of the diffusion model without any iterative optimization. We demonstrate that our method achieves an impressive performance and enables controllable editing on in-the-wild images with high fidelity.
The original codebase can be found at , and you can try it out in a .
PaintByExample is supported by the official checkpoint. The checkpoint is warm-started from to inpaint partly masked images conditioned on example and reference images.
Make sure to check out the Schedulers to learn how to explore the tradeoff between scheduler speed and quality, and see the section to learn how to efficiently load the same components into multiple pipelines.
( vae: AutoencoderKLimage_encoder: PaintByExampleImageEncoderunet: UNet2DConditionModelscheduler: typing.Union[diffusers.schedulers.scheduling_ddim.DDIMScheduler, diffusers.schedulers.scheduling_pndm.PNDMScheduler, diffusers.schedulers.scheduling_lms_discrete.LMSDiscreteScheduler]safety_checker: StableDiffusionSafetyCheckerfeature_extractor: CLIPImageProcessorrequires_safety_checker: bool = False )
Parameters
vae () β Variational Auto-Encoder (VAE) model to encode and decode images to and from latent representations.
image_encoder (PaintByExampleImageEncoder) β Encodes the example input image. The unet is conditioned on the example image instead of a text prompt.
tokenizer (CLIPTokenizer) β A CLIPTokenizer to tokenize text.
unet () β A UNet2DConditionModel to denoise the encoded image latents.
scheduler () β A scheduler to be used in combination with unet to denoise the encoded image latents. Can be one of , , or .
safety_checker (StableDiffusionSafetyChecker) β Classification module that estimates whether generated images could be considered offensive or harmful. Please refer to the for more details about a modelβs potential harms.
feature_extractor (CLIPImageProcessor) β A CLIPImageProcessor to extract features from generated images; used as inputs to the safety_checker.
π§ͺ This is an experimental feature!
Pipeline for image-guided image inpainting using Stable Diffusion.
__call__
Parameters
example_image (torch.FloatTensor or PIL.Image.Image or List[PIL.Image.Image]) β An example image to guide image generation.
image (torch.FloatTensor or PIL.Image.Image or List[PIL.Image.Image]) β Image or tensor representing an image batch to be inpainted (parts of the image are masked out with mask_image and repainted according to prompt).
mask_image (torch.FloatTensor or PIL.Image.Image or List[PIL.Image.Image]) β Image or tensor representing an image batch to mask image. White pixels in the mask are repainted, while black pixels are preserved. If mask_image is a PIL image, it is converted to a single channel (luminance) before use. If itβs a tensor, it should contain one color channel (L) instead of 3, so the expected shape would be (B, H, W, 1).
height (int, optional, defaults to self.unet.config.sample_size * self.vae_scale_factor) β The height in pixels of the generated image.
width (int, optional, defaults to self.unet.config.sample_size * self.vae_scale_factor) β The width in pixels of the generated image.
num_inference_steps (int, optional, defaults to 50) β The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference.
guidance_scale (float, optional, defaults to 7.5) β A higher guidance scale value encourages the model to generate images closely linked to the text prompt at the expense of lower image quality. Guidance scale is enabled when guidance_scale > 1.
negative_prompt (str or List[str], optional) β The prompt or prompts to guide what to not include in image generation. If not defined, you need to pass negative_prompt_embeds instead. Ignored when not using guidance (guidance_scale < 1).
num_images_per_prompt (int, optional, defaults to 1) β The number of images to generate per prompt.
latents (torch.FloatTensor, optional) β Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor is generated by sampling using the supplied random generator.
output_type (str, optional, defaults to "pil") β The output format of the generated image. Choose between PIL.Image or np.array.
callback (Callable, optional) β A function that calls every callback_steps steps during inference. The function is called with the following arguments: callback(step: int, timestep: int, latents: torch.FloatTensor).
callback_steps (int, optional, defaults to 1) β The frequency at which the callback function is called. If not specified, the callback is called at every step.
Returns
The call function to the pipeline for generation.
Example:
Copied
( images: typing.Union[typing.List[PIL.Image.Image], numpy.ndarray]nsfw_content_detected: typing.Optional[typing.List[bool]] )
Parameters
images (List[PIL.Image.Image] or np.ndarray) β List of denoised PIL images of length batch_size or NumPy array of shape (batch_size, height, width, num_channels).
nsfw_content_detected (List[bool]) β List indicating whether the corresponding generated image contains βnot-safe-for-workβ (nsfw) content or None if safety checking could not be performed.
Output class for Stable Diffusion pipelines.
This model inherits from . Check the superclass documentation for the generic methods implemented for all pipelines (downloading, saving, running on a particular device, etc.).
( example_image: typing.Union[torch.FloatTensor, PIL.Image.Image]image: typing.Union[torch.FloatTensor, PIL.Image.Image]mask_image: typing.Union[torch.FloatTensor, PIL.Image.Image]height: typing.Optional[int] = Nonewidth: typing.Optional[int] = Nonenum_inference_steps: int = 50guidance_scale: float = 5.0negative_prompt: typing.Union[str, typing.List[str], NoneType] = Nonenum_images_per_prompt: typing.Optional[int] = 1eta: float = 0.0generator: typing.Union[torch._C.Generator, typing.List[torch._C.Generator], NoneType] = Nonelatents: typing.Optional[torch.FloatTensor] = Noneoutput_type: typing.Optional[str] = 'pil'return_dict: bool = Truecallback: typing.Union[typing.Callable[[int, int, torch.FloatTensor], NoneType], NoneType] = Nonecallback_steps: int = 1 ) β or tuple
eta (float, optional, defaults to 0.0) β Corresponds to parameter eta (Ξ·) from the paper. Only applies to the , and is ignored in other schedulers.
generator (torch.Generator or List[torch.Generator], optional) β A to make generation deterministic.
return_dict (bool, optional, defaults to True) β Whether or not to return a instead of a plain tuple.
or tuple
If return_dict is True, is returned, otherwise a tuple is returned where the first element is a list with the generated images and the second element is a list of bools indicating whether the corresponding generated image contains βnot-safe-for-workβ (nsfw) content.