SDXL-Lightning

SDXL-Lightning is a lightning-fast text-to-image generation model. It can generate high-quality 1024px images in a few steps. For more information, please refer to our research paper: [SDXL-Lightning: Progressive Adversarial Diffusion Distillation](https://arxiv.org/abs/2402.13929). We open-source the model as part of the research. Our models are distilled from [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0). This repository contains checkpoints for 1-step, 2-step, 4-step, and 8-step distilled models. The generation quality of our 2-step, 4-step, and 8-step model is amazing. Our 1-step model is more experimental. We provide both full UNet and LoRA checkpoints. The full UNet models have the best quality while the LoRA models can be applied to other base models. * Generate with all configurations, best quality: [Demo](https://huggingface.co/spaces/ByteDance/SDXL-Lightning) * `sdxl_lightning_Nstep.safetensors`: All-in-one checkpoint, for ComfyUI. * `sdxl_lightning_Nstep_unet.safetensors`: UNet checkpoint only, for Diffusers. * `sdxl_lightning_Nstep_lora.safetensors`: LoRA checkpoint, for Diffusers and ComfyUI. Please always use the correct checkpoint for the corresponding inference steps. import torch from diffusers import StableDiffusionXLPipeline, UNet2DConditionModel, EulerDiscreteScheduler from huggingface_hub import hf_hub_download from safetensors.torch import load_file base = "stabilityai/stable-diffusion-xl-base-1.0" repo = "ByteDance/SDXL-Lightning" ckpt = "sdxl_lightning_4step_unet.safetensors" # Use the correct ckpt for your step setting! unet = UNet2DConditionModel.from_config(base, subfolder="unet").to("cuda", torch.float16) unet.load_state_dict(load_file(hf_hub_download(repo, ckpt), device="cuda")) pipe = StableDiffusionXLPipeline.from_pretrained(base, unet=unet, torch_dtype=torch.float16, variant="fp16").to("cuda") pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config, timestep_spacing="trailing") pipe("A girl smiling", num_inference_steps=4, guidance_scale=0).images[0].save("output.png") Use LoRA only if you are using non-SDXL base models. Otherwise use our UNet checkpoint for better quality. import torch from diffusers import StableDiffusionXLPipeline, EulerDiscreteScheduler from huggingface_hub import hf_hub_download base = "stabilityai/stable-diffusion-xl-base-1.0" repo = "ByteDance/SDXL-Lightning" ckpt = "sdxl_lightning_4step_lora.safetensors" # Use the correct ckpt for your step setting! pipe = StableDiffusionXLPipeline.from_pretrained(base, torch_dtype=torch.float16, variant="fp16").to("cuda") pipe.load_lora_weights(hf_hub_download(repo, ckpt)) pipe.fuse_lora() pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config, timestep_spacing="trailing") pipe("A girl smiling", num_inference_steps=4, guidance_scale=0).images[0].save("output.png") The 1-step model is only experimental and the quality is much less stable. Consider using the 2-step model for much better quality. The 1-step model uses "sample" prediction instead of "epsilon" prediction! The scheduler needs to be configured correctly. import torch from diffusers import StableDiffusionXLPipeline, UNet2DConditionModel, EulerDiscreteScheduler from huggingface_hub import hf_hub_download from safetensors.torch import load_file base = "stabilityai/stable-diffusion-xl-base-1.0" repo = "ByteDance/SDXL-Lightning" ckpt = "sdxl_lightning_1step_unet_x0.safetensors" # Use the correct ckpt for your step setting! unet = UNet2DConditionModel.from_config(base, subfolder="unet").to("cuda", torch.float16) unet.load_state_dict(load_file(hf_hub_download(repo, ckpt), device="cuda")) pipe = StableDiffusionXLPipeline.from_pretrained(base, unet=unet, torch_dtype=torch.float16, variant="fp16").to("cuda") pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config, timestep_spacing="trailing", prediction_type="sample") pipe("A girl smiling", num_inference_steps=1, guidance_scale=0).images[0].save("output.png") Please always use the correct checkpoint for the corresponding inference steps. Please use Euler sampler with sgm_uniform scheduler. 1. Download the full checkpoint (`sdxl_lightning_Nstep.safetensors`) to `/ComfyUI/models/checkpoints`. 1. Download our [ComfyUI full workflow](comfyui/sdxl_lightning_workflow_full.json). Use LoRA only if you are using non-SDXL base models. Otherwise use our full checkpoint for better quality. 1. Prepare your own base model. 1. Download the LoRA checkpoint (`sdxl_lightning_Nstep_lora.safetensors`) to `/ComfyUI/models/loras` 1. Download our [ComfyUI LoRA workflow](comfyui/sdxl_lightning_workflow_lora.json). The 1-step model is only experimental and the quality is much less stable. Consider using the 2-step model for much better quality. 1. Update your ComfyUI to the latest version. 1. Download the full checkpoint (`sdxl_lightning_1step_x0.safetensors`) to `/ComfyUI/models/checkpoints`. 1. Download our [ComfyUI full 1-step workflow](comfyui/sdxl_lightning_workflow_full_1step.json). @misc{lin2024sdxllightning, title={SDXL-Lightning: Progressive Adversarial Diffusion Distillation}, author={Shanchuan Lin and Anran Wang and Xiao Yang}, year={2024}, eprint={2402.13929}, archivePrefix={arXiv}, primaryClass={cs.CV} }