Generate a LoRA fine-tuning pipeline for Stable Diffusion with dataset curation, training, and inference integration.
You are a generative AI engineer. Create a LoRA fine-tuning pipeline for Stable Diffusion. ## Configuration - **Base Model**: [BASE e.g. SDXL 1.0 / SD 3.5 / Flux.1-dev] - **Subject**: [SUBJECT e.g. specific person / art style / product / character] - **Training Images**: [NUM_IMAGES e.g. 15-30 images] - **Hardware**: [HARDWARE e.g. RTX 4090 24GB / A100 40GB] - **LoRA Rank**: [RANK e.g. 32 / 64 / 128] ## Pipeline ### 1. Dataset Preparation (`dataset/`) - Image preprocessing: resize, center crop, face alignment (if applicable) - Automatic captioning using BLIP-2 or CogVLM - Caption editing: add trigger word [TRIGGER e.g. sks_style] - Tag-based captions for Danbooru-style training - Regularization images: generate class-specific images from base model - Dataset validation: check image quality, resolution, diversity ### 2. Training Configuration (`config/`) - LoRA config: rank=[RANK], alpha=[ALPHA e.g. rank*2], target modules - Training params: - Learning rate: [LR e.g. 1e-4] with cosine annealing - Steps: [STEPS e.g. 1500] - Batch size: 1 with gradient accumulation [ACCUM e.g. 4] - Resolution: [RES e.g. 1024x1024] - Mixed precision: fp16/bf16 - Prior preservation loss with weight=[PRIOR_WEIGHT e.g. 1.0] - Text encoder training: frozen / partial / full - Noise scheduler: DDPM with offset noise [OFFSET e.g. 0.1] ### 3. Training Script (`train.py`) - Based on diffusers train_dreambooth_lora.py - Custom callbacks: sample generation every N steps - Checkpoint saving with LoRA weights only - W&B logging with generated sample images - GPU memory optimization: xformers, gradient checkpointing - Support for both Kohya-ss and diffusers training backends ### 4. Inference (`inference/`) - Load LoRA weights with configurable strength [STRENGTH e.g. 0.8] - Multiple LoRA merging and switching - ComfyUI workflow export - A1111 WebUI compatible LoRA format - Batch generation with prompt templates ### 5. Evaluation (`eval/`) - CLIP score for text-image alignment - FID score against reference images - Human evaluation template - Overfitting detection: diversity metrics Provide all scripts, configs, and a troubleshooting guide for common training issues.
Free to copy and use. Compatible with Claude 4 Opus, GPT-5, Gemini 2.5 Pro.
Collect 15-30 high-quality images of your subject. Run dataset preparation with auto-captioning. Adjust LoRA rank and learning rate based on your hardware. Monitor sample generations during training to catch overfitting.
Initial release
Sign in and download this prompt to leave a review.