Fine-tune an embedding model for domain-specific semantic search with hard negative mining, evaluation, and deployment.
You are a search and retrieval engineer. Fine-tune an embedding model for domain-specific search. ## Configuration - **Base Model**: [BASE e.g. BAAI/bge-large-en-v1.5 / intfloat/e5-large-v2 / all-MiniLM-L6-v2] - **Domain**: [DOMAIN e.g. legal / medical / e-commerce / technical documentation] - **Training Data**: [DATA e.g. query-document pairs / click logs / relevance judgments] - **Evaluation Set**: [EVAL e.g. annotated query-document pairs with relevance scores] - **Hardware**: [HARDWARE e.g. single A100 / 4x V100] ## Pipeline ### 1. Training Data Preparation (`data/`) - Load and clean [DATA] - Generate training triplets: (query, positive, negative) - **Hard negative mining**: Use base model to find similar but irrelevant documents - **In-batch negatives**: Leverage batch composition for negative sampling - **LLM-generated pairs**: Use LLM to create synthetic query-document pairs from corpus - Data deduplication and quality filtering - Split into train/validation sets ### 2. Training Script (`train.py`) - Load [BASE] with sentence-transformers - Loss function selection: - MultipleNegativesRankingLoss for pairs - TripletLoss with margin for triplets - Contrastive loss for binary pairs - InfoNCE loss with temperature scaling - Training configuration: - Learning rate: [LR e.g. 2e-5] - Batch size: [BATCH e.g. 32] (larger is better for in-batch negatives) - Epochs: [EPOCHS e.g. 3-5] - Warmup: 10% of training steps - FP16/BF16 mixed precision - Gradient caching for large batch simulation - Matryoshka representation learning for flexible dimensions ### 3. Evaluation (`evaluate.py`) - Information Retrieval metrics: - NDCG@1, @5, @10 - MRR@10 - Recall@10, @100 - Comparison against base model (before fine-tuning) - Per-query analysis: improved, degraded, unchanged - Embedding space visualization with t-SNE/UMAP - Semantic Textual Similarity (STS) benchmarks ### 4. Deployment (`deploy/`) - Export to ONNX for fast inference - Quantization: INT8 for latency reduction - Batch encoding with optimal padding - API server with FastAPI - A/B testing framework for old vs new embeddings - Re-indexing pipeline for vector store migration Provide complete code, training configs, and evaluation reports.
Free to copy and use. Compatible with Claude 4 Opus, GPT-5, Gemini 2.5 Pro.
Prepare query-document pairs from your domain. Run hard negative mining with the base model. Fine-tune and evaluate against your test set. Re-index your vector store with the new model.
Initial release
Sign in and download this prompt to leave a review.