Build an end-to-end sentiment analysis pipeline using Hugging Face Transformers with multi-language support and aspect-based analysis.
You are an NLP engineer. Create a production sentiment analysis pipeline.
## Requirements
- **Languages**: [LANGUAGES e.g. English, Hindi, Tamil]
- **Granularity**: [GRANULARITY e.g. document-level / sentence-level / aspect-based]
- **Categories**: [CATEGORIES e.g. positive, negative, neutral / 5-star scale]
- **Base Model**: [MODEL e.g. xlm-roberta-large / bert-base-multilingual]
- **Domain**: [DOMAIN e.g. product reviews / social media / financial news]
## Pipeline Architecture
### 1. Data Processing (`processing/`)
- Text cleaning: remove URLs, emails, special chars while preserving emoticons
- Language detection using fasttext lid.176.bin
- Tokenization with [MODEL] tokenizer
- Handle code-mixed text (e.g., Hinglish) with transliteration
- Aspect term extraction using dependency parsing for aspect-based mode
### 2. Model Training (`training/`)
- Fine-tune [MODEL] on [DOMAIN] data
- Implement custom training loop with:
- Label smoothing (epsilon=0.1)
- Focal loss for class imbalance
- Gradient accumulation for effective batch size
- Mixed precision (FP16) training
- Multi-task learning: joint sentiment + aspect prediction
- K-fold cross-validation with stratified splits
### 3. Inference Engine (`inference/`)
- Batch prediction with dynamic padding
- Confidence calibration using temperature scaling
- Ensemble predictions from multiple checkpoints
- Explainability: attention visualization and LIME/SHAP explanations
- Handle long documents with sliding window approach
### 4. API Service (`api/`)
- FastAPI endpoints:
- POST /analyze - Single text analysis
- POST /batch - Bulk analysis with async processing
- GET /explain/{id} - Get explanation for a prediction
- Request validation and rate limiting
- Response caching with Redis
- Prometheus metrics endpoint
### 5. Monitoring (`monitoring/`)
- Track prediction distribution drift
- Alert on confidence score degradation
- Log low-confidence predictions for human review
- A/B testing framework for model versions
Provide complete code with Dockerfile, requirements, and deployment configs.Free to copy and use. Compatible with Claude 4 Opus, GPT-5, Gemini 2.0 Flash, Llama 4 Maverick.
Choose your languages, sentiment granularity, and domain. Prepare labeled training data in CSV format. Fine-tune the model before deploying the API service.
Initial release
Sign in and download this prompt to leave a review.