Build production ML pipelines with Kubeflow including data processing, training, evaluation, and model serving stages.
You are an MLOps architect. Build a production ML pipeline with Kubeflow Pipelines. ## Configuration - **ML Task**: [TASK e.g. text classification / object detection / recommendation] - **Data Source**: [SOURCE e.g. BigQuery / S3 / PostgreSQL / Kafka] - **Training Framework**: [FRAMEWORK e.g. PyTorch / TensorFlow / XGBoost] - **Serving Platform**: [SERVING e.g. KServe / Seldon / TF Serving / Triton] - **Cluster**: [CLUSTER e.g. GKE / EKS / on-premise k8s] ## Pipeline Components ### 1. Data Pipeline Component (`components/data/`) - Data extraction from [SOURCE] - Data validation with Great Expectations or TensorFlow Data Validation - Feature engineering and transformation - Train/validation/test split with stratification - Data versioning with DVC or artifact tracking - Schema drift detection ### 2. Training Component (`components/train/`) - Configurable hyperparameters via pipeline parameters - [FRAMEWORK] training with GPU support - Distributed training for large models - Experiment tracking integration (MLflow/W&B) - Checkpoint saving and resumption - Resource requests/limits for cost optimization ### 3. Evaluation Component (`components/evaluate/`) - Model performance metrics for [TASK] - Comparison against baseline model - Fairness and bias evaluation - Performance threshold gates (pass/fail) - Confusion matrix and ROC curve artifacts - Model card generation ### 4. Model Registry Component (`components/registry/`) - Register model with metadata and metrics - Version tagging and lineage tracking - Promotion workflow: staging → production - Model comparison dashboard ### 5. Serving Component (`components/serve/`) - Deploy to [SERVING] with canary rollout - A/B testing configuration - Autoscaling based on request volume - Request/response logging - Model monitoring for drift detection ### 6. Pipeline Definition (`pipeline.py`) - KFP v2 SDK pipeline definition - Conditional execution based on evaluation results - Caching for expensive components - Pipeline parameters for experimentation - Scheduled execution (daily/weekly retraining) - Slack/email notifications on completion/failure Provide all component containers, pipeline definition, and deployment manifests for [CLUSTER].
Free to copy and use. Compatible with Claude 4 Opus, GPT-5, Gemini 2.5 Pro.
Set up Kubeflow Pipelines on your cluster. Build component Docker images and push to your registry. Define pipeline parameters and run an initial test pipeline with a small dataset.
Initial release
Sign in and download this prompt to leave a review.