ml-pipeline-workflow

• The task is unrelated to ml pipeline workflow
• You need a different domain or tool outside this scope

• Clarify goals, constraints, and required inputs.
• Apply relevant best practices and validate outcomes.
• Provide actionable steps and verification.
• If detailed examples are required, open resources/implementation-playbook.md.

This skill provides comprehensive guidance for building production ML pipelines that handle the full lifecycle: data ingestion → preparation → training → validation → deployment → monitoring.

• Building new ML pipelines from scratch
• Designing workflow orchestration for ML systems
• Implementing data → model → deployment automation
• Setting up reproducible training workflows
• Creating DAG-based ML orchestration
• Integrating ML components into production systems

Core Capabilities

1. Pipeline Architecture

- End-to-end workflow design

- DAG orchestration patterns (Airflow, Dagster, Kubeflow)

- Component dependencies and data flow

- Error handling and retry strategies

1. Data Preparation

- Data validation and quality checks

- Feature engineering pipelines

- Data versioning and lineage

- Train/validation/test splitting strategies

1. Model Training

- Training job orchestration

- Hyperparameter management

- Experiment tracking integration

- Distributed training patterns

1. Model Validation

- Validation frameworks and metrics

- A/B testing infrastructure

- Performance regression detection

- Model comparison workflows

1. Deployment Automation

- Model serving patterns

- Canary deployments

- Blue-green deployment strategies

- Rollback mechanisms

Reference Documentation

See the references/ directory for detailed guides:

• data-preparation.md - Data cleaning, validation, and feature engineering
• model-training.md - Training workflows and best practices
• model-validation.md - Validation strategies and metrics
• model-deployment.md - Deployment patterns and serving architectures

Assets and Templates

The assets/ directory contains:

• pipeline-dag.yaml.template - DAG template for workflow orchestration
• training-config.yaml - Training configuration template
• validation-checklist.md - Pre-deployment validation checklist

Basic Pipeline Setup

```python

# 1. Define pipeline stages

stages = [

"data_ingestion",

"data_validation",

"feature_engineering",

"model_training",

"model_validation",

"model_deployment"

]

# 2. Configure dependencies

# See assets/pipeline-dag.yaml.template for full example

```

Production Workflow

1. Data Preparation Phase

- Ingest raw data from sources

- Run data quality checks

- Apply feature transformations

- Version processed datasets

1. Training Phase

- Load versioned training data

- Execute training jobs

- Track experiments and metrics

- Save trained models

1. Validation Phase

- Run validation test suite

- Compare against baseline

- Generate performance reports

- Approve for deployment

1. Deployment Phase

- Package model artifacts

- Deploy to serving infrastructure

- Configure monitoring

- Validate production traffic

Pipeline Design

• Modularity: Each stage should be independently testable
• Idempotency: Re-running stages should be safe
• Observability: Log metrics at every stage
• Versioning: Track data, code, and model versions
• Failure Handling: Implement retry logic and alerting

Data Management

• Use data validation libraries (Great Expectations, TFX)
• Version datasets with DVC or similar tools
• Document feature engineering transformations
• Maintain data lineage tracking

Model Operations

• Separate training and serving infrastructure
• Use model registries (MLflow, Weights & Biases)
• Implement gradual rollouts for new models
• Monitor model performance drift
• Maintain rollback capabilities

Deployment Strategies

• Start with shadow deployments
• Use canary releases for validation
• Implement A/B testing infrastructure
• Set up automated rollback triggers
• Monitor latency and throughput

Orchestration Tools

• Apache Airflow: DAG-based workflow orchestration
• Dagster: Asset-based pipeline orchestration
• Kubeflow Pipelines: Kubernetes-native ML workflows
• Prefect: Modern dataflow automation

Experiment Tracking

• MLflow for experiment tracking and model registry
• Weights & Biases for visualization and collaboration
• TensorBoard for training metrics

Deployment Platforms

• AWS SageMaker for managed ML infrastructure
• Google Vertex AI for GCP deployments
• Azure ML for Azure cloud
• Kubernetes + KServe for cloud-agnostic serving

Start with the basics and gradually add complexity:

1. Level 1: Simple linear pipeline (data → train → deploy)
2. Level 2: Add validation and monitoring stages
3. Level 3: Implement hyperparameter tuning
4. Level 4: Add A/B testing and gradual rollouts
5. Level 5: Multi-model pipelines with ensemble strategies

Batch Training Pipeline

```yaml

# See assets/pipeline-dag.yaml.template

stages:

- name: data_preparation

dependencies: []

- name: model_training

dependencies: [data_preparation]

- name: model_evaluation

dependencies: [model_training]

- name: model_deployment

dependencies: [model_evaluation]

```

Real-time Feature Pipeline

```python

# Stream processing for real-time features

# Combined with batch training

# See references/data-preparation.md

```

Continuous Training

```python

# Automated retraining on schedule

# Triggered by data drift detection

# See references/model-training.md

```

Common Issues

• Pipeline failures: Check dependencies and data availability
• Training instability: Review hyperparameters and data quality
• Deployment issues: Validate model artifacts and serving config
• Performance degradation: Monitor data drift and model metrics

Debugging Steps

1. Check pipeline logs for each stage
2. Validate input/output data at boundaries
3. Test components in isolation
4. Review experiment tracking metrics
5. Inspect model artifacts and metadata

After setting up your pipeline:

1. Explore hyperparameter-tuning skill for optimization
2. Learn experiment-tracking-setup for MLflow/W&B
3. Review model-deployment-patterns for serving strategies
4. Implement monitoring with observability tools

• experiment-tracking-setup: MLflow and Weights & Biases integration
• hyperparameter-tuning: Automated hyperparameter optimization
• model-deployment-patterns: Advanced deployment strategies