senior-data-engineer

Main Capabilities

```bash

# Core Tool 1

python scripts/pipeline_orchestrator.py --input data/ --output results/

# Core Tool 2

python scripts/data_quality_validator.py --target project/ --analyze

# Core Tool 3

python scripts/etl_performance_optimizer.py --config config.yaml --deploy

```

This skill covers world-class capabilities in:

• Advanced production patterns and architectures
• Scalable system design and implementation
• Performance optimization at scale
• MLOps and DataOps best practices
• Real-time processing and inference
• Distributed computing frameworks
• Model deployment and monitoring
• Security and compliance
• Cost optimization
• Team leadership and mentoring

Languages: Python, SQL, R, Scala, Go

ML Frameworks: PyTorch, TensorFlow, Scikit-learn, XGBoost

Data Tools: Spark, Airflow, dbt, Kafka, Databricks

LLM Frameworks: LangChain, LlamaIndex, DSPy

Deployment: Docker, Kubernetes, AWS/GCP/Azure

Monitoring: MLflow, Weights & Biases, Prometheus

Databases: PostgreSQL, BigQuery, Snowflake, Pinecone

1. Data Pipeline Architecture

Comprehensive guide available in references/data_pipeline_architecture.md covering:

• Advanced patterns and best practices
• Production implementation strategies
• Performance optimization techniques
• Scalability considerations
• Security and compliance
• Real-world case studies

2. Data Modeling Patterns

Complete workflow documentation in references/data_modeling_patterns.md including:

• Step-by-step processes
• Architecture design patterns
• Tool integration guides
• Performance tuning strategies
• Troubleshooting procedures

3. Dataops Best Practices

Technical reference guide in references/dataops_best_practices.md with:

• System design principles
• Implementation examples
• Configuration best practices
• Deployment strategies
• Monitoring and observability

Pattern 1: Scalable Data Processing

Enterprise-scale data processing with distributed computing:

• Horizontal scaling architecture
• Fault-tolerant design
• Real-time and batch processing
• Data quality validation
• Performance monitoring

Pattern 2: ML Model Deployment

Production ML system with high availability:

• Model serving with low latency
• A/B testing infrastructure
• Feature store integration
• Model monitoring and drift detection
• Automated retraining pipelines

Pattern 3: Real-Time Inference

High-throughput inference system:

• Batching and caching strategies
• Load balancing
• Auto-scaling
• Latency optimization
• Cost optimization

Development

• Test-driven development
• Code reviews and pair programming
• Documentation as code
• Version control everything
• Continuous integration

Production

• Monitor everything critical
• Automate deployments
• Feature flags for releases
• Canary deployments
• Comprehensive logging

Team Leadership

• Mentor junior engineers
• Drive technical decisions
• Establish coding standards
• Foster learning culture
• Cross-functional collaboration

Latency:

• P50: < 50ms
• P95: < 100ms
• P99: < 200ms

Throughput:

• Requests/second: > 1000
• Concurrent users: > 10,000

Availability:

• Uptime: 99.9%
• Error rate: < 0.1%

• Authentication & authorization
• Data encryption (at rest & in transit)
• PII handling and anonymization
• GDPR/CCPA compliance
• Regular security audits
• Vulnerability management

```bash

# Development

python -m pytest tests/ -v --cov

python -m black src/

python -m pylint src/

# Training

python scripts/train.py --config prod.yaml

python scripts/evaluate.py --model best.pth

# Deployment

docker build -t service:v1 .

kubectl apply -f k8s/

helm upgrade service ./charts/

# Monitoring

kubectl logs -f deployment/service

python scripts/health_check.py

```

• Advanced Patterns: references/data_pipeline_architecture.md
• Implementation Guide: references/data_modeling_patterns.md
• Technical Reference: references/dataops_best_practices.md
• Automation Scripts: scripts/ directory

As a world-class senior professional:

1. Technical Leadership

- Drive architectural decisions

- Mentor team members

- Establish best practices

- Ensure code quality

1. Strategic Thinking

- Align with business goals

- Evaluate trade-offs

- Plan for scale

- Manage technical debt

1. Collaboration

- Work across teams

- Communicate effectively

- Build consensus

- Share knowledge

1. Innovation

- Stay current with research

- Experiment with new approaches

- Contribute to community

- Drive continuous improvement

1. Production Excellence

- Ensure high availability

- Monitor proactively

- Optimize performance

- Respond to incidents