mlflow
π―SkillInstallation
npx skills add g1joshi/agent-skills --skill mlflowRun it in a terminal; it asks which AI coding tools to install it for (Claude Code, Codex, Cursor and more).
Uses skills.sh Β· Node.js 22.20+
After installing
Ask for related work and the agent uses it on its own, or call it yourself.
/mlflowInstall by hand
Copy the whole skill folder from GitHub to the place below. Copying only SKILL.md can leave out files the skill uses.
Open the skill folder β- This project
.claude/skills/mlflow/ - All projects
~/.claude/skills/mlflow/
These paths are for Claude Code. For other tools, the command above puts it in the right place.
SKILL.md
View on GitHub βExpert MLflow assistance covering experiment tracking, autologging, model registry, and MLflow recipes. Use when tracking ML metrics, versioning model artifacts, or serving production models.
MLflow
MLflow is an open-source platform for managing the end-to-end machine learning lifecycle, including experiment tracking, model registry, artifact storage, and LLM tracing evaluation.
When to Use
- Machine Learning Experiment Tracking: Logging parameters, metrics, code versions, and artifacts across runs.
- Model Registry & Governance: Managing model lifecycle stages (Staging, Production, Archived) with lineage.
- LLM Evaluation & Prompt Engineering: Evaluating LLMs, RAG applications, and prompts using MLflow Evaluate.
- Unified Model Deployment: Packaging models into self-contained flavors (Python function, PyTorch, ONNX) for one-click deployment.
Quick Start
import mlflow
import mlflow.sklearn
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error
mlflow.set_experiment("housing_price_prediction")
with mlflow.start_run():
n_estimators = 50
model = RandomForestRegressor(n_estimators=n_estimators)
model.fit(X_train, y_train)
preds = model.predict(X_val)
mse = mean_squared_error(y_val, preds)
mlflow.log_param("n_estimators", n_estimators)
mlflow.log_metric("mse", mse)
mlflow.sklearn.log_model(model, "random_forest_model")
Core Concepts
Experiment Tracking & Autologging
Logging parameters, evaluation metrics, and model weights automatically:
import mlflow
import mlflow.sklearn
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
# Configure remote or local tracking URI
mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("iris_classification_prod")
# Enable automatic framework logging
mlflow.sklearn.autolog(log_model_signatures=True)
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
with mlflow.start_run(run_name="rf_n100_d5") as run:
params = {"n_estimators": 100, "max_depth": 5, "random_state": 42}
mlflow.log_params(params)
clf = RandomForestClassifier(**params)
clf.fit(X_train, y_train)
score = clf.score(X_test, y_test)
mlflow.log_metric("accuracy", score)
print(f"Logged run: {run.info.run_id} with Accuracy: {score:.4f}")
Model Registry & Production Staging
Registering and promoting versioned models:
from mlflow import MlflowClient
client = MlflowClient()
# Register model from a completed run artifact
model_uri = f"runs:/{run.info.run_id}/model"
registered_model = mlflow.register_model(model_uri, "CustomerChurnPredictor")
# Assign an alias (MLflow 2.x recommended pattern)
client.set_registered_model_alias(
name="CustomerChurnPredictor",
alias="champion",
version=registered_model.version
)
# Load champion model in production serving microservice
champion_model = mlflow.pyfunc.load_model("models:/CustomerChurnPredictor@champion")
predictions = champion_model.predict(X_test)
LLM Evaluation with mlflow.evaluate
Benchmarking RAG outputs against ground truth datasets:
import mlflow
import pandas as pd
eval_df = pd.DataFrame({
"inputs": ["What is MLflow?", "How does autologging work?"],
"ground_truth": [
"MLflow is an open-source platform for managing the end-to-end ML lifecycle.",
"Autologging automatically captures metrics, parameters, and models without explicit log statements."
],
"predictions": [
"MLflow manages machine learning experiments, models, and deployments.",
"It automatically tracks parameters and metrics during training."
]
})
with mlflow.start_run():
results = mlflow.evaluate(
data=eval_df,
targets="ground_truth",
predictions="predictions",
model_type="text-summarization",
evaluators="default"
)
print("Evaluation Metrics:", results.metrics)
Common Patterns
Autologging Framework Integrations
Problem: Writing manual mlflow.log_metric lines for every epoch and hyperparameter.
Solution: Enable automatic framework-level logging:
import mlflow
# Autolog PyTorch, TensorFlow, Scikit-Learn, LightGBM, or XGBoost
mlflow.autolog()
# Subsequent model.fit() calls automatically log all parameters, metrics, and models
model.fit(X_train, y_train)
Best Practices
Do:
- Target MLflow 2.15+ utilizing model aliases (
@champion,@challenger) rather than legacy stage transitions. - Log model signatures (
mlflow.models.infer_signature) to ensure input/output schema validation at deployment time. - Use
mlflow.start_run()inside Python context managers to ensure runs are reliably closed on errors. - Back up the remote backend store (PostgreSQL) and artifact repository (S3/GCS) regularly.
Don't:
- Store large training datasets directly as artifacts; log dataset hashes and S3 URIs via
mlflow.data. - Use local filesystem tracking URIs in production or collaborative team environments.
- Hardcode tracking URIs; configure via environment variable
MLFLOW_TRACKING_URI.
Troubleshooting
| Error | Cause | Solution |
|---|---|---|
MlflowException: Could not find experiment with ID | Experiment deleted or connecting to mismatched tracking URI. | Set explicit tracking URI: mlflow.set_tracking_uri("http://localhost:5000"). |
Artifact transfer failed | S3 / GCS bucket permissions missing on client runner. | Ensure AWS/GCP credentials with write permissions are active in environment. |
Schema enforcement error on model load | Input DataFrame columns mismatch model signature logged at training. | Align DataFrame columns and types with logged ModelSignature. |
References
More from this repository10
A template skill providing a starter framework for creating new agent skills with structured sections for triggers, quick start examples, and capability descriptions.
Skill
Skill
Skill
Skill
Skill
Skill
Skill
Skill
Skill