prometheus
π―SkillInstallation
npx skills add g1joshi/agent-skills --skill prometheusRun it in a terminal; it asks which AI coding tools to install it for (Claude Code, Codex, Cursor and more).
Uses skills.sh Β· Node.js 22.20+
After installing
Ask for related work and the agent uses it on its own, or call it yourself.
/prometheusInstall by hand
Copy the whole skill folder from GitHub to the place below. Copying only SKILL.md can leave out files the skill uses.
Open the skill folder β- This project
.claude/skills/prometheus/ - All projects
~/.claude/skills/prometheus/
These paths are for Claude Code. For other tools, the command above puts it in the right place.
SKILL.md
View on GitHub βExpert Prometheus monitoring assistance covering PromQL, metric types (Counter, Gauge, Histogram), scrape configs, and Alertmanager. Use when collecting and querying operational metrics across microservices.
Prometheus
Prometheus is the CNCF graduated monitoring and alerting toolkit, featuring a multi-dimensional data model with time-series metrics, PromQL, and native OpenTelemetry ingestion.
When to Use
- Cloud-Native Time-Series Monitoring: Scraping, storing, and querying numerical metrics from infrastructure and applications.
- PromQL Quantitative Querying: Computing request rates, 99th percentile latencies, error ratios, and saturations.
- Automated Service Discovery: Dynamically discovering Kubernetes pods, EC2 instances, and Consul nodes to scrape.
- Alerting Rules & Alertmanager Integration: Evaluating threshold rules and dispatching alerts to on-call engineers.
Quick Start
# prometheus.yml
global:
scrape_interval: 15s
scrape_configs:
- job_name: "node"
static_configs:
- targets: ["localhost:9100"]
Core Concepts
Prometheus Scrape Configuration (prometheus.yml)
Configuring scrape jobs with relabeling:
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
- "alert_rules.yml"
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
- job_name: "api-microservices"
scrape_interval: 10s
metrics_path: "/metrics"
kubernetes_sd_configs:
- role: pod
namespaces:
names: ["production"]
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: "true"
- source_labels: [__meta_kubernetes_pod_container_port_number]
action: keep
regex: "8080"
Production Alerting Rules (alert_rules.yml)
Defining SLO-based alert thresholds:
groups:
- name: API_SLO_Alerts
rules:
- alert: HighHttpErrorRate
expr: |
(
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
) * 100 > 5
for: 2m
labels:
severity: critical
annotations:
summary: "High HTTP 5xx error rate on {{ $labels.service }}"
description: 'Service {{ $labels.service }} error rate is {{ $value | printf "%.2f" }}% (> 5% SLO threshold).'
Exposing Custom Metrics in Python
Instrumentation using official Prometheus Python client:
from prometheus_client import start_http_server, Counter, Histogram
import time
REQUEST_COUNTER = Counter(
'http_requests_total',
'Total count of HTTP requests processed',
['method', 'endpoint', 'status']
)
REQUEST_DURATION = Histogram(
'http_request_duration_seconds',
'Histogram of HTTP request latency in seconds',
['endpoint'],
buckets=(0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0)
)
@REQUEST_DURATION.labels(endpoint='/api/orders').time()
def process_order():
time.sleep(0.08)
REQUEST_COUNTER.labels(method='POST', endpoint='/api/orders', status='200').inc()
# Start metrics endpoint server on port 8000
start_http_server(8000)
Common Patterns
High-Throughput Scrape Config with Kubernetes Service Discovery
Problem: Static IP scraping configurations fail as pods scale dynamically in Kubernetes.
Solution:
Use kubernetes_sd_configs with metric path annotations:
scrape_configs:
- job_name: "kubernetes-pods"
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels:
[__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
Best Practices
Do:
- Adhere strictly to the Four Golden Signals (Latency, Traffic, Errors, Saturation) when building PromQL alert rules.
- Configure appropriate histogram buckets around your service SLO targets (e.g. 100ms, 200ms, 500ms).
- Use recording rules (
record: job:metric:rate5m) for complex PromQL expressions queried by dashboards. - Store long-term historical metrics using remote-write storage (Thanos, Cortex, or VictoriaMetrics).
Don't:
- Introduce high-cardinality label values (user IDs, email addresses, order IDs) into metric labels.
- Set scrape intervals too low (< 5s) across large fleets; it overloads target scrapers and storage.
- Alert on simple transient spikes; always use
for: 2morfor: 5mto filter temporary noise.
Troubleshooting
| Error | Cause | Solution |
|---|---|---|
Target shows DOWN in Prometheus Targets UI | Scrape target port firewalled or application metrics path crashing. | Test target endpoint manually: curl http://<pod-ip>:<port>/metrics. |
High memory usage / Prometheus OOM | High churn in metric series (high cardinality tags like UUIDs or user emails). | Audit and drop unbounded labels using metric_relabel_configs. |
PromQL: many-to-many matching not allowed | Vector matching join (on(...)) without group_left or group_right modifiers. | Add group_left or group_right to specify many-to-one relationship. |
References
More from this repository10
A template skill providing a starter framework for creating new agent skills with structured sections for triggers, quick start examples, and capability descriptions.
Skill
Skill
Skill
Skill
Skill
Skill
Skill
Skill
Skill