🎯

prometheus

🎯Skill
g1joshi/agent-skills1 Installs9Last Updated Feb 10, 2026

Installation

$npx skills add g1joshi/agent-skills --skill prometheus

Run it in a terminal; it asks which AI coding tools to install it for (Claude Code, Codex, Cursor and more).

Uses skills.sh Β· Node.js 22.20+

After installing

Ask for related work and the agent uses it on its own, or call it yourself.

Run it (Claude Code)/prometheus
Install by hand

Copy the whole skill folder from GitHub to the place below. Copying only SKILL.md can leave out files the skill uses.

Open the skill folder β†’
  • This project .claude/skills/prometheus/
  • All projects ~/.claude/skills/prometheus/

These paths are for Claude Code. For other tools, the command above puts it in the right place.

Expert Prometheus monitoring assistance covering PromQL, metric types (Counter, Gauge, Histogram), scrape configs, and Alertmanager. Use when collecting and querying operational metrics across microservices.

Prometheus

Prometheus is the CNCF graduated monitoring and alerting toolkit, featuring a multi-dimensional data model with time-series metrics, PromQL, and native OpenTelemetry ingestion.

When to Use

  • Cloud-Native Time-Series Monitoring: Scraping, storing, and querying numerical metrics from infrastructure and applications.
  • PromQL Quantitative Querying: Computing request rates, 99th percentile latencies, error ratios, and saturations.
  • Automated Service Discovery: Dynamically discovering Kubernetes pods, EC2 instances, and Consul nodes to scrape.
  • Alerting Rules & Alertmanager Integration: Evaluating threshold rules and dispatching alerts to on-call engineers.

Quick Start

# prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "node"
    static_configs:
      - targets: ["localhost:9100"]

Core Concepts

Prometheus Scrape Configuration (prometheus.yml)

Configuring scrape jobs with relabeling:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "alert_rules.yml"

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "api-microservices"
    scrape_interval: 10s
    metrics_path: "/metrics"
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: ["production"]
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
      - source_labels: [__meta_kubernetes_pod_container_port_number]
        action: keep
        regex: "8080"

Production Alerting Rules (alert_rules.yml)

Defining SLO-based alert thresholds:

groups:
  - name: API_SLO_Alerts
    rules:
      - alert: HighHttpErrorRate
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) * 100 > 5
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "High HTTP 5xx error rate on {{ $labels.service }}"
          description: 'Service {{ $labels.service }} error rate is {{ $value | printf "%.2f" }}% (> 5% SLO threshold).'

Exposing Custom Metrics in Python

Instrumentation using official Prometheus Python client:

from prometheus_client import start_http_server, Counter, Histogram
import time

REQUEST_COUNTER = Counter(
    'http_requests_total',
    'Total count of HTTP requests processed',
    ['method', 'endpoint', 'status']
)

REQUEST_DURATION = Histogram(
    'http_request_duration_seconds',
    'Histogram of HTTP request latency in seconds',
    ['endpoint'],
    buckets=(0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0)
)

@REQUEST_DURATION.labels(endpoint='/api/orders').time()
def process_order():
    time.sleep(0.08)
    REQUEST_COUNTER.labels(method='POST', endpoint='/api/orders', status='200').inc()

# Start metrics endpoint server on port 8000
start_http_server(8000)

Common Patterns

High-Throughput Scrape Config with Kubernetes Service Discovery

Problem: Static IP scraping configurations fail as pods scale dynamically in Kubernetes.

Solution: Use kubernetes_sd_configs with metric path annotations:

scrape_configs:
  - job_name: "kubernetes-pods"
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels:
          [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__

Best Practices

Do:

  • Adhere strictly to the Four Golden Signals (Latency, Traffic, Errors, Saturation) when building PromQL alert rules.
  • Configure appropriate histogram buckets around your service SLO targets (e.g. 100ms, 200ms, 500ms).
  • Use recording rules (record: job:metric:rate5m) for complex PromQL expressions queried by dashboards.
  • Store long-term historical metrics using remote-write storage (Thanos, Cortex, or VictoriaMetrics).

Don't:

  • Introduce high-cardinality label values (user IDs, email addresses, order IDs) into metric labels.
  • Set scrape intervals too low (< 5s) across large fleets; it overloads target scrapers and storage.
  • Alert on simple transient spikes; always use for: 2m or for: 5m to filter temporary noise.

Troubleshooting

ErrorCauseSolution
Target shows DOWN in Prometheus Targets UIScrape target port firewalled or application metrics path crashing.Test target endpoint manually: curl http://<pod-ip>:<port>/metrics.
High memory usage / Prometheus OOMHigh churn in metric series (high cardinality tags like UUIDs or user emails).Audit and drop unbounded labels using metric_relabel_configs.
PromQL: many-to-many matching not allowedVector matching join (on(...)) without group_left or group_right modifiers.Add group_left or group_right to specify many-to-one relationship.

References

More from this repository10