eks-observability

Complete observability solution for Amazon EKS using AWS-native managed services and open-source tools. This skill implements the three-pillar approach (metrics, logs, traces) with 2025 best practices including ADOT, Amazon Managed Prometheus, Fluent Bit, and OpenTelemetry.

Keywords: EKS monitoring, CloudWatch Container Insights, Prometheus, Grafana, ADOT, Fluent Bit, X-Ray, OpenTelemetry, distributed tracing, log aggregation, metrics collection, observability stack

Status: Production-ready with 2025 best practices

• Setting up monitoring for EKS clusters
• Implementing centralized logging pipelines
• Configuring distributed tracing
• Building production dashboards in Grafana
• Troubleshooting application performance
• Establishing SLOs and error budgets
• Optimizing observability costs
• Migrating from X-Ray SDKs to OpenTelemetry
• Correlating metrics, logs, and traces
• Setting up alerting and on-call runbooks

1. Metrics

CloudWatch Container Insights + Amazon Managed Prometheus (AMP)

• Dual monitoring provides complete visibility
• CloudWatch for AWS-native integration and quick setup
• Prometheus for advanced queries and community dashboards
• Amazon Managed Grafana for visualization

2. Logs

Fluent Bit → CloudWatch Logs

• Lightweight log forwarder (AWS deprecated FluentD in Feb 2025)
• DaemonSet deployment for automatic collection
• Structured logging with JSON parsing
• Optional aggregation to OpenSearch for analytics

3. Traces

ADOT → AWS X-Ray

• OpenTelemetry standard (X-Ray SDKs entering maintenance mode 2026)
• ADOT Collector converts OTLP to X-Ray format
• Distributed tracing across microservices
• Integration with CloudWatch ServiceLens

Step 1: Enable CloudWatch Container Insights

Using EKS Add-on (Recommended):

```bash

# Create IAM policy for CloudWatch access

aws iam create-policy \

--policy-name CloudWatchAgentServerPolicy \

--policy-document file://cloudwatch-policy.json

# Create IRSA for CloudWatch

eksctl create iamserviceaccount \

--name cloudwatch-agent \

--namespace amazon-cloudwatch \

--cluster my-cluster \

--attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \

--approve \

--override-existing-serviceaccounts

# Install Container Insights add-on

aws eks create-addon \

--cluster-name my-cluster \

--addon-name amazon-cloudwatch-observability \

--service-account-role-arn arn:aws:iam::ACCOUNT_ID:role/CloudWatchAgentRole

```

Verify Installation:

```bash

# Check add-on status

aws eks describe-addon \

--cluster-name my-cluster \

--addon-name amazon-cloudwatch-observability

# Verify pods running

kubectl get pods -n amazon-cloudwatch

```

What You Get:

• Node-level metrics (CPU, memory, disk, network)
• Pod-level metrics (resource usage, restart counts)
• Namespace-level aggregations
• Automatic CloudWatch Logs integration
• Pre-built CloudWatch dashboards

Step 2: Deploy Amazon Managed Prometheus

Create AMP Workspace:

```bash

# Create workspace

aws amp create-workspace \

--alias my-cluster-metrics \

--region us-west-2

# Get workspace ID

WORKSPACE_ID=$(aws amp list-workspaces \

--alias my-cluster-metrics \

--query 'workspaces[0].workspaceId' \

--output text)

# Create IRSA for AMP ingestion

eksctl create iamserviceaccount \

--name amp-ingest \

--namespace prometheus \

--cluster my-cluster \

--attach-policy-arn arn:aws:iam::aws:policy/AmazonPrometheusRemoteWriteAccess \

--approve

```

Deploy kube-prometheus-stack:

```bash

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm repo update

# Install with AMP remote write

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \

--namespace prometheus \

--create-namespace \

--set prometheus.prometheusSpec.remoteWrite[0].url=https://aps-workspaces.us-west-2.amazonaws.com/workspaces/${WORKSPACE_ID}/api/v1/remote_write \

--set prometheus.prometheusSpec.remoteWrite[0].sigv4.region=us-west-2 \

--set prometheus.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"="arn:aws:iam::ACCOUNT_ID:role/AMPIngestRole"

```

What You Get:

• Prometheus Operator for CRD-based monitoring
• Node Exporter for hardware metrics
• kube-state-metrics for cluster state
• Alertmanager for alert routing
• 100+ pre-built Grafana dashboards

Step 3: Deploy Fluent Bit for Logging

Create IRSA for Fluent Bit:

```bash

eksctl create iamserviceaccount \

--name fluent-bit \

--namespace logging \

--cluster my-cluster \

--attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \

--approve

```

Deploy Fluent Bit:

```bash

helm repo add fluent https://fluent.github.io/helm-charts

helm install fluent-bit fluent/fluent-bit \

--namespace logging \

--create-namespace \

--set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"="arn:aws:iam::ACCOUNT_ID:role/FluentBitRole" \

--set cloudWatch.enabled=true \

--set cloudWatch.region=us-west-2 \

--set cloudWatch.logGroupName=/aws/eks/my-cluster/logs \

--set cloudWatch.autoCreateGroup=true

```

What You Get:

• Automatic log collection from all pods
• Structured JSON log parsing
• CloudWatch Logs integration
• Multi-line log support
• Kubernetes metadata enrichment

Step 4: Deploy ADOT for Distributed Tracing

Install ADOT Operator:

```bash

# Create IRSA for ADOT

eksctl create iamserviceaccount \

--name adot-collector \

--namespace adot \

--cluster my-cluster \

--attach-policy-arn arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess \

--attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \

--approve

# Install ADOT add-on

aws eks create-addon \

--cluster-name my-cluster \

--addon-name adot \

--service-account-role-arn arn:aws:iam::ACCOUNT_ID:role/ADOTCollectorRole

```

Deploy ADOT Collector:

```yaml

# adot-collector.yaml

apiVersion: opentelemetry.io/v1alpha1

kind: OpenTelemetryCollector

metadata:

name: adot-collector

namespace: adot

spec:

mode: deployment

serviceAccount: adot-collector

config: |

receivers:

otlp:

protocols:

grpc:

endpoint: 0.0.0.0:4317

http:

endpoint: 0.0.0.0:4318

processors:

batch:

timeout: 30s

send_batch_size: 50

memory_limiter:

check_interval: 1s

limit_mib: 512

exporters:

awsxray:

region: us-west-2

awsemf:

region: us-west-2

namespace: EKS/Observability

service:

pipelines:

traces:

receivers: [otlp]

processors: [memory_limiter, batch]

exporters: [awsxray]

metrics:

receivers: [otlp]

processors: [memory_limiter, batch]

exporters: [awsemf]

```

```bash

kubectl apply -f adot-collector.yaml

```

What You Get:

• OTLP receiver for OpenTelemetry traces
• Automatic X-Ray integration
• Service map visualization
• Trace sampling and filtering
• CloudWatch ServiceLens integration

Step 5: Setup Amazon Managed Grafana

Create AMG Workspace:

```bash

# Create workspace (via AWS Console recommended)

# Or use AWS CLI:

aws grafana create-workspace \

--workspace-name my-cluster-grafana \

--account-access-type CURRENT_ACCOUNT \

--authentication-providers AWS_SSO \

--permission-type SERVICE_MANAGED

```

Add Data Sources:

1. Navigate to AMG workspace URL
2. Configuration → Data Sources → Add data source
3. Add Amazon Managed Service for Prometheus

- Region: us-west-2

- Workspace: Select your AMP workspace

1. Add CloudWatch

- Default region: us-west-2

- Namespaces: ContainerInsights, EKS/Observability

1. Add AWS X-Ray

- Default region: us-west-2

Import Dashboards:

```bash

# EKS Container Insights Dashboard

Dashboard ID: 16028

# Node Exporter Full Dashboard

Dashboard ID: 1860

# Kubernetes Cluster Monitoring

Dashboard ID: 15760

```

Infrastructure

• [ ] CloudWatch Container Insights enabled (EKS add-on)
• [ ] Amazon Managed Prometheus workspace created
• [ ] kube-prometheus-stack deployed with remote write
• [ ] Fluent Bit DaemonSet running on all nodes
• [ ] ADOT Collector deployed (deployment or daemonset)
• [ ] Amazon Managed Grafana workspace created
• [ ] All IRSA roles configured with least-privilege policies

Configuration

• [ ] Prometheus scrape configs include all targets
• [ ] Fluent Bit log groups created and structured
• [ ] ADOT sampling configured (5-10% for high traffic)
• [ ] Grafana data sources connected (AMP, CloudWatch, X-Ray)
• [ ] Log retention policies set (7-90 days typical)
• [ ] Metric retention configured (AMP default 150 days)

Dashboards

• [ ] Cluster overview dashboard (nodes, pods, namespaces)
• [ ] Application performance dashboard (latency, errors, throughput)
• [ ] Resource utilization dashboard (CPU, memory, disk)
• [ ] Cost monitoring dashboard (resource waste, right-sizing)
• [ ] Network performance dashboard (CNO metrics)

Alerting

• [ ] Critical alerts: Pod crash loops, node not ready
• [ ] Performance alerts: High latency, error rate spikes
• [ ] Resource alerts: CPU/memory pressure, disk full
• [ ] Cost alerts: Budget thresholds, waste detection
• [ ] SNS topics configured for notifications
• [ ] PagerDuty/Opsgenie integration (optional)

Application Instrumentation

• [ ] OpenTelemetry SDK integrated in applications
• [ ] Trace context propagation configured
• [ ] Custom metrics exported via OTLP
• [ ] Structured logging with JSON format
• [ ] Log correlation with trace IDs

```

┌─────────────────────────────────────────────────────────────┐

│ EKS Cluster │

│ │

│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │

│ │ Application │ │ Application │ │ Application │ │

│ │ + OTel SDK │ │ + OTel SDK │ │ + OTel SDK │ │

│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │

│ │ │ │ │

│ └──────────────────┴──────────────────┘ │

│ │ │

│ ┌────────▼────────┐ │

│ │ ADOT Collector │ │

│ │ (OTel) │ │

│ └────────┬────────┘ │

│ │ │

│ ┌──────────────────┼──────────────────┐ │

│ │ │ │ │

│ ┌────▼─────┐ ┌────▼─────┐ ┌────▼─────┐ │

│ │Prometheus│ │Fluent Bit│ │Container │ │

│ │ (local) │ │DaemonSet │ │ Insights │ │

│ └────┬─────┘ └────┬─────┘ └────┬─────┘ │

└─────────┼──────────────────┼──────────────────┼────────────┘

│ │ │

│ │ │

┌─────▼─────┐ ┌────▼─────┐ ┌────▼─────┐

│ AMP │ │CloudWatch│ │ X-Ray │

│(Managed │ │ Logs │ │ │

│Prometheus)│ └────┬─────┘ └────┬─────┘

└─────┬─────┘ │ │

│ │ │

└─────────────────┴──────────────────┘

│

┌────────▼────────┐

│Amazon Managed │

│ Grafana │

└─────────────────┘

```

For comprehensive guides on each observability component:

• Metrics Collection: [references/metrics.md](references/metrics.md)

- CloudWatch Container Insights setup

- Amazon Managed Prometheus configuration

- kube-prometheus-stack deployment

- Custom metrics and ServiceMonitors

- Cost optimization strategies

• Centralized Logging: [references/logging.md](references/logging.md)

- Fluent Bit configuration and parsers

- CloudWatch Logs integration

- OpenSearch aggregation (optional)

- Log retention and lifecycle policies

- Troubleshooting log collection

• Distributed Tracing: [references/tracing.md](references/tracing.md)

- ADOT Collector deployment patterns

- OpenTelemetry SDK instrumentation

- X-Ray integration and migration

- Trace sampling strategies

- ServiceLens and trace analysis

Metrics

• Sample high-cardinality metrics (5-10% of labels)
• Use metric relabeling to drop unnecessary labels
• Aggregate metrics before remote write to AMP
• Set appropriate retention periods (30-90 days typical)

Logs

• Implement log sampling for verbose applications
• Use CloudWatch Logs Insights instead of exporting to S3
• Set aggressive retention for debug logs (7 days)
• Keep audit logs longer (90+ days)

Traces

• Sample traces based on traffic (5-10% default)
• Increase sampling for errors (100%)
• Use tail-based sampling for important transactions
• Clean up old X-Ray traces (default 30 days)

Typical Monthly Costs:

• Small cluster (10 nodes): $50-150/month
• Medium cluster (50 nodes): $200-500/month
• Large cluster (200+ nodes): $1000-2000/month

Correlation Between Pillars

Metrics → Logs:

```promql

# Find pods with high error rates

rate(http_requests_total{status=~"5.."}[5m]) > 0.1

# Then search CloudWatch Logs for those pod names

```

Logs → Traces:

```json

// Include trace_id in structured logs

{

"timestamp": "2025-01-27T10:30:00Z",

"level": "error",

"message": "Database connection failed",

"trace_id": "1-67a2f3b1-12456789abcdef012345678",

"span_id": "abcdef0123456789"

}

```

Traces → Metrics:

• Use trace data to identify slow endpoints
• Create SLIs from trace latency percentiles
• Alert on trace error rates

CloudWatch ServiceLens

Unified view combining:

• X-Ray traces (request flow)
• CloudWatch metrics (performance)
• CloudWatch Logs (detailed context)

```bash

# Enable ServiceLens (automatic with Container Insights + X-Ray)

aws servicelens get-service-lens-metrics \

--service-name my-app \

--start-time 2025-01-27T00:00:00Z \

--end-time 2025-01-27T23:59:59Z

```

| Issue | Cause | Fix |

|-------|-------|-----|

| No metrics in AMP | Missing IRSA or remote write config | Check Prometheus pod logs, verify IAM role |

| Logs not appearing | Fluent Bit not running or wrong IAM | kubectl logs -n logging fluent-bit-xxx |

| Traces not in X-Ray | ADOT not deployed or app not instrumented | Verify ADOT pods, check OTel SDK setup |

| High costs | Too much data ingestion | Enable sampling, reduce log verbosity |

| Missing pod metrics | kube-state-metrics not running | Check kube-prometheus-stack installation |

| Grafana can't connect | Data source IAM permissions | Add CloudWatch/AMP read policies to AMG role |

Incident Response

1. Check Grafana overview dashboard - Identify affected services
2. Review X-Ray service map - Find bottleneck in request flow
3. Query CloudWatch Logs Insights - Get detailed error messages
4. Correlate with metrics spike - Understand timeline and scope
5. Execute remediation - Scale, restart, or rollback

Performance Investigation

1. Start with RED metrics (Rate, Errors, Duration)
2. Check USE metrics (Utilization, Saturation, Errors) for infrastructure
3. Analyze trace percentiles (p50, p95, p99)
4. Review log patterns during slow periods
5. Identify optimization opportunities

Define SLIs (Service Level Indicators):

```yaml

# Availability SLI

• metric: probe_success

target: 99.9%

window: 30d

# Latency SLI

• metric: http_request_duration_seconds

percentile: p99

target: < 500ms

window: 30d

# Error Rate SLI

• metric: http_requests_total{status=~"5.."}

target: < 0.1%

window: 30d

```

Calculate Error Budget:

```

Error Budget = 100% - SLO Target

Example: 99.9% SLO = 0.1% error budget

= 43.2 minutes downtime/month

```

Burn Rate Alerts:

```promql

# Fast burn (5% budget in 1 hour)

(1 - slo:availability:ratio_rate_1h) > 0.05

# Slow burn (10% budget in 6 hours)

(1 - slo:availability:ratio_rate_6h) > 0.1

```

1. Use Dual Monitoring: CloudWatch Container Insights + Prometheus
2. Standardize on OpenTelemetry: Future-proof instrumentation
3. Enable IRSA for Everything: No node IAM roles
4. Deploy ADOT Collector: Vendor-neutral observability
5. Sample Intelligently: 5-10% traces, 100% errors
6. Structure Your Logs: JSON format with trace correlation
7. Set Retention Policies: Balance cost and compliance
8. Build Actionable Dashboards: Focus on SLIs and anomalies
9. Implement Progressive Alerting: Warn before critical
10. Regularly Review Costs: Optimize based on actual usage

---

Stack: CloudWatch Container Insights, AMP, Fluent Bit, ADOT, AMG, X-Ray

Standards: OpenTelemetry, IRSA, EKS Add-ons

Last Updated: January 2025 (2025 Best Practices)