ack-resources

AWS Controllers for Kubernetes (ACK) enables you to:

• Define AWS resources as Kubernetes manifests (S3 buckets, RDS databases, SQS queues, etc.)
• Use kubectl to manage AWS infrastructure
• Implement end-to-end GitOps for applications and infrastructure
• Leverage Kubernetes RBAC for AWS resource access control
• Automatically reconcile drift between desired and actual AWS state
• Adopt existing AWS resources without recreation

Architecture Pattern:

```

Git (manifests) → ArgoCD/Flux → Kubernetes (ACK CRDs) → AWS APIs → AWS Resources

↓

Continuous Reconciliation

```

Key Characteristics:

• Kubernetes Native: Uses standard CRDs and the operator pattern
• Direct API Integration: Calls AWS APIs directly (not CloudFormation)
• Modular: Install only the service controllers you need
• GitOps-First: Designed for declarative infrastructure management
• Multi-Account: Supports cross-account resource management (CARM)

Perfect For

✅ EKS Workloads Needing AWS Services

• Applications requiring RDS databases, S3 buckets, SQS queues
• Unified control plane for apps and their infrastructure dependencies
• Tight coupling between Kubernetes workloads and AWS resources

✅ GitOps Infrastructure Workflows

• Version-controlled AWS infrastructure in Git
• PR-based review process for infrastructure changes
• ArgoCD/Flux automated synchronization

✅ Self-Service Developer Platforms

• Developers provision AWS resources via kubectl
• Namespace-based RBAC controls access
• Platform teams define policies and quotas

✅ Multi-Account/Multi-Tenant Environments

• Different teams/namespaces map to different AWS accounts
• Centralized control plane with isolated AWS resources
• Cost allocation per namespace/account

Not Ideal For

❌ Multi-Cloud Infrastructure

• ACK is AWS-only (use Crossplane for multi-cloud)

❌ Comprehensive AWS Coverage Required

• Limited to 14+ GA controllers (use Terraform for broader coverage)

❌ Non-Kubernetes Environments

• Requires Kubernetes cluster (use Terraform/CDK for AWS-only infrastructure)

❌ Stable Production APIs Only

• Many controllers still in alpha (v1alpha1)

Generally Available Controllers

| Service | Controller | Common Use Cases |

|---------|-----------|------------------|

| Amazon S3 | s3-controller | Application data storage, static assets |

| Amazon RDS | rds-controller | PostgreSQL, MySQL, Oracle databases |

| Amazon DynamoDB | dynamodb-controller | NoSQL tables for applications |

| Amazon SQS | sqs-controller | Message queues for async processing |

| Amazon SNS | sns-controller | Notifications and pub/sub messaging |

| AWS Lambda | lambda-controller | Serverless functions |

| Amazon ECR | ecr-controller | Container image repositories |

| Amazon EKS | eks-controller | Additional EKS clusters |

| Amazon EC2 | ec2-controller | VPCs, subnets, security groups |

| AWS IAM | iam-controller | Roles, policies for applications |

| Amazon EFS | efs-controller | Shared file systems |

| Amazon ElastiCache | elasticache-controller | Redis/Memcached clusters |

| Amazon MSK | msk-controller | Managed Kafka clusters |

| API Gateway V2 | apigatewayv2-controller | HTTP/WebSocket APIs |

| Amazon SageMaker | sagemaker-controller | ML model endpoints |

| Amazon Athena | athena-controller | SQL queries on S3 data |

Additional controllers available in preview/beta. See references for complete list.

1. Understand Your Objective

What are you trying to accomplish?

• Setup: Install ACK controllers in your EKS cluster → See [Controller Setup](#2-setup-install-ack-controller)
• Create Resources: Define AWS resources as Kubernetes manifests → See [Create AWS Resources](#3-create-aws-resources)
• Adopt Existing: Import existing AWS resources into ACK → See [Adopt Existing Resources](#5-adopt-existing-aws-resources)
• GitOps: Integrate with ArgoCD/Flux → See [GitOps Integration](#6-gitops-integration)
• Multi-Account: Manage resources across AWS accounts → See [Cross-Account Management](#7-cross-account-management-carm)
• Troubleshoot: Debug ACK resource issues → See [Troubleshooting](#8-troubleshooting)

2. Setup: Install ACK Controller

Prerequisites:

• EKS cluster (Kubernetes 1.16+)
• kubectl configured
• Helm 3.8+
• AWS CLI

Installation Steps:

#### Step 1: Create OIDC Provider (IRSA)

```bash

# Enable IRSA for your EKS cluster

eksctl utils associate-iam-oidc-provider \

--cluster=my-cluster \

--region=us-east-1 \

--approve

```

#### Step 2: Create IAM Policy for Controller

Example for S3 controller:

```json

{

"Version": "2012-10-17",

"Statement": [

{

"Effect": "Allow",

"Action": [

"s3:CreateBucket",

"s3:DeleteBucket",

"s3:ListBucket",

"s3:GetBucket*",

"s3:PutBucket*",

"s3:DeleteBucket*"

],

"Resource": "*"

}

]

}

```

```bash

# Create the IAM policy

aws iam create-policy \

--policy-name ACK-S3-Controller-Policy \

--policy-document file://s3-policy.json

```

#### Step 3: Create IAM Role with IRSA

```bash

# Create service account with IAM role

eksctl create iamserviceaccount \

--name ack-s3-controller \

--namespace ack-system \

--cluster my-cluster \

--attach-policy-arn arn:aws:iam::123456789012:policy/ACK-S3-Controller-Policy \

--approve \

--override-existing-serviceaccounts

```

#### Step 4: Install Controller via Helm

```bash

export HELM_EXPERIMENTAL_OCI=1

# Install S3 controller

helm install ack-s3-controller \

oci://public.ecr.aws/aws-controllers-k8s/s3-chart \

--version=v0.1.7 \

--namespace ack-system \

--create-namespace \

--set=aws.region=us-east-1 \

--set=serviceAccount.create=false \

--set=serviceAccount.name=ack-s3-controller

```

#### Step 5: Verify Installation

```bash

# Check controller is running

kubectl get pods -n ack-system

# List installed CRDs

kubectl get crds | grep s3.services.k8s.aws

```

See reference: references/controller-setup.md for detailed installation guides for all controllers.

3. Create AWS Resources

Define AWS resources using Kubernetes manifests:

#### Example: S3 Bucket

```yaml

apiVersion: s3.services.k8s.aws/v1alpha1

kind: Bucket

metadata:

name: my-app-bucket

namespace: production

spec:

name: my-app-bucket-unique-12345

versioning:

status: Enabled

```

```bash

kubectl apply -f s3-bucket.yaml

# Check status

kubectl get buckets.s3.services.k8s.aws -n production

kubectl describe bucket my-app-bucket -n production

```

#### Example: RDS PostgreSQL Instance

```yaml

apiVersion: rds.services.k8s.aws/v1alpha1

kind: DBInstance

metadata:

name: myapp-db

namespace: production

spec:

dbInstanceIdentifier: myapp-db

dbInstanceClass: db.t3.medium

engine: postgres

engineVersion: "15.4"

allocatedStorage: 20

storageType: gp3

storageEncrypted: true

masterUsername: postgres

masterUserPassword:

name: db-credentials # Kubernetes secret

key: password

backupRetentionPeriod: 7

multiAZ: true

publiclyAccessible: false

dbSubnetGroupName: my-subnet-group

vpcSecurityGroupIDs:

- sg-0123456789abcdef0

```

#### Example: SQS Queue

```yaml

apiVersion: sqs.services.k8s.aws/v1alpha1

kind: Queue

metadata:

name: orders-queue

namespace: production

spec:

queueName: orders-queue

visibilityTimeout: 30

messageRetentionPeriod: 345600 # 4 days

receiveMessageWaitTimeSeconds: 20 # Long polling

```

See reference: references/resource-definitions.md for comprehensive examples of all supported AWS services.

4. Reference and Export Resource Values

ACK resources automatically populate status fields with AWS resource details. Use these in two ways:

#### Pattern 1: Resource References (Cross-Resource)

Reference one ACK resource from another using *Ref fields:

```yaml

# Create API Gateway API

apiVersion: apigatewayv2.services.k8s.aws/v1alpha1

kind: API

metadata:

name: my-api

namespace: default

spec:

name: my-api

protocolType: HTTP

---

# Create Integration referencing the API

apiVersion: apigatewayv2.services.k8s.aws/v1alpha1

kind: Integration

metadata:

name: my-integration

namespace: default

spec:

apiRef:

from:

name: my-api # References API by name

integrationType: AWS_PROXY

integrationURI: arn:aws:lambda:us-east-1:123456789012:function:my-function

```

#### Pattern 2: Field Exports (to ConfigMap/Secret)

Export ACK resource values for use by Kubernetes workloads:

```yaml

# Export S3 bucket ARN to ConfigMap

apiVersion: services.k8s.aws/v1alpha1

kind: FieldExport

metadata:

name: export-bucket-arn

namespace: production

spec:

from:

resource:

group: s3.services.k8s.aws

kind: Bucket

name: my-app-bucket

path: ".status.arn"

to:

kind: configmap

name: app-config

key: S3_BUCKET_ARN

```

Use exported value in pod:

```yaml

apiVersion: v1

kind: Pod

metadata:

name: my-app

namespace: production

spec:

containers:

- name: app

image: my-app:latest

envFrom:

- configMapRef:

name: app-config # Contains S3_BUCKET_ARN

```

5. Adopt Existing AWS Resources

Import existing AWS resources (created outside ACK) without recreation:

```yaml

apiVersion: services.k8s.aws/v1alpha1

kind: AdoptedResource

metadata:

name: adopt-prod-bucket

namespace: production

spec:

aws:

nameOrID: existing-prod-bucket-name # Existing AWS resource

kubernetes:

group: s3.services.k8s.aws

kind: Bucket

metadata:

name: prod-bucket # Name in Kubernetes

namespace: production

```

Process:

1. Apply AdoptedResource manifest
2. ACK describes the existing AWS resource
3. Creates corresponding Kubernetes resource with full spec/status
4. Future changes managed via the ACK resource

Benefits:

• No resource recreation (zero downtime)
• Gradual migration to ACK
• Preserve existing configurations

6. GitOps Integration

Manage ACK resources through Git with ArgoCD or Flux:

#### ArgoCD Application

```yaml

apiVersion: argoproj.io/v1alpha1

kind: Application

metadata:

name: aws-infrastructure

namespace: argocd

spec:

project: default

source:

repoURL: https://github.com/myorg/infrastructure

path: ack-resources/production

targetRevision: main

destination:

server: https://kubernetes.default.svc

namespace: production

syncPolicy:

automated:

prune: false # Safety: don't auto-delete AWS resources

selfHeal: true # Auto-correct drift

syncOptions:

- CreateNamespace=true

```

#### Flux Kustomization

```yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1

kind: Kustomization

metadata:

name: aws-infrastructure

namespace: flux-system

spec:

interval: 10m

path: ./ack-resources/production

prune: false # Safety: don't auto-delete

sourceRef:

kind: GitRepository

name: infrastructure

targetNamespace: production

```

Workflow:

1. Developers create/modify ACK resources in Git
2. Submit PR for review
3. Merge to main triggers ArgoCD/Flux sync
4. ACK controllers create/update AWS resources
5. Continuous reconciliation ensures consistency

See reference: references/gitops-patterns.md for comprehensive GitOps workflows and best practices.

7. Cross-Account Management (CARM)

Manage AWS resources in multiple accounts from a single Kubernetes cluster:

#### Step 1: Create ConfigMap with Account Role Mappings

```yaml

apiVersion: v1

kind: ConfigMap

metadata:

name: ack-role-account-map

namespace: ack-system

data:

"111122223333": "arn:aws:iam::111122223333:role/ack-controller-role"

"444455556666": "arn:aws:iam::444455556666:role/ack-controller-role"

```

#### Step 2: Annotate Namespace with Account ID

```yaml

apiVersion: v1

kind: Namespace

metadata:

name: team-a

annotations:

services.k8s.aws/owner-account-id: "111122223333"

```

#### Step 3: Deploy Resources (Uses Namespace Account)

```yaml

apiVersion: s3.services.k8s.aws/v1alpha1

kind: Bucket

metadata:

name: team-a-bucket

namespace: team-a # Uses account 111122223333

spec:

name: team-a-bucket-unique

```

Benefits:

• Multi-tenancy with separate billing
• RBAC-based access control
• Security boundaries per team/account

8. Troubleshooting

#### Check Resource Status

```bash

# View resource status

kubectl get buckets.s3.services.k8s.aws -n production

kubectl describe bucket my-bucket -n production

# Check status conditions

kubectl get bucket my-bucket -n production -o jsonpath='{.status.conditions}'

```

Status Conditions:

• ACK.ResourceSynced: True - Resource matches AWS state
• ACK.Terminal: True - Resource in error state
• ACK.Recovering: True - Resource recovering from error

#### Check Controller Logs

```bash

# View controller logs

kubectl logs -n ack-system deployment/ack-s3-controller -f

# Filter for specific resource

kubectl logs -n ack-system deployment/ack-s3-controller | grep my-bucket

```

#### Common Issues

Issue: Resource stuck in pending state

```bash

# Check events

kubectl describe bucket my-bucket -n production

# Common causes:

# - IAM permissions missing

# - AWS API throttling

# - Invalid resource configuration

```

Issue: Drift reconciliation not working

```bash

# Verify controller is running

kubectl get pods -n ack-system

# Check reconciliation interval (default ~10 minutes)

# Force reconciliation by annotating resource

kubectl annotate bucket my-bucket -n production force-sync="$(date +%s)"

```

Issue: Cross-account resources failing

```bash

# Verify namespace annotation

kubectl get namespace team-a -o yaml | grep owner-account-id

# Verify ConfigMap has role mapping

kubectl get configmap ack-role-account-map -n ack-system -o yaml

# Check controller has assume role permissions

```

1. Deletion Policies

Protect critical resources from accidental deletion:

```yaml

apiVersion: rds.services.k8s.aws/v1alpha1

kind: DBInstance

metadata:

name: production-db

namespace: production

annotations:

services.k8s.aws/deletion-policy: retain # Keep AWS resource on delete

spec:

dbInstanceIdentifier: production-db

# ... spec

```

Policy hierarchy:

1. Resource annotation (highest priority)
2. Namespace annotation
3. Controller flag (lowest priority)

Policy values:

• delete (default): Delete AWS resource when K8s resource deleted
• retain: Keep AWS resource when K8s resource deleted

2. Secrets Management

Never store passwords in plain text:

```yaml

# Create secret

apiVersion: v1

kind: Secret

metadata:

name: db-credentials

namespace: production

type: Opaque

stringData:

password: "SecurePassword123!"

---

# Reference in RDS instance

apiVersion: rds.services.k8s.aws/v1alpha1

kind: DBInstance

metadata:

name: mydb

spec:

masterUserPassword:

name: db-credentials

key: password

```

Better: Use External Secrets Operator with AWS Secrets Manager for secret rotation.

3. IAM Least Privilege

Grant minimum required permissions per controller:

```json

{

"Version": "2012-10-17",

"Statement": [

{

"Effect": "Allow",

"Action": [

"s3:CreateBucket",

"s3:DeleteBucket",

"s3:GetBucket*",

"s3:PutBucket*"

],

"Resource": "arn:aws:s3:::prefix-*" // Limit to specific prefix

}

]

}

```

4. Resource Quotas

Limit resource creation per namespace:

```yaml

apiVersion: v1

kind: ResourceQuota

metadata:

name: aws-resource-quota

namespace: production

spec:

hard:

count/buckets.s3.services.k8s.aws: "10"

count/dbinstances.rds.services.k8s.aws: "5"

count/queues.sqs.services.k8s.aws: "20"

```

5. Monitoring

Enable Prometheus metrics for ACK controllers:

```yaml

apiVersion: monitoring.coreos.com/v1

kind: ServiceMonitor

metadata:

name: ack-s3-controller

namespace: ack-system

spec:

selector:

matchLabels:

app.kubernetes.io/name: ack-s3-controller

endpoints:

- port: metrics

interval: 30s

```

Key metrics:

• ack_resource_reconcile_duration_seconds: Reconciliation latency
• ack_resource_reconcile_errors_total: Error count
• ack_resource_synced: Resource sync status

6. High Availability

Run multiple controller replicas with leader election:

```yaml

# Helm values

replicaCount: 3

affinity:

podAntiAffinity:

requiredDuringSchedulingIgnoredDuringExecution:

- labelSelector:

matchLabels:

app.kubernetes.io/name: ack-s3-controller

topologyKey: kubernetes.io/hostname

```

ACK vs Terraform

| Aspect | ACK | Terraform |

|--------|-----|-----------|

| Cloud Support | AWS only | Multi-cloud (AWS, Azure, GCP, 300+ providers) |

| Paradigm | Kubernetes-native, continuous reconciliation | State-based, apply-driven |

| Maturity | Newer, many alpha APIs | Mature, stable, widely adopted |

| Service Coverage | 14+ GA, 12+ preview | 1000+ AWS resources |

| Drift Handling | Automatic continuous reconciliation | Detect on plan/apply, manual fix |

| Best For | EKS workloads, GitOps, K8s-centric teams | Multi-cloud, comprehensive coverage |

Selective Approach: Use ACK for application-tied resources (RDS, SQS, S3 for apps) and Terraform for foundational infrastructure (VPCs, IAM, networking).

ACK vs Crossplane

| Aspect | ACK | Crossplane |

|--------|-----|-----------|

| Scope | AWS only | Multi-cloud |

| Abstraction | None (1:1 AWS API) | Compositions (custom abstractions) |

| Maturity | AWS-sponsored | CNCF project |

| Best For | AWS-only, simple use cases | Multi-cloud, platform engineering |

Note: Crossplane provider-aws uses ACK code generation under the hood.

ACK vs AWS CDK

| Aspect | ACK | AWS CDK |

|--------|-----|---------|

| Language | YAML (K8s manifests) | TypeScript, Python, Java, C#, Go |

| Output | Direct AWS API calls | CloudFormation templates |

| Execution | Continuous (K8s reconciliation) | On-demand (CDK deploy) |

| Best For | K8s-native workflows, GitOps | Application developers, reusable patterns |

Pattern 1: Application Stack

Define entire application stack in one manifest:

```yaml

# database.yaml

apiVersion: rds.services.k8s.aws/v1alpha1

kind: DBInstance

metadata:

name: app-db

namespace: production

spec:

dbInstanceIdentifier: app-db

# ... database config

---

# queue.yaml

apiVersion: sqs.services.k8s.aws/v1alpha1

kind: Queue

metadata:

name: app-queue

namespace: production

spec:

queueName: app-queue

---

# Export values for app

apiVersion: services.k8s.aws/v1alpha1

kind: FieldExport

metadata:

name: export-db-endpoint

spec:

from:

resource:

group: rds.services.k8s.aws

kind: DBInstance

name: app-db

path: ".status.endpoint.address"

to:

kind: secret

name: app-config

key: DB_HOST

```

Pattern 2: Multi-Region Setup

Deploy same resources across regions:

```yaml

# us-east-1-bucket.yaml

apiVersion: s3.services.k8s.aws/v1alpha1

kind: Bucket

metadata:

name: multi-region-bucket-east

namespace: production

annotations:

services.k8s.aws/region: us-east-1

spec:

name: my-bucket-us-east-1-unique

---

# us-west-2-bucket.yaml

apiVersion: s3.services.k8s.aws/v1alpha1

kind: Bucket

metadata:

name: multi-region-bucket-west

namespace: production

annotations:

services.k8s.aws/region: us-west-2

spec:

name: my-bucket-us-west-2-unique

```

Pattern 3: Environment Promotion

Use Kustomize overlays for environment-specific configurations:

```

ack-resources/

├── base/

│ ├── kustomization.yaml

│ ├── rds-instance.yaml

│ └── s3-bucket.yaml

├── overlays/

│ ├── dev/

│ │ └── kustomization.yaml # db.t3.small, single-AZ

│ ├── staging/

│ │ └── kustomization.yaml # db.t3.medium, multi-AZ

│ └── production/

│ └── kustomization.yaml # db.r5.large, multi-AZ, encrypted

```

For Installation & Configuration

→ See references/controller-setup.md:

• Detailed installation for all 14+ controllers
• IAM policy templates per service
• IRSA configuration examples
• Controller configuration options
• Multi-controller setup
• Upgrade procedures

For Resource Definitions & Examples

→ See references/resource-definitions.md:

• Comprehensive examples for all AWS services
• S3, RDS, DynamoDB, SQS, SNS, Lambda, ECR, IAM, EC2, EKS
• Production configurations with encryption, backups, HA
• Cross-resource references
• Field exports to ConfigMaps/Secrets
• Adopting existing AWS resources

For GitOps Integration

→ See references/gitops-patterns.md:

• ArgoCD integration patterns
• Flux CD integration patterns
• Multi-cluster GitOps
• Environment promotion strategies
• Drift detection and reconciliation
• Disaster recovery procedures
• Secrets management with External Secrets Operator

Official Documentation:

• ACK Community: https://aws-controllers-k8s.github.io/community/
• GitHub: https://github.com/aws-controllers-k8s
• Service List: https://aws-controllers-k8s.github.io/community/docs/community/services/

Container Images:

• ECR Public Gallery: https://gallery.ecr.aws/aws-controllers-k8s

Learning:

• EKS Workshop: https://www.eksworkshop.com/docs/automation/controlplanes/ack/
• AWS Blog (ACK): https://aws.amazon.com/blogs/containers/ (filter: ACK)

Common Commands:

```bash

# List all ACK resources in namespace

kubectl get services.k8s.aws -n

# Describe ACK resource with events

kubectl describe bucket.s3.services.k8s.aws -n

# View controller logs

kubectl logs -n ack-system deployment/ack-s3-controller -f

# Export resource to YAML

kubectl get bucket.s3.services.k8s.aws -n -o yaml

# Validate resource (dry-run)

kubectl apply --dry-run=server -f bucket.yaml

```

ACK brings AWS resource management into the Kubernetes ecosystem, enabling true infrastructure-as-code with GitOps workflows. It's ideal for teams running EKS workloads that need tight integration between applications and AWS services, with the flexibility to use familiar Kubernetes tools and processes.

Best practice: Use ACK selectively for application-tied resources (databases, queues, storage for apps) while using Terraform for foundational infrastructure (VPCs, IAM roles, networking). This hybrid approach combines the strengths of both tools.