Managing Kubernetes platform components with Terragrunt
Introduction
A Kubernetes platform brings together tools with shared configuration and deployment dependencies. Terragrunt can keep those settings in one place and make the order of operations explicit.
This walkthrough uses the Kubernetes Platform Terragrunt Configuration repository to connect core platform tools, networking, and monitoring.
The Big Picture: Architecture
Below is a high-level architecture that illustrates the essential modules and their relationships.
Key Highlights:
- Core Platform: Manages vital capabilities like node provisioning (Karpenter), DNS integration (External DNS), certificate management (Cert Manager), and secret management (External Secrets).
- Networking & Service Mesh: Tightly integrates Istio, Kong Gateway, and Jaeger for routing and distributed tracing.
- Observability: Implements Loki for logs and Kubecost for cost insights.
- Platform Tools: Offers GitOps (ArgoCD), Terraform collaboration (Atlantis), workflow management (Airflow), and advanced secret management (Vault).
Why Terragrunt?
Terragrunt extends Terraform’s functionality by introducing:
- DRY (Don’t Repeat Yourself): Centralize standard configurations (like common.hcl) and reduce duplication across multiple modules.
- Automated Dependencies: Terragrunt’s dependency management ensures proper deployment order.
- Consistent Environment Management: Standardize your environment configurations (like platform_vars.yaml) for dev, stage, and production.
Repository Structure
.
├── core-platform/ # Core platform components
│ ├── cert-manager # Certificate management
│ ├── external-dns # DNS automation
│ ├── external-secrets # Secrets management
│ └── karpenter # Kubernetes node provisioning
├── service-mesh/ # Service mesh components
│ ├── istio # Service mesh control plane
│ ├── jeager # Distributed tracing
│ └── kong-gw # API gateway
├── observability/ # Monitoring and observability
│ ├── kubecost # Cost monitoring
│ └── loki-stack # Log aggregation
├── platform-tools/ # Platform utilities
│ ├── airflow # Workflow automation
│ ├── argocd # GitOps deployment
│ ├── atlantis # Terraform automation
│ └── vault # Secrets management
└── ci-cd-templates/ # Reusable CI/CD workflows
This modular layout allows teams to maintain, iterate, and scale independently on each platform component.
Prerequisites
To get started, ensure you have the following tools properly installed and configured:
- Terragrunt >= v0.60.0
- Terraform >= v1.5.0
- AWS CLI configured
- kubectl configured
- Helm v3.x
Getting Started
1. Configure Common Settings
In common.hcl, you’ll find shared values, like the EKS cluster name and AWS region:
locals {
platform_vars = yamldecode(file(("platform_vars.yaml")))
eks_cluster_name = local.platform_vars.common.eks_cluster_name
environment = get_env("ENV", "dev")
aws_region = local.platform_vars.common.aws_region
tags = local.platform_vars.common.common_tags
}
2. Define Platform Variables
Inside platform_vars.yaml, you set environment-specific details like domain name, AWS region, and tagging:
aws_region: "us-east-2"
eks_cluster_name: "dev-eks-cluster"
environment: "dev"
domain_name: "cloudon.work"
common_tags:
Environment: "dev"
Owner: "cloudon"
ManagedBy: "Terragrunt"
Team: "platform"
ClusterName: "dev-eks-cluster"
...
3. Deployment Order
1. Core Platform
terragrunt run-all apply --terragrunt-working-dir core-platform
2. Service Mesh & Networking
terragrunt run-all apply --terragrunt-working-dir service-mesh
3. Observability
terragrunt run-all apply --terragrunt-working-dir observability
4. Platform Tools
terragrunt run-all apply --terragrunt-working-dir platform-tools
Day 2 Operations
Updating a Component
cd component-name
terragrunt apply
This allows you to quickly apply changes for a single module without affecting the rest of the platform.
Backing Up Terraform State
terragrunt state pull > backup.tfstate
Always keep regular state backups to ensure a smooth recovery if something goes wrong.
Security Considerations
- IRSA: Ensure each Kubernetes service account has the minimum privileges needed.
- Network Policies: Use Istio and Kong to manage traffic flow and encrypt communication.
- Vault & External Secrets: Keep sensitive data out of your Git repository and manage secrets securely.
Monitoring & Observability
Leverage best-in-class tools right out of the box:
- Loki Stack for log aggregation
- Jaeger for distributed tracing
- Kubecost for granular cost monitoring
- Grafana dashboards for custom insights
CI/CD Integration
The repository’s ci-cd-templates folder includes reusable workflows for:
- Docker Builds
- Terragrunt Plan/Apply
- Environment Variable Management
Additionally, you’ll find test coverage action templates for multiple languages like Java, .NET, Node.js, and Python.
Ready to Dive In?
This guide scratches what’s possible with a well-structured Terragrunt + Kubernetes setup.
For detailed configurations, advanced workflows, and troubleshooting tips, check out the k8s-platform-tools GitHub repository.
Contributing
We welcome contributions! Fork the repo, make changes, and open a Pull Request to get your updates reviewed and merged.
License
This project is available under the MIT License. Details can be found in the repository’s LICENSE file.
Final Thoughts
The configuration gives each platform component its own module while sharing environment settings. Start with a small set of components and check their dependencies before expanding the platform.
Questions and proposed improvements are welcome in the repository’s issues and pull requests.
CloudOn engineering
Practical notes on AWS, Google Cloud, Kubernetes, and infrastructure code from the CloudOn engineering blog.