Kubernetes Terraform DevOps Platform Engineering

Building a Kubernetes platform with Terraform modules

CO
CloudOn
|
Read as Markdown JSON
In this article
  1. Why Use These Modules
  2. Core Modules at a Glance
  3. Architecture Overview
  4. Getting Started
  5. Security Built In
  6. Observability and Monitoring
  7. Best Practices
  8. Contributing and Support
  9. Ready to Explore

A Kubernetes platform needs more than a cluster. Autoscaling, networking, secrets, monitoring, and deployments each bring their own settings and dependencies. This collection groups those configurations into Terraform modules that you can use independently.

Why Use These Modules

Reusable configuration — Each module handles the installation and configuration of one platform component. Start with the modules you need and review their defaults for your environment.

Consistency — Modules enforce naming conventions, label standards, and configuration patterns across every environment, reducing drift between development, staging, and production clusters.

Modularity — Compose only the components you need. Each module is independently usable; there is no requirement to adopt the entire collection.

Security First — RBAC, TLS, network policies, and secret management are built into the modules by default rather than treated as afterthoughts.

Core Modules at a Glance

Core Infrastructure

ModulePurpose
KarpenterNode autoprovisioning based on pending pod scheduling constraints
External DNSAutomatic DNS record management from Kubernetes Service and Ingress resources
Cert ManagerAutomated TLS certificate issuance and renewal via Let’s Encrypt or private CAs
External SecretsSynchronize secrets from AWS Secrets Manager, GCP Secret Manager, or HashiCorp Vault into Kubernetes Secrets

Service Mesh and Networking

ModulePurpose
IstiomTLS between services, traffic management, and fine-grained authorization policies
KongAPI gateway with rate limiting, authentication plugins, and developer portal
JaegerDistributed tracing with OpenTelemetry-compatible instrumentation

Observability

ModulePurpose
LokiLog aggregation and querying with Grafana integration
KubecostReal-time Kubernetes cost allocation and showback
ArgoCDGitOps continuous delivery with drift detection and automated sync

Platform Tools

ModulePurpose
AirflowWorkflow orchestration for data pipelines and scheduled jobs
AtlantisTerraform pull request automation with plan/apply workflows
VaultSecrets management, dynamic credentials, and PKI

Architecture Overview

The modules are organized into five logical layers that map to the platform stack from bottom to top:

LayerComponents
Infrastructure ManagementKarpenter, External DNS, Cert Manager, External Secrets
Service Mesh and SecurityIstio, Kong, Jaeger
ObservabilityLoki, Kubecost, Prometheus, Grafana
Automation and CI/CDArgoCD, Atlantis, Airflow
External AccessIngress controllers, API gateways, load balancers

Each layer depends only on layers below it, making it straightforward to add or replace components at any level without affecting the rest of the stack.

Getting Started

1. Clone the Repository

git clone https://github.com/cloudon-one/k8s-platform-modules.git
cd k8s-platform-modules

2. Pick Your Modules

Reference only the modules you need from your root Terraform configuration:

module "cert_manager" {
  source  = "./modules/cert-manager"
  version = "~> 1.0"

  cluster_name      = var.cluster_name
  lets_encrypt_email = var.lets_encrypt_email
}

module "external_secrets" {
  source = "./modules/external-secrets"

  cluster_name    = var.cluster_name
  aws_region      = var.aws_region
  service_account = "external-secrets-sa"
}

3. Configure Your Modules

Each module exposes a focused set of input variables. For example, to configure the Loki module:

module "loki" {
  source = "./modules/loki"

  cluster_name       = var.cluster_name
  storage_bucket_name = var.loki_bucket
  retention_days     = 30
  replicas           = 2
}

4. Deploy

terraform init
terraform plan -out=tfplan
terraform apply tfplan

Security Built In

RBAC — Every module creates the minimum set of ClusterRoles and RoleBindings required for the component to function. Service accounts are namespaced and never granted cluster-admin.

TLS — Modules that expose endpoints configure TLS by default using Cert Manager. Istio enforces mTLS in STRICT mode across all mesh-enrolled namespaces.

Network Policies — Modules ship with default NetworkPolicy resources that restrict ingress and egress to only the traffic paths each component legitimately requires.

Secret Management — Sensitive values (credentials, tokens, certificates) are never stored in Terraform state in plaintext. Modules integrate with External Secrets or Vault to pull secrets at runtime.

Observability and Monitoring

The observability modules provide a cohesive monitoring stack out of the box:

  • Prometheus scrapes metrics from all platform components using pre-built ServiceMonitor and PodMonitor resources.
  • Grafana ships with pre-configured dashboards for each installed module, giving immediate visibility into cluster health, cost allocation, and application performance.
  • Alarms — Key SLO-aligned alerting rules are included for each component, covering availability, latency, and error rate thresholds.

Best Practices

  • Pin each module to a specific version tag rather than using a branch reference to ensure reproducible deployments.
  • Use separate Terraform workspaces or state backends for each environment (development, staging, production).
  • Run terraform plan in CI on every pull request and require approval before terraform apply reaches production — the Atlantis module automates this workflow.
  • Review Kubecost dashboards weekly to identify namespace-level cost anomalies early.
  • Rotate secrets and certificates on a defined schedule; use Vault’s dynamic secrets capabilities wherever possible to reduce the blast radius of a compromised credential.

Contributing and Support

Contributions are welcome. To propose a new module or an improvement to an existing one:

  1. Open an issue describing the use case and the component you want to add or change.
  2. Fork the repository and implement the change following the existing module structure.
  3. Submit a pull request with a description of what the module does, the variables it exposes, and a working example in the examples/ directory.

For questions and discussion, open a GitHub issue on the repository.

Ready to Explore

Start with the module that addresses your next platform task. Read its inputs and example configuration, then test it in a development cluster before using it elsewhere.

CO

CloudOn engineering

Practical notes on AWS, Google Cloud, Kubernetes, and infrastructure code from the CloudOn engineering blog.