Kubernetes Networking Unlocked: Why You Need Both a CNI and a Service Mesh

Are you curious why your Kubernetes setup still needs Istio if you’ve already installed Calico or Flannel? This guide demystifies the roles of CNIs and Service mesh, showing how they work together to deliver seamless connectivity, robust security, and advanced traffic management in your cluster.

When setting up a Kubernetes cluster, you likely installed a Container Network Interface (CNI) plugin such as Calico or Flannel. You may also be advised to install a Service Mesh like Istio. This article explains why a Service Mesh can benefit even if you already have a CNI. We’ll provide an overview of Kubernetes networking to help distinguish between a CNI solution (e.g., Calico) and a Service Mesh (e.g., Istio).

Kubernetes Network Layers

Kubernetes networking can be thought of as four distinct layers or networks:

1. Host Network

This is the underlying network connecting all hosts (nodes) in your cluster. IP addresses here are typically allocated within your Virtual Private Cloud (VPC).

2. Cluster Network

All Pods communicate with each other seamlessly in this network without Network Address Translation (NAT). In effect, Pods behave as if they’re virtual machines on a shared subnet.

You can find the subnet CIDR with:

kubectl cluster-info dump | grep -m 1 cluster-cidr

This might return something like:

"--cluster-cidr=192.168.0.0/16"

3. Service Network

Because Pods are ephemeral, Kubernetes uses Services to provide a stable endpoint. Each Service gets an IP address from the Service Network.

To find this IP range:

kubectl cluster-info dump | grep -m 1 service-cluster-ip-range

Example output:

"--service-cluster-ip-range=10.96.0.0/12",

Ensure the Service Network CIDR does not overlap with the Cluster Network CIDR. Both are defined when the cluster is created and cannot be changed later.

4. Container Network

Although most Pods have only one container, they can contain multiple containers (e.g., sidecars). All containers within a single Pod share the same network stack and IP address. They communicate internally via localhost and must use different ports to avoid conflicts.

Container Network Interface (CNI)

Kubernetes requires that all Pods communicate with each other over the Cluster Network. As new Pods start and new nodes join the cluster, the networking must be automatically configured. This is where the CNI comes in.

When a Pod is scheduled on a node, the node’s kubelet process calls the CNI plugin to:

• Assign an IP address to the Pod.
• Configure network interfaces to ensure Pod connectivity.

Several CNI plugins (Flannel, Calico, etc.) offer different implementations and security features. However, a CNI plugin is mandatory for Kubernetes to function regardless of your choice.

Service Mesh

A Service Mesh adds another layer to the Cluster Network. Instead of allowing open, unencrypted communication between all Pods, it controls and secures traffic between specific services.

A typical Service Mesh deploys a “sidecar” proxy container alongside each main container in the Pod. This proxy handles the Pod’s inbound and outbound network traffic over localhost. You configure the proxies using Kubernetes manifests, which a control plane component translates into rules and distributes to the proxies.

Benefits of a Service Mesh

1. Security: Implements routing rules, mutual TLS (mTLS), and encryption.
2. Load Balancing: Manages traffic distribution across multiple service instances.
3. Monitoring: Collects metrics and logs for network traffic.
4. Separation of Concerns: Abstracts network policy and security away from the application code.
5. Reliability: Retries requests automatically when failures occur.

Do You Need Both a CNI and a Service Mesh?

In short, yes:

• CNI is required for Pods to communicate within the cluster network. It ensures each Pod gets an IP and traffic flows at the network level.
• Service Mesh is optional but highly beneficial. It secures communications, provides observability, handles mTLS, and allows traffic management — features not provided out of the box by the CNI.

The Need for an API Gateway

You might wonder if you still need an API Gateway once a Service Mesh is installed. While a Service Mesh can provide some API Gateway-like features, it’s not explicitly designed for external (north/south) traffic management.

An API Gateway typically handles:

• Ingress routing (including load balancing)
• Authentication and authorization
• Circuit breaking
• Service discovery
• Traffic control for external clients

So, the API Gateway often remains a separate but complementary component.

Overlaps in Networking Solutions

CNI, Service Mesh, and API Gateway can overlap in functionality. However, each serves a distinct primary purpose. While there’s no harm in using all three, complexity and costs may increase.

Further Developments and Confusion

This area is rapidly evolving. For instance:

• Istio (a Service Mesh) now offers Istio CNI as a plugin.
• Calico (a CNI) offers its own Pod-to-Pod security policies.
• eBPF-based solutions are becoming more efficient replacements for traditional proxy or iptables mechanisms.

Popular Service Mesh Options

Several Service Mesh technologies are available:

• Istio
• Linkerd
• Cilium (also provides eBPF-based features)
• Consul Connect
• Cloud-Specific Solutions (e.g., GCP Anthos, AWS App Mesh, Azure Open Service Mesh)

Many are adding enterprise features or charging for advanced capabilities. Istio remains free and open source but can be intricate to set up and manage.

Summary

In this article, we covered:

• The four layers of Kubernetes networking
• The role of the Container Network Interface (CNI)
• How a Service Mesh adds security, observability, and traffic management
• Why you need a CNI even if you install a Service Mesh
• How an API Gateway fits into the mix
• Overlapping functionalities among CNI, Service Mesh, and API Gateway
• A glance at different Service Mesh technologies

Stay tuned for an upcoming article on installing and configuring Istio.