Skip to main content

Traffic Mirroring with Istio

· 5 min read
tanzhuo

Once Istio is deployed as a Service Mesh in a k8s cluster, the Envoy proxy that forwards traffic can also mirror it — a few extra lines in a config file are all it takes. Mirrored responses never reach the client, so it's completely transparent to your application code, and used well it can deliver a lot of value in production.

What Is Traffic Mirroring​

Traffic mirroring is also known as shadow traffic: while forwarding a client request to the real service, Envoy sends a copy of it to a designated mirror service. Two things matter here:

  1. Mirrored requests are fire-and-forget. Envoy never waits for the mirror service's response, let alone returns it to the client. The mirror service can be slow, throw errors, or even go down without affecting the main path at all.

  2. Mirroring happens before traffic reaches the application container — it's handled by the sidecar proxy. Your application code needs zero changes; you just deploy a separate instance to receive the mirrored traffic.

Official docs: Mirroring

Typical Use Cases​

  1. Testing environments: a test build can run against real production traffic without touching the production critical path. A staging environment, for instance, gets much better real-time validation, fewer surprises after release, and a development team with far more confidence going live 😂 — releases become almost boringly predictable, no more late nights pushing hotfixes.

  2. Data collection: capture request data in parallel and feed it to other containers for risk analysis or logging, building out user profiles along the way.

  3. Performance and compatibility validation: a new service version, a refactored API, or an implementation on a new framework or dependency can "shadow-run" against mirrored traffic for a while. Compare logs and metrics on both sides, confirm the behavior matches, and only then cut over for real.

  4. Reproducing bugs: those rare, intermittent bad requests in production are nearly impossible to reconstruct in a test environment. Mirror the traffic to an instance with verbose logging or debug flags enabled and you get a first-hand capture of the scene.

Configuration​

In Istio, traffic mirroring is declared via the mirror field of a VirtualService. Suppose the httpbin service has two versions, v1 and v2, and you want traffic hitting v1 to also be mirrored to v2:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: httpbin
spec:
  hosts:
    - httpbin
  http:
    - route:
        # Real traffic: all forwarded to v1, response returned to the client
        - destination:
            host: httpbin
            subset: v1
          weight: 100
      # Mirrored traffic: a copy is sent to v2, response discarded
      mirror:
        host: httpbin
        subset: v2
      # Sampling percentage for mirroring, defaults to 100% if omitted
      mirrorPercentage:
        value: 100.0

What the fields mean:

  1. route.destination is the primary route. It decides where the real request goes, and its response is what the client receives.

  2. mirror specifies the mirror target, also as a host + subset pair. The subset must be defined in a DestinationRule beforehand.

  3. mirrorPercentage controls the sampling rate. When the mirror target has less capacity than production, ramping up gradually — 1%, then 10% — is far safer than going straight to 100%.

Apply the config with kubectl apply -f and it takes effect immediately, no application Pod restarts required — one of the perks of the sidecar model.

Pitfalls and Caveats​

  1. Mirrored traffic gets a -shadow suffix appended to its Host header — httpbin:8000 becomes httpbin-shadow:8000. If the mirror service validates or routes on the Host header, handle this difference up front. On the flip side, the suffix is a handy way to tell real traffic from shadow traffic in your logs.

  2. What gets mirrored is the request, not idempotency. A mirrored write endpoint will execute again in the shadow environment, and if the mirror service talks to the same database or the same downstream, you can end up with duplicate writes or duplicate third-party calls. Either point the mirror service at an isolated data source, or intercept write operations on the mirror side.

  3. Mirroring amplifies egress bandwidth and Envoy overhead — especially noticeable for services with large request bodies or high QPS. Validate with a small mirrorPercentage first, then scale up.

  4. Errors in the mirror service never reach the client. That's a strength, but also a blind spot — when the shadow environment goes down, no user notices. Give it its own monitoring and alerting, or the "shadow run" may have silently stopped long ago without anyone finding out.

Wrapping Up​

Traffic mirroring turns "validate with production traffic" — normally a high-risk endeavor — into a low-cost operation: one VirtualService config, zero application changes, zero impact on the main path. Staging validation, risk analysis, and bug reproduction all benefit. The main things to watch are side effects of write operations, the -shadow Host header suffix, and monitoring on the mirror side itself. Handle those three well, and it's a tool worth keeping within arm's reach.

COMMENTS