A First Look at Service Mesh

This tech talk is about the big-picture ideas and mental models only; specific configuration details and deployment steps are out of scope.

Introduction

The discussion revolves around three points:

1. The limitations of microservices
2. What Service Mesh does and how it is implemented
3. Serverless platforms built on top of Service Mesh today

The stacks we know best—Java, Spring Cloud, Spring Cloud Alibaba, and the like—belong to the traditional microservice architecture. This design suffers from tight SDK coupling: every update to the underlying SDK version can easily destabilize services at the business layer. When the SDK is tightly coupled to the code environment, many constraints follow—you cannot fully leverage the different languages and their ecosystems in the domains where they excel. For example: the nimble Go and Python languages with their goroutines, web scraping, and neural network libraries; or the rigorous Rust language with no garbage collection overhead, better suited to enterprise middleware scenarios that demand high efficiency and a low memory footprint. Every language has its own strengths and ecosystem, and looking further ahead, it is a clear trend that large industry leaders will spread their business deployments across different languages as they iterate.

The traditional microservice stack:

Moreover, the cross-OS portability the JVM once prided itself on has been superseded by container technologies and ecosystems such as K8S, Docker, Containerd, CRI-O, and Kata. Advanced features like garbage collection, dynamic proxies, agents, and ZGC have also been borrowed by other languages. What props up Java's position today is the ecosystem accumulated over the years (open source communities from Apache, Spring, Alibaba, Huawei, and others keep injecting new middleware—the well-known Kafka, Hadoop, HBase, RocketMQ, Flink, Seata, and so on—plus whatever comes next to keep the ecosystem alive). But over time, in the long run (20–30 years or more—a subjective prediction, take it for what it's worth), these advantages may be claimed by rising newcomers, and the next generation of microservice architecture is bound to iterate on the strengths of many languages.

A personal take: Java is now owned by Oracle, and Oracle is not a technologically aggressive company, so it may well constrain Java's development going forward. There is an open source edition, but open source efforts are less concentrated than corporate ones. Meanwhile Google's Go language has risen rapidly and already taken a slice of Java's market—likely one of the reasons companies like ByteDance, Bilibili, Tencent, Baidu, JD.com, and Xiaomi have adopted it.

Service Mesh

Overview

Back to our topic: Service Mesh. Its dominant implementation approach today is the Sidecar pattern. In K8S the Pod is the basic deployment unit; the Sidecar approach deploys a Proxy container inside the same Pod as the business Server container (Pod container injection), and the Proxy container intercepts all of the Server container's traffic.

The big-picture architecture of Istio's Service Mesh implementation:

citadel: the core security component responsible for identity authentication and certificate management.

galley: the configuration management component—it validates the format and content of configuration files and feeds that configuration to pilot and mixer.

pilot: the control hub, covering service discovery plus rule translation and distribution.

proxy: implemented by the C++-based Envoy together with Pilot-agent, providing dynamic service discovery, load balancing, TLS, circuit breaking, health checks, traffic splitting, canary rollouts, and more. It also produces telemetry data, giving microservices observability.

Ingressgateway: the gateway at the entrance—traffic from outside the mesh reaches services inside the mesh through this gateway.

With the Proxy container, business code is fully decoupled from its SDK, and you gain a long list of capabilities: traffic mirroring, canary releases, service registration and discovery, remote invocation, circuit breaking, degradation, distributed tracing, control plane management, and encrypted in-cluster transport (distributed transactions have no dedicated solution here, since transactions strictly speaking do not belong to the service governance layer). All of this is completely transparent to programmers: developers do not need to know the deployment details and can code almost exactly as they would for a single machine, with no language restrictions and no dependency on any microservice SDK. Companies are then free to develop different functional modules in different languages (which may bring higher code-management and communication costs, and a broader spread of languages the team must master).

The Official Demo

The official demo, the Book-Info microservice app, can be deployed to a K8S environment with one Helm command so you can experience these features firsthand (Book-Info architecture diagram):

The Bookinfo microservices are deployed with Istio sidecars and are written in 4 different languages, and the Java-based Reviews service ships 3 different versions of the application—an excellent showcase of Istio's capabilities in multi-version control, traffic management, traffic migration, and integration. Deploy it yourself and you will get a very direct feel for how powerful Service Mesh is and how flexible the configuration can be (all changes take effect within seconds, with imperceptible traffic migration).

The Most Common Implementations

Istio

Istio is an open source Service Mesh framework initiated by Google, IBM, and Lyft. The project launched in 2017 and released version 1.0 in July 2018; as of 2022-12-06 the current version is 1.16.0. Istio is the flagship Service Mesh implementation today. If the Sidecar is the data plane of the whole Service Mesh, Istio's main improvements are on the control plane. Istio uses Envoy as the sidecar, and the entire control plane is written in Golang, bringing a significant performance boost. Istio is first and foremost a service mesh, but it is more than that: on top of classic service meshes like Linkerd and Envoy, Istio provides a complete solution that delivers behavioral insight and operational control over the entire mesh, meeting the diverse needs of microservice applications.

Istio

Linkerd2

Linkerd is a high-performance network proxy open-sourced by Buoyant in 2016 and the industry's first Service Mesh framework, built to address the problems services face when communicating in distributed environments—unreliable and insecure networks, latency, packet loss, and so on. Linkerd was originally written in Scala; its successor, Linkerd2, was rewritten in Go and Rust.

The world’s lightest, fastest service mesh.

Conduit

Conduit, released in December 2017, is another open source project sponsored by Buoyant after Linkerd, positioned as a standalone Kubernetes-focused edition of Linkerd. Conduit aims to radically simplify the complexity of running a service mesh on Kubernetes and improve the user experience, rather than optimizing for every platform the way Linkerd did. Its main goals are to be lightweight, high-performance, secure, and very easy to understand and use. Like Linkerd and Istio, Conduit consists of a data plane and a control plane: the data plane is written in Rust, which lets Conduit run on very little memory, while the control plane is written in Go.

Buoyant. All of the service mesh. None of the service mess.

Architecture Comparison

Recap

Kubernetes has already solved most operational problems—container deployment, high availability, multiple replicas, container migration, elastic scaling, rolling updates, image version management, liveness probing, compute resource allocation, node monitoring—with an excellent automated operations machinery. But K8S does not provide monitoring and traffic management for application-layer Service containers, nor inter-container invocation, monitoring, registration, configuration, or traffic governance. Service Mesh technology emerged to fill that gap and extend K8S in this area, so that microservices no longer worry about specific configuration or environment details, focus on implementing business features, and are freed from language constraints.

Serverless

Overview

As the microservice philosophy took hold, more and more companies gradually moved their applications from monoliths to microservice architectures, and the arrival of container technology accelerated that transition. Containers effectively solved the many runtime-environment discrepancies between services, but as the number of services grew, container orchestration and management became the new problem. Kubernetes then answered the challenge of orchestrating and deploying microservice containers at scale, making the industry realize that PaaS could actually land in practice. And as the container count in a microservice system kept climbing, service governance and traffic management became unavoidable—hence Istio, which, through its strategy of separating the network proxy from control, enables effective and sensible management of service control policies.

At this point the architectural evolution seems to have reached a lovely place:

• Microservices: solve the problem of bloated, over-coupled applications.
• Containers: solve runtime-environment discrepancies and deployment.
• Kubernetes: solves orchestration and "aggregate" deployment of huge numbers of micro-containers.
• Istio: solves the traffic, release, and governance problems services face once they go live.

At first glance, building a container cloud now has a complete chain and solution—everything looks so "perfect."

But step back and dig deeper: does service interaction under the microservice model still have problems? First, whether over HTTP or RPC, it is fundamentally one service remotely calling another; within application code, services cannot be truly transparent to each other. The consequence: no matter how "finely" the business is decomposed into microservices, the business units still cannot run and evolve independently—they are never fully decoupled—and when branching into different development domains, you cannot pick the most suitable implementation for each. So we want to standardize development environments through "templates" and "configuration," and provide an "event" mechanism to reduce the coupling of service-to-service interaction to a minimum. Second, there is the problem of dynamic scaling in production: elasticity in today's Kubernetes environments requires customers to collect monitoring data and implement scaling manually, but we would rather have production services scale automatically and intelligently.

Finally there is service standardization. We want the internal service model to be standard—quickly replicable, quickly buildable; service communication to be standard—standard protocols, standard formats; and the runtime environment to be standard—fast to deploy, fast to migrate.

Knative arrived to solve exactly these problems: direct remote invocation, automatic online scaling, version snapshots, and a whole series of standardization issues.

Knative

Knative is the weapon Google unveiled for the Serverless world in 2018. It brings serverless architecture to any public or private cloud, letting users adopt serverless programming techniques. The main participating companies are Google, Pivotal, IBM, and Red Hat; it was announced on July 24, 2018 and is still evolving rapidly. Unlike Kubernetes, which must keep at least one Pod instance running to serve an application, Knative can scale down to zero (cold/warm start technology). Only when a client request hits your application does Knative actually spin up the application's Pods. This can save a great deal of the money spent keeping applications running year-round. For instance, infrequently accessed, low-traffic functional modules are good candidates for cold-started applications to save memory and compute (container startup time can be optimized to reduce cold-start latency—down to milliseconds at best; in China, Alibaba Cloud is probably the leader here).

Official sites:

Documentation at knative.dev/docs and github.com/knative

Knative's goal is to support the entire development lifecycle on top of Kubernetes. Concretely: first, it lets you as a developer write code in whatever language and style you want; second, it helps you build and package the application; and finally it helps you run and scale it. Knative is built around three core components—Build, Serving, and Eventing—and it is these three that drive the great Serverless ship of Knative forward.

Build

• In-cluster builds: builds run inside Kubernetes, compiling functions and shipping them quickly, tightly integrated with the whole Kubernetes ecosystem.
• Standardization: it aims to be a general, standardized build component usable as part of larger systems; deployment scripts are standardized and structured, making services easier to migrate and deploy.

Serving

• Fast deployment: rapidly deploy serverless containers.
• Scale on demand: automatic scaling up and down, including scale-to-zero.
• Routing policies: routing and network programming built on Istio.
• Version snapshots: deployment snapshots (production container snapshots that can be kept long-term and restored to any snapshot version at will).

Eventing

Source, Channel, Subscription

The eventing system makes producing and consuming events easy: it abstracts event sources and lets operators use the messaging layer of their choice. The heart of Serverless is event-based triggering—when something happens, a particular function fires. The event concept decouples functions from their callers: a deployed function does not care who will call it, and an event source does not care who will handle it. In short, our code no longer names the concrete callee Service; it only concerns itself with emitting events and handling events (anyone familiar with MQ queues will feel at home here, though Eventing's system is a bit more complete, with several event-processing patterns). When our Service makes a remote RPC or HTTP call, it needs neither knowledge of a concrete service instance nor a subscription to a registry—just emit the event source, and Eventing handles all of the returned data. This achieves true transparency between services and solves the coupling problem. (As the saying goes, coupling and decoupling are often just one indirection layer apart.)

Overall Strengths

Convenience: Knative uses Kubernetes as its underlying framework, so any Kubernetes cluster—online or on-premises, a managed cloud service or self-built—can become a Serverless platform quickly by installing the Knative add-on.

Standardization: together with the CNCF, Knative standardized all events into CloudEvents, giving events cross-platform portability while decoupling functions from their concrete callers.

Inter-service decoupling: with Knative, applications are no longer hard-bound to underlying dependent services and can interoperate across clouds.

A mature ecosystem: Knative is built on the Kubernetes stack and integrates tightly with its ecosystem.

Autoscaling: it monitors application requests and scales automatically, and—thanks to Istio (or ambassador, gloo, etc.)—natively supports blue-green deployments and rollbacks, streamlining the release process.

Application monitoring: log collection, search, and analysis are supported, along with metrics dashboards and call-chain tracing.

Snapshot deployment: every release is recorded as a snapshot that can be retained long-term, and you can seamlessly restore to any snapshot version at any point in time.

Summary

Serverless has become the new technology hotspot, evolved on top of traditional container technology and the Service Mesh. Serverless cloud functions free developers from server deployment and operations: develop just the core business logic or function and it is live, automatically gaining distributed disaster tolerance and load-based autoscaling. On public clouds you are billed by actual invocation count, execution duration, and compute consumed—no wasted resources, and better cost savings for the business.

Serverless broadly falls into two categories: BaaS (Backend as a Service) and FaaS (Function as a Service).

BaaS: "serverless" was first used to describe applications that significantly or fully incorporate third-party, cloud-hosted applications and services to manage server-side logic and state. These are typically "rich client" applications—single-page web apps or mobile apps—that use the vast ecosystem of cloud-accessible databases (e.g. Parse, Firebase), authentication services (e.g. Auth0, AWS Cognito), and so on. These kinds of services were previously described as "Backend as a Service"—think of the containers and service middleware we commonly use on the backend: Spring Boot, Tomcat, Dubbo, WebLogic, Gin, Flask.

FaaS: serverless can also mean applications where server-side logic is still written by the application developer. Unlike traditional architectures, it runs in stateless compute containers that are event-triggered—you implement just a function without worrying about the environment—ephemeral (possibly lasting a single invocation), and fully managed by a third party. One way to think of it is "functions as a service," or FaaS. Abroad, AWS Lambda is one of the most popular FaaS platforms; in China, Alibaba Cloud's Function Compute (FC) offers FaaS today.

In short, serverless architecture did not appear to replace traditional applications. Starting from its highly flexible usage model and event-driven nature, it helps us reduce deployment effort, improve scalability, and shrink the burden of maintaining the infrastructure behind the code—giving us more options to pick the deployment approach that best fits the business.

Further Questions

1. How will deployment architectures evolve? How will the 3H traits—high availability, high performance, high concurrency—spawn new technology stacks? How does an enterprise private cloud reach 6 to 12 nines of availability? (6 nines: (1-99.9999%)*365*24*60*60 = 31 seconds—no more than 31 seconds of downtime per year. But 12 nines is quite a challenge; from a cost perspective it is unnecessary and perhaps impossible: annual downtime must not exceed (1-99.9999999999%)*365*24*60*60 = 0.00003 seconds.) Combined with multi-region active-active architectures, how many new middleware technologies might that spawn?

2. Does this architecture really have only upsides and no downsides? Are its overhead, footprint, and controllability suitable for every company? Is it sound for a company to jump straight to Serverless without considering its business scenarios?

3. Have you noticed that container ecosystems in software and hyper-converged infrastructure in hardware are strikingly similar? Both aim for elasticity, migration, cost savings, flexibility, agility, and greater reliability.

Corrections and additions are welcome—feel free to leave a comment.