The gRPC Study Guide

01What gRPC actually is

gRPC is a high-performance, contract-first Remote Procedure Call framework. The mental model is the key starting point: instead of thinking in terms of resources and verbs (REST), you think in terms of methods you call on a remote object. You define a service interface once, and the client calls stub.GetUser(req) as if it were a local function. The framework handles serialization, transport, and the network round-trip.

Three pillars hold it together, and you should be able to recite them:

• Protocol Buffers — the Interface Definition Language (IDL) and the binary serialization format. The contract is the source of truth; client and server code are generated from it.
• HTTP/2 — the transport. Multiplexing, streaming, header compression, and binary framing are not optional extras; gRPC is designed around them.
• Generated code + a runtime — protoc plus a language plugin emits type-safe stubs; the runtime handles channels, flow control, deadlines, and retries.

gRPC vs REST — the honest comparison

02Protocol Buffers

Protobuf is two things at once: a schema language for describing messages and services, and a wire format for serializing them efficiently. Modern gRPC uses proto3.

protobufsyntax = "proto3";
package user.v1;

service UserService {
  // Unary: one request, one response
  rpc GetUser(GetUserRequest) returns (User);

  // Server streaming: one request, a stream of responses
  rpc ListUsers(ListUsersRequest) returns (stream User);

  // Bidirectional streaming
  rpc Chat(stream ChatMessage) returns (stream ChatMessage);
}

message User {
  string id   = 1;   // field number, NOT a value
  string name = 2;
  repeated string roles = 3;
  int64  created_at_unix = 4;
}

The wire format, briefly

Each field is encoded as a tag (field number + wire type) followed by the value. Integers use varint encoding (small numbers take fewer bytes). Strings/bytes/messages are length-delimited. There are no field names, no whitespace, no quotes — that's why protobuf is dramatically smaller and faster to parse than JSON.

proto3 semantics you must know

• Defaults are invisible. A scalar set to its zero value (0, "", false) is not serialized. You cannot, by default, distinguish "unset" from "set to zero."
• Use optional (re-introduced in proto3) when you genuinely need presence semantics — it adds a hidden has-bit.
• Backwards compatibility: add fields freely; never change types or numbers; treat unknown fields as pass-through.

03HTTP/2 — the part everyone skips

You cannot reason about gRPC in production without understanding what's happening at the HTTP/2 layer. Every gRPC call is one HTTP/2 stream, and many streams ride a single TCP connection.

The anatomy of a unary call on the wire

1. HEADERS frame — carries :method: POST, :path: /user.v1.UserService/GetUser, content-type: application/grpc, plus any custom metadata and the grpc-timeout.
2. DATA frame(s) — the message, length-prefixed: 1 byte compression flag + 4 bytes big-endian length + the protobuf bytes.
3. Trailing HEADERS (trailers) — grpc-status and grpc-message. This is critical: the actual success/failure of a gRPC call lives in HTTP/2 trailers, sent after the body.

HTTP/2 features gRPC depends on

• Multiplexing — concurrent streams without head-of-line blocking at the HTTP layer.
• Flow control — per-stream and per-connection windows; controls backpressure (relevant when tuning streaming throughput).
• HPACK — header compression, so repeated metadata is cheap.
• GOAWAY — graceful connection shutdown; the server tells clients to stop opening new streams (important for rolling deploys).

04The four RPC types

05Channels, stubs & the call lifecycle

• Channel — the client's virtual connection to a logical server (a target string like dns:///user-svc:50051). A channel manages one or more real subchannels (TCP connections), name resolution, and load-balancing state. Channels are expensive; create one and reuse it.
• Stub — the generated, type-safe client bound to a channel. Cheap to create.
• Subchannel — a connection to a single backend address, with its own connectivity state (IDLE → CONNECTING → READY → TRANSIENT_FAILURE).

go// Create ONE channel, reuse across the process.
conn, err := grpc.NewClient(
    "dns:///user-svc.default.svc.cluster.local:50051",
    grpc.WithTransportCredentials(insecure.NewCredentials()),
    grpc.WithDefaultServiceConfig(`{"loadBalancingConfig":[{"round_robin":{}}]}`),
)
defer conn.Close()

client := userv1.NewUserServiceClient(conn)
resp, err := client.GetUser(ctx, &userv1.GetUserRequest{Id: "42"})

06Metadata, deadlines & cancellation

Metadata

Key-value pairs sent as HTTP/2 headers (request) or trailers (response). Used for auth tokens, request IDs, tracing context. Binary values use a -bin suffix on the key.

Deadlines — not timeouts

gRPC strongly favors deadlines (an absolute point in time) over timeouts (a duration). The client encodes the remaining time as a grpc-timeout header. Crucially, deadlines propagate: if Service A calls B with a 2s deadline and B calls C, C inherits the remaining budget. This prevents work being done on requests the caller has already given up on.

Cancellation

If the client cancels (or the deadline expires), gRPC sends an RST_STREAM and the server's context is cancelled — well-behaved servers stop work immediately. This is cooperative; your handlers must check ctx.Done().

07Status codes & the error model

gRPC has its own status code space — 17 codes, distinct from HTTP status codes. Knowing the retry-relevant ones cold is essential for configuring meshes and clients.

08Interceptors

Interceptors are gRPC's middleware. They wrap every call, on both client and server sides, for unary and streaming. This is where you put cross-cutting concerns: auth, logging, metrics, tracing, panic recovery, rate limiting.

go// Server-side unary interceptor: log + inject metrics
func MetricsInterceptor(
    ctx context.Context, req any,
    info *grpc.UnaryServerInfo, handler grpc.UnaryHandler,
) (any, error) {
    start := time.Now()
    resp, err := handler(ctx, req)        // call the actual RPC
    code := status.Code(err)
    rpcLatency.WithLabelValues(info.FullMethod, code.String()).
        Observe(time.Since(start).Seconds())
    return resp, err
}

09Name resolution & load balancing

This section is the bridge to the Kubernetes story. gRPC's LB model is client-side by default, and that design decision is what collides with how Kubernetes Services work.

Name resolution

The channel target uses a scheme: dns:///host:port, passthrough:///ip:port, or xds:///. The DNS resolver resolves the host to a set of addresses and watches for changes. The trailing details matter:

• dns:///user-svc:50051 — resolve user-svc via DNS, get back all A records.
• For this to return all pod IPs in Kubernetes, you need a headless Service (clusterIP: None) — a normal ClusterIP Service returns a single virtual IP.

Load-balancing policies

• pick_first (default) — connect to the first working address; all traffic to one backend.
• round_robin — open a subchannel to every resolved address and rotate. This is what you want for client-side LB.
• Look-aside / xDS — an external control plane (e.g. via the gRPC xDS API, the same one Envoy uses) hands out endpoints and policy.

10Retries, keepalive & health checking

Retries & hedging (service config)

gRPC supports declarative retry policy via the channel's service config (JSON) — max attempts, backoff, and which status codes are retryable. Hedging sends the same request to multiple backends and takes the first response (only for idempotent calls).

json{
  "methodConfig": [{
    "name": [{"service": "user.v1.UserService"}],
    "retryPolicy": {
      "maxAttempts": 4,
      "initialBackoff": "0.1s",
      "maxBackoff": "1s",
      "backoffMultiplier": 2,
      "retryableStatusCodes": ["UNAVAILABLE"]
    }
  }]
}

Keepalive

gRPC sends HTTP/2 PING frames to detect dead connections and to keep NAT/firewall mappings alive. Misconfigured keepalive is a frequent cause of GOAWAY: too_many_pings errors — the client pings more aggressively than the server's PermitWithoutStream / min-time policy allows.

Health checking

The standard grpc.health.v1.Health/Check service returns SERVING / NOT_SERVING. This is what Kubernetes probes and load balancers query (see §14). Reflection (grpc.reflection.v1) lets tools like grpcurl discover services without the .proto file — invaluable for debugging, but consider disabling it in production.

11Security & auth

• Channel credentials — TLS/mTLS at the transport. In a mesh you typically delegate this to the sidecar (the app speaks plaintext to its local Envoy, Envoy does mTLS on the wire).
• Call credentials — per-RPC tokens (OAuth2 / JWT bearer tokens) carried in metadata. This composes with channel creds.
• ALTS — Google's mutual-auth scheme, mostly GCP-internal.

12Kubernetes: the L4 load-balancing trap

This is the canonical gRPC-on-Kubernetes problem, and it follows directly from §03 and §09. Here is the failure mode in one breath:

13Fixing load balancing without a mesh

There are three families of fix. Pick based on whether you have a mesh.

Option A — Headless Service + client-side LB

Make the Service headless so DNS returns all pod IPs, then configure the gRPC client to use round_robin. The client now holds a subchannel to every pod and balances per-request.

yamlapiVersion: v1
kind: Service
metadata:
  name: user-svc
spec:
  clusterIP: None          # ← headless: DNS returns every pod IP
  selector:
    app: user
  ports:
    - name: grpc           # name the port (matters for Istio too)
      port: 50051
      appProtocol: grpc      # explicit protocol hint

Client target: dns:///user-svc.default.svc.cluster.local:50051 with round_robin. The catch: the DNS resolver only re-resolves periodically, so newly added pods aren't picked up instantly — tune the resolver refresh, or rely on the mesh instead.

Option B — A look-aside / xDS balancer

Run a control plane that feeds endpoints to clients over the gRPC xDS API. This is powerful but operationally heavy if you don't already run such infrastructure.

Option C — An L7 proxy / service mesh

Put an HTTP/2-aware L7 proxy (Envoy) in the path. It terminates the client's connection and does per-request balancing to backends. This is the Istio answer, and it's why a mesh makes the whole problem disappear — see Part III.

14Health probes

An HTTP probe can't check a gRPC-only server. Two correct approaches:

Native gRPC probes (Kubernetes 1.24+, GA)

yamlreadinessProbe:
  grpc:
    port: 50051
    service: "user.v1.UserService"   # optional; checks Health/Check
  initialDelaySeconds: 5
  periodSeconds: 10

The kubelet calls the standard grpc.health.v1.Health/Check service. The server must implement and register it.

Legacy: grpc_health_probe binary

Before 1.24, you shipped the grpc_health_probe binary in the image and used an exec probe. Still seen in older charts.

15Istio: how Envoy handles gRPC

Istio injects an Envoy sidecar next to each pod. All inbound/outbound traffic is transparently redirected through it (via iptables or the newer ambient mode). Envoy is a full L7, HTTP/2-native proxy — it parses gRPC framing, understands trailers, reads grpc-status, and balances per request.

16Protocol detection — get this right

For Envoy to apply HTTP/2-aware (L7) handling, Istio must know the port speaks gRPC. If it thinks the port is plain TCP, you fall back to L4 balancing — the very problem you're trying to avoid.

Two ways to declare it, in order of preference:

1. appProtocol: grpc on the Service port — the explicit, modern, unambiguous way.
2. Port name prefix — name the port grpc, grpc-web, or grpc-anything. Istio reads the prefix.

yamlports:
  - name: grpc-user        # prefix recognised by Istio
    port: 50051
    appProtocol: grpc        # preferred explicit signal

17Traffic management for gRPC

Because Envoy understands gRPC, VirtualService and DestinationRule work at the method level. Routing can match on the gRPC path (/package.Service/Method) and on metadata headers.

yamlapiVersion: networking.istio.io/v1
kind: VirtualService
metadata: { name: user-svc }
spec:
  hosts: [ user-svc ]
  http:                        # gRPC is HTTP/2 → use the http block
    - match:
        - uri:
            prefix: "/user.v1.UserService/"
      route:
        - destination: { host: user-svc, subset: v2 }
      timeout: 2s

18Retries, outlier detection & connection pools

gRPC-aware retries

Istio retries can key off gRPC status codes, not just HTTP codes. The retryOn field accepts gRPC conditions like cancelled, deadline-exceeded, internal, resource-exhausted, and unavailable.

yaml  http:
    - route: [ { destination: { host: user-svc } } ]
      retries:
        attempts: 3
        perTryTimeout: 1s
        retryOn: "unavailable,deadline-exceeded,resource-exhausted"

Outlier detection (passive health)

In a DestinationRule, outlier detection ejects misbehaving pods. For gRPC, Envoy maps certain statuses (e.g. UNAVAILABLE, INTERNAL) into the 5xx bucket that consecutive5xxErrors counts.

yamlapiVersion: networking.istio.io/v1
kind: DestinationRule
metadata: { name: user-svc }
spec:
  host: user-svc
  trafficPolicy:
    loadBalancer: { simple: ROUND_ROBIN }
    connectionPool:
      http:
        http2MaxRequests: 1000          # max concurrent streams to backend
        maxRequestsPerConnection: 0     # 0 = unlimited (no forced cycling)
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50

19mTLS & telemetry

Automatic mTLS

Because gRPC is just HTTP/2 to Envoy, Istio's automatic mutual TLS applies with zero app changes. The app speaks plaintext HTTP/2 to its sidecar; the sidecars negotiate mTLS on the wire using SPIFFE identities. Enforce with PeerAuthentication (set mode STRICT) and authorize peer-to-peer access with AuthorizationPolicy — which can match on the gRPC method path under operation.paths.

yamlapiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata: { name: user-svc-rbac }
spec:
  selector: { matchLabels: { app: user } }
  action: ALLOW
  rules:
    - from: [ { source: { principals: [ "cluster.local/ns/web/sa/frontend" ] } } ]
      to:   [ { operation: { paths: [ "/user.v1.UserService/GetUser" ] } } ]

Telemetry

Envoy emits per-call gRPC telemetry without app instrumentation: request counts, durations, and the gRPC response status as a dimension. These flow into your metrics backend (Prometheus/Mimir) as istio_requests_total with a grpc_response_status label, and Envoy propagates tracing headers (B3 / W3C) so spans stitch across hops into Tempo.

20Streaming & the gotchas list

The full gotcha checklist

★One-page cheat sheet

The five facts to never forget

1. 1 RPC = 1 HTTP/2 stream; many streams = 1 TCP connection. Everything else follows.
2. gRPC status lives in HTTP/2 trailers — L4 LBs are blind to it.
3. Default k8s ClusterIP balances at L4 → all gRPC traffic pins to one pod.
4. Fix without mesh: headless Service + round_robin. Fix with mesh: nothing — Envoy balances per request.
5. Always set a deadline; always declare the port as gRPC.

Debugging toolkit

grpcurl introspect & call (needs reflection) · grpc_health_probe probe health · ghz load testing · istioctl proxy-config inspect Envoy clusters/routes · channelz in-process connection state · GRPC_GO_LOG_SEVERITY_LEVEL=info client tracing

Status codes worth memorizing

0 OK3 INVALID_ARGUMENT4 DEADLINE_EXCEEDED8 RESOURCE_EXHAUSTED13 INTERNAL14 UNAVAILABLE — retry16 UNAUTHENTICATED