The idea: one front door for many services
A system split into services (users, orders, catalog, …) does not want every client to know every service's address, and it
does not want every service to check passwords, count requests and log calls on its own. So it puts an API gateway
in front: clients call one address, api.example.com, and the gateway sends each request to the right service. On the
way it runs the checks that every request needs, once, in a fixed order. Products: Kong, Envoy, AWS API Gateway,
Apigee, Tyk, nginx with plugins.
On the canvas, the clients are on the left: Alice's mobile app, which sends a JWT; Partner Co's server, which sends an API key; and Mallory, whose token has expired. The gateway's stages run along the top of the middle panel, with its tables below: routes, consumers with their token buckets, the response cache and the cached signing keys. The services and the identity provider are on the right. The request box moves from stage to stage, and its text changes when the gateway rewrites it.
Routing and path rewriting
The route stage matches the method and the path against the route table: GET /api/orders goes to
orders-svc, GET /api/catalog/items to catalog-svc. No match means 404, answered by the gateway itself
(Demo 2). The route also says how to rewrite the request: here the /api prefix is stripped, so orders-svc
receives GET /orders and never needs to know the public URL layout. Routing happens first because everything after it
(which credentials, which scope, which limits, cache or not) is configured per route.
Authentication: API keys, JWT and JWKS
Two kinds of credentials are common. An API key (apikey: pk_live_42) is looked up in the gateway's
consumer table; it says who is calling and nothing more. A JWT, issued by an identity provider after a login (see
OAuth 2.0 and JWT vs Session), carries its claims inside: who
(sub), until when (exp) and what it may do (scope). The gateway checks its signature with the
provider's public keys, published as a JWKS at /.well-known/jwks.json. It fetches them once and caches
them, so a JWT check costs no network call afterwards (Demo 1: 72 ms the first time, 42 ms the second).
A missing, forged or expired credential gives 401 Unauthorized (Demo 2, Mallory). A known caller who lacks the
route's scope gets 403 Forbidden (Partner Co trying POST /api/orders). After the check the gateway removes the
credential and adds X-Consumer-Id, so services trust the gateway rather than the client.
Rate limiting per consumer
Each consumer has a token bucket whose size depends on its plan: gold holds 5 tokens, free holds 3, and both get
1 token back every 5 s. A request takes a token; with none left the answer is 429 Too Many Requests with
Retry-After, and every answer carries X-RateLimit-Remaining (Demo 3: of a burst of 5, Partner Co gets
3 answers and 2 × 429). See Token Bucket vs Leaky Bucket for the bucket itself.
The order of stages matters. In Kong the rate-limiting plugin runs before the proxy cache, so a cached answer still costs a token: the limit protects the plan, not only the services. With several gateway nodes, the counters live in a shared store such as Redis, or each node gets a share of the limit.
Response caching and transforms
For a GET whose answer is the same for everyone, the gateway can keep the answer: GET /api/catalog/items is cached for
30 s. A hit is answered at the gateway in 2 ms instead of 122 ms, and catalog-svc does no work (Demo 4). Answers that depend on
the caller, such as /api/orders, are not cached. The transform stage rewrites the request: strip the prefix, add
X-Request-Id so one request can be traced through every service's logs, add the consumer, drop the credential.
Timeouts, 502 and 504
The gateway waits at most 3000 ms for a service (Kong read_timeout, Envoy route timeout). Some statuses
are written by the gateway, not by a service:
| Status | Stage | Meaning |
|---|---|---|
| 404 Not Found | Route | no route for this method and path |
| 401 Unauthorized | Auth | no credential, a bad signature, or an expired token |
| 403 Forbidden | Authz | known caller, missing scope |
| 429 Too Many Requests | Rate limit | bucket empty; retry after Retry-After seconds |
| 502 Bad Gateway | Forward | the service refused the connection or sent a broken answer |
| 504 Gateway Timeout | Forward | the service did not answer in time |
In Demo 5, catalog-svc becomes slow: a cached answer is still served, but once it expires the request ends with
504 after 3000 ms. With orders-svc down, the gateway answers 502 at once. See
Circuit Breaker for how to stop sending requests to a service that keeps failing.
Aggregation
A mobile screen often needs data from several services. Instead of the phone making several round trips over a slow network, the
gateway calls the services in parallel and merges the answers: GET /api/me/dashboard calls users-svc and orders-svc and
takes as long as the slower one (Demo 6). A gateway built for one kind of client this way is called a
backend for frontend (BFF).
Gateway, load balancer, service mesh
All three sit in the path of a request, but at different places and for different jobs.
| API gateway | Load balancer | Service mesh | |
|---|---|---|---|
| sits | at the edge, for outside clients | in front of the copies of one service | next to every service (sidecar) |
| traffic | north-south | north-south or internal | east-west (service to service) |
| decides on | route, consumer, plan, scope | which copy is free and healthy | retries, timeouts, mTLS identity |
| examples | Kong, AWS API Gateway, Apigee | HAProxy, nginx, AWS ALB/NLB | Istio, Linkerd (Envoy sidecars) |
They are often stacked: a CDN in front (see How a CDN and Route 53 Work), then the gateway, then a load balancer per service (see Load Balancer: L4 vs L7). Envoy can play all three roles.
A gateway is often where a trace begins: see Distributed Tracing with OpenTelemetry for how one request is followed through the services behind it.
See also Monolith vs Microservices: the same shop as one process and as services behind a gateway, side by side.
The round trips a gateway saves are counted in REST vs GraphQL vs gRPC.
An API gateway is a reverse proxy with extra rules; see Forward proxy vs Reverse proxy for how the two kinds of proxy differ.
What the page leaves out
- mTLS between the gateway and the services, and request validation against an OpenAPI schema.
- Token introspection for opaque tokens, OAuth flows, and a web application firewall (WAF).
- Rate-limit counters shared between gateway nodes, usage plans and billing, and quotas per day or month.
- Retries, circuit breaking, canary routing by weight or header, gRPC and WebSocket routes.
- Real latencies vary; the numbers here are fixed so every demo gives the same result.