Draw the Line: What Belongs in an API Gateway

ThoughtWorks put “overambitious API gateways” on hold in their Technology Radar. The phrase they used stuck with us: a gateway is “transport software with ambitions to run critical application logic.”

We’ve been drawing that line for 20 years. It’s the same line, every project, and it moves in the same direction: teams keep pushing logic into the gateway because it’s convenient, and every time it becomes the bottleneck the whole system works around.

Here is where the line goes, and why it belongs there.

What a gateway is for

An API gateway sits in the request path. It’s the first thing traffic hits and the last thing that touches a response on the way out. That position gives it exactly four honest jobs.

Routing. Match an incoming request to a backend. Path, host, header, weighted split for a canary. This is the whole reason the gateway exists.

TLS termination. Decrypt inbound, re-encrypt to the backend if you need mutual TLS internally. Certificate handling in one place instead of scattered across services.

Authentication. Validate the token. Verify a JWT signature, check an API key against a store, confirm the request carries a valid session. Note the word: validate. Not issue, not rotate, not manage the lifecycle. Confirm the credential is real and current, then pass the identity downstream.

Rate limiting and shedding. Protect what’s behind you. Per-consumer quotas, burst limits, circuit breaking when a backend is unhealthy. The gateway is the only place that sees all traffic, so it’s the right place to throttle it.

Add request logging and basic metrics and you have the full list. Everything on it shares one property: it’s a transport concern. None of it knows what your business does.

What a gateway is not for

The trouble starts when a request needs shaping and the gateway is right there, holding the request.

Data transformation. A partner sends a SOAP envelope, your backend speaks JSON. Tempting to map it in the gateway. Don’t. That mapping encodes business meaning: which field is the customer number, what a missing value implies, how a legacy status code translates to a modern one. That knowledge belongs in a service you can test, not in a config block in the request path.

Business rules. “Reject orders over €10,000 unless the account is flagged enterprise.” That’s a rule. It has edge cases, it changes, it needs test coverage. In the gateway it has none of those things.

Composition across backends. Calling three services and stitching one response is orchestration. It has failure semantics: what happens when the second call succeeds and the third times out. A gateway config has no good way to express a compensating action. A service does.

ThoughtWorks is blunt about the reason. Domain logic in middleware becomes “a single point of failure requiring specialized expertise.” When the rule lives in the gateway, changing it means touching shared infrastructure. Every team that shares the gateway is now coupled to your business rule. Deploys get scary. The one engineer who understands the gateway config becomes the constraint on everyone’s release.

The test is simple: if you can’t unit-test it, it doesn’t belong in the gateway. Routing rules and rate limits are declarative and boring. Business logic is neither.

The pattern beneath the products

Gateways look new but the shape is old. InfoQ traces the line cleanly: hardware load balancers, then application delivery controllers, then software API gateways, then cloud-native gateways built on the same primitives as the rest of your infrastructure. Every generation did the same core job (get a request to the right backend, safely) with more programmability than the last.

Understanding that lineage changes how you evaluate a gateway. You’re not buying a platform. You’re buying a fast, reliable router with an auth stage and a throttle. When a vendor’s pitch is mostly about the transformation engine and the visual orchestration canvas, they’re selling you the part that will hurt you.

Where you put the gateway

Solo.io lays out five deployment options, and the choice matters as much as the feature list. In practice it collapses to three decisions.

One central edge gateway. All external traffic through a single tier. Simple to reason about, simple to secure. The risk is that it becomes a shared chokepoint: every team’s routing rules in one place, every team blocked on one deploy pipeline. Fine for a handful of services. Painful at fifty.

Two-tier. An edge gateway for cross-cutting concerns (TLS, coarse auth, DDoS protection) and a second tier of smaller gateways closer to each domain or team. The edge stays thin and stable; teams own their own routing without fighting over one config. This is where most of our clients land once they outgrow the single edge.

Per-service or sidecar. A microgateway next to each service, or a service mesh handling gateway concerns as sidecars. Maximum isolation, maximum operational surface. Worth it when you have the platform team to run a mesh. A tax you’ll resent if you don’t.

There’s no universal answer. There is a wrong answer, and it’s picking the mesh because it’s fashionable when a two-tier setup would have carried you for years.

The migration we keep doing

We’ve moved clients off overambitious gateways more than a dozen times. The story doesn’t vary much.

It starts convenient. A transformation here, a validation there, a small piece of routing logic that’s really a business rule. Eighteen months later the gateway config is thousands of lines nobody wants to touch, the one person who understands it has become a bottleneck, and a routine change to a business rule requires a change window on shared infrastructure.

The fix is never clever. Pull the logic out into services. Give the gateway back its four jobs. Make the business rules testable and deployable on their own schedule. It’s unglamorous work, and it’s a large fraction of what integration engineering actually is: undoing the convenient decision that quietly became the expensive one.

Draw the line early and you never pay for the migration. The gateway routes, authenticates, and rate-limits. Your services do the thinking. Keep those two things apart and most of the pain in this part of your stack simply never shows up.