Services that prove who they are: mutual TLS between microservices

It is easy to assume that anything inside the private network can be trusted. That is exactly the assumption an attacker hopes for. One compromised container, or one mistaken route, and every internal service becomes open to it.

So we do not trust the network. Our domain services are never reachable from outside. The only public door is the reverse proxy, then the gateway, then one backend-for-frontend. Behind that door, every service has to prove who it is to every other service.

A caller is checked twice, and the user is checked separately.

• The handshake. The caller must present a certificate signed by our own internal authority. With no certificate, the connection is refused before any request is read.
• The allow-list. The certificate carries a name, and each service keeps a short list of names that may call it. A valid certificate that is not on the list gets a 403.
• The user. The end user's signed token is verified by every service itself, exactly as before. mTLS says which service is calling. The token says which person it is calling for.

Most of the work is making sure no call is forgotten. Each kind of outbound client needs the certificate wired in, and so do health checks and the metrics scraper. A new service or a new caller needs an identity issued and a line added to the callee's allow-list. We keep that as a written checklist, because the easy mistake is a call that works locally and fails in the deployed setup.

A few rules keep it honest:

• No certificates inside application jars. They are mounted from a folder, and passwords come from the environment.
• No trust-all shortcuts, and never a global setting that replaces the default trust store, because that would break calls to services like Google.
• Certificates last a year and are rotated by regenerating them and restarting.
• The service registry and the config server cannot read their allow-list from the config server, so theirs lives in their own jar.

In tests we switch the check off for the web test layer, because it has no TLS to speak of, and cover it with its own tests: allowed caller, unlisted caller, no certificate, and the check disabled.

It is more setup, and a call that is not on the list fails by design. That can feel annoying the first time a new service cannot reach another. We think that annoyance is the point. A forgotten allow-list entry shows up in minutes in development. The alternative is a gap that shows up much later, in a much worse way.