Four Silent Gates: Routing to an Allowlisted Endpoint with a Tailscale App Connector

We had a staging OAuth endpoint behind an internet-facing AWS load balancer. The load balancer was locked down with loadBalancerSourceRanges, an IP allowlist that held only the address blocks of one third-party partner. Engineers could not reach it from their laptops. This post is about how we fixed that with a Tailscale app connector, and about the four separate policy gates that each had to open before a single packet got through. None of the four gates gave an error at the client. Traffic just hung.

The problem

A laptop resolves the hostname to the public IPs of the load balancer. The security group drops the SYN because the laptop IP is not on the list. Because the packet is dropped and not rejected, the client sees a connect timeout, not a connection refused. That detail matters later, because every failure in this story looks the same from the client side.

The obvious fix is to add each engineer’s IP as a /32. I did not want to do that. Home IPs change, the list grows with the team, and every entry widens an internet-facing surface.

The design

A Tailscale app connector is a tailnet node that routes traffic for a set of domains, not a set of CIDRs. When a tailnet client resolves a configured domain, the connector resolves it too, advertises the resulting IPs as /32 routes, and the client sends that traffic through the connector.

We already ran the Tailscale Kubernetes operator in the staging cluster for subnet routers, so the connector is a pod in the staging VPC. Its egress leaves through the staging NAT gateways. We added the three NAT Elastic IPs to the load balancer allowlist. So instead of N laptop IPs, the list gained three stable egress IPs that the team does not need to touch again.

laptop (tailnet) --> app connector pod --> NAT gateway (EIP) --> public ELB --> oauth
                     (staging VPC)          (allowlisted)

The OAuth endpoints still enforce client authentication. The allowlist is network-layer defense in depth, not the primary control.

Gate 1: the operator must be allowed to mint the tag

The operator provisions the connector pod and registers it as a tailnet device with the tag tag:appconnector-staging. The operator authenticates with an OAuth client, and that client was scoped to mint auth keys for tag:staging only. Device registration failed with a 400:

requested tags [tag:appconnector-staging] are invalid or not permitted

The Connector resource went to ConnectorCreationFailed. Nothing at the client side showed this. You only see it if you read the operator logs or the resource status.

There are two parts to the fix. First, the policy file must let the operator own the new tag:

"tagOwners": {
  "tag:staging-operator":     [],
  "tag:staging":              ["tag:staging-operator"],
  "tag:appconnector-staging": ["tag:staging-operator"],
},

Second, the OAuth client itself must list the tag. We manage that client in Terraform, and the tags attribute is force-new. So the apply replaced the client and rotated the credentials in SSM. The operator does not reload credentials on its own, so it also needed a kubectl rollout restart deploy/operator.

Gate 2: the connector and the domain binding

The connector is a Connector custom resource with an empty appConnector block. Our Helm chart renders it from values:

apiVersion: tailscale.com/v1alpha1
kind: Connector
metadata:
  name: oauth-app-connector
spec:
  tags:
    - "tag:appconnector-staging"
  replicas: 2
  appConnector: {}

The empty block means “discover routes dynamically”. The list of domains does not live in the cluster. It lives in the tailnet policy, as a nodeAttrs entry that binds domains to connector tags:

"nodeAttrs": [
  {
    // target must be "*" for app-connector attrs. The "connectors"
    // field below scopes which nodes act as connectors.
    "target": ["*"],
    "app": {
      "tailscale.com/app-connectors": [
        {
          "name":       "oauth-stage",
          "connectors": ["tag:appconnector-staging"],
          "domains":    ["oauth.stage.example.com"],
        },
      ],
    },
  },
],

After gate 1 was open, the connector authenticated. Its logs showed it resolve the domain and advertise three /32 routes, one for each load balancer IP. That looked like success. It was not.

Gate 3: route auto-approval

Advertised routes are inert until the control plane approves them. Our subnet routers work because their CIDRs are in autoApprovers.routes. The connector routes are different: they are the current IPs of an ELB, and AWS rotates those. A fixed list of /32s would break the next time the ELB moved.

The answer is to auto-approve any route that the connector tag advertises:

"autoApprovers": {
  "routes": {
    "0.0.0.0/0": ["tag:appconnector-staging"],
    "::/0":      ["tag:appconnector-staging"],
  },
},

This reads like a default route, and the first reaction in review was alarm. It is not a default route. It says “approve any route that a node with this tag advertises”. The connector only advertises routes for its configured domains, so the scope stays tight. The real boundary is who can carry the tag, and gate 1 already controls that.

Gate 4: the client must be allowed to route through the connector

This was the subtle one. We had an ACL that let engineers reach the connector node:

{"action": "accept",
 "src": ["group:eng", "group:staging"],
 "dst": ["tag:appconnector-staging:*"]},

That rule covers the DNS discovery traffic to the connector itself. It does not cover the destinations the connector routes to. Subnet routers have the same split: you grant access to the router, and you grant access to <CIDR>:* behind it. For an app connector the destinations are dynamic public IPs, so there is no CIDR to name.

Tailscale models this with a via grant. It means “you may reach the internet, but only through this node”:

"grants": [
  {
    "src": ["group:eng", "group:staging"],
    "dst": ["autogroup:internet"],
    "ip":  ["*"],
    "via": ["tag:appconnector-staging"],
  },
],

My first attempt put this in the legacy acls block. The policy validator rejected it with json: unknown field "via", because via exists only in grants. The fix was to add a top-level grants block. Tailscale evaluates acls and grants side by side, so you do not need to migrate the rest of the policy to use one grant.

Again, this is not general internet access. autogroup:internet is scoped by via to a node that only advertises routes for one domain, so only that traffic goes through it.

After gate 4, curl https://oauth.stage.example.com/... from a laptop returned a response.

Two red herrings

CI passed at every step. We validate the policy file in CI with Tailscale’s policy check. It passed for every one of the four changes, including the ones that did not work. The check validates syntax and references. It cannot tell you that the operator’s OAuth client will refuse a tag (gate 1), or that a route will not propagate to a client (gates 3 and 4). A green check here means “the control plane will accept this file”, not “traffic will flow”.

tailscale whois still says “peer not found”. Even now that the path works, tailscale whois <elb-ip> reports no peer, because whois does not reflect routing that a via grant enables. I lost time treating that output as a signal. The only reliable functional test was a real request from a real client.

One more thing that looks like a fault and is not: we run two connector replicas for HA, and only the primary advertises routes. The standby correctly reports that it “does not expose any routes”.

What I would do differently

Read the gates as a checklist before the first apply, not one at a time after each hang:

  1. Can the operator’s OAuth client mint the connector tag? Does tagOwners agree?
  2. Does a nodeAttrs entry bind the domain to the connector tag?
  3. Will the control plane auto-approve the routes the connector advertises?
  4. Does a via grant let clients route through the connector to autogroup:internet?

Then test with a real request, not with whois and not with CI.

The result: engineers on the tailnet reach the staging OAuth endpoint through the connector, the NAT gateway, and the allowlisted load balancer. The public allowlist grew by three stable IPs, once, and it does not need to grow again.

Author | Joe O'Pecko

Software engineer