For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Configure backends to route traffic to hostnames, LLM providers, and MCP servers.

Agentgateway backendsBackendA destination service that receives traffic from agentgateway. Backends can be static hosts, MCP servers, LLM providers, or other services. control where traffic is routed to. Agentgateway supports a variety of backends, such as simple hostnames and IP addresses, LLM providersProviderA service that provides LLM capabilities, such as OpenAI, Anthropic, or Azure. Agentgateway supports multiple LLM providers and can route to different providers based on configuration., and MCP servers.

Static Hosts

The simplest form of backend is a static hostname or IP address. Static hosts are a routing-based backend, so they are configured in a routes entry; the simplified llm and mcp modes model only LLM providers and MCP targets. For example:

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
gateways:
  default:
    port: 3000
    protocol: HTTP
routes:
- backends:
  - host: example.com:8080
    weight: 1
  - host: 127.0.0.1:80
    weight: 9

MCP Servers

The MCP backend allows you to connect to an MCP server. Below shows a simple example, exposing a local and remote MCP server. See the MCP connectivity guide for more information.

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
mcp:
  port: 3000
  targets:
  - name: stdio-server
    stdio:
      cmd: npx
      args: ["@modelcontextprotocol/server-everything"]
  - name: http-server
    mcp:
      host: https://example.com/mcp

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
gateways:
  default:
    port: 3000
routes:
- backends:
  - mcp:
      targets:
      - name: stdio-server
        stdio:
          cmd: npx
          args: ["@modelcontextprotocol/server-everything"]
      - name: http-server
        mcp:
          host: https://example.com/mcp

Session routing

By default, MCP backends use stateful session routing, where the gateway tracks session IDs and routes subsequent requests to the same upstream. For upstreams that do not maintain server-side session state, you can set statefulMode: stateless. In stateless mode, the gateway automatically wraps each request with an initialization sequence, so the upstream server processes every request independently.

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
mcp:
  port: 3000
  statefulMode: stateless
  targets:
  - name: openapi-server
    openapi:
      host: petstore3.swagger.io:443
      schema:
        url: https://petstore3.swagger.io/api/v3/openapi.json

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
gateways:
  default:
    port: 3000
routes:
- backends:
  - mcp:
      statefulMode: stateless
      targets:
      - name: openapi-server
        openapi:
          host: petstore3.swagger.io:443
          schema:
            url: https://petstore3.swagger.io/api/v3/openapi.json

LLM Providers

Agentgateway natively supports connecting to LLM providers, such as OpenAI and Anthropic. Below shows a simple example, connecting to OpenAI. See the LLM consumption guide for more information.

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
llm:
  models:
  - name: openai
    provider: openAI
    params:
      model: gpt-3.5-turbo
      apiKey: "$OPENAI_API_KEY"

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
gateways:
  default:
    port: 3000
routes:
- backends:
  - ai:
      name: openai
      provider:
        openAI:
          model: gpt-3.5-turbo
  policies:
    backendAuth:
      key: "$OPENAI_API_KEY"

AWS AgentCore

The AWS backend routes requests to an Amazon Bedrock AgentCore agent runtime. Configure the AWS backend inline in a routes entry, as in the following example, or as a named top-level backends entry that several routes reference.

Agentgateway derives the connection details from the agentRuntimeArn value: requests are sent over TLS to the bedrock-agentcore endpoint in the runtime’s AWS region, with the path set to the runtime’s invocation endpoint. Agentgateway signs each request with AWS SigV4 under the bedrock-agentcore signing name, by using the standard AWS credential lookup from the environment. To authenticate to a runtime that uses a JWT authorizer instead, set a policies.backendAuth.key policy, which replaces the SigV4 signing. For the steps to set up both modes, see Connect to AWS Bedrock AgentCore.

The following configuration is from the traffic-aws-agentcore example in the agentgateway repository.

# AgentCore with static user-id header forwarding.
binds:
- port: 3000
  listeners:
  - routes:
    - matches:
      - path:
          pathPrefix: /supply-chain-agent
      backends:
      - aws:
          agentCore:
            agentRuntimeArn: "arn:aws:bedrock-agentcore:us-west-2:606469916935:runtime/a2a_sca_iam-4rLvS1BRqq"
        policies:
          requestHeaderModifier:
            set:
              X-Amzn-Bedrock-AgentCore-Runtime-User-Id: "user-foo"
              X-Amzn-Bedrock-AgentCore-Runtime-Custom-User-Id: "user-foo"

Session affinity

When a backend resolves to more than one endpoint, agentgateway load balances across them, and two requests from the same client can land on different endpoints. Set the sessionAffinity backend policy to send every request that carries the same value to the same endpoint.

A source CEL expression selects the value. Agentgateway hashes it and maps the hash to an endpoint with weighted rendezvous hashing, so each replica picks the same endpoint for the same value without sharing any state with the other replicas.

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
gateways:
  default:
    port: 3000
routes:
- backends:
  - host: localhost:8080
    policies:
      sessionAffinity:
        source: request.headers["x-session-id"]

Common expressions for source include the following.

What session affinity does not do

Session affinity is best-effort, and it is not session persistence. Agentgateway does not record which endpoint a value was sent to. It recomputes the mapping for each request from the value and the set of healthy endpoints, which has two consequences.

• The mapping moves when the endpoint set changes. Adding, removing, or losing an endpoint remaps some values, so a client can be moved to a different endpoint mid-session. Rendezvous hashing keeps that disruption small, because only the values that mapped to the changed endpoint move, but it is not zero.
• A request that produces no usable value is not pinned. Agentgateway falls back to normal load balancing when the expression fails to evaluate, returns a value that is not a string or bytes, or returns an empty value, such as a header the client did not send. The request still succeeds.

Do not use session affinity to hold server-side state that only one endpoint has. Use it to improve cache hit rates, to keep a conversation on one replica when that is a preference rather than a requirement, or to make debugging easier.

Two other features choose an endpoint before affinity does, and they win when they apply: inference routing, and a stateful MCP session that is already pinned to an upstream. In practice they do not conflict, because they target different backends.