For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Detect model fallback by injecting the requested and actual LLM model names as response headers using llm.requestModel and llm.responseModel CEL variables.

Use LLM request transformations to dynamically compute and set fields in LLM requests using Common Expression Language (CEL)CEL (Common Expression Language)A simple expression language used throughout agentgateway to enable flexible configuration. CEL expressions can access request context, JWT claims, and other variables to make dynamic decisions. expressions. Transformations let you enforce policies such as capping token usage or conditionally modifying request parameters, without changing client code.

To learn more about CEL, see the following resources:

• CEL expression reference
• cel.dev tutorial

Before you begin

1. Set up an agentgateway proxy.
2. Set up access to the OpenAI LLM provider.

Configure LLM request transformations

1. Create an AgentgatewayPolicy resource to apply an LLM request transformation. The following example limits max_completion_tokens to no more than 10. If the client requests less than 10 tokens, this number is applied. If the client requests more than 10 tokens, the maximum number of 10 is applied.

   kubectl apply -f- <<EOF
   apiVersion: agentgateway.dev/v1alpha1
   kind: AgentgatewayPolicy
   metadata:
     name: cap-max-tokens
     namespace: agentgateway-system
     labels:
       app: agentgateway
   spec:
     targetRefs:
     - group: gateway.networking.k8s.io
       kind: HTTPRoute
       name: openai
     backend:
       ai:
         transformations:
         - field: max_completion_tokens
           expression: "min(llmRequest.max_completion_tokens, 10)"
   EOF

   Setting	Description
   backend.ai.transformations	A list of LLM request field transformations.
   field	The name of the LLM request field to set. Maximum 256 characters.
   expression	A CEL expression that computes the value for the field. Use the llmRequest variable to access the original LLM request body. Maximum 16,384 characters.

   Note

   You can specify up to 64 transformations per policy. Transformations take priority over overrides for the same field. If an expression fails to evaluate, the field is silently removed from the request.

   Thinking budget fields, such as reasoning_effort and thinking_budget_tokens can also be set or capped by using transformations. This way, operators can enforce reasoning limits centrally without requiring client changes. For example, use "field": "reasoning_effort" with the expression "medium" to cap all requests to medium reasoning efforts regardless of what the client sends.

2. Verify that the AgentgatewayPolicy is accepted.

   kubectl get AgentgatewayPolicy cap-max-tokens -n agentgateway-system -o jsonpath='{.status.ancestors[0].conditions[?(@.type=="Accepted")].status}'

3. Send a request with max_completion_tokens set to a value greater than 10. The transformation limits it to 10 before the request reaches the LLM provider. Verify that the completion_tokens value in the response is 10 or fewer and the finish_reason is set to length.

   Note

   Some older OpenAI models use max_tokens instead of max_completion_tokens. If the transformation does not appear to take effect, check the model’s API documentation for the correct field name and update the transformation’s field value accordingly.

   Cloud Provider LoadBalancerPort-forward for local testing

   curl "$INGRESS_GW_ADDRESS/v1/chat/completions" \
   -H "content-type: application/json" \
   -d '{
     "model": "gpt-3.5-turbo",
     "max_completion_tokens": 5000,
     "messages": [
       {
         "role": "user",
         "content": "Tell me a short story"
       }
     ]
   }' | jq 

   curl "localhost:8080/v1/chat/completions" \
   -H "content-type: application/json" \
   -d '{
     "model": "gpt-3.5-turbo",
     "max_completion_tokens": 5000,
     "messages": [
       {
         "role": "user",
         "content": "Tell me a short story"
       }
     ]
   }' | jq 

   Example output:

   {
     "model": "gpt-3.5-turbo-0125",
     "usage": {
       "prompt_tokens": 12,
       "completion_tokens": 10,
       "total_tokens": 22,
       "completion_tokens_details": {
         "reasoning_tokens": 0,
         "audio_tokens": 0,
         "accepted_prediction_tokens": 0,
         "rejected_prediction_tokens": 0
       },
       "prompt_tokens_details": {
         "cached_tokens": 0,
         "audio_tokens": 0
       }
     },
     "choices": [
       {
         "message": {
           "content": "Once upon a time, in a small village nestled",
           "role": "assistant",
           "refusal": null,
           "annotations": []
         },
         "index": 0,
         "logprobs": null,
         "finish_reason": "length"
       }
     ],
     ...
   }
   

Transform requests after provider conversion

The transformations field runs before agentgateway converts the request into the format that the provider expects. To write one, you must know how the conversion works, and you cannot change a field that the conversion itself adds.

The finalTransformations field runs after the conversion instead. You only need to know the shape of the target API.

The difference shows in the field names. When a client sends max_tokens to an OpenAI provider, the conversion renames the field to max_completion_tokens, so the provider receives the following request body.

{
  "model": "gpt-4",
  "max_completion_tokens": 5000,
  "messages": [{"role": "user", "content": "Tell me a short story"}]
}

A transformations entry must target max_tokens, the name that the client sent. A finalTransformations entry must target max_completion_tokens, the name that the provider receives.

1. Create an AgentgatewayPolicy resource that caps the converted field and removes a field from the provider request. The min() expression caps max_completion_tokens at 10. The fail("remove") expression always fails to evaluate, which deletes reasoning_effort from the request.

   kubectl apply -f- <<EOF
   apiVersion: agentgateway.dev/v1alpha1
   kind: AgentgatewayPolicy
   metadata:
     name: cap-converted-tokens
     namespace: agentgateway-system
     labels:
       app: agentgateway
   spec:
     targetRefs:
     - group: gateway.networking.k8s.io
       kind: HTTPRoute
       name: openai
     backend:
       ai:
         finalTransformations:
         - field: max_completion_tokens
           expression: "min(llmRequest.max_completion_tokens, 10)"
         - field: reasoning_effort
           expression: 'fail("remove")'
   EOF

   Setting	Description
   backend.ai.finalTransformations	A list of transformations that apply to the converted provider request. You can specify up to 64 entries per policy. Entries take priority over overrides for the same field.
   field	The name of the field to set in the converted request. Maximum 256 characters.
   expression	A CEL expression that computes the value for the field. Maximum 16,384 characters.

   Warning

   In a finalTransformations expression, llmRequest is the converted request body, not the request that the client sent. An expression that reads a field which the converted body does not have, such as llmRequest.max_tokens for an OpenAI provider, fails to evaluate. A failed expression removes the target field, so a mistyped field name silently deletes the field that you meant to set. For more information about fail(), see Validate and set request body defaults.

2. Verify that the AgentgatewayPolicy is accepted.

   kubectl get AgentgatewayPolicy cap-converted-tokens -n agentgateway-system -o jsonpath='{.status.ancestors[0].conditions[?(@.type=="Accepted")].status}'

3. Send a request that sets max_tokens and reasoning_effort. The conversion renames max_tokens to max_completion_tokens, and the transformation then caps that field at 10 and drops reasoning_effort. Verify that the completion_tokens value in the response is 10 or fewer and the finish_reason is set to length.