Audio models

Step 1: Deploy the audio model (self-hosted)

Deploy the audio model inside your cluster (e.g., Voxtral Small via vLLM) and expose it via a Service.

1. Deploy Voxtral Small using the official vLLM OpenAI-compatible image.

   kubectl apply -f- <<EOF
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: voxtral-vllm
     namespace: agentgateway-system
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: voxtral-vllm
     template:
       metadata:
         labels:
           app: voxtral-vllm
       spec:
         containers:
         - name: voxtral-vllm
           image: vllm/vllm-openai:latest
           args:
           - "vllm"
           - "serve"
           - "mistralai/Voxtral-Small-24B-2507"
           - "--tokenizer_mode"
           - "mistral"
           - "--config_format"
           - "mistral"
           - "--load_format"
           - "mistral"
           - "--tool-call-parser"
           - "mistral"
           - "--enable-auto-tool-choice"
           ports:
           - containerPort: 8000
             name: http
           resources:
             limits:
               nvidia.com/gpu: "1"
             requests:
               nvidia.com/gpu: "1"
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: voxtral-service
     namespace: agentgateway-system
   spec:
     selector:
       app: voxtral-vllm
     ports:
     - port: 80
       targetPort: 8000
       protocol: TCP
   EOF

   The deployment uses:

   • vllm/vllm-openai:latest — the official vLLM OpenAI-compatible server image
   • vllm serve mistralai/Voxtral-Small-24B-2507 with Mistral-specific configuration
   • Port 8000 internally (vLLM default) exposed as port 80 on the Service
   • NVIDIA GPU reservation (1 GPU)

2. Wait for the pod to be ready.

   kubectl wait --for=condition=ready pod \
     -l app=voxtral-vllm \
     -n agentgateway-system \
     --timeout=300s

Step 2: Verify the model is responding

Verify that the model is responding by listing the models that it serves. The voxtral-service Service is a ClusterIP Service, so send the request from a pod inside the cluster.

kubectl run curl-audio --rm -i --restart=Never \
  -n agentgateway-system \
  --image=curlimages/curl -- \
  curl -s http://voxtral-service.agentgateway-system.svc.cluster.local:80/v1/models

You should see a JSON response listing the model.

Step 3: Create the LLM backend with Passthrough routes

Create an AgentgatewayBackend resource with policies.ai.routes to forward audio endpoints via Passthrough processing.

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: voxtral-audio
  namespace: agentgateway-system
spec:
  ai:
    provider:
      openai:
        model: voxtral-small-24b-2507
      host: voxtral-service.agentgateway-system.svc.cluster.local
      port: 80
  policies:
    ai:
      routes:
        "/v1/audio/transcriptions": "Passthrough"
        "/v1/models": "Passthrough"
        "*": "Passthrough"
EOF

Review the following table to understand this configuration. For more information, see the API reference.

Step 4: Create an HTTPRoute

Create an HTTPRoute that routes traffic to the audio backend.

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: voxtral-audio-route
  namespace: agentgateway-system
spec:
  parentRefs:
    - name: agentgateway-proxy
      namespace: agentgateway-system
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /audio
      filters:
        - type: URLRewrite
          urlRewrite:
            path:
              type: ReplacePrefixMatch
              replacePrefixMatch: /
      backendRefs:
        - name: voxtral-audio
          namespace: agentgateway-system
          group: agentgateway.dev
          kind: AgentgatewayBackend
EOF

Step 5: Send audio transcription requests

Create a sample WAV file to transcribe. If you already have an audio file, use its path in the following requests instead.

mkdir -p testdata
python3 -c "
import wave
w = wave.open('testdata/sample-audio.wav', 'wb')
w.setnchannels(1); w.setsampwidth(2); w.setframerate(16000)
w.writeframes(b'\x00\x00' * 16000)
w.close()
"

Then, send a transcription request to the audio model through the gateway.

export INGRESS_GW_ADDRESS=$(kubectl get svc -n agentgateway-system agentgateway-proxy -o=jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")

curl --request POST \
  --url "http://${INGRESS_GW_ADDRESS}:80/audio/v1/audio/transcriptions" \
  --header 'Content-Type: multipart/form-data' \
  --form model=voxtral-small-24b-2507 \
  --form 'file=@./testdata/sample-audio.wav' | jq

kubectl port-forward -n agentgateway-system svc/agentgateway-proxy 8080:80

curl --request POST \
  --url "http://localhost:8080/audio/v1/audio/transcriptions" \
  --header 'Content-Type: multipart/form-data' \
  --form model=voxtral-small-24b-2507 \
  --form 'file=@./testdata/sample-audio.wav' | jq