Skip to content
agentgateway has joined the Agentic AI FoundationLearn more

For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Data plane metrics

Page as Markdown

View, customize, and reference data plane metrics emitted by the agentgateway proxy.

The agentgateway proxy exposes a Prometheus-compatible metrics endpoint on port 15020. Metrics are collected automatically for every request that passes through the gateway and follow the OpenMetrics format. All agentgateway metrics use the agentgateway_ prefix.

To set up automatic scraping of these metrics with Prometheus, see Enable metrics scraping.

View data plane metrics

  1. Port-forward the agentgateway proxy.

    kubectl port-forward -n agentgateway-system deploy/agentgateway-proxy 15020:15020
  2. Query the metrics endpoint.

    curl http://localhost:15020/metrics
  3. Enable metrics scraping for data plane metrics with the OTel stack so that you can export and visualize metrics in monitoring tools, such as Prometheus and Grafana. For more information, see Scrape metrics for querying and visualization.

Add custom metric labels

You can enrich all metrics with custom labels computed from CEL expressions by using an AgentgatewayPolicy. Labels are added to every metric that carries route identifier labels.

The following example adds two labels to all metrics:

  • team: A dynamic label computed from a JWT claim, so each metric reflects the team that owns the request.
  • org: A dynamic label read from the x-org-id request header.
kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: metrics-labels
  namespace: agentgateway-system
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: agentgateway-proxy
  frontend:
    metrics:
      attributes:
        add:
        - name: team
          expression: jwt.team
        - name: org
          expression: 'request.headers["x-org-id"]'
EOF

Warning

High-cardinality labels, such as per-user IDs, can significantly increase Prometheus storage and memory usage. Prefer low-cardinality dimensions such as team or environment.

Note

The Kubernetes API only supports adding labels. Removing default metric labels or an entire metric is not supported via AgentgatewayPolicy resource.

Data plane metrics reference

Note

Counter and histogram metrics only appear after the first request of that type. For example, agentgateway_mcp_requests_total is not present until MCP traffic is received. This is normal Prometheus behavior — counters and histograms are not pre-initialized.

XDS

These metrics track communication with the agentgateway control plane over XDS. They are only present when agentgateway runs in Kubernetes mode and do not appear on a standalone process.

MetricTypeUnitDescription
agentgateway_xds_connection_terminations_totalCounterThe total number of completed connections to xds server (unstable).
agentgateway_xds_message_bytes_totalCounterbytesTotal number of bytes received (unstable).
agentgateway_xds_message_totalCounterTotal number of messages received (unstable).

HTTP

MetricTypeUnitDescription
agentgateway_request_duration_secondsHistogramsecondsDuration of HTTP requests.
agentgateway_request_processing_secondsHistogramsecondsDuration from receiving an HTTP request to sending the primary outbound call.
agentgateway_requests_shed_totalCounterTotal downstream requests rejected by the in-flight request limit.
agentgateway_requests_totalCounterThe total number of HTTP requests sent.
agentgateway_response_bytes_totalCounterbytesTotal HTTP response bytes received.
agentgateway_response_processing_secondsHistogramsecondsDuration from receiving the primary outbound response to sending the HTTP response.
agentgateway_retries_totalCounterThe total number of request retries.

TCP

MetricTypeUnitDescription
agentgateway_downstream_connections_shed_totalCounterTotal downstream connections closed by the active connection limit.
agentgateway_downstream_connections_totalCounterThe total number of downstream connections established.
agentgateway_downstream_received_bytes_totalCounterbytesTotal TCP bytes received per connection labels.
agentgateway_downstream_sent_bytes_totalCounterbytesTotal TCP bytes transmitted per connection labels.
agentgateway_tls_handshake_duration_secondsHistogramsecondsDuration to complete inbound TLS/HTTPS handshake.
agentgateway_upstream_connect_duration_secondsHistogramsecondsDuration to establish upstream connection.

MCP

MetricTypeUnitDescription
agentgateway_mcp_requests_totalCounterTotal number of MCP requests.

LLM

These metrics follow the OpenTelemetry semantic conventions for generative AI.

MetricTypeUnitDescription
agentgateway_cost_catalog_lookups_totalCounterTotal number of model cost catalog lookups by resolution status.
agentgateway_gen_ai_client_cost_usd_totalCounterusdCumulative USD cost of generative AI requests.
agentgateway_gen_ai_client_token_usageHistogramNumber of tokens used per request.
agentgateway_gen_ai_server_request_durationHistogramDuration of generative AI request.
agentgateway_gen_ai_server_time_per_output_tokenHistogramTime to generate each output token for a given request.
agentgateway_gen_ai_server_time_to_first_tokenHistogramTime to generate the first token for a given request.
agentgateway_guardrail_checks_totalCounterTotal number of guardrail checks.

Cgroup memory

These metrics track Linux cgroup v2 memory statistics for the agentgateway process. They are available whenever agentgateway runs in a Linux environment, including standalone deployments, and are not specific to Kubernetes mode.

MetricTypeUnitDescription
agentgateway_cgroup_usageGaugebytesCurrent memory usage.
agentgateway_cgroup_working_setGaugebytesCurrent working set.
agentgateway_cgroup_anonGaugebytesCurrent anonymous memory usage.
agentgateway_cgroup_active_anonGaugebytesCurrent active anonymous memory usage.
agentgateway_cgroup_inactive_anonGaugebytesCurrent inactive anonymous memory usage.
agentgateway_cgroup_fileGaugebytesCurrent file memory usage.
agentgateway_cgroup_file_mappedGaugebytesCurrent mapped file memory usage.
agentgateway_cgroup_active_fileGaugebytesCurrent active file memory usage.
agentgateway_cgroup_inactive_fileGaugebytesCurrent inactive file memory usage.
agentgateway_cgroup_shmemGaugebytesCurrent shared memory usage.
agentgateway_cgroup_kernelGaugebytesCurrent kernel memory usage.
agentgateway_cgroup_kernel_stackGaugebytesCurrent kernel stack memory usage.
agentgateway_cgroup_pagetablesGaugebytesCurrent page tables memory usage.
agentgateway_cgroup_percpuGaugebytesCurrent per-CPU memory usage.
agentgateway_cgroup_sockGaugebytesCurrent socket memory usage.
agentgateway_cgroup_slabGaugebytesCurrent slab memory usage.
agentgateway_cgroup_slab_reclaimableGaugebytesCurrent reclaimable slab memory usage.
agentgateway_cgroup_slab_unreclaimableGaugebytesCurrent unreclaimable slab memory usage.
agentgateway_cgroup_pgfault_totalCounterTotal cgroup page faults.
agentgateway_cgroup_pgmajfault_totalCounterTotal cgroup major page faults.
agentgateway_cgroup_workingset_refault_anon_totalCounterTotal anonymous working set refaults.
agentgateway_cgroup_workingset_refault_file_totalCounterTotal file working set refaults.
agentgateway_cgroup_workingset_activate_anon_totalCounterTotal anonymous working set activations.
agentgateway_cgroup_workingset_activate_file_totalCounterTotal file working set activations.
agentgateway_cgroup_workingset_restore_anon_totalCounterTotal anonymous working set restores.
agentgateway_cgroup_workingset_restore_file_totalCounterTotal file working set restores.

Process memory

These metrics track process-level memory for the agentgateway process, sourced from /proc/self/smaps. They are available whenever agentgateway runs in a Linux environment, including standalone deployments, and are not specific to Kubernetes mode.

MetricTypeUnitDescription
agentgateway_process_rssGaugebytesRSS (resident set size) memory usage.
agentgateway_process_pssGaugebytesPSS (proportional set size) memory usage.
agentgateway_process_pss_dirtyGaugebytesDirty PSS memory usage.
agentgateway_process_shared_cleanGaugebytesShared clean memory usage.
agentgateway_process_shared_dirtyGaugebytesShared dirty memory usage.
agentgateway_process_private_cleanGaugebytesPrivate clean memory usage.
agentgateway_process_private_dirtyGaugebytesPrivate dirty memory usage.
agentgateway_process_referencedGaugebytesReferenced memory usage.
agentgateway_process_anonymousGaugebytesAnonymous memory usage.
agentgateway_process_lazy_freeGaugebytesLazy free memory.
agentgateway_process_anon_huge_pagesGaugebytesAnonymous huge pages usage.
agentgateway_process_shmem_huge_pagesGaugebytesShared memory huge pages usage.
agentgateway_process_shmem_pmd_mappedGaugebytesShared memory PMD-mapped usage.
agentgateway_process_file_pmd_mappedGaugebytesFile PMD-mapped usage.
agentgateway_process_shared_hugetlbGaugebytesShared hugetlb usage.
agentgateway_process_private_hugetlbGaugebytesPrivate hugetlb usage.
agentgateway_process_swapGaugebytesProcess swap usage.
agentgateway_process_swap_pssGaugebytesProcess proportional swap usage.
agentgateway_process_lockedGaugebytesProcess locked memory usage.

Misc

The following metrics reflect the health of the agentgateway process and appear on every startup regardless of traffic.

MetricTypeUnitDescription
agentgateway_build_infoInfoAgentgateway build information.
agentgateway_config_synchronizedGaugeWhether the last configuration load or reload was successful.
agentgateway_upstream_call_duration_secondsHistogramsecondsDuration of outbound calls made by agentgateway.
agentgateway_tokio_num_workersGaugeThe number of worker threads the async runtime uses.
agentgateway_tokio_num_alive_tasksGaugeThe number of tasks currently alive in the async runtime.
agentgateway_tokio_global_queue_depthGaugeThe number of tasks currently scheduled in the async runtime’s global queue.

Example PromQL queries

Use these queries in Prometheus or as the basis for Grafana panels and alerts. To enter a raw PromQL query in Grafana, switch the query editor from Builder to Code mode.

Use casePromQL query
Request raterate(agentgateway_requests_total[5m])
Error raterate(agentgateway_requests_total{status=~"5.."}[5m]) / rate(agentgateway_requests_total[5m])
LLM token usage (input)sum by (gen_ai_system, gen_ai_request_model) (rate(agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="input"}[5m]))
Time to first token (p95)histogram_quantile(0.95, rate(agentgateway_gen_ai_server_time_to_first_token_bucket[5m]))
MCP tool call rate by toolsum by (server, resource) (rate(agentgateway_mcp_requests_total{method="tools/call"}[5m]))
Was this page helpful?
Agentgateway assistant

Ask me anything about agentgateway configuration, features, or usage.

Note: AI-generated content might contain errors; please verify and test all returned information.

Tip: one topic per conversation gives the best results. Use the + button in the chat header to start a new conversation.

Switching topics? Starting a new conversation improves accuracy.
↑↓ navigate select esc dismiss

What could be improved?

Your feedback helps us improve assistant answers and identify docs gaps we should fix.

Need more help? Join us on Discord: https://discord.gg/y9efgEmppm

Want to use your own agent? Add the Solo MCP server to query our docs directly. Get started here: https://search.solo.io/.