AI Inference Eagle Eye ​

Feature Introduction ​

Eagle Eye is an observability system for AI inference scenarios, providing full-link metric collection, near-real-time transmission, intelligent diagnosis, and end-to-end distributed tracing capabilities.

In terms of metric monitoring, it combines Prometheus's periodic metric collection with a distributed message queue system's low-latency push mechanism, achieving comprehensive observability from AI gateway, inference engine, Mooncake to infrastructure (Ray, K8s, hardware). It supports both trend analysis for scaling decisions and the needs of modules with high time-sensitivity requirements (such as intelligent routing) for second-level data updates. Through an independent hardware health diagnosis module, it continuously monitors and identifies anomalies in low-level metrics such as NPU/GPU, temperature, power consumption, and error codes, building a closed-loop monitoring capability of "collection — transmission — diagnosis — assessment".

In terms of tracing, based on OpenTelemetry, it provides non-intrusive end-to-end tracing capability, tracking the complete execution path of inference requests from AI gateway to inference engine (vLLM-Ascend), recording the duration and key business attributes of each stage, supporting performance tuning and fault diagnosis.

Application Scenarios ​

• System Resource Health Monitoring: Real-time monitoring of infrastructure (Ray, K8s, hardware) to ensure the health and stability of system resources, timely detect and resolve resource bottlenecks, and ensure efficient system operation.
• Inference Process Performance Optimization: Real-time monitoring of performance metrics (such as latency, throughput) and resource usage at each stage of the inference process (such as prefill, decode), identifying and analyzing performance bottlenecks, optimizing model execution efficiency, and improving the response speed and computational efficiency of inference tasks.
• Hardware Fault Diagnosis and Repair: View anomaly analysis reports provided by the hardware diagnosis module, which include fault pattern identification and remediation suggestions, helping to quickly locate and resolve hardware faults. The system can monitor hardware status such as NPU/GPU, temperature, and power consumption in real time, generate detailed fault analysis reports, provide specific fault causes and repair solutions, and ensure hardware stability and reliability.
• Automatic Scaling Decisions: Obtain SLA-related metrics (such as throughput rate, latency, etc.) and use this data as the basis for automatic scaling decisions, ensuring that inference services dynamically expand or shrink based on load and performance requirements, achieving the goal of elastic scaling.
• Intelligent Routing Decisions: Achieve near-real-time data updates through a distributed message queue system, enabling intelligent routing to make decisions quickly based on the latest data, thereby optimizing response speed during AI inference.
• Weight Distribution Acceleration Decisions: Achieve near-real-time data transmission through a distributed message queue system, obtaining real-time network dynamic performance metrics (such as actual transmission rate of node RDMA NICs, remaining available bandwidth, etc.), combined with static metrics (such as theoretical transmission speed of node RDMA NICs, NIC PCIe bandwidth, etc.) attached to nodes as labels by NPU Feature Discovery. Based on these metrics, the module dynamically evaluates node performance and ultimately selects the node with optimal performance for task assignment, thereby ensuring optimal resource utilization and maximizing system stability and processing efficiency.
• Inference Request Fault Quick Localization: When inference requests encounter issues such as slow response, timeout, or failure, view the complete call chain through the Jaeger UI to quickly locate whether the bottleneck is at the AI gateway, routing decision, or a specific stage of the inference engine. Analyze the duration and key attributes of each stage (KVCache hit rate, inference execution duration, TTFT (Time to First Token) / ITL (Inter-Token Latency) and other performance metrics) to precisely identify the root cause of the fault.

Capability Scope ​

• Multi-layer Metric Coverage: Covers AI gateway (such as performance, resource consumption, security and compliance audit, governance policy execution records), inference engine (API Server, model input/output, inference process, inference engine status), Mooncake (Mooncake master, transfer engine, Mooncake client), and infrastructure (Ray, K8s, hardware), achieving full-link observability.
• Near-real-time Metric Transmission: For modules with high time-sensitivity requirements, achieves second-level metric push through a distributed message queue system, ensuring that metrics can be promptly perceived and influence decisions.
• Scaling Decision Support: Synchronously reports collected system and runtime metrics to Prometheus for periodic computation and trend evaluation.
• Hardware Health Check and Diagnosis: Builds an independent hardware health diagnosis module that periodically collects low-level metrics such as NPU/GPU temperature, power consumption, and error codes, and reports them in real time through a distributed message queue system. The diagnosis module subscribes to and analyzes the collected data, combining device model, driver, and firmware information, and based on threshold rules and anomaly metric analysis, identifies typical fault patterns and outputs diagnostic conclusions and remediation suggestions, achieving a closed loop from data collection to health assessment.
• End-to-end Distributed Tracing: Based on the OpenTelemetry standard, provides non-intrusive request-level tracing capability from AI gateway to inference engine (vLLM-Ascend), supporting collection, storage, and querying, recording key performance metrics and business attributes.

Implementation Principle ​

Metric Monitoring ​

Logical View ​

Figure 1 Level-0 Logical View

The monitoring system is divided into backend layer and component layer by business hierarchy.

Backend Layer

• Hardware Health Monitor: The hardware health monitoring module, as an independently running collection component, actively executes metric collection and reporting as periodic tasks. During operation, the module calls low-level interfaces (DCMI, NVML) or parses system logs (dmesg) at fixed collection intervals to obtain device running status and health information. Collection results are published in real time to the diagnosis module through a distributed message queue system, achieving decoupling of collection and diagnosis.
• Hardware Diagnosis: The diagnosis module subscribes to metric data published by the collection module through a distributed message queue system, combining device model, driver, and firmware information to perform real-time analysis of hardware health status. The module supports threshold judgment and anomaly detection, identifies typical fault patterns, and outputs diagnostic conclusions and remediation suggestions, achieving a closed loop from data collection to health assessment.

Component Layer

The component layer provides underlying metric collection, transmission, and presentation capabilities, covering the following key modules: metric collection (Exporter), high-performance distributed message queue system, metric storage (Prometheus), and presentation (Grafana).

Data Flow Diagram ​

NATS ​

Figure 2 Data Flow Diagram - NATS

ZMQ ​

Figure 3 Data Flow Diagram - ZMQ

Full Data List ​

Table 1 Full Data List

Notice: network performance exporter, hardware monitor, hardware diagnosis, and npu exporter all need to enable privileged containers during deployment; otherwise, DCMI cannot be loaded, resulting in data not being correctly collected.

End-to-End Tracing ​

Logical View ​

Figure 4 Level-0 Logical View

vLLM-Ascend Span Structure ​

vllm_ascend.request (API process - full request lifecycle)
├─ vllm_ascend.prefill_execution (API process - Prefill phase, created post-hoc)
│  ├─ vllm_ascend.prefill.schedule (Prefill scheduling aggregate statistics, created post-hoc)
│  ├─ vllm_ascend.prefill.kv_cache (Prefill allocation aggregate statistics, created post-hoc)
│  └─ vllm_ascend.prefill.ascend_store (AscendStore save operation, created post-hoc)
└─ vllm_ascend.decode_phase (API process - Decode phase, created post-hoc)
 ├─ vllm_ascend.decode.schedule (Decode scheduling aggregate statistics, created post-hoc)
 ├─ vllm_ascend.decode.kv_cache (Decode allocation aggregate statistics, created post-hoc)

vLLM-Ascend Span Description ​

Table 2 vLLM-Ascend Span Description Table

Relationship with Related Features ​

• Tracing: Depends on the inference engine (e.g., vLLM-Ascend 0.18.0).

Installation ​

Prerequisites ​

Hardware Requirements ​

Eagle Eye itself has no special hardware environment requirements.

Software Requirements ​

Kubernetes v1.33.0 or above.

Starting Installation ​

openFuyao Platform ​

1. Log in to the openFuyao management plane.

2. Select "Application Market > Application List" from the left navigation bar to navigate to the "Application List" page.

3. Check "Artificial Intelligence/Machine Learning" under "Scenarios" on the left, and find the "eagle-eye" card. Or enter "eagle-eye" in the search box.

4. Click "Deploy" to enter the "Deploy" page.

5. Enter the application name, select the installation version and namespace. You can choose an existing namespace or create a new one; for creating a namespace, see Namespace.

6. Configure eagle-eye in the "values.yaml" under parameter configuration.

   Notice:networkPerformanceExporter.collectInterval is used to configure the network-performance-exporter collection interval (in seconds), with a default value of 15s. If this value is set too short, it may cause the next collection round to be triggered before the previous one completes, resulting in incomplete metric data. It is recommended not to go below the default value of 15s.

7. Click "Deploy" to complete the installation.

Standalone Deployment ​

In addition to openFuyao platform installation and deployment, this feature also provides standalone deployment functionality through the following two methods:

• Obtain the project installation package from the openFuyao official image repository.

1. Execute the following command to pull the project installation package.

   helm pull oci://cr.openfuyao.cn/charts/eagle-eye --version xxx

   Where xxx needs to be replaced with the specific project installation package version, such as 0.0.0-latest. The obtained installation package is in compressed package form.

2. Execute the following command to extract the installation package.

   cd

   tar -zxvf eagle-eye

3. Install and deploy.

   Execute the following command in the same directory as eagle-eye.

   helm install eagle-eye -n xxxxxx .

• Obtain from the openFuyao GitCode repository.

1. Execute the following command to pull the project from the repository.

   git clone https://gitcode.com/openFuyao/eagle-eye.git

2. Install and deploy. Execute the following command in the same directory as eagle-eye.

   helm install eagle-eye -n xxxxxx .

Using Metric Monitoring ​

High Real-time Metric Collection and Reporting for Intelligent Routing ​

NATS ​

To meet the high time-sensitivity requirements of modules such as intelligent routing, the system uses NATS for second-level push of key performance metrics. This enables metrics such as waiting request count and NPU/GPU KVCache utilization to be perceived in real time, thereby dynamically adjusting decisions.

Currently only vLLM deployment is supported.

Prerequisites

• Kubernetes cluster is deployed and accessible.
• kubectl command-line tool is installed with cluster access permissions configured.
• Ascend driver and related dependencies are installed on the nodes.
• NATS is deployed and running normally.

Deployment Steps

The NATS publish/subscribe model is selected as the metric transmission method. CustomStatLogger is a custom statistics data logger implemented in vLLM by inheriting from the StatLoggerBase abstract class. As a publisher, CustomStatLogger actively generates structured runtime metric messages after each decode batch ends and publishes them to the specified NATS topic; Router as a subscriber only needs to register a listener on that topic to receive messages the moment they are published and trigger subsequent processing logic.

To avoid losing the first batch of messages, the subscriber (Router) must complete connection and subscription before the publisher (CustomStatLogger).

1. Router (Subscriber).

   Router as a subscriber is mainly responsible for receiving published runtime metric messages and triggering corresponding processing logic. Refer to the following code example to implement Router and deploy it to the Kubernetes cluster.

   import logging
   import json
   from nats.aio.client import Client as NATS_Client
   
   logger = logging.getLogger()
   
   class RouterSubscriber:
       def __init__(self, server_address: str, topic: str):
           """
           Initialize RouterSubscriber instance
           :param server_address: NATS server address
           :param topic: Topic to subscribe to
           """
           self.server_address = server_address
           self.topic = topic
           self.client = NATS_Client()
   
       async def connect(self):
           """
           Establish connection with NATS server
           """
           try:
               # Establish NATS persistent connection
               await self.client.connect(self.server_address)
               logger.info("Connected to NATS server at %s", self.server_address)
           except Exception as e:
               logger.error("Failed to connect to NATS server: %s", e)
               raise e
   
       async def subscribe(self, topic: str, message_handler):
           """
           Subscribe to the specified topic and pass received messages to business logic processing
           :param message_handler: Business logic processing function for handling received messages
           """
           try:
               async def on_message(msg):
                   """
                   Message callback function, handles subscribed messages
                   :param msg: NATS message
                   """
                   logger.info("Received message on topic %s", topic)
                   await message_handler(msg)
   
               # Ensure NATS connection
               if not self.client.is_connected:
                   await self.connect()
   
               # Register subscription topic and bind message callback function
               await self.client.subscribe(topic, cb=on_message)
               logger.info("Successfully subscribed to NATS topic %s", topic)
   
           except Exception as e:
               logger.error("Failed to subscribe to NATS topic %s: %s", topic, e)
               raise e
               
       async def message_handler(self, msg) -> None:
           """
           NATS raw callback, responsible for:
           1. Decoding data from msg;
           2. Calling parse_message for JSON deserialization;
           3. Passing parsed structured data to subsequent business logic processing.
           """
           try:
               data = msg.data.decode("utf-8")
           except Exception as exc:
               logger.error("Failed to decode NATS message: %s", exc)
               return
   
           data_dict = self.parse_message(data)
           if not data_dict:
               # Return directly on parse failure to avoid subsequent logic exceptions
               return
   
           # TODO: Handle specific business logic here, e.g., make routing decisions based on metrics
           ...
   
       def parse_message(self, data: str) -> Dict[str, Any]:
           """
           Deserialize NATS message body from JSON string to dictionary.
           Logs and returns empty dictionary on parse failure.
           """
           try:
               return json.loads(data)
           except Exception as exc:
               logger.error("Failed to parse message: %s, raw: %r", exc, data)
               return {}

2. Customstatlogger (Publisher).

   CustomStatLogger is the publisher, responsible for generating and publishing real-time runtime metric data. After each decode batch ends, CustomStatLogger publishes structured metric messages to the specified NATS topic (this topic should match the subscriber's topic, currently eagle_eye.routing_metrics) for Router to subscribe.

   2.1 Package source code.

   Execute the following command in the project root directory to package the source code into a compressed file.

   bash

   tar -czf source_code.tar.gz src/ requirements.txt

   2.2 Create ConfigMap.

   Import the compressed package as a binary file into a ConfigMap.

   bash

   # Check if namespace exists, create it if not
   kubectl get namespace ai-inference || kubectl create namespace ai-inference
   
   # If a ConfigMap with the same name already exists, delete it first
   kubectl delete configmap vllm-source-code -n ai-inference --ignore-not-found
   
   # Create new ConfigMap
   kubectl create configmap vllm-source-code \
     --from-file=source_code.tar.gz=source_code.tar.gz \
     -n ai-inference

   2.3 Deploy the service.

   Apply the deployment YAML file.

   bash

   kubectl apply -f ./docs/routing-metrics/vllm_eagle_eye.yaml

   2.4 Check deployment status.

   bash

   # View Pod running status
   kubectl get pods -n ai-inference -l app=vllm-eagle-eye
   
   # View Pod details
   kubectl describe pod -n ai-inference -l app=vllm-eagle-eye
   
   # View Pod logs
   kubectl logs -n ai-inference -l app=vllm-eagle-eye -f

3. Verification steps.

   3.1 Method 1: Verify using a temporary Pod.

   3.1.1 Get the service Pod IP address.

   bash

   kubectl get pod -n ai-inference -l app=vllm-eagle-eye -o wide

   Record the IP address from the output (e.g., 10.244.1.5).

   3.1.2 Start a temporary test Pod.

   bash

   kubectl run curl-test --image=curlimages/curl -n ai-inference -it --rm --restart=Never -- sh

   3.1.3 Send a test request from the temporary Pod.

   Execute in the temporary Pod's shell (replace <POD_IP> with the actual IP address):

   bash

   curl -X POST http://<POD_IP>:8000/generate \
     -H "Content-Type: application/json" \
     -d '{"prompt": "Hello AI"}'

   3.2 Method 2: Verify using port-forward.

   bash

   # Port forwarding
   kubectl port-forward -n ai-inference deployment/vllm-eagle-eye 8000:8000
   
   # Send request from another terminal
   curl -X POST http://localhost:8000/generate \
     -H "Content-Type: application/json" \
     -d '{"prompt": "Hello AI"}'

4. Clean up resources.

   bash

   # Delete Deployment
   kubectl delete -f ./docs/routing-metrics/vllm_eagle_eye.yaml
   
   # Delete ConfigMap
   kubectl delete configmap vllm-source-code -n ai-inference
   
   # Delete temporary Pod
   kubectl delete pod -n ai-inference curl-test

Common Issues

1. Pod fails to start.

   bash

   # View Pod events
   kubectl describe pod -n ai-inference -l app=vllm-eagle-eye
   
   # View init container logs
   kubectl logs -n ai-inference <pod-name> -c code-extractor

2. Dependency installation failure.

   If the cluster cannot access the internet, dependencies need to be pre-packaged.

   bash

   # Download dependencies locally
   pip download -r requirements.txt -d packages/
   
   # Include dependencies when packaging
   tar -czf source_code.tar.gz src/ requirements.txt packages/

Using Tracing ​

vLLM-Ascend Trace Data Reporting Configuration ​

Eagle Eye automatically installs tracing backend components (Jaeger + Elasticsearch) when deployed via Helm Chart. Users need to configure the OTLP reporting address on the inference engine side to report Trace data to Eagle Eye's Jaeger.

Prerequisites

• Kubernetes cluster is deployed and accessible.
• kubectl command-line tool is installed with cluster access permissions configured.
• Ascend driver and related dependencies are installed on the nodes.
• Eagle Eye has been successfully deployed via Helm Chart (Eagle Eye tracing backend is enabled by default, including Jaeger and Elasticsearch).
• The inference engine is vLLM-Ascend.

Deployment Steps

1. Get the Jaeger service address.

   bash

   kubectl get svc -n <release-namespace> | grep eagle-eye-tracing-collector

   Record the Jaeger Collector service port (default OTLP/gRPC port is 4317).

2. Modify vLLM-Ascend deployment configuration.

   Add tracing-related configuration to the vLLM-Ascend deployment configuration, including initContainers, environment variables, startup parameters, volume mounts, etc.

   Complete example (replace placeholders with actual values):

   yaml

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: vllm-ascend
     namespace: ai-inference
   spec:
     template:
       spec:
         # initContainer: copy tracing files to shared directory
         initContainers:
         - name: install-tracing
           image: cr.openfuyao.cn/openfuyao/eagle-eye-tracing:latest
           imagePullPolicy: IfNotPresent
           command: ["cp", "-r", "/tracing-source/.", "/tracing-install/"]
           volumeMounts:
           - name: tracing-install
             mountPath: /tracing-install
   
         containers:
         - name: vllm
           image: <your-vllm-ascend-image>:<tag>
           imagePullPolicy: IfNotPresent
           command: ["/bin/bash", "-c"]
           args:
           - |
             set -euo pipefail
   
             # Install tracing files to Python environment
             /tracing-install/install.sh
   
             # Other initialization commands...
   
             # Start vLLM, add --otlp-traces-endpoint parameter
             exec vllm serve <model-name> \
               --served-model-name <model-name> \
               --trust-remote-code \
               --otlp-traces-endpoint http://eagle-eye-tracing-collector.<release-namespace>.svc.cluster.local:4317 \
               --port 8000
               # Other parameters...
   
           # Add environment variables
           env:
           - name: OTEL_SERVICE_NAME
             valueFrom:
               fieldRef:
                 fieldPath: metadata.name
           - name: VLLM_ASCEND_TRACING_ENABLED
             value: "1"
           - name: VLLM_ASCEND_TRACING_LOG_LEVEL
             value: "INFO"
   
           # Add volume mounts
           volumeMounts:
           - name: tracing-install
             mountPath: /tracing-install
             readOnly: true
   
         # Add volume definition
         volumes:
         - name: tracing-install
           emptyDir: {}

   Note: Placeholders such as <your-vllm-ascend-image>:<tag>, <model-name>, <release-namespace> in the configuration need to be replaced with actual values.

   Configuration Description:

   Table 3 vLLM-Ascend Tracing Configuration Description

   Configuration Item	Description
   initContainers.install-tracing	Copies files from tracing image to shared volume.
   --otlp-traces-endpoint	OTLP receiving address of Jaeger Collector.
   OTEL_SERVICE_NAME	OpenTelemetry service name, uses Pod name.
   VLLM_ASCEND_TRACING_ENABLED	Enable vLLM-Ascend tracing.
   VLLM_ASCEND_TRACING_LOG_LEVEL	Tracing log level.
   /tracing-install/install.sh	Installs tracing module to Python site-packages.

3. Apply configuration.

   bash

   kubectl apply -f vllm-ascend-deployment.yaml

   Wait for the Pod to start, check Pod running status:

   bash

   kubectl get pods -n ai-inference | grep vllm-ascend

   Confirm Pod status is Running and all initContainers have completed successfully.

Querying Jaeger UI ​

1. Confirm tracing-related Pods are running normally.

   bash

   kubectl get pods -n <release-namespace> | grep eagle-eye-tracing

   Confirm all related Pod statuses are Running.

2. Send inference requests.

   bash

   curl -X POST http://<vllm-service-address>:8000/v1/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model": "<model-name>",
       "prompt": "Hello, how are you?",
       "max_tokens": 50
     }'

   Replace <vllm-service-address> with the actual vLLM service address.

3. Get the Jaeger UI access address.

   bash

   kubectl get svc -n <release-namespace> | grep eagle-eye-tracing-query

   Record the NodePort port number (e.g., 30686) and node IP address from the output.

4. Access Jaeger UI.

   Access http://<node-ip>:<node-port> (e.g., http://192.168.1.100:30686) in the browser to enter the Jaeger UI.

5. View Trace data.

   Select vllm-ascend from the Service drop-down list in the Jaeger UI, click the Find Traces button, and confirm that the Trace list is visible.

6. View Trace details.

   Click a Trace to expand and view detailed information, confirming that you can see Spans for each stage:

   • vllm_ascend.request: Full request lifecycle.
   • vllm_ascend.prefill_execution: Prefill phase.
   • vllm_ascend.decode_phase: Decode phase.
   • And sub-Spans for each stage (scheduling, KVCache allocation, etc.).