AI Inference Eagle Eye ​

Feature Introduction ​

Eagle Eye is an observability system designed for AI inference scenarios, implementing full-chain metric collection, near real-time transmission, and intelligent diagnosis from AI gateway, inference engine, Mooncake to infrastructure (Ray, K8s, hardware). This system integrates Prometheus periodic metric collection with distributed message queue system low-latency push mechanisms, supporting both trend analysis for scaling decisions and meeting the needs of modules with high timeliness requirements (such as intelligent routing) for second-level data updates. Through an independent hardware health diagnosis module, it achieves continuous monitoring and anomaly identification of underlying metrics such as NPU/GPU, temperature, power consumption, and error codes, constructing a closed-loop monitoring capability of "collection — transmission — diagnosis — assessment", providing solid data support for the stability, performance optimization, and resource scheduling of AI inference systems.

Application Scenarios ​

• System Resource Health Monitoring: Real-time monitoring of infrastructure (Ray, K8s, hardware), ensuring health and stability of system resources, timely discovering and resolving resource bottlenecks, ensuring efficient system operation.
• Inference Process Performance Optimization: Real-time monitoring of performance metrics (such as latency, throughput) and resource usage at various stages of the inference process (such as prefill, decode, etc.), identifying and analyzing performance bottlenecks, optimizing model execution efficiency, improving response speed and computational efficiency of inference tasks.
• Hardware Fault Diagnosis and Repair: Viewing anomaly analysis reports provided by the hardware diagnosis module, which include fault pattern identification and handling recommendations, helping quickly locate and resolve hardware faults. The system can monitor NPU/GPU, temperature, power consumption, and other hardware states in real-time, generate detailed fault analysis reports, provide specific fault causes and repair solutions, ensuring hardware stability and reliability.
• Auto Scaling Decision: Obtaining SLA-related metrics (such as throughput rate, latency, etc.), using these data as the basis for auto scaling decisions, ensuring inference services dynamically expand or shrink according to load and performance requirements, achieving elastic scaling goals.
• Intelligent Routing Decision: Achieving near real-time data updates through distributed message queue system, enabling intelligent routing to make decisions quickly based on the latest data, thereby optimizing response speed during AI inference processes.
• Weight Distribution Acceleration Decision: Achieving near real-time data transmission through distributed message queue system, obtaining real-time network dynamic performance metrics (such as actual transmission rate of node RDMA network cards, remaining available bandwidth, etc.), and combining static metrics (such as theoretical transmission speed of node RDMA network cards, network card PCIe bandwidth, etc.) attached to nodes by NPU Feature Discovery in label form. Based on these metrics, the module dynamically evaluates node performance, ultimately selecting the node with optimal performance for task assignment, thereby ensuring optimal resource utilization and maximizing system stability and processing efficiency.

Capability Scope ​

• Multi-layer Metric Coverage: Covers AI gateway (such as performance, resource consumption, security and compliance auditing, governance policy execution records), inference engine (API Server, model input/output, inference process, inference engine status), Mooncake (Mooncake master, transfer engine, Mooncake client), and infrastructure (Ray, K8S, hardware), achieving full-chain observability.
• Near Real-time Metric Transmission: For modules with high timeliness requirements, achieving second-level metric push through distributed message queue system, ensuring metrics can be timely perceived and influence decisions.
• Scaling Decision Support: Synchronizing collected system and runtime metrics to Prometheus for periodic calculation and trend assessment.
• Hardware Health Check and Diagnosis: Building an independent hardware health diagnosis module, periodically collecting underlying metrics such as NPU/GPU temperature, power consumption, error codes, etc., and reporting them in real-time through distributed message queue system. The diagnosis module subscribes to and analyzes collected data, combining device model, driver and firmware information, based on threshold rules and abnormal metric analysis, identifying typical fault patterns and outputting diagnosis conclusions and handling recommendations, achieving a closed loop from data collection to health assessment.

Implementation Principle ​

Logical View ​

Figure 1 Layer 0 Logical View

The monitoring system is divided into backend layer and component layer according to business hierarchy.

Backend Layer

• Hardware Health Monitor: The hardware health monitoring module operates as an independent collection component, actively executing metric collection and reporting as periodic tasks. During operation, the module calls underlying interfaces (DCMI, NVML) or parses system logs (dmesg) at fixed collection intervals to obtain device running status and health information. Collection results are published in real-time through distributed message queue system to the diagnosis module, achieving decoupling of collection and diagnosis.
• Hardware Diagnosis: The diagnosis module subscribes to metric data published by the collection module through distributed message queue system, combining device model, driver and firmware information to perform real-time analysis of hardware health status. The module supports threshold judgment and anomaly identification, identifying typical fault patterns and outputting diagnosis conclusions and handling recommendations, achieving a closed loop from data collection to health assessment.

Component Layer

The component layer provides underlying metric collection, transmission, and display capabilities, covering the following key modules: metric collection (Exporter), high-performance distributed message queue system, metric storage (Prometheus), and display (Grafana).

Data Flow Diagram ​

NATS ​

Figure 2 Data Flow Diagram - NATS

ZMQ ​

Figure 3 Data Flow Diagram - ZMQ

Full Data List ​

Table 1 Full Data List

Notice: network performance exporter, hardware monitor, hardware diagnosis, and npu exporter must be deployed with privileged containers enabled, otherwise DCMI cannot be loaded, causing data to not be collected correctly.

Relationship with Related Features ​

Depends on inference engine (such as vLLM).

Installation ​

Prerequisites ​

Hardware Requirements ​

Eagle Eye itself has no special hardware environment requirements.

Software Requirements ​

Kubernetes v1.33.0 or above.

Start Installation ​

openFuyao Platform ​

1. Log in to openFuyao management plane.

2. Select "Application Marketplace > Application List" in the left navigation bar to jump to the "Application List" interface.

3. Check "Artificial Intelligence / Machine Learning" in "Scenarios" on the left side, find the "eagle-eye" card. Or enter "eagle-eye" in the search box.

4. Click "Deploy" to enter the "Deploy" interface.

5. Enter application name, select installation version and namespace. Namespace can use existing namespace or create a new namespace, for creating namespace please refer to Namespace.

6. Configure eagle-eye in the "values.yaml" in parameter configuration.

   Notice: networkPerformanceExporter.collectInterval is used to configure network-performance-exporter collection interval (unit: seconds), default value is 15s. If this value is configured too short, it may cause the next collection task to trigger before the previous one completes, resulting in incomplete metric data. It is recommended not to be lower than the default value of 15s.

7. Click "Deploy" to complete installation.

Standalone Deployment ​

Besides openFuyao platform installation and deployment, this feature also provides standalone deployment functionality, with the following two ways:

• Obtain project installation package from openFuyao official image repository.

1. Execute the following command to pull project installation package.

   helm pull oci://cr.openfuyao.cn/charts/eagle-eye --version xxx

   Where xxx needs to be replaced with specific project installation package version, such as 0.0.0-latest. The pulled installation package is in compressed package format.

2. Execute the following command to extract installation package.

   cd

   tar -zxvf eagle-eye

3. Install and deploy.

   Execute the following command in the same level directory as eagle-eye.

   helm install eagle-eye -n xxxxxx .

• Obtain from openFuyao GitCode repository.

1. Execute the following command to clone project from repository.

   git clone https://gitcode.com/openFuyao/eagle-eye.git

2. Install and deploy. Execute the following command in the same level directory as eagle-eye.

   helm install eagle-eye -n xxxxxx .

High Real-time Metric Collection and Reporting for Intelligent Routing ​

NATS ​

To meet the high timeliness requirements of modules such as intelligent routing, the system uses NATS for second-level push of key performance metrics. This enables metrics such as waiting request count and NPU/GPU KVCache utilization to be perceived in real-time, thereby dynamically adjusting decisions.

Currently only supports vLLM deployment.

Prerequisites ​

• Kubernetes cluster has been deployed and can be accessed normally
• kubectl command line tool has been installed and cluster access permissions configured
• Ascend driver and related dependencies have been installed on nodes
• NATS has been deployed and running normally

Deployment Steps ​

Select NATS publish/subscribe model as the metric transmission method. CustomStatLogger is a custom statistics data recorder implemented in vLLM by inheriting the StatLoggerBase abstract class. As the publisher, CustomStatLogger generates structured runtime metric messages after each decode batch ends, and publishes them to the specified NATS topic; Router as the subscriber only needs to register listening on that topic to receive messages instantly upon publication and trigger subsequent processing logic.

To avoid first message loss, the subscriber (Router) must complete connection and subscription before the publisher (CustomStatLogger).

1. Router (Subscriber).

   Router as the subscriber mainly receives published runtime metric messages and triggers corresponding processing logic. Refer to the following code example to implement Router and deploy it to the Kubernetes cluster.

   import logging
   import json
   from nats.aio.client import Client as NATS_Client
   
   logger = logging.getLogger()
   
   class RouterSubscriber:
       def __init__(self, server_address: str, topic: str):
           """
           Initialize RouterSubscriber instance
           :param server_address: NATS server address
           :param topic: Topic to subscribe to
           """
           self.server_address = server_address
           self.topic = topic
           self.client = NATS_Client()
   
       async def connect(self):
           """
           Establish connection with NATS server
           """
           try:
               # Establish NATS persistent connection
               await self.client.connect(self.server_address)
               logger.info("Connected to NATS server at %s", self.server_address)
           except Exception as e:
               logger.error("Failed to connect to NATS server: %s", e)
               raise e
   
       async def subscribe(self, topic: str, message_handler):
           """
           Subscribe to specified topic and hand received messages to business logic processing
           :param message_handler: Business logic processing function for handling received messages
           """
           try:
               async def on_message(msg):
                   """
                   Message callback function, handles subscribed messages
                   :param msg: NATS message
                   """
                   logger.info("Received message on topic %s", topic)
                   await message_handler(msg)
   
               # Ensure NATS connection
               if not self.client.is_connected:
                   await self.connect()
   
               # Register subscription topic and bind message callback function
               await self.client.subscribe(topic, cb=on_message)
               logger.info("Successfully subscribed to NATS topic %s", topic)
   
           except Exception as e:
               logger.error("Failed to subscribe to NATS topic %s: %s", topic, e)
               raise e
               
       async def message_handler(self, msg) -> None:
           """
           NATS raw callback, responsible for:
           1. Decoding data from msg;
           2. Calling parse_message for JSON deserialization;
           3. Handing parsed structured data to subsequent business logic processing.
           """
           try:
               data = msg.data.decode("utf-8")
           except Exception as exc:
               logger.error("Failed to decode NATS message: %s", exc)
               return
   
           data_dict = self.parse_message(data)
           if not data_dict:
               # Return directly on parse failure, avoid subsequent logic exceptions
               return
   
           # TODO: Handle specific business logic here, such as making routing decisions based on metrics
           ...
   
       def parse_message(self, data: str) -> Dict[str, Any]:
           """
           Deserialize NATS message body from JSON string to dictionary.
           Log and return empty dictionary on parse failure.
           """
           try:
               return json.loads(data)
           except Exception as exc:
               logger.error("Failed to parse message: %s, raw: %r", exc, data)
               return {}

2. Customstatlogger (Publisher).

   CustomStatLogger is the publisher, responsible for generating and publishing real-time runtime metric data. After each decode batch ends, CustomStatLogger publishes structured metric messages to the specified NATS topic (this topic should match the subscriber's topic, current topic eagle_eye.routing_metrics), for Router to subscribe.

   2.1 Package source code.

   Execute the following command in the project root directory to package source code into a compressed file.

   bash

   tar -czf source_code.tar.gz src/ requirements.txt

   2.2 Create ConfigMap.

   Import the compressed package as a binary file into ConfigMap.

   bash

   # Check if namespace exists, create it if not
   kubectl get namespace ai-inference || kubectl create namespace ai-inference
   
   # If ConfigMap with same name already exists, delete it first
   kubectl delete configmap vllm-source-code -n ai-inference --ignore-not-found
   
   # Create new ConfigMap
   kubectl create configmap vllm-source-code \
     --from-file=source_code.tar.gz=source_code.tar.gz \
     -n ai-inference

   2.3 Deploy service.

   Apply deployment YAML file.

   bash

   kubectl apply -f ./docs/routing-metrics/vllm_eagle_eye.yaml

   2.4 Check deployment status.

   bash

   # View Pod running status
   kubectl get pods -n ai-inference -l app=vllm-eagle-eye
   
   # View Pod detailed information
   kubectl describe pod -n ai-inference -l app=vllm-eagle-eye
   
   # View Pod logs
   kubectl logs -n ai-inference -l app=vllm-eagle-eye -f

3. Verification steps.

   3.1 Method 1: Use temporary Pod for verification.

   3.1.1 Get service Pod IP address.

   bash

   kubectl get pod -n ai-inference -l app=vllm-eagle-eye -o wide

   Record the IP address in the output (for example: 10.244.1.5).

   3.1.2 Start temporary test Pod.

   bash

   kubectl run curl-test --image=curlimages/curl -n ai-inference -it --rm --restart=Never -- sh

   3.1.3 Send test request in temporary Pod.

   Execute in the temporary Pod's shell (replace <POD_IP> with actual IP address):

   bash

   curl -X POST http://<POD_IP>:8000/generate \
     -H "Content-Type: application/json" \
     -d '{"prompt": "Hello AI"}'

   3.2 Method 2: Use port-forward for verification.

   bash

   # Port forwarding
   kubectl port-forward -n ai-inference deployment/vllm-eagle-eye 8000:8000
   
   # Send request in another terminal
   curl -X POST http://localhost:8000/generate \
     -H "Content-Type: application/json" \
     -d '{"prompt": "Hello AI"}'

4. Clean up resources.

   bash

   # Delete Deployment
   kubectl delete -f ./docs/routing-metrics/vllm_eagle_eye.yaml
   
   # Delete ConfigMap
   kubectl delete configmap vllm-source-code -n ai-inference
   
   # Delete temporary Pod
   kubectl delete pod -n ai-inference curl-test

Common Issues ​

1. Pod startup failure.

   bash

   # View Pod events
   kubectl describe pod -n ai-inference -l app=vllm-eagle-eye
   
   # View init container logs
   kubectl logs -n ai-inference <pod-name> -c code-extractor

2. Dependency installation failure.

   If the cluster cannot access the internet, dependencies need to be pre-packaged into the compressed package.

   bash

   # Download dependencies locally
   pip download -r requirements.txt -d packages/
   
   # Include dependencies when packaging
   tar -czf source_code.tar.gz src/ requirements.txt packages/