Eagle Eye for AI Inference ​

Feature Overview ​

Eagle Eye is an observability system for AI inference scenarios, implementing full-link metric collection from AI gateways, inference engines, Mooncake to infrastructure (Ray, K8s, hardware), near real-time transmission and intelligent diagnosis. The system integrates Prometheus's periodic metric collection with the low-latency push mechanism of distributed message queue systems, supporting both trend analysis for scaling decisions and meeting the requirements of modules with high timeliness needs (such as intelligent routing) for second-level data updates. Through an independent hardware health diagnosis module, continuous monitoring and anomaly identification of underlying metrics such as NPU/GPU, temperature, power consumption, and error codes are achieved, building a closed-loop monitoring capability of "collection—transmission—diagnosis—evaluation" to provide solid data support for AI inference system stability, performance optimization, and resource scheduling.

Application Scenarios ​

• System Resource Health Monitoring: Real-time monitoring of infrastructure (Ray, K8s, hardware) to ensure system resource health and stability, promptly discovering and resolving resource bottlenecks to ensure system efficiency.
• Inference Process Performance Optimization: Real-time monitoring of performance metrics (such as latency, throughput) and resource usage at each stage of the inference process (such as prefill, decode, etc.), identifying and analyzing performance bottlenecks, optimizing model execution efficiency, and improving response speed and computational efficiency of inference tasks.
• Hardware Fault Diagnosis and Repair: View abnormality analysis reports provided by the hardware diagnosis module, which include fault pattern identification and handling recommendations to help quickly locate and resolve hardware faults. The system can monitor NPU/GPU, temperature, power consumption, and other hardware states in real time, generate detailed fault analysis reports, provide specific fault causes and repair solutions, ensuring hardware stability and reliability.
• Automatic Scaling Decisions: Obtain SLA-related metrics (such as throughput rate, latency, etc.) and use this data as the basis for automatic scaling decisions, ensuring that inference services dynamically expand or shrink according to load and performance requirements to achieve elastic scaling goals.
• Intelligent Routing Decisions: Implement near real-time data updates through distributed message queue systems, enabling intelligent routing to make decisions quickly based on the latest data, thereby optimizing response speed in the AI inference process.
• Weight Distribution Acceleration Decisions: Implement near real-time data transmission through distributed message queue systems to obtain network dynamic performance metrics in real time (such as actual transmission rate of node RDMA network cards, remaining available bandwidth, etc.), and combine with static metrics attached to nodes in label form by NPU Feature Discovery (such as theoretical transmission speed of node RDMA network cards, PCIe bandwidth of network cards, etc.). Based on these metrics, the module dynamically evaluates node performance and ultimately selects the node with the best performance for task allocation, ensuring optimal resource utilization and maximizing system stability and processing efficiency.

Capability Scope ​

• Multi-layer Metric Coverage: Covers AI gateways (such as performance, resource consumption, security and compliance audits, governance policy execution tracking), inference engines (API Server, model input/output, inference process, inference engine status), Mooncake (Mooncake master, transfer engine, Mooncake client) and infrastructure (Ray, K8S, hardware), achieving full-link observation.
• Near Real-time Metric Transmission: For modules with high timeliness requirements, second-level metric push is implemented through distributed message queue systems to ensure metrics can be perceived promptly and influence decisions.
• Scaling Decision Support: Synchronize collected system and runtime metrics to Prometheus for periodic calculation and trend evaluation.
• Hardware Health Check and Diagnosis: Build an independent hardware health diagnosis module that periodically collects underlying metrics such as NPU/GPU temperature, power consumption, and error codes, and reports them in real time through distributed message queue systems. The diagnosis module subscribes to and analyzes collected data, combines device model, driver, and firmware information, identifies typical fault patterns based on threshold rules and abnormal metric analysis, and outputs diagnostic conclusions and handling recommendations, achieving a closed loop from data collection to health assessment.

Implementation Principle ​

Logical View ​

Figure 1 Layer 0 Logical View

The monitoring system is divided into backend layer and component layer according to business hierarchy.

Backend Layer

• Hardware Health Monitor: The hardware health detection module operates as an independent collection component, actively executing metric collection and reporting in a periodic task manner. During operation, the module calls underlying interfaces (DCMI, NVML) or parses system logs (dmesg) according to a fixed collection cycle to obtain device running status and health information. Collection results are published in real time to the diagnosis module through distributed message queue systems, achieving decoupling of collection and diagnosis.
• Hardware Diagnosis: The diagnosis module subscribes to metric data published by the collection module through distributed message queue systems, combines device model, driver, and firmware information to analyze hardware health status in real time. The module supports threshold judgment and anomaly detection, identifies typical fault patterns and outputs diagnostic conclusions and handling recommendations, achieving a closed loop from data collection to health assessment.

Component Layer

The component layer provides underlying metric collection, transmission, and display capabilities, covering the following key modules: metric collection (Exporter), high-performance distributed message queue systems, metric storage (Prometheus), and display (Grafana).

Data Flow Diagram ​

NATS ​

Figure 2 Data Flow Diagram-NATS

ZMQ ​

Figure 3 Data Flow Diagram-ZMQ

Complete Data List ​

Table 1 Complete Data List

Relationship with Related Features ​

Depends on inference engines (such as vLLM).

Installation ​

Prerequisites ​

Hardware Requirements ​

Eagle Eye itself has no special hardware environment requirements.

Software Requirements ​

Kubernetes v1.33.0 and above.

Installation Steps ​

openFuyao Platform ​

1. Log in to the openFuyao management dashboard.
2. Select "Application Market > Application List" in the left navigation bar to jump to the "Application List" interface.
3. Check "Artificial Intelligence/Machine Learning" in the left "Scenario" section and find the "eagle-eye" card. Or enter "eagle-eye" in the search box.
4. Click "Deploy" to enter the "Deploy" interface.
5. Enter the application name, select the installation version and namespace. You can select an existing namespace or create a new namespace. For creating namespaces, refer to Namespaces.
6. Configure routing in "values.yaml" in parameter configuration.
7. Click "Deploy" to complete installation.

Standalone Deployment ​

In addition to openFuyao platform installation and deployment, this feature also provides standalone deployment functionality, with the following two methods:

• Obtain the project installation package from the openFuyao official image repository.

1. Execute the following command to pull the project installation package.

   bash

   helm pull oci://cr.openfuyao.cn/charts/eagle-eye --version xxx

   Where xxx needs to be replaced with the specific project installation package version, such as 0.0.0-latest. The pulled installation package is in compressed package form.

2. Execute the following command to extract the installation package.

   bash

   tar -zxvf eagle-eye

3. Install and deploy.

   Execute the following command in the same directory as eagle-eye.

   bash

   helm install eagle-eye -n xxxxxx .

• Obtain from the openFuyao GitCode repository.

1. Execute the following command to pull the project from the repository.

   bash

   git clone https://gitcode.com/openFuyao/eagle-eye.git

2. Install and deploy. Execute the following command in the same directory as eagle-eye.

   bash

   helm install eagle-eye -n xxxxxx .

High Real-time Metric Collection and Reporting for Intelligent Routing ​

NATS ​

To meet the high timeliness requirements of modules such as intelligent routing, the system uses NATS for second-level push of key performance metrics. This enables metrics such as waiting request count and NPU/GPU KVCache utilization to be perceived in real time, thereby dynamically adjusting decisions.

Currently only vLLM deployment is supported.

Prerequisites ​

• Kubernetes cluster is deployed and accessible
• kubectl command line tool is installed and cluster access permissions are configured
• Ascend driver and related dependencies are installed on nodes
• NATS is deployed and running normally

Deployment Steps ​

The NATS publish/subscribe model is used as the metric transmission method. CustomStatLogger is a custom statistics data logger implemented in vLLM by inheriting the StatLoggerBase abstract class. As a publisher, CustomStatLogger actively generates structured operation metric messages after each decoding batch ends and publishes them to the specified NATS topic; Router, as a subscriber, only needs to register a listener on this topic to receive and trigger subsequent processing logic at the moment the message is published.

To avoid the loss of the first batch of messages, the subscriber end (Router) needs to complete connection and subscription before the publisher end (CustomStatLogger).

1. Router (Subscriber).

   Router, as a subscriber, is mainly responsible for receiving published operation metric messages and triggering corresponding processing logic. Refer to the following code example to implement Router and deploy it to the Kubernetes cluster.

   python

   import logging
   import json
   from nats.aio.client import Client as NATS_Client
   
   logger = logging.getLogger()
   
   class RouterSubscriber:
       def __init__(self, server_address: str, topic: str):
           """
           Initialize RouterSubscriber instance
           :param server_address: NATS server address
           :param topic: Topic to subscribe to
           """
           self.server_address = server_address
           self.topic = topic
           self.client = NATS_Client()
   
       async def connect(self):
           """
           Establish connection with NATS server
           """
           try:
               # Establish NATS long connection
               await self.client.connect(self.server_address)
               logger.info("Connected to NATS server at %s", self.server_address)
           except Exception as e:
               logger.error("Failed to connect to NATS server: %s", e)
               raise e
   
       async def subscribe(self, topic: str, message_handler):
           """
           Subscribe to specified topic and pass received messages to business logic processing
           :param message_handler: Business logic processing function for handling received messages
           """
           try:
               async def on_message(msg):
                   """
                   Message callback function to handle subscribed messages
                   :param msg: NATS message
                   """
                   logger.info("Received message on topic %s", topic)
                   await message_handler(msg)
   
               # Ensure NATS connection
               if not self.client.is_connected:
                   await self.connect()
   
               # Register subscription topic and bind message callback function
               await self.client.subscribe(topic, cb=on_message)
               logger.info("Successfully subscribed to NATS topic %s", topic)
   
           except Exception as e:
               logger.error("Failed to subscribe to NATS topic %s: %s", topic, e)
               raise e
               
       async def message_handler(self, msg) -> None:
           """
           NATS original callback, responsible for:
           1. Decoding data from msg;
           2. Calling parse_message for JSON deserialization;
           3. Passing parsed structured data to subsequent business logic processing.
           """
           try:
               data = msg.data.decode("utf-8")
           except Exception as exc:
               logger.error("Failed to decode NATS message: %s", exc)
               return
   
           data_dict = self.parse_message(data)
           if not data_dict:
               # Return directly if parsing fails to avoid subsequent logic exceptions
               return
   
           # TODO: Process specific business logic here, such as making routing decisions based on metrics
           ...
   
       def parse_message(self, data: str) -> Dict[str, Any]:
           """
           Deserialize NATS message body from JSON string to dictionary.
           Log and return empty dictionary if parsing fails.
           """
           try:
               return json.loads(data)
           except Exception as exc:
               logger.error("Failed to parse message: %s, raw: %r", exc, data)
               return {}

2. Customstatlogger (Publisher).

   CustomStatLogger is the publisher, responsible for generating and publishing real-time operation metric data. After each decoding batch ends, CustomStatLogger publishes structured metric messages to the specified NATS topic (this topic should match the subscriber's topic, currently topic eagle_eye.routing_metrics) for Router subscription.

   2.1 Package source code.

   Execute the following command in the project root directory to package the source code into a compressed file.

   bash

   tar -czf source_code.tar.gz src/ requirements.txt

   2.2 Create ConfigMap.

   Import the compressed package into ConfigMap as a binary file.

   bash

   # Check if namespace exists, create it if it doesn't exist
   kubectl get namespace ai-inference || kubectl create namespace ai-inference
   
   # Delete if ConfigMap with the same name already exists
   kubectl delete configmap vllm-source-code -n ai-inference --ignore-not-found
   
   # Create new ConfigMap
   kubectl create configmap vllm-source-code \
     --from-file=source_code.tar.gz=source_code.tar.gz \
     -n ai-inference

   2.3 Deploy service.

   Apply deployment YAML file.

   bash

   kubectl apply -f ./docs/routing-metrics/vllm_eagle_eye.yaml

   2.4 Check deployment status.

   bash

   # Check Pod running status
   kubectl get pods -n ai-inference -l app=vllm-eagle-eye
   
   # Check Pod detailed information
   kubectl describe pod -n ai-inference -l app=vllm-eagle-eye
   
   # Check Pod logs
   kubectl logs -n ai-inference -l app=vllm-eagle-eye -f

3. Verification steps.

   3.1 Method 1: Verify using temporary Pod.

   3.1.1 Get service Pod IP address.

   bash

   kubectl get pod -n ai-inference -l app=vllm-eagle-eye -o wide

   Record the IP address in the output (for example: 10.244.1.5).

   3.1.2 Start temporary test Pod.

   bash

   kubectl run curl-test --image=curlimages/curl -n ai-inference -it --rm --restart=Never -- sh

   3.1.3 Send test request in temporary Pod.

   Execute in the temporary Pod's shell (replace <POD_IP> with actual IP address):

   bash

   curl -X POST http://<POD_IP>:8000/generate \
     -H "Content-Type: application/json" \
     -d '{"prompt": "Hello AI"}'

   3.2 Method 2: Verify using port-forward.

   bash

   # Port forwarding
   kubectl port-forward -n ai-inference deployment/vllm-eagle-eye 8000:8000
   
   # Send request in another terminal
   curl -X POST http://localhost:8000/generate \
     -H "Content-Type: application/json" \
     -d '{"prompt": "Hello AI"}'

4. Clean up resources.

   bash

   # Delete Deployment
   kubectl delete -f ./docs/routing-metrics/vllm_eagle_eye.yaml
   
   # Delete ConfigMap
   kubectl delete configmap vllm-source-code -n ai-inference
   
   # Delete temporary Pod
   kubectl delete pod -n ai-inference curl-test

Common Issues ​

1. Pod startup failure.

   bash

   # Check Pod events
   kubectl describe pod -n ai-inference -l app=vllm-eagle-eye
   
   # Check init container logs
   kubectl logs -n ai-inference <pod-name> -c code-extractor

2. Dependency installation failure.

   If the cluster cannot access the internet, dependency packages need to be packed into the compressed package in advance.

   bash

   # Download dependencies locally
   pip download -r requirements.txt -d packages/
   
   # Include dependencies when packaging
   tar -czf source_code.tar.gz src/ requirements.txt packages/