AI Inference Integrated Deployment

Feature Introduction

AI Inference Integrated Deployment (InferNex) is an end-to-end integrated deployment solution designed for optimizing AI inference services in cloud-native environments. This solution is built on Kubernetes Gateway API Inference Extension (GIE) and mainstream LLM technology stacks, seamlessly integrating core acceleration modules including open-source gateways, intelligent routing, high-performance inference backends, global KVCache management, scaling decision frameworks, and inference observability systems through Helm Charts. It provides a complete acceleration chain from request ingress, dynamic routing, inference execution to resource management and monitoring, aiming to improve inference throughput and reduce TTFT/TPOT latency, achieving an efficient one-stop AI service deployment experience.

Application Scenarios

Aggregate Inference Scenarios: Supports aggregate inference architecture, suitable for small to medium-scale inference scenarios.
PD Separation Inference Scenarios: Supports Prefill-Decode separation inference architecture, suitable for large-scale, high-throughput inference scenarios.
AI Inference Software Suite Scenarios: Supports the original AI inference software suite mode, suitable for software deployment in integrated machine scenarios. For specific feature usage methods, please refer to the Guide.

Capability Scope

Supports users to optionally install open-source gateways, intelligent routing, KVCache index management, and inference observability components according to scenario requirements.
Intelligent routing supports integration with multiple open-source gateways, which need to adapt to GIE.
Intelligent routing provides advanced routing strategies such as KVCache aware, PD bucket scheduling, and latency prediction, supporting optimized inference request scheduling in various scenarios.
Intelligent routing provides disaster recovery capabilities, including automatic traffic switching, fault awareness, and request retry.
KVCache index management maintains a two-layer KVCache view of inference instance VRAM (L1) and external cache management component-managed memory (L3), supporting intelligent routing to obtain comprehensive and accurate request KVCache hit rates.
Supports deploying inference engines in both aggregate/PD separation modes using LeaderWorkerSet (LWS), and can configure the number of inference engine nodes.
Supports configuring Mooncake as the distributed KVCache management backend.
Supports appending additional startup parameters on top of built-in vLLM startup commands and regular configuration items (model length, batch size, parallel strategy, external DP load balancing, memory utilization, block size, etc.).
Supports configuring different versions of vLLM inference engines.
Supports fine-grained resource configuration at the inference engine node level, including CPU limits, memory limits, environment variables, volume mounts, etc.
Adapts to Huawei Ascend 910B4 chip inference acceleration.
Supports user-configured inference chips.
Implements full-chain metric collection from AI gateway, inference engine, Mooncake to infrastructure.
- AI gateway: Performance, resource consumption, security and compliance auditing, etc.
- Inference engine: API Server, model input/output, inference process, etc.
- Mooncake: Mooncake master, Mooncake client, and transfer engine.
- Infrastructure: Ray, K8s, and hardware.
Provides an independent hardware health diagnosis module that periodically collects underlying metrics such as NPU/GPU temperature, power consumption, error codes, and reports them in real-time through a distributed message queue system. The diagnosis module subscribes to and analyzes collected data, combines device model, driver and firmware information, identifies typical fault patterns based on threshold rules and abnormal metric analysis, and outputs diagnosis conclusions and handling recommendations, achieving a closed loop from data collection to health assessment.
Provides SLA-related metrics (such as throughput rate, latency, etc.) to support automatic scaling decisions for inference services, achieving elastic scaling based on load and performance.
Provides network performance metrics (such as actual transmission rate of node RDMA network cards, remaining available bandwidth, etc.) to support weight distribution acceleration modules in implementing node selection based on network performance.
Provides tidal algorithm, supporting pulling up/deleting business resources during specified time periods.
Provides scaling decision framework, supporting metric-driven and event-driven control of resource replica scaling, and supports flexible extension of user-defined scaling decision algorithms and custom resource management logic.
Provides dynamic PD group scaling capability, using abstract resource management objects to manage PD instances, achieving proportional dynamic PD scaling.
Supports one-click deployment of inference clusters in K8s environment using Helm.
Currently provides the following inference interfaces: /v1/chat/completions, /v1/completions. The interfaces do not involve authentication and log auditing capabilities; user management capabilities are provided uniformly by the upstream user management plane. For details, see External Interface Description

Key Features

Component Optional Installation and Decoupling: Open-source gateways, intelligent routing, KVCache index management, and inference observability components all use optional installation design. Users can enable or disable them as needed according to actual scenarios, and support replacement with self-developed or third-party components with equivalent capabilities.
Open-Source Gateway Capability Integration: Supports integration with open-source gateways adapted to GIE (Istio, Envoy AI Gateway, etc.), possessing key gateway capabilities such as service discovery, fault awareness, request retry, and traffic control.
KVCache Aware Routing Strategy: Compared to traditional load balancing, achieves more intelligent request routing by sensing the KVCache status of global inference nodes, reducing repeated KVCache computation.
PD-Bucket Routing Strategy: Bucket scheduling strategy in PD separation architecture, improving inference throughput in mixed long and short request scenarios with medium to high concurrency.
Latency Prediction Routing Strategy: Makes routing decisions based on real-time instance metrics, cache status, and latency prediction results, helping to further optimize inference latency and resource utilization.
Prefill-Decode Separation Architecture: Supports the industry's advanced PD separation architecture, significantly improving LLM inference throughput.
LWS Deployment Orchestration: Inference engines are deployed by LWS, natively supporting multi-DP coordination, and can configure DP load balancing strategies through dataParallelSize and dataParallelSizeLocal.
vLLM Mooncake Integration: vLLM v1 architecture integrates Mooncake distributed KVCache management system, providing distributed KVCache pooled storage and cross-instance KVCache high-speed transmission, improving cache reuse efficiency.
Second-Level Metric Push: Integrates NATS distributed message queue system, achieving efficient second-level metric push. After the collection module obtains hardware health data, it immediately pushes the data to the diagnosis module through NATS. This mechanism ensures that the diagnosis module can quickly receive the latest status information for timely anomaly detection and fault analysis.
PD-Orchestrator Feature: Integrates three capabilities of tidal algorithm, scaling decision framework, and dynamic PD scaling, covering multiple scenarios such as independent scaling of PD instances, proportional scaling of PD instance groups, metric-driven scaling, and tidal business time-triggered scaling, ensuring service availability during traffic spikes.
One-Click Deployment: Achieves one-click integrated deployment of three major components in K8s environment through Helm Chart.

Implementation Principle

Figure 1 AI Inference Integration Component Diagram

Hermes-router: Intelligent routing component. Responsible for receiving user requests and forwarding them to the optimal inference backend service according to routing strategies. For implementation principles, see AI Inference Hermes Routing.
cache-indexer: KVCache index management component, providing L1/L3 two-level KVCache hit rate query services for intelligent routing. For implementation principles, see AI Inference KVCache Index Management.
inference-backend: Inference backend component, providing high-performance large model inference services based on vLLM, consisting of 1 ProxyServer instance, n vLLM Prefill inference engine instances, and n vLLM Decode inference engine instances.
vLLM: vLLM inference engine instance.
Mooncake: Distributed KVCache pooled storage and providing KVCache P2P high-speed transmission between PD instances.
eagle-eye: Provides near real-time observability metric publish-subscribe mechanism, ensuring millisecond-level latency for key metrics; covers business running state, system running state, and hardware health metrics in inference scenarios, and provides hardware fault awareness and diagnosis modules. For implementation principles, see AI Inference Eagle Eye.
PD-Orchestrator: Composed of Tidal Controller, Elastic Scaler, and RSG three components, providing tidal time-based scaling, metric-driven scaling, and multi-resource proportional scaling capabilities. For implementation principles, see AI Inference Elastic Scaling, AI Inference Tidal Algorithm, Resource Scaling Group.

Component Initialization Flow:

Open-source gateway (default Istio):
1. After user deploys Istiod control plane, Istiod starts and begins listening to Kubernetes Gateway API related CRDs (GatewayClass, Gateway, HTTPRoute, etc.).
2. Istio automatically creates GatewayClass resource, declaring itself as the gateway controller.
3. After user configures Gateway CR, Istiod generates corresponding Envoy configuration, automatically creates Envoy Proxy Deployment and Service in the target namespace, as the actual gateway entry point.
4. Data plane completes configuration loading, gateway begins normally forwarding external traffic.
Intelligent routing:
1. Intelligent routing reads configuration.
2. Starts inference backend service discovery module (periodically updates backend list).
3. Starts inference backend metric collection module (periodically updates backend load metrics).
Inference backend:
1. Uses LWS to deploy vLLM inference engine instances according to configuration.
2. vLLM inference engine instances bind to hardware devices, load models.
3. Each vLLM inference engine instance starts Mooncake client, registers memory/SSD storage pool.
4. Prefill inference engine instances and Decode inference engine instances establish connections with each other's exposed KV connector ports.
5. In PD separation deployment mode, ProxyServer component starts, begins automatic discovery of inference engine instances (periodically updates inference engine instance list).
KVCache index management:
1. Loads configuration from ConfigMap, initializes L1/L3 indexers, block hash builders, and scoring services, starts HTTP service.
2. Starts inference engine instance auto-discovery module, periodically updates vLLM Pod list and Mooncake Master Pod.
3. Establishes ZMQ SUB connection for each vLLM Pod, subscribes to KV events and writes to L1 indexer.
4. Periodically polls KVCache information from Mooncake Master Pod and writes to L3 indexer.
5. Automatically cancels corresponding subscriptions when instances go offline, cleans up index records of offline instances.
PD-Orchestrator:
1. Starts tidal-scheduler controller, elastic-scaler controller, ResourceScalingGroup controller according to configuration.
2. Deploys default ElasticScaler CR instance and ResourceScalingGroup CR instance according to configuration, binds to vLLM inference backend.
3. Performs PD group scaling for vLLM inference backend based on CPU utilization and other metrics.
eagle-eye
1. Starts kube-prometheus-stack, nats, hardware-diagnosis, hardware-monitor, network-performance-exporter according to configuration.
2. After hardware-monitor starts, begins periodically collecting underlying metrics such as NPU/GPU temperature, power consumption, error codes, and publishes them to the diagnosis module in real-time through nats.
3. hardware-diagnosis subscribes to nats message queue, receives collected data and performs health status analysis combining device model, driver and firmware information, identifies fault patterns and outputs diagnosis conclusions.
4. After network-performance-exporter starts, begins periodically collecting network performance metrics such as actual transmission rate of node RDMA network cards, remaining available bandwidth, etc.
5. Prometheus begins scraping metric data from various exporters for periodic calculation and trend assessment.

Request Flow:

Request Ingress: User requests first reach GIE open-source gateway Istio.
KVCache Query: Gateway plugin Hermes-router queries cache-indexer for this request's KVCache hit rate in the cluster.
Routing Decision: Hermes-router selects the optimal inference backend service based on KVCache hit status, GPU utilization, and other metrics.
Request Forwarding: Open-source gateway forwards requests to the selected inference backend service.
KVCache Retrieval: Prefill node inference engine attempts to retrieve cached prefix KVCache from Mooncake KVCache pool.
Prefill Computation: Prefill node inference engine executes prefill computation for missed KVCache parts of tokens, and writes newly generated KVCache to Mooncake KVCache pool.
KVCache Transfer: Prefill node inference engine high-speed transfers request's KVCache to Decode node inference engine through Mooncake.
Inference Generation: Prefill inference engine generates first token result, Decode inference engine continuously generates subsequent tokens based on received KVCache.
Inference Instance Dynamic Scaling: When inference backend service pressure increases/decreases, ElasticScaler monitors corresponding metrics and calculates the number of replicas to scale up/down, achieving inference backend service scaling up/down.
Result Return: Inference results generated by inference engine are returned to open-source gateway, then returned to client.
Global KVCache Management Async Update: Prefill inference engine produces KV Event during prefill process, cache-indexer subscribes to this KV Event and real-time updates global KVCache metadata.

Intelligent routing component Hermes-router depends on inference engine (such as vLLM) to provide inference services and metric interfaces.
KVCache index management component cache-indexer depends on inference engine and distributed cache management system (such as Mooncake) to provide KVCache storage and removal events.
Tidal algorithm Tidal-scheduler needs to depend on elastic scaling framework elastic-scaler to modify the replica count of resource objects that need time-based scaling.

InferNex default configuration example reference: values.yaml

Installation

Prerequisites

Hardware Requirements

At least one inference chip per inference node.
At least 32GB memory and 4 CPU cores per inference node.

Software Requirements

Kubernetes v1.33.0 or above.
npu-operator component installed.
LWS component installed: Inference backend is orchestrated and deployed by LWS, must deploy LWS CRD and LWS Operator in the target cluster before installing InferNex (recommended v0.8.0 or above); for installation details, see LWS Official Installation Documentation.
InferNex will one-click deploy open-source gateway, default uses Istio, need to ensure environment has no conflicts.
Components included in InferNex have not been installed and deployed in the target namespace: hermes-router, Inference Backend, cache-indexer.
Components included in InferNex have not been installed and deployed in eagle-eye and nats namespaces: eagle-eye.

Network Requirements

Online installation requires access to image repository: oci://cr.openfuyao.cn.

Permission Requirements

Users have permissions to create RBAC resources.

Start Installation

Standalone Deployment

This feature can be deployed standalone in the following two ways:

Obtain Project Installation Package from openFuyao Official Image Repository

Pull project installation package.
bash
```
helm pull oci://cr.openfuyao.cn/charts/infernex --version 26.6.0
```
The pull result is a tgz compressed package. --version is used to specify the installation package version, corresponding one-to-one with InferNex release version: 0.0.0-latest indicates the latest build from the master branch, other version numbers see InferNex Version List.
Extract installation package.
bash
```
tar -xzvf infernex-26.6.0.tgz
```
Where 26.6.0 can be replaced with specific project installation package version.
Install and deploy.
Taking namespace ai-inference, release name infernex as an example, execute the following command in the same level directory as infernex:
bash
```
helm install -n ai-inference infernex ./infernex --create-namespace
```

Obtain Complete Project from openFuyao GitCode Repository

Clone project from repository.

bash

git clone -b release-26.6.0 https://gitcode.com/openFuyao/InferNex.git

Install and deploy.
Taking namespace ai-inference, release name infernex as an example, execute the following command in the same level directory as InferNex:
bash
```
cd InferNex/charts/infernex
helm dependency build
helm install -n ai-inference infernex . --create-namespace
```

Offline Installation

Note:
InferNex 26.6.0 does not offer a offline package. The latest offline package is 0.22.2, released in March 2026. If it is desired, please contact the InferNex team or follow the Offline Package Creation Guide to build it.

Obtain InferNex 0.22.2 offline installation package from openFuyao artifact repository.
bash
```
wget https://openfuyao.obs.cn-north-4.myhuaweicloud.com/openFuyao/ext-components/InferNex/openFuyao-infernex-offline-v26.03.tar.gz
```
Note:
If users want to manually create InferNex offline package, can refer to Offline Package Creation Guide.

Extract offline installation package.

bash

tar -xzvf openFuyao-infernex-offline-v26.03.tar.gz

Extract helm chart installation package, extract images to local repository, extract InferNex built-in model cache files.
bash
```
cd openFuyao-infernex-offline-v26.03 && bash install.sh
```
Install and deploy.
If users want to use custom model files, please refer to Custom Model Directory Configuration.
Taking namespace ai-inference, release name infernex as an example, execute the following command in the same level directory as chart package file infernex:
bash
```
helm install -n ai-inference infernex ./infernex --create-namespace
```

Configuring AI Inference Integrated Deployment

Prerequisites

Obtained InferNex project files.

Operation Steps

Prepare values.yaml configuration file.
Refer to Start Installation Section to find the values.yaml configuration file.
Configure global configuration.
- global.image.pullPolicy: Controls the pull policy for all images deployed by InferNex, default is IfNotPresent for online deployment, default is Never for offline deployment.
- global.imagePullSecrets: Secret configuration for private image repositories, used for pulling images from private image repositories. For example: [{"name": "registry-secret"}].
- global.env: Defines default environment variables for all inference-backend service instances, these environment variables will be injected into all inference engine containers and cache-indexer containers. Common configurations include HuggingFace offline download switch, Huggingface access K8s Secret.
- global.autoDownloadModel: Whether to automatically download Huggingface models through InferNex. If want to use local non-Huggingface models or in offline environment, need to set to false.
- global.modelName: Inference model name (required). If global.modelPath is not configured, vLLM will load weights via Huggingface model name. If global.modelPath is configured, vLLM will use this configuration as model alias for matching the model field in inference requests. Default inference model used is "Qwen/Qwen3-8B".
- global.modelPath: Inference model weight local path (optional). If configured, vLLM starts using local path, otherwise uses the model name configured in global.modelName to load Huggingface model. Current version this configuration item cannot be used together with cache-indexer component. Since model weights are mounted to the container's /root/.cache directory via global.cachePath, this configuration should be /root/.cache/{model directory after cachePath}.
- global.cachePath: Path on host machine storing HuggingFace model cache, structure is {cache directory}/huggingface/hub/{model directory}, this path will be mounted to /root/.cache directory of all inference engine containers and cache-indexer containers via hostPath. For detailed instructions and examples on custom model directory configuration, please refer to Custom Model Directory Configuration. Default host machine model cache path configured is /home/llm_cache.
  Local non-Huggingface model configuration example: Host machine local model path /mnt/public/models/my_models/Qwen3-8B-W8A8/, then InferNex configuration is:
  yaml
```
global:
  env:
    - name: HF_HUB_OFFLINE
      value: "0"
  autoDownloadModel: false
  modelName: "Qwen3-8B-W8A8"
  modelPath: "/root/.cache/Qwen3-8B-W8A8/"
  cachePath: "/mnt/public/models/my_models/"
```
Configure intelligent routing parameters.
Configure Gateway Parameters
- inferenceGateway.enabled: Open-source gateway switch. true enables, false disables.
- inferenceGateway.name: Gateway resource name. Must be consistent with hermes-router.httpRoute.parentRef.name.
- inferenceGateway.className: Gateway class name. Default value `"istio", indicates using Istio control plane management.
- inferenceGateway.listeners: Defines listener configuration.
  - name: http: Listener name.
  - port: 80: Listener port.
  - protocol: HTTP: Protocol type.
Configure Routing Image Pull
- hermes-router.enabled: Intelligent routing switch. true enables, false disables. Note, since intelligent routing itself is an extension plugin, cannot be used standalone without gateway.
- hermes-router.image.repository: Intelligent routing image address.
- hermes-router.image.tag: Intelligent routing image version.
Configure Routing Strategy
- hermes-router.inferenceExtension.replicas: Intelligent routing replica count. Default is 1.
- hermes-router.inferenceExtension.pluginsConfigFile: Currently recommended to keep as default-plugins.yaml, used for loading Hermes-router built-in routing strategy configuration.
- hermes-router.inferenceExtension.routing.deploymentMode: Used to declare inference backend deployment mode, options are aggregate or pd.
- hermes-router.inferenceExtension.routing.profile: Used to declare routing strategy type, options are random, kv-cache-aware, bucket, prediction. Where bucket only applies to pd deployment mode.
- Currently supports the following preset routing strategy combinations:
  - aggregate-random.yaml: Aggregate architecture random routing.
  - aggregate-kv-cache-aware.yaml: Aggregate architecture KVCache aware routing.
  - aggregate-prediction.yaml: Aggregate architecture latency prediction routing.
  - pd-random.yaml: PD architecture random routing.
  - pd-kv-cache-aware.yaml: PD architecture KVCache aware routing.
  - pd-bucket.yaml: PD bucket scheduling routing.
  - pd-prediction.yaml: PD architecture latency prediction routing.
- For complete reference configuration of Hermes-router built-in routing strategies, see hermes-router routing strategy reference configuration.
- If need to customize plugin chain, can continue to extend EndpointPickerConfig content through hermes-router.inferenceExtension.pluginsCustomConfig; at this time pluginsConfigFile should be consistent with custom configuration file name.
Configure InferencePool
- hermes-router.inferencePool.modelServers.matchLabels: Inference service label selector, used to select Pods joining InferencePool, Pods must satisfy all labels simultaneously, and only match within the same namespace, cross-namespace not supported.
- hermes-router.inferencePool.targetPorts: Actual listening port number of each inference service in InferencePool, used for processing inference traffic. Default inference port is 8000.
- hermes-router.inferencePool.modelServerType: Inference engine type. Default is vllm.
Configure HTTPRoute
- hermes-router.httpRoute.inferenceGatewayName: Gateway resource name. Must be consistent with inferenceGateway.name.
Configure Request Retry
- hermes-router.provider.retryConfig: Defines request retry configuration.
  - enabled: Request retry switch. true enables, false disables. Default false.
  - retryOn: List of error types that trigger retry, typical optional values include: connect-failure, refused-stream, unavailable, cancelled, retriable-status-codes, 5xx, reset, etc., can be combined as needed. Default selects all.
  - numRetries: Maximum retry count allowed for a single request. Default count is 3.
- hermes-router.provider.istio.destinationRule.tls: Defines communication rules between Istio gateway and inference backend.
  - mode: TLS mode when Istio communicates with backend, options: DISABLE (TLS not enabled), SIMPLE (one-way TLS), MUTUAL/ISTIO_MUTUAL (mutual TLS, relies on certificates or identity provided by Istio). Default SIMPLE.
  - insecureSkipVerify: Whether to skip backend service certificate verification, options: true (skip), false (do not skip). Default true.
Configure inference backend parameters.
Configure Inference Backend Image Parameters
- inference-backend.images.inferenceEngine: Configure inference engine image (repository, tag). InferNex default configuration uses quay.io/ascend/vllm-ascend:v0.18.0 inference engine; multi-DP scenarios require vLLM version in image not lower than 0.10.0, to support chart auto-injected hybrid and multi-DP related startup parameters.
- inference-backend.images.proxyServer: Configure ProxyServer image (repository, tag).
Configure Inference Backend Environment Variables
- inference-backend.env: Inference backend environment variable total configuration, these environment variables will be injected into all inference service's inference engine containers (Prefill, Decode) and ProxyServer containers. Users can configure according to actual needs referring to vllm-ascend environment variable configuration instructions documentation.
Configure Inference Backend File Mount Parameters
- inference-backend.volumeMounts: volumeMounts configuration mounted by all vLLM inference engine Pods pulled up by LWS. Default configuration includes Ascend device related volumeMounts. Users can add more detailed mount items according to needs, detailed configuration instructions please refer to Inference Backend Default Mount Configuration.
- inference-backend.volumes: volumes configuration mounted by all vLLM inference engine Pods pulled up by LWS. Default configuration includes Ascend device related volumes. Detailed configuration instructions please refer to Inference Backend Default Mount Configuration.
Configure Inference Services
- inference-backend.services: Inference service configuration, supports users to configure multiple independent vLLM inference services. Each service can independently configure model, deployment mode (aggregate mode or PD separation mode), resource and other parameters, achieving mixed deployment of multiple deployment form inference services. Inference engine underlying workload is LWS: when mode: pd, Prefill and Decode each correspond to one LWS (such as {service name}-prefill, {service name}-decode); when mode: aggregated, corresponds to one LWS (such as {service name}-aggregated).
  Basic Configuration
  - name: Inference service name.
  - enabled: Whether to deploy this service. InferNex requires inference backend to have at least one inference service enabled.
  - mode: Inference backend service mode, options are aggregated and pd, representing aggregate architecture and PD separation architecture respectively; default pd.
  - service.port: Service service port, default 8000.
  - pdGroupID: PD mode group ID, needs to be set in PD separation architecture, indicates that all ProxyServer, Prefill, Decode inference backends under this inference service are within this group, for intelligent routing to discover and filter by labels like openfuyao.com/pdGroupID.
  Configure Inference Engine Connector
  - kvTransferConfig.connectorConfig: Configure KVCache Connector, used to define KVCache reuse method between Prefill and Decode stages. Provides YAML format configuration object, configuration will be automatically converted to JSON format according to key-value pairs, and kv_role, kv_rank, engine_id, tp_size, dp_size and other prefill/decode node specific fields will be automatically filled, no need to manually configure (user manual configuration can override auto-fill). PD separation mode deployment kv_connector recommends using MultiConnector (combining MooncakeConnectorV1 and AscendStoreConnector), aggregate mode deployment recommends using AscendStoreConnector.
    Due to different vllm/vllm-ascend versions, connector names and detailed configuration items are different, for detailed configuration instructions please refer to target vllm/vllm-ascend version documentation. InferNex default configuration uses vllm-ascend:v0.18.0 inference engine, users can refer to vllm-ascend documentation KV pool and PD separation related sections.
  - kvTransferConfig.mooncake.configPath: When using Mooncake as inference engine's KVCache management system, configuration item file for Mooncake client startup inside inference engine. Default path is "/app/mooncake.json".
  - kvTransferConfig.mooncake.use_store: When using Mooncake as inference engine's KVCache management system, used to control whether to use Mooncake Store mode. If above connector type applied Mooncake Store type Connector, need to enable this configuration.
  - kvTransferConfig.mooncake.config: Mooncake client configuration file content (yaml format). Inference engine pod's initContainer will automatically convert and generate mooncake.json configuration file for Mooncake client inside inference engine to use directly. For detailed configuration item instructions please refer to Mooncake documentation.
  Configure Inference Engine Startup Items in PD Separation Mode
  - pd.prefill.replicas: Prefill side LWS parallel group sets count, default 2.
  - pd.prefill.cardCount: Used to specify the number of inference cards allocated to Prefill engine; when not configured, default takes tp*dp*pp. In multi-machine deployment single model scenario, must manually set this parameter, otherwise default calculation based on tp*dp*pp will produce deviation.
  - pd.prefill.tensorParallelSize: Prefill engine tensor parallelism.
  - pd.prefill.pipelineParallelSize: Prefill engine pipeline parallelism, currently only supports configuration as 1.
  - pd.prefill.dataParallelSize: Prefill engine data parallelism (logical DP rank total count), default 1.
  - pd.prefill.dataParallelSizeLocal: Prefill engine single node local DP rank count, default 1. dataParallelSize is inference service DP rank total count, dataParallelSizeLocal is local rank count on a single inference engine node worker; the two must divide evenly, quotient is the number of workers within this LWS group (inference engine node count). chart automatically injects vLLM multi-DP startup parameters based on LWS group internal environment variables; when dataParallelSizeLocal is 1, it is close to External DP load balancing, when equal to dataParallelSize, it is close to Internal DP load balancing.
  - pd.prefill.dataParallelRpcPort: Prefill engine DP RPC port; required when dataParallelSize is greater than 1, default 12890.
    yaml
```
pd:
  prefill:
    replicas: 1                 
    dataParallelSize: 4         
    dataParallelSizeLocal: 2    
    dataParallelRpcPort: 12890
    tensorParallelSize: 1
    pipelineParallelSize: 1
```
    Deployment topology (replicas=1): LWS leaderWorkerTemplate.size = 4 / 2 = 2 (2 inference engine nodes).
    Node 0 (LWS_WORKER_INDEX=0): Pod×2 → DP rank 0、1 (--data-parallel-start-rank=0)
    Node 1 (LWS_WORKER_INDEX=1): Pod×2 → DP rank 2、3 (--data-parallel-start-rank=2)
  - pd.prefill.enablePrefixCaching: Whether Prefill engine enables prefix caching.
  - pd.prefill.maxModelLen: Prefill engine maximum model length.
  - pd.prefill.maxNumBatchedTokens: Prefill engine maximum batch processing token count.
  - pd.prefill.gpuMemoryUtilization: Prefill engine VRAM utilization.
  - pd.prefill.blockSize: Prefill engine block size.
  - pd.prefill.env: Prefill engine container environment variable list. Can append role-specific variables as needed.
  - pd.prefill.volumeMounts: Prefill engine container volumeMounts list, used to configure volume mounts specifically needed by this inference service Prefill inference engine Pod.
  - pd.prefill.volumes: Prefill engine container volumes list, used to configure volumes specifically needed by this inference service Prefill inference engine Pod.
  - pd.prefill.extraArgs: Prefill engine extra startup parameter list. Users need performance tuning startup command specific configuration in some scenarios, this configuration is responsible for appending extra configuration items to inference engine startup command.
  - pd.decode.replicas: Decode side LWS parallel group sets count, default 2.
  - pd.decode.cardCount: Used to specify the number of inference cards allocated to Decode engine; when not configured, default takes tp*dp*pp. In multi-machine deployment single model scenario, must manually set this parameter, otherwise default calculation based on tp*dp*pp will produce deviation.
  - pd.decode.tensorParallelSize: Decode engine tensor parallelism.
  - pd.decode.pipelineParallelSize: Decode engine pipeline parallelism, currently only supports configuration as 1.
  - pd.decode.dataParallelSize: Decode engine data parallelism (logical DP rank total count), default 1.
  - pd.decode.dataParallelSizeLocal: Decode engine single node local DP rank count, default 1; meaning same as pd.prefill, see above configuration example.
  - pd.decode.dataParallelRpcPort: Decode engine DP RPC port; required when dataParallelSize is greater than 1, default 12777.
  - pd.decode.enablePrefixCaching: Whether Decode engine enables prefix caching.
  - pd.decode.maxModelLen: Decode engine maximum model length.
  - pd.decode.maxNumBatchedTokens: Decode engine maximum batch processing token count.
  - pd.decode.gpuMemoryUtilization: Decode engine VRAM utilization.
  - pd.decode.blockSize: Decode engine block size.
  - pd.decode.env: Decode engine container environment variable list, used to configure environment variables specifically needed by this inference service Decode inference engine Pod.
  - pd.decode.volumeMounts: Decode engine container volumeMounts list, used to configure volume mounts specifically needed by this inference service Decode inference engine Pod.
  - pd.decode.volumes: Decode engine container volumes list, used to configure volumes specifically needed by this inference service Decode inference engine Pod.
  - pd.decode.extraArgs: Decode engine extra startup parameter list.
  - pd.proxyServer.enabled: Whether to deploy ProxyServer corresponding to this inference service, default true in PD separation mode. ProxyServer filters pdGroupID during inference backend node service discovery, when deploying multiple inference services and want them to be uniformly managed and routed by one ProxyServer, can set each inference-backend.services[i].pdGroupID to be consistent, and only one inference service sets pd.proxyServer.enabled to true.
  - pd.proxyServer.discoveryInterval: ProxyServer inference backend service discovery interval (unit: seconds), default 10.
  Configure Inference Engine in Aggregate Mode
  - aggregated.replicas: Aggregate mode LWS parallel group sets count.
  - aggregated.cardCount: Used to specify the number of inference cards allocated to aggregate mode inference engine; when not configured, default takes tp*dp*pp. In multi-machine deployment single model scenario, must manually set this parameter, otherwise default calculation based on tp*dp*pp will produce deviation.
  - aggregated.enablePrefixCaching: Whether aggregate mode inference engine enables prefix caching.
  - aggregated.tensorParallelSize: Aggregate mode inference engine tensor parallelism.
  - aggregated.pipelineParallelSize: Aggregate mode inference engine pipeline parallelism, currently only supports configuration as 1.
  - aggregated.dataParallelSize: Aggregate mode inference engine data parallelism (logical DP rank total count), default 1; LWS and multi-DP parameter meaning see PD mode above pd.prefill configuration example.
  - aggregated.dataParallelSizeLocal: Aggregate mode single node local DP rank count, default 1.
  - aggregated.dataParallelRpcPort: Aggregate mode DP RPC port; required when dataParallelSize is greater than 1, default 12890.
  - aggregated.gpuMemoryUtilization: Aggregate mode inference engine VRAM utilization.
  - aggregated.blockSize: Aggregate mode inference engine block size.
  - aggregated.maxModelLen: Aggregate mode inference engine maximum model length.
  - aggregated.maxNumBatchedTokens: Aggregate mode inference engine maximum batch processing token count.
  - aggregated.env: Aggregate mode inference engine container environment variable list, used to configure environment variables specifically needed by this inference service inference engine Pod.
  - aggregated.volumeMounts: Aggregate mode inference engine container volumeMounts list, used to configure volume mounts specifically needed by this inference service inference engine Pod.
  - aggregated.volumes: Aggregate mode inference engine container volumes list, used to configure volumes specifically needed by this inference service inference engine Pod.
  - aggregated.extraArgs: Aggregate mode inference engine extra startup parameter list.
  Configure Inference Engine Resources
  - resources.requests: Inference engine request resources. Default example is CPU 4, memory 64Gi.
  - resources.limits: Inference engine limit resources. Default example is CPU 8, memory 128Gi.
Configure KVCache index management parameters.
Configure Whether to Enable KVCache Index Management Component
- cache-indexer.enabled: cache-indexer is an InferNex optional component, default is true.
Configure KVCache Index Management Component Image Pull
- cache-indexer.image.repository: cache-indexer image address.
- cache-indexer.image.tag: cache-indexer image tag.
Configure KVCache Index Management Component Resources
- cache-indexer.resources.requests: cache-indexer Pod resource request value, default CPU 100m, memory 128Mi.
- cache-indexer.resources.limits: cache-indexer Pod resource limit value, default CPU 1, memory 2Gi.
Configure KVCache Index Management Component Logs
- cache-indexer.log.level: cache-indexer log level, options debug, info, error, default info.
Configure KVCache Index Management Component K8s Service
- cache-indexer.service.name: cache-indexer K8s Service name, default is cache-indexer-service.
- cache-indexer.service.port: cache-indexer K8s Service port, default is 8080.
Configure KVCache Index Management Component Inference Backend Service Discovery
- cache-indexer.discovery.labels.engineKey/engineValue: vLLM Pod discovery label, default openfuyao.com/engine=vllm.
- cache-indexer.discovery.labels.pdRoleKey/pdRoleValue: Inference role label participating in hit rate calculation, default openfuyao.com/pdRole, values prefill and aggregate.
- cache-indexer.discovery.labels.kvManagerKey/kvManagerValue: Mooncake Master discovery label, default openfuyao.com/kvmanager=mooncake.
Configure KVCache Index Management Component Block Hash Calculation Parameters
- cache-indexer.blockKey.pythonHashSeed: Must be consistent with vLLM PYTHONHASHSEED, default "0".
- cache-indexer.blockKey.prefixCachingHashAlgo: Must be consistent with vLLM --prefix-caching-hash-algo, default sha256_cbor.
- cache-indexer.blockKey.useIntBlockHashes: Must be consistent with vLLM VLLM_KV_EVENTS_USE_INT_BLOCK_HASHES, default true.
Configure AI inference observability parameters.
Configure whether to enable AI inference observability
- eagle-eye.enabled: Controls whether to enable AI inference observability.
Configure Hardware Health Monitoring Image Pull
- hardware-monitor.images.core.repository: hardware-monitor image address.
- hardware-monitor.images.core.tag: hardware-monitor image tag.
- hardware-monitor.images.core.pullPolicy: hardware-monitor image pull policy.
Configure Hardware Diagnosis Image Pull
- hardware-diagnosis.images.core.repository: hardware-diagnosis image address.
- hardware-diagnosis.images.core.tag: hardware-diagnosis image tag.
- hardware-diagnosis.images.core.pullPolicy: hardware-diagnosis image pull policy.
Configure Network Performance Collection Component Image Pull
- network-performance-exporter.images.core.repository: network-performance-exporter image address.
- network-performance-exporter.images.core.tag: network-performance-exporter image tag.
- network-performance-exporter.images.core.pullPolicy: network-performance-exporter image pull policy.
- network-performance-exporter.networkPerformanceExporter.metricsPort: network-performance-exporter metric exposure port.
- network-performance-exporter.networkPerformanceExporter.collectInterval: network-performance-exporter metric collection interval.
Configure PD-Orchestrator parameters.
Configure Whether to Enable Elastic Scaling Framework
- elastic-scaler.enabled: Controls whether to enable elastic-scaler component
Configure Elastic Scaling Framework Image Pull
- elastic-scaler.images.repository: elastic-scaler image address.
- elastic-scaler.images.tag: elastic-scaler image tag.
Configure PD-Orchestrator Component Installation Namespace
- elastic-scaler.namespace.name: Namespace where PD-Orchestrator component controller is located.
Configure Elastic Scaling Framework Default CR Instance
- elastic-scaler.elasticScaler.enabled: Controls whether to deploy default ElasticScaler CR instance.
- elastic-scaler.targetRef.kind: Type of actual resource object controlled by default ElasticScaler CR, can be Kubernetes native resources like Deployment, StatefulSet, etc. In InferNex configuration, this configuration default is resourcescalinggroup.
- elastic-scaler.targetRef.name: Name of actual resource object controlled by default ElasticScaler CR. In InferNex configuration, this field needs to correspond to RSG resource object name. Example: If RSG resource object configuration is:
  yaml
```
resourcescalinggroup:
  instanceConfig:
    name: rsg
```
  Then this attribute needs to be configured as elastic-scaler.targetRef.name: rsg.
- elastic-scaler.targetRef.apiVersion: API version of actual resource object controlled by default ElasticScaler CR. In InferNex configuration, this field needs to correspond to RSG resource object API version, for example autoscaling.openfuyao.com/v1alpha1.
- elastic-scaler.minReplicas: Minimum replica count of actual resource object controlled by default ElasticScaler CR.
- elastic-scaler.maxReplicas: Maximum replica count of actual resource object controlled by default ElasticScaler CR.
- elastic-scaler.trigger.scalingAlgorithm: Algorithm triggering actual resource object scaling, default configuration is apa.
- elastic-scaler.trigger.resource.metricsName: Metric name triggering actual resource object scaling, default configuration is CPU, indicating scaling module will calculate resource replica count based on CPU metric.
- elastic-scaler.trigger.resource.targetType: Metric type triggering actual resource object scaling, default is utilization, combined with elastic-scaler.trigger.resource.metricsName field, indicating CPU utilization is the trigger type for resource object replica scaling.
- elastic-scaler.trigger.resource.targetValue: Metric threshold triggering actual resource object scaling.
Configure Whether to Enable PD Scaling Resource Management Object Component
- resourcescalinggroup.enabled: Controls whether to enable ResourceScalingGroup component
Configure PD Scaling Resource Management Object Component Image Pull
- resourcescalinggroup.images.repository: ResourceScalingGroup image address.
- resourcescalinggroup.images.tag: ResourceScalingGroup image tag.
Configure PD Scaling Resource Management Object Component Installation Namespace
- resourcescalinggroup.namespace.name: Namespace where ResourceScalingGroup Controller is deployed, default is scaling-system.
- resourcescalinggroup.namespace.create: Whether to automatically create namespace specified by resourcescalinggroup.namespace.name. If this namespace does not exist in cluster, need to set to true; otherwise need to create manually beforehand. Current default value is false, default does not create this namespace.
Configure PD Scaling Resource Management Object Component Running Parameters
- resourcescalinggroup.prometheus.url: Prometheus query address, used by scaleDown.metric scaling down strategy, will actually inject environment variable RSG_PROMETHEUS_URL. This feature needs to be used with prometheus related configuration, for details please refer to Resource Scaling Group.
Configure PD Scaling Resource Management Object Default CR Instance
- ResourceScalingGroup CR instance configuration
  Basic Configuration
  - resourcescalinggroup.instanceConfig.enabled: Controls whether to deploy default ResourceScalingGroup CR instance.
  - resourcescalinggroup.instanceConfig.name: Name of default ResourceScalingGroup CR instance.
  - resourcescalinggroup.instanceConfig.scalingStrategy.type: Scaling strategy type used by default ResourceScalingGroup CR instance, is GroupReplication or InplaceScaling. Current default configuration is GroupReplication, if need to change to InplaceScaling mode, please refer to Resource Scaling Group.
  - resourcescalinggroup.instanceConfig.scalingStrategy.groupConfig.groupName: Group name prefix of default ResourceScalingGroup CR instance; when empty, default uses RSG name.
  - resourcescalinggroup.instanceConfig.scalingStrategy.groupConfig.replicas: Desired group count of default ResourceScalingGroup CR instance, this attribute is limited by elastic-scaler.minReplicas and elastic-scaler.maxReplicas fields. Example: If configured as:
    yaml
```
elastic-scaler:
  minReplicas: 1
  maxReplicas: 10
```
    Then through APA scaling, the minimum number of resource groups created is 1, maximum is 10.
  - resourcescalinggroup.instanceConfig.scalingStrategy.groupConfig.scaleDown.metric.expr: Prometheus metric expression or metric name referenced for scaling down by default ResourceScalingGroup CR instance.
  - resourcescalinggroup.instanceConfig.scalingStrategy.groupConfig.scaleDown.metric.type: Scaling down metric type for default ResourceScalingGroup CR instance, such as Counter, Gauge, Histogram.
  - resourcescalinggroup.instanceConfig.scalingStrategy.groupConfig.scaleDown.metric.window: Statistical window used when scaling down metric type is Counter or Histogram.
  - resourcescalinggroup.instanceConfig.scalingStrategy.groupConfig.scaleDown.metric.aggregator: Aggregation method for metric data during scaling down, such as avg, sum, max.
  - resourcescalinggroup.instanceConfig.scalingStrategy.groupConfig.scaleDown.order: Group scaling down order, usually supports ascending or descending.
  Scaling Resource Object Configuration
  - Below gives all related field configurations for resourcescalinggroup.instanceConfig.targetResources in InferNex configuration:
    yaml
```
resourcescalinggroup:
  instanceConfig:
    name: rsg # ResourceScalingGroup CR instance name, needs to be same as elastic-scaler.targetRef.name
    targetResources:
    - name: prefill # This name can be customized by user
      resourceRef:
        apiVersion: leaderworkerset.x-k8s.io/v1
        kind: LeaderWorkerSet # Resource type of prefill nodes created by inference backend
        name: qwen3-8b-2p2d-01-prefill # Must correspond to inference-backend.services[].name: {service name}-prefill
    - name: decode
      resourceRef:
        apiVersion: leaderworkerset.x-k8s.io/v1
        kind: LeaderWorkerSet
        name: qwen3-8b-2p2d-01-decode # Must be consistent with {service name}-decode
```
  - resourcescalinggroup.instanceConfig.targetResources.name: Logical name of target resource within default ResourceScalingGroup CR instance, used for scaling strategy configuration reference, must be unique within same resource group.
  - resourcescalinggroup.instanceConfig.targetResources.resourceRef.apiVersion: API version of target workload in default ResourceScalingGroup CR instance. In InferNex configuration, since inference backend default configuration object is Deployment, default value is apps/v1.
  - resourcescalinggroup.instanceConfig.targetResources.resourceRef.kind: Target workload type in default ResourceScalingGroup CR instance, such as Deployment, StatefulSet, LeaderWorkerSet. In InferNex configuration, default value is Deployment, this type needs to be same as inference backend resource type created by inference-backend.
  - resourcescalinggroup.instanceConfig.targetResources.resourceRef.name: Target workload name in default ResourceScalingGroup CR instance. In InferNex configuration, needs to correspond to inference-backend.services.name.
  - resourcescalinggroup.instanceConfig.targetResources.resourceRef.namespace: Namespace where target workload in default ResourceScalingGroup CR instance is located, when empty default uses RSG located namespace.
Configure Whether to Enable Tidal Algorithm Component
- tidal.enabled: Controls whether to enable TidalScheduler component, used to adjust target workload replica count or ResourceScalingGroup group count according to time rules.
Configure Tidal Algorithm Component Image Configuration
- tidal.images.repository: tidal image address.
- tidal.images.tag: tidal image tag.
Note:
TidalScheduler is only responsible for calculating the desired replica count based on time rules; actual scaling still relies on ElasticScaler. If you primarily use Tidal for tidal scaling, it is recommended to keep elastic-scaler.enabled=true and disable the default ElasticScaler and ResourceScalingGroup example CRs as needed, to avoid conflicts with business-custom ElasticScaler or ResourceScalingGroup resources.
Apply configuration
User refers to Installation Section to deploy and apply configuration.

Using AI Inference

Prerequisites

Hardware Requirements

At least one inference chip per inference node.
At least 32GB memory and 4 CPU cores per inference node.
In PD separation scenario when Mooncake transfers KVCache, if using HCCS protocol, host machine /etc/hccn.conf file needs to have correct configuration for inference device IP address and mask on host machine, hccn.conf file device information configuration example script please refer to Ascend HCCS Device IP Address Configuration Example.

Software Requirements

Kubernetes v1.33.0 or above.
npu-operator component installed.
LWS component installed.
metrics server v0.8.0 or above version needs to be installed in cluster.
InferNex included necessary components have been installed: inference-backend, PD-Orchestrator.

Network Requirements

In PD separation scenario when Mooncake transfers KVCache, if using HCCS protocol for cross-machine high-speed communication transfer, need HCCS device or RDMA device support.

Background Information

None.

Usage Limitations

Currently only supports vLLM/vLLM-Ascend inference engines.
Currently only verified on Ascend910B4 inference chip.
Currently only supports AI inference scenarios, does not support AI training scenarios.

Operation Steps

Taking release name infernex and deployment namespace ai-inference as example:

Get service access address.
1.1 Check hermes-router service access address:
bash
```
kubectl get svc -n ai-inference
```
1.2 Record gateway IP address and port. hermes-router service name is inference-gateway-istio.

Send inference request (using curl to send request as example).

2.1 Send non-streaming inference request:

bash

curl -X POST http://[routing service IP address]:[routing service port]/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "messages": [{"role": "user", "content": "Please introduce openFuyao open source community"}],
    "stream": false
  }'

2.2 Send streaming inference request:

bash

curl -X POST http://[routing service IP address]:[routing service port]/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "messages": [{"role": "user", "content": "Please introduce openFuyao open source community"}],
    "stream": true
  }'

Receive inference results.
3.1 Non-streaming response returns complete result at once.
3.2 Streaming response returns JSON objects starting with data: block by block.

Taking release name infernex and namespace ai-inference as example:

Check Deployment Status:

bash

helm status infernex -n ai-inference

Uninstall System:

bash

helm uninstall infernex -n ai-inference

Export Configuration:

bash

helm get values infernex -n ai-inference > current-values.yaml

FAQ

Hermes-router and ProxyServer have error messages in initial phase.
Phenomenon Description: Through viewing Pod logs found that initially Hermes-router and ProxyServer have many error request messages.
Handling Steps: Normal phenomenon. Hermes-router periodically sends inference engine metric query requests to ProxyServer after initialization is complete, since inference engine instances take long time to load large models during initialization, metric query requests cannot be correctly responded at this time. Wait for inference engine instances to start completely, this problem will not appear.
Hermes-router and cache-indexer cannot discover inference services.
Phenomenon Description: Through viewing Pod logs found that Hermes-router and cache-indexer auto-discover inference service instances with errors.
Handling Steps: Possibly because service label selector configuration error. hermes-router.app.discovery.labelSelector.app and cache-indexer.app.serviceDiscovery.labelSelector attributes represent hermes-router and cache-indexer inference backend K8s Service resource service discovery configuration, need to be consistent with inference-backend.service.label attribute.
Gateway cannot work normally after deployment is complete.
Phenomenon Description: After installation is complete, Istio Envoy proxy reports errors or cannot forward requests to backend services, but backend inference service running status is normal.
Handling Steps: Check whether HTTPRoute resource's parentRef.name is consistent with Gateway resource name (needs to match with inferenceGateway.name configuration item).
PD separation mode Decode service does not normally use Mooncake for transfer.
Phenomenon Description: After PD separation architecture deployment, Decode inference service is abnormal or performance is poor.
Handling Steps: Check Decode inference service prefix cache configuration. In PD separation architecture using Mooncake for transfer, Decode inference service should not enable prefix cache. Please confirm pd.decode.enablePrefixCaching configuration item is set to false, or use --no-enable-prefix-caching parameter to start Decode service.
PD separation mode Decode service keeps computing output tokens, or inference request result is empty string.
Phenomenon Description: Configured to use HCCS for KVCache transfer, after PD separation architecture deployment, Decode node keeps computing returned tokens, output does not stop.
Handling Steps: Check whether host machine's hccn.conf file is mounted, and confirm whether hccn.conf file has correct configuration for inference device IP address and mask on host machine. hccn.conf file device information configuration example script please refer to Ascend HCCS Device IP Address Configuration Example.
Using non-HuggingFace models (such as quantized models) causes inference engine Pod cannot be deployed.
Phenomenon Description: When user uses non-HuggingFace models (such as quantized model Qwen3-8B-W4A8), inference engine backend Pod is in CrashLoopBackOff status.
Handling Steps: Current cache-indexer component is not compatible with non-Huggingface models. Configure InferNex default configuration's cache-indexer.enabled: true to cache-indexer.enabled: false, or modify intelligent routing's routing strategy to non-kv-aware strategy. Inference backend component can normally deploy non-HuggingFace models.
After vllm-ascend 0.14.0 dense models (such as Qwen3-8B) configured with multi-DP inference engine Pod fails to start.
Phenomenon Description: Using InferNex default inference engine image vllm-ascend:v0.18.0 to deploy Qwen/Qwen3-8B and other non-MoE dense models, if in values.yaml configure dataParallelSize to greater than 1 (for example dataParallelSize: 4, dataParallelSizeLocal: 2), after helm install inference backend Pod repeatedly restarts. In logs can see similar errors:
```
ValueError: KV transfer 'decode' config has a conflicting data parallel size. Expected 1, but got 4.
```
Reason Explanation: Since vLLM 0.14.0, non-MoE models no longer support single inference service internal multi-DP (dataParallelSize greater than 1) deployment; in PD separation scenario KV transfer configuration's decode side DP scale must match engine actual DP, dense models only support dataParallelSize: 1.
Handling Steps: For dense models like Qwen3-8B, multi-DP deployment and multi-instance deployment functionality have no difference, please keep Prefill/Decode (or Aggregated) dataParallelSize, dataParallelSizeLocal both as 1, expand inference throughput by increasing replicas, for example:
yaml
```
inference-backend:
  services:
    - name: qwen3-8b-2p2d-01
      pd:
        prefill:
          replicas: 4
          dataParallelSize: 1
          dataParallelSizeLocal: 1
        decode:
          replicas: 4
          dataParallelSize: 1
          dataParallelSizeLocal: 1
```
Resource replicas scaled by ResourceScalingGroup will not be deleted by helm uninstall.
Phenomenon Description: When using ResourceScalingGroup, if new LWS and other resources are scaled out, at this time user executes helm uninstall command to uninstall components, scaled out LWS and other resources will not be deleted.
Handling Steps: Before executing helm uninstall command, execute the following commands to delete ResourceScalingGroup CR and scaled out LWS resources.
bash
```
kubectl delete resourcescalinggroup [rsg-name] -n [namespace]
kubectl delete lws -n [namespace] -l 'rsg.io/name=[rsg-name],rsg.io/group-id!=0'
```
Where rsg-name is CR instance name, namespace is instance located namespace.
Using HPA algorithm to scale ResourceScalingGroup resource cannot scale as expected.
Phenomenon Description: When scaling ResourceScalingGroup through pd-orchestrator's HPA algorithm, its managed LWS inference backend instance count does not match expectation, for example metrics not reaching threshold but triggers scaling.
Note: The calculation methods for ResourceScalingGroup and HPA replica counts are inconsistent; scaling cannot work as expected temporarily, and support is planned for subsequent versions.
Using InferNex to deploy inference on Ascend Atlas 300I (310P).
Deployment Key Points:
- In values.yaml configure inference cards mounted by inference service:
```
inference-backend:
  inferenceDevice: "huawei.com/Ascend310P"
```
- Ascend 310P card deploying inference service needs to use inference engine vLLM-Ascend's 310P exclusive image, for example quay.io/ascend/vllm-ascend:v0.18.0rc1-310p-openeuler, please adjust inference engine image to corresponding exclusive version in deployment used values.yaml.
- vLLM-Ascend's 310P exclusive image does not have built-in Mooncake, cannot complete KVCache direct transfer between Prefill and Decode in PD separation architecture, must adopt aggregate mode to deploy InferNex, can refer to InferNex repository example examples/vllm-aggregated-random-values.yaml. When using this example, delete inference-backend.services[0].kvTransferConfig configuration section, can disable Mooncake related capabilities.
- 310P card does not support bfloat16 data type and npu_dynamic_quant operator, can add the following parameters in values.yaml for inference-backend.services[0].aggregated.extraArgs:
  yaml
```
extraArgs:
  - "--enforce-eager"
  - "--dtype float16"
```
Note:
For inference engine complete Ascend 310P usage instructions please refer to vLLM-Ascend official documentation: Atlas 300I Online Inference on NPU.
InferNex through LWS+multi-DP deployment cache-indexer cannot normally subscribe inference instances
Phenomenon Description: cache-indexer when discovering multi-instances only tries to subscribe to one instance or appears L1 subscription connection refused error.
Note: Current version cache-indexer temporarily does not support vLLM ZMQ port offset subscription in LWS+DP scenario. It will directly read Pod spec's zmq-pub port name and assume corresponding port is available, while in multi-DP rank vLLM runtime port will increment from default 5557, offset to 5558 and other non-default ports, causing cache-indexer cannot correctly recognize.
Recommendation: This problem involves InferNex LWS+multi-DP deployment logic and cache-indexer L1 subscription logic adjustment, temporarily not recommended to enable cache-indexer in LWS+DP scenario.

Appendix

Custom Model Directory Configuration

When downloading models from HuggingFace, model files can be placed in custom folders. Just need to ensure mounted folder format is huggingface/hub/{model directory}, InferNex can normally recognize and use this model. For example, for model Qwen/Qwen3-8B, custom folder structure should be: {cache directory}/huggingface/hub/models--Qwen--Qwen3-8B (note / in model name will be converted to --). When configuring global.cachePath, just specify to {cache directory}, system will automatically recognize models in huggingface/hub directory.

The following script example downloads Qwen/Qwen3-8B model to mounted directory /home/llm_cache/.

python3 -c "
  import os
  from transformers import AutoModelForCausalLM, AutoTokenizer

  model_name = \"Qwen/Qwen3-8B\"

  tokenizer = AutoTokenizer.from_pretrained(
      model_name,
      cache_dir='/home/llm_cache/huggingface/hub',  
      force_download=True,  
      resume_download=True  # Resume download
  )

  model = AutoModelForCausalLM.from_pretrained(
      model_name,
      cache_dir='/home/llm_cache/huggingface/hub',  
      force_download=True,
      resume_download=True
  )
"

Inference Backend Default Mount Configuration

InferNex default configured inference backend's volumeMounts and volumes, including Ascend device related mounts. Below is default configuration example:

Default volumeMounts:

yaml

volumeMounts:
  ascend: # Ascend device related volumeMounts
    enable: true # Whether to enable Ascend related volumeMounts
    mounts:
      - name: shm
        mountPath: /dev/shm
      - name: dcmi
        mountPath: /usr/local/dcmi
      - name: npusmi
        mountPath: /usr/local/bin/npu-smi
      - name: lib64
        mountPath: /usr/local/Ascend/driver/lib64
      - name: version
        mountPath: /usr/local/Ascend/driver/version.info
      - name: installinfo
        mountPath: /etc/ascend_install.info
      - name: hccnconf
        mountPath: /etc/hccn.conf

Default volumes:

yaml

volumes:
  ascend: # Ascend device related volumes
    enable: true # Whether to enable Ascend related volumes
    mounts:
      - name: shm
        emptyDir:
          medium: Memory
          sizeLimit: "24Gi"
      - name: dcmi
        hostPath:
          path: /usr/local/dcmi
      - name: npusmi
        hostPath:
          path: /usr/local/bin/npu-smi
          type: File
      - name: lib64
        hostPath:
          path: /usr/local/Ascend/driver/lib64
      - name: version
        hostPath:
          path: /usr/local/Ascend/driver/version.info
          type: File
      - name: installinfo
        hostPath:
          path: /etc/ascend_install.info
          type: File
      - name: hccnconf
        hostPath:
          path: /etc/hccn.conf
          type: File

Ascend HCCS Device IP Address Configuration Example

The following script example shows how to configure device IP addresses and mask information in hccn.conf for multiple Ascend devices, can adjust according to actual device count and network planning:

bash

#!/bin/bash
for i in {0..7}
do
  hccn_tool -i $i -ip -s address 192.168.102.$i netmask 255.255.255.0
done

Offline Package Creation Guide

Obtain online chart package.
1.1 Obtain project installation package from openFuyao official image repository:
bash
```
helm pull oci://cr.openfuyao.cn/charts/infernex --version xxx
```
Where xxx needs to be replaced with specific project installation package version, such as 0.22.1. The pulled installation package is in compressed package format.
1.2 Extract installation package:
bash
```
tar -xzvf infernex-xxx.tgz
```
Where xxx needs to be replaced with specific project installation package version, such as 0.22.1.
Modify values.yaml configuration file.
In extracted chart package file infernex find values.yaml file, perform the following configuration modifications:
- Change global.image.pullPolicy value to Never, make system use local images instead of pulling from remote repository.
- Set global.env's HF_HUB_OFFLINE environment variable value to 1, enable HuggingFace offline mode, prevent inference engine startup from requesting Huggingface to download models.
Add image files.
Compress required images to tar.gz format files through nerdctl save command:
bash
```
nerdctl save -o xxx.tar.gz xxx
```
InferNex 0.22.1 version offline package default image list:
- cr.openfuyao.cn/openfuyao/eagle-eye-hardware-diagnosis:0.22.0
- cr.openfuyao.cn/openfuyao/eagle-eye-hardware-monitor:0.22.0
- cr.openfuyao.cn/openfuyao/npu-exporter:v7.2.RC1-of.1
- cr.openfuyao.cn/openfuyao/hermes-router:0.21.0
- cr.openfuyao.cn/openfuyao/cache-indexer:0.21.1
- cr.openfuyao.cn/openfuyao/huggingface-download:0.22.1
- hub.oepkgs.net/openfuyao/redis:8.6.1
- hub.oepkgs.net/openfuyao/mikefarah/yq:4.50.1
- cr.openfuyao.cn/openfuyao/elastic-scaler:0.20.0
- cr.openfuyao.cn/openfuyao/resource-scaling-group:0.20.0
- cr.openfuyao.cn/openfuyao/tidal:0.20.0
- hub.oepkgs.net/openfuyao/alpine/kubectl:1.34.2
- hub.oepkgs.net/openfuyao/prometheus/node-exporter:v1.8.2
- hub.oepkgs.net/openfuyao/kube-state-metrics/kube-state-metrics:v2.14.0
- hub.oepkgs.net/openfuyao/prometheus/alertmanager:v0.28.0
- hub.oepkgs.net/openfuyao/prometheus-operator/admission-webhook:v0.80.0
- hub.oepkgs.net/openfuyao/ingress-nginx/kube-webhook-certgen:v1.5.1
- hub.oepkgs.net/openfuyao/prometheus-operator/prometheus-operator:v0.80.0
- hub.oepkgs.net/openfuyao/prometheus-operator/prometheus-config-reloader:v0.80.0
- hub.oepkgs.net/openfuyao/thanos/thanos:v0.37.2
- hub.oepkgs.net/openfuyao/prometheus/prometheus:v3.1.0
- hub.oepkgs.net/openfuyao/nats:2.12.1-alpine
- hub.oepkgs.net/openfuyao/natsio/nats-server-config-reloader:0.20.1
- hub.oepkgs.net/openfuyao/natsio/prometheus-nats-exporter:0.17.3
- hub.oepkgs.net/openfuyao/busybox:1.36.1
- hub.oepkgs.net/openfuyao/istio/pilot:1.28.0
- hub.oepkgs.net/openfuyao/istio/proxyv2:1.28.0
- hub.oepkgs.net/openfuyao/ascend/vllm-ascend:v0.13.0
Add local model files.
Download model files according to Custom Model Directory Configuration instructions, and configure global.cachePath parameter to point to model directory.
Create offline package.
Package the following contents to create offline installation package:
- chart package file: Contains modified values.yaml configuration file.
- image files: Image tar.gz files compressed through nerdctl save command.
- model cache files: Compressed model directory files.

AI Inference Software Suite Mode Deployment

In openFuyao v26.03 version, AI inference software suite (AI inference software suite) related functionality has been merged into Infernex and continues to evolve. AI inference software suite originally positioned as lightweight inference software deployment solution facing integrated machine scenarios, based on openFuyao platform application marketplace achieving one-click installation deployment, supporting Kunpeng, Ascend affinity and mainstream CPU computing scenarios. After merging, users can deploy inference engines in aggregate mode through InferNex configuration items, achieving lightweight inference deployment capability equivalent to original AI inference software suite.

Below provides core specifications of original AI inference software suite and migration guide to InferNex.

Original AI Inference Software Suite Specifications Overview

Application Scenarios: Web scenarios and API interface scenarios, supporting calling large model inference capabilities through openAI API.
Deployment Method: One-click deployment of aiaio-installer application through openFuyao platform application marketplace.
Core Components: NPU Operator (or GPU Operator), KubeRay Operator.
Inference Engine: Based on vLLM, supporting vLLM v1 version.
Hardware Support: Ascend 910B/910B4, NVIDIA V100.
Model Support: HuggingFace existing models, such as DeepSeek-R1-Distill series (1.5B~70B).
API Interface: Follows openAI API specification, provides /v1/chat/completions interface.

Migration Guide

From AI inference software suite migration to InferNex, mainly involves deployment method and configuration method changes, inference API interface maintains compatibility.

Deployment Method Change
Original AI inference software suite one-click deploys aiaio-installer application through openFuyao platform application marketplace, after migration uses InferNex Helm Chart deployment. For specific deployment steps please refer to Installation Section.

Configuration Parameter Mapping

Original AI inference software suite's values.yaml configuration parameters and InferNex configuration parameters correspondence relationship as follows:

Table 1 AI Inference Software Suite and Infernex Configuration Parameter Mapping

Original AI Inference Software Suite Parameter	InferNex Configuration Parameter	Description
accelerator.NPU / accelerator.GPU	inference-backend.inferenceDevice	Specify inference chip type, NPU corresponds to `huawei.com/Ascend910`, GPU temporarily not supported.
accelerator.type	-	Current Infernex temporarily only supports NPU.
accelerator.num	-	Infernex supports automatic calculation of required accelerator count.
service.model	global.modelName	Inference model name.
service.tensor_parallel_size	aggregated.tensorParallelSize	Tensor parallelism.
service.pipeline_parallel_size	aggregated.pipelineParallelSize	Pipeline parallelism.
service.max_model_len	aggregated.maxModelLen	Model maximum sequence length.
service.vllm_use_v1	-	InferNex default uses vLLM v1 engine.
storage.size	-	Current Infernex supports direct mount of host directory, no need to fill in.

Model Recommended Configuration Mapping

Below is original AI inference software suite model recommended configuration corresponding configuration example in InferNex:

Table 2 AI Inference Software Suite Recommended Configuration and Infernex Configuration Correspondence Table

Model Scale	global.modelName	aggregated.tensorParallelSize	aggregated.pipelineParallelSize	Recommended Storage Size
1.5B	deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B	1	1	10Gi
7B	deepseek-ai/DeepSeek-R1-Distill-Qwen-7B	1	1	20Gi
8B	deepseek-ai/DeepSeek-R1-Distill-Llama-8B	1	1	25Gi
14B	deepseek-ai/DeepSeek-R1-Distill-Qwen-14B	2	1	40Gi
32B	deepseek-ai/DeepSeek-R1-Distill-Qwen-32B	4	1	80Gi
70B	deepseek-ai/DeepSeek-R1-Distill-Llama-70B	8	1	160Gi

API Interface Compatibility
After migration inference API interface maintains compatibility, still follows openAI API specification. Users can access /v1/chat/completions interface through InferNex deployed inference service address, request and response format consistent with original AI inference software suite. For specific usage please refer to Using AI Inference Section.
Note:
Original AI inference software suite used aiaio-installer application is no longer maintained after v26.03 version. If need to use integrated machine scenario lightweight inference deployment capability, please use InferNex aggregate mode deployment.

Configuration Example

This section gives configuration file using Infernex to deploy DeepSeek-R1-Distill-Qwen-7B, this file can also be obtained in openFuyao/InferNex repository examples/ai_software_suite directory.

yaml

inferenceGateway:
  enabled: false

global:
  image:
    pullPolicy: IfNotPresent
  imagePullSecrets: [] # Private image repository Secret, for example: [{"name": "registry-secret"}]

  env:
    - name: HF_HUB_OFFLINE # HuggingFace Hub offline switch (1=offline; 0=online)
      value: "0"

  modelName: "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B" # Inference model name
  cachePath: "/home/llm_cache" # Inference side cache directory (such as HuggingFace / vLLM download and compile cache)

hermes-router: 
  enabled: false

inference-backend:
  images: 
    inferenceEngine:
      repository: "hub.oepkgs.net/openfuyao/ascend/vllm-ascend"
      tag: "v0.13.0"

  inferenceDevice: "huawei.com/Ascend910"

  services:
  - name: vllm-aggregated-tp2-01
    enabled: true
    mode: aggregated # Inference backend uses aggregate mode
    service:
      port: 8000

    aggregated:
      replicas: 1
      enablePrefixCaching: true
      tensorParallelSize: 2
      pipelineParallelSize: 1
      dataParallelSize: 1
      gpuMemoryUtilization: 0.8
      blockSize: 128
      maxModelLen: 10000
      maxNumBatchedTokens: 40960
      extraArgs: [] # aggregated node extra vLLM startup parameters, for example: ["--dtype float16", "--max-num-seqs 256"]

    resources: # aggregated node resource configuration
      requests:
        cpu: "4"
        memory: "32Gi"
      limits:
        cpu: "8"
        memory: "64Gi"

# cache indexer
cache-indexer:
  enabled: false

eagle-eye:
  enabled: false

pd-orchestrator:
  elastic-scaler:
    enabled: false
  resourcescalinggroup:
    enabled: false
  tidal:
    enabled: false

External Interface Description

Table 3 InferNex External Interface Description

Interface Address	Access Method	Reason for Unable to Record Operation Logs	Custom Development Operation Log Entry	Other Description
/v1/completions	POST	InferNex itself does not provide user management capability, needs to connect to inference service management plane to have user information.	Audit tracing: needs to supplement user field in request body. Authentication authorization: needs to complete through HTTP header `Authorization: Bearer <API_KEY>` .	None
/v1/chat/completions	POST	InferNex itself does not provide user management capability, needs to connect to inference service management plane to have user information.	Audit tracing: needs to supplement user field in request body. Authentication authorization: needs to complete through HTTP header `Authorization: Bearer <API_KEY>` .	None

View source on GitCode

Bug

AI Inference Integrated Deployment ​

Feature Introduction ​

Application Scenarios ​

Capability Scope ​

Key Features ​

Implementation Principle ​

Relationship with Related Features ​

Related Examples ​

Installation ​

Prerequisites ​

Hardware Requirements ​

Software Requirements ​

Network Requirements ​

Permission Requirements ​

Start Installation ​

Standalone Deployment ​

Offline Installation ​

Configuring AI Inference Integrated Deployment ​

Prerequisites ​

Operation Steps ​

Using AI Inference ​

Prerequisites ​

Hardware Requirements ​

Software Requirements ​

Network Requirements ​

Background Information ​

Usage Limitations ​

Operation Steps ​

Related Operations ​

FAQ ​

Appendix ​

Custom Model Directory Configuration ​

Inference Backend Default Mount Configuration ​

Ascend HCCS Device IP Address Configuration Example ​

Offline Package Creation Guide ​

AI Inference Software Suite Mode Deployment ​

Original AI Inference Software Suite Specifications Overview ​

Migration Guide ​

Configuration Example ​

External Interface Description ​

AI Inference Integrated Deployment

Feature Introduction

Application Scenarios

Capability Scope

Key Features

Implementation Principle

Relationship with Related Features

Related Examples

Installation

Prerequisites

Hardware Requirements

Software Requirements

Network Requirements

Permission Requirements

Start Installation

Standalone Deployment

Offline Installation

Configuring AI Inference Integrated Deployment

Prerequisites

Operation Steps

Using AI Inference

Prerequisites

Hardware Requirements

Software Requirements

Network Requirements

Background Information

Usage Limitations

Operation Steps

Related Operations

FAQ

Appendix

Custom Model Directory Configuration

Inference Backend Default Mount Configuration

Ascend HCCS Device IP Address Configuration Example

Offline Package Creation Guide

AI Inference Software Suite Mode Deployment

Original AI Inference Software Suite Specifications Overview

Migration Guide

Configuration Example

External Interface Description