Backend Declarative Upgrade Guide ​

Backend Declarative Upgrade Introduction ​

Application Scenarios ​

Declarative upgrade is a cluster version upgrade method provided by openFuyao based on declarative APIs. Users only need to declare the desired target version, and the system will automatically complete the entire process including upgrade path validation, upgrade resource preparation, orderly component upgrades, and status tracking. Through backend upgrades, users can view phase information during the upgrade process and understand the upgrade execution steps.

Applicable to the following scenarios:

• Upgrading the openFuyao version of a cluster from v26.06-rc.2 and subsequent versions to a higher version.
• Cluster version upgrades in both online and offline environments.

Capability Scope ​

• Declarative upgrade trigger: Modify the desiredVersion field of ClusterVersion CR (Custom Resource) to trigger the upgrade, and the system automatically issues upgrade instructions.
• Automatic upgrade path validation: Based on UpgradePath rules, automatically validates whether the upgrade path from the current version to the target version is valid.
• Automatic multi-hop upgrade execution: When no direct upgrade path exists, the system automatically executes upgrades hop by hop through intermediate versions.
• DAG (Directed Acyclic Graph) driven parallel upgrade: Upgrade components build a DAG topology based on dependency relationships, supporting parallel upgrades of components in the same batch.
• Both online and offline scenarios are supported.
• Both management clusters and business clusters can be upgraded. Business cluster upgrades must be performed on the cluster managing their lifecycle (management cluster or bootstrap node).

Note:
Declarative upgrade is triggered by modifying ClusterVersion.spec.desiredVersion; after validation, ClusterVersionReconciler writes the cvo.openfuyao.cn/upgrade-ready annotation on BKECluster, and BKEClusterReconciler executes the DAG accordingly. For controller parameters and annotations, see Controller Parameters and Upgrade Annotations below.

Highlight Features ​

• Declarative trigger: Only need to declare the target version (modify the desiredVersion field of ClusterVersion CR), without worrying about upgrade execution details.
• DAG parallel upgrade: Compared to the legacy serial PhaseFlow upgrade, declarative upgrade supports components executing in parallel based on dependency topology, reducing overall upgrade time.
• Automatic multi-hop execution: During cross-version upgrades, the system automatically calculates intermediate hop versions and executes hop by hop, requiring no manual intervention from users.

Basic Concepts ​

Note:
For upgrade component types (inline / yaml / helm), see Before You Upgrade - Upgrade Component Types.

The trigger and execution flow of declarative upgrade is as follows:

Flow step descriptions:

1. Modify desiredVersion: Modify the spec.desiredVersion field of ClusterVersion CR to the target version.
2. Detect Version Change: ClusterVersionReconciler detects that desiredVersion is inconsistent with currentVersion, triggering the upgrade flow.
3. Query Upgrade Path: ClusterVersionReconciler queries the UpgradePath CR, validating whether the upgrade path from the current version to the target version is valid.
4. Ensure Resource Exists: Ensure the ReleaseImage CR exists and its status is Valid; the system automatically pulls and parses it from the OCI image repository.
5. Write Upgrade Annotation: Write the upgrade-ready annotation on BKECluster, triggering BKEClusterReconciler to enter the upgrade flow.
6. Detect Upgrade Annotation: BKEClusterReconciler detects the upgrade-ready annotation on BKECluster and enters the declarative upgrade flow.
7. Parse Upgrade Components: Parse ReleaseImage to obtain the upgrade component list and dependency relationships.
8. Build DAG Topology: Build the DAG topology based on the upgrade component list from ReleaseImage and the dependency relationships from ComponentVersion.
9. Execute Upgrades in Parallel: Execute component upgrades in parallel by DAG topology batches; inline, yaml, and helm components are dispatched to their respective executors based on spec.type.
10. Determine Upgrade Success: Check whether all components have been upgraded successfully.
11. Update Status: After successful upgrade, update BKECluster and ClusterVersion status; update DeclarativeUpgradeStatus after each component completes.
12. Record Failed Components: When upgrade fails, record the failed component information.
13. Determine Whether Target Version Reached: Check whether the current version equals the target version; if not, continue executing the next hop upgrade.
14. Upgrade Complete: The current version equals the target version; all upgrades are complete, and upgrade annotations are cleared.

Controller Parameters and Upgrade Annotations ​

Declarative upgrade is triggered by the ClusterVersion controller and executed by the BKECluster controller according to the DAG defined in ReleaseImage. After the user modifies ClusterVersion.spec.desiredVersion, ClusterVersionReconciler validates UpgradePath and ensures ReleaseImage is valid, then writes upgrade-ready and related annotations to BKECluster; BKEClusterReconciler executes the DAG when non-empty upgrade-ready is detected.

Table 1 Declarative Upgrade Parameters and Annotations

Collaboration

• Execution gate: shouldUseDeclarativeUpgrade only checks whether cvo.openfuyao.cn/upgrade-ready is non-empty.
• Helm components: HelmComponentEnabled uses annotation cvo.openfuyao.cn/helm-component when present; otherwise falls back to the --helm-component-support global flag.
• Cleanup: After a hop completes, is cancelled, or fails, the controller removes upgrade-ready, cluster-version, and upgrade-path annotations together.

Usage Restrictions ​

Notice:
For the upgrade system overview and guide navigation, see Before You Upgrade. For upgrading from v26.03 to v26.06, refer to Upgrade Guide. The following guide applies to upgrading from v26.06-rc.2 and subsequent versions.

Online Scenario Upgrade Operations ​

Prerequisites ​

• The management cluster is running normally.
• The management cluster can access the remote version repository.
• The cluster to be upgraded has a status of Healthy.

Note:
If the target version includes a bkeagent upgrade (EnsureAgentUpgrade in the upgrade DAG), download the target-version bkeagent binaries (amd64/arm64) in advance, then copy them to the bootstrap node's /bke/mount/source_registry/files/ directory. The following example uses v26.09 (component version 26.9.0); replace the version number with the component version of your target release:

bash

curl -L -o bkeagent-26.9.0-linux-arm64 https://openfuyao.obs.cn-north-4.myhuaweicloud.com/openFuyao/cluster-api-provider-bke/releases/download/26.9.0/bkeagent_linux_arm64
curl -L -o bkeagent-26.9.0-linux-amd64 https://openfuyao.obs.cn-north-4.myhuaweicloud.com/openFuyao/cluster-api-provider-bke/releases/download/26.9.0/bkeagent_linux_amd64
cp bkeagent-26.9.0-linux-arm64 bkeagent-26.9.0-linux-amd64 /bke/mount/source_registry/files/

Step 1: Check Cluster Status ​

# View business cluster status
kubectl get bkecluster <cluster-name> -n <namespace> 

# Expected output: Healthy
# If the output is not Healthy, troubleshoot and fix cluster issues first

Step 2: Query Upgradable Paths ​

# View upgrade path rules in UpgradePath CR
kubectl get upgradepath openfuyao-upgrade-paths -o yaml

Step 3: Trigger Upgrade ​

Operate ClusterVersion CR via kubectl

If you want to upgrade to v26.09, you can execute the following operations to trigger the upgrade.

# Patch update desiredVersion
kubectl patch clusterversion <cluster-name> -n <namespace> --type merge -p '{"spec":{"desiredVersion":"v26.09"}}'
# Or edit the clusterversion resource, changing desiredVersion to v26.09
kubectl edit clusterversion <cluster-name> -n <namespace>

Note:
Modifying the desiredVersion field of ClusterVersion CR is the only way to trigger declarative upgrade. After the system detects that desiredVersion is inconsistent with currentVersion, it automatically enters the upgrade flow.

Step 4: Monitor Upgrade Progress ​

View various statuses during the upgrade process via kubectl:

# View ClusterVersion status
kubectl get clusterversion <cluster-name> -n <namespace> -w

ClusterVersion phase field meanings:

# View overall cluster health status
kubectl get bkecluster <cluster-name> -n <namespace> -w

Declarative upgrade phase sequence (executed by DAG topology):

Note:
1. Components within the same batch execute in parallel; different batches execute serially in dependency order. Batches 1~6 are automatically determined by DAG topology sorting; dependency relationships come from the spec.dependencies field of each ComponentVersion in ReleaseImage.
2. In Batch 5, Master Upgrade and Worker Upgrade execute in parallel; in Batch 6, kube-proxy, coredns, and calico execute in parallel.
3. In multi-hop upgrade scenarios, each hop executes all the above phases before automatically proceeding to the next hop.
4. After provider self-upgrade completes, a new Pod will take over the subsequent upgrade process, and the old Pod will automatically exit.

Step 5: Confirm Upgrade Completion ​

# Confirm ClusterVersion currentVersion equals desiredVersion
kubectl get clusterversion <cluster-name> -n <namespace> -o jsonpath='{.status.currentVersion}'

# Confirm ClusterVersion phase is Ready
kubectl get clusterversion <cluster-name> -n <namespace> -o jsonpath='{.status.phase}'

# Confirm cluster status has returned to Healthy
kubectl get bkecluster <cluster-name> -n <namespace> -o jsonpath='{.status.clusterHealthState}'

Offline Scenario Upgrade Operations ​

Prerequisites ​

• The management cluster is running normally and in an offline environment.
• The cluster is in offline mode.
• The cluster to be upgraded has a status of Healthy.
• The version configuration file has been prepared (for preparing the full offline package); offline package preparation must be done in an online environment, and the prepared offline package is then uploaded to the bootstrap node.

Offline Package Preparation Step 1: Prepare Version Configuration File ​

1. Download the openFuyao version index file to obtain the correspondence between versions and file names.

   bash

   wget https://openfuyao.obs.cn-north-4.myhuaweicloud.com/openFuyao/version-config/index.yaml

   Index file sample content:

   yaml

   - openFuyaoVersion: v26.09
     filePath: ./Core-VersionConfig-v26.09.yaml

2. Download the target version configuration file based on the index file.

   bash

   wget https://openfuyao.obs.cn-north-4.myhuaweicloud.com/openFuyao/version-config/Core-VersionConfig-v26.09.yaml

Offline Package Preparation Step 2: Prepare Full Offline Package ​

Based on the version configuration file downloaded in Offline Package Preparation Step 1: Prepare Version Configuration File, prepare the full offline package required for upgrade.

Prepare the full offline package based on the version configuration file. Here, v26.09 is used as an example; replace the file name and offline package name with the actual target version:

bke build patch -f Core-VersionConfig-v26.09.yaml -t offline-v26.09.tar.gz --strategy=oci

Note:
Here, v26.09 is used as an example; please replace the version number in the command with your actual target upgrade version number.

The full offline package contains the following:

Note:
1. The version configuration file defines all images, binary files, and dependency packages required for the full offline package, serving as the input configuration for offline package preparation.
2. The full offline package ensures that all resources required for upgrade are available in the offline environment, avoiding upgrade failures due to missing dependencies.

Offline Upgrade Step 1: Upload Offline Package to Bootstrap Node ​

1. Place the full offline package file in the /etc/openFuyao directory on the bootstrap node.

2. Synchronize the offline package to the bootstrap node:

   bash

   # Log in to the bootstrap node
   
   # Extract the full offline package
   tar -xzvf /etc/openFuyao/offline-v26.09.tar.gz
   
   # Synchronize images to the local image source
   bke registry patch --source ./offline-v26.09 --target <bootstrap-node-IP>:40443
   
   # Copy binary files to the specified directory
   # Copy the binary files required for upgrade from ./offline-v26.09/volumes/ to /bke/mount/source_registry/files/, example command as follows
   cp ./offline-v26.09/volumes/containerd-*.tar.gz /bke/mount/source_registry/files/

   Note:
   1. Here, v26.09 is used as an example; please replace the version number in the command with your actual target upgrade version number.
   2. If the target version includes a bkeagent upgrade, also confirm that versioned bkeagent binaries exist under /bke/mount/source_registry/files/ (see the preparation note in Online Scenario Prerequisites); if the offline package does not include them, download and copy them separately as described there.

Offline Upgrade Step 2: Check Cluster Status ​

Same as the online scenario:

kubectl get bkecluster <cluster-name> -n <namespace>

Offline Upgrade Step 3: Trigger Upgrade ​

Same as the online scenario:

If you want to upgrade to v26.09, you can execute the following operations to trigger the upgrade.

# Patch update desiredVersion
kubectl patch clusterversion <cluster-name> -n <namespace> --type merge -p '{"spec":{"desiredVersion":"v26.09"}}'
# Or edit the clusterversion resource, changing desiredVersion to v26.09
kubectl edit clusterversion <cluster-name> -n <namespace>

Offline Upgrade Step 4: Monitor Upgrade Progress ​

Same as the online scenario, refer to Online Scenario - Monitor Upgrade Progress.

Offline Upgrade Step 5: Confirm Upgrade Completion ​

Same as the online scenario, refer to Online Scenario - Confirm Upgrade Completion.

Upgrade Failure Handling ​

View Failure Details ​

# View ClusterVersion status
kubectl get clusterversion <cluster-name> -n <namespace> -o yaml

# View failure records in DeclarativeUpgradeStatus
kubectl get bkecluster <cluster-name> -n <namespace> -o jsonpath='{.status.declarativeUpgrade.lastFailure}'

# View cluster status
kubectl get bkecluster <cluster-name> -n <namespace> -o jsonpath='{.status.clusterHealthState}'

Common Failure Causes ​

FAQ ​

• What is the difference between declarative upgrade and legacy PhaseFlow upgrade?

  Comparison Item	Legacy PhaseFlow	Declarative Upgrade
  Trigger Method	Modify BKECluster.spec.openFuyaoVersion	Modify ClusterVersion.spec.desiredVersion
  Execution Method	Serial execution of all upgrade phases	DAG topology driven parallel execution
  Component Order	Fixed sequence	Dynamically determined by dependency relationships
  Multi-hop Upgrade	Not supported	Supports automatic hop-by-hop execution

• Why is the upgrade version list empty?

  bash

  # Check whether UpgradePath CR exists
  kubectl get upgradepath
  
  # Check path rules
  kubectl get upgradepath openfuyao-upgrade-paths -o jsonpath='{.spec.paths}'

  Possible causes: 1) The current version is already the latest version; 2) The UpgradePath rules have no reachable upgrade paths for the current version; 3) In online mode, the remote version repository is unreachable.

• What is multi-hop upgrade?

  When no direct upgrade path exists from the current version to the target version (such as v26.06-rc.2 to v26.06), the system automatically calculates the intermediate hop version path (such as v26.06-rc.2 -> v26.06-rc.3 -> v26.06) and executes upgrades hop by hop. After each hop completes, it automatically proceeds to the next hop until reaching the final target version.

• Why is there a brief interruption after provider self-upgrade during the upgrade process?

  Provider self-upgrade uses the Kubernetes Deployment rolling update mechanism: after the old Pod's image is patched, K8s creates a new Pod, and after the new Pod is Ready, the old Pod is terminated. The new Pod will automatically take over the subsequent upgrade process after starting. This process typically completes within 1-2 minutes.