Cluster Deployment Issue Diagnosis Guide ​

This document aims to provide preliminary guidance on diagnosing cluster deployment issues for users deploying clusters using the openFuyao cluster installation tool.

Background ​

The bke cluster installation tool provided by the openFuyao community offers the following features:

• Bootstrap node initialization, using nerdctl to start containers such as k3s, registry, and nginx. The k3s container further starts the k3s cluster with the built-in default flannel network plugin. The registry and nginx containers serve as the image source and file source services, respectively.
• The bootstrap node starts the management cluster and business clusters using the calico network plugin, assigns Pod IP addresses, and uses containerd runtime to start containers.
• The management cluster starts business clusters.

Prerequisites ​

Clusters deployed using the openFuyao community's bke cluster installation tool.

Restrictions ​

None.

Usage Scenarios ​

• Offline installation package preparation failure.
• Bootstrap node initialization failure.
• Bootstrap node K8s cluster deployment failure.

Reference Documentation ​

For detailed issues related to installation and deployment, please refer to the Installation and Deployment FAQ.

Offline Installation Package Preparation Failure ​

When deploying clusters in an offline environment using the openFuyao installation tool, offline installation packages need to be prepared in advance.

Common Issues and Information Collection ​

The offline installation package preparation process is uninterrupted and completed in one go. When an interruption occurs, it indicates preparation failure. The main diagnostic input information is as follows:

• Terminal output logs near the interruption point during offline package preparation.
• docker pull cr.openfuyao.cn/openfuyao/installer-service:latest Manually attempt to pull the image and provide terminal output results.

Bootstrap Node Initialization Failure ​

Execute the following command to initialize the bootstrap node.

# Online environment, if you don't need to install the openFuyao management plane, add the parameter --installConsole=false
bke init --onlineImage cr.openfuyao.cn/openfuyao/bke-online-installed:1.2.2

# Offline environment, if you don't need to install the openFuyao management plane, add the parameter --installConsole=false
bke init

After successful bootstrap cluster installation, execute the following commands to view container and cluster Pod information.

# View containers started by nerdctl, all containers should have UP status
nerdctl ps -a
# You can use the following command to enter a specific container
nerdctl exec -it <container-id> -- sh

# View k3s cluster Pod information, kubeconfig is copied from /etc/rancher/k3s/k3s.yaml to /root/.kube/config
kubectl get pod -A -owide
# Note: When viewing k3s cluster pods during co-deployment of k3s cluster and k8s cluster, use the following command
kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get pod -A -owide

Common Issues and Information Collection ​

This section introduces some common errors and diagnostic input information to be collected.

nerdctl Container Startup Failure ​

Bootstrap node initialization uses nerdctl to start containers, as described below:

• bocloud_nfs_registry: Provides network file management functionality, currently unused, startup failure does not affect operation.
• bocloud_chart_registry: Provides chart source service, local http chart source, commonly used during offline installation.
• bocloud_yum_registry: Provides file source service, essentially an nginx server, required for both online and offline installation.
• bocloud_image_registry: Provides image source service, image source during offline installation.
• kubernetes: k3s container, used to subsequently start cluster Pods, uses flannel network plugin by default.

If container startup fails, the main diagnostic input information is as follows:

• Terminal output logs during init initialization.
• nerdctl ps -a provides container status information.
• nerdctl inspect container-id provides detailed information about the abnormal container.

k3s Cluster Pod Startup Failure ​

After the k3s container successfully starts, k3s cluster Pods will be subsequently deployed. A brief introduction by namespace follows:

• openfuyao-system: Namespace where openFuyao components are located, mainly including authentication, user management, plugin management, console, and cluster management related Pods, optional deployment.
• ingress-nginx: Provides ingress entry service, works with Pods in the openfuyao-system namespace to provide openFuyao management plane functionality.
• kube-system: This namespace only has coredns, works with Pods in the openfuyao-system namespace to provide openFuyao management plane functionality.
• cluster-system: Deploys cluster-api reconciler related Pods, provides cluster management capabilities, must be deployed.

If Pod startup fails, the main diagnostic input information is as follows:

• kubectl get pod -A -owide provides Pod status information.
• kubectl logs -n ns pod-name provides log information for the abnormal Pod.

Bootstrap Node K8s Cluster Installation ​

Execute the following command to install the K8s cluster.

# Install K8s cluster using backend, users need to replace the files specified by -f and -n parameters
bke cluster create -f /bke/cluster/1master-cluster.yaml -n /bke/cluster/1master-node.yaml

# Install using the bootstrap cluster's openFuyao management plane, directly log in to https://<bootstrap cluster ip>:30010, enter cluster information to install

The simplified process for bootstrap node K8s cluster installation is as follows:

Figure 1 Simplified Process for Bootstrap Node K8s Cluster Installation

After successful K8s cluster installation, execute the following commands to view container and cluster Pod information.

# View containers started by containerd, each node can only view containers on its own node
crictl ps -a

# View k8s cluster Pod information
kubectl get pod -A -owide

Common Issues and Information Collection ​

Execute kubectl get bc -A on the bootstrap node to view cluster status and determine the current failure stage.

PushAgent Failure ​

The bootstrap node pushes bkeagent node agent to each node of the cluster to be installed. bkeagent runs as a system service and listens to the bootstrap node's APIServer.

If bkeagent push fails, the main diagnostic input information is as follows:

• Logs from the bke-controller-manager-xxxxxx Pod on the bootstrap node, obtained by executing the following command:

  bash

  # Note: bke-controller-manager-xxxxxx needs to be replaced with the actual Pod name
  kubectl logs -n cluster-system bke-controller-manager-xxxxxx -c manager

• bkeagent logs on each cluster node, for new versions the log path is /var/log/openFuyao/bkeagent.log, for versions v25.12 and earlier the log path is /var/log/bkeagent.log.

NodeEnvInit Failure ​

After successfully pushing bkeagent, the bootstrap node issues commands (a custom CRD), and the node's bkeagent monitors the commands to perform environment initialization operations, mainly as follows:

• Install containerd.
• Download scripts and perform pre-processing operations.
• Set kernel parameters.

When environment initialization fails, it is mainly due to containerd installation failure. The main diagnostic input information is as follows:

• systemctl status containerd query containerd status.
• journalctl -xu containerd -n 200 view containerd service logs.
• bkeagent logs on each cluster node, for new versions the log path is /var/log/openFuyao/bkeagent.log, for versions v25.12 and earlier the log path is /var/log/bkeagent.log.

MasterInit Failure ​

After successful environment initialization, master node initialization operations continue, mainly as follows:

• Generate static Pod yaml files in the /etc/kubernetes/manifests directory, including etcd, kube-apiserver, kube-controller-manager, and kube-scheduler.
• Install kubelet.
• Pre-pull images for k8s components, including etcd, kube-apiserver, kube-controller-manager, and kube-scheduler.

For MasterInit failure, the main diagnostic input information is as follows:

• Execute command systemctl status kubelet to query kubelet status.
• Execute command journalctl -xu kubelet -n 200 to view kubelet service logs.
• Execute command crictl images to query pre-pulled images, crictl ps -a to view container status.
• Execute command crictl logs container-id to query abnormal container logs.

AddonDeploy Failure ​

After successful master node initialization, addon installation continues. The K8s cluster started by openFuyao starts Pods at the addon granularity. Common community addons include calico, coredns, bkeagent-deployer, cluster-api, and openfuyao-system-controller. User-defined addons can also be installed. This step operation retrieves the addon yaml files from the bke-controller-manager Pod on the bootstrap node, renders parameters, and fulfills them to the APIServer through the K8s cluster's k8sClient, thus starting the corresponding Pods.

For AddonDeploy failure, the main diagnostic input information is as follows:

• Logs from the bke-controller-manager-xxxxxx Pod on the bootstrap node, obtained by executing the following command:

  bash

  # Note: bke-controller-manager-xxxxxx needs to be replaced with the actual Pod name
  kubectl logs -n cluster-system bke-controller-manager-xxxxxx -c manager

• Execute the following commands on K8s cluster nodes to view information:

  bash

  # View Pod information
  kubectl get pod -A -owide
  
  # View Pod logs, replace ns-name with actual namespace, pod-name with actual Pod name
  kubectl logs -n ns-name pod-name
  
  # View detailed Pod information, replace ns-name with actual namespace, pod-name with actual Pod name
  kubectl describe pod -n ns-name pod-name