Addon User Guide ​

Background ​

The openFuyao community installs extensions through placing addon data (in yaml and chart forms) in BKECluster. The reconciler can perform extension installation when processing BKECluster. For an introduction to addons, see the Appendix section.

Prerequisites ​

None.

Restrictions ​

• For chart form addons, only provides chart installation, uninstallation, and upgrade configuration declaration framework. Host configuration modifications before and after chart addon installation need to be handled by users themselves.
• When a deployed chart addon contains sub-charts, decompressing the chart and modifying sub-chart values.yaml during deployment is not supported.
• In offline scenarios, users need to ensure that offline image registry and offline chart repository have the images and charts required by addons to be install

ed.

Procedure ​

Follow these steps to install addon plugins. Operations are identical for online and offline scenarios.

yaml Form Addons ​

1. Edit bc CR, add required addon components and required parameters.

   Need to declare custom addon parameters in Spec.ClusterConfig.Addons of the BKECluster CRD to ensure the reconciler can recognize and parse these parameters.

   Parameter format: Use the Param field to pass key-value pairs.

   Example: Customize an extension requiring image version, replica count, and custom labels:

   yaml

   # Configure custom addon in BKECluster CR
   spec:
     clusterConfig:
       addons:
       - name: "my-addon"
         version: "v1.0.0"
         param:
           image: "my-registry/my-addon:v1.0.0"
           replicas: "3"
           labels: "app=my-addon, env=prod"
         block: true # Whether to block entire process on deployment failure

   Parameter parsing: Code in cluster-api-provider-bke/pkg/kube injects these parameters into YAML templates through the prepareAddonParam function. No need to modify CRD structure, directly use the Param field to pass.

2. Prepare addon YAML templates.

   Need to place addon deployment YAML (such as Deployment, Service, etc.) in the specified directory to ensure the reconciler can traverse and render these templates.

   Directory structure: Under the /etc/openFuyao/addons/manifests/kubernetes directory, create directory <addon-name>/<version>/ and place custom yaml files, for example:

   text

   kubernetes/
     my-addon/
       v1.0.0/
         01-deployment.yaml
         02-service.yaml

   Template variables: Dynamic parameters (such as image address, replica count) that need replacement in YAML should use template placeholders. The reconciler will inject parameters through functions like parseFabricParam.

   Note:
   File name prefixes (such as 01-, 02-) affect deployment order (executed in ascending order), while deletion is executed in descending order.

   Below shows the 01-deployment.yaml file where template placeholders are injected with parameters.

   yaml

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: my-addon
     namespace: {{.namespace}} # Injected with cluster namespace by reconciler
   spec:
     replicas: {{.replicas}} # References replicas from addon parameters
     selector:
       matchLabels:
         app: my-addon
     template:
       metadata:
         labels: {{.labels}} # References labels from addon parameters
       spec:
         containers:
         - name: my-addon
           image: {{.image}} # References image from addon parameters

3. Handle addon special logic (optional).

   If addon requires pre/post operations (such as creating Secrets, configuring permissions, depending on other components), custom logic needs to be added to reconciler code.

   • Pre-operations: Add branches in EnsureAddonDeploy.addonBeforeCreateCustomOperate to handle pre-deployment preparation work (such as creating storage directories, configuring RBAC).
   • Post-operations: Add branches in EnsureAddonDeploy.addonAfterCreateCustomOperate to handle post-deployment finishing work (such as verifying status, registering resources).

4. Execute deployment.

   Configure custom addon in Spec.ClusterConfig.Addons of BKECluster CR. The reconciler will detect the new addon during the EnsureAddonDeploy stage and automatically execute the deployment process.
   4.1. Compare Spec (desired state) and Status.AddonStatus (actual state), trigger deployment on finding differences.
   4.2. Parse parameters and render YAML templates.
   4.3. Deploy to target cluster through kubectl apply via kube.InstallAddon.
   4.4. Update Status.AddonStatus to record deployment results.

5. Verify deployment.

   • Check whether corresponding resources are created in target cluster (kubectl get deployment my-addon -n <namespace>).
   • View BKECluster status (kubectl describe bkecluster <name> -n <namespace>), confirm ClusterAddonCondition is True and Status.AddonStatus contains record for my-addon.

6. Addon update and uninstallation (optional).

   • Update: Modify version or param of addon in BKECluster. Reconciler will automatically trigger update process (re-render YAML and fulfill).
   • Uninstallation: Remove addon from Spec.ClusterConfig.Addons. Reconciler will execute deletion operations in descending order by YAML file name (kubectl delete -f <file>).

chart Form Addons ​

Notice:
This function does not perform any security, integrity or compliance validation of chart package contents. Please ensure chart package security before deployment.

1. Edit bc CR, add chart repo.

   Need to declare chart repo in spec.clusterConfig.cluster.chartRepo of BKECluster CRD and declare chart repo authentication information.

   Example: Declare a private chart repo with self-signed certificate, example as follows.

   yaml

   # Configure custom addon in BKECluster CR
   spec:
     clusterConfig:
       cluster:
         chartRepo: 
           domain: my.chartrepo.cn  # Domain name, can fill IP address when chart repository is not configured with domain name 
           ip: 192.168.0.2          # IP address
           port: "443"              # Open port number
           prefix: openfuyao        # Chart domain prefix for pulling, for example this repository pulls chart url as my.chartrepo.cn/openfuyao
           insecureSkipTLSVerify: false     # Whether to skip TLS verification, default is false
           authSecretRef:                   # Chart repository authentication information, can ignore this field for public chart repositories
             name: chart-repo-auth-secret   # Secret name
             namespace: bke-cluster         # Secret namespace
             usernameKey: username          # Key storing username in Secret, defaults to username if not filled
             passwordKey: password          # Key storing password in Secret, defaults to password if not filled
           tlsSecretRef:                    # Chart repository TLS authentication information, can ignore this field when no TLS verification
             name: chart-repo-tls-secret    # Secret name
             namespace: bke-cluster         # Secret namespace
             certKey: tls.crt               # Key storing certificate in Secret, defaults to tls.crt if not filled
             caKey: ca.crt                  # Key storing CA in Secret, defaults to ca.crt if not filled
             keyKey: tls.key                # Key storing private key in Secret, defaults to tls.key if not filled

   Note:
   Supports configuring public chart repositories, private chart repositories, and chart repositories with TLS authentication. Supports pulling oci format charts and traditional format charts. Only one chart repo can be configured in one bc CR, pulling charts from multiple chart repos at once is not supported.

   If connecting to private chart repository, login authentication information needs to be saved in management cluster Secret. Secret structure example as follows.

   yaml

   # Secret YAML definition corresponding to authSecretRef
   apiVersion: v1
   kind: Secret
   metadata:
      name: chart-repo-auth-secret
      namespace: bke-cluster
   type: Opaque
   data:
      username: <base64-encoded-username>  # Example: Replace with actual base64 encoded username, e.g. echo -n 'your-username' | base64
      password: <base64-encoded-password>  # Example: Replace with actual base64 encoded password, e.g. echo -n 'your-password' | base64

   If connecting to self-signed certificate chart repository, TLS certificate authentication information needs to be saved in management cluster Secret. Secret structure example as follows.

   yaml

   # Secret YAML definition corresponding to tlsSecretRef
   apiVersion: v1
   kind: Secret
   metadata:
     name: chart-repo-tls-secret
     namespace: bke-cluster
   type: Opaque
   data:
     tls.crt: <base64-encoded-client-certificate>  # Example: Replace with actual base64 encoded client certificate content, e.g. cat client.crt | base64 -w 0
     ca.crt: <base64-encoded-ca-certificate>       # Example: Replace with actual base64 encoded CA certificate content, e.g. cat ca.crt | base64 -w 0
     tls.key: <base64-encoded-private-key>         # Example: Replace with actual base64 encoded private key content, e.g. cat client.key | base64 -w 0

2. Edit bc CR, add chart addon.

   Need to declare chart addon configuration in spec.clusterConfig.addons of BKECluster CRD.

   Example: Customize a logging-package addon and configure custom values.yaml, example as follows.

   yaml

   # Configure custom addon in BKECluster CR
   spec:
     clusterConfig:
       addons:
       - name: logging-package     # Chart name, will pull chart based on this name, required
         release: logging          # Release name, will create release based on this name, optional, defaults to addon name
         type: chart               # Plugin type, chart plugin needs to be specified as chart
         namespace: logging        # Namespace where chart is installed, required
         timeout: 5                # Chart installation, upgrade timeout, optional, defaults to 5 minutes
         valuesRefConfigMap:       # Chart package values.yaml reference configuration, optional, defaults to using chart package default values.yaml
           name: logging           # configmap name, required
           namespace: bke-cluster  # configmap namespace, required
           valuesKey: values.yaml  # values.yaml filename, optional, defaults to values.yaml
         version: 0.0.0-latest     # Chart version, required
         block: false              # Whether to block other addon deployment during deployment, optional, defaults to false

   If configuring custom values.yaml, values.yaml needs to be saved in management cluster ConfigMap. ConfigMap structure example as follows.

   Note:
   The values.yaml in the example needs to be modified to actual configuration. Actual values.yaml can be obtained from logging-operator repository.

   yaml

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: logging
     namespace: bke-cluster
   data:
     values.yaml: |  # The following values.yaml content is example configuration, needs to be modified to actual configuration
       replicaCount: 2
       image:
         repository: cr.openfuyao.cn/openfuyao
         tag: "2.4"
       service:
         type: ClusterIP
         port: 80

   Notice:
   Modifying the configMap storing values.yaml after deploying chart will not trigger reconciler to update chart addon. For offline scenarios, users need to modify image pull addresses in values.yaml themselves.

3. Handle addon special logic (optional).

   If addon requires pre/post operations (such as creating Secrets, configuring permissions, depending on other components), custom logic needs to be added to reconciler code.

   • Pre-operations: Add branches in EnsureAddonDeploy.addonBeforeCreateCustomOperate to handle pre-deployment preparation work (such as creating storage directories, configuring RBAC).
   • Post-operations: Add branches in EnsureAddonDeploy.addonAfterCreateCustomOperate to handle post-deployment finishing work (such as verifying status, registering resources).

4. Execute deployment.

   Configure custom addon in spec.clusterConfig.addons of BKECluster CR. The reconciler will detect the new addon during the EnsureAddonDeploy stage and automatically execute the deployment process.
   4.1. Compare Spec (desired state) and Status.AddonStatus (actual state), trigger deployment on finding differences.
   4.2. Pull chart package to local temporary file through helm sdk via kube.InstallAddon based on addon name and version from bkemanifests.
   4.3. Deploy chart to target cluster via helm sdk.
   4.4. Update Status.AddonStatus to record deployment results.

5. Verify deployment.

   • Check whether corresponding resources are created in target cluster (kubectl get deployment my-addon -n <namespace>).
   • View BKECluster status (kubectl describe bkecluster <name> -n <namespace>), confirm ClusterAddonCondition is True and Status.AddonStatus contains record for my-addon.

6. Addon update and uninstallation (optional).

   • Update: Modify version of addon in BKECluster. Reconciler will automatically trigger update process.
   • Uninstallation: Remove addon from spec.clusterConfig.addons. Reconciler will automatically delete chart addon.

Appendix ​

Below introduces common extension situations.

• K8s Ecosystem Components

  yaml

  - name: kubeproxy
    version: v1.33.1-of.1
    block: true # Optional, defaults to false. Setting to true means blocking, i.e. waiting for this component to start before deploying subsequent extensions
    param: # Parameters required for deploying this component
      clusterNetworkMode: calico
  - name: calico
    version: v3.25.0
    param:
      calicoMode: bgp
  - name: coredns
    version: v1.10.1
  - name: nfs-csi
    version: v4.1.0
    block: true
    param:
      nfsServer: 172.28.8.186

  • The reconciler deploys extensions in configuration order.
  • If an extension's installation strongly depends on services started by preceding extensions, the block parameter of preceding extensions can be set to true for blocking installation.
  • If non-network extensions need to be installed, place them after network extensions.

• Cluster Network Components

  Note:
  1. Please use different combinations according to the examples below based on actual situation for network components, and write them at the front of addons.
  2. To ensure subsequent components can start normally, network-related addons shown below all need to add the block: true parameter.
  3. For detailed addon parameters and parameter descriptions, please see the examples below.

  Below introduces two common network plugins, Fabric and Calico.

  Note:
  IPVS is only supported when using Calico plus kube-proxy. Fabric does not support IPVS.

  • Fabric

    Fabric supports both Overlay and Underlay modes.

    • Overlay Mode

      Pods use independent CIDR, communicate across nodes through VXLAN tunnel encapsulation, suitable for scenarios with limited underlying networks.

      yaml

      addons:
      - name: fabric
        block: true
        param:
          fabricMode: overlay
          cidrBlock: "10.250.0.0/16" # Fill according to actual situation, after filling also set bkecluster.spec.clusterConfig.cluster.networking.podSubnet to match
          excludeIps: "" # Fill with "" if none, supports IP ranges and single IPs, separated by English comma. eg:"10.250.0.1-10.250.0.10,10.250.0.55"
          tunnelType: vxlan # geneve gre erspan vxlan
          subCIDRMask: "24" # Fill according to actual situation
        version: 2.6.2
      - name: coredns
        block: true
        version: v1.10.1

    • Underlay Mode

      Pods directly use VPC subnet IP addresses, rely on underlying network for communication, no encapsulation, high performance.

      yaml

      addons:
      - name: fabric
        block: true
        param:
          fabricMode: underlay
          cidrBlock: "10.250.0.0/16" # Fill according to actual situation, after filling also set bkecluster.spec.clusterConfig.cluster.networking.podSubnet to match
          excludeIps: "" # Fill with "" if none, supports IP ranges and single IPs, separated by English comma. eg:"10.250.0.1-10.250.0.10,10.250.0.55"
          vlanID: "" # Fill according to actual situation
          gateway: "10.250.0.254" # Fill according to actual situation
          mask: "24" # Fill according to actual situation
          dataPlaneUnified: "eth1" # Fill according to actual situation
        version: 2.6.2
      - name: coredns
        block: true
        version: v1.10.1

  • Calico

    Calico is Kubernetes' network and network policy plugin. Calico-Typha is its intermediate cache proxy component used to reduce data storage load in large-scale clusters. By aggregating Felix connections, filtering irrelevant updates and horizontal scaling, it significantly reduces Kubernetes API Server/etcd pressure and optimizes network policy distribution efficiency. Parameters can be adjusted according to actual business needs below.

    • calicoMode: Used to set Calico's network mode, options are vxlan, bgp, ipip.
    • proxyMode: Used to set kube-proxy's forwarding mode, options are iptables, ipvs, this parameter is optional.
    • allowTypha: Used to configure whether to enable Typha. Typha is used to reduce data storage cache. The mode selection of calicoMode and proxyMode does not affect Calico-Typha usage.
    • typhaReplicas: Used to set Typha replica count. According to calico official documentation, it is recommended to enable Calico-Typha when cluster scale exceeds 50 nodes. For every additional 200 nodes, at least 1 Typha replica should be added, while ensuring total Typha replicas is less than total cluster nodes.
    yaml

    # kube-proxy version matches k8s version
    - name: kubeproxy
      param:
        proxyMode: iptables # iptables, ipvs
      version: v1.33.1-of.1
      block: true
    - name: calico
      param:
        calicoMode: vxlan # vxlan bgp ipip
        allowTypha: true
        typhaReplicas: 2 # Fill according to actual situation
      version: v3.31.3 # Only version 3.31.3 and above support typha related configuration, otherwise ineffective.
      block: true
    - name: coredns
      version: v1.10.1
      block: true

• DNS Cache Components

  DNS cache components are middleware between DNS clients and DNS servers, used to cache DNS query results, reducing repeated query latency and network overhead.

  • node-local-dns

    node-local-dns as a Kubernetes cluster internal DNS resolution enhancement component, can cache DNS query results at node level, effectively reducing CoreDNS load, improving service discovery speed and reducing network latency. Also supports automatic retry and failover to enhance DNS resolution reliability, improving cluster business stability and concurrent processing capability in very large scale clusters. Adjust configuration according to actual business needs through the following parameters:

    • localdns: Used to configure listening address. In IPTABLES mode, node-local-dns Pod will listen to both kube-dns service IP address and <localdns> address, so Pods can use either IP address to query DNS records. In IPVS mode, because the interface used by IPVS load balancing has already occupied this address, node-local-dns Pod only listens to <localdns> address, so node-local-dns interface cannot bind kube-dns cluster IP address.
    yaml

    - name: nodelocaldns            
      param:
        localdns: 10.96.0.10 # Fill according to actual situation, be careful to avoid conflicts with already used ports
      version: v1.26.4