Binary File Custom Configuration Guide

Background

This document describes how to use Custom Resources (CR) to dynamically configure Kubelet and Containerd components. In bc, there are references to kubelet and containerd configuration CRs. The kubelet CR is abbreviated as kct, and the containerd CR is abbreviated as cct. You can edit kct and cct to import custom kubelet and containerd configuration items.

Prerequisites

Use CRs to complete Kubelet and Containerd configuration during cluster creation.

Restrictions

None.

Guide

KubeletConfig CR Configuration

Overview

The KubeletConfig CR allows you to dynamically configure Kubelet's configuration file and systemd service file through Kubernetes resources, without manually editing files on nodes.

CR Structure

bash

apiVersion: bke.bocloud.com/v1beta1
kind: KubeletConfig
metadata:
  name: <config-name>
  namespace: <namespace>  # Default is bke-system
spec:
  kubeletConfig:
    kubelet.conf:
      raw: |
        # Kubelet configuration file YAML content
  kubeletService:
    service:
      # Kubelet systemd service configuration

spec Field Description

spec.kubeletConfig

Used to configure Kubelet's configuration file (/var/lib/kubelet/config.yaml).

Parameter	Type	Required	Description
`kubelet.conf`	`map[string]RawExtension`	Yes	Configuration object containing `raw` field
`kubelet.conf.raw`	`string`	Yes	Complete YAML content of Kubelet configuration file

raw Field Format:

The raw field should contain the complete Kubelet configuration file YAML content, for example:

yaml

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
clusterDNS:
  - 10.96.0.10
clusterDomain: cluster.local
...

kubelet.config implements passthrough of the filled kubelet.conf.raw, and users need to ensure that the imported configuration items are correct.

spec.kubeletService

Used to configure Kubelet's systemd service file (/etc/systemd/system/kubelet.service).

Parameter	Type	Required	Description
`service`	`KubeletService`	No	Kubelet systemd service configuration object

kubeletService Field Description:

Parameter	Type	Required	Description
`ExecStart`	`string`	No	Service startup command
`Restart`	`string`	No	Restart policy (e.g., always, on-failure)
`RestartSec`	`string`	No	Restart interval (e.g., 5s, 10s)
`StartLimitInterval`	`string`	No	Start limit time window
`StartLimitBurst`	`int`	No	Start limit count
`KillMode`	`string`	No	Termination mode (e.g., control-group, process)
`StandardOutput`	`string`	No	Standard output redirect (e.g., journal, null)
`StandardError`	`string`	No	Standard error redirect
`SyslogIdentifier`	`string`	No	Syslog identifier
`WorkingDirectory`	`string`	No	Working directory
`User`	`string`	No	Running user
`Group`	`string`	No	Running group
`ExecStartPre`	`[]string`	No	Command list to execute before startup
`Environment`	`[]string`	No	Environment variable list (format: KEY=VALUE)
`EnvironmentFile`	`[]string`	No	Environment variable file path list
`CustomExtra`	`map[string]string`	No	Custom extra configuration items (key-value pairs)

kubelet.service supports configuration items for kubelet service startup.

kubelet Dynamic Value Passing Feature

The dynamic value passing mechanism allows the use of variables in configuration files, which are replaced with actual values during deployment. This provides great flexibility, allowing the same configuration to be applicable to different nodes and environments.

Enabling Variable Substitution

During the BKECluster deployment process, the system automatically detects KubeletConfigRef and enables variable substitution. Variable substitution takes effect in the following two places:

kubelet.conf configuration
kubelet.service systemd service file

Expression variable: ${EXPR|command|END}

Execute shell commands and use the output as variable values.

Syntax:

${EXPR|shell_command|END}

Example:

yaml

spec:  kubeletConfig:    
  kubelet.conf:      
    raw: |        
        apiVersion: kubelet.config.k8s.io/v1beta1
        kind: KubeletConfiguration
        # Get CPU core count
        cpuManagerPolicy: "static"
        reservedSystemCPUs: "${EXPR|nproc|END}"
        # Get memory size (MB)
        evictionHard:
          memory.available: "${EXPR|free -m | grep Mem | awk '{print $2}'|END}Mi"
        # Get hostname
        hostnameOverride: "${EXPR|hostname|END}"

Note:
Commands are executed on the target node
If command execution fails, the variable remains unchanged (not replaced)
Command output will have leading and trailing whitespace removed

kubelet Configuration File kct Reference

my-kubelet.yaml

yaml

apiVersion: bke.bocloud.com/v1beta1
kind: KubeletConfig
metadata:
  name: my-kubelet
  namespace: my-kubelet
  labels:
    test: "true"
    component: "kubelet"
spec:

  kubeletConfig:
    kubelet.conf:
      raw: |
        apiVersion: kubelet.config.k8s.io/v1beta1
        kind: KubeletConfiguration
        authentication:
          anonymous:
            enabled: false
          webhook:
            cacheTTL: 2m0s
            enabled: true
          x509:
            clientCAFile: /etc/kubernetes/pki/ca.crt
        authorization:
          mode: Webhook
          webhook:
            cacheAuthorizedTTL: 5m0s
            cacheUnauthorizedTTL: 30s
        address: "${EXPR|ip -4 addr show | awk '/inet / {split($2,a,"/"); print a[1]}' | grep -vE '^127\.0\.0\.1|^169\.254\.' | head -n1|END}"
        port: 10250
        readOnlyPort: 0
        cgroupDriver: systemd
        hairpinMode: promiscuous-bridge
        serializeImagePulls: false
        clusterDNS:
          - "10.96.0.10"
        clusterDomain: "cluster.local"
        healthzBindAddress: 127.0.0.1
        healthzPort: 10248
        rotateCertificates: true
        staticPodPath: /etc/kubernetes/manifests
        streamingConnectionIdleTimeout: 4h0m0s
        syncFrequency: 1m0s
        volumeStatsAggPeriod: 1m0s

  kubeletService:
    enabled: true
    serviceName: "kubelet"
    unit:
      description: "Kubernetes Kubelet (my-Test)"
      documentation: "https://kubernetes.io/docs/concepts/overview/components/#kubelet"
      after:
        - "containerd.service"
        - "network-online.target"
      wants:
        - "network-online.target"
      requires:
        - "containerd.service"
    service:
      workingDirectory: "/var/lib/kubelet"
      execStartPre:
        - "/usr/bin/mkdir -p /sys/fs/cgroup/cpuacct/system.slice/kubelet.service"
        - "/usr/bin/mkdir -p /sys/fs/cgroup/hugetlb/system.slice/kubelet.service"
      execStart: "/usr/bin/kubelet \
      --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf \
      --kubeconfig=/etc/kubernetes/admin.conf \
      --config=/var/lib/kubelet/config.yaml \
      --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
      --pod-infra-container-image=registry.k8s.io/pause:3.9 \
      --cgroup-driver=systemd \
      --register-node=true \
      --v=2 \
      --node-ip=${EXPR|ip -4 addr show scope global | awk '/inet /{print $2}' | cut -d'/' -f1|END} \
      --hostname-override=${EXPR|hostname -i | awk '{print $1}'|END}
      "
      restart: "always"
      restartSec: 10
      startLimitInterval: 30
      startLimitBurst: 3
      killMode: "process"
      standardOutput: "syslog"
      standardError: "syslog"
      syslogIdentifier: "kubelet"
      user: "root"
      group: "root"

    install:
      wantedBy:
        - "multi-user.target"
      requiredBy: []

    variables:
      kubeletBinaryPath: "/usr/bin/kubelet"
      kubeletConfigPath: "/etc/kubernetes/kubelet.conf"
      nodeName: "test-node-01"
      nodeIP: "xx"
      containerdEndpoint: "unix:///run/containerd/containerd.sock"
      pauseImage: "registry.aliyuncs.com/google_containers/pause:3.9"

ContainerdConfig CR Configuration

Overview

The ContainerdConfig CR allows you to dynamically configure Containerd's main configuration file, service file, and image registry configuration through Kubernetes resources.

CR Structure

yaml

apiVersion: bke.bocloud.com/v1beta1
kind: ContainerdConfig
metadata:
  name: <config-name>
  namespace: <namespace>
spec:
  script:
    # Script configuration
  service:
    # Containerd systemd service configuration
  main:
    # Containerd main configuration file (config.toml)
  registry:
    # Image registry configuration

spec Field Description

spec.script

Used to execute custom scripts before configuring Containerd.

Parameter	Type	Required	Description
`Content`	`string`	No	Script content (inline script)
`Path`	`string`	No	Script file path (local file)
`Interpreter`	`string`	No	Interpreter (e.g., /bin/bash, /bin/sh), default is /bin/bash
`Args`	`[]string`	No	List of arguments passed to the script

Note:
Content and Path are mutually exclusive. If both are provided, Content takes precedence.

spec.service

Used to configure Containerd's systemd service drop-in file (/etc/systemd/system/containerd.service.d/10-override.conf).

Parameter	Type	Required	Description
`ExecStart`	`string`	No	Service startup command
`Slice`	`string`	No	systemd slice unit
`KillMode`	`string`	No	Termination mode
`Restart`	`string`	No	Restart policy
`RestartSec`	`string`	No	Restart interval
`TimeoutStopSec`	`string`	No	Stop timeout
`StartLimitInterval`	`string`	No	Start limit time window
`StartLimitBurst`	`int`	No	Start limit count
`Logging`	`LoggingConfig`	No	Logging configuration
`CustomExtra`	`map[string]string`	No	Custom extra configuration items

LoggingConfig Fields:

Parameter	Type	Required	Description
`StandardOutput`	`string`	No	Standard output redirect
`StandardError`	`string`	No	Standard error redirect
`SyslogIdentifier`	`string`	No	Syslog identifier
`LogLevelMax`	`string`	No	Maximum log level

spec.main

Used to configure Containerd's main configuration file (/etc/containerd/config.toml).

Parameter	Type	Required	Description
`SandboxImage`	`string`	No	Pod sandbox image (e.g., pause image)
`Root`	`string`	No	Containerd data root directory, default is `/var/lib/containerd`
`State`	`string`	No	Containerd state directory, default is `/run/containerd`
`ConfigPath`	`string`	No	Configuration file path, default is `/etc/containerd/certs.d`
`MetricsAddress`	`string`	No	Metrics monitoring address

Note:
If the main field is specified, the system will generate the config.toml file based on the template. If not specified, the default configuration is used.

spec.registry

Used to configure the image registry's hosts.toml file.

Parameter	Type	Required	Description
`ConfigPath`	`string`	No	hosts.toml configuration directory, default is `/etc/containerd/certs.d`
`Configs`	`map[string]RegistryHostConfig`	Yes	Registry configuration mapping, key is registry name

RegistryHostConfig Fields:

Parameter	Type	Required	Description
`Host`	`string`	Yes	Registry host address
`Capabilities`	`[]string`	No	Supported capabilities (e.g., pull, push, resolve)
`SkipVerify`	`bool`	No	Skip TLS verification
`PlainHTTP`	`bool`	No	Use HTTP instead of HTTPS
`Insecure`	`bool`	No	Insecure connection
`OverridePath`	`bool`	No	Override path
`TLS`	`TLSConfig`	No	TLS configuration
`Auth`	`RegistryAuthConfig`	No	Authentication configuration
`Header`	`map[string][]string`	No	Custom HTTP headers

TLSConfig Fields:

Parameter	Type	Required	Description
`CAFile`	`string`	No	CA certificate file path
`CertFile`	`string`	No	Client certificate file path
`KeyFile`	`string`	No	Client private key file path
`InsecureSkipVerify`	`bool`	No	Skip TLS verification

RegistryAuthConfig Fields:

Parameter	Type	Required	Description
`Auth`	`string`	No	Base64 encoded authentication string
`Username`	`string`	No	Username
`Password`	`string`	No	Password
`IdentityToken`	`string`	No	Identity token
`RegistryToken`	`string`	No	Registry token

Binary File Custom Configuration Guide ​

Background ​

Prerequisites ​

Restrictions ​

Guide ​

KubeletConfig CR Configuration ​

Overview ​

CR Structure ​

ContainerdConfig CR Configuration ​

Overview ​

CR Structure ​

Binary File Custom Configuration Guide

Background

Prerequisites

Restrictions

Guide

KubeletConfig CR Configuration

Overview

CR Structure

ContainerdConfig CR Configuration

Overview

CR Structure