CPU Acceleration During Container Startup

Feature Overview

The service startup phase consumes more CPU resources than the stable state. This feature ensures that the startup speed is increased while the resource usage in the stable state is reduced, meeting service requirements.

Application Scenarios

• After this feature is enabled and resize.on.starting is correctly configured for startup, a Pod is started based on ResizeLimit during startup. After the Pod is started, limit reverts to the value of the stable state configuration.

• After this feature is enabled and resize.on.starting is correctly configured for startup, if a container is abnormally killed after startup, the container is restarted based on ResizeLimit. After the container is started, limit reverts to the value of the stable state configuration.

• After this feature is enabled and resize.on.starting is correctly configured for startup, if a Pod is abnormally killed after startup, its containers are started based on ResizeLimit when the Pod is restarted. After the containers are started, limit reverts to the value of the stable state configuration.

Supported Capabilities

The CPU limit can be adjusted during startup to meet the startup acceleration requirement. The memory limit cannot be adjusted during startup.

Highlights

Java and other programs consume many resources during startup. After the startup is complete, the resource usage decreases. If the resources configured for startup are small, the startup takes a long time. This feature ensures a high startup speed while reducing the steady-state resource usage.

Implementation Principles

Design Principles

• By default, the community-native capabilities are retained, and new functions are disabled by default.
• The changes to the native code are minimized.
• The basic information about the request access is recorded at the minimum storage cost.

New Parameter

Parameter	Default Value	Description
resize.on.starting	An empty string	CPU limit when the service is started

Process Sorting

Figure 1 CPU acceleration process during container startup

Implementation

1. Check whether the configuration value is valid. The value must be greater than the stable state limit. If the value is invalid, the CPU value is not dynamically adjusted and a log is recorded.
2. In the SyncPod method for processing Pods, the resource size of the non-ready containers in a non-terminated Pod is patched based on the configuration in the annotation, and then the patch is delivered to the runtime Pod for startup.
3. After the startup is successful, the Pod status is updated based on the probe result. If the probe is normal, the container status is updated to ready. In this case, the native logic syncCh restores the CPU limit to its steady state.
4. Repeat the preceding operations each time a Pod is re-created or restarted.

Relationship with Related Features

InPlacePodVerticalScaling: The startup adjustment capability depends on the InPlacePodVerticalScaling feature. Only the CPU upper limit can be adjusted at the startup phase.

Creating a Pod and Checking CPU Acceleration During Pod Startup

Prerequisites

• This feature is disabled by default and is enabled or disabled based on the product requirements.

• The startup adjustment capability depends on the InPlacePodVerticalScaling feature. Only the CPU upper limit can be adjusted at the startup phase.

• The containers must be ready when the readiness probe result is successful. The probe result is the only basis for determining whether to adjust the resources to the stable state.

• The QoS level of a Pod cannot be guaranteed.

Context

You need to understand how to enable the InPlacePodVerticalScaling feature of the kube-apiserver and kubelet components.

Constraints

• The ResizeLimit value cannot be less than the CPU limit in the stable state during startup. If the ResizeLimit value is less than the CPU limit in the stable state, the configuration does not take effect, and logs are recorded and displayed.
• If the feature is enabled during the upgrade, containers will be restarted, causing service interruption. Currently, the feature cannot be enabled during the upgrade.
• Due to insufficient resources, a Pod may not be accelerated even if the CPU limit is adjusted after the Pod is scheduled.

Procedure

1. To use the startup adjustment capability, enable the InPlacePodVerticalScaling feature gate of the kubelet and kube-apiserver components. For details, see the following links:

   kube-apiserver | Kubernetes

   Feature Gates | Kubernetes

   Example:

   Add InPlacePodVerticalScaling=true to feature-gates in the kube-apiserver configuration file and restart kube-apiserver.

   Add InPlacePodVerticalScaling: true to featureGate in the kubelet configuration file and restart kubelet.

2. When creating a workload (such as a Deployment or StatefulSet), configure a steady-state CPU request and limit in the resource field, and configure the CPU value required for starting each container in the annotations of the Pod metadata.

   resize.on.starting: '[{"name":"container1", "cpu":"400m"},{"name":"container2", "cpu":"600m"}]'

   The following is an example of correct Deployment configuration:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: my-pod
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: my-pod
     template:
       metadata:
         labels:
           app: my-pod
         annotations:
           resize.on.starting: '[{"name":"my-container", "cpu":"4"}]'        # Enable the resize feature. Multiple containers can be configured.
       spec:
         restartPolicy: Always
         containers:
           - name: my-container
             image: registry.k8s.io/e2e-test-images/nginx:1.15-2
             resources:
               limits:
                 cpu: 300m
             ports:
               - containerPort: 80
             readinessProbe:
               httpGet:
                 path: /
                 port: 80
               initialDelaySeconds: 1
               periodSeconds: 10
               timeoutSeconds: 1
               failureThreshold: 3

3. Self-check: Run the crictl inspect containerID |grep cpuQuota command to check the CPU usage during container creation and after the container is in the stable state. The CPU usage during container startup is the size specified by resize.on.starting, and the CPU usage after the container is in the stable state is the size specified by cpuLimit.