Kunpeng Confidential Container

Feature Introduction

Kunpeng Confidential Container is built on the Kunpeng TEE technology, using the complete software stack of k8s+containerd+Kata+QEMU+KVM+CoCo to implement confidential container deployment. Kunpeng Confidential Container has been customized and adapted on the basis of the open-source Kata/CoCo community, featuring security capabilities such as remote attestation, image signing and encryption, and confidential container device passthrough.

Application Scenarios

When a business requires high confidentiality and needs to be deployed in a secure container, this feature provides strong isolation similar to traditional virtual machines, preventing security issues between different containers.

Capability Scope

Provides fast compilation and build capabilities for Kunpeng Confidential Container related components.
Provides an Operator customized and optimized for Kunpeng Confidential Container to enable fast installation and deployment of the Kunpeng Confidential Container environment in Kubernetes clusters managed by openFuyao.
Establishes Kunpeng Confidential Container feature test methods.

Key Features

The official Kata/CoCo community only supports confidential containers on Intel, AMD, IBM, and other hardware platforms, but does not support Kunpeng Confidential Container. The openFuyao platform adds a fast deployment solution for the Kunpeng Confidential Container environment.

Installation

Prerequisites

Kunpeng Confidential Container Component Build

The build.sh script compiles and builds Kunpeng Confidential Container related components based on the TEE technology provided by Kunpeng virtCCA.

Compiles and builds the kata-deploy image for the confidential container runtime from source code, including components such as shim-v2, qemu, and kernel.
Compiles trustee components from source code, building AS, KBS, and RVPS images.

Environment Preparation

The build environment uses an ARM operating system with a kernel version of v5.0 or above.

Prerequisites

The build environment requires git, make, openssl, and docker to be installed.
Docker must be able to pull images. You can test this by pulling the following image.
shell
```
docker pull ubuntu:22.04
```

Start Building

Use the root user for compilation and building.

shell

git clone https://gitcode.com/openFuyao/confidential-containers-deployment.git
cd confidential-containers-deployment/build/kunpeng
chmod 755 ./build.sh

Run the following command to prepare the source code.
shell
```
./build.sh prepare_src
```
Run the following command to compile all kata-deploy components and build the kata-deploy image.
shell
```
./build.sh compile_kdeploy_all
```

The final build artifact is the kata-deploy image kata-deploy_3.15.0.tar, located in: ./kata-containers/tools/packaging/kata-deploy

Run the following command, enter a component name, and perform single-point compilation of kata-deploy components.

shell

./build.sh compile_kdeploy_components [component-name]

# Available component names:
kernel                       - make kernel-tarball
kernel-virtcca-confidential  - make kernel-virtcca-confidential-tarball
nydus                        - make nydus-tarball
shim-v2                      - make shim-v2-tarball
virtiofsd                    - make virtiofsd-tarball
qemu                         - make qemu-tarball
qemu-virtcca-experimental    - make qemu-virtcca-experimental-tarball
agent                        - make agent-tarball
pause-image                  - make pause-image-tarball
coco-guest-components        - make coco-guest-components-tarball
rootfs-image                 - make rootfs-image-tarball
rootfs-image-experimental    - make rootfs-image-experimental-tarball
merge-builds                 - make merge-builds to combine all tarballs

When a component name is specified, the build artifact is the component package kata-static-[component-name].tar.xz, located in: ./kata-containers/tools/packaging/kata-deploy/local-build/build

When merge-builds is specified, all required components are checked and the kata-deploy image kata-deploy_3.15.0.tar is built, located in: ./kata-containers/tools/packaging/kata-deploy

Run the following command to compile the remote attestation component images.
shell
```
./build.sh compile_coco
```

The build artifacts are rvps-grpc_v0.12.0.tar, kbs-grpc-as_v0.12.0.tar, and as-grpc_v0.12.0.tar, located in: ./kata-containers/build/trustee/output

Pre-deployment Preparation

Install a K8s cluster. Because this depends on the Remote Snapshotter feature of containerd, the container runtime must be containerd with version no lower than v1.7.
The following container images are required for deployment. Ensure that the cluster nodes can pull them successfully.
- Operator Controller image: quay.io/confidential-containers/operator:v0.13.0
- Pre-install Payload image: quay.io/confidential-containers/reqs-payload:latest
The kata-deploy image must be imported into the deployment environment. For the build process, refer to the Kunpeng Confidential Container Component Build section.
shell
```
# Example: importing into the containerd image repository
ctr -n k8s.io i import kata-deploy_3.15.0.tar
```

Start Installation

Deploy cc-Operator

Run the following command to deploy cc-operator-controller-manager.

shell

kubectl apply -k github.com/confidential-containers/operator/config/release?ref=v0.13.0

Run the following command to deploy CcRuntime.

shell

# Get the CcRuntime deployment yaml for virtCCA
wget https://raw.gitcode.com/openeuler/virtCCA_sdk/raw/master/kata-v3.15.0/conf/virtcca-kata-deploy.yaml
# The payloadImage field specifies the kata-deploy image path, e.g., docker.io/library/kata-deploy:3.15.0. Modify it according to your imported image repository.
kubectl apply -f virtcca-kata-deploy.yaml

After successful creation, the confidential-containers-system namespace contains one cc-operator-controller and two daemonset-managed Pods.

shell

# kubectl get pod -n confidential-containers-system -owide
NAMESPACE                           NAME
confidential-containers-system      cc-operator-controller-manager-685bdf4fbb-mllt5
confidential-containers-system      cc-operator-daemon-install-h4v42
confidential-containers-system      cc-operator-pre-install-daemon-rcfdt

The RuntimeClass corresponding to virtCCA is registered in the K8s cluster.

shell

# kubectl get runtimeclass
NAME                  HANDLER
kata                  kata-qemu
kata-qemu             kata-qemu
kata-qemu-coco-dev    kata-qemu-coco-dev
kata-qemu-virtcca     kata-qemu-virtcca

Create Confidential Container

Deploy a test Pod to verify that the Kunpeng Confidential Container runtime is correctly configured and in effect.

Deploy a test Pod

Run a confidential container Pod based on the kata-qemu-virtcca container runtime:

yaml

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: test-kata-qemu-virtcca
  annotations:
    io.containerd.cri.runtime-handler: "kata-qemu-virtcca"
    io.katacontainers.config.hypervisor.kernel_params: "agent.debug_console agent.log=debug"
spec:
  runtimeClassName: kata-qemu-virtcca
  terminationGracePeriodSeconds: 5
  containers:
  - name: box-1
    image: registry.hw.com:5000/busybox:latest
    imagePullPolicy: Always
    command:
    - sh
    tty: true
EOF

Deploy trustee-Operator

Install kustomize Tool

To facilitate customization of the Operator deployment manifests (such as replacing image addresses), install the kustomize tool.

shell

# Download the kustomize compressed package
wget -O "kustomize_v5.4.3_linux_arm64.tar.gz" "https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv5.4.3/kustomize_v5.4.3_linux_arm64.tar.gz"

# Extract to the system executable directory
tar -zxf "kustomize_v5.4.3_linux_arm64.tar.gz" -C /usr/local/bin

Generate Deployment yaml

Generate deploy.yaml and kbsconfig.yaml required for trustee-operator deployment.

shell

# Clone trustee-operator code
git clone https://github.com/confidential-containers/trustee-operator.git -b v0.3.0
cd /workdir/trustee-operator/config/manager
kustomize edit set image controller=quay.io/confidential-containers/trustee-operator:v0.3.0
cd /workdir/trustee-operator && kustomize build "config/default" >/workdir/deploy.yaml
cd /workdir/trustee-operator/config/samples/microservices 
openssl genpkey -algorithm ed25519 >privateKey
openssl pkey -in privateKey -pubout -out kbs.pem
kustomize build . >/workdir/kbsconfig.yaml

Modify the image names of the three remote attestation components in deploy.yaml.

For building the remote attestation component images, refer to the Kunpeng Confidential Container Component Build section.

yaml

...
env:
- name: KBS_IMAGE_NAME
  value: docker.io/library/kbs-grpc-as:v0.12.0
- name: AS_IMAGE_NAME
  value: docker.io/library/as-grpc:v0.12.0
- name: RVPS_IMAGE_NAME
  value: docker.io/library/rvps:v0.12.0
...

Modify the configuration in kbsconfig.yaml.

shell

# Modify configmap: as-config-grpc, update the attestation_token_broker configuration as follows
"attestation_token_broker": {
    "type": "Simple",
    "duration_min": 5
}

# Modify configmap: rvps-config-grpc, update the storage configuration as follows
"storage": {
    "type": "LocalFs",
    "file_path": "/opt/confidential-containers/attestation-service/rvps"
}

# Modify configmap: kbs-config-grpc, remove the following configuration
[policy_engine]
policy_path = "/opt/confidential-containers/opa/policy.rego"

Deploy operator

Deploy trustee-operator.

shell

kubectl apply -f /workdir/deploy.yaml
# After successful creation, the following pod is present in the cluster; kubectl get pod -n trustee-operator-system
NAMESPACE                   NAME
trustee-operator-system     trustee-operator-controller-manager-6fcbf48f96-pgwg5

Deploy the remote attestation components.

shell

kubectl apply -f /workdir/kbsconfig.yaml
# After successful creation, the following pod is present in the cluster; kubectl get pod -n trustee-operator-system
NAMESPACE                   NAME
trustee-operator-system     trustee-deployment-694bf879d5-7wzwp

Modify the trustee-deployment deployment file.

shell

# kubectl edit deployment -n trustee-operator-system trustee-deployment

# Add the following environment variable to all 3 containers in spec.template.spec.containers
...
env:
- name: RUST_LOG
  value: DEBUG
  
# Add volume mounts to the kbs container's volumeMounts
- mountPath: /etc/attestation/attestation-service/verifier/virtcca/
  name: verifier
  
# Add volume mounts to the as container's volumeMounts
- mountPath: /etc/attestation/attestation-service/verifier/virtcca/
  name: verifier
- mountPath: /opt/confidential-containers/attestation-service
  name: attestation-service
 
# Add volume mounts to the rvps container's volumeMounts
- mountPath: /etc/attestation/attestation-service/verifier/virtcca/
  name: verifier
- mountPath: /opt/confidential-containers/attestation-service
  name: attestation-service

# Add volumes to the pod
- hostPath:
    path: /opt/confidential-containers/kbs/repository/default
    type: Directory
    name: default
- hostPath:
    path: /etc/attestation/attestation-service/verifier/virtcca
    type: Directory
    name: verifier
- hostPath:
    path: /opt/confidential-containers/attestation-service
    type: Directory
    name: attestation-service

Using Image Encryption and Image Signing

Using the image encryption and image signing features of confidential containers requires the remote attestation service and installation of the image signing tool.

Configure Remote Attestation Service

Procedure

Configure the policy for verifying VirtCCA measurement reports.

shell

mkdir -p /opt/confidential-containers/attestation-service/token/simple/policies/opa
vim /opt/confidential-containers/attestation-service/token/simple/policies/opa/default.rego

# default.rego content:
package policy
import future.keywords.every
import future.keywords.if
default allow := false
allow if {
    print("Full Input:", input)
    print("Rim:", input["virtcca.realm.rim"])
    print("Ref:", data.reference)
    input["virtcca.realm.rim"] in data.reference["virtcca.realm.rim"]
}

Pre-provision the remote attestation root certificate and intermediate CA certificate files on the remote attestation service node.
shell
```
mkdir -p /etc/attestation/attestation-service/verifier/virtcca/ && cd /etc/attestation/attestation-service/verifier/virtcca/
```
Visit the websites to download the certificates and place the following certificate files in the current directory, keeping the downloaded certificate names unchanged.
shell
```
Huawei Equipment Root CA.pem
Huawei IT Product CA.pem
```

RSA root certificate download link: https://download.huawei.com/dl/download.do?actionFlag=download&nid=PKI1000000002&partNo=3001&mid=SUP_PKI
RSA intermediate CA certificate download link: https://download.huawei.com/dl/download.do?actionFlag=download&nid=PKI1000000040&partNo=3001&mid=SUP_PKI
ECC root certificate download link: https://download.huawei.com/dl/download.do?actionFlag=download&nid=PKI1100000224&partNo=3001&mid=SUP_PKI
ECC intermediate CA certificate download link: https://download.huawei.com/dl/download.do?actionFlag=download&nid=PKI1100000225&partNo=3001&mid=SUP_PKI

Add RIM baseline values using the rvps tool.

shell

cd /opt/confidential-containers/attestation-service
cat << EOF > sample
{
    "virtcca.realm.rim": [
        "0f7733fcbaa9059d4d579fab25743868a2b4027290b09dfbc59964fc4b642kkk"
    ]
}
EOF
provenance=$(cat sample | base64 --wrap=0)
cat << EOF > message
{
    "version" : "0.1.0",
    "type": "sample",
    "payload": "$provenance"
}
EOF

# Register the baseline values with the rvps service
kubectl exec -it -n trustee-operator-system trustee-deployment-694bf879d5-7wzwp -c rvps -- bash
./rvps-tool register --path /opt/confidential-containers/attestation-service/message --addr http://127.0.0.1:50003

Note:
The RIM baseline values (list) added to virtcca.realm.rim can be generated using the baseline value generation tool.

Install Basic Tools

Procedure

Install golang.

shell

wget https://golang.google.cn/dl/go1.22.4.linux-arm64.tar.gz
tar -zxvf go1.22.4.linux-arm64.tar.gz -C /usr/local
echo "export GO_HOME=/usr/local/go" >> /etc/profile
echo "export GOPATH=/home/work/go" >> /etc/profile
source /etc/profile
echo "export PATH=${GO_HOME}/bin:${PATH}" >> /etc/profile
source /etc/profile

Install the cosign image signing tool.

shell

wget https://github.com/sigstore/cosign/releases/download/v2.5.0/cosign-linux-arm64
chmod +x cosign-linux-arm64
mv cosign-linux-arm64 /usr/local/bin/cosign

Compile and install skopeo.

shell

git clone -b v1.15.1 https://github.com/containers/skopeo.git $GOPATH/src/github.com/containers/skopeo
yum install -y gpgme-devel device-mapper-devel
cd $GOPATH/src/github.com/containers/skopeo && make bin/skopeo
cp bin/skopeo /usr/local/bin

# Test pulling an image locally; the image should be pulled successfully
mkdir -p /home/work/images && cd /home/work/images
skopeo copy --insecure-policy   
docker://docker.io/library/busybox:latest dir:busybox:latest

Container Image Encryption/Decryption

Official reference documentation: https://github.com/confidential-containers/guest-components/tree/main/attestation-agent/coco_keyprovider https://confidentialcontainers.org/docs/features/encrypted-images/

Prerequisites

Completed Configure Remote Attestation Service
Completed Install Basic Tools

Procedure

Generate an encryption key and encrypt the container image.

shell

# Generate an encryption key
KEY_FILE="image_key"
head -c 32 /dev/urandom | openssl enc > "$KEY_FILE"

# Create the encrypted image output directory
mkdir output

# Create a container and use the generated key to encrypt the plaintext image inside the container (-s specifies the original image to be encrypted)
docker run -v "$PWD/output:/output" \\
-e http_proxy="http://IP:PORT" \\
-e https_proxy="http://IP:PORT" \\
ghcr.io/confidential-containers/staged-images/coco-keyprovider:latest /encrypt.sh \\
-k "$(base64 < image_key)" \\
-i kbs:///default/image_key/busybox_enc \\
-s docker://busybox:latest \\
-d dir:/output

# Push the encrypted image to the local image repository
skopeo copy dir:output docker://registry.hw.com:5000/busybox_enc:latest

# Use the following command to verify whether the image was encrypted successfully
skopeo inspect docker://registry.hw.com:5000/busybox_enc:latest

# Copy the encryption key to the specified path
cp ./image_key /opt/confidential-containers/kbs/repository/default/image_key/busybox_enc

Start the encrypted image container.

shell

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: enc-test
  annotations:
    io.containerd.cri.runtime-handler: "kata-qemu-virtcca"
    io.katacontainers.config.hypervisor.kernel_params: "agent.debug_console agent.log=debug agent.image_policy_file=kbs:///default/security-policy/test agent.enable_signature_verification=true agent.guest_components_rest_api=all agent.aa_kbc_params=cc_kbc::http:[attestation-service-pod-IP]:8080"
spec:
  runtimeClassName: kata-qemu-virtcca
  terminationGracePeriodSeconds: 5
  containers:
  - name: box
    image: registry.hw.com:5000/busybox_enc:latest
    imagePullPolicy: Always
    command:
    - sh
    tty: true
EOF

Fill in the agent.aa_kbc_params parameter in pod.metadata.annotations with the Pod IP address of the remote attestation service. Example:

shell

# View the remote attestation service pod: kubectl get pod -n trustee-operator-system -owide
NAMESPACE                  NAME                                    READY   STATUS    IP
trustee-operator-system    trustee-deployment-694bf879d5-7wzwp     3/3     Running   10.244.0.231

# In pod annotations: agent.aa_kbc_params=cc_kbc::http:10.244.0.231:8080"

Container Image Signing

CoCo community reference documentation: https://confidentialcontainers.org/docs/features/signed-images/

Prerequisites

Completed Configure Remote Attestation Service
Completed Install Basic Tools

Procedure

Sign the image.

shell

# Generate a key pair (press Enter twice to skip setting a password, depending on your requirements)
cosign generate-key-pair

# --tlog-upload=false means do not upload to the official cosign server (i.e., offline signing, depending on your requirements)
cosign sign --key cosign.key --tlog-upload=false registry.hw.com:5000/busybox:latest

# Deploy the signing public key
mkdir -p /opt/confidential-containers/kbs/repository/default/cosign-key
cp ./cosign.pub /opt/confidential-containers/kbs/repository/default/cosign-key/1

Specify the image repository verification policy.

shell

vim /opt/confidential-containers/kbs/repository/default/security-policy/test

json

{
    "default": [
        {
            "type": "reject"
        }
    ],
    "transports": {
        "docker": {
            "registry.hw.com:5000": [
                {
                    "type": "sigstoreSigned",
                    "keyPath": "kbs:///default/cosign-key/1"
                }
            ]
        }
    }
}

Note:
Set the type field to specify the verification requirement: "reject" rejects any image, "sigstoreSigned" requires signature verification, and "insecureAcceptAnything" performs no verification.

Start the container with signature verification.

Signing and verification demo:

yaml

apiVersion: v1
kind: Pod
metadata:
  name: sign-test
  annotations:
    io.containerd.cri.runtime-handler: "kata-qemu-virtcca"
    io.katacontainers.config.hypervisor.kernel_params: "agent.debug_console agent.log=debug agent.image_policy_file=kbs:///default/security-policy/test agent.enable_signature_verification=true agent.guest_components_rest_api=all agent.aa_kbc_params=cc_kbc::http:[attestation-service-pod-IP]:8080"
spec:
  runtimeClassName: kata-qemu-virtcca
  terminationGracePeriodSeconds: 5
  containers:
  - name: box
    image: registry.hw.com:5000/busybox:latest
    imagePullPolicy: Always
    command:
      - sh
    tty: true

Fill in the agent.aa_kbc_params parameter in pod.metadata.annotations with the Pod IP address of the remote attestation service. Example:

shell

# View the remote attestation service pod: kubectl get pod -n trustee-operator-system -owide
NAMESPACE                  NAME                                    READY   STATUS    IP
trustee-operator-system    trustee-deployment-694bf879d5-7wzwp     3/3     Running   10.244.0.231

# In pod annotations: agent.aa_kbc_params=cc_kbc::http:10.244.0.231:8080"

View source on GitCode

Kunpeng Confidential Container ​

Feature Introduction ​

Application Scenarios ​

Capability Scope ​

Key Features ​

Installation ​

Prerequisites ​

Kunpeng Confidential Container Component Build ​

Environment Preparation ​

Prerequisites ​

Start Building ​

Pre-deployment Preparation ​

Start Installation ​

Deploy cc-Operator ​

Create Confidential Container ​

Deploy trustee-Operator ​

Install kustomize Tool ​

Generate Deployment yaml ​

Deploy operator ​

Using Image Encryption and Image Signing ​

Configure Remote Attestation Service ​

Procedure ​

Install Basic Tools ​

Procedure ​

Container Image Encryption/Decryption ​

Prerequisites ​

Procedure ​

Container Image Signing ​

Prerequisites ​

Procedure ​

Kunpeng Confidential Container

Feature Introduction

Application Scenarios

Capability Scope

Key Features

Installation

Prerequisites

Kunpeng Confidential Container Component Build

Environment Preparation

Prerequisites

Start Building

Pre-deployment Preparation

Start Installation

Deploy cc-Operator

Create Confidential Container

Deploy trustee-Operator

Install kustomize Tool

Generate Deployment yaml

Deploy operator

Using Image Encryption and Image Signing

Configure Remote Attestation Service

Procedure

Install Basic Tools

Procedure

Container Image Encryption/Decryption

Prerequisites

Procedure

Container Image Signing

Prerequisites

Procedure