kube-log-runner Log Rotation Reliability Enhancement

Feature Overview

The native kube-log-runner function of the Kubernetes community is simple and provides only log redirection. This feature extends the native mechanism to provide byte-level log rotation and service continuity when the disk space is full.

Application Scenarios

• kube-log-runner: enables the custom parameter --fy-ignore-nospace-error=true.

  If the disk space is full and logs fail to be written, kube-log-runner ignores the error, discards the logs without service interruptions, and records the fault occurrence time.

  If the disk space is released, kube-log-runner writes logs successfully, the fault is rectified, and the fault occurrence time and recovery time are recorded in logs.

• kube-log-runner enables the custom parameter --fy-rotate-enable=true and sets a series of log rotation parameters.

  When the size of a log file reaches the threshold, log rotation is enabled, and the log file that exceeds the threshold is removed for dump and compression.

• kube-log-runner enables the custom parameters --fy-rotate-enable=true and --fy-check-log-deleted-period=5 and checks whether log files exist every 5 seconds.

  When a log file is deleted, the system detects that the log file does not exist, creates a log file, and records the fault occurrence time in the new log file.

Supported Capabilities

• When the disk space is full, logs are discarded without service interruptions, and the fault occurrence time is recorded. When the fault is rectified, the fault rectification time is recorded in the logs.

• When the log size threshold is reached, the log file that exceeds the threshold is removed for dump and compression.

• When a log file is deleted, a new log file is created and the fault occurrence time is recorded in the new log file.

Highlights

• Services are not interrupted in the fault scenario. After the fault is rectified, the fault occurrence time is recorded.

• If the size of a log is too large, the log is compressed for storage to reduce the storage pressure.

• Logs are automatically restored when they are deleted by mistake.

Implementation Principles

A log file is wrapped as an extensible writer to implement the write method with subsequent processing. The rotation is implemented by an independent coroutine to ensure that logs are continuously written without being blocked.

The related code segment is staging/src/k8s.io/component-base/fylogs/file_wrap.go:33.

func (f *fileWrap) Write(p []byte) (int, error) {
 f.writtenBytes, f.writtenErr = f.target.Write(p)
 if f.writtenErr == nil {
  for _, f := range f.normalFunc {
   f()
  }
 } else {
  for _, f := range f.onErrFunc {
   f()
  }
  f.writtenBytes = len(p)
 }
 return f.writtenBytes, f.writtenErr
}

This feature ensures that only one rotation coroutine is running at a time when the log write rate is extremely high.

func (r *rotator) startRotateLog() {
 r.rotateLock.Lock()
 r.rotateCounting++ // Number of rotation requests
 r.rotateLock.Unlock()

 // Ensure only one rotation at a time
 if r.rotateCounting != 1 { // The rotation is started.
  return
 }

 var err error
 defer r.afterRotateLog(err)
 err = r.rotateLogs()
 if err != nil {
  r.tryPrintErr(err)
 }
}

Relationship with Related Features

None

Installation

No installation is required. You can enable the kube-log-runner functions by configuring parameters when starting kube-log-runner.

Using kube-log-runner

Constraints

kube-log-runner redirects and rotates only stdout and stderr generated by subprocesses. External injection of logs to log files does not trigger log rotation.

Procedure

Enable or adjust the log rotation functions of kube-log-runner by configuring startup parameters.

Table 1 Parameters related to kube-log-runner

Parameter	Default Value	Description	Configuration Example
fy-file-permission	644	Log file permission.	--fy-file-permission=750
fy-compress-permission	440	Compressed file permission.	--fy-compress-permission=500
fy-ignore-nospace-error	false	Whether to discard logs when the disk space is full without service interruptions.	--fy-ignore-nospace-error=true
fy-rotate-enable	false	Whether to enable log rotation.	--fy-rotate-enable=true

The log rotation parameters when the log rotation is enabled (--fy-rotate-enable=true) are listed in the following table.

Table 2 Performance parameters related to kube-log-runner log rotation

Rotation Parameter	Default Value	Description	Configuration Example
fy-each-file-size	20 (MB)	Maximum size of a single log file. The value ranges from 5 MB to 100 MB.	--fy-each-file-size=100
fy-total-size-limit	150 (MB)	Upper limit of the total size of log files and dump files. The value 0 indicates that there is no upper limit.	--fy-total-size-limit=100
fy-total-count-limit	10	Upper limit of the total number of log files and dump files. The value 0 indicates that there is no upper limit.	--fy-total-count-limit=0
fy-compress-enable	true	Whether to enable dump file compression.	--fy-compress-enable=true
fy-check-log-deleted-period	0 (not check)	Period for checking whether log files are deleted. The value 0 indicates that the check is not performed.	--fy-check-log-deleted-period=5

Related Operations

None