As world enters the era of artificial intelligence, the demand of parallel computing power(which is largely based on GPU) is growing in an unprecedented manner.

However, powerful GPU is usually expensive and can not be equipped on each device. Thus, current industry trend is to move GPU computational power from client’s machine to data center and cloud.

Famous cloud providers such as AWS and GCP already enable GPU instance in their virtual machine renting services. NVIDIA also develops cloud services for GPU-specific workload such as gaming and deep learning.

GPU cloud architecture

The core value of cloud is resource sharing and renting. We can assign hardware resources based on needs, which in turn optimizes efficiency and minimizes cost.

A virtual machine is the essential building block to divide resources. GPU cloud is no different. Unlike other hardwares like CPU, memory and storage, extra effort is needed to bring GPU into VM. The rest of the article will explain in detail how GPU resource is virtualized.

Following is the architecture of a typical GPU cloud:

The diagram depicts structure of one specific zone. Admin host manages the entire data center. GPU host provides GPU-powered VM instances to customers.

Linux virtualization stack

Linux’s virtualization stack contains several layers:

client

This layer consists of users of virtualization. Examples are command-line tools, GUI application, and various cloud service providers.

libvirt

This is the API provided by Red Hat to unify management of different virtualization back-ends.

QEMU

QEMU is a hardware emulator, it provides a set of different emulated hardware and device models for the virtual machine.

KVM runs in kernel space and contains only core virtualization logic such as vCPU and memory mapping. Thus, QEMU is a good complement for providing peripheral support.

When the guest OS operates emulated hardware, QEMU translates the instruction and gets it executed on host.

KVM

KVM is the virtualization module shipped with Linux kernel. Since released, it gains more and more popularity.

Large cloud service providers such as AWS are actively migrating from existing virtualization solution like Xen towards KVM.

Overview

KVM itself is a Linux kernel module. KVM and its interaction with guest and host is as follows:

Project Structure

KVM source code is located in Linux main branch at /virt/kvm. Most of the logic is inside kvm_main.c file.

Process of creating a VM

Creating a VM is one of the most typical workflows of KVM. Let’s walk through its process:

1. Open KVM device

KVM itself is represented as a device in Linux. In order for it to operate VM, it has to be opened first:

kvm_fd=open("/dev/kvm")

2. Create VM

After KVM device is opened, we use ioctl to send command to the device asking for creating a VM:

vm_fd = ioctl(kvm_fd, KVM_CREATE_VM, 0)

3. Create virtual CPU for the VM

The core component of a VM is CPU and memory. a vCPU is represented as a file as well and created using ioctl.

vcpu_fd = ioctl(vm_fd, KVM_CREATE_VCPU, 0)

4. Set up memory for the VM

The memory of VM is created in user space and then mapped to VM.

void *mem = mmap(0, mem_size, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS, -1, 0);
struct kvm_userspace_memory_region region = {.userspace_addr = mem, .memory_size = mem_size};
ioctl(vm_fd, KVM_SET_USER_MEMORY_REGION, &region);

5. Run VM

After everything is set up, we run vCPU inside VM and wait for it to exit:

while(true) {
    ioctl(vcpu_fd, KVM_RUN, 0);
}

Tips

KVM API is mainly used by guest framework such as QEMU. Client usually controls behaviors of VM through QEMU command line parameters and seldom calls KVM API directly.

GPU resource sharing

Introduction

The core value of cloud is sharing resources. For CPU/memory/network, this is relatively easy to implement, we can assign any fraction of resource to a VM.

For GPU, things are different. Most consumer grade GPUs don’t support dividing GPU resources. This means GPU can only be exclusively assigned to one VM as a whole. We can add many GPUs to PCIe slots and assign one GPU per VM, but it will be obviously annoying to manage and maintain.

vGPU

Nvidia’s vGPU technology is used for resolving this problem. Workstation GPUs such as Quadro and Tesla support this feature.

mediated device framework(mdev)

This is the mechanism for implementing vGPU. In general, it is compatible with VFIO UAPI, but managed by vendor driver which controls resource sharing logic.

Multiple virtual devices can be generated by Mdev, each contains a fraction of GPU resource. The generated virtual device is located inside /sys/bus/mdev/devices/ and ready to be attached to VM.

GPU sharing for container

Containerization gains more and more popularity nowadays, and it is natural to come up with the demand to share GPU among containers.

As of 2020, it remains an open question: Is sharing GPU to multiple containers feasible?

The root cause is, there is no way of partitioning GPU resources (CUDA cores, memory), or even assigning priorities for most non-datacenter GPUs. So before NVIDIA provides hardware support for resource sharing, we can do nothing about it🙃

GPU Passthrough: access GPU from VM

GPU is viewed as a PCIe device. It is normally attached to and accessed by host driver. In order for it to be used by VM, A special driver called vfio driver is needed to control GPU from inside VM.

how to enable GPU passthrough

bind vfio driver to GPU

Adding GRUB parameters is a convenient way to bind vfio driver:

GRUB_CMDLINE_LINUX_DEFAULT="vfio-pci.ids=10de:2484"

Attach GPU to VM

This can be done via command line tool or GUI application. The GPU is added to VM as a PCIe device.

The client calls libvirt library and libvirt adds relevant parameter indicating attached device to start qemu process. It is like:

$ ps -ef|grep qemu
libvirt+ 1351222       1 99 16:14 ?        00:02:21 /usr/bin/qemu-system-x86_64 ... -device vfio-pci,host=01:00.0

vfio driver

Background

In the past, specific device assignment code has to be written for different PCI devices.

In order to unify the device access process of VM, VFIO driver frameworks is developed.

Mechanism

In general, vfio is a re-mapper of device address. When guest accesses device, vfio is responsible for remapping the address to real, physical device address. Thus, the host is bypassed and guest can access physical PCIe device directly.

VFIO operates devices in the granularity of IOMMU group. IOMMU is used for connecting physical device’s I/O bus to main memory.

An IOMMU group combines multiple devices(which are usually logically related) into the same set. A typical example is NVIDIA’s Video and Audio device.

You can use lspci command to inspect IOMMU info of a pci device:

$ lspci -vvv
08:00.0 VGA compatible controller: NVIDIA Corporation Device 2484 (rev a1) (prog-if 00 [VGA controller])
    Subsystem: NVIDIA Corporation Device 146b
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0
    Interrupt: pin A routed to IRQ 79
    IOMMU group: 13
    Region 0: Memory at f5000000 (32-bit, non-prefetchable) [size=16M]
    Region 1: Memory at e0000000 (64-bit, prefetchable) [size=256M]
    Region 3: Memory at f0000000 (64-bit, prefetchable) [size=32M]
    Region 5: I/O ports at d000 [size=128]
    Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
    Capabilities: <access denied>
    Kernel driver in use: nvidia
    Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

Source code analysis

VFIO is a PCIe driver, so its source code is located at /drivers/vfio/pci. Main logic is located at vfio_pci.c.

VFIO device is viewed just like a file which we can open/release/read/write. The operations are like:

//source: https://github.com/torvalds/linux/blob/139711f033f636cc78b6aaf7363252241b9698ef/drivers/vfio/pci/vfio_pci.c#L1884
static const struct vfio_device_ops vfio_pci_ops = {
    .name		= "vfio-pci",
    .open		= vfio_pci_open,
    .release	= vfio_pci_release,
    .ioctl		= vfio_pci_ioctl,
    .read		= vfio_pci_read,
    .write		= vfio_pci_write,
    .mmap		= vfio_pci_mmap,
    .request	= vfio_pci_request,
    .match		= vfio_pci_match,
};

VFIO read/write shares the same backend called vfio_pci_rw. Its code is like:

//source: https://github.com/torvalds/linux/blob/139711f033f636cc78b6aaf7363252241b9698ef/drivers/vfio/pci/vfio_pci.c#L1407
static ssize_t vfio_pci_rw(void *device_data, char __user *buf,
               size_t count, loff_t *ppos, bool iswrite) 
{
    unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
    struct vfio_pci_device *vdev = device_data;
    switch (index) {
    case VFIO_PCI_CONFIG_REGION_INDEX:
        return vfio_pci_config_rw(vdev, buf, count, ppos, iswrite);

    case VFIO_PCI_BAR0_REGION_INDEX ... VFIO_PCI_BAR5_REGION_INDEX:
        return vfio_pci_bar_rw(vdev, buf, count, ppos, iswrite);
    default:
        index -= VFIO_PCI_NUM_REGIONS;
        return vdev->region[index].ops->rw(vdev, buf,
                           count, ppos, iswrite);
    }
    return -EINVAL;
}

We can see that different operations apply to different region indexes.

For pci_bar_rw, it generally contains 2 steps, first setup IO mapping, then do actual read/write.

//source: https://github.com/torvalds/linux/blob/master/drivers/vfio/pci/vfio_pci_rdwr.c#L227
ssize_t vfio_pci_bar_rw(struct vfio_pci_device *vdev, char __user *buf,
            size_t count, loff_t *ppos, bool iswrite)
{
    ...
    int ret = vfio_pci_setup_barmap(vdev, bar);
    ...
    do_io_rw(vdev, res->flags & IORESOURCE_MEM, io, buf, pos,
            count, x_start, x_end, iswrite);
    ...
}

The data is transferred between user space and PCIe device, with VFIO driver as the medium:

//source: https://github.com/torvalds/linux/blob/f6e1ea19649216156576aeafa784e3b4cee45549/drivers/vfio/pci/vfio_pci_rdwr.c#L97
static ssize_t do_io_rw(struct vfio_pci_device *vdev, bool test_mem,
            void __iomem *io, char __user *buf,
            loff_t off, size_t count, size_t x_start,
            size_t x_end, bool iswrite)
{
    ssize_t done = 0;
    int ret;
    while (count) {
        size_t fillable, filled;
        ...
        if (fillable >= 4 && !(off % 4)) {
            u32 val;
            if (iswrite) {
                if (copy_from_user(&val, buf, 4))
                    return -EFAULT;
                ret = vfio_pci_iowrite32(vdev, test_mem,
                             val, io + off);
            } else {
                ret = vfio_pci_ioread32(vdev, test_mem,
                            &val, io + off);
                if (copy_to_user(buf, &val, 4))
                    return -EFAULT;
            }
            filled = 4;
        }
        ...
        count -= filled;
        done += filled;
        off += filled;
        buf += filled;
    }

    return done;
}

References

1. https://en.wikipedia.org/wiki/Kernel-based_Virtual_Machine
2. https://www.linux-kvm.org/images/5/59/02x03-Neo_Jia_and_Kirti_Wankhede-vGPU_on_KVM-A_VFIO_based_Framework.pdf
3. https://www.kernel.org/doc/Documentation/vfio.txt
4. https://david942j.blogspot.com/2018/10/note-learning-kvm-implement-your-own.html

开车离家后，一路向西。旧金山湾中间的平地被一圈山岭围绕，所以要到海边必须走上好长一段山路。路两侧壮观的红杉林虽美，但弯弯绕绕的盘山公路的驾驶体验实在不是很好，都要晕车了。

湾区附近的海滩中，以半月湾(Half Moon Bay)最为热门。想要避开人群，因此特地选了附近的另一个San Gregorio海滩。原以为隆冬时节没人会来海边， 来到目的地后，发现还是有稀稀疏疏的游人，点缀在金色的沙滩上。

加州海岸多为浪蚀而成。海浪常年冲击侵蚀海岸，因此在岸边形成了陡峭的悬崖。San Gregorio海滩边也有一面崖壁。顶部平坦，形成了一座天然的观景台，可以观赏海滩全景:

全景图拍得比较渣，可能是手机握的不稳的缘故，用三脚架固定可能效果更好些。不过这光影的震颤变幻以及支离破碎的贴图，是不是有点印象派和立体主义的风格？哈哈，希望莫奈、毕加索不要跳起来打我。

到海边时刚好赶上涨潮，隆隆的潮声从浩渺的太平洋上传来，蔚为雄壮。沙滩上聚集着成群结队的海鸥，和旅人们一样，多面朝大海，静立在沙滩上，思绪似乎随潮声飘向远方。

原本最大的愿望就是拍摄海上日落，结果到了之后才发现乌云完全遮住了西方天空，甚是失望。日落后，沿着蜿蜒海岸线的加州1号公路回家, 结果在路上，绚烂的火烧云显现在天空中。

如果日落时在海滩多呆半小时，就可以完整拍到海边晚霞之景，也算不虚此行了。可见世上诸事，行百里者半九十。在这个纷繁变幻的年代，恒心才是最宝贵的东西。

Felt bored and drove out without destination last weekend, I luckily arrived at the summit of Windy Hill, a hill in western Silicon Valley that has very good viewshed of the entire bay area.

The whole San Francisco Bay is surrounded by hills. The climate of eastern half is dry, seldom rains. The symbol color of California is golden. Everybody will agree on this when the distant, golden-hued and vegetation-lacking hills come into sight after leaving airport and driving on the highway.

The western half, where Windy Hill is located, is quite different. It faces Pacific Ocean directly. Moisture continuously flows from coast, thus the climate is humid enough for California redwood to survive.

California redwood is one of the tallest species on Earth, reaching up to 100 meters, shading most sunlight from reaching ground.

In addition, the environment is always foggy, mainly due to cool coastal air and mountainous terrain. All these factors depict a serene, even gloomy scene inside redwood forest.

Above is when i drove through a redwood forest of Windy Hill. Although living just next to Redwood City, this is the first time I’m able to see the forest of California redwood.

Kubernetes is largely a resource manager for cluster. Among all hardware resources, only CPU and memory are natively supported.

Since there are lots of hardware vendors, it is unrealistic to add each hardware’s specific code to k8s main project.

Thus, a device plugin framework is developed to provide support for different devices. Hardware vendors need to implement “driver” for their devices to run on k8s cluster.

Use of device plugin

A device plugin is deployed as a DaemonSet on each node. It monitors device on each node and collaborates with kubelet to run a container with device enabled.

A sample yaml file for NVIDIA GPU device plugin is as follows:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
spec:
    template:
        metadata:
            labels:
                - name: device-plugin
        spec:
            containers:
                name: device-plugin-ctr
                image: NVIDIA/device-plugin:1.0
                volumeMounts:
                  - mountPath: /device-plugin
                  - name: device-plugin
            volumes:
             - name: device-plugin
               hostPath:
                   path: /var/lib/kubelet/device-plugins

After device plugin is deployed, a pod can request device from cluster, just as other types of resources. Sample YAML file is as follows:

apiVersion: v1
kind: Pod
metadata:
  name: demo-pod
spec:
  containers:
    - name: demo-container-1
      image: k8s.gcr.io/pause:2.0
      resources:
        limits:
          nvidia.com/gpu: 2

Device plugin framework

Device plugin framework is implemented in kubelet to provide support of managing different device plugins. It does following work:

1. Register device plugin.
2. Allocate device to container.

plugin registration

A device plugin has to be registered to kubelet first in order for it to be used.

The registration communication is done via grpc. Plugin framework serves as a grpc server and device plugin is the client.

The registration protobuf message is as follows:

//source: https://github.com/kubernetes/kubernetes/blob/296f7c91bb52cd724ce6d6d120d5d41ed459d677/staging/src/k8s.io/kubelet/pkg/apis/deviceplugin/v1beta1/api.proto#L23
service Registration {
	rpc Register(RegisterRequest) returns (Empty) {}
}

message RegisterRequest {
	// Version of the API the Device Plugin was built against
	string version = 1;
	// Name of the unix socket the device plugin is listening on
	// PATH = path.Join(DevicePluginPath, endpoint)
	string endpoint = 2;
	// Schedulable resource name. As of now it's expected to be a DNS Label
	string resource_name = 3;
	// Options to be communicated with Device Manager
	DevicePluginOptions options = 4;
}

From RegisterRequest message we can see that a version, a grpc endpoint, a resource name and some options are needed to represent a device plugin.

Register method is implemented in plugin framework:

//source: https://github.com/kubernetes/kubernetes/blob/4eadf404480e0653e29a9367841080d94ea4017c/pkg/kubelet/cm/devicemanager/manager.go#L312
func (m *ManagerImpl) RegisterPlugin(pluginName string, endpoint string, versions []string) error {
	klog.V(2).Infof("Registering Plugin %s at endpoint %s", pluginName, endpoint)

	e, err := newEndpointImpl(endpoint, pluginName, m.callback)
	if err != nil {
		return fmt.Errorf("failed to dial device plugin with socketPath %s: %v", endpoint, err)
	}

	options, err := e.client.GetDevicePluginOptions(context.Background(), &pluginapi.Empty{})
	if err != nil {
		return fmt.Errorf("failed to get device plugin options: %v", err)
	}

	m.registerEndpoint(pluginName, options, e)
	go m.runEndpoint(pluginName, e)

	return nil
}

In general, an endpoint is registered and run for the plugin. The term endpoint represents a single registered device plugin. Its definition is as follows:

//source: https://github.com/kubernetes/kubernetes/blob/abf87c99c63984ba426239e0aed657bf9a8a9054/pkg/kubelet/cm/devicemanager/endpoint.go#L35
type endpoint interface {
	run()
	stop()
	allocate(devs []string) (*pluginapi.AllocateResponse, error)
    callback(resourceName string, devices []pluginapi.Device)
    ...
}

type endpointImpl struct {
	client     pluginapi.DevicePluginClient
	clientConn *grpc.ClientConn

	socketPath   string
	resourceName string

	mutex sync.Mutex
	cb    monitorCallback
}

It contains necessary methods/fields for plugin framework to communicate with plugin.

device discovery

After plugin is registered, kubelet will start watching for device changes. The logic is as follows:

//source: https://github.com/kubernetes/kubernetes/blob/abf87c99c63984ba426239e0aed657bf9a8a9054/pkg/kubelet/cm/devicemanager/endpoint.go#L96
func (e *endpointImpl) run() {
	stream, err := e.client.ListAndWatch(context.Background(), &pluginapi.Empty{})
	for {
		response, err := stream.Recv()
		devs := response.Devices
		var newDevs []pluginapi.Device
		for _, d := range devs {
			newDevs = append(newDevs, *d)
		}

		e.callback(e.resourceName, newDevs)
	}
}

Device is watched through ListAndWatch grpc call. Whenever there is device changes, it is received by kubelet and a callback is called to record device info.

device allocation

The key usage of a device plugin is to have device allocated to a container.

It is done through endpoint::allocate function:

// allocate issues Allocate gRPC call to the device plugin.
func (e *endpointImpl) allocate(devs []string) (*pluginapi.AllocateResponse, error) {
	return e.client.Allocate(context.Background(), &pluginapi.AllocateRequest{
		ContainerRequests: []*pluginapi.ContainerAllocateRequest{
			{DevicesIDs: devs},
		},
	})
}

The function sends grpc request with ContainerAllocateRequest and return with ContainerAllocateResponse. They are both protobuf message, definition is as follows:

// source: https://github.com/kubernetes/kubernetes/blob/296f7c91bb52cd724ce6d6d120d5d41ed459d677/staging/src/k8s.io/kubelet/pkg/apis/deviceplugin/v1beta1/api.proto#L162
message ContainerAllocateRequest {
	repeated string devicesIDs = 1;
}
message ContainerAllocateResponse {
  	// List of environment variable to be set in the container to access one of more devices.
	map<string, string> envs = 1;
	// Mounts for the container.
	repeated Mount mounts = 2;
	// Devices for the container.
	repeated DeviceSpec devices = 3;
	// Container annotations to pass to the container runtime
	map<string, string> annotations = 4;
}

The request message is simple: if container want to use some device, it just sends device’s ID to the plugin.

The response message indicates how to use this device in container.

DeviceSpec specifies core device attribute including path/permission/etc.

mounts is needed for device driver/library to be mounted into container.

Also, environment variable/annotations help access of device in container as well.

nvidia device plugin

Introduction

NVIDIA’s k8s device plugin is crucial for bringing GPU workload to Kubernetes. It serves similar purpose as nvidia-docker.

what nvidia-device-plugin does

In general, nvidia-device-plugin is a GPU device manager for Kubernetes cluster. It will:

1. respond to grpc requests from plugin-framework;
2. monitor all gpus on node;
3. return device for allocation;

Project Architecture

server.go

nvidia-device-plugin is basically a grpc server. server.go implements all rpc functions and server related logic.

nvidia.go

The operations of GPU are implemented in this file which are used by server.go.

gpu-monitoring-tools

This project provides golang bindings of lower level management libraries which is used by nvidia-device-plugin project.

NVML(NVIDIA Management Library)

This library provides C-API for monitoring/management of NVIDIA GPU devices. It is used by nvidia-smi and other libraries including gpu-monitoring-tools.

ListAndWatch

This function monitors GPU devices on node.

//source: https://gitlab.com/nvidia/kubernetes/device-plugin/blob/4167bfd7fdfdbec6a5378af3589650714cf2ab3f/server.go#L218
func (m *NvidiaDevicePlugin) ListAndWatch(e *pluginapi.Empty, s pluginapi.DevicePlugin_ListAndWatchServer) error {
    s.Send(&pluginapi.ListAndWatchResponse{Devices: m.apiDevices()})
	for {
		select {
		case <-m.stop:
			return nil
		case d := <-m.health:
			d.Health = pluginapi.Unhealthy
			log.Printf("'%s' device marked unhealthy: %s", m.resourceName, d.ID)
			s.Send(&pluginapi.ListAndWatchResponse{Devices: m.apiDevices()})
		}
	}
}

The logic is clear. The function keeps checking state of devices. Whenever there is an update of health, a notification is sent to kubelet through grpc.

The health check is as follows:

//source: https://gitlab.com/nvidia/kubernetes/device-plugin/blob/4167bfd7fdfdbec6a5378af3589650714cf2ab3f/nvidia.go#L159
func checkHealth(stop <-chan interface{}, devices []*Device, unhealthy chan<- *Device) {
	...
	for _, d := range devices {
		gpu, _, _, err := nvml.ParseMigDeviceUUID(d.ID)
		if err != nil {
			gpu = d.ID
		}
		err = nvml.RegisterEventForDevice(eventSet, nvml.XidCriticalError, gpu)
		if err != nil && strings.HasSuffix(err.Error(), "Not Supported") {
			log.Printf("Warning: %s is too old to support healthchecking: %s. Marking it unhealthy.", d.ID, err)
			unhealthy <- d
			continue
		}
	}

	for {
		e, err := nvml.WaitForEvent(eventSet, 5000)
		if err != nil && e.Etype != nvml.XidCriticalError {
			continue
		}
		...
		for _, d := range devices {
			// Please see https://github.com/NVIDIA/gpu-monitoring-tools/blob/148415f505c96052cb3b7fdf443b34ac853139ec/bindings/go/nvml/nvml.h#L1424
			// for the rationale why gi and ci can be set as such when the UUID is a full GPU UUID and not a MIG device UUID.
			gpu, gi, ci, err := nvml.ParseMigDeviceUUID(d.ID)
			if err != nil {
				gpu = d.ID
				gi = 0xFFFFFFFF
				ci = 0xFFFFFFFF
			}

			if gpu == *e.UUID && gi == *e.GpuInstanceId && ci == *e.ComputeInstanceId {
				log.Printf("XidCriticalError: Xid=%d on Device=%s, the device will go unhealthy.", e.Edata, d.ID)
				unhealthy <- d
			}
		}
	}
}

The GPU event is registered and monitored through NVML’s binding function. When there is an unhealthy-GPU event, it is reported back to ListAndWatch through unhealthy channel.

Allocate

Let’s see what info is sent back to kubelet in order for container to use GPU device:

//source: https://gitlab.com/nvidia/kubernetes/device-plugin/blob/4167bfd7fdfdbec6a5378af3589650714cf2ab3f/server.go#L265
func (m *NvidiaDevicePlugin) Allocate(ctx context.Context, reqs *pluginapi.AllocateRequest) (*pluginapi.AllocateResponse, error) {
	responses := pluginapi.AllocateResponse{}
	for _, req := range reqs.ContainerRequests {
		for _, id := range req.DevicesIDs {
			if !m.deviceExists(id) {
				return nil, fmt.Errorf("invalid allocation request for '%s': unknown device: %s", m.resourceName, id)
			}
		}
		response := pluginapi.ContainerAllocateResponse{}
		if *deviceListStrategyFlag == DeviceListStrategyVolumeMounts {
			response.Envs = m.apiEnvs(m.deviceListEnvvar, []string{deviceListAsVolumeMountsContainerPathRoot})
			response.Mounts = m.apiMounts(req.DevicesIDs)
		}
		if *passDeviceSpecs {
			response.Devices = m.apiDeviceSpecs(req.DevicesIDs)
		}
		responses.ContainerResponses = append(responses.ContainerResponses, &response)
	}
	return &responses, nil
}

As code indicates, DeviceSpec, Mount and Envs consist of response.

Here is what DeviceSpec looks like:

//source: https://gitlab.com/nvidia/kubernetes/device-plugin/blob/4167bfd7fdfdbec6a5378af3589650714cf2ab3f/server.go#L351
func (m *NvidiaDevicePlugin) apiDeviceSpecs(filter []string) []*pluginapi.DeviceSpec {
	var specs []*pluginapi.DeviceSpec

	paths := []string{
		"/dev/nvidiactl",
		"/dev/nvidia-uvm",
		"/dev/nvidia-uvm-tools",
		"/dev/nvidia-modeset",
	}

	for _, p := range paths {
		if _, err := os.Stat(p); err == nil {
			spec := &pluginapi.DeviceSpec{
				ContainerPath: p,
				HostPath:      p,
				Permissions:   "rw",
			}
			specs = append(specs, spec)
		}
	}
    ...
	return specs
}

We can see that all NVIDIA related device paths have been attached to response. Kubelet will mount all these paths to enable GPU inside container.

Kubelet is a node agent running on each Kubernetes node. It is basically a pod manager controlling pods running on the node.

PodSpec yaml file is provided by client and received through API server, then kubelet will update pod on scheduled node.

Kubelet acts as a gRPC client communicating with container runtime, instruct runtime to do the actual container operation.

Following is the structure of kubelet:

Project Architecture

kubelet is one of the core components of Kubernetes. Its source code sits directly inside main project at pkg/kubelet folder.

Following are important sub-folders of kubelet project:

server

Kubelet communicates with control plane through http call. Thus kubelet itself is an http server.

config

The main input of kubelet server is “config” of Pod. config folder contains Object-Oriented abstraction of different types of config.

pod/images/network/volumemanager

These folders contain management code of different aspects of a Pod. They determine how a Pod is run.

cm

One of the main goal of Kubernetes is resource management. Kubelet should be able to allocate proper resources specified by PodSpec.

In kubelet project this is implemented inside cm folder, aka container manager. Resources like cpu, memory and device will be managed by ContainerManager.

container/kuberuntime

These two folders together serve as the interface of container runtime.

In order to operate a container, kubelet calls functions inside container folder, which then calls functions inside kuberuntime.

kuberuntime folder contains grpc services that communicate with underlying container runtime such as Docker’s containerd.

prober/stats/metrics/logs

kubelet provides various ways to increase the observability of managed pods. These folders contain code which collects info that is later fetched by control plane.

Process of creating a Pod

One of the typical use cases of Kubernetes is to start a Pod. The steps are:

1. User uses kubectl to communicate with API server asking creation of a resource;
2. Scheduler schedules Pod to a proper node;
3. Pod is started on this node by kubelet;

Kubelet is responsible for step 3 of pod creation. Let’s walk through the code to see how it is done.

start kubelet

//source: https://github.com/kubernetes/kubernetes/blob/75242fce7aa8a8f9e703b8602587900ca5aaf937/cmd/kubelet/app/server.go#L1178
func startKubelet(k kubelet.Bootstrap, podCfg *config.PodConfig, kubeCfg *kubeletconfiginternal.KubeletConfiguration, kubeDeps *kubelet.Dependencies, enableCAdvisorJSONEndpoints, enableServer bool) {
	// start the kubelet
	go k.Run(podCfg.Updates())

	// start the kubelet server
	if enableServer {
		go k.ListenAndServe(net.ParseIP(kubeCfg.Address), uint(kubeCfg.Port), kubeDeps.TLSOptions, kubeDeps.Auth,
			enableCAdvisorJSONEndpoints, kubeCfg.EnableDebuggingHandlers, kubeCfg.EnableContentionProfiling, kubeCfg.EnableSystemLogHandler)

	}
	...
}

The main kubelet logic is clear, there are two things to do:

1. Start kubelet server which listens to incoming instructions;
2. Start a goroutine which handles instructions of pod;

Following is the pod instruction handling logic:

//source: https://github.com/kubernetes/kubernetes/blob/75242fce7aa8a8f9e703b8602587900ca5aaf937/pkg/kubelet/kubelet.go#L1306
func (kl *Kubelet) Run(updates <-chan kubetypes.PodUpdate) {
	if kl.logServer == nil {
		kl.logServer = http.StripPrefix("/logs/", http.FileServer(http.Dir("/var/log/")))
	}
	if err := kl.initializeModules(); err != nil {
		kl.recorder.Eventf(kl.nodeRef, v1.EventTypeWarning, events.KubeletSetupFailed, err.Error())
		klog.Fatal(err)
	}

	// Start volume manager
	go kl.volumeManager.Run(kl.sourcesReady, wait.NeverStop)

	if kl.kubeClient != nil {
		// Start syncing node status immediately, this may set up things the runtime needs to run.
		go wait.Until(kl.syncNodeStatus, kl.nodeStatusUpdateFrequency, wait.NeverStop)
		go kl.fastStatusUpdateOnce()

		// start syncing lease
		go kl.nodeLeaseController.Run(wait.NeverStop)
	}

	// Start a goroutine responsible for killing pods (that are not properly
	// handled by pod workers).
	go wait.Until(kl.podKiller.PerformPodKillingWork, 1*time.Second, wait.NeverStop)

	// Start component sync loops.
	kl.statusManager.Start()
	kl.probeManager.Start()

	// Start syncing RuntimeClasses if enabled.
	if kl.runtimeClassManager != nil {
		kl.runtimeClassManager.Start(wait.NeverStop)
	}

	// Start the pod lifecycle event generator.
	kl.pleg.Start()
	kl.syncLoop(updates, kl)
}

We can see that it consists of all kinds of “managers” which handle logic like podUpdate, volume, status, logs, liveness probe, etc.

PodConfig

The specification of pod is stored in a struct PodConfig:

//source: https://github.com/kubernetes/kubernetes/blob/0ed41c3f1036785c6c86dd35d20412c8387cf382/pkg/kubelet/config/config.go#L56
type PodConfig struct {
	pods *podStorage
	mux  *config.Mux

	// the channel of denormalized changes passed to listeners
	updates chan kubetypes.PodUpdate

	// contains the list of all configured sources
	sourcesLock sync.Mutex
	sources     sets.String
}

It contains a list of pods and a channel updates which listens to update of pods.

The update of pod is represented as follows:

//source: https://github.com/kubernetes/kubernetes/blob/0ed41c3f1036785c6c86dd35d20412c8387cf382/pkg/kubelet/types/pod_update.go#L74
type PodUpdate struct {
	Pods   []*v1.Pod
	Op     PodOperation
	Source string
}

Whenever there is an update of Pod, it is sent through updates channel and received by handler, which performs pod update on the node.

PodConfig synchronizer

Among all handling logic, syncLoop is the main loop processing update of Pods. It syncs from running state to desired state.

//source: https://github.com/kubernetes/kubernetes/blob/8c724d793370605d0c474eb6e4fb74779212ff1d/pkg/kubelet/kubelet.go#L1785
func (kl *Kubelet) syncLoop(updates <-chan kubetypes.PodUpdate, handler SyncHandler) {
	klog.Info("Starting kubelet main sync loop.")
	// The syncTicker wakes up kubelet to checks if there are any pod workers
	// that need to be sync'd. A one-second period is sufficient because the
	// sync interval is defaulted to 10s.
	syncTicker := time.NewTicker(time.Second)
	defer syncTicker.Stop()
	housekeepingTicker := time.NewTicker(housekeepingPeriod)
	defer housekeepingTicker.Stop()
	plegCh := kl.pleg.Watch()
	const (
		base   = 100 * time.Millisecond
		max    = 5 * time.Second
		factor = 2
	)
	duration := base
	for {
		if err := kl.runtimeState.runtimeErrors(); err != nil {
			klog.Errorf("skipping pod synchronization - %v", err)
			// exponential backoff
			time.Sleep(duration)
			duration = time.Duration(math.Min(float64(max), factor*float64(duration)))
			continue
		}
		// reset backoff if we have a success
		duration = base

		kl.syncLoopMonitor.Store(kl.clock.Now())
		if !kl.syncLoopIteration(updates, handler, syncTicker.C, housekeepingTicker.C, plegCh) {
			break
		}
		kl.syncLoopMonitor.Store(kl.clock.Now())
	}
}

Basically it is a non-ending loop listening to config update. It is also woken up periodically to sync to last known desired state.

Based on type of Pod update, the update event is dispatched to appropriate handler:

//source: https://github.com/kubernetes/kubernetes/blob/8c724d793370605d0c474eb6e4fb74779212ff1d/pkg/kubelet/kubelet.go#L1859
func (kl *Kubelet) syncLoopIteration(configCh <-chan kubetypes.PodUpdate, handler SyncHandler,
	syncCh <-chan time.Time, housekeepingCh <-chan time.Time, plegCh <-chan *pleg.PodLifecycleEvent) bool {
	select {
	case u, open := <-configCh:
		// Update from a config source; dispatch it to the right handler
		// callback.
		if !open {
			klog.Errorf("Update channel is closed. Exiting the sync loop.")
			return false
		}

		switch u.Op {
		case kubetypes.ADD:
			klog.V(2).Infof("SyncLoop (ADD, %q): %q", u.Source, format.Pods(u.Pods))
			// After restarting, kubelet will get all existing pods through
			// ADD as if they are new pods. These pods will then go through the
			// admission process and *may* be rejected. This can be resolved
			// once we have checkpointing.
			handler.HandlePodAdditions(u.Pods)
		case kubetypes.UPDATE:
			klog.V(2).Infof("SyncLoop (UPDATE, %q): %q", u.Source, format.PodsWithDeletionTimestamps(u.Pods))
			handler.HandlePodUpdates(u.Pods)
		case kubetypes.REMOVE:
			klog.V(2).Infof("SyncLoop (REMOVE, %q): %q", u.Source, format.Pods(u.Pods))
			handler.HandlePodRemoves(u.Pods)
		case kubetypes.RECONCILE:
			klog.V(4).Infof("SyncLoop (RECONCILE, %q): %q", u.Source, format.Pods(u.Pods))
			handler.HandlePodReconcile(u.Pods)
		case kubetypes.DELETE:
			klog.V(2).Infof("SyncLoop (DELETE, %q): %q", u.Source, format.Pods(u.Pods))
			// DELETE is treated as a UPDATE because of graceful deletion.
			handler.HandlePodUpdates(u.Pods)
		case kubetypes.SET:
			// TODO: Do we want to support this?
			klog.Errorf("Kubelet does not support snapshot update")
		default:
			klog.Errorf("Invalid event type received: %d.", u.Op)
		}
	...
	}
	return true
}

Pod Update Handler

According to the above code, a pod-ADD operation is handled by HandlePodAdditions:

//source: https://github.com/kubernetes/kubernetes/blob/8c724d793370605d0c474eb6e4fb74779212ff1d/pkg/kubelet/kubelet.go#L2010
func (kl *Kubelet) HandlePodAdditions(pods []*v1.Pod) {
	start := kl.clock.Now()
	sort.Sort(sliceutils.PodsByCreationTime(pods))
	for _, pod := range pods {
		existingPods := kl.podManager.GetPods()
		// Always add the pod to the pod manager. Kubelet relies on the pod
		// manager as the source of truth for the desired state. If a pod does
		// not exist in the pod manager, it means that it has been deleted in
		// the apiserver and no action (other than cleanup) is required.
		kl.podManager.AddPod(pod)
		...
		kl.dispatchWork(pod, kubetypes.SyncPodCreate, mirrorPod, start)
		kl.probeManager.AddPod(pod)
	}
}

The pod is first added to podManager indicating the desired state. Then actual pod creation work is dispatched to a pod worker and asynchronously handled.

The podWorkers struct has UpdatePod function which creates a goroutine handling pod-update work.

//source: https://github.com/kubernetes/kubernetes/blob/442a69c3bdf6fe8e525b05887e57d89db1e2f3a5/pkg/kubelet/pod_workers.go#L220
go func() {
	defer runtime.HandleCrash()
	p.managePodLoop(podUpdates)
}()

A syncPodFn function is used to do the actual pod operation. It is configured when kubelet starts.

//source:https://github.com/kubernetes/kubernetes/blob/442a69c3bdf6fe8e525b05887e57d89db1e2f3a5/pkg/kubelet/pod_workers.go#L175
err = p.syncPodFn(syncPodOptions{
	mirrorPod:      update.MirrorPod,
	pod:            update.Pod,
	podStatus:      status,
	killPodOptions: update.KillPodOptions,
	updateType:     update.UpdateType,
})

//source: https://github.com/kubernetes/kubernetes/blob/8c724d793370605d0c474eb6e4fb74779212ff1d/pkg/kubelet/kubelet.go#L1438
func (kl *Kubelet) syncPod(o syncPodOptions) error {
	...
	result := kl.containerRuntime.SyncPod(pod, podStatus, pullSecrets, kl.backOff)
	...
}

//source: https://github.com/kubernetes/kubernetes/blob/e6c67c32e140f88c923499b2a35fb96b34fdfdd2/pkg/kubelet/kuberuntime/kuberuntime_manager.go#L661
func (m *kubeGenericRuntimeManager) SyncPod(pod *v1.Pod, podStatus *kubecontainer.PodStatus, pullSecrets []v1.Secret, backOff *flowcontrol.Backoff) (result kubecontainer.PodSyncResult) {
	...
}

The pod operation is backed by container runtime. The function of starting a container is as follows:

//source: https://github.com/kubernetes/kubernetes/blob/6d001ebb68efd8a499c07b37b9b59158ca6159c8/pkg/kubelet/kuberuntime/kuberuntime_container.go#L134
func (m *kubeGenericRuntimeManager) startContainer(podSandboxID string, podSandboxConfig *runtimeapi.PodSandboxConfig, spec *startSpec, pod *v1.Pod, podStatus *kubecontainer.PodStatus, pullSecrets []v1.Secret, podIP string, podIPs []string) (string, error) {
	container := spec.container

	// Step 1: pull the image.
	imageRef, msg, err := m.imagePuller.EnsureImageExists(pod, container, pullSecrets, podSandboxConfig)
	if err != nil {
		s, _ := grpcstatus.FromError(err)
		m.recordContainerEvent(pod, container, "", v1.EventTypeWarning, events.FailedToCreateContainer, "Error: %v", s.Message())
		return msg, err
	}

	// Step 2: create the container.
	// For a new container, the RestartCount should be 0
	...
	containerID, err := m.runtimeService.CreateContainer(podSandboxID, containerConfig, podSandboxConfig)
	if err != nil {
		s, _ := grpcstatus.FromError(err)
		m.recordContainerEvent(pod, container, containerID, v1.EventTypeWarning, events.FailedToCreateContainer, "Error: %v", s.Message())
		return s.Message(), ErrCreateContainer
	}
	err = m.internalLifecycle.PreStartContainer(pod, container, containerID)

	// Step 3: start the container.
	err = m.runtimeService.StartContainer(containerID)
	if err != nil {
		s, _ := grpcstatus.FromError(err)
		m.recordContainerEvent(pod, container, containerID, v1.EventTypeWarning, events.FailedToStartContainer, "Error: %v", s.Message())
		return s.Message(), kubecontainer.ErrRunContainer
	}
	m.recordContainerEvent(pod, container, containerID, v1.EventTypeNormal, events.StartedContainer, fmt.Sprintf("Started container %s", container.Name))

	// Step 4: execute the post start hook.
	...
}

This is the lowest point of pod creation logic in kubelet. Kubelet would use grpc client to communicate with Container Runtime(e.g Docker containerd) and instruct it to do the actual container operation.

kubectl/kube-apiserver together form the client-server communication model of Kubernetes. As interface of k8s, it is the entry point to dive into the whole system.

kubectl

Kubectl, the command line client of k8s, is usually the first tool people use to get in touch with k8s. In essence it is just an http client requesting different APIs and getting results from k8s API server.

Project Structure

In the good old days when k8s project was relatively small, kubectl source code was directly inside kubernetes main project.

For the purpose of making projects more modularized, kubectl was recently factored out of main k8s project and became a stand-alone project kubernetes/kubectl. kubernetes/pkg/kubectl/ directory now contains only code which forwards logic to kubectl library.

The kubectl project structure is as follows:

staging/src/k8s.io/kubectl/pkg
├── apply
├── apps
├── cmd
├── describe
├── drain
├── explain
├── generate
├── generated
├── metricsutil
├── polymorphichelpers
├── proxy
├── rawhttp
├── scale
├── scheme
├── util
└── validation

We can see that contents of kubectl project are basically implementations of all kinds of kubectl commands.

Process of “kubectl get”

Here is what the most common kubectl operation get looks like:

First, in k8s main project, we call NewCmdGet from kubectl library:

// source: https://github.com/kubernetes/kubernetes/blob/1b32dfdafdcd6cce21415c75385970a9ae5b0f01/pkg/kubectl/cmd/cmd.go#L432
func NewKubectlCommand(in io.Reader, out, err io.Writer) *cobra.Command {
    ...
    groups := templates.CommandGroups{
		{
			Message: "Basic Commands (Intermediate):",
			Commands: []*cobra.Command{
				explain.NewCmdExplain("kubectl", f, ioStreams),
				get.NewCmdGet("kubectl", f, ioStreams),
				edit.NewCmdEdit(f, ioStreams),
				delete.NewCmdDelete(f, ioStreams),
			},
		},
    }
    ...
    templates.ActsAsRootCommand(cmds, filters, groups...)
    ...
    return cmds
}

get is a sub-command of kubectl, so its implementation is similar to NewKubectlCommand:

func NewCmdGet(parent string, f cmdutil.Factory, streams genericclioptions.IOStreams) *cobra.Command {
	o := NewGetOptions(parent, streams)
	cmd := &cobra.Command{
		Use:                   "get [(-o|--output=)json|yaml|wide|custom-columns=...|custom-columns-file=...|go-template=...|go-template-file=...|jsonpath=...|jsonpath-file=...] (TYPE[.VERSION][.GROUP] [NAME | -l label] | TYPE[.VERSION][.GROUP]/NAME ...) [flags]",
		DisableFlagsInUseLine: true,
		Short:                 i18n.T("Display one or many resources"),
		Long:                  getLong + "\n\n" + cmdutil.SuggestAPIResources(parent),
		Example:               getExample,
		Run: func(cmd *cobra.Command, args []string) {
			cmdutil.CheckErr(o.Complete(f, cmd, args))
			cmdutil.CheckErr(o.Validate(cmd))
			cmdutil.CheckErr(o.Run(f, cmd, args))
		},
		SuggestFor: []string{"list", "ps"},
	}
    ...
	return cmd
}

The actual get operation is inside function GetOptions.Run:

//source: https://github.com/kubernetes/kubernetes/blob/b326948a9a317dbc17c6f32dfeea26e090bde3b0/staging/src/k8s.io/kubectl/pkg/cmd/get/get.go#L448
func (o *GetOptions) Run(f cmdutil.Factory, cmd *cobra.Command, args []string) error {
	r := f.NewBuilder().
		Unstructured().
		NamespaceParam(o.Namespace).DefaultNamespace().AllNamespaces(o.AllNamespaces).
		FilenameParam(o.ExplicitNamespace, &o.FilenameOptions).
		LabelSelectorParam(o.LabelSelector).
		FieldSelectorParam(o.FieldSelector).
		RequestChunksOf(chunkSize).
		ResourceTypeOrNameArgs(true, args...).
		ContinueOnError().
		Latest().
		Flatten().
		TransformRequests(o.transformRequests).
        Do()
    ...
    infos, err := r.Infos()
    ...
    printer.PrintObj(info.Object, w)
	...

We can see that builder pattern is used. All command line options correspond to part of build pipeline.

In Do function, Visitor pattern is used. A visitor is responsible for iterating all resources fetched from API server.

//source: https://github.com/kubernetes/kubernetes/blob/c386fb09a7bde5924a07bd271f6dbb5f4e698aa8/staging/src/k8s.io/cli-runtime/pkg/resource/builder.go#L919
func (b *Builder) visitByResource() *Result {
    ...
	// retrieve one client for each resource
	mappings, err := b.resourceTupleMappings()
	if err != nil {
		result.err = err
		return result
	}
	clients := make(map[string]RESTClient)
	for _, mapping := range mappings {
		s := fmt.Sprintf("%s/%s", mapping.GroupVersionKind.GroupVersion().String(), mapping.Resource.Resource)
		if _, ok := clients[s]; ok {
			continue
		}
		client, err := b.getClient(mapping.GroupVersionKind.GroupVersion())
		if err != nil {
			result.err = err
			return result
		}
		clients[s] = client
	}
	items := []Visitor{}
	for _, tuple := range b.resourceTuples {
		mapping, ok := mappings[tuple.Resource]
		if !ok {
			return result.withError(fmt.Errorf("resource %q is not recognized: %v", tuple.Resource, mappings))
		}
		s := fmt.Sprintf("%s/%s", mapping.GroupVersionKind.GroupVersion().String(), mapping.Resource.Resource)
		client, ok := clients[s]
		if !ok {
			return result.withError(fmt.Errorf("could not find a client for resource %q", tuple.Resource))
		}
		selectorNamespace := b.namespace
		info := &Info{
			Client:    client,
			Mapping:   mapping,
			Namespace: selectorNamespace,
			Name:      tuple.Name,
		}
		items = append(items, info)
	}
	...
	result.sources = items
	return result
}

First a RESTClient is retrieved. Then use this client to fetch resource and save/return result.

RESTClient is implemented in client-go library, this library is provided to developers to write customized client on their own.

//source: https://github.com/kubernetes/kubernetes/blob/b1098bd0d53658bfb945e485683d543ab7dc73ba/staging/src/k8s.io/client-go/rest/client.go#L107
func NewRESTClient(baseURL *url.URL, versionedAPIPath string, config ClientContentConfig, rateLimiter flowcontrol.RateLimiter, client *http.Client) (*RESTClient, error) {
	if len(config.ContentType) == 0 {
		config.ContentType = "application/json"
	}

	base := *baseURL
	if !strings.HasSuffix(base.Path, "/") {
		base.Path += "/"
	}
	base.RawQuery = ""
	base.Fragment = ""

	return &RESTClient{
		base:             &base,
		versionedAPIPath: versionedAPIPath,
		content:          config,
		createBackoffMgr: readExpBackoffConfig,
		rateLimiter:      rateLimiter,

		Client: client,
	}, nil
}

In order to build a RESTClient, we need a URL(base + versionedAPIPath) and some configs(content, rateLimiter, etc.) And the client itself is based on Golang’s http.Client library.

Summary

So that’s it. We type some commands, which are converted to REST APIs. Result is fetched and stored in some resource struct, and finally printed to terminal. This is the typical workflow of kubectl.

kube-apiserver

kube-apiserver is the core communication gateway between client/k8s-cluster and components inside k8s cluster.

Overview

Let’s have a quick review of what an http server usually does:

1. open a socket listening to incoming request;
2. route incoming request to a proper handler;
3. handler processes request;
4. read/save result to persistent storage;

Kube-apiserver’s core logic is of no difference.

In addition to the above, kube-apiserver also provides detailed implementation of authN/authZ/admission that regulates incoming request.

In general, kube-apiserver’s logic can be divided into two parts: setup server, and run server. Let’s take a look at them:

Server Setup

Before API server starts to serve requests, it has to be properly setup.

What setup is needed for a server? If we recall behavior of some famous web servers like Nginx, there are 2 types of setup: handler of request, and various server configurations.

Register Handler

The handlers are installed per APIGroup. Following is the function used to install all handlers in API Group:

//source: https://github.com/kubernetes/kubernetes/blob/4362d613f243a02558f03e90b8fcb58b4c6efb06/staging/src/k8s.io/apiserver/pkg/server/genericapiserver.go#L453
func (s *GenericAPIServer) InstallAPIGroups(apiGroupInfos ...*APIGroupInfo) error {
	...
	for _, apiGroupInfo := range apiGroupInfos {
		if err := s.installAPIResources(APIGroupPrefix, apiGroupInfo, openAPIModels); err != nil {
			return fmt.Errorf("unable to install api resources: %v", err)
		}
	...
	}
	return nil
}

After several levels of decomposition, a specific type of API Resource is bound to Storage via a handler. It is done through following function:

//source: https://github.com/kubernetes/kubernetes/blob/c522ee08a3d248ec1097e3673119ffa7a4e1ef7b/staging/src/k8s.io/apiserver/pkg/endpoints/installer.go#L97
func (a *APIInstaller) Install() ([]metav1.APIResource, *restful.WebService, []error) {
	var apiResources []metav1.APIResource
	var errors []error
	ws := a.newWebService()

	// Register the paths in a deterministic (sorted) order to get a deterministic swagger spec.
	paths := make([]string, len(a.group.Storage))
	var i int = 0
	for path := range a.group.Storage {
		paths[i] = path
		i++
	}
	sort.Strings(paths)
	for _, path := range paths {
		apiResource, err := a.registerResourceHandlers(path, a.group.Storage[path], ws)
		if err != nil {
			errors = append(errors, fmt.Errorf("error in registering resource: %s, %v", path, err))
		}
		if apiResource != nil {
			apiResources = append(apiResources, *apiResource)
		}
	}
	return apiResources, ws, errors
}

The actual CRUD logic is implemented inside registerResourceHandlers function:

//source: https://github.com/kubernetes/kubernetes/blob/c522ee08a3d248ec1097e3673119ffa7a4e1ef7b/staging/src/k8s.io/apiserver/pkg/endpoints/installer.go#L185
func (a *APIInstaller) registerResourceHandlers(path string, storage rest.Storage, ws *restful.WebService) (*metav1.APIResource, error) {
	...
	getter, isGetter := storage.(rest.Getter)
	...
	actions := []action{}
	actions = appendIf(actions, action{"LIST", resourcePath, resourceParams, namer, false}, isLister)
	actions = appendIf(actions, action{"POST", resourcePath, resourceParams, namer, false}, isCreater)
	actions = appendIf(actions, action{"GET", itemPath, nameParams, namer, false}, isGetter)
	...
	for _, action := range actions {
		...
		routes := []*restful.RouteBuilder{}
		switch action.Verb {
		case "GET":
			if isGetterWithOptions {
				handler = restfulGetResourceWithOptions(getterWithOptions, reqScope, isSubresource)
			} else {
				handler = restfulGetResource(getter, exporter, reqScope)
			}
			...
			route := ws.GET(action.Path).To(handler).
				Doc(doc).
				Param(ws.QueryParameter("pretty", "If 'true', then the output is pretty printed.")).
				Operation("read"+namespaced+kind+strings.Title(subresource)+operationSuffix).
				Produces(append(storageMeta.ProducesMIMETypes(action.Verb), mediaTypes...)...).
				Returns(http.StatusOK, "OK", producedObject).
				Writes(producedObject)
			...
			routes = append(routes, route)
			...
	}
	...
}

Different CRUD actions register with different handlers. All the handlers form routes which will be routed by server.

Register Filter

A configuration is defined as a filter. The setup of filters is as follows:

//source: https://github.com/kubernetes/kubernetes/blob/d74ab9e1a4929be208d4529fd12b76d3fcd5d546/staging/src/k8s.io/apiserver/pkg/server/config.go#L671
func DefaultBuildHandlerChain(apiHandler http.Handler, c *Config) http.Handler {
	handler := genericapifilters.WithAuthorization(apiHandler, c.Authorization.Authorizer, c.Serializer)
	handler = genericapifilters.WithImpersonation(handler, c.Authorization.Authorizer, c.Serializer)
	handler = genericapifilters.WithAudit(handler, c.AuditBackend, c.AuditPolicyChecker, c.LongRunningFunc)
	failedHandler := genericapifilters.Unauthorized(c.Serializer)
	failedHandler = genericapifilters.WithFailedAuthenticationAudit(failedHandler, c.AuditBackend, c.AuditPolicyChecker)
	handler = genericapifilters.WithAuthentication(handler, c.Authentication.Authenticator, failedHandler, c.Authentication.APIAudiences)
	handler = genericfilters.WithCORS(handler, c.CorsAllowedOriginList, nil, nil, nil, "true")
	handler = genericfilters.WithTimeoutForNonLongRunningRequests(handler, c.LongRunningFunc, c.RequestTimeout)
	handler = genericfilters.WithWaitGroup(handler, c.LongRunningFunc, c.HandlerChainWaitGroup)
	handler = genericapifilters.WithRequestInfo(handler, c.RequestInfoResolver)
	handler = genericapifilters.WithAuditAnnotations(handler, c.AuditBackend, c.AuditPolicyChecker)
	handler = genericapifilters.WithWarningRecorder(handler)
	handler = genericapifilters.WithCacheControl(handler)
	handler = genericapifilters.WithRequestReceivedTimestamp(handler)
	handler = genericfilters.WithPanicRecovery(handler)
	return handler
}

In general, configuration is done by linking various filters together.

Serve

In essence, serve is the action of providing proper response given a request. Let’s see what serve process actually looks like in kube-apiserver:

//source: https://github.com/kubernetes/kubernetes/blob/13b6a929bc945f2bb97dbf7cd7f0fdd02b49bc0f/cmd/kube-apiserver/app/server.go#L161
func Run(completeOptions completedServerRunOptions, stopCh <-chan struct{}) error {
	server, err := CreateServerChain(completeOptions, stopCh)
	if err != nil {
		return err
	}

	prepared, err := server.PrepareRun()
	if err != nil {
		return err
	}

	return prepared.Run(stopCh)
}

The Run function is associated with cobra.Command which forms kube-apiserver binary. It provides the main serve logic.

The process is clear in code. First do necessary configuration(register handler, filter, etc), then some preparation work, finally actually run the server.

The prepared server is run as follows:

//source: https://github.com/kubernetes/kubernetes/blob/4362d613f243a02558f03e90b8fcb58b4c6efb06/staging/src/k8s.io/apiserver/pkg/server/genericapiserver.go#L316
func (s preparedGenericAPIServer) Run(stopCh <-chan struct{}) error {
	...

	// close socket after delayed stopCh
	stoppedCh, err := s.NonBlockingRun(delayedStopCh)
	if err != nil {
		return err
	}

	<-stopCh

	// run shutdown hooks directly. This includes deregistering from the kubernetes endpoint in case of kube-apiserver.
	err = s.RunPreShutdownHooks()
	if err != nil {
		return err
	}
	...
	// Wait for all requests to finish, which are bounded by the RequestTimeout variable.
	s.HandlerChainWaitGroup.Wait()

	return nil
}

We can see that server is run in a non-blocking fashion. Also there are post-start/pre-shutdown hooks which provide extra customizability. There is also code to ensure graceful shutdown of server.

Finally, the core serving logic:

//source: https://github.com/kubernetes/kubernetes/blob/2c3687c255c014f7049eed159de30a82082656b6/staging/src/k8s.io/apiserver/pkg/server/secure_serving.go#L147
func (s *SecureServingInfo) Serve(handler http.Handler, shutdownTimeout time.Duration, stopCh <-chan struct{}) (<-chan struct{}, error) {
	if s.Listener == nil {
		return nil, fmt.Errorf("listener must not be nil")
	}

	tlsConfig, err := s.tlsConfig(stopCh)
	if err != nil {
		return nil, err
	}

	secureServer := &http.Server{
		Addr:           s.Listener.Addr().String(),
		Handler:        handler,
		MaxHeaderBytes: 1 << 20,
		TLSConfig:      tlsConfig,
	}
	...
	// use tlsHandshakeErrorWriter to handle messages of tls handshake error
	tlsErrorWriter := &tlsHandshakeErrorWriter{os.Stderr}
	tlsErrorLogger := log.New(tlsErrorWriter, "", 0)
	secureServer.ErrorLog = tlsErrorLogger

	klog.Infof("Serving securely on %s", secureServer.Addr)
	return RunServer(secureServer, s.Listener, shutdownTimeout, stopCh)
}
//source: https://github.com/kubernetes/kubernetes/blob/2c3687c255c014f7049eed159de30a82082656b6/staging/src/k8s.io/apiserver/pkg/server/secure_serving.go#L207
func RunServer(
	server *http.Server,
	ln net.Listener,
	shutDownTimeout time.Duration,
	stopCh <-chan struct{},
) (<-chan struct{}, error) {
	...
	go func() {
		defer utilruntime.HandleCrash()

		var listener net.Listener
		listener = tcpKeepAliveListener{ln}
		if server.TLSConfig != nil {
			listener = tls.NewListener(listener, server.TLSConfig)
		}

		err := server.Serve(listener)

		msg := fmt.Sprintf("Stopped listening on %s", ln.Addr().String())
		select {
		case <-stopCh:
			klog.Info(msg)
		default:
			panic(fmt.Sprintf("%s due to error: %v", msg, err))
		}
	}()
	return stoppedCh, nil
}

Apart from auxiliary code such as TLS config and graceful-shutdown, we can see that core serving logic is straight forward. Golang’s default http.Server is used to handle incoming request. Whenever there is a new connection, a new go routine is created to serve it. No thread pool, no task queue, just pure concurrency.

Why so simple? Because goroutine hides most concurrency details for us. Unlike an OS thread which is quite primitive and bare-metal, goroutine implements a user space “green-thread” which has many powerful features. Examples are channels for thread communication, goroutine scheduling/multiplex, etc.

Thus, user can write simple/clean concurrency code without trapped in multi-threading messes. That may be the reason why golang is such popular in distributed system and the cornerstone projects like Docker/Kubernetes are all written in go.

Kubernetes, an open-sourced container orchestration platform praised as the future of cloud infrastructure, gains more and more attention nowadays. It extends resource management from a single machine to a cluster of machines, and thus can be seen as the operating system of cloud.

This article series will briefly introduce K8s project stack and mechanism of important components.

Kubernetes project stack

Before deep diving into core logic of Kubernetes, let’s have quick overview of what k8s project stack consists of.

K8S is one of the best open source projects in terms of project organization and support. It sets an outstanding example of how a totally open-sourced project should be grouped, organized and hosted over Internet.

K8s project is located at here.

kubernetes

This is the core project of K8S stack. Will discuss later on.

community

Each engineer engaged in an open-sourced project once felt the pain of reading obscure code alone without any help.

Developers are usually from different places of the world and it is hard for them to sit together in reality and discuss like team/friends. This is the most challenging part for open-source projects.

For an open source project to succeed, not only code, but people need to be organized as well. That’s why k8s provide tremendous support for community.

Special Interest Groups (SIGs)

SIG is a persistent open groups that focus on a part of the project. Example SIGs are sig-node, sig-scheduling, etc.

Each SIG corresponds to a folder in community project, thus you can easily find all the docs/events/etc you want of a SIG.

Working Groups(WG)

WGs mainly focus on addressing issues across SIGs.

Communication

Instant Messaging

Slack is popular among k8s communities nowadays. Just pick a SIG you are interested in and join that channel.

Conference

KubeCon, powered by CNCF, holds regular meetings each year where people share thoughts and innovation about Kubernetes.

enhancements

When developing a brand new feature for a project, it’s usually not very stable and not totally ready for GA yet. Thus we need a “staging” space to put new features temporarily.

After the feature is stable enough, we move the feature to the main repo. Kubernete’s enhancements repo serves as this purpose. It functions much like OpenCV’s opencv_contrib repo.

test-infra

Everybody once submitted a PR to K8S project knows how powerful(even verbose :wink:) the code submission process is. This project contains all tools/scripts to make K8S projects’ CI/CD pipeline as complete and reliable as possible. Each PR triggers several checks/tests and you can interact with k8s-bot on github directly.

minikube

Would like to try K8S but no machines at hand? Minikube is your best friend. Minikube is most suitable for a newbie to try out k8s. It can also serve as a playground to test configuration, deployment, etc.

dashboard

Like any mature projects, k8s provides a dashboard to visualize/operate its internal state as well. It is an convenient out-of-the-box solution, but not very flexible and customizable.

On many k8s-based platforms it is replaced with third-party choices. I personally use Grafana for metrics visualization and Vscode k8s extension for cluster operation/inspection.

website

The k8s homepage https://kubernetes.io/ is open-sourced as well, try submitting a PR for typos when you catch one:sunglasses:.

Kubernetes core project

The kubernetes/kubernetes project contains core functionalities of k8s.

Its file structure is as follows:

build

This folder contains all the build scripts. The main script for building components binaries is build/run.sh. The build process runs in a Docker container to provide consistent build environment.

cmd

This folder contains source code of all the executables(kubectl, kubelet, etc). It uses cobra.Command library to handle logic of creating command-line binary and setup options and configs.

pkg

This folder is where most k8s core logic live. Each components corresponds to a sub-folder. Start here if you would like to deep-dive into one specific k8s component.

staging

As Kubernetes project becomes larger and larger, different components are no more suitable to live in a single project. However, scattering source code everywhere will make it painful to lookup.

K8S project resolves this problem by introducing staging folder. The source code of different components are still checked into kubernetes/kubernetes project at staging folder, but they are later on published to external k8s.io repositories(e.g. k8s.io/kube-scheduler) and referenced by other projects through k8s.io repo.

NVIDIA Docker is a project providing GPU support to Docker container. It is the cornerstone of NVIDIA NGC and all container-based AI platforms.

Why NVIDIA Docker is needed

NVIDIA Docker drastically simplifies deployment of GPU based application. Everybody using Linux knows how painful it is to handle NVIDIA driver stack on Linux(remember the famous dirty words Linus Torvalds threw to NVIDIA? ;-P).

With NVIDIA Docker, we can “pass-through” GPU from host to container, thus eliminate the work needed to manually configure GPU inside container.

Installation

In general, we need to install Docker engine, NVIDIA driver and NVIDIA Docker library on host.

See this link for detailed instructions: Installation guide

Usage

Add --gpus option to DockerCLI command when starting a container. The started container will have access to host’s GPUs. Example:

$docker run --rm --gpus all ubuntu nvidia-smi
#output:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06    Driver Version: 450.51.06    CUDA Version: N/A      |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 1060    On   | 00000000:01:00.0 Off |                  N/A |
| N/A   52C    P8     6W /  N/A |     11MiB /  6078MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

Overview of Docker Stack

Before diving into mechanism of NVIDIA Docker, let’s have a quick review of what is under the hood of Docker:

Architecture

In general there are 2 layers. Upper layer interacts with user, lower layer handles core logic of container.

DockerCLI/Dockerd

These two components, with client-server pattern, together form the user interface of Docker.

DockerCLI is the client user interacts with Docker. Dockerd is a daemon server listening to client requests and forwarding them to containerd.

containerd

Containerd is the engine of Docker Container. It implements every and only logic about container including container lifecycle management, image management, etc.

runc

runc is a CLI tool for spawning and running containers according to the OCI specification. Containerd uses this tool to actually start a new container.

OCI(Open Container Initiative) runtime-spec

In order to run a container, user must provide runtime-spec(a config.json file) to runc describing the process to be run.

Config file example

{
    "ociVersion": "1.0.1",
    "process": {
        "terminal": true,
        "user": {
            "uid": 1,
            "gid": 1,
        },
        "args": [
            "sh"
        ],
        "env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "TERM=xterm"
        ],
        "cwd": "/",
        "capabilities": {
            "bounding": [
                "CAP_AUDIT_WRITE",
                "CAP_KILL",
                "CAP_NET_BIND_SERVICE"
            ],
        },
        "rlimits": [
            {
                "type": "RLIMIT_CORE",
                "hard": 1024,
                "soft": 1024
            },
        ],
    },
    "root": {
        "path": "rootfs",
        "readonly": true
    },
    "mounts": [
        {
            "destination": "/proc",
            "type": "proc",
            "source": "proc"
        },
    ],
    "hooks": {
        "prestart": [
            {
                "path": "/usr/bin/fix-mounts",
                "args": [
                    "fix-mounts",
                    "arg1",
                    "arg2"
                ],
                "env": [
                    "key1=value1"
                ]
            },
            {
                "path": "/usr/bin/setup-network"
            }
        ]
    }
}

From the example, we can see that:

1. Container is nothing but a process. Thus a process field is required.
   1.1 cwd: Working directory of the process;
   1.2 args: Command and arguments to be executed;
   1.3 env: environment variables;
2. We need a root field specifying root filesystem the container process is running in. mount field specifies extra filesystem to be mounted.
3. Hooks: Hooks are actions that can be executed at different stages of container life-cycle. It can be used to customize container’s behavior. NVIDIA Docker uses hooks to inject GPU functionalities into container.

Mechanism of NVIDIA Docker

Architecture

From previous discussion, we can see that the key point is to provide a hook to inject GPU into container.

In general, containerd inside Docker Engine needs to set up hook given the gpu options from DockerCLI. The gpu hooking logic itself is provided by a library called libnvidia-container.

libnvidia-container

This project provides a command line program nvidia-container-cli.

The cli has several commands. list command lists the components required in order to configure a container with GPU support. configure command does the actual GPU set-up inside container.

Following is the example output of list:

$nvidia-container-cli list
#output:
/dev/nvidiactl
/dev/nvidia-uvm
/dev/nvidia-uvm-tools
/dev/nvidia-modeset
/dev/nvidia0
/usr/bin/nvidia-smi
/usr/bin/nvidia-debugdump
/usr/bin/nvidia-persistenced
/usr/bin/nvidia-cuda-mps-control
/usr/bin/nvidia-cuda-mps-server
/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.450.51.06
/usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.450.51.06
/usr/lib/x86_64-linux-gnu/libcuda.so.450.51.06
/usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.450.51.06
/usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.450.51.06
/usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.450.51.06
/usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.450.51.06
/usr/lib/x86_64-linux-gnu/libnvidia-ngx.so.450.51.06
/usr/lib/x86_64-linux-gnu/libnvidia-encode.so.450.51.06
/usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.450.51.06
/usr/lib/x86_64-linux-gnu/libnvcuvid.so.450.51.06
/usr/lib/x86_64-linux-gnu/libnvidia-eglcore.so.450.51.06
/usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.450.51.06
/usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.450.51.06
/usr/lib/x86_64-linux-gnu/libEGL_nvidia.so.450.51.06
/usr/lib/x86_64-linux-gnu/libGLESv2_nvidia.so.450.51.06
/usr/lib/x86_64-linux-gnu/libGLESv1_CM_nvidia.so.450.51.06

We can see that there are different GPU related components such as nvidia-smi, cuda and gpu driver.

In general, what nvidia-container-cli configure does is to mount above components to enable GPU support of container.

Code Analysis

Whoa, coding time! Let’s read the actual source code that implements nvidia-docker.

Process of enabling GPU support of a container

containerd

The --gpus option is parsed when containerd creates a new container:

//source: https://github.com/containerd/containerd/blob/bc4c3813997554d14449b34d336bca2513e84f96/cmd/ctr/commands/run/run_unix.go#L74
func NewContainer(ctx gocontext.Context, client *containerd.Client, context *cli.Context) (containerd.Container, error) {
    ...
    var (
		opts  []oci.SpecOpts
		cOpts []containerd.NewContainerOpts
		spec  containerd.NewContainerOpts
    )
    ...
    if context.IsSet("gpus") {
        opts = append(opts, nvidia.WithGPUs(nvidia.WithDevices(context.Int("gpus")), nvidia.WithAllCapabilities))
    }
    ...
    spec = containerd.WithSpec(&s, opts...)
    ...
    return client.NewContainer(ctx, id, cOpts...)
}

--gpus option corresponds to the function WithGPUs. WithGPUs returns a function that sets the NVIDIA hooks.

//source: https://github.com/containerd/containerd/blob/bc4c3813997554d14449b34d336bca2513e84f96/contrib/nvidia/nvidia.go#L68
const NvidiaCLI = "nvidia-container-cli"
...
func WithGPUs(opts ...Opts) oci.SpecOpts {
	return func(_ context.Context, _ oci.Client, _ *containers.Container, s *specs.Spec) error {
		...
		nvidiaPath, err := exec.LookPath(NvidiaCLI)
		...
		s.Hooks.Prestart = append(s.Hooks.Prestart, specs.Hook{
			Path: c.OCIHookPath,
			Args: append([]string{
				"containerd",
				"oci-hook",
				"--",
				nvidiaPath,
				// ensures the required kernel modules are properly loaded
				"--load-kmods",
			}, c.args()...),
			Env: os.Environ(),
		})
		return nil
	}
}

libnvidia-container

The configuration options is passed to nvidia-container-cli as command line options. It looks like:

//source: https://github.com/NVIDIA/libnvidia-container/blob/e6e1c4860d9694608217737c31fc844ef8b9dfd7/src/cli/configure.c#L18
(const struct argp_option[]){
    {NULL, 0, NULL, 0, "Options:", -1},
    {"pid", 'p', "PID", 0, "Container PID", -1},
    {"device", 'd', "ID", 0, "Device UUID(s) or index(es) to isolate", -1},
    {"require", 'r', "EXPR", 0, "Check container requirements", -1},
    {"ldconfig", 'l', "PATH", 0, "Path to the ldconfig binary", -1},
    {"compute", 'c', NULL, 0, "Enable compute capability", -1},
    {"utility", 'u', NULL, 0, "Enable utility capability", -1},
    {"video", 'v', NULL, 0, "Enable video capability", -1},
    {"graphics", 'g', NULL, 0, "Enable graphics capability", -1},
    {"display", 'D', NULL, 0, "Enable display capability", -1},
    {"ngx", 'n', NULL, 0, "Enable ngx capability", -1},
    {"compat32", 0x80, NULL, 0, "Enable 32bits compatibility", -1},
    {"mig-config", 0x81, "ID", 0, "Enable configuration of MIG devices", -1},
    {"mig-monitor", 0x82, "ID", 0, "Enable monitoring of MIG devices", -1},
    {"no-cgroups", 0x83, NULL, 0, "Don't use cgroup enforcement", -1},
    {"no-devbind", 0x84, NULL, 0, "Don't bind mount devices", -1},
    {0},
}

We can choose what GPU capabilities are needed by container through specifying args.

Then options are configured in configure_command function:

//source: https://github.com/NVIDIA/libnvidia-container/blob/e6e1c4860d9694608217737c31fc844ef8b9dfd7/src/cli/configure.c#L187
int configure_command(const struct context *ctx)
{
    ...
    if (nvc_driver_mount(nvc, cnt, drv) < 0) {
        warnx("mount error: %s", nvc_error(nvc));
        goto fail;
    }
    ...
}

Options are passed to nvc_driver_mount for actual mount operations. Here is the nvc_driver_mount function:

//source: https://github.com/NVIDIA/libnvidia-container/blob/b2fd9616cd544f780b8d63357e747e7e96281743/src/nvc_mount.c#L709
int
nvc_driver_mount(struct nvc_context *ctx, const struct nvc_container *cnt, const struct nvc_driver_info *info)
{
    ...
    /* Procfs mount */
    ...

    /* Application profile mount */
    ...
    /* Host binary and library mounts */
    if (info->bins != NULL && info->nbins > 0) {
        if ((tmp = (const char **)mount_files(&ctx->err, ctx->cfg.root, cnt, cnt->cfg.bins_dir, info->bins, info->nbins)) == NULL)
                goto fail;
        ptr = array_append(ptr, tmp, array_size(tmp));
        free(tmp);
    }
    ...

    /* IPC mounts */
    for (size_t i = 0; i < info->nipcs; ++i) {
        if ((*ptr++ = mount_ipc(&ctx->err, ctx->cfg.root, cnt, info->ipcs[i])) == NULL)
                goto fail;
    }

    /* Device mounts */
    for (size_t i = 0; i < info->ndevs; ++i) {
        if (!(cnt->flags & OPT_NO_DEVBIND)) {
            if ((*ptr++ = mount_device(&ctx->err, ctx->cfg.root, cnt, &info->devs[i])) == NULL)
                    goto fail;
        }
    }
}

mount_* functions are thin wrappers of Linux system call mount.

See? Nothing fancy, we just mount relevant binaries/devices one by one to container’s filesystem. This is the core logic of NVIDIA Docker.

Summary

Here are the key takeaways of this article:

1. NVIDIA Docker provides full GPU support to container by a single --gpus option;
2. NVIDIA Docker serves as a hook of containerd providing customized functionalities(e.g. GPU support) to Docker container.
3. NVIDIA Docker’s core library libnvidia-container is implemented by mounting host OS’s NVIDIA GPU driver components inside Docker Container’s filesystem.

Computer Vision is one of the main applications of current deep-learning-based AI wave. Compared to many AI research fields which are still in lab, CV is already largely deployed in production and used in all kinds of scenarios.

Originally developed by Intel and age 20 years now, OpenCV is a perfect tools for computer vision tasks.

In addition to traditional image processing use-cases such as image smoothing, edge detection, etc., OpenCV can also do Deep Neural Network inference. Thus, we can apply state-of-the-art computer vision NN models to project with the help of OpenCV.

Why choose OpenCV?

I have a hobby real-time object-detection project written in C++. After video frame is obtained, I’m facing the choice of frameworks suitable for object-detection on frame.

After investigation, I found that popular AI frameworks such as Tensorflow, Pytorch are stronger at training models. However, for an AI application, only inference part is important.

Since the project already uses OpenCV for other frame handling work, there is no need to import another AI framework and external project dependency is minimized.

Our friend of this task is DNN module. It provides support for deep learning inference. In the past, OpenCV only supports CPU inference which limits its usage, especially for real-time cases.

Fortunately, a sub-module named cuda4dnn is added recently which provides CUDA support for DNN. This sub-module is backed by NVIDIA’s cuDNN library, so inference performance on NVIDIA GPU shall be guaranteed.

code walk-through

The overall inference process is as follows:

1. load module

DNN supports multiple module formats including .pb(tensorflow), .onnx, etc. YOLO’s darknet model is also supported. To start, download pre-trained .weights file here.

Load the pre-trained weights with readNet:

cv::dnn::Net net_ = cv::dnn::readNet(weights_location, cfg_location);

2. set backend and target

DNN is a generalized inference engine supporting different backends including OpenCL, CUDA, FPGA, etc. Here we set CUDA as backend to use GPU for inference.

net_.setPreferableBackend(cv::dnn::Backend::DNN_BACKEND_CUDA);
net_.setPreferableTarget(cv::dnn::Target::DNN_TARGET_CUDA);

3. do inference

Use image blob as the input of DNN Net, do computation, then save the results in outs array:

cv::Mat blob = cv::dnn::blobFromImage(img, 1.0 / 255, cv::Size(320, 320), cv::Scalar(), true, false, CV_32F);
net_.setInput(blob);
std::vector<cv::Mat> outs;
net_.forward(outs, net_.getUnconnectedOutLayersNames());

Terminology

Mat

This is OpenCV world’s tensor/numpy-array, representing an n-dimensional array.

Data is wrapped inside Mat object and no manual garbage-collection is needed, just like std::vector. Data manipulation thus is much simpler than raw array.

Blob

The NCHW ordered 4-dimensional Mat transformed from input image. Blob served as the input of DNN Net.

4. NMS(Non-maximum Suppression) filtering

Object detection inference generates lots of similar candidate boxes. NMS is a technique that filters candidates. Here is what NMS process looks like:

5. For each detection, draw a bounding box on frame

The following code loops over all detections and draws a bounding box with classification name and confidence on the frame.

for (size_t i = 0; i < indices.size(); ++i)
{
    int idx = indices[i];
    cv::Rect2d box = boxes[idx];
    float conf = confidences[idx];
    int class_id = classIds[idx];
    add_bounding_box(class_names_, classIds[idx], confidences[idx], box.x, box.y, box.x + box.width, box.y + box.height, img);
}

We can use OpenCV to draw bounding boxes as well!

add_bounding_box(std::vector<std::string> &class_names, int classId, float conf, int left, int top, int right, int bottom, cv::Mat& frame)
{
    cv::rectangle(frame, cv::Point(left, top), cv::Point(right, bottom), cv::Scalar(0, 255, 0));

    std::string label = cv::format("%.2f", conf);
    label = class_names[classId] + ": " + label;
    int baseLine;
    cv::Size labelSize = cv::getTextSize(label, cv::FONT_HERSHEY_SIMPLEX, 0.5, 1, &baseLine);

    top = cv::max(top, labelSize.height);
    cv::rectangle(frame, cv::Point(left, top - labelSize.height),
            cv::Point(left + labelSize.width, top + baseLine), cv::Scalar::all(255), cv::FILLED);
    cv::putText(frame, label, cv::Point(left, top), cv::FONT_HERSHEY_SIMPLEX, 0.5, cv::Scalar());
}

cv::rectangle and cv:putText are 2 useful functions to help us draw bounding boxes and labels.

How DNN module is implemented internally

Let’s deep dive into OpenCV’s source code to see how DNN is implemented.

Net

This class represents an artificial neural network model.

readNet

This function loads model file into Net instance.

forward

The actual inference is done by this function. Let’s see what it looks like:

//source file: https://github.com/opencv/opencv/blob/7ce518106ba041f3cbb27cafda9eb670e5bb99f3/modules/dnn/src/dnn.cpp#L3802
void Net::forward(OutputArrayOfArrays outputBlobs, const String& outputName)
{
    ...
    impl->setUpNet(pins);
    impl->forwardToLayer(impl->getLayerData(layerName));
    ...
}

setUpNet is used to construct DNN Net for computation. The forwardToLayer function is backed by specific backend node which does the concrete computation.

Layer

This class represents an abstract Neural Network Node. It is the building block of DNN Net.

BackendNode

Different backends inherits this class to provide different backend support of Layer.

How NVIDIA GPU is utilized by DNN module

As the above explanation indicates, cuda4dnn submodule provides CUDA backend support for DNN module. It implements abstract interfaces like BackendNode, and wrap concrete computation code inside.

Example: ReLU

ReLU is a typical activation function defined as: $f(x) = max(0,x)$

Let’s see how ReLU is implemented in cuda4dnn:

ReLUOp

ReLUOp is an implementation of CUDABackendNode.

//source file: opencv/modules/dnn/src/cuda4dnn/primitives/activation.hpp
template <class T>
class ReLUOp final : public CUDABackendNode {
public:
    ...
    void forward(
        const std::vector<cv::Ptr<BackendWrapper>>& inputs,
        const std::vector<cv::Ptr<BackendWrapper>>& outputs,
        csl::Workspace& workspace) override
    {
        for (int i = 0; i < inputs.size(); i++)
        {
            kernels::relu<T>(stream, output, input, slope);
        }
    }
    ...
};

When forward function is called, ReLUOp class proxies computation to kernels::relu. Definition of kernels::relu is as follows:

//source: https://github.com/opencv/opencv/blob/8808aaccffaec43d5d276af493ff408d81d4593c/modules/dnn/src/cuda/activations.cu#L126
template <class T>
void relu(const Stream& stream, Span<T> output, View<T> input, T slope) {
    generic_op<T, relu_functor>(stream, output, input, slope);
}

CUDA kernel

A CUDA kernel of type relu_functor will then be generated by following function:

//source: https://github.com/opencv/opencv/blob/8808aaccffaec43d5d276af493ff408d81d4593c/modules/dnn/src/cuda/activations.cu#L30
template <class T, class Functor, std::size_t N, class ...FunctorArgs>
__global__ void generic_op_vec(Span<T> output, View<T> input, FunctorArgs ...functorArgs) {
    using vector_type = get_vector_type_t<T, N>;

    auto output_vPtr = vector_type::get_pointer(output.data());
    auto input_vPtr = vector_type::get_pointer(input.data());

    Functor functor(functorArgs...);

    for (auto i : grid_stride_range(output.size() / vector_type::size())) {
        vector_type vec;
        v_load(vec, input_vPtr[i]);
        for (int j = 0; j < vector_type::size(); j++)
            vec.data[j] = functor(vec.data[j]);
        v_store(output_vPtr[i], vec);
    }
}

Launch kernel

Finally, the kernel will be launched by following function:

//source: https://github.com/opencv/opencv/blob/8808aaccffaec43d5d276af493ff408d81d4593c/modules/dnn/src/cuda/execution.hpp#L64
template <class Kernel, typename ...Args> inline
void launch_kernel(Kernel kernel, Args ...args) {
    auto policy = make_policy(kernel);
    kernel <<<policy.grid, policy.block>>> (args...);
}

ReLU Functor

Let’s see what the core ReLU logic looks like in relu_functor:

//source: https://github.com/opencv/opencv/blob/d981d04c76821037f745a3684533e753d6951e21/modules/dnn/src/cuda/functors.hpp#L92
template <class T>
struct relu_functor {
    __device__ relu_functor(T slope_) : slope{slope_} { }
    __device__ T operator()(T value) {
        return value >= T(0) ? value : slope * value;
    }

    T slope;
};

We can see that cuda4dnn module actually implements Leaky ReLU variant. The signal can be leaked “backward” when input is from negative direction.

Further Reading

1. Sample source code

cgroups is a Linux kernel feature which isolates and limits computer resources(e.g CPU, memory, disk, network, etc).

It is the cornerstone of hottest containerization/orchestration technologies including Docker, Kubernetes, etc.

Background

The essence of any virtualization technique is about isolation and management of something, cgroup is of no exception.

Like the invention of process implements management/isolation of machine code execution, cgroup implements management/isolation of a group of processes.

Real-world example

I was working on an old service platform before. The whole stack was developed since prehistory when containerization was not trending yet. Each service was running as a plain executable inside a VM.

At one time I found that low-memory alerts was constantly triggered by the service. After investigation, turned out that a network issue triggered many error logs. Embedded fluentd log agent consumed too much logs and ate up all RAM processing logs.

We can see that this system is quite fragile, even logging process can crash the whole service. If cgroup-based container and appropriate resource limit is applied, we can confine the problem inside log container and prevent logging issue from affecting service’s main logic.

Play around Linux command

TL;DR. Let’s play with cgroups command to get a more intuitive impression.

Find cgroup of a process

cgroup in essence is an attribute of a process. Thus cgroup info can be found inside /proc/PID/ directory.

cat /proc/self/cgroup

Sample output:

2:blkio:/init.scope
1:name=systemd:/init.scope
0::/init.scope

List all cgroups

Use systemctl status to get cgroups hierarchy. The output is like:

CGroup: /
        ├─user.slice 
        │ └─user-1000.slice 
        │   ├─user@1000.service 
        │   │ ├─gnome-shell-wayland.service 
        │   │ │ ├─ 1129 /usr/bin/gnome-shell
        │   │ ├─gnome-terminal-server.service 
        │   
        ├─init.scope 
        │ └─1 /sbin/init
        └─system.slice 
            ├─systemd-udevd.service 
            │ └─285 /usr/lib/systemd/systemd-udevd
            ├─systemd-journald.service 
            │ └─272 /usr/lib/systemd/systemd-journald
            ├─NetworkManager.service 
            │ └─656 /usr/bin/NetworkManager --no-daemon

We can see that there are 3 big category: init, system and user.

Find resource usage of cgroup

Use systemd-cgtop to find resource usage of each group. Sample output:

Control Group                 Tasks   %CPU   Memory  Input/s Output/s
 /                            2031   76.8    17.7G        -        -
user.slice                    1660   64.1    14.7G        -        -
system.slice                   196    3.4     2.4G        -        -

How cgroup works internally

File-based design

Like most Linux components, cgroup follows the famous rule of Unix: everything is a file. Linux creates a filesytem in /sys/fs/cgroup to represent cgroup. The hierarchy of cgroup mirrors structure of this directory.

Controller

Each folder in cgroup filesystem is called a Controller. I.e. cpu controller is just the folder /sys/fs/cgroup/cpu. If you would like to use a controller, just mount the directory to cgroup filesystem:

mount -t cgroup -o cpu none /sys/fs/cgroup/cpu

Common controllers

cpu

This controller limits CPU time a process can use.

cpuacct

cgroup also provides stats of the group.

memory

This controller limits memory used by process.

Move a process to cgroup

All processes in a cgroup is stored in cgroup.proc. Just write the pid to this file to add a process to cgroup.

echo $$ > /sys/fs/cgroup/cpu/cg1/cgroup.procs

Kernel code analysis

We can not fully understand cgroup without reading source code directly. Let’s dive into Linux kernel source code to see how cgroup is implemented.

source file

The cgroup source code is located at: linux/linux/kernel/cgroup/

Overview

Kernel basically needs to do 2 things about cgroup:

1. In linux/init/main.c, call cgroup_init() to read/initialize root cgroups at system boot;
2. For each process created, make sure it is assigned appropriate cgroup.

data structure

cgroup

Info in cgroup filesystem is loaded into this data structure.

struct cgroup {
    ...
    int level;
    /* Maximum allowed descent tree depth */
    int max_depth;
    int nr_descendants;
    int nr_dying_descendants;
    int max_descendants;

    struct kernfs_node *kn;		/* cgroup kernfs entry */
    struct cgroup_file procs_file;	/* handle for "cgroup.procs" */
    struct cgroup_file events_file;	/* handle for "cgroup.events" */
    ...
};

cgroup_subsys(css)

This is one of the core data structure of cgroup implementation. It represents a specific controller.
Following is the simplified cgroup_subsys struct definition(some detailed code removed):

struct cgroup_subsys {
	struct cgroup_subsys_state *(*css_alloc)(struct cgroup_subsys_state *parent_css);
	int (*css_online)(struct cgroup_subsys_state *css);
	void (*css_offline)(struct cgroup_subsys_state *css);
	void (*css_released)(struct cgroup_subsys_state *css);
	void (*css_free)(struct cgroup_subsys_state *css);
	void (*css_reset)(struct cgroup_subsys_state *css);
	void (*fork)(struct task_struct *task);
	void (*exit)(struct task_struct *task);
	void (*release)(struct task_struct *task);
	void (*bind)(struct cgroup_subsys_state *root_css);
};

The most important function is fork, which is used by kernel to assign necessary cgroup to process.

Example: cpuset

cpuset is one typical cgroup controller. It controls processor placement(a.k.a. Processor affinity) of process. A typical flow of forking a new process with cpuset cgroup is like:

_do_fork(linux/kernel/fork.c)
↓
cgroup_fork
↓
cpuset_fork

_do_fork

_do_fork is the backend of fork system call. The main logic is copying content of current process to a new process.

long _do_fork(unsigned long clone_flags,
	      unsigned long stack_start,
	      unsigned long stack_size,
	      int __user *parent_tidptr,
	      int __user *child_tidptr,
	      unsigned long tls)
{
    ...
    p = copy_process(clone_flags, stack_start, stack_size,
			 child_tidptr, NULL, trace, tls, NUMA_NO_NODE);
    ...
}

For copy_process, basically we need to do some configuration first, then schedule the fork process.

static __latent_entropy struct task_struct *copy_process(
					unsigned long clone_flags,
					unsigned long stack_start,
					unsigned long stack_size,
					int __user *child_tidptr,
					struct pid *pid,
					int trace,
					unsigned long tls,
					int node)
{
    ...
    cgroup_fork(p);
    ...
    retval = sched_fork(clone_flags, p);
    ...
}

cgroup_fork

cgroup_fork does initialization of cgroup data structure, the main cgroup logic is inside cgroup_post_fork:

void cgroup_post_fork(struct task_struct *child) {
    struct cgroup_subsys *ss;
    do_each_subsys_mask(ss, i, have_fork_callback) {
        ss->fork(child);
    } while_each_subsys_mask();
}

As code indicates, fork function of all subsystem(controller) will be called. cpuset is one subsystem with its own version of cgroup_subsys that has all functions implemented:

struct cgroup_subsys cpuset_cgrp_subsys = {
	...
    .css_alloc	= cpuset_css_alloc,
    .css_free	= cpuset_css_free,
    .fork		= cpuset_fork,
    ...
};

cpuset_fork

The cpuset_fork implementation is as follows:

//source file: linux/kernel/cgroup/cpuset.c
static void cpuset_fork(struct task_struct *task)
{
	if (task_css_is_root(task, cpuset_cgrp_id))
		return;
	set_cpus_allowed_ptr(task, &current->cpus_allowed);
	task->mems_allowed = current->mems_allowed;
}

Finally, we reach our travel destination, set_cpus_allowed_ptr. This is the core logic of what cpuset is supposed to do: change process’s CPU affinity.

//source file: linux/kernel/sched/core.c
static int __set_cpus_allowed_ptr(struct task_struct *p,
				  const struct cpumask *new_mask, bool check)
{
    const struct cpumask *cpu_valid_mask = cpu_active_mask;
    unsigned int dest_cpu;
    dest_cpu = cpumask_any_and(cpu_valid_mask, new_mask);
    if (task_running(rq, p) || p->state == TASK_WAKING) {
        struct migration_arg arg = { p, dest_cpu };
        /* Need help from migration thread: drop lock and wait. */
        task_rq_unlock(rq, p, &rf);
        stop_one_cpu(cpu_of(rq), migration_cpu_stop, &arg);
        tlb_migrate_finish(p->mm);
        return 0;
    } else if (task_on_rq_queued(p)) {
        rq = move_queued_task(rq, &rf, p, dest_cpu);
    }
}