说明

k8s v1.24.0 版本之后,删除了docker的代码,运行时推荐使用containerd.悲伤的是没有docker build命令,无法build镜像,需要使用 nerdctl + buildkit组合实现build功能.

镜像差异

nerdctl 不会从本地寻找镜像,例如已经存在 test:1.0.0 的image,如果dockerfile里 from test:1.0.0 docker可以正常从本地获取镜像,nerdctl会从公网查找镜像,需要把镜像发布到registry内网仓库,通过https访问获取.
crictl images是k8s的使用的镜像,和nerdctl images不是一个,需要单独清理.也可以本地导入镜像

# docker
docker save -o nginx.tar nginx:1.24.0
docker load -i nginx.tar
# containerd
ctr -n=k8s.io image export  nginx.tar nginx:1.24.0
ctr -n=k8s.io image import  nginx.tar

安装containerd和配置

yum安装

# 安装需要的软件包, yum-util 提供yum-config-manager功能,另外两个是devicemapper驱动依赖的
yum install -y yum-utils device-mapper-persistent-data lvm2
# 设置 yum 源
# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum install -y containerd

二进制安装

### 安装runc
yum install -y runc

### 下载
wget https://github.com/containerd/containerd/releases/download/v1.7.8/containerd-1.7.8-linux-amd64.tar.gz

### 解压
tar -zxvf containerd-1.7.8-linux-amd64.tar.gz

### copy到 /usr/bin/
cp -rf ./bin/* /usr/bin/

### 生成默认的配置文件
mkdir /etc/containerd
containerd config default > /etc/containerd/config.toml

#注意修改sandbox_image的pause镜像版本最好和K8S的保持一致!!!!

镜像加速,修改/etc/containerd/config.toml

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""

      [plugins."io.containerd.grpc.v1.cri".registry.auths]

      [plugins."io.containerd.grpc.v1.cri".registry.configs]
	    ### 跳过tls验证
        [plugins."io.containerd.grpc.v1.cri".registry.configs."registry.jiagou.com:5000".tls]
          insecure_skip_verify = true

      [plugins."io.containerd.grpc.v1.cri".registry.headers]

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]

        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.jiagou.com:5000"]
          endpoint = ["https://registry.jiagou.com:5000"]
        ### 表示需要配置 mirror 的镜像仓库原镜像仓库,endpoint表示提供 mirror 的镜像加速服务.
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["https://xuachqgw.mirror.aliyuncs.com"]

编写服务脚本 /usr/lib/systemd/system/containerd.service

[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target

注册服务: systemctl enable containerd
启动服务: systemctl start containerd

如果断电重启,可能会出现类似问题: https://github.com/containerd/containerd/issues/3347

find /var/lib/containerd/ -type f -size -5M -name '*.db' |grep -v overlay
##/var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db

### 会丢失镜像记录,需要重新pull镜像!!!!!!
mv /var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db /var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db.bak

### 重启
systemctl restart containerd

安装buildkit

mkdir -p /usr/local/buildkit/
cd /usr/local/buildkit/
wget https://github.com/moby/buildkit/releases/download/v0.12.3/buildkit-v0.12.3.linux-amd64.tar.gz

tar -zxvf buildkit-v0.12.3.linux-amd64.tar.gz 

### 创建软连接
ln -s /usr/local/buildkit/bin/buildctl /usr/local/bin/buildctl

配置/etc/buildkit/buildkitd.toml,非root用户配置文件是~/.config/buildkit/buildkitd.toml

[worker.oci]
  enabled = false

[worker.containerd]
  enabled = true
  # 设置默认命名空间
  #namespace = "default"

   
# optionally mirror configuration can be done by defining it as a registry.
[registry."registry.jiagou.com:5000"]
  ###允许http访问
  http = true
  ###忽略检查https证书,用于自签证书
  insecure=true

编写服务脚本 /etc/systemd/system/buildkit.service

[Unit]
Description=BuildKit
Documentation=https://github.com/moby/buildkit

[Service]
Type=notify
NotifyAccess=all
ExecStart=/usr/local/buildkit/bin/buildkitd --oci-worker=false --containerd-worker=true

[Install]
WantedBy=multi-user.target

注册服务: systemctl enable buildkit
启动服务: systemctl start buildkit

安装nerdctl

wget https://github.com/containerd/nerdctl/releases/download/v1.6.2/nerdctl-1.6.2-linux-amd64.tar.gz

tar -zxvf nerdctl-1.6.2-linux-amd64.tar.gz

cp -rf ./nerdctl /usr/local/bin/nerdctl
### 伪装成 docker命令
ln -s /usr/local/bin/nerdctl /usr/local/bin/docker

##使用 docker pull 和 docker push 时如果出现X509异常,可以添加 --insecure-registry 参数,使用自签证书.
## docker --insecure-registry  pull/push

安装配置registry

nerdctl 默认不解析本地已经存在的镜像,会从公网搜索,所以还是要使用镜像仓库,默认使用registry,配置https证书.

生成SANs https证书

高版本的Chrome浏览器会要求设置subjectAltName,如果没有设置SAN会报证书错误
参考openssl配置文件,Linux服务器上通常在/etc/pki/tls/openssl.cnf
新建文件jiagou.conf

[ req ]
default_bits = 4096
distinguished_name = req_distinguished_name
req_extensions = v3_req

[req_distinguished_name]
countryName = country - CN
stateOrProvinceName = province - henan
localityName = city - zhengzhou
organizationName = company name - jiaogu
commonName = domain name or ip - *.jiagou.com

[ v3_req ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1=*.jiagou.com #可以使用通配符
#IP.1=xxx.xxx.xxx.xxx

使用openssl生成证书

#1.生成根证书密钥
openssl genrsa -out ca.key 4096   #建议长度为4096,1024长度已经被列为不安全.
#2.生成自签名根证书
openssl req -new -x509 -days 36500 -key ca.key -out ca.crt
#3.生成证书密钥
openssl genrsa -out jiagou.key 4096
#向根证书请求签名一个新的证书,由于用户信任了你的根证书,所以根证书签名的其它证书也会被信任
#4.生成csr 注意要使用sha256算法(推荐是sha256算法,默认算法浏览器会报弱加密算法错误)
openssl req -new -key jiagou.key -out jiagou.csr -config ./jiagou.conf -sha256
#5.使用根证书按照csr给证书签名,生成新证书jiagou.crt
openssl x509 -req -days 36500 -in jiagou.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out jiagou.crt -extfile ./jiagou.conf -extensions v3_req

#6.查看证书信息
openssl x509 -text -in jiagou.crt

配置registry.yaml

    spec:
      containers:
      - name: registry
        image: registry:2.8.1
        ###
        imagePullPolicy: IfNotPresent
        env:
        ###设置时区###
        - name: TZ
          value: Asia/Shanghai  

        ###配置生成的域名证书,文件外部挂载
        - name: REGISTRY_HTTP_TLS_CERTIFICATE
          value: /certs/jiagou.crt                     
        - name: REGISTRY_HTTP_TLS_KEY
          value: /certs/jiagou.key

本地解析

修改/etc/hosts,增加本地的域名解析

10.98.239.102 registry.jiagou.com

​