centos 9 - k8s 구성하기 (master, worker01,worker02)

장비 사양

• core : 24
• mem : 62
• disk : 866G
• linux : Rocky Linux release 9.4 (Blue Onyx)
• kernel : 5.14.0-427.13.1.el9_4.x86_64

모든 작업은 root 계정에서 진행

 

모든 노드에서 공통 작업

환경설정

host 설정

vi /etc/hosts


192.168.0.97 tech07 master kubemaster
192.168.0.93 tech03 worker01 kubeworker01
192.168.0.94 tech04 worker02 kubeworker02

 

 

 

swap off

swapon --show
swapoff -a
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
cat /etc/fstab

[root@tech7 ~]#  swapon --show
NAME      TYPE      SIZE USED PRIO
/dev/dm-1 partition  64G   0B   -2
[root@tech7 ~]# swapoff -a
[root@tech7 ~]# sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
[root@tech7 ~]#  cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Thu Aug  1 12:34:37 2024
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/rl-root     /                       xfs     defaults        0 0
UUID=a648de7a-6dcf-4ea2-90cd-967b447f3b7c /boot                   xfs     defaults        0 0
UUID=0D99-6F7D          /boot/efi               vfat    umask=0077,shortname=winnt 0 2
#/dev/mapper/rl-swap     none                    swap    defaults        0 0

 

 

 

커널 모듈 추가 

 

/etc/modules-load.d/containerd.conf 에 overlay, br_netfilter 설정

• overlay : 파일 시스템은 여러 개의 디렉토리를 하나의 디렉토리로 결합하여 읽기-쓰기 가능한 파일 시스템을 제공
• br_netfilter : 네트워크 브리지에서 패킷 필터링을 가능하게 하여 Kubernetes 네트워킹을 지원

sudo tee /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF

 

sudo modprobe overlay            # overlay
sudo modprobe br_netfilter        # kubernetes pod 간의 VxLAN Protocol 통신과 masquerading을 위해

 

 

 

 

커널 파라미터 설정 

sudo tee /etc/sysctl.d/kubernetes.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF



sysctl --system

 

 

Containerd 설치

- runtime 프로그램이 필요한데 도커 말고 containerd를 설치한다 

# 위 페이지에서 Copy한 link의 파일을 wget으로 다운받는다.
wget https://github.com/containerd/containerd/releases/download/v1.7.1/containerd-1.7.1-linux-amd64.tar.gz

# /usr/local 아래에 압축을 풀어준다.
tar Cxzvf /usr/local containerd-1.7.1-linux-amd64.tar.gz

# root로 사용하기 위해 진행했지만 아래부터는 생략해도 무관하다.
mkdir -p ~/.local/bin & cd ~/.local/bin
ln -s /usr/local/bin/containerd containerd
ln -s /usr/local/bin/ctr ctr

# containerd 버전 확인
containerd -v
# containerd github.com/containerd/containerd v1.7.1 1677a17964311325ed1c31e2c0a3589ce6d5c30d

 

containerd config

디렉토리 생성 후 config 등록

mkdir /etc/containerd
containerd config default > /etc/containerd/config.toml

 

 

systemctl srervice에 등록

service 등록을 위해 /usr/lib/systemd/system/containerd.service 생성.

vi /usr/lib/systemd/system/containerd.service



# Copyright The containerd Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
#uncomment to enable the experimental sbservice (sandboxed) version of containerd/cri integration
#Environment="ENABLE_CRI_SANDBOXES=sandboxed"
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target

 

 

 

Containerd systemd cgroup driver 설정

kubernetes v1.22 이상에서는 cgroup driver를 default로 systemd를 사용하도록 코드가 변경되었기 때문에 systemd를 cgroup driver로 사용할 수 있도록 containerd 설정을 변경을 해줘야 합니다. /etc/containerd/config.toml 파일을 열어서 아래와 같이 SystemdCgroup?을 true로 설정해줍니다.

vi /etc/containerd/config.toml 



          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            BinaryName = ""
            CriuImagePath = ""
            CriuPath = ""
            CriuWorkPath = ""
            IoGid = 0
            IoUid = 0
            NoNewKeyring = false
            NoPivotRoot = false
            Root = ""
            ShimCgroup = ""
            SystemdCgroup = true  <-- 이부분false 에서 true로 변경

 

 

 

 

데몬 실행하기 

systemctl daemon-reload
systemctl enable --now containerd	# systemctl start containerd (containerd 시작만 원할시)

# containerd status 확인
systemctl status containerd



[root@tech7 bin]# systemctl status containerd
● containerd.service - containerd container runtime
     Loaded: loaded (/usr/lib/systemd/system/containerd.service; enabled; preset: disabled)
     Active: active (running) since Fri 2024-08-02 10:53:23 KST; 23s ago
       Docs: https://containerd.io
    Process: 53368 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 53369 (containerd)
      Tasks: 18
     Memory: 23.0M
        CPU: 167ms
     CGroup: /system.slice/containerd.service
             └─53369 /usr/local/bin/containerd

 8월 02 10:53:23 tech7 containerd[53369]: time="2024-08-02T10:53:23.101694934+09:00" level=info msg="Start subscribing containerd event"
 8월 02 10:53:23 tech7 containerd[53369]: time="2024-08-02T10:53:23.101738305+09:00" level=info msg="Start recovering state"
 8월 02 10:53:23 tech7 containerd[53369]: time="2024-08-02T10:53:23.101810750+09:00" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
 8월 02 10:53:23 tech7 containerd[53369]: time="2024-08-02T10:53:23.101812833+09:00" level=info msg="Start event monitor"
 8월 02 10:53:23 tech7 containerd[53369]: time="2024-08-02T10:53:23.101837600+09:00" level=info msg="Start snapshots syncer"
 8월 02 10:53:23 tech7 containerd[53369]: time="2024-08-02T10:53:23.101846917+09:00" level=info msg="Start cni network conf syncer for default"
 8월 02 10:53:23 tech7 containerd[53369]: time="2024-08-02T10:53:23.101855563+09:00" level=info msg="Start streaming server"
 8월 02 10:53:23 tech7 containerd[53369]: time="2024-08-02T10:53:23.101846546+09:00" level=info msg=serving... address=/run/containerd/containerd.sock
 8월 02 10:53:23 tech7 containerd[53369]: time="2024-08-02T10:53:23.101902240+09:00" level=info msg="containerd successfully booted in 0.023380s"
 8월 02 10:53:23 tech7 systemd[1]: Started containerd container runtime.

 

 

 

kubeadm, kubelet, kubectl 설치

yum repo 변경

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.29/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.29/rpm/repodata/repomd.xml.key
EOF

 

 

selinux 설정

# permissive 모드로 SELinux 설정(효과적으로 비활성화)
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

 

 

 

설치 진행 

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

 

 

 

runc 설치 

yum install -y runc

 

 

 

Master에서 진행

 

control-plane  설정

control-plane config 받기

 kubeadm config print init-defaults > kubeadm-init.yaml

 

 

control-plane config 수정

vi kubeadm-init.yaml


apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.0.97                  <--- host ip
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  name: tech7                         <--- hostname 
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: 1.29.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12    <--- 대역대 확인

 

 sudo cat << EOF | cat >> kubeadm-init.yaml
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
EOF

 

 

 

초기화 진행

kubeadm init --config=kubeadm-init.yaml

..
...
....


otstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.0.97:6443 --token abcdef.0123456789abcdef \
	--discovery-token-ca-cert-hash sha256:256877167dd45da9186e69ae7cb9a20f832ff74bfb14c179577a37254de6c634

아래에 가이드대로 mkdir 폴더 만들고 권한 및 환경변수 설정해주면 된다 

 

 

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
export KUBECONFIG=/etc/kubernetes/admin.conf

 

 

calio 올림

wget https://docs.projectcalico.org/manifests/calico.yaml --no-check-certificate

kubectl apply -f calico.yaml

 

 

 

 

Worker node 에서 진행 

 

join용 config 만듬

apiVersion: kubeadm.k8s.io/v1beta3
kind: JoinConfiguration
discovery:
  bootstrapToken:
    apiServerEndpoint: "192.168.0.97:6443"
    token: "abcdef.0123456789abcdef"
    caCertHashes:
      - "sha256:256877167dd45da9186e69ae7cb9a20f832ff74bfb14c179577a37254de6c634"
nodeRegistration:
  name: "worker01"

 

apiServerEndpoint : master의 ip port

token : master init 할떄 나온 token 정보

caCertHashes : master init 할떄 나온 discovery-token-ca-cert-hash 정보

nodeRegistration : worker node 의 정보 

 

kubeadm token list

으로 master의 정보를 볼 수 있

 

 

Kubectl 에서 Tab 하면 자동완성하게 설정

리눅스에서 bash 자동 완성 사용하기 | Kubernetes

리눅스에서 bash 자동 완성 사용하기

 

yum install bash-completion

ls -al /usr/share/bash-completion/bash_completion
source /usr/share/bash-completion/bash_completion


## bash 등록
kubectl completion bash | sudo tee /etc/bash_completion.d/kubectl > /dev/null

echo 'alias k=kubectl' >>~/.bashrc
echo 'complete -o default -F __start_kubectl k' >>~/.bashrc

exec bash

 

 

 

 

참고한 블로그

kubernetes 설치 [CentOS 9/v1.27.4] (tistory.com)

kubernetes 설치 [CentOS 9/v1.27.4]

Kubernetes + containerd 설치 ( centos8 ) (tistory.com)