[k8s] operator-sdk(Go)로 memcahce 오퍼레이터 구축하기 - 1

 

Quickstart for Go-based Operators | Operator SDK

Quickstart for Go-based Operators

 

필수 체크 요소 

- cluster-admin 권한이 있어야함 

- example.com 이 기본 namespace로 된다 

 

프로젝트 생성과 초기화 

operator-sdk 초기화

mkdir memcached-operator
cd memcached-operator
operator-sdk init --domain example.com --repo github.com/example/memcached-operator

 

trace

[root@tech7 memcached-operator]# operator-sdk init --domain example.com --repo github.com/example/memcached-operator
INFO[0000] Writing kustomize manifests for you to edit...
INFO[0000] Writing scaffold for you to edit...
INFO[0000] Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.17.3

INFO[0002] Update dependencies:
$ go mod tidy
Next: define a resource with:
$ operator-sdk create api

 

 

 

Memcached API 생성

Api 와 Controller 를 생성한다 

Operator가 어떤 리소스를 관리할지 정의한다 

operator-sdk create api --group cache --version v1alpha1 --kind Memcached --resource --controller

- group : CR의 그룹이름 (cache)

- version : API 버전 (v1alpha1)

- kind : CR의 이름 (Memcached)

 

trace

-bash: 예기치 않은 토큰 ';' 근처에서 문법 오류
[root@tech7 memcached-operator]# ls
Dockerfile  Makefile  PROJECT  README.md  cmd  config  go.mod  go.sum  hack  test
[root@tech7 memcached-operator]# cd ../
[root@tech7 Operator]# ls
kopf  kopf-02  memcached-operator  operator-sdk  test
[root@tech7 Operator]# cd memcached-operator/
[root@tech7 memcached-operator]# ls
Dockerfile  Makefile  PROJECT  README.md  cmd  config  go.mod  go.sum  hack  test
[root@tech7 memcached-operator]# operator-sdk create api --group cache --version v1alpha1 --kind Memcached --resource --controller
INFO[0000] Writing kustomize manifests for you to edit...
INFO[0000] Writing scaffold for you to edit...
INFO[0000] api/v1alpha1/memcached_types.go
INFO[0000] api/v1alpha1/groupversion_info.go
INFO[0000] internal/controller/suite_test.go
INFO[0000] internal/controller/memcached_controller.go
INFO[0000] internal/controller/memcached_controller_test.go
INFO[0000] Update dependencies:
$ go mod tidy
INFO[0000] Running make:
$ make generate
mkdir -p /root/k8s/Operator/memcached-operator/bin
Downloading sigs.k8s.io/controller-tools/cmd/controller-gen@v0.14.0
/root/k8s/Operator/memcached-operator/bin/controller-gen-v0.14.0 object:headerFile="hack/boilerplate.go.txt" paths="./..."
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests

 

 

api 폴더 안에 v1alpha 폴더와 bin 폴더가  생성되었다 

- before

[root@tech7 memcached-operator]# ls
Dockerfile  Makefile  PROJECT  README.md  cmd  config  go.mod  go.sum  hack  test

- after

[root@tech7 memcached-operator]# ls
Dockerfile  Makefile  PROJECT  README.md  api  bin  cmd  config  go.mod  go.sum  hack  internal  test

 

api 폴더 안에는 go 파일들이 생성이 된다 

[root@tech7 v1alpha1]# pwd
/root/k8s/Operator/memcached-operator/api/v1alpha1

[root@tech7 v1alpha1]# ls
groupversion_info.go  memcached_types.go  zz_generated.deepcopy.go

 

bin 폴더 안에는 controller 실행파일이 생성디 되었다 

[root@tech7 bin]# pwd
/root/k8s/Operator/memcached-operator/bin
[root@tech7 bin]# ls
controller-gen-v0.14.0

 

 

API 코드 살펴보기

경로 : */memcached-operator/api/v1alpha1

파일 

1. groupversion_info.go  

- API그룹과 Version 정보가 있다 

- SchemeBuilder로 CR을 k8s에 등록할때 필요한 객체를 정의한다 

- k8s가 API인식할 수 있게 한다  

2. memcached_types.go

- 중요

- CRD의 spec 과 status 정의

- spec에선 replica 와 img 등을 정의하고

- Status에서는 현재 리소스의 상태를 관리 

3. zz_generated.deepcopy.go:

- DeepCopy 함수를 자동으로 생성한 코드

- Golang에선 객체 복사할때 Deep Copy 함수가 필요하다

- Operator가 Reconcile 함수에서 spec과 status를 업데이트 할 때 필요하다 

 

1. groupversion_info.go

package v1alpha1

import (
        "k8s.io/apimachinery/pkg/runtime/schema"
        "sigs.k8s.io/controller-runtime/pkg/scheme"
)

var (
        // GroupVersion is group version used to register these objects
        GroupVersion = schema.GroupVersion{Group: "cache.example.com", Version: "v1alpha1"}

        // SchemeBuilder is used to add go types to the GroupVersionKind scheme
        SchemeBuilder = &scheme.Builder{GroupVersion: GroupVersion}

        // AddToScheme adds the types in this group-version to the given scheme.
        AddToScheme = SchemeBuilder.AddToScheme
)

group 명 (cache) 과 group 버전(v1alpha1)으로 schema를 생성

 

2. memcached_types.go

package v1alpha1

import (
        metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!
// NOTE: json tags are required.  Any new fields you add must have json tags for the fields to be serialized.

// MemcachedSpec defines the desired state of Memcached
type MemcachedSpec struct {
        // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
        // Important: Run "make" to regenerate code after modifying this file

        // Foo is an example field of Memcached. Edit memcached_types.go to remove/update
        Foo string `json:"foo,omitempty"`
}

// MemcachedStatus defines the observed state of Memcached
type MemcachedStatus struct {
        // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
        // Important: Run "make" to regenerate code after modifying this file
}

//+kubebuilder:object:root=true
//+kubebuilder:subresource:status

// Memcached is the Schema for the memcacheds API
type Memcached struct {
        metav1.TypeMeta   `json:",inline"`
        metav1.ObjectMeta `json:"metadata,omitempty"`

        Spec   MemcachedSpec   `json:"spec,omitempty"`
        Status MemcachedStatus `json:"status,omitempty"`
}

//+kubebuilder:object:root=true

// MemcachedList contains a list of Memcached
type MemcachedList struct {
        metav1.TypeMeta `json:",inline"`
        metav1.ListMeta `json:"metadata,omitempty"`
        Items           []Memcached `json:"items"`
}

func init() {
        SchemeBuilder.Register(&Memcached{}, &MemcachedList{})

 이 부분에서 CR의 스펙과 Status를 정의할 수 있다 .  

구조체 : MemcachedSpec , MemcachedStatus , Memcached , MemcachedList 

함수 :  init 

 

 

3. zz_generated.deepcopy.go

// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *Memcached) DeepCopyInto(out *Memcached) {
	*out = *in
	out.TypeMeta = in.TypeMeta
	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
	out.Spec = in.Spec
	out.Status = in.Status
}

// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Memcached.
func (in *Memcached) DeepCopy() *Memcached {
	if in == nil {
		return nil
	}
	out := new(Memcached)
	in.DeepCopyInto(out)
	return out
}

// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
func (in *Memcached) DeepCopyObject() runtime.Object {
	if c := in.DeepCopy(); c != nil {
		return c
	}
	return nil
}

// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *MemcachedList) DeepCopyInto(out *MemcachedList) {
	*out = *in
	out.TypeMeta = in.TypeMeta
	in.ListMeta.DeepCopyInto(&out.ListMeta)
	if in.Items != nil {
		in, out := &in.Items, &out.Items
		*out = make([]Memcached, len(*in))
		for i := range *in {
			(*in)[i].DeepCopyInto(&(*out)[i])
		}
	}
}

// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MemcachedList.
func (in *MemcachedList) DeepCopy() *MemcachedList {
	if in == nil {
		return nil
	}
	out := new(MemcachedList)
	in.DeepCopyInto(out)
	return out
}

// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
func (in *MemcachedList) DeepCopyObject() runtime.Object {
	if c := in.DeepCopy(); c != nil {
		return c
	}
	return nil
}

// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *MemcachedSpec) DeepCopyInto(out *MemcachedSpec) {
	*out = *in
}

// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MemcachedSpec.
func (in *MemcachedSpec) DeepCopy() *MemcachedSpec {
	if in == nil {
		return nil
	}
	out := new(MemcachedSpec)
	in.DeepCopyInto(out)
	return out
}

// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *MemcachedStatus) DeepCopyInto(out *MemcachedStatus) {
	*out = *in
}

// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MemcachedStatus.
func (in *MemcachedStatus) DeepCopy() *MemcachedStatus {
	if in == nil {
		return nil
	}
	out := new(MemcachedStatus)
	in.DeepCopyInto(out)
	return out
}

 

 

docker 이미지 build 하고 push 하기 

operaotr를 Docker 이미지로 build 하고 push 한다 

k8s 가 Operator 실행할 때 필요한 이미지. 

 

경로

[root@tech7 memcached-operator]# pwd
/root/k8s/Operator/memcached-operator
[root@tech7 memcached-operator]# ls
Dockerfile  Makefile  PROJECT  README.md  api  bin  cmd  config  go.mod  go.sum  hack  internal  test

 

실행하기

make docker-build docker-push IMG="example.com/memcached-operator:v0.0.1"

 

에러 발생

[root@tech7 memcached-operator]# make docker-build docker-push IMG="example.com/memcached-operator:v0.0.1"
docker build -t example.com/memcached-operator:v0.0.1 .
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
[1/2] STEP 1/11: FROM golang:1.21 AS builder
✔ registry.access.redhat.com/golang:1.21
Trying to pull registry.access.redhat.com/golang:1.21...

Error: creating build container: initializing source docker://registry.access.redhat.com/golang:1.21: reading manifest 1.21 in registry.access.redhat.com/golang: name unknown: Repo not found
make: *** [Makefile:145: docker-build] 오류 125

podname login docker.io 로 login을 한다 

 

 

 

OLM은 무엇인가 

Operator Lifecycle Manager

Operator의 배포와 관리를 쉽게 하기 위한 툴

k8s에 Operator 설치와 업데이트, 의존성 , 권한 관리 가능 

 - CSV파일로 Operator를 관리할 수 있다

bundle 로 묶어서 여러 이미지를 한번에 배포하는 걸로 보인다 

 

배포하기 

OLM 없이 직접 배포를 해보겠다 

 

 

1. operator를 k8s에 배포한다 

make deploy IMG="example.com/memcached-operator:v0.0.1"

 

에러 (go 버전이 맞지 않는다)

# internal/race
compile: version "go1.23.2" does not match go tool version "go1.21.11 (Red Hat 1.21.11-1.el9_4)"

[root@tech7 memcached-operator]# go version
go version go1.21.11 (Red Hat 1.21.11-1.el9_4) linux/amd64

 

버전을 맞춰줬다

[root@tech7 memcached-operator]# go version
go version go1.23.2 linux/amd64

 

해결

[root@tech7 memcached-operator]# make deploy IMG="suamj/memcached-operator:v0.0.1"
/root/k8s/Operator/memcached-operator/bin/controller-gen-v0.14.0 rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
Downloading sigs.k8s.io/kustomize/kustomize/v5@v5.3.0
cd config/manager && /root/k8s/Operator/memcached-operator/bin/kustomize-v5.3.0 edit set image controller=suamj/memcached-operator:v0.0.1
/root/k8s/Operator/memcached-operator/bin/kustomize-v5.3.0 build config/default | kubectl apply -f -
namespace/memcached-operator-system created
customresourcedefinition.apiextensions.k8s.io/memcacheds.cache.example.com created
serviceaccount/memcached-operator-controller-manager created
role.rbac.authorization.k8s.io/memcached-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/memcached-operator-manager-role created
clusterrole.rbac.authorization.k8s.io/memcached-operator-memcached-editor-role created
clusterrole.rbac.authorization.k8s.io/memcached-operator-memcached-viewer-role created
clusterrole.rbac.authorization.k8s.io/memcached-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/memcached-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/memcached-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/memcached-operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/memcached-operator-proxy-rolebinding created
service/memcached-operator-controller-manager-metrics-service created
deployment.apps/memcached-operator-controller-manager created

 

 

 

2. CR을 K8S에 생성한다 

명령 수행에 앞서 파일에 뭐가있는지 본다

kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml

 

- cache_v1alpha1_memcached.yaml 

[root@tech7 samples]# pwd
/root/k8s/Operator/memcached-operator/config/samples


[root@tech7 samples]# cat cache_v1alpha1_memcached.yaml
apiVersion: cache.example.com/v1alpha1  # API 버전
kind: Memcached                         # 리소스의 종류, 여기서는 Memcached CR
metadata:
  labels:
    app.kubernetes.io/name: memcached-operator
    app.kubernetes.io/managed-by: kustomize
  name: memcached-sample                 # 리소스의 이름
spec:
  # TODO(user): Add fields here          # 사용자가 정의할 추가 설정 필드

 

- kustomization.yaml

[root@tech7 samples]# cat kustomization.yaml
## Append samples of your project ##
resources:
- cache_v1alpha1_memcached.yaml
#+kubebuilder:scaffold:manifestskustomizesamples

 

 

CR을 등록

[root@tech7 memcached-operator]#  kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml
memcached.cache.example.com/memcached-sample created