Kubernetes์—์„œ AlloyDB Omni์šฉ ์ปค๋„ ํŒŒ๋ผ๋ฏธํ„ฐ ๊ตฌ์„ฑ

๋ฌธ์„œ ๋ฒ„์ „์„ ์„ ํƒํ•ฉ๋‹ˆ๋‹ค.

์ด ํŽ˜์ด์ง€์—์„œ๋Š” AlloyDB Omni Kubernetes ์—ฐ์‚ฐ์ž์™€ AlloyDB Omni ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค ์—”์ง„์˜ ์ตœ์  ์„ฑ๋Šฅ์„ ์œ„ํ•ด AlloyDB Omni ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค ํด๋Ÿฌ์Šคํ„ฐ๋ฅผ ํ˜ธ์ŠคํŒ…ํ•˜๋Š” Kubernetes ๋…ธ๋“œ๋ฅผ ์„ค์ •ํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ์„ค๋ช…ํ•ฉ๋‹ˆ๋‹ค.

์ปค๋„ ๋งค๊ฐœ๋ณ€์ˆ˜๋ฅผ ๊ตฌ์„ฑํ•˜๋ฉด AlloyDB Omni๊ฐ€ ๊ณผ๋„ํ•œ ์›Œํฌ๋กœ๋“œ๋ฅผ ์ฒ˜๋ฆฌํ•  ๋•Œ ์‹œ์Šคํ…œ ๋ฉ”๋ชจ๋ฆฌ์™€ IO ๋ฆฌ์†Œ์Šค๋ฅผ ๋” ํšจ์œจ์ ์œผ๋กœ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๊ธฐ๋ณธ ์š”๊ฑด

์‹œ์ž‘ํ•˜๊ธฐ ์ „์— Kubernetes ๋…ธ๋“œ๊ฐ€ Linux ์ปค๋„ 6.1 ์ด์ƒ, ํŠนํžˆ MADV_COLLAPSE ๋ฐ MADV_POPULATE_WRITE ํ”Œ๋ž˜๊ทธ๋ฅผ ์ง€์›ํ•˜๋Š” ์ปค๋„์„ ์‹คํ–‰ํ•˜๋Š”์ง€ ํ™•์ธํ•˜์„ธ์š”. ์ด๋Ÿฌํ•œ ํ”Œ๋ž˜๊ทธ์— ๋Œ€ํ•œ ์ž์„ธํ•œ ๋‚ด์šฉ์€ Linux madwise ๋ฌธ์„œ๋ฅผ ์ฐธ๊ณ ํ•˜์„ธ์š”.

๋‹ค์Œ ํ‘œ์—๋Š” ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค ํด๋Ÿฌ์Šคํ„ฐ ํฌ๋“œ๋ฅผ ์‹คํ–‰ํ•˜๋Š” Kubernetes ๋…ธ๋“œ์— ํ•„์š”ํ•œ ์ปค๋„ ๋งค๊ฐœ๋ณ€์ˆ˜์™€ ํ•ด๋‹น ๊ฐ’์ด ๋‚˜์™€ ์žˆ์Šต๋‹ˆ๋‹ค.

ํŒŒ์ผ ๊ฐ’
/sys/kernel/mm/transparent_hugepage/shmem_enabled
๊ณต์œ  ๋ฉ”๋ชจ๋ฆฌ์— ๊ด€ํ•œ ์ž์„ธํ•œ ๋‚ด์šฉ์€ Transparent Hugepage Support๋ฅผ ์ฐธ๊ณ ํ•˜์„ธ์š”.
within_size ๋˜๋Š” always
/proc/sys/vm/max_map_count
ํ”„๋กœ์„ธ์Šค์—์„œ ๋งŒ๋“ค ์ˆ˜ ์žˆ๋Š” ๋ฉ”๋ชจ๋ฆฌ ๋งต ์˜์—ญ์˜ ์ˆ˜์— ๊ด€ํ•œ ์ž์„ธํ•œ ๋‚ด์šฉ์€ /proc/sys/vm ๋ฌธ์„œ๋ฅผ ์ฐธ๊ณ ํ•˜์„ธ์š”.
1073741824 ์ด์ƒ

DaemonSet๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ Kubernetes ๋…ธ๋“œ์—์„œ ์ปค๋„ ๋งค๊ฐœ๋ณ€์ˆ˜๋ฅผ ๊ตฌ์„ฑํ•˜๋ ค๋ฉด ๋‹ค์Œ ๋‹จ๊ณ„๋ฅผ ๋”ฐ๋ฅด์„ธ์š”.

  1. ๋…ธ๋“œ์— ๋ผ๋ฒจ ์ถ”๊ฐ€์˜ ์•ˆ๋‚ด์— ๋”ฐ๋ผ ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค ํด๋Ÿฌ์Šคํ„ฐ๋ฅผ ์‹คํ–‰ํ•  ๋…ธ๋“œ์— ๋ผ๋ฒจ์„ ์ ์šฉํ•ฉ๋‹ˆ๋‹ค.

  2. LABEL_KEY=LABEL_VALUE ๋ผ๋ฒจ์ด ์ง€์ •๋œ ๋…ธ๋“œ์—์„œ ์ปค๋„ ๋งค๊ฐœ๋ณ€์ˆ˜๋ฅผ ์„ค์ •ํ•˜๋ ค๋ฉด DaemonSet๋ฅผ ๋งŒ๋“œ์„ธ์š”.

    cat << EOF | kubectl apply -f -
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: alloydb-omni-kernel-tuning
      namespace: DS_NAMESPACE
    spec:
      selector:
        matchLabels:
          name: alloydb-omni-kernel-tuning
      template:
        metadata:
          labels:
            name: alloydb-omni-kernel-tuning
        spec:
          volumes:
            - name: host-sys
              hostPath:
                path: /sys
          nodeSelector:
            LABEL_KEY: "LABEL_VALUE"
          restartPolicy: Always
          terminationGracePeriodSeconds: 1
          initContainers:
            - name: enable-thp-mmc
              image: INIT_IMAGE
              volumeMounts:
                - name: host-sys
                  mountPath: /sys
              securityContext:
                privileged: true
              command: ["sh", "-c", "sysctl -w vm.max_map_count=MAX_MAP_COUNT && echo within_size >/sys/kernel/mm/transparent_hugepage/shmem_enabled"]
          containers:
            - name: CONTAINER_NAME
              image: CONTAINER_IMAGE
              command: ["watch", "-n", "600", "cat", "/sys/kernel/mm/transparent_hugepage/shmem_enabled"]
    EOF
    

    ๋‹ค์Œ์„ ๋ฐ”๊ฟ‰๋‹ˆ๋‹ค.

    • DS_NAMESPACE: DaemonSet์„ ๋ฐฐํฌํ•  ๋„ค์ž„์ŠคํŽ˜์ด์Šค์ž…๋‹ˆ๋‹ค(์˜ˆ: default).
    • LABEL_KEY: ๋ผ๋ฒจ์˜ ์‹๋ณ„์ž์ž…๋‹ˆ๋‹ค(์˜ˆ: workload).
    • LABEL_VALUE: ๋ผ๋ฒจ์˜ ์‹๋ณ„์ž์™€ ์—ฐ๊ฒฐ๋œ ๋ฐ์ดํ„ฐ์ž…๋‹ˆ๋‹ค(์˜ˆ: database).
    • INIT_IMAGE: init ์ปจํ…Œ์ด๋„ˆ์— ์‚ฌ์šฉ๋œ ์ด๋ฏธ์ง€์˜ ์ด๋ฆ„์ž…๋‹ˆ๋‹ค.
    • MAX_MAP_COUNT: ํ”„๋กœ์„ธ์Šค๊ฐ€ ๋งŒ๋“ค ์ˆ˜ ์žˆ๋Š” ์ตœ๋Œ€ ๋ฉ”๋ชจ๋ฆฌ ๋งต ์˜์—ญ ์ˆ˜์ž…๋‹ˆ๋‹ค(์˜ˆ: 1073741824).
    • CONTAINER_NAME: ๊ธฐ๋ณธ ์ปจํ…Œ์ด๋„ˆ์˜ ์ด๋ฆ„์ž…๋‹ˆ๋‹ค(์˜ˆ: main).
    • CONTAINER_IMAGE: ๊ธฐ๋ณธ ์ปจํ…Œ์ด๋„ˆ์— ์‚ฌ์šฉ๋˜๋Š” ์ด๋ฏธ์ง€์˜ ์ด๋ฆ„์ž…๋‹ˆ๋‹ค(์˜ˆ: latest).
  3. DaemonSet๋ฅผ ๋ฐฐํฌํ•œ ํ›„ Kubernetes ํด๋Ÿฌ์Šคํ„ฐ์˜ ๋ชจ๋“  ํฌ๋“œ๊ฐ€ Running ์ƒํƒœ์ด๊ณ  LABEL_KEY=LABEL_VALUE ๋ผ๋ฒจ์ด ์ง€์ •๋œ ๊ฐ ๋…ธ๋“œ์— ํฌ๋“œ๊ฐ€ ํ•˜๋‚˜์”ฉ ์žˆ๋Š”์ง€ ํ™•์ธํ•˜๋ ค๋ฉด ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‚ฌ์šฉํ•˜์„ธ์š”.

    kubectl get pods -l LABEL_KEY=LABEL_VALUE
    

    ์ถœ๋ ฅ ์˜ˆ์‹œ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.

    NAME                               READY   STATUS    RESTARTS   AGE
    alloydb-omni-kernel-tuning-2dkwh   1/1     Running   0          22s
    alloydb-omni-kernel-tuning-pgkbj   1/1     Running   0          19s
    
  4. ์ปค๋„ ํ”Œ๋ž˜๊ทธ๊ฐ€ ์„ฑ๊ณต์ ์œผ๋กœ ์ ์šฉ๋˜์—ˆ๋Š”์ง€ ํ™•์ธํ•˜๋ ค๋ฉด DaemonSet์„ ๋ฐฐํฌํ•œ ํ›„ ์‹๋ณ„๋œ ๊ฐ ํฌ๋“œ์— ๋Œ€ํ•ด ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค.

    kubectl logs POD_NAME
    

    POD_NAME์„ ๋กœ๊ทธ๋ฅผ ๊ฒ€์‚ฌํ•˜๋ ค๋Š” ํฌ๋“œ์˜ ์ด๋ฆ„์œผ๋กœ ๋ฐ”๊ฟ‰๋‹ˆ๋‹ค.

    ์ถœ๋ ฅ ์˜ˆ์‹œ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.

    always [within_size] advise never deny force
    

๊ถŒ์žฅ ์ปค๋„ ๋งค๊ฐœ๋ณ€์ˆ˜๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ Kubernetes ๋…ธ๋“œ์— ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค ํด๋Ÿฌ์Šคํ„ฐ๋ฅผ ๋ฐฐํฌํ•˜๋ ค๋ฉด ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค ํด๋Ÿฌ์Šคํ„ฐ ๋งค๋‹ˆํŽ˜์ŠคํŠธ์˜ ๋‹ค์Œ ์„น์…˜ ์ค‘ ํ•˜๋‚˜์— nodeAffinity ์„น์…˜์„ ์ถ”๊ฐ€ํ•˜์„ธ์š”.

  • ๊ธฐ๋ณธ ์ธ์Šคํ„ด์Šค์˜ ๊ฒฝ์šฐ primarySpec.schedulingConfig
  • ์ฝ๊ธฐ ํ’€ ์ธ์Šคํ„ด์Šค์˜ ๊ฒฝ์šฐ spec.schedulingConfig
      nodeaffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: LABEL_KEY
              operator: In
              values:
              - "LABEL_VALUE"

ํŠน์ • Kubernetes ๋…ธ๋“œ์—์„œ ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค ํด๋Ÿฌ์Šคํ„ฐ๋ฅผ ์‹คํ–‰ํ•˜๋Š” ๋ฐฉ๋ฒ•์— ๋Œ€ํ•œ ์ž์„ธํ•œ ๋‚ด์šฉ์€ ์˜ˆ์•ฝ์„ ์‚ฌ์šฉํ•˜์—ฌ ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค ํด๋Ÿฌ์Šคํ„ฐ์— ๋…ธ๋“œ ํ• ๋‹น์„ ์ฐธ๊ณ ํ•˜์„ธ์š”.

์ปค๋„ ๋งค๊ฐœ๋ณ€์ˆ˜ ์ ์šฉ ํ™•์ธ

์ปค๋„ ๋งค๊ฐœ๋ณ€์ˆ˜๊ฐ€ ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค ํฌ๋“œ์— ์ ์šฉ๋˜์—ˆ๋Š”์ง€ ํ™•์ธํ•˜๋ ค๋ฉด ๋‹ค์Œ์„ ์‹คํ–‰ํ•˜์„ธ์š”.

  1. ๊ณ ๊ฐ€์šฉ์„ฑ์ด ์‚ฌ์šฉ ์„ค์ •๋œ ๊ฒฝ์šฐ(ํŠนํžˆ spec.availability.numberOfStandbys์ด 0๋ณด๋‹ค ํฐ ๊ฐ’์œผ๋กœ ์„ค์ •๋œ ๊ฒฝ์šฐ) ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค ๋งž์ถค ๋ฆฌ์†Œ์Šค๊ฐ€ DBCLUSTERPHASE ์—ด์— DBClusterReady์„ ํ‘œ์‹œํ•˜๊ณ  HAREADYSTATUS ์—ด์— True์„ ํ‘œ์‹œํ•˜๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

    kubectl get dbcluster -n DBCLUSTER_NAMESPACE DBCLUSTER_NAME
    

    ๋‹ค์Œ์„ ๋ฐ”๊ฟ‰๋‹ˆ๋‹ค.

    • DBCLUSTER_NAME: ํ™•์ธํ•˜๋Š” ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค ํด๋Ÿฌ์Šคํ„ฐ์˜ ์ด๋ฆ„์ž…๋‹ˆ๋‹ค.
    • DBCLUSTER_NAMESPACE: ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค ํด๋Ÿฌ์Šคํ„ฐ๊ฐ€ ์žˆ๋Š” ํŠน์ • ๋„ค์ž„์ŠคํŽ˜์ด์Šค์˜ ์ด๋ฆ„์ž…๋‹ˆ๋‹ค.

    ์ถœ๋ ฅ ์˜ˆ์‹œ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.

    NAME               PRIMARYENDPOINT   PRIMARYPHASE   DBCLUSTERPHASE   HAREADYSTATUS   HAREADYREASON
    dbcluster-sample   10.29.21.240      Ready          DBClusterReady   True            Ready
    
  2. ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค ํด๋Ÿฌ์Šคํ„ฐ์— ์†ํ•˜๋Š” Kubernetes ํฌ๋“œ๋ฅผ ๋‚˜์—ดํ•ฉ๋‹ˆ๋‹ค.

    kubectl get pods -n DBCLUSTER_NAMESPACE -l alloydbomni.internal.dbadmin.goog/dbcluster=DBCLUSTER_NAME
    
  3. ๊ณต์œ  ๋ฉ”๋ชจ๋ฆฌ ์ •๋ณด๋ฅผ ํ™•์ธํ•˜๋ ค๋ฉด ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค.

    kubectl exec -n DBCLUSTER_NAMESPACE POD_NAME -- grep Shmem /proc/meminfo
    
  4. ์ถœ๋ ฅ์˜ ๋ชจ๋“  ํ•ญ๋ชฉ์— 0์ด ์•„๋‹Œ ์ˆซ์ž๊ฐ€ ํฌํ•จ๋˜์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

    ์ถœ๋ ฅ ์˜ˆ์‹œ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.

    Shmem:          126255872 kB
    ShmemHugePages: 18403328 kB
    ShmemPmdMapped:  2961408 kB
    

    ํ•ญ๋ชฉ์— 0 kB๊ฐ€ ํ‘œ์‹œ๋˜๋ฉด ํ•ด๋‹น ํฌ๋“œ๋ฅผ ๋‹ค์‹œ ์‹œ์ž‘ํ•ฉ๋‹ˆ๋‹ค.

    kubectl delete pod -n DBCLUSTER_NAMESPACE POD_NAME
    
  5. ํฌ๋“œ๊ฐ€ Running ์ƒํƒœ๊ฐ€ ๋˜๋ฉด 1~5๋‹จ๊ณ„๋ฅผ ๋ฐ˜๋ณตํ•ฉ๋‹ˆ๋‹ค.