๊ด€๋ฆฌ์ž ํด๋Ÿฌ์Šคํ„ฐ์˜ ์ œ์–ด ์˜์—ญ VM ๋ณต๊ตฌ

Google Distributed Cloud ๊ตฌํ˜„์—์„œ ๊ด€๋ฆฌ์ž ํด๋Ÿฌ์Šคํ„ฐ์˜ ์ œ์–ด ์˜์—ญ VM์—๋Š” ๋‘ ๊ฐœ์˜ ์—ฐ๊ฒฐ๋œ ๋””์Šคํฌ๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค.

  • ๋ถ€ํŒ… ๋””์Šคํฌ์—๋Š” VM์šฉ ์šด์˜์ฒด์ œ๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค.

  • ๋ฐ์ดํ„ฐ ๋””์Šคํฌ์—๋Š” ๊ด€๋ฆฌ์ž ํด๋Ÿฌ์Šคํ„ฐ์˜ ์ƒํƒœ๋ฅผ ์ €์žฅํ•˜๋Š” ์‚ฌ์šฉ์ž ์ธ์ฆ ์ •๋ณด์™€ etcd ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค. ์ฆ‰, ๋ฐ์ดํ„ฐ ๋””์Šคํฌ๋Š” ๊ด€๋ฆฌ์ž ํด๋Ÿฌ์Šคํ„ฐ์˜ ๋ชจ๋“  Kubernetes ๊ฐ์ฒด๋ฅผ ์ €์žฅํ•ฉ๋‹ˆ๋‹ค.

์ด ํŽ˜์ด์ง€์—์„œ๋Š” ์ œ์–ด ์˜์—ญ VM์ด ์†์‹ค๋˜๊ฑฐ๋‚˜ ๋ถ€ํŒ… ๋””์Šคํฌ๊ฐ€ ์†์ƒ๋œ ๊ฒฝ์šฐ ๋ณต๊ตฌํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ๋ณด์—ฌ์ค๋‹ˆ๋‹ค. ์˜ˆ๋ฅผ ๋“ค๋ฉด ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.

  • ์ŠคํŒธ ์ €๋„ ๋กœ๊ทธ๋กœ ์ธํ•ด ๋ถ€ํŒ… ๋””์Šคํฌ๊ฐ€ ์ฝ๊ธฐ ์ „์šฉ์ด ๋ฉ๋‹ˆ๋‹ค.
  • Docker ์˜ค๋ฒ„๋ ˆ์ด ํŒŒ์ผ ์‹œ์Šคํ…œ์ด ์†์ƒ๋ฉ๋‹ˆ๋‹ค.

์ด ํŽ˜์ด์ง€์—์„œ๋Š” ๋ฐ์ดํ„ฐ ๋””์Šคํฌ ๋ณต๊ตฌ์— ๋Œ€ํ•ด์„œ๋Š” ๋‹ค๋ฃจ์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ๋ฐ์ดํ„ฐ ๋””์Šคํฌ๋ฅผ ๋ณต๊ตฌํ•˜๋Š” ๋ฐฉ๋ฒ•์€ ๊ด€๋ฆฌ์ž ํด๋Ÿฌ์Šคํ„ฐ ๋ณต์›์„ ์ฐธ์กฐํ•˜์„ธ์š”.

์ œ์–ด ์˜์—ญ VM ๋ณต๊ตฌ

๊ด€๋ฆฌ์ž ํด๋Ÿฌ์Šคํ„ฐ์˜ ์ œ์–ด ์˜์—ญ VM์„ ๋ณต๊ตฌํ•˜๊ธฐ ์œ„ํ•œ ๋‹จ๊ณ„๋Š” ๋ณด์œ ํ•˜๊ณ  ์žˆ๋Š” ๊ด€๋ฆฌ์ž ํด๋Ÿฌ์Šคํ„ฐ๊ฐ€ ๊ณ ๊ฐ€์šฉ์„ฑ(HA) ๊ด€๋ฆฌ์ž ํด๋Ÿฌ์Šคํ„ฐ์ธ์ง€ ์•„๋‹Œ์ง€์— ๋”ฐ๋ผ ์•ฝ๊ฐ„ ๋‹ฌ๋ผ์ง‘๋‹ˆ๋‹ค.

HA

HA ๊ด€๋ฆฌ์ž ํด๋Ÿฌ์Šคํ„ฐ์—๋Š” 3๊ฐœ์˜ ์ปจํŠธ๋กค ํ”Œ๋ ˆ์ธ VM์ด ์žˆ์Šต๋‹ˆ๋‹ค. ํด๋Ÿฌ์Šคํ„ฐ ์ปจํŠธ๋กค ํ”Œ๋ ˆ์ธ์„ ๊ฐ€์ ธ์˜ค๋ ค๋ฉด VM์ด ์ตœ์†Œ 2๊ฐœ ์ด์ƒ ์žˆ์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. 3๊ฐœ์˜ VM์ด ์‹คํŒจํ•œ ๊ฒฝ์šฐ ์‹คํŒจํ•œ VM์„ ํ•œ ๋ฒˆ์— ํ•˜๋‚˜์”ฉ ๋ณต๊ตฌํ•ฉ๋‹ˆ๋‹ค. ๋‘ ๋ฒˆ์งธ VM์ด ๋ณต๊ตฌ๋˜๊ณ  ์‹คํ–‰๋˜๋ฉด ํด๋Ÿฌ์Šคํ„ฐ ์ปจํŠธ๋กค ํ”Œ๋ ˆ์ธ์ด ๋‹ค์‹œ ์ž‘๋™ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

  1. ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค.

    gkectl repair admin-master --config ADMIN_CLUSTER_CONFIG --kubeconfig ADMIN_CLUSTER_KUBECONFIG
    

    ๋‹ค์Œ์„ ๋ฐ”๊ฟ‰๋‹ˆ๋‹ค.

    • ADMIN_CLUSTER_CONFIG: ๊ด€๋ฆฌ์ž ํด๋Ÿฌ์Šคํ„ฐ ๊ตฌ์„ฑ ํŒŒ์ผ์˜ ๊ฒฝ๋กœ

    • ADMIN_CLUSTER_KUBECONFIG๋ฅผ ๊ด€๋ฆฌ์ž ํด๋Ÿฌ์Šคํ„ฐ์˜ kubeconfig ํŒŒ์ผ ๊ฒฝ๋กœ๋กœ ๋ฐ”๊ฟ‰๋‹ˆ๋‹ค.

    ์ด ๋ช…๋ น์–ด ์ถœ๋ ฅ์€ ๋‹ค์Œ๊ณผ ๋น„์Šทํ•ฉ๋‹ˆ๋‹ค.

    Please select the control plane VM template to be used for re-creating the admin cluster's control plane VM.
    [1] VM template:         /atl-qual-vc07/vm/gke-admin-57f8g-fx9f4c729448z2v8-2-tmpl
        GKE on-prem version: 1.16.0-gke.550
        Creation time:       2023-07-25 01:52:51.815518 +0000 UTC
        CPU:                 4 CPU(s)
        Memory:              16384 MB
        Data disk:           [vsanDatastore] 37a73d64-b823-47cd-2e0c-00620b9189a0/gke-admin-57f8g/default/gke-admin-57f8g-2-data.vmdk
    
    [2] VM template:         /atl-qual-vc07/vm/gke-admin-57f8g-fx9f4c729448z2v8-0-tmpl
        GKE on-prem version: 1.16.0-gke.550
        Creation time:       2023-07-25 01:52:54.228252 +0000 UTC
        CPU:                 4 CPU(s)
        Memory:              16384 MB
        Data disk:           [vsanDatastore] 37a73d64-b823-47cd-2e0c-00620b9189a0/gke-admin-57f8g/default/gke-admin-57f8g-0-data.vmdk
    
    [3] VM template:         /atl-qual-vc07/vm/gke-admin-57f8g-fx9f4c729448z2v8-1-tmpl
        GKE on-prem version: 1.16.0-gke.550
        Creation time:       2023-07-25 01:52:54.210705 +0000 UTC
        CPU:                 4 CPU(s)
        Memory:              16384 MB
        Data disk:           [vsanDatastore] 37a73d64-b823-47cd-2e0c-00620b9189a0/gke-admin-57f8g/default/gke-admin-57f8g-1-data.vmdk
    
    Please enter your numeric choice:
    
  2. ๋ณต๊ตฌํ•  VM์˜ ๋ฒˆํ˜ธ๋ฅผ ์ž…๋ ฅํ•ฉ๋‹ˆ๋‹ค. ์ถœ๋ ฅ์— VM์ด ํ‘œ์‹œ๋˜์ง€ ์•Š์œผ๋ฉด Google Cloud ์ง€์›ํŒ€์— ๋ฌธ์˜ํ•˜์„ธ์š”.

    ๋ณต๊ตฌํ•ด์•ผ ํ•˜๋Š” VM์ด 3๊ฐœ ์žˆ๋Š” ๊ฒฝ์šฐ gkectl repair admin-master๋Š” ์ฒซ ๋ฒˆ์งธ VM์„ ๋ณต๊ตฌํ•œ ํ›„ ๋‹ค์Œ๊ณผ ์œ ์‚ฌํ•œ ์˜ค๋ฅ˜ ๋ฉ”์‹œ์ง€๋ฅผ ์ถœ๋ ฅํ•ฉ๋‹ˆ๋‹ค.

    If you are repairing admin control plane VM for HA admin cluster,
    it's possible that the API server is still down after repairing one
    of the VMs. Try continue fixing other control plane VMs listed to
    recover the quorum of control plane.
    

    ์ด ๊ฒฝ์šฐ ๋ช…๋ น์–ด๋ฅผ ๋‹ค์‹œ ์‹คํ–‰ํ•˜์—ฌ ๋‘ ๋ฒˆ์งธ VM์„ ๋ณต๊ตฌํ•ฉ๋‹ˆ๋‹ค.

HA ์ด์™ธ

๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค.

gkectl repair admin-master \
  --config ADMIN_CLUSTER_CONFIG \
  --kubeconfig ADMIN_CLUSTER_KUBECONFIG

๋‹ค์Œ์„ ๋ฐ”๊ฟ‰๋‹ˆ๋‹ค.

  • ADMIN_CLUSTER_CONFIG: ๊ด€๋ฆฌ์ž ํด๋Ÿฌ์Šคํ„ฐ ๊ตฌ์„ฑ ํŒŒ์ผ์˜ ๊ฒฝ๋กœ
  • ADMIN_CLUSTER_KUBECONFIG๋ฅผ ๊ด€๋ฆฌ์ž ํด๋Ÿฌ์Šคํ„ฐ์˜ kubeconfig ํŒŒ์ผ ๊ฒฝ๋กœ๋กœ ๋ฐ”๊ฟ‰๋‹ˆ๋‹ค.

๊ด€๋ฆฌ์ž ํด๋Ÿฌ์Šคํ„ฐ์˜ ์ œ์–ด ์˜์—ญ VM์€ VM์„ ๋‹ค์‹œ ์ƒ์„ฑํ•˜๋Š” ๋ฐ ํ•„์š”ํ•œ ๋ชจ๋“  ์ •๋ณด๊ฐ€ ๋“ค์–ด ์žˆ๋Š” VM ํ…œํ”Œ๋ฆฟ์— ํด๋ก ๋ฉ๋‹ˆ๋‹ค. gkectl repair admin-master ๋ช…๋ น์–ด๋Š” VM ํ…œํ”Œ๋ฆฟ์„ ์‚ฌ์šฉํ•˜์—ฌ ์ƒˆ VM์„ ๋งŒ๋“ญ๋‹ˆ๋‹ค. ๊ทธ๋Ÿฐ ๋‹ค์Œ ์ƒˆ ๋ถ€ํŒ… ๋””์Šคํฌ์™€ ๊ธฐ์กด ๋ฐ์ดํ„ฐ ๋””์Šคํฌ๋ฅผ ์—ฐ๊ฒฐํ•ฉ๋‹ˆ๋‹ค.

ํด๋Ÿฌ์Šคํ„ฐ ๋…ธ๋“œ๊ฐ€ DHCP ์„œ๋ฒ„์—์„œ ์ฃผ์†Œ๋ฅผ ๊ฐ€์ ธ์˜ฌ ๊ฒฝ์šฐ ์ƒˆ VM์€ ์›๋ณธ VM๊ณผ ๋‹ค๋ฅธ IP ์ฃผ์†Œ๋ฅผ ๋ณด์œ ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋‹ค์Œ ๋‹จ๊ณ„

  • ์ถ”๊ฐ€ ์ง€์›์ด ํ•„์š”ํ•˜๋ฉด Cloud Customer Care์— ์—ฐ๋ฝํ•ฉ๋‹ˆ๋‹ค.

    ๋‹ค์Œ์„ ๋น„๋กฏํ•œ ์ง€์› ๋ฆฌ์†Œ์Šค์— ๊ด€ํ•œ ์ž์„ธํ•œ ๋‚ด์šฉ์€ ์ง€์› ๋ฐ›๊ธฐ๋ฅผ ์ฐธ์กฐํ•˜์„ธ์š”.