Envoy ๋ฐฐํฌ ๋ฌธ์ œ ํ•ด๊ฒฐ

์ด ๊ฐ€์ด๋“œ์—์„œ๋Š” Google API๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ Cloud Service Mesh๋ฅผ ์‹คํ–‰ํ•  ๋•Œ Envoy ํด๋ผ์ด์–ธํŠธ ๊ตฌ์„ฑ ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜๋Š” ๋ฐ ๋„์›€์ด ๋˜๋Š” ์ •๋ณด๋ฅผ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค. Client State Discovery Service(CSDS) API๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ Cloud Service Mesh ๋ฌธ์ œ๋ฅผ ์กฐ์‚ฌํ•˜๋Š” ๋ฐฉ๋ฒ•์€ Cloud Service Mesh ํด๋ผ์ด์–ธํŠธ ์ƒํƒœ ์ดํ•ด๋ฅผ ์ฐธ์กฐํ•˜์„ธ์š”.

VM์— ์„ค์น˜๋œ Envoy ๋ฒ„์ „ ํ™•์ธ

๋‹ค์Œ ์•ˆ๋‚ด์— ๋”ฐ๋ผ ๊ฐ€์ƒ ๋จธ์‹ (VM) ์ธ์Šคํ„ด์Šค์—์„œ ์‹คํ–‰ ์ค‘์ธ Envoy ๋ฒ„์ „์„ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

Envoy ๋ฒ„์ „์„ ํ™•์ธํ•˜๊ฑฐ๋‚˜ ๊ฒ€์‚ฌํ•˜๋ ค๋ฉด ๋‹ค์Œ ์ค‘ ํ•˜๋‚˜๋ฅผ ์ˆ˜ํ–‰ํ•˜๋ฉด ๋ฉ๋‹ˆ๋‹ค.

gce-service-proxy/proxy-version ๊ฒฝ๋กœ์—์„œ VM์˜ ๊ฒŒ์ŠคํŠธ ์†์„ฑ์„ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

  gcloud compute --project cloud-vm-mesh-monitoring instances get-guest-attributes INSTANCE_NAME 
--zone ZONEc --query-path=gce-service-proxy/proxy-version

NAMESPACE KEY VALUE gce-service-proxy proxy-version dc78069b10cc94fa07bb974b7101dd1b42e2e7bf/1.15.1-dev/Clean/RELEASE/BoringSSL

๋‹ค์Œ๊ณผ ๊ฐ™์€ ์ฟผ๋ฆฌ๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ Google Cloud Console์˜ VM ์ธ์Šคํ„ด์Šค ์„ธ๋ถ€์ •๋ณด Logging ํŽ˜์ด์ง€์—์„œ Cloud Logging ์ธ์Šคํ„ด์Šค ๋กœ๊ทธ๋ฅผ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

  resource.type="gce_instance"
  resource.labels.instance_id="3633122484352464042"
  jsonPayload.message:"Envoy version"
  

๋‹ค์Œ๊ณผ ๊ฐ™์€ ์‘๋‹ต์ด ์ˆ˜์‹ ๋ฉ๋‹ˆ๋‹ค.

  {
    "insertId": "9zy0btf94961a",
    "jsonPayload": {
      "message": "Envoy Version: dc78069b10cc94fa07bb974b7101dd1b42e2e7bf/1.15.1-dev/Clean/RELEASE/BoringSSL",
      "localTimestamp": "2021-01-12T11:39:14.3991Z"
    },
    "resource": {
      "type": "gce_instance",
      "labels": {
        "zone": "asia-southeast1-b",
        "instance_id": "3633122484352464042",
        "project_id": "cloud-vm-mesh-monitoring"
      }
    },
    "timestamp": "2021-01-12T11:39:14.399200504Z",
    "severity": "INFO",
    "logName": "projects/cloud-vm-mesh-monitoring/logs/service-proxy-agent",
    "receiveTimestamp": "2021-01-12T11:39:15.407023427Z"
  }
  

SSH๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ VM์— ์—ฐ๊ฒฐํ•˜๊ณ  ๋ฐ”์ด๋„ˆ๋ฆฌ ๋ฒ„์ „์„ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

  YOUR_USER_NAME@backend-mig-5f5651e1-517a-4269-b457-f6bdcf3d98bc-m3wt:~$ /usr/local/bin/envoy --version

/usr/local/bin/envoy version: dc78069b10cc94fa07bb974b7101dd1b42e2e7bf/1.15.1-dev/Clean/RELEASE/BoringSSL

SSH๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ VM ๋ฐ ๊ด€๋ฆฌ ์ธํ„ฐํŽ˜์ด์Šค์— ๋ฃจํŠธ๋กœ ์—ฐ๊ฒฐํ•ฉ๋‹ˆ๋‹ค.

  root@backend-mig-5f5651e1-517a-4269-b457-f6bdcf3d98bc-m3wt:~# curl localhost:15000/server_info
  {
   "version": "dc78069b10cc94fa07bb974b7101dd1b42e2e7bf/1.15.1-dev/Clean/RELEASE/BoringSSL",
   "state": "LIVE",
   "hot_restart_version": "disabled",
   ...
  }
  

Envoy ๋กœ๊ทธ ์œ„์น˜

์ผ๋ถ€ ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜๋ ค๋ฉด Envoy ํ”„๋ก์‹œ ๋กœ๊ทธ๋ฅผ ์กฐ์‚ฌํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

SSH๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ VM ์ธ์Šคํ„ด์Šค์— ์—ฐ๊ฒฐํ•ด ๋กœ๊ทธ ํŒŒ์ผ์„ ๊ฐ€์ ธ์˜ฌ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๊ฒฝ๋กœ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

  /var/log/envoy/envoy.err.log
  

ํ”„๋ก์‹œ๊ฐ€ Cloud Service Mesh์— ์—ฐ๊ฒฐ๋˜์ง€ ์•Š์Œ

ํ”„๋ก์‹œ๊ฐ€ Cloud Service Mesh์— ์—ฐ๊ฒฐ๋˜์ง€ ์•Š์œผ๋ฉด ๋‹ค์Œ์„ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค.

  • Envoy ํ”„๋ก์‹œ ๋กœ๊ทธ์— trafficdirector.googleapis.com ์—ฐ๊ฒฐ ๊ด€๋ จ ์˜ค๋ฅ˜๊ฐ€ ์žˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

  • ๋ชจ๋“  ํŠธ๋ž˜ํ”ฝ์„ Envoy ํ”„๋ก์‹œ๋กœ ๋ฆฌ๋””๋ ‰์…˜ํ•˜๋„๋ก iptables๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ netfilter๋ฅผ ์„ค์ •ํ•œ ๊ฒฝ์šฐ ํ”„๋ก์‹œ๋ฅผ ์‹คํ–‰ํ•˜๋Š” ์‚ฌ์šฉ์ž(UID)๊ฐ€ ๋ฆฌ๋””๋ ‰์…˜์—์„œ ์ œ์™ธ๋˜๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค. ๊ทธ๋ ‡์ง€ ์•Š์œผ๋ฉด ํŠธ๋ž˜ํ”ฝ์ด ์ง€์†์ ์œผ๋กœ ํ”„๋ก์‹œ๋กœ ๋˜๋Œ์•„๊ฐ‘๋‹ˆ๋‹ค.

  • ํ”„๋กœ์ ํŠธ์— Cloud Service Mesh API๋ฅผ ์‚ฌ์šฉ ์„ค์ •ํ–ˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค. ํ”„๋กœ์ ํŠธ์˜ API ๋ฐ ์„œ๋น„์Šค์—์„œ Cloud Service Mesh API ์˜ค๋ฅ˜๋ฅผ ์ฐพ์Šต๋‹ˆ๋‹ค.

  • VM์„ ๋งŒ๋“ค ๋•Œ ๋‹ค์Œ์„ ์ง€์ •ํ•˜์—ฌ VM์˜ API ์•ก์„ธ์Šค ๋ฒ”์œ„๊ฐ€Google Cloud API์— ๋Œ€ํ•œ ์ „์ฒด ์•ก์„ธ์Šค๋ฅผ ํ—ˆ์šฉํ•˜๋„๋ก ์„ค์ •๋˜์–ด ์žˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

    --scopes=https://www.googleapis.com/auth/cloud-platform
    
  • ์„œ๋น„์Šค ๊ณ„์ •์— ์˜ฌ๋ฐ”๋ฅธ ๊ถŒํ•œ์ด ์žˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค. ์ž์„ธํ•œ ๋‚ด์šฉ์€ Traffic Director API์— ์•ก์„ธ์Šคํ•˜๋„๋ก ์„œ๋น„์Šค ๊ณ„์ • ์‚ฌ์šฉ ์„ค์ •์„ ์ฐธ์กฐํ•˜์„ธ์š”.

  • VM์—์„œ trafficdirector.googleapis.com:443์— ์•ก์„ธ์Šคํ•  ์ˆ˜ ์žˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค. ์ด ์•ก์„ธ์Šค์— ๋ฌธ์ œ๊ฐ€ ์žˆ๋Š” ๊ฒฝ์šฐ ๋ฐฉํ™”๋ฒฝ ํฌํŠธ๊ฐ€ TCP ํฌํŠธ 443์„ ํ†ตํ•ด trafficdirector.googleapis.com์— ์•ก์„ธ์Šคํ•˜์ง€ ๋ชปํ•˜๋„๋ก ํ•˜๊ฑฐ๋‚˜ trafficdirector.googleapis.com ํ˜ธ์ŠคํŠธ ์ด๋ฆ„์˜ DNS ํ™•์ธ ๋ฌธ์ œ์ผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

  • ์‚ฌ์ด๋“œ์นด ํ”„๋ก์‹œ์— Envoy๋ฅผ ์‚ฌ์šฉํ•˜๋Š” ๊ฒฝ์šฐ Envoy ์ถœ์‹œ ๋ฒ„์ „์ด 1.24.9 ์ด์ƒ์ธ์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

Cloud Service Mesh๋กœ ๊ตฌ์„ฑ๋œ ์„œ๋น„์Šค์— ์—ฐ๊ฒฐํ•  ์ˆ˜ ์—†์Œ

Cloud Service Mesh๋กœ ๊ตฌ์„ฑ๋œ ์„œ๋น„์Šค์— ์—ฐ๊ฒฐํ•  ์ˆ˜ ์—†์œผ๋ฉด ์‚ฌ์ด๋“œ์นด ํ”„๋ก์‹œ๊ฐ€ ์‹คํ–‰ ์ค‘์ด๊ณ  Cloud Service Mesh์— ์—ฐ๊ฒฐํ•  ์ˆ˜ ์žˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

Envoy๋ฅผ ์‚ฌ์ด๋“œ์นด ํ”„๋ก์‹œ๋กœ ์‚ฌ์šฉํ•˜๋Š” ๊ฒฝ์šฐ ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‹คํ–‰ํ•˜์—ฌ ์ด๋ฅผ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

  1. ๋ช…๋ น์ค„์—์„œ Envoy ํ”„๋กœ์„ธ์Šค๊ฐ€ ์‹คํ–‰ ์ค‘์ธ์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

    ps aux | grep envoy
    
  2. Envoy์˜ ๋Ÿฐํƒ€์ž„ ๊ตฌ์„ฑ์„ ๊ฒ€์‚ฌํ•˜์—ฌ Cloud Service Mesh์—์„œ ๋™์  ๋ฆฌ์†Œ์Šค๋ฅผ ๊ตฌ์„ฑํ–ˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค. ๊ตฌ์„ฑ์„ ๋ณด๋ ค๋ฉด ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค.

    curl http://localhost:15000/config_dump
    
  3. ์‚ฌ์ด๋“œ์นด ํ”„๋ก์‹œ์— ๋Œ€ํ•œ ํŠธ๋ž˜ํ”ฝ ๊ฐ€๋กœ์ฑ„๊ธฐ๊ฐ€ ์˜ฌ๋ฐ”๋ฅด๊ฒŒ ์„ค์ •๋˜์—ˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค. iptables๋กœ ๋ฆฌ๋””๋ ‰์…˜์„ ์„ค์ •ํ•˜๋ ค๋ฉด iptables ๋ช…๋ น์–ด๋ฅผ ์‹คํ–‰ํ•œ ํ›„ ์ถœ๋ ฅ์„ grepํ•˜์—ฌ ๊ทœ์น™์ด ์žˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

    sudo iptables -t nat -S | grep ISTIO
    

    ๋‹ค์Œ์€ ๊ฐ€์ƒ IP ์ฃผ์†Œ(VIP) 10.0.0.1/32๋ฅผ ๊ฐ€๋กœ์ฑ„์„œ ํฌํŠธ 15001์— UID 1006์œผ๋กœ ์ „๋‹ฌํ•˜๋Š” iptables์˜ ์ถœ๋ ฅ ์˜ˆ์‹œ์ž…๋‹ˆ๋‹ค.

    -N ISTIO_IN_REDIRECT
    -N ISTIO_OUTPUT
    -N ISTIO_REDIRECT
    -A OUTPUT -p tcp -j ISTIO_OUTPUT
    -A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15001
    -A ISTIO_OUTPUT -m owner --uid-owner 1006 -j RETURN
    -A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
    -A ISTIO_OUTPUT -d 10.0.0.1/32 -j ISTIO_REDIRECT
    -A ISTIO_OUTPUT -j RETURN
    

Google Cloud Console์„ ํ†ตํ•ด VM ์ธ์Šคํ„ด์Šค๋ฅผ ๋งŒ๋“  ๊ฒฝ์šฐ ์ผ๋ถ€ IPv6 ๊ด€๋ จ ๋ชจ๋“ˆ์€ ๋‹ค์‹œ ์‹œ์ž‘ํ•˜๊ธฐ ์ „์— ์„ค์น˜ ๋ฐ ์ œ๊ณต๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ์ด๋กœ ์ธํ•ด ์ข…์† ํ•ญ๋ชฉ์ด ๋ˆ„๋ฝ๋˜์–ด iptables๊ฐ€ ์‹คํŒจํ•ฉ๋‹ˆ๋‹ค. ์ด ๊ฒฝ์šฐ VM์„ ๋‹ค์‹œ ์‹œ์ž‘ํ•˜๊ณ  ์„ค์ • ํ”„๋กœ์„ธ์Šค๋ฅผ ๋‹ค์‹œ ์‹คํ–‰ํ•˜๋ฉด ๋ฌธ์ œ๊ฐ€ ํ•ด๊ฒฐ๋ฉ๋‹ˆ๋‹ค. Google Cloud CLI๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ๋งŒ๋“  Compute Engine VM์—์„œ๋Š” ์ด๋Ÿฌํ•œ ๋ฌธ์ œ๊ฐ€ ๋ฐœ์ƒํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.

Envoy ์•ก์„ธ์Šค ๋กœ๊น…์ด ๊ตฌ์„ฑ๋œ ๊ฒฝ์šฐ ์„œ๋น„์Šค์— ์—ฐ๊ฒฐํ•  ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค.

Cloud Service Mesh์šฉ Envoy ๋ถ€ํŠธ์ŠคํŠธ๋žฉ ์†์„ฑ ๊ตฌ์„ฑ์˜ ์„ค๋ช…๋Œ€๋กœ TRAFFICDIRECTOR_ACCESS_LOG_PATH๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ Envoy ์•ก์„ธ์Šค ๋กœ๊ทธ๋ฅผ ๊ตฌ์„ฑํ•œ ๊ฒฝ์šฐ Envoy ํ”„๋ก์‹œ๋ฅผ ์‹คํ–‰ํ•˜๋Š” ์‹œ์Šคํ…œ ์‚ฌ์šฉ์ž์—๊ฒŒ ์ง€์ •๋œ ์•ก์„ธ์Šค ๋กœ๊ทธ ์œ„์น˜์— ์ž‘์„ฑํ•  ์ˆ˜ ์žˆ๋Š” ๊ถŒํ•œ์ด ์žˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

ํ•„์š”ํ•œ ๊ถŒํ•œ์„ ์ œ๊ณตํ•˜์ง€ ์•Š์œผ๋ฉด ๋ฆฌ์Šค๋„ˆ๊ฐ€ ํ”„๋ก์‹œ์—์„œ ํ”„๋กœ๊ทธ๋ž˜๋ฐ๋˜์ง€ ์•Š์œผ๋ฉฐ Envoy ํ”„๋ก์‹œ ๋กœ๊ทธ์—์„œ ๋‹ค์Œ ์˜ค๋ฅ˜ ๋ฉ”์‹œ์ง€๋ฅผ ํ™•์ธํ•˜์—ฌ ๊ฐ์ง€ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected:
Error adding/updating listener(s) TRAFFICDIRECTOR_INTERCEPTION_PORT:
unable to open file '/var/log/envoy.log': Permission denied

๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜๊ธฐ ์œ„ํ•ด Envoy ์‚ฌ์šฉ์ž๊ฐ€ ์•ก์„ธ์Šค ๋กœ๊ทธ์— ์ž‘์„ฑํ•  ์ˆ˜ ์žˆ๋„๋ก ์„ ํƒํ•œ ํŒŒ์ผ์˜ ๊ถŒํ•œ์„ ๋ณ€๊ฒฝํ•ฉ๋‹ˆ๋‹ค.

๊ตฌ์„ฑ ๋ฌธ์ œ๋ฅผ ํ‘œ์‹œํ•˜๋Š” Envoy ๋กœ๊ทธ์˜ ์˜ค๋ฅ˜ ๋ฉ”์‹œ์ง€

์ด ์„น์…˜์€ ๋ถ€ํ•˜ ๋ถ„์‚ฐ API๋ฅผ ์‚ฌ์šฉํ•˜๋Š” ๋ฐฐํฌ์— ์ ์šฉ๋ฉ๋‹ˆ๋‹ค.

Cloud Service Mesh ๊ตฌ์„ฑ์— ๋ฌธ์ œ๊ฐ€ ์žˆ์œผ๋ฉด Envoy ๋กœ๊ทธ์— ๋‹ค์Œ๊ณผ ๊ฐ™์€ ์˜ค๋ฅ˜ ๋ฉ”์‹œ์ง€๊ฐ€ ํ‘œ์‹œ๋  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

  • warning envoy config    StreamAggregatedResources gRPC config stream closed:
    5, Cloud Service Mesh configuration was not found for network "VPC_NAME" in
    project "PROJECT_NUMBER".
  • warning envoy upstream  StreamLoadStats gRPC config stream closed:
    5, Cloud Service Mesh configuration was not found for network "VPC_NAME" in
    project "PROJECT_NUMBER".
  • warning envoy config    StreamAggregatedResources gRPC config stream closed:
    5, Requested entity was not found.
  • warning envoy upstream  StreamLoadStats gRPC config stream closed:
    5, Requested entity was not found.
  • Cloud Service Mesh configuration was not found.

๋งˆ์ง€๋ง‰ ์˜ค๋ฅ˜ ๋ฉ”์‹œ์ง€(Traffic Director configuration was not found)๋Š” ์ผ๋ฐ˜์ ์œผ๋กœ Envoy๊ฐ€ Cloud Service Mesh์—์„œ ๊ตฌ์„ฑ์„ ์š”์ฒญํ•˜์ง€๋งŒ ์ผ์น˜ํ•˜๋Š” ๊ตฌ์„ฑ์„ ์ฐพ์„ ์ˆ˜ ์—†์Œ์„ ๋‚˜ํƒ€๋ƒ…๋‹ˆ๋‹ค. Envoy๊ฐ€ Cloud Service Mesh์— ์—ฐ๊ฒฐํ•˜๋ฉด VPC ๋„คํŠธ์›Œํฌ ์ด๋ฆ„(์˜ˆ: my-network)์ด ํ‘œ์‹œ๋ฉ๋‹ˆ๋‹ค. ๊ทธ๋Ÿฐ ๋‹ค์Œ Cloud Service Mesh์—์„œ INTERNAL_SELF_MANAGED ๋ถ€ํ•˜ ๋ถ„์‚ฐ ์Šคํ‚ด์ด ์žˆ๊ณ  ๊ฐ™์€ VPC ๋„คํŠธ์›Œํฌ ์ด๋ฆ„์„ ์ฐธ์กฐํ•˜๋Š” ์ „๋‹ฌ ๊ทœ์น™์„ ์ฐพ์Šต๋‹ˆ๋‹ค.

์ด ์˜ค๋ฅ˜๋ฅผ ๊ณ ์น˜๋ ค๋ฉด ๋‹ค์Œ ์•ˆ๋‚ด๋ฅผ ๋”ฐ๋ฅด์„ธ์š”.

  1. ๋„คํŠธ์›Œํฌ์— ๋ถ€ํ•˜ ๋ถ„์‚ฐ ์Šคํ‚ด INTERNAL_SELF_MANAGED๊ฐ€ ์žˆ๋Š” ์ „๋‹ฌ ๊ทœ์น™์ด ์žˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค. ์ „๋‹ฌ ๊ทœ์น™์˜ VPC ๋„คํŠธ์›Œํฌ ์ด๋ฆ„์„ ๊ธฐ๋กํ•ฉ๋‹ˆ๋‹ค.

  2. Compute Engine์—์„œ ์ž๋™ํ™”๋œ Envoy ๋ฐฐํฌ์™€ ํ•จ๊ป˜ Cloud Service Mesh๋ฅผ ์‚ฌ์šฉํ•˜๋Š” ๊ฒฝ์šฐ --service-proxy:network ํ”Œ๋ž˜๊ทธ์— ์ œ๊ณต๋œ ๊ฐ’์ด ์ „๋‹ฌ ๊ทœ์น™์˜ VPC ๋„คํŠธ์›Œํฌ ์ด๋ฆ„๊ณผ ์ผ์น˜ํ•˜๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

  3. Compute Engine์—์„œ ์ˆ˜๋™ Envoy ๋ฐฐํฌ์™€ ํ•จ๊ป˜ Cloud Service Mesh๋ฅผ ์‚ฌ์šฉํ•˜๋Š” ๊ฒฝ์šฐ ๋‹ค์Œ๊ณผ ๊ฐ™์ด Envoy ๋ถ€ํŠธ์ŠคํŠธ๋žฉ ํŒŒ์ผ์„ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

    1. TRAFFICDIRECTOR_NETWORK_NAME ๋ณ€์ˆ˜ ๊ฐ’์ด ์ „๋‹ฌ ๊ทœ์น™์˜ VPC ๋„คํŠธ์›Œํฌ ์ด๋ฆ„๊ณผ ์ผ์น˜ํ•˜๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.
    2. ํ”„๋กœ์ ํŠธ ๋ฒˆํ˜ธ๊ฐ€ TRAFFICDIRECTOR_GCP_PROJECT_NUMBER ๋ณ€์ˆ˜์— ์„ค์ •๋˜์–ด ์žˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.
  4. GKE์— ๋ฐฐํฌํ•˜๊ณ  ์ž๋™ ์ธ์ ํ„ฐ๋ฅผ ์‚ฌ์šฉํ•˜๊ณ  ์žˆ๋Š” ๊ฒฝ์šฐ ํ”„๋กœ์ ํŠธ ๋ฒˆํ˜ธ์™€ VPC ๋„คํŠธ์›Œํฌ ์ด๋ฆ„์ด ์ž๋™ Envoy ์‚ฝ์ž…์œผ๋กœ GKE ํฌ๋“œ์šฉ Cloud Service Mesh ์„ค์ •์˜ ์•ˆ๋‚ด์— ๋”ฐ๋ผ ์˜ฌ๋ฐ”๋ฅด๊ฒŒ ๊ตฌ์„ฑ๋˜์—ˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

Compute Engine ๋ฌธ์ œ ํ•ด๊ฒฐ

์ด ์„น์…˜์—์„œ๋Š” Compute Engine์˜ Envoy ๋ฐฐํฌ ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜๊ธฐ ์œ„ํ•œ ์•ˆ๋‚ด๋ฅผ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.

Envoy ๋ฐ VM ๋ถ€ํŠธ์ŠคํŠธ๋žฉ ํ”„๋กœ์„ธ์Šค ๋ฐ ์ถ”๊ฐ€ ์ˆ˜๋ช… ์ฃผ๊ธฐ ๊ด€๋ฆฌ ์ž‘์—…์€ ์ผ์‹œ์ ์ธ ์—ฐ๊ฒฐ ๋ฌธ์ œ, ์ €์žฅ์†Œ ์†์ƒ, ๋ถ€ํŠธ์ŠคํŠธ๋žฉ ์Šคํฌ๋ฆฝํŠธ ๋ฐ VM ๊ธฐ๋ฐ˜ ์—์ด์ „ํŠธ์˜ ๋ฒ„๊ทธ, ์˜ˆ๊ธฐ์น˜ ์•Š์€ ์‚ฌ์šฉ์ž ์ž‘์—… ๋“ฑ ์—ฌ๋Ÿฌ ๊ฐ€์ง€ ์ด์œ ๋กœ ์‹คํŒจํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋ฌธ์ œ ํ•ด๊ฒฐ์„ ์œ„ํ•œ ์ปค๋ฎค๋‹ˆ์ผ€์ด์…˜ ์ฑ„๋„

Google Cloud ๋Š” ๋ถ€ํŠธ์ŠคํŠธ๋žฉ ํ”„๋กœ์„ธ์Šค ๋ฐ VM์— ์žˆ๋Š” ๊ตฌ์„ฑ์š”์†Œ์˜ ํ˜„์žฌ ์ƒํƒœ๋ฅผ ์ดํ•ดํ•˜๋Š” ๋ฐ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ๋Š” ํ†ต์‹  ์ฑ„๋„์„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.

๊ฐ€์ƒ ์ง๋ ฌ ํฌํŠธ ์ถœ๋ ฅ ๋กœ๊น…

VM์˜ ์šด์˜์ฒด์ œ, BIOS, ๊ธฐํƒ€ ์‹œ์Šคํ…œ ์ˆ˜์ค€ ํ•ญ๋ชฉ์€ ์ผ๋ฐ˜์ ์œผ๋กœ ์ง๋ ฌ ํฌํŠธ์— ์ถœ๋ ฅ์„ ์ž‘์„ฑํ•ฉ๋‹ˆ๋‹ค. ์ด ์ถœ๋ ฅ ๊ฒฐ๊ณผ๋Š” ์‹œ์Šคํ…œ ์žฅ์• , ๋ถ€ํŒ… ์‹คํŒจ, ์‹œ์ž‘ ๋ฌธ์ œ, ์ข…๋ฃŒ ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜๋Š”๋ฐ ์œ ์šฉํ•ฉ๋‹ˆ๋‹ค

Compute Engine ๋ถ€ํŠธ์ŠคํŠธ๋žฉ ์—์ด์ „ํŠธ๋Š” ์ˆ˜ํ–‰๋œ ๋ชจ๋“  ์ž‘์—…์„ ์‹œ์Šคํ…œ ํฌํŠธ 1์— ๊ธฐ๋กํ•ฉ๋‹ˆ๋‹ค. ์—ฌ๊ธฐ์—๋Š” ์ธ์Šคํ„ด์Šค์˜ ๋ฉ”ํƒ€๋ฐ์ดํ„ฐ ์„œ๋ฒ„, iptables ๊ตฌ์„ฑ, Envoy ์„ค์น˜ ์ƒํƒœ๋กœ๋ถ€ํ„ฐ ๋ฐ์ดํ„ฐ๋ฅผ ๊ฐ€์ ธ์™€ ๊ธฐ๋ณธ ํŒจํ‚ค์ง€ ์„ค์น˜๋ถ€ํ„ฐ ์‹œ์ž‘ํ•˜๋Š” ์‹œ์Šคํ…œ ์ด๋ฒคํŠธ๊ฐ€ ํฌํ•จ๋ฉ๋‹ˆ๋‹ค.

VM ๊ธฐ๋ฐ˜ ์—์ด์ „ํŠธ๋Š” Envoy ํ”„๋กœ์„ธ์Šค ์ƒํƒœ, ์ƒˆ๋กœ ๋ฐœ๊ฒฌ๋œ Cloud Service Mesh ์„œ๋น„์Šค, VM ๋ฌธ์ œ๋ฅผ ์กฐ์‚ฌํ•  ๋•Œ ์œ ์šฉํ•  ์ˆ˜ ์žˆ๋Š” ๊ธฐํƒ€ ๋ชจ๋“  ์ •๋ณด๋ฅผ ๋กœ๊น…ํ•ฉ๋‹ˆ๋‹ค.

Cloud Monitoring ๋กœ๊น…

์ง๋ ฌ ํฌํŠธ ์ถœ๋ ฅ์— ๋…ธ์ถœ๋œ ๋ฐ์ดํ„ฐ๋Š” Monitoring์— ๋กœ๊น…๋ฉ๋‹ˆ๋‹ค. ์ด๋Š” Golang ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋ฅผ ์‚ฌ์šฉํ•˜๊ณ  ๋กœ๊ทธ๋ฅผ ๋ณ„๋„์˜ ๋กœ๊ทธ๋กœ ๋‚ด๋ณด๋‚ด ๋…ธ์ด์ฆˆ๋ฅผ ์ค„์ž…๋‹ˆ๋‹ค. ์ด ๋กœ๊ทธ๋Š” ์ธ์Šคํ„ด์Šค ์ˆ˜์ค€ ๋กœ๊ทธ์ด๋ฏ€๋กœ, ๋‹ค๋ฅธ ์ธ์Šคํ„ด์Šค ๋กœ๊ทธ์™€ ๊ฐ™์ด ๋™์ผํ•œ ํŽ˜์ด์ง€์—์„œ ์„œ๋น„์Šค ํ”„๋ก์‹œ ๋กœ๊ทธ๋ฅผ ์ฐพ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

VM ๊ฒŒ์ŠคํŠธ ์†์„ฑ

๊ฒŒ์ŠคํŠธ ์†์„ฑ์€ ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜์ด ์ธ์Šคํ„ด์Šค์—์„œ ์‹คํ–‰๋˜๋Š” ๋™์•ˆ ์“ธ ์ˆ˜ ์žˆ๋Š” ํŠน์ • ์œ ํ˜•์˜ ์ปค์Šคํ…€ ๋ฉ”ํƒ€๋ฐ์ดํ„ฐ์ž…๋‹ˆ๋‹ค. ์ธ์Šคํ„ด์Šค์˜ ๋ชจ๋“  ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜ ๋˜๋Š” ์‚ฌ์šฉ์ž๋Š” ์ด๋Ÿฌํ•œ ๊ฒŒ์ŠคํŠธ ์†์„ฑ ๋ฉ”ํƒ€๋ฐ์ดํ„ฐ ๊ฐ’์„ ์ฝ๊ณ  ์—ฌ๊ธฐ์— ๋ฐ์ดํ„ฐ๋ฅผ ์“ธ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

Compute Engine Envoy ๋ถ€ํŠธ์ŠคํŠธ๋žฉ ์Šคํฌ๋ฆฝํŠธ์™€ VM ์—์ด์ „ํŠธ๋Š” ๋ถ€ํŠธ์ŠคํŠธ๋žฉ ์ฒ˜๋ฆฌ ๋ฐ Envoy์˜ ํ˜„์žฌ ์ƒํƒœ์— ๋Œ€ํ•œ ์ •๋ณด์™€ ํ•จ๊ป˜ ์†์„ฑ์„ ๋…ธ์ถœํ•ฉ๋‹ˆ๋‹ค. ๋ชจ๋“  ๊ฒŒ์ŠคํŠธ ์†์„ฑ์€ gce-service-proxy ๋„ค์ž„์ŠคํŽ˜์ด์Šค์— ๋…ธ์ถœ๋ฉ๋‹ˆ๋‹ค.

gcloud compute instances get-guest-attributes INSTANCE_NAME  \
    --query-path=gce-service-proxy/ \
    --zone=ZONE

๋ฌธ์ œ๊ฐ€ ๋ฐœ๊ฒฌ๋˜๋ฉด ๊ฒŒ์ŠคํŠธ ์†์„ฑ bootstrap-status ๋ฐ bootstrap-last-failure์˜ ๊ฐ’์„ ํ™•์ธํ•˜๋Š” ๊ฒƒ์ด ์ข‹์Šต๋‹ˆ๋‹ค. FINISHED ์ด์™ธ์˜ ๋ชจ๋“  bootstrap-status ๊ฐ’์€ Envoy ํ™˜๊ฒฝ์ด ์•„์ง ๊ตฌ์„ฑ๋˜์ง€ ์•Š์•˜์Œ์„ ๋‚˜ํƒ€๋ƒ…๋‹ˆ๋‹ค. bookstrap-last-failure ๊ฐ’์€ ๋ฌธ์ œ๊ฐ€ ๋ฌด์—‡์ธ์ง€ ๋‚˜ํƒ€๋‚ผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

์„œ๋น„์Šค ํ”„๋ก์‹œ๊ฐ€ ์‚ฌ์šฉ ์„ค์ •๋œ ์ธ์Šคํ„ด์Šค ํ…œํ”Œ๋ฆฟ์„ ์‚ฌ์šฉํ•˜์—ฌ ๋งŒ๋“  VM์—์„œ Cloud Service Mesh ์„œ๋น„์Šค์— ์—ฐ๊ฒฐํ•  ์ˆ˜ ์—†์Œ

์ด ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜๋ ค๋ฉด ๋‹ค์Œ ๋‹จ๊ณ„๋ฅผ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค.

  1. VM์— ์„œ๋น„์Šค ํ”„๋ก์‹œ ๊ตฌ์„ฑ์š”์†Œ ์„ค์น˜๊ฐ€ ์™„๋ฃŒ๋˜์ง€ ์•Š์•˜๊ฑฐ๋‚˜ ์‹คํŒจํ–ˆ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ๋ชจ๋“  ๊ตฌ์„ฑ์š”์†Œ๊ฐ€ ์ œ๋Œ€๋กœ ์„ค์น˜๋˜์—ˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

    gcloud compute instances get-guest-attributes INSTANCE_NAME \
        --query-path=gce-service-proxy/ \
        --zone=ZONE
    

    bootstrap-status ๊ฒŒ์ŠคํŠธ ์†์„ฑ์€ ๋‹ค์Œ ์ค‘ ํ•˜๋‚˜๋กœ ์„ค์ •๋ฉ๋‹ˆ๋‹ค.

    • [none]์€ ์„ค์น˜๊ฐ€ ์•„์ง ์‹œ์ž‘๋˜์ง€ ์•Š์•˜์Œ์„ ๋‚˜ํƒ€๋ƒ…๋‹ˆ๋‹ค. VM์ด ์•„์ง ๋ถ€ํŒ… ์ค‘์ผ ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค. ์ž ์‹œ ํ›„ ์ƒํƒœ๋ฅผ ๋‹ค์‹œ ํ™•์ธํ•ด ๋ณด์„ธ์š”.
    • IN PROGRESS๋Š” ์„œ๋น„์Šค ํ”„๋ก์‹œ ๊ตฌ์„ฑ์š”์†Œ์˜ ์„ค์น˜ ๋ฐ ๊ตฌ์„ฑ์ด ์•„์ง ์™„๋ฃŒ๋˜์ง€ ์•Š์•˜์Œ์„ ๋‚˜ํƒ€๋ƒ…๋‹ˆ๋‹ค. ํ”„๋กœ์„ธ์Šค ์—…๋ฐ์ดํŠธ์— ๋Œ€ํ•œ ์ƒํƒœ ํ™•์ธ์„ ๋ฐ˜๋ณตํ•ฉ๋‹ˆ๋‹ค.
    • FAILED๋Š” ๊ตฌ์„ฑ์š”์†Œ ์„ค์น˜ ๋˜๋Š” ๊ตฌ์„ฑ์ด ์‹คํŒจํ–ˆ์Œ์„ ๋‚˜ํƒ€๋ƒ…๋‹ˆ๋‹ค. gce-service-proxy/bootstrap-last-failure1 ์†์„ฑ์„ ์ฟผ๋ฆฌํ•˜์—ฌ ์˜ค๋ฅ˜ ๋ฉ”์‹œ์ง€๋ฅผ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.
    • FINISHED๋Š” ์„ค์น˜ ๋ฐ ๊ตฌ์„ฑ ํ”„๋กœ์„ธ์Šค๊ฐ€ ์˜ค๋ฅ˜ ์—†์ด ์™„๋ฃŒ๋˜์—ˆ์Œ์„ ๋‚˜ํƒ€๋ƒ…๋‹ˆ๋‹ค. ๋‹ค์Œ ์•ˆ๋‚ด์— ๋”ฐ๋ผ ํŠธ๋ž˜ํ”ฝ ๊ฐ€๋กœ์ฑ„๊ธฐ์™€ Envoy ํ”„๋ก์‹œ๊ฐ€ ์˜ฌ๋ฐ”๋ฅด๊ฒŒ ๊ตฌ์„ฑ๋˜์—ˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.
  2. VM์˜ ํŠธ๋ž˜ํ”ฝ ๊ฐ€๋กœ์ฑ„๊ธฐ๊ฐ€ Cloud Service Mesh ๊ธฐ๋ฐ˜ ์„œ๋น„์Šค์— ์˜ฌ๋ฐ”๋ฅด๊ฒŒ ๊ตฌ์„ฑ๋˜์ง€ ์•Š์•˜์Šต๋‹ˆ๋‹ค. VM์— ๋กœ๊ทธ์ธํ•˜๊ณ  iptables ๊ตฌ์„ฑ์„ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

    gcloud compute ssh INSTANCE_NAME \
        --zone=ZONE \
        sudo iptables -L -t nat
    

    ๋‹ค์Œ๊ณผ ๊ฐ™์€ SERVICE_PROXY_REDIRECT ํ•ญ๋ชฉ์˜ SERVICE_PROXY_SERVICE_CIDRS ์ฒด์ธ์„ ์‚ดํŽด๋ด…๋‹ˆ๋‹ค.

    Chain SERVICE_PROXY_SERVICE_CIDRS (1 references)
    target                   prot opt source         destination ...
    SERVICE_PROXY_REDIRECT   all  --  anywhere       10.7.240.0/20
    

    ๊ฐ ์„œ๋น„์Šค๋Š” destination ์—ด์— ์ผ์น˜ํ•˜๋Š” IP ์ฃผ์†Œ๋‚˜ CIDR์ด ์žˆ์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ๊ฐ€์ƒ IP ์ฃผ์†Œ(VIP)์— ๋Œ€ํ•œ ํ•ญ๋ชฉ์ด ์—†๋Š” ๊ฒฝ์šฐ ์ด๋Š” Cloud Service Mesh์—์„œ Envoy ํ”„๋ก์‹œ ๊ตฌ์„ฑ์„ ์ฑ„์šฐ๋Š” ๋ฐ ๋ฌธ์ œ๊ฐ€ ์žˆ๊ฑฐ๋‚˜ VM ๊ธฐ๋ฐ˜ ์—์ด์ „ํŠธ๊ฐ€ ์‹คํŒจํ•œ ๊ฒƒ์ž…๋‹ˆ๋‹ค.

  3. Envoy ํ”„๋ก์‹œ๊ฐ€ ์•„์ง Cloud Service Mesh์—์„œ ๊ตฌ์„ฑ์„ ๋ฐ›์ง€ ๋ชปํ–ˆ์Šต๋‹ˆ๋‹ค. VM์— ๋กœ๊ทธ์ธํ•˜์—ฌ Envoy ํ”„๋ก์‹œ ๊ตฌ์„ฑ์„ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

    gcloud compute ssh INSTANCE_NAME \
        --zone=ZONE \
        sudo curl localhost:15000/config_dump
    

    Cloud Service Mesh์—์„œ ๋ฐ›์€ ๋ฆฌ์Šค๋„ˆ ๊ตฌ์„ฑ์„ ๊ฒ€์‚ฌํ•ฉ๋‹ˆ๋‹ค. ์˜ˆ๋ฅผ ๋“ค๋ฉด ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.

    "dynamic_active_listeners": [
      ...
      "filter_chains": [{
        "filter_chain_match": {
          "prefix_ranges": [{
            "address_prefix": "10.7.240.20",
            "prefix_len": 32
          }],
          "destination_port": 80
        },
      ...
        "route_config_name": "URL_MAP/PROJECT_NUMBER.td-routing-rule-1"
      ...
    ]
    

    address_prefix๋Š” Cloud Service Mesh ์„œ๋น„์Šค์˜ ๊ฐ€์ƒ IP ์ฃผ์†Œ(VIP)์ž…๋‹ˆ๋‹ค. td-routing-rule-1์ด๋ผ๋Š” URL ๋งต์„ ๊ฐ€๋ฆฌํ‚ต๋‹ˆ๋‹ค. ์—ฐ๊ฒฐํ•˜๋ ค๋Š” ์„œ๋น„์Šค๊ฐ€ ๋ฆฌ์Šค๋„ˆ ๊ตฌ์„ฑ์— ์ด๋ฏธ ํฌํ•จ๋˜์–ด ์žˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

  4. VM ๊ธฐ๋ฐ˜ ์—์ด์ „ํŠธ๊ฐ€ ์‹คํ–‰๋˜๊ณ  ์žˆ์ง€ ์•Š์Šต๋‹ˆ๋‹ค. VM ๊ธฐ๋ฐ˜ ์—์ด์ „ํŠธ๋Š” ์ƒˆ Cloud Service Mesh ์„œ๋น„์Šค๊ฐ€ ์ƒ์„ฑ๋  ๋•Œ ํŠธ๋ž˜ํ”ฝ ๊ฐ€๋กœ์ฑ„๊ธฐ๋ฅผ ์ž๋™์œผ๋กœ ๊ตฌ์„ฑํ•ฉ๋‹ˆ๋‹ค. ์—์ด์ „ํŠธ๊ฐ€ ์‹คํ–‰๋˜๊ณ  ์žˆ์ง€ ์•Š์œผ๋ฉด ์ƒˆ ์„œ๋น„์Šค์— ๋Œ€ํ•œ ๋ชจ๋“  ํŠธ๋ž˜ํ”ฝ์ด Envoy ํ”„๋ก์‹œ์™€ ํƒ€์ž„์•„์›ƒ์„ ์šฐํšŒํ•˜์—ฌ VIP๋กœ ๋ฐ”๋กœ ์ „๋‹ฌ๋ฉ๋‹ˆ๋‹ค.

    1. ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‹คํ–‰ํ•˜์—ฌ VM์˜ ์—์ด์ „ํŠธ ์ƒํƒœ๋ฅผ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

      gcloud compute instances get-guest-attributes INSTANCE_NAME \
         --query-path=gce-service-proxy/ \
         --zone=ZONE
      
    2. VM ๊ธฐ๋ฐ˜ ์—์ด์ „ํŠธ ์†์„ฑ์„ ๊ฒ€์‚ฌํ•ฉ๋‹ˆ๋‹ค. agent-heartbeat ์†์„ฑ์˜ ๊ฐ’์—๋Š” ์—์ด์ „ํŠธ๊ฐ€ ๋งˆ์ง€๋ง‰์œผ๋กœ ์ž‘์—… ๋˜๋Š” ํ™•์ธ์„ ์ˆ˜ํ–‰ํ•œ ์‹œ๊ฐ„์ด ์žˆ์Šต๋‹ˆ๋‹ค. ์ด ๊ฐ’์ด 5๋ถ„์„ ์ดˆ๊ณผํ•˜๋ฉด ์—์ด์ „ํŠธ๊ฐ€ ์ค‘๋‹จ๋ฉ๋‹ˆ๋‹ค. ๊ทธ๋Ÿฌ๋ฉด ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ VM์„ ๋‹ค์‹œ ๋งŒ๋“ค์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

      gcloud compute instance-groups managed recreate-instance
      
    3. agent-last-failure ์†์„ฑ์€ ์—์ด์ „ํŠธ์—์„œ ๋งˆ์ง€๋ง‰์œผ๋กœ ๋ฐœ์ƒํ•œ ์˜ค๋ฅ˜๋ฅผ ๋…ธ์ถœํ•ฉ๋‹ˆ๋‹ค. ์ด๋Š” ๋‹ค์Œ ๋ฒˆ์— ์—์ด์ „ํŠธ๊ฐ€ ํ•ด๋‹น ์˜ค๋ฅ˜๊ฐ€ Cannot reach the Cloud Service Mesh API server์ธ์ง€ ํ˜น์€ ์˜๊ตฌ์ ์ธ ์˜ค๋ฅ˜์ธ์ง€ ํ™•์ธํ•  ๋•Œ ํ•ด๊ฒฐ๋˜๋Š” ์ผ์‹œ์ ์ธ ๋ฌธ์ œ์ผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋ช‡ ๋ถ„ ๊ธฐ๋‹ค๋ฆฐ ํ›„์— ์˜ค๋ฅ˜๋ฅผ ๋‹ค์‹œ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

์›Œํฌ๋กœ๋“œ ํฌํŠธ์— ์ธ๋ฐ”์šด๋“œ ํŠธ๋ž˜ํ”ฝ ๊ฐ€๋กœ์ฑ„๊ธฐ๊ฐ€ ๊ตฌ์„ฑ๋˜์–ด ์žˆ์ง€๋งŒ VM ์™ธ๋ถ€์—์„œ๋Š” ํฌํŠธ์— ์—ฐ๊ฒฐํ•  ์ˆ˜ ์—†์Œ

์ด ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜๋ ค๋ฉด ๋‹ค์Œ ๋‹จ๊ณ„๋ฅผ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค.

  1. VM์— ์„œ๋น„์Šค ํ”„๋ก์‹œ ๊ตฌ์„ฑ์š”์†Œ ์„ค์น˜๊ฐ€ ์™„๋ฃŒ๋˜์ง€ ์•Š์•˜๊ฑฐ๋‚˜ ์‹คํŒจํ–ˆ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ๋ชจ๋“  ๊ตฌ์„ฑ์š”์†Œ๊ฐ€ ์ œ๋Œ€๋กœ ์„ค์น˜๋˜์—ˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

    gcloud compute instances get-guest-attributes INSTANCE_NAME \
        --query-path=gce-service-proxy/ \
        --zone=ZONE
    

    bootstrap-status ๊ฒŒ์ŠคํŠธ ์†์„ฑ์€ ๋‹ค์Œ ์ค‘ ํ•˜๋‚˜๋กœ ์„ค์ •๋ฉ๋‹ˆ๋‹ค.

    • [none]์€ ์„ค์น˜๊ฐ€ ์•„์ง ์‹œ์ž‘๋˜์ง€ ์•Š์•˜์Œ์„ ๋‚˜ํƒ€๋ƒ…๋‹ˆ๋‹ค. VM์ด ์•„์ง ๋ถ€ํŒ… ์ค‘์ผ ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค. ์ž ์‹œ ํ›„ ์ƒํƒœ๋ฅผ ๋‹ค์‹œ ํ™•์ธํ•ด ๋ณด์„ธ์š”.
    • IN PROGRESS๋Š” ์„œ๋น„์Šค ํ”„๋ก์‹œ ๊ตฌ์„ฑ์š”์†Œ์˜ ์„ค์น˜ ๋ฐ ๊ตฌ์„ฑ์ด ์•„์ง ์™„๋ฃŒ๋˜์ง€ ์•Š์•˜์Œ์„ ๋‚˜ํƒ€๋ƒ…๋‹ˆ๋‹ค. ํ”„๋กœ์„ธ์Šค ์—…๋ฐ์ดํŠธ์— ๋Œ€ํ•œ ์ƒํƒœ ํ™•์ธ์„ ๋ฐ˜๋ณตํ•ฉ๋‹ˆ๋‹ค.
    • FAILED๋Š” ๊ตฌ์„ฑ์š”์†Œ ์„ค์น˜ ๋˜๋Š” ๊ตฌ์„ฑ์ด ์‹คํŒจํ–ˆ์Œ์„ ๋‚˜ํƒ€๋ƒ…๋‹ˆ๋‹ค. gce-service-proxy/bootstrap-last-failure1 ์†์„ฑ์„ ์ฟผ๋ฆฌํ•˜์—ฌ ์˜ค๋ฅ˜ ๋ฉ”์‹œ์ง€๋ฅผ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.
    • FINISHED๋Š” ์„ค์น˜ ๋ฐ ๊ตฌ์„ฑ ํ”„๋กœ์„ธ์Šค๊ฐ€ ์˜ค๋ฅ˜ ์—†์ด ์™„๋ฃŒ๋˜์—ˆ์Œ์„ ๋‚˜ํƒ€๋ƒ…๋‹ˆ๋‹ค. ๋‹ค์Œ ์•ˆ๋‚ด์— ๋”ฐ๋ผ ํŠธ๋ž˜ํ”ฝ ๊ฐ€๋กœ์ฑ„๊ธฐ์™€ Envoy ํ”„๋ก์‹œ๊ฐ€ ์˜ฌ๋ฐ”๋ฅด๊ฒŒ ๊ตฌ์„ฑ๋˜์—ˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.
  2. VM์˜ ํŠธ๋ž˜ํ”ฝ ๊ฐ€๋กœ์ฑ„๊ธฐ๊ฐ€ ์ธ๋ฐ”์šด๋“œ ํŠธ๋ž˜ํ”ฝ์— ๋งž๊ฒŒ ์˜ฌ๋ฐ”๋ฅด๊ฒŒ ๊ตฌ์„ฑ๋˜์ง€ ์•Š์•˜์Šต๋‹ˆ๋‹ค. VM์— ๋กœ๊ทธ์ธํ•˜๊ณ  iptables ๊ตฌ์„ฑ์„ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

    gcloud compute ssh INSTANCE_NAME \
        --zone=ZONE \
        sudo iptables -L -t nat
    

    ๋‹ค์Œ๊ณผ ๊ฐ™์€ SERVICE_PROXY_IN_REDIRECT ํ•ญ๋ชฉ์˜ SERVICE_PROXY_INBOUND ์ฒด์ธ์„ ์‚ดํŽด๋ด…๋‹ˆ๋‹ค.

    Chain SERVICE_PROXY_INBOUND (1 references)
    target                      prot opt source       destination ...
    SERVICE_PROXY_IN_REDIRECT   tcp  --  anywhere     anywhere  tcp dpt:mysql
    

    service-proxy:serving-ports์— ์ •์˜๋œ ๊ฐ ํฌํŠธ์— ๋Œ€ํ•ด destination ์—ด์— ์ผ์น˜ํ•˜๋Š” ํฌํŠธ๊ฐ€ ์žˆ์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ํฌํŠธ ํ•ญ๋ชฉ์ด ์—†๋Š” ๊ฒฝ์šฐ ๋ชจ๋“  ์ธ๋ฐ”์šด๋“œ ํŠธ๋ž˜ํ”ฝ์€ Envoy ํ”„๋ก์‹œ๋ฅผ ์šฐํšŒํ•˜์—ฌ ์ด ํฌํŠธ๋กœ ์ง์ ‘ ์ด๋™ํ•ฉ๋‹ˆ๋‹ค.

    ์ด ํฌํŠธ ๋˜๋Š” ํŠน์ • ํฌํŠธ๋ฅผ ์ œ์™ธํ•œ ๋ชจ๋“  ํฌํŠธ๋กœ ํŠธ๋ž˜ํ”ฝ์„ ์‚ญ์ œํ•˜๋Š” ๋‹ค๋ฅธ ๊ทœ์น™์ด ์—†๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

  3. Envoy ํ”„๋ก์‹œ๊ฐ€ ์•„์ง Cloud Service Mesh์—์„œ ์ธ๋ฐ”์šด๋“œ ํฌํŠธ ๊ตฌ์„ฑ์„ ๋ฐ›์ง€ ๋ชปํ–ˆ์Šต๋‹ˆ๋‹ค. VM์— ๋กœ๊ทธ์ธํ•˜์—ฌ Envoy ํ”„๋ก์‹œ ๊ตฌ์„ฑ์„ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

    gcloud compute ssh INSTANCE_NAME \
        --zone=ZONE \
        sudo curl localhost:15000/config_dump
    

    Cloud Service Mesh์—์„œ ๋ฐ›์€ ์ธ๋ฐ”์šด๋“œ ๋ฆฌ์Šค๋„ˆ ๊ตฌ์„ฑ์„ ์ฐพ์Šต๋‹ˆ๋‹ค.

    "dynamic_active_listeners": [
      ...
      "filter_chains": [{
        "filter_chain_match": {
          "prefix_ranges": [{
            "address_prefix": "10.0.0.1",
            "prefix_len": 32
          }],
          "destination_port": 80
        },
      ...
        "route_config_name": "inbound|default_inbound_config-80"
      ...
    ]
    

    inbound๋กœ ์‹œ์ž‘ํ•˜๋Š” route_config_name๋Š” ์ธ๋ฐ”์šด๋“œ ํŠธ๋ž˜ํ”ฝ ๊ฐ€๋กœ์ฑ„๊ธฐ์šฉ์œผ๋กœ ์ƒ์„ฑ๋œ ํŠน์ˆ˜ ์„œ๋น„์Šค๋ฅผ ๋‚˜ํƒ€๋ƒ…๋‹ˆ๋‹ค. ์—ฐ๊ฒฐํ•˜๋ ค๋Š” ํฌํŠธ๊ฐ€ destination_port์˜ ๋ฆฌ์Šค๋„ˆ ๊ตฌ์„ฑ์— ์ด๋ฏธ ํฌํ•จ๋˜์–ด ์žˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

์—ฐ๊ฒฐ์—์„œ ์„œ๋ฒ„ ์šฐ์„  ํ”„๋กœํ† ์ฝœ์„ ์‚ฌ์šฉํ•  ๋•Œ ๋ฐœ์ƒํ•˜๋Š” ๋ฌธ์ œ

MySQL๊ณผ ๊ฐ™์€ ์ผ๋ถ€ ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜์€ ์„œ๋ฒ„๊ฐ€ ์ฒซ ๋ฒˆ์งธ ํŒจํ‚ท์„ ์ „์†กํ•˜๋Š” ํ”„๋กœํ† ์ฝœ์„ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ์ฆ‰, ์ตœ์ดˆ ์—ฐ๊ฒฐ ์‹œ์— ์„œ๋ฒ„๋Š” ์ฒซ ๋ฒˆ์งธ ๋ฐ”์ดํŠธ๋ฅผ ๋ณด๋ƒ…๋‹ˆ๋‹ค. ์ด๋Ÿฌํ•œ ํ”„๋กœํ† ์ฝœ๊ณผ ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜์€ Cloud Service Mesh์—์„œ ์ง€์›๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.

์„œ๋น„์Šค ๋ฉ”์‹œ ์ƒํƒœ ๋ฌธ์ œ ํ•ด๊ฒฐ

์ด ๊ฐ€์ด๋“œ์—์„œ๋Š” Cloud Service Mesh ๊ตฌ์„ฑ ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜๋Š” ๋ฐ ๋„์›€์ด ๋˜๋Š” ์ •๋ณด๋ฅผ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.

๋Œ€๋ถ€๋ถ„์˜ ์—”๋“œํฌ์ธํŠธ๊ฐ€ ๋น„์ •์ƒ์ธ ๊ฒฝ์šฐ์˜ Cloud Service Mesh ๋™์ž‘

์—”๋“œํฌ์ธํŠธ์˜ 99%๊ฐ€ ๋น„์ •์ƒ์ด๋ฉด ์‹ ๋ขฐ์„ฑ์„ ๋†’์ด๊ธฐ ์œ„ํ•ด Cloud Service Mesh์—์„œ ๋ฐ์ดํ„ฐ ์˜์—ญ์„ ๊ตฌ์„ฑํ•˜์—ฌ ์—”๋“œํฌ์ธํŠธ์˜ ์ •์ƒ ์ƒํƒœ๋ฅผ ๋ฌด์‹œํ•ฉ๋‹ˆ๋‹ค. ๋Œ€์‹  ๋ฐ์ดํ„ฐ ์˜์—ญ์€ ๋ชจ๋“  ์—”๋“œํฌ์ธํŠธ ๊ฐ„์— ํŠธ๋ž˜ํ”ฝ์„ ๋ถ„์‚ฐํ•˜๋ฉฐ ์ด๋Š” ์„œ๋น™ ํฌํŠธ๊ฐ€ ๊ณ„์† ์ž‘๋™ํ•  ์ˆ˜ ์žˆ๊ธฐ ๋•Œ๋ฌธ์ž…๋‹ˆ๋‹ค.

๋น„์ •์ƒ ๋ฐฑ์—”๋“œ๋กœ ์ธํ•ด ํŠธ๋ž˜ํ”ฝ ๋ถ„์‚ฐ์ด ์ตœ์ ํ™”๋˜์ง€ ์•Š์Œ

Cloud Service Mesh๋Š” ๋ฐฑ์—”๋“œ ์„œ๋น„์Šค์— ์—ฐ๊ฒฐ๋œ HealthCheck ๋ฆฌ์†Œ์Šค์˜ ์ •๋ณด๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ๋ฐฑ์—”๋“œ ์ƒํƒœ๋ฅผ ํ‰๊ฐ€ํ•ฉ๋‹ˆ๋‹ค. Cloud Service Mesh๋Š” ์ด ์ •์ƒ ์ƒํƒœ๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ํŠธ๋ž˜ํ”ฝ์„ ๊ฐ€์žฅ ๊ฐ€๊นŒ์šด ์ •์ƒ ๋ฐฑ์—”๋“œ๋กœ ๋ผ์šฐํŒ…ํ•ฉ๋‹ˆ๋‹ค. ์ผ๋ถ€ ๋ฐฑ์—”๋“œ๊ฐ€ ๋น„์ •์ƒ์ด๋ฉด ํŠธ๋ž˜ํ”ฝ์ด ๊ณ„์† ์ฒ˜๋ฆฌ๋˜๋”๋ผ๋„ ๋ถ„ํฌ๊ฐ€ ์ตœ์ ์ด ์•„๋‹ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์˜ˆ๋ฅผ ๋“ค์–ด ํŠธ๋ž˜ํ”ฝ์ด ์ •์ƒ ๋ฐฑ์—”๋“œ๊ฐ€ ๊ณ„์† ์žˆ๋Š” ๋ฆฌ์ „์— ์ „๋‹ฌ๋  ์ˆ˜ ์žˆ์ง€๋งŒ ํด๋ผ์ด์–ธํŠธ์—์„œ ๋ฉ€๋ฆฌ ๋–จ์–ด์ ธ ์žˆ์–ด ์ง€์—ฐ ์‹œ๊ฐ„์ด ๋ฐœ์ƒํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋ฐฑ์—”๋“œ ์ •์ƒ ์ƒํƒœ๋ฅผ ์‹๋ณ„ํ•˜๊ณ  ๋ชจ๋‹ˆํ„ฐ๋งํ•˜๋ ค๋ฉด ๋‹ค์Œ ๋‹จ๊ณ„๋ฅผ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค.

๋‹ค์Œ ๋‹จ๊ณ„