Google Kubernetes Engine(GKE)์šฉ Eventarc ๋ฌธ์ œ ํ•ด๊ฒฐ

์ด ํŽ˜์ด์ง€์—์„œ๋Š” GKE ํด๋Ÿฌ์Šคํ„ฐ์—์„œ ์‹คํ–‰๋˜๋Š” ๋น„๊ณต๊ฐœ ๋ฐ ๊ณต๊ฐœ ์„œ๋น„์Šค์˜ ๊ณต๊ฐœ ์—”๋“œํฌ์ธํŠธ๋ฅผ ํฌํ•จํ•˜์—ฌ GKE์šฉ Eventarc๋ฅผ ์‚ฌ์šฉํ•  ๋•Œ ๋ฐœ์ƒํ•  ์ˆ˜ ์žˆ๋Š” ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ๋ณด์—ฌ์ค๋‹ˆ๋‹ค.

๋ฐœ์ƒํ•  ์ˆ˜ ์žˆ๋Š” ๋‹ค๋ฅธ ๋ฌธ์ œ๋Š” ๋‹ค์Œ ๋ฌธ์ œ ํ•ด๊ฒฐ ํŽ˜์ด์ง€๋ฅผ ์ฐธ์กฐํ•˜์„ธ์š”.

๋ฆฌ์†Œ์Šค ์ƒ์„ฑ ๊ถŒํ•œ ์˜ค๋ฅ˜๋กœ ํŠธ๋ฆฌ๊ฑฐ ์ƒ์„ฑ ์‹คํŒจ

๋‹ค์Œ ์ค‘ ํ•˜๋‚˜์™€ ์œ ์‚ฌํ•œ ์˜ค๋ฅ˜ ๋ฉ”์‹œ์ง€๊ฐ€ ํ‘œ์‹œ๋ฉ๋‹ˆ๋‹ค.

Error applying IAM policy for service account 'projects/PROJECT_ID/serviceAccounts/SERVICE_ACCOUNT':
Error setting IAM policy for service account 'projects/PROJECT_ID/serviceAccounts/SERVICE_ACCOUNT':
googleapi: Error 403: Permission iam.serviceAccounts.setIamPolicy is required to perform this
operation on service account SERVICE_ACCOUNT., forbidden

Error when reading or editing Container Cluster "cluster": googleapi:
Error 403: Required "container.clusters.get" permission(s) for "..."., forbidden
Error reading instance group manager returned as an instance group URL:
"googleapi: Error 403: Required 'compute.instanceGroupManagers.get' permission
for '...', forbidden"

์ด ์˜ค๋ฅ˜๋Š” Eventarc์—์„œ Google Kubernetes Engine(GKE) ํด๋Ÿฌ์Šคํ„ฐ์˜ ๋ฆฌ์†Œ์Šค๋ฅผ ๊ด€๋ฆฌํ•˜๋Š” ๋ฐ ํ•„์š”ํ•œ Eventarc ์„œ๋น„์Šค ๊ณ„์ •์ด ์˜ฌ๋ฐ”๋ฅด๊ฒŒ ์„ค์ •๋˜์ง€ ์•Š์€ ๊ฒฝ์šฐ์— ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค.

์ด ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜๋ ค๋ฉด Eventarc ์„œ๋น„์Šค ๊ณ„์ •์ด ๋ฆฌ์†Œ์Šค๋ฅผ ์ƒ์„ฑํ•  ์ˆ˜ ์žˆ๋Š” ์ ์ ˆํ•œ ๊ถŒํ•œ์œผ๋กœ ์˜ฌ๋ฐ”๋ฅด๊ฒŒ ๊ตฌ์„ฑ๋˜์—ˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค. ์ž์„ธํ•œ ๋‚ด์šฉ์€ ํŠน์ • ์ œ๊ณต์—…์ฒด, ์ด๋ฒคํŠธ ์œ ํ˜•, GKE ๋Œ€์ƒ์— ๋Œ€ํ•œ ์•ˆ๋‚ด๋ฅผ ๋”ฐ๋ฅด์„ธ์š”.

๋Œ€์ƒ ํด๋Ÿฌ์Šคํ„ฐ๋ฅผ ์ฐพ์„ ์ˆ˜ ์—†๊ฑฐ๋‚˜ GKE์šฉ ์›Œํฌ๋กœ๋“œ ์•„์ด๋ดํ‹ฐํ‹ฐ ์ œํœด๊ฐ€ ์‚ฌ์šฉ ์„ค์ •๋˜์–ด ์žˆ์ง€ ์•Š์•„ ํŠธ๋ฆฌ๊ฑฐ ์ƒ์„ฑ์ด ์‹คํŒจํ•จ

๋‹ค์Œ ์ค‘ ํ•˜๋‚˜์™€ ์œ ์‚ฌํ•œ ์˜ค๋ฅ˜ ๋ฉ”์‹œ์ง€๊ฐ€ ํ‘œ์‹œ๋ฉ๋‹ˆ๋‹ค.

ERROR: (gcloud.eventarc.triggers.create) NOT_FOUND: Not found: projects/PROJECT_ID/locations/
LOCATION/clusters/CLUSTER_NAME.
...
ERROR: (gcloud.eventarc.triggers.create) INVALID_ARGUMENT: workload identity is not enabled on
cluster CLUSTER_NAME in LOCATION
...

์ด ์˜ค๋ฅ˜๋Š” Eventarc์—์„œ ๋Œ€์ƒ GKE ํด๋Ÿฌ์Šคํ„ฐ๋ฅผ ์ฐพ์„ ์ˆ˜ ์—†๊ฑฐ๋‚˜ ํด๋Ÿฌ์Šคํ„ฐ์— GKE์šฉ ์›Œํฌ๋กœ๋“œ ์•„์ด๋ดํ‹ฐํ‹ฐ ์ œํœด๊ฐ€ ์‚ฌ์šฉ ์„ค์ •๋˜์ง€ ์•Š์€ ๊ฒฝ์šฐ ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค.

์ด ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜๋ ค๋ฉด ๋Œ€์ƒ ํด๋Ÿฌ์Šคํ„ฐ๊ฐ€ ์กด์žฌํ•˜๊ณ  ์›Œํฌ๋กœ๋“œ ์•„์ด๋ดํ‹ฐํ‹ฐ๊ฐ€ ์‚ฌ์šฉ ์„ค์ •๋˜์–ด ์žˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

์ž˜๋ชป๋œ ๋ฆฌ์†Œ์Šค ์ƒํƒœ๋กœ ์ธํ•ด ํŠธ๋ฆฌ๊ฑฐ ์ƒ์„ฑ ์‹คํŒจ

ํ”„๋กœ์ ํŠธ์— ๋‹ค์Œ๊ณผ ์œ ์‚ฌํ•œ ์˜ค๋ฅ˜ ๋ฉ”์‹œ์ง€๊ฐ€ ํ‘œ์‹œ๋ฉ๋‹ˆ๋‹ค.

ERROR: (gcloud.eventarc.triggers.create) Invalid resource state for projects/PROJECT_ID/locations/LOCATION/triggers/TRIGGER_ID.
Ensure that you have enabled your GKE destination and assigned required permissions to the service account.
For more information, see https://cloud.google.com/eventarc/standard/docs/gke/troubleshooting.

์ด ์˜ค๋ฅ˜๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์€ ๊ฒฝ์šฐ์— ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค.

  • GKE ๋Œ€์ƒ์ด ์‚ฌ์šฉ ์„ค์ •๋˜์ง€ ์•Š์€ ๊ฒฝ์šฐ
  • Eventarc์—์„œ GKE ๋Œ€์ƒ์˜ ์ด๋ฒคํŠธ๋ฅผ ๊ด€๋ฆฌํ•˜๋Š” ๋ฐ ํ•„์š”ํ•œ ์—ญํ•  ๋ฐ ๊ถŒํ•œ์ด ์„œ๋น„์Šค ๊ณ„์ •์— ๋ถ€์—ฌ๋˜์ง€ ์•Š์€ ๊ฒฝ์šฐ

์ด ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜๋ ค๋ฉด ๋‹ค์Œ ๋‹จ๊ณ„๋ฅผ ๋”ฐ๋ฅด์„ธ์š”.

  1. GKE ๋Œ€์ƒ์„ ์‚ฌ์šฉ ์„ค์ •ํ–ˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.
    gcloud eventarc gke-destinations init
    ์ž์„ธํ•œ ๋‚ด์šฉ์€ "GKE ๋Œ€์ƒ ์‚ฌ์šฉ ์„ค์ •" ์„น์…˜์—์„œ ํŠน์ • ์ œ๊ณต์—…์ฒด ๋ฐ ์ด๋ฒคํŠธ ์œ ํ˜•์— ๋Œ€ํ•œ ์•ˆ๋‚ด๋ฅผ ๋”ฐ๋ฅด์„ธ์š”.
  2. Eventarc ์„œ๋น„์Šค ๊ณ„์ •์ด ๋ฆฌ์†Œ์Šค๋ฅผ ๋งŒ๋“ค ์ˆ˜ ์žˆ๋Š” ์ ์ ˆํ•œ ๊ถŒํ•œ์œผ๋กœ ๊ตฌ์„ฑ๋˜์—ˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค. ์„œ๋น„์Šค ๊ณ„์ •์— ์ ์ ˆํ•œ ์—ญํ• ์„ ๋ถ€์—ฌํ•˜๋Š” ๋ฐฉ๋ฒ•์€ "ํŠธ๋ฆฌ๊ฑฐ ์ƒ์„ฑ ์ค€๋น„" ์„น์…˜์˜ ํŠน์ • ์ œ๊ณต์—…์ฒด ๋ฐ ์ด๋ฒคํŠธ ์œ ํ˜•์— ๋Œ€ํ•œ ์•ˆ๋‚ด๋ฅผ ๋”ฐ๋ฅด์„ธ์š”.
  3. ์˜ค๋ฅ˜๊ฐ€ ๊ณ„์†๋˜๋ฉด ์ง€์›ํŒ€์— ๋ฌธ์˜ํ•˜์„ธ์š”.

ํŠธ๋ฆฌ๊ฑฐ๊ฐ€ ์ƒ์„ฑ๋˜์—ˆ์ง€๋งŒ ๋Œ€์ƒ์—์„œ ์ด๋ฒคํŠธ๋ฅผ ์ˆ˜์‹ ํ•˜์ง€ ์•Š์Œ

ํŠธ๋ฆฌ๊ฑฐ๋ฅผ ๋งŒ๋“  ํ›„ ์ถฉ๋ถ„ํ•œ ์‹œ๊ฐ„์ด ๊ฒฝ๊ณผํ–ˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค. ์ด๋ฒคํŠธ๊ฐ€ ์ „์†ก๋˜๋Š” ๋ฐ ์ตœ๋Œ€ 2๋ถ„์ด ๊ฑธ๋ฆด ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

  1. ์ด๋ฒคํŠธ๊ฐ€ ๋Œ€์ƒ์—์„œ ์ „์†ก๋˜์ง€ ์•Š์œผ๋ฉด Pub/Sub์—์„œ ๋Œ€์ƒ์œผ๋กœ ์ „์†ก๋œ ์ด๋ฒคํŠธ๊ฐ€ ์‚ญ์ œ๋  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

    ์ด๋ฒคํŠธ๊ฐ€ ์‚ญ์ œ๋˜์ง€ ์•Š๋„๋ก Pub/Sub ๊ตฌ๋… ์žฌ์‹œ๋„ ์ •์ฑ…์„ ์„ค์ •ํ•˜๊ฑฐ๋‚˜ ์ „์†ก๋˜์ง€ ์•Š์€ ๋ฉ”์‹œ์ง€๋ฅผ ๋ฐ๋“œ ๋ ˆํ„ฐ ์ฃผ์ œ(๋ฐ๋“œ ๋ ˆํ„ฐ ํ๋ผ๊ณ ๋„ ํ•จ)๋กœ ์ „๋‹ฌํ•ฉ๋‹ˆ๋‹ค.

    ๋ฐ๋“œ ๋ ˆํ„ฐ ์ฃผ์ œ๋ฅผ ์„ค์ •ํ•˜๊ธฐ ์ „์— ํŠธ๋ฆฌ๊ฑฐ์˜ ์ฃผ์ œ ๋ฐ ๊ตฌ๋…์„ ๊ฒ€์ƒ‰ํ•ฉ๋‹ˆ๋‹ค.

    gcloud eventarc triggers describe TRIGGER \
      --location=LOCATION

    ๋‹ค์Œ์„ ๋ฐ”๊ฟ‰๋‹ˆ๋‹ค.

    • TRIGGER: ํŠธ๋ฆฌ๊ฑฐ์˜ ID ๋˜๋Š” ์ •๊ทœํ™”๋œ ์‹๋ณ„์ž
    • LOCATION: Eventarc ํŠธ๋ฆฌ๊ฑฐ ์œ„์น˜
  2. ํŠธ๋ฆฌ๊ฑฐ๊ฐ€ ๊ณ„์† ์ž‘๋™ํ•˜์ง€ ์•Š์œผ๋ฉด Eventarc ์ด๋ฒคํŠธ ์ „๋‹ฌ์ž ๊ตฌ์„ฑ์š”์†Œ์˜ ์ƒํƒœ๋ฅผ ํ™•์ธํ•˜์—ฌ ์˜ฌ๋ฐ”๋ฅด๊ฒŒ ๋ฐฐํฌ๋˜์—ˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

    1. ์ด๋ฆ„์— event-forwarder๊ฐ€ ์žˆ๋Š” ๋ชจ๋“  Pod๋ฅผ ๋‚˜์—ดํ•˜์—ฌ ํด๋Ÿฌ์Šคํ„ฐ ๋„ค์ž„์ŠคํŽ˜์ด์Šค๋ฅผ ์‹๋ณ„ํ•ฉ๋‹ˆ๋‹ค.
      kubectl get pods --all-namespaces | grep event-forwarder
      ์ถœ๋ ฅ์€ ๋‹ค์Œ๊ณผ ๋น„์Šทํ•ฉ๋‹ˆ๋‹ค.
      event-forwarder-namespace-hash1   event-forwarder-hash1   1/1   Running   0   5d2h
      event-forwarder-namespace-hash2   event-forwarder-hash2   1/1   Running   0   2d2h
      
      ๋„ค์ž„์ŠคํŽ˜์ด์Šค์˜ ์ฒ˜์Œ 30์ž(์˜๋ฌธ ๊ธฐ์ค€)๋Š” ์‚ฌ์šฉ์ž๊ฐ€ ์ƒ์„ฑํ•œ Eventarc ํŠธ๋ฆฌ๊ฑฐ์˜ ์ด๋ฆ„๊ณผ ์ผ์น˜ํ•˜๋ฉฐ ๊ทธ ๋’ค์— ํ•ด์‹œ๊ฐ€ ์ถ”๊ฐ€๋˜์–ด ์ฒ˜์Œ 30์ž๊ฐ€ ๋™์ผํ•œ ํŠธ๋ฆฌ๊ฑฐ์™€ ๊ตฌ๋ณ„๋ฉ๋‹ˆ๋‹ค.
    2. ๋ฐฐํฌ ์„ธ๋ถ€์ •๋ณด๋ฅผ ๊ฒ€์ƒ‰ํ•ฉ๋‹ˆ๋‹ค.
      kubectl get deployments -n EVENT_FORWARDER_NAMESPACE event-forwarder
    3. ํฌ๋“œ ์„ธ๋ถ€์ •๋ณด๋ฅผ ๊ฒ€์ƒ‰ํ•ฉ๋‹ˆ๋‹ค.
      kubectl describe pod -n EVENT_FORWARDER_NAMESPACE EVENT_FORWARDER_HASH
    4. ํฌ๋“œ ๋กœ๊ทธ๋ฅผ ์ถœ๋ ฅํ•˜์—ฌ ์˜ค๋ฅ˜๊ฐ€ ๋ฐœ์ƒํ•œ ์ด์œ ๋‚˜ ์œ„์น˜๋ฅผ ์กฐ์‚ฌํ•ฉ๋‹ˆ๋‹ค.
      kubectl logs -f -n EVENT_FORWARDER_NAMESPACE EVENT_FORWARDER_HASH
      ์˜ˆ๋ฅผ ๋“ค์–ด ๋‹ค์Œ ๋กœ๊ทธ์—์„œ ์„œ๋น„์Šค ๊ณ„์ •์— Pub/Sub Subscriber ์—ญํ• ์ด ์ทจ์†Œ๋˜์—ˆ์Šต๋‹ˆ๋‹ค.
      {"error":"generic::permission_denied: missing 'pubsub.subscriptions.consume' permission",
      "message":"Permissions check failed","severity":"fatal","timestamp":"2021-07-07T14:57:49.038877483Z"}
      
  • GKE์šฉ Eventarc์— ๋Œ€ํ•ด ์ž์„ธํžˆ ์•Œ์•„๋ณด์„ธ์š”.
  • ์ถ”๊ฐ€ ๋„์›€์„ ๋ฐ›์œผ๋ ค๋ฉด ์ง€์›์„ ์ฐธ์กฐํ•˜์„ธ์š”.