ํƒ€์‚ฌ ๋ ˆ์ง€์ŠคํŠธ๋ฆฌ์—์„œ ์ปจํ…Œ์ด๋„ˆ ๋งˆ์ด๊ทธ๋ ˆ์ด์…˜

ํƒ€์‚ฌ ๋ ˆ์ง€์ŠคํŠธ๋ฆฌ์—์„œ ์ปจํ…Œ์ด๋„ˆ ์ด๋ฏธ์ง€ ์ผ๋ถ€๋ฅผ ์ง์ ‘ ๊ฐ€์ ธ์™€ Google Kubernetes Engine ๋˜๋Š” Cloud Run๊ณผ ๊ฐ™์€ Google Cloud ํ™˜๊ฒฝ์— ๋ฐฐํฌํ•˜๋ฉด ์ด๋ฏธ์ง€ ๊ฐ€์ ธ์˜ค๊ธฐ ๋˜๋Š” ํƒ€์‚ฌ ์„œ๋น„์Šค ์ค‘๋‹จ์— ๋Œ€ํ•œ ๋น„์œจ ์ œํ•œ์— ๋”ฐ๋ผ ๋นŒ๋“œ์™€ ๋ฐฐํฌ๊ฐ€ ์ค‘๋‹จ๋  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด ํŽ˜์ด์ง€์—์„œ๋Š” ํ†ตํ•ฉ์ ์ด๊ณ  ์ผ๊ด€์ ์ธ ์ปจํ…Œ์ด๋„ˆ ์ด๋ฏธ์ง€ ๊ด€๋ฆฌ๋ฅผ ์œ„ํ•ด ์ด๋ฏธ์ง€๋ฅผ ํ†ตํ•ฉํ•ด์„œ Google Cloud์˜ ๋ ˆ์ง€์ŠคํŠธ๋ฆฌ์— ๋ณต์‚ฌํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ์„ค๋ช…ํ•ฉ๋‹ˆ๋‹ค.

๋˜ํ•œ ์ทจ์•ฝ์  ์Šค์บ”์„ ์‚ฌ์šฉํ•œ ์†Œํ”„ํŠธ์›จ์–ด ๊ณต๊ธ‰๋ง ๋ณด์•ˆ ๋ฐ Binary Authorization์„ ์‚ฌ์šฉํ•œ ๋ฐฐํฌ ์ •์ฑ… ์ ์šฉ์„ ํฌํ•จํ•œ ๊ธฐํƒ€ ๊ธฐ๋Šฅ์„ ํ™œ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋ ˆ์ง€์ŠคํŠธ๋ฆฌ ์„ ํƒ

Artifact Registry๋Š” Google Cloud์—์„œ ์ปจํ…Œ์ด๋„ˆ ์ด๋ฏธ์ง€ ๋ฐ ๊ธฐํƒ€ ๋นŒ๋“œ ์•„ํ‹ฐํŒฉํŠธ ์ €์žฅ ๋ฐ ๊ด€๋ฆฌ๋ฅผ ์œ„ํ•ด ๊ถŒ์žฅ๋˜๋Š” ์„œ๋น„์Šค์ž…๋‹ˆ๋‹ค.

  • Container Registry๋ฅผ ์‚ฌ์šฉํ•˜์ง€ ์•Š๋Š” ๊ฒฝ์šฐ ๋Œ€์‹  ์ด๋ฏธ์ง€๋ฅผ Artifact Registry๋กœ ๋งˆ์ด๊ทธ๋ ˆ์ด์…˜ํ•˜์„ธ์š”. Artifact Registry๋Š” ๋ฉ€ํ‹ฐ ๋ฆฌ์ „ ๋Œ€์‹  ๋‹จ์ผ ๋ฆฌ์ „์— ์ด๋ฏธ์ง€ ์ €์žฅ, ๋ณด๋‹ค ์„ธ๋ถ€์ ์ธ ์•ก์„ธ์Šค ์ œ์–ด, ๊ธฐํƒ€ ์•„ํ‹ฐํŒฉํŠธ ํ˜•์‹ ์ง€์›์„ ํฌํ•จํ•˜์—ฌ ๋›ฐ์–ด๋‚œ ์œ ์—ฐ์„ฑ๊ณผ ์ œ์–ด ๊ธฐ๋Šฅ์„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.
  • Container Registry๋ฅผ ์‚ฌ์šฉํ•˜๋Š” ๊ฒฝ์šฐ Artifact Registry๋กœ ์ „ํ™˜ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋งˆ์ด๊ทธ๋ ˆ์ด์…˜ ๊ฐœ์š”

์ปจํ…Œ์ด๋„ˆ ์ด๋ฏธ์ง€ ๋งˆ์ด๊ทธ๋ ˆ์ด์…˜์—๋Š” ๋‹ค์Œ ๋‹จ๊ณ„๊ฐ€ ํฌํ•จ๋ฉ๋‹ˆ๋‹ค.

  1. ๊ธฐ๋ณธ ์š”๊ฑด์„ ์„ค์ •ํ•ฉ๋‹ˆ๋‹ค.
  2. ๋งˆ์ด๊ทธ๋ ˆ์ด์…˜ํ•  ์ด๋ฏธ์ง€๋ฅผ ์‹๋ณ„ํ•ฉ๋‹ˆ๋‹ค.
    • ํƒ€์‚ฌ ๋ ˆ์ง€์ŠคํŠธ๋ฆฌ์— ๋Œ€ํ•œ ์ฐธ์กฐ๋กœ Dockerfile ํŒŒ์ผ๊ณผ ๋ฐฐํฌ ๋งค๋‹ˆํŽ˜์ŠคํŠธ ๊ฒ€์ƒ‰
    • Cloud Logging ๋ฐ BigQuery๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ํƒ€์‚ฌ ๋ ˆ์ง€์ŠคํŠธ๋ฆฌ์—์„œ ์ด๋ฏธ์ง€ ๊ฐ€์ ธ์˜ค๊ธฐ ๋นˆ๋„๋ฅผ ๊ฒฐ์ •ํ•ฉ๋‹ˆ๋‹ค.
  3. Container Registry์— ์‹๋ณ„๋œ ์ด๋ฏธ์ง€๋ฅผ ๋ณต์‚ฌํ•ฉ๋‹ˆ๋‹ค.
  4. ๋ ˆ์ง€์ŠคํŠธ๋ฆฌ์— ๋Œ€ํ•œ ๊ถŒํ•œ์ด ์˜ฌ๋ฐ”๋ฅด๊ฒŒ ๊ตฌ์„ฑ๋˜์—ˆ๋Š”์ง€ ํŠนํžˆ Container Registry์™€ Google Cloud ๋ฐฐํฌ ํ™˜๊ฒฝ์ด ๋‹ค๋ฅธ ํ”„๋กœ์ ํŠธ์— ์žˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.
  5. ๋ฐฐํฌ ๋งค๋‹ˆํŽ˜์ŠคํŠธ๋ฅผ ์—…๋ฐ์ดํŠธํ•ฉ๋‹ˆ๋‹ค.
  6. ์›Œํฌ๋กœ๋“œ๋ฅผ ๋‹ค์‹œ ๋ฐฐํฌํ•ฉ๋‹ˆ๋‹ค.
  7. (์„ ํƒ์‚ฌํ•ญ) ํƒ€์‚ฌ ์†Œ์Šค์˜ ์ด๋ฏธ์ง€ ๋ฐฐํฌ๋ฅผ ์ฐจ๋‹จํ•ฉ๋‹ˆ๋‹ค.

Container Registry๋Š” Container Registry๋กœ ๋ณต์‚ฌํ•˜๋Š” ์ด๋ฏธ์ง€ ์—…๋ฐ์ดํŠธ์— ์‚ฌ์šฉ๋˜๋Š” ํƒ€์‚ฌ ๋ ˆ์ง€์ŠคํŠธ๋ฆฌ๋ฅผ ๋ชจ๋‹ˆํ„ฐ๋งํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ์ตœ์‹  ๋ฒ„์ „์˜ ์ด๋ฏธ์ง€๋ฅผ ํŒŒ์ดํ”„๋ผ์ธ์— ํ†ตํ•ฉํ•˜๋ ค๋ฉด Container Registry๋กœ ํ‘ธ์‹œํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

์‹œ์ž‘ํ•˜๊ธฐ ์ „์—

  1. ๊ถŒํ•œ์„ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค. ์ด๋ฏธ์ง€๋ฅผ Container Registry๋กœ ๋งˆ์ด๊ทธ๋ ˆ์ด์…˜ํ•˜๋ ค๋Š” ํ”„๋กœ์ ํŠธ์— ์†Œ์œ ์ž ๋˜๋Š” ํŽธ์ง‘์ž IAM ์—ญํ• ์ด ์žˆ์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.
  2. ํ”„๋กœ์ ํŠธ ์„ ํƒ๊ธฐ ํŽ˜์ด์ง€๋กœ ์ด๋™

    1. Container Registry๋ฅผ ์‚ฌ์šฉํ•  Google Cloud ํ”„๋กœ์ ํŠธ๋ฅผ ์„ ํƒํ•ฉ๋‹ˆ๋‹ค.
    2. Google Cloud ์ฝ˜์†”์—์„œ Cloud Shell๋กœ ์ด๋™ํ•ฉ๋‹ˆ๋‹ค.
    3. ํ”„๋กœ์ ํŠธ ID๋ฅผ ์ฐพ์•„์„œ Cloud Shell์—์„œ ์„ค์ •ํ•ฉ๋‹ˆ๋‹ค. YOUR_PROJECT_ID๋ฅผ ํ”„๋กœ์ ํŠธ ID๋กœ ๋ฐ”๊ฟ‰๋‹ˆ๋‹ค.

      gcloud config set project YOUR_PROJECT_ID
      
  3. ๋‹ค์Œ ํ™˜๊ฒฝ ๋ณ€์ˆ˜๋ฅผ ๋‚ด๋ณด๋ƒ…๋‹ˆ๋‹ค.

      export PROJECT=$(gcloud config get-value project)
    
  4. ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ BigQuery, Container Registry, Cloud Monitoring API๋ฅผ ์‚ฌ์šฉ ์„ค์ •ํ•ฉ๋‹ˆ๋‹ค.

    gcloud services enable \
    containerregistry.googleapis.com \
    stackdriver.googleapis.com \
    logging.googleapis.com \
    monitoring.googleapis.com
    
  5. Go ๋ฒ„์ „ 1.13 ์ด์ƒ์ด ์„ค์น˜๋˜์–ด ์žˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

    • ๋‹ค์Œ ๋ช…๋ น์–ด๋กœ ๊ธฐ์กด Go ์„ค์น˜ ๋ฒ„์ „์„ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

      go version
      
    • Go๋ฅผ ์„ค์น˜ํ•˜๊ฑฐ๋‚˜ ์—…๋ฐ์ดํŠธํ•ด์•ผ ํ•˜๋Š” ๊ฒฝ์šฐ Go ์„ค์น˜ ๋ฌธ์„œ๋ฅผ ์ฐธ์กฐํ•˜์„ธ์š”.

๋น„์šฉ

์ด ๊ฐ€์ด๋“œ์—์„œ๋Š” ๋น„์šฉ์ด ์ฒญ๊ตฌ๋  ์ˆ˜ ์žˆ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์€ Google Cloud ๊ตฌ์„ฑ์š”์†Œ๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

๋งˆ์ด๊ทธ๋ ˆ์ด์…˜ํ•  ์ด๋ฏธ์ง€ ์‹๋ณ„

ํƒ€์‚ฌ ๋ ˆ์ง€์ŠคํŠธ๋ฆฌ์— ๋Œ€ํ•œ ์ฐธ์กฐ์šฉ ์ปจํ…Œ์ด๋„ˆ ์ด๋ฏธ์ง€๋ฅผ ๋นŒ๋“œ ๋ฐ ๋ฐฐํฌํ•˜๋Š” ๋ฐ ์‚ฌ์šฉํ•œ ํŒŒ์ผ์„ ๊ฒ€์ƒ‰ํ•œ ํ›„ ์ด๋ฏธ์ง€๋ฅผ ๊ฐ€์ ธ์˜ค๋Š” ๋นˆ๋„๋ฅผ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

Dockerfile์—์„œ ์ฐธ์กฐ ์‹๋ณ„

Dockerfile์ด ์ €์žฅ๋œ ์œ„์น˜์—์„œ ์ด ๋‹จ๊ณ„๋ฅผ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค. ํŒŒ์ผ์„ VM์—์„œ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ๋Š” ๊ฒฝ์šฐ ์ฝ”๋“œ๊ฐ€ ๋กœ์ปฌ์—์„œ ์ฒดํฌ์•„์›ƒ๋˜๊ฑฐ๋‚˜ Cloud Shell์— ์žˆ๋Š” ์œ„์น˜์ผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

Dockerfile์ด ์žˆ๋Š” ๋””๋ ‰ํ„ฐ๋ฆฌ์—์„œ ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค.

grep -inr -H --include Dockerfile\* "FROM" . | grep -i -v -E 'docker.pkg.dev|gcr.io'

์ถœ๋ ฅ์€ ๋‹ค์Œ ์˜ˆ์‹œ์™€ ๊ฐ™์ด ํ‘œ์‹œ๋ฉ๋‹ˆ๋‹ค.

./code/build/baseimage/Dockerfile:1:FROM debian:stretch
./code/build/ubuntubase/Dockerfile:1:FROM ubuntu:latest
./code/build/pythonbase/Dockerfile:1:FROM python:3.5-buster

์ด ๋ช…๋ น์–ด๋Š” ๋””๋ ‰ํ„ฐ๋ฆฌ์˜ ๋ชจ๋“  Dockerfile์„ ๊ฒ€์ƒ‰ํ•˜๊ณ  "FROM" ์ค„์„ ์‹๋ณ„ํ•ฉ๋‹ˆ๋‹ค. Dockerfile์„ ์ €์žฅํ•˜๋Š” ๋ฐฉ๋ฒ•๊ณผ ์ผ์น˜ํ•˜๋„๋ก ๋ช…๋ น์–ด๋ฅผ ์กฐ์ •ํ•ฉ๋‹ˆ๋‹ค.

๋งค๋‹ˆํŽ˜์ŠคํŠธ์—์„œ ์ฐธ์กฐ ์‹๋ณ„

GKE ๋˜๋Š” Cloud Run ๋งค๋‹ˆํŽ˜์ŠคํŠธ๊ฐ€ ์ €์žฅ๋˜๋Š” ์œ„์น˜์—์„œ ์ด ๋‹จ๊ณ„๋ฅผ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค. ํŒŒ์ผ์„ VM์—์„œ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ๋Š” ๊ฒฝ์šฐ ์ฝ”๋“œ๊ฐ€ ๋กœ์ปฌ์—์„œ ์ฒดํฌ์•„์›ƒ๋˜๊ฑฐ๋‚˜ Cloud Shell์— ์žˆ๋Š” ์œ„์น˜์ผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

  1. GKE ๋˜๋Š” Cloud Run ๋งค๋‹ˆํŽ˜์ŠคํŠธ๊ฐ€ ์žˆ๋Š” ๋””๋ ‰ํ„ฐ๋ฆฌ์—์„œ ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค.

    grep -inr -H --include \*.yaml "image:" . | grep -i -v -E 'docker.pkg.dev|gcr.io'
    

    ์ƒ˜ํ”Œ ๊ฒฐ๊ณผ:

    ./code/deploy/k8s/ubuntu16-04.yaml:63: image: busybox:1.31.1-uclibc
    ./code/deploy/k8s/master.yaml:26:      image: kubernetes/redis:v1
    

    ์ด ๋ช…๋ น์–ด๋Š” ๋””๋ ‰ํ„ฐ๋ฆฌ์˜ ๋ชจ๋“  YAML ํŒŒ์ผ์„ ํ™•์ธํ•˜๊ณ  image: ์ค„์„ ์‹๋ณ„ํ•œ ํ›„ ๋งค๋‹ˆํŽ˜์ŠคํŠธ๊ฐ€ ์ €์žฅ๋˜๋Š” ๋ฐฉ์‹์— ๋”ฐ๋ผ ์กฐ์ •ํ•ฉ๋‹ˆ๋‹ค.

  2. ํ˜„์žฌ ํด๋Ÿฌ์Šคํ„ฐ์—์„œ ์‹คํ–‰ ์ค‘์ธ ์ด๋ฏธ์ง€๋ฅผ ๋‚˜์—ดํ•˜๋ ค๋ฉด ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค.

      kubectl get all --all-namespaces -o yaml | grep image: | grep -i -v -E 'docker.pkg.dev|gcr.io'
    

    ์ด ๋ช…๋ น์–ด๋Š” ํ˜„์žฌ ์„ ํƒ๋œ Kubernetes ํด๋Ÿฌ์Šคํ„ฐ์—์„œ ์‹คํ–‰ ์ค‘์ธ ๋ชจ๋“  ๊ฐ์ฒด๋ฅผ ๋ฐ˜ํ™˜ํ•˜๊ณ  ์ด๋ฏธ์ง€ ์ด๋ฆ„์„ ๊ฐ€์ ธ์˜ต๋‹ˆ๋‹ค.

    ์ƒ˜ํ”Œ ๊ฒฐ๊ณผ:

    - image: nginx
      image: nginx:latest
        - image: nginx
        - image: nginx
    

์ „์ฒด Google Cloud ํ”„๋กœ์ ํŠธ์˜ ๋ชจ๋“  GKE ํด๋Ÿฌ์Šคํ„ฐ์— ์ด ๋ช…๋ น์–ด๋ฅผ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค.

ํƒ€์‚ฌ ๋ ˆ์ง€์ŠคํŠธ๋ฆฌ์—์„œ ๊ฐ€์ ธ์˜ค๊ธฐ ๋นˆ๋„ ์‹๋ณ„

ํƒ€์‚ฌ ๋ ˆ์ง€์ŠคํŠธ๋ฆฌ์—์„œ ๊ฐ€์ ธ์˜ค๋Š” ํ”„๋กœ์ ํŠธ์—์„œ ์ด๋ฏธ์ง€ ๊ฐ€์ ธ์˜ค๊ธฐ ๋นˆ๋„์— ๋Œ€ํ•œ ์ •๋ณด๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ์‚ฌ์šฉ๋Ÿ‰์ด ํƒ€์‚ฌ ๋ ˆ์ง€์ŠคํŠธ๋ฆฌ์—์„œ ์ ์šฉํ•˜๋Š” ๋น„์œจ ์ œํ•œ์— ๊ทผ์ ‘ํ•˜๊ฑฐ๋‚˜ ์ด๋ฅผ ์ดˆ๊ณผํ•˜๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

๋กœ๊ทธ ๋ฐ์ดํ„ฐ ์ˆ˜์ง‘

๋กœ๊ทธ ์‹ฑํฌ๋ฅผ ๋งŒ๋“ค์–ด BigQuery๋กœ ๋ฐ์ดํ„ฐ๋ฅผ ๋‚ด๋ณด๋ƒ…๋‹ˆ๋‹ค. ๋กœ๊ทธ ์‹ฑํฌ์—๋Š” ๋‚ด๋ณด๋‚ผ ๋กœ๊ทธ ํ•ญ๋ชฉ์„ ์„ ํƒํ•˜๋Š” ๋Œ€์ƒ๊ณผ ์ฟผ๋ฆฌ๊ฐ€ ํฌํ•จ๋ฉ๋‹ˆ๋‹ค. ๊ฐœ๋ณ„ ํ”„๋กœ์ ํŠธ๋ฅผ ์ฟผ๋ฆฌํ•˜์—ฌ ์‹ฑํฌ๋ฅผ ๋งŒ๋“ค๊ฑฐ๋‚˜ ์Šคํฌ๋ฆฝํŠธ๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ํ”„๋กœ์ ํŠธ ๊ฐ„์— ๋ฐ์ดํ„ฐ๋ฅผ ์ˆ˜์ง‘ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋‹จ์ผ ํ”„๋กœ์ ํŠธ์˜ ์‹ฑํฌ๋ฅผ ๋งŒ๋“ค๋ ค๋ฉด ๋‹ค์Œ ์•ˆ๋‚ด๋ฅผ ๋”ฐ๋ฅด์„ธ์š”.

๋‹ค์Œ ์•ˆ๋‚ด๋Š” Logging ๋ฏธ๋ฆฌ๋ณด๊ธฐ ์ธํ„ฐํŽ˜์ด์Šค์— ์ ์šฉ๋ฉ๋‹ˆ๋‹ค.

  1. ๋กœ๊ทธ ํƒ์ƒ‰๊ธฐ๋กœ ์ด๋™

  2. Google Cloud ํ”„๋กœ์ ํŠธ๋ฅผ ์„ ํƒํ•ฉ๋‹ˆ๋‹ค.

  3. ์ฟผ๋ฆฌ ๋นŒ๋” ํƒญ์—์„œ ๋‹ค์Œ ์ฟผ๋ฆฌ๋ฅผ ์ž…๋ ฅํ•ฉ๋‹ˆ๋‹ค.

      resource.type="k8s_pod"
      jsonPayload.reason="Pulling"
    
  4. ๋ณ€๊ฒฝ ๋‚ด์—ญ ํ•„ํ„ฐ๋ฅผ ์ง€๋‚œ 1์‹œ๊ฐ„์—์„œ ์ง€๋‚œ 7์ผ๋กœ ๋ณ€๊ฒฝํ•ฉ๋‹ˆ๋‹ค.

    ์ด๋ฏธ์ง€

  5. ์ฟผ๋ฆฌ ์‹คํ–‰์„ ํด๋ฆญํ•ฉ๋‹ˆ๋‹ค.

  6. ๊ฒฐ๊ณผ๊ฐ€ ์˜ฌ๋ฐ”๋ฅด๊ฒŒ ํ‘œ์‹œ๋˜๋Š”์ง€ ํ™•์ธํ•œ ํ›„ ์ž‘์—… > ์‹ฑํฌ ๋งŒ๋“ค๊ธฐ๋ฅผ ํด๋ฆญํ•ฉ๋‹ˆ๋‹ค.

  7. ์‹ฑํฌ ๋ชฉ๋ก์—์„œ BigQuery ๋ฐ์ดํ„ฐ ์„ธํŠธ๋ฅผ ์„ ํƒํ•œ ํ›„ ๋‹ค์Œ์„ ํด๋ฆญํ•ฉ๋‹ˆ๋‹ค.

  8. ์‹ฑํฌ ์ˆ˜์ • ํŒจ๋„์—์„œ ๋‹ค์Œ ๋‹จ๊ณ„๋ฅผ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค.

    • ์‹ฑํฌ ์ด๋ฆ„ ํ•„๋“œ์— image_pull_logs๋ฅผ ์ž…๋ ฅํ•ฉ๋‹ˆ๋‹ค.
    • ์‹ฑํฌ ๋Œ€์ƒ ํ•„๋“œ์—์„œ ์ƒˆ ๋ฐ์ดํ„ฐ ์„ธํŠธ๋ฅผ ๋งŒ๋“ค๊ฑฐ๋‚˜ ๋‹ค๋ฅธ ํ”„๋กœ์ ํŠธ์˜ ๋Œ€์ƒ ๋ฐ์ดํ„ฐ ์„ธํŠธ๋ฅผ ์„ ํƒํ•ฉ๋‹ˆ๋‹ค.
  9. ์‹ฑํฌ ๋งŒ๋“ค๊ธฐ๋ฅผ ํด๋ฆญํ•ฉ๋‹ˆ๋‹ค.

์—ฌ๋Ÿฌ ํ”„๋กœ์ ํŠธ์˜ ์‹ฑํฌ๋ฅผ ๋งŒ๋“ค๋ ค๋ฉด ๋‹ค์Œ ์•ˆ๋‚ด๋ฅผ ๋”ฐ๋ฅด์„ธ์š”.

  1. Cloud Shell์„ ์—ฝ๋‹ˆ๋‹ค.

  2. Cloud Shell์—์„œ ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค.

    PROJECTS="PROJECT-LIST"
    DESTINATION_PROJECT="DATASET-PROJECT"
    DATASET="DATASET-NAME"
    
    for source_project in $PROJECTS
    do
      gcloud logging --project="${source_project}" sinks create image_pull_logs bigquery.googleapis.com/projects/${DESTINATION_PROJECT}/datasets/${DATASET} --log-filter='resource.type="k8s_pod" jsonPayload.reason="Pulling"'
    done
    

    ๊ฐ ํ•ญ๋ชฉ์˜ ์˜๋ฏธ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.

    • PROJECT-LIST๋Š” ๊ณต๋ฐฑ์œผ๋กœ ๊ตฌ๋ถ„๋œ Google Cloud ํ”„๋กœ์ ํŠธ ID์˜ ๋ชฉ๋ก์ž…๋‹ˆ๋‹ค. ์˜ˆ๋ฅผ ๋“ค๋ฉด project1 project2 project3์ž…๋‹ˆ๋‹ค.
    • DATASET-PROJECT๋Š” ๋ฐ์ดํ„ฐ ์„ธํŠธ๋ฅผ ์ €์žฅํ•  ํ”„๋กœ์ ํŠธ์ž…๋‹ˆ๋‹ค.
    • DATASET-NAME์€ ๋ฐ์ดํ„ฐ ์„ธํŠธ ์ด๋ฆ„์ž…๋‹ˆ๋‹ค(์˜ˆ: image_pull_logs).

์‹ฑํฌ๋ฅผ ๋งŒ๋“  ํ›„ ์ด๋ฏธ์ง€๋ฅผ ๊ฐ€์ ธ์˜ค๋Š” ๋นˆ๋„์— ๋”ฐ๋ผ BigQuery ํ…Œ์ด๋ธ”๋กœ ๋ฐ์ดํ„ฐ๋ฅผ ์ „์†กํ•˜๋Š” ๋ฐ ์‹œ๊ฐ„์ด ๊ฑธ๋ฆฝ๋‹ˆ๋‹ค.

๊ฐ€์ ธ์˜ค๊ธฐ ๋นˆ๋„ ์ฟผ๋ฆฌ

๋นŒ๋“œ๊ฐ€ ์ˆ˜ํ–‰ํ•˜๋Š” ์ด๋ฏธ์ง€ ๊ฐ€์ ธ์˜ค๊ธฐ์˜ ๋Œ€ํ‘œ ์ƒ˜ํ”Œ์ด ์žˆ์œผ๋ฉด ๊ฐ€์ ธ์˜ค๊ธฐ ๋นˆ๋„์˜ ์ฟผ๋ฆฌ๋ฅผ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค.

  1. BigQuery ์ฝ˜์†”๋กœ ์ด๋™ํ•ฉ๋‹ˆ๋‹ค.

  2. ๋‹ค์Œ ์ฟผ๋ฆฌ๋ฅผ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค.

    SELECT
      REGEXP_EXTRACT(jsonPayload.message, r'"(.*?)"') AS imageName,
      COUNT(*) AS numberOfPulls
    FROM
          `DATASET-PROJECT.DATASET-NAME.events_*`
    GROUP BY
          imageName
    ORDER BY
          numberOfPulls DESC
    

    ๊ฐ ํ•ญ๋ชฉ์˜ ์˜๋ฏธ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.

    • DATASET-PROJECT๋Š” ๋ฐ์ดํ„ฐ ์„ธํŠธ๊ฐ€ ํฌํ•จ๋œ ํ”„๋กœ์ ํŠธ์ž…๋‹ˆ๋‹ค.
    • DATASET-NAME๋Š” ๋ฐ์ดํ„ฐ ์„ธํŠธ ์ด๋ฆ„์ž…๋‹ˆ๋‹ค.

๋‹ค์Œ ์˜ˆ์‹œ๋Š” ์ฟผ๋ฆฌ์˜ ์ถœ๋ ฅ์„ ๋ณด์—ฌ์ค๋‹ˆ๋‹ค. imageName ์—ด์—์„œ Container Registry ๋˜๋Š” Artifact Registry์— ์ €์žฅ๋˜์ง€ ์•Š์€ ์ด๋ฏธ์ง€์˜ ๊ฐ€์ ธ์˜ค๊ธฐ ๋นˆ๋„๋ฅผ ๊ฒ€ํ† ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

์ด๋ฏธ์ง€

Container Registry๋กœ ์ด๋ฏธ์ง€ ๋ณต์‚ฌ

ํƒ€์‚ฌ ๋ ˆ์ง€์ŠคํŠธ๋ฆฌ์—์„œ ์ด๋ฏธ์ง€๋ฅผ ์‹๋ณ„ํ•˜๋ฉด ์ด๋ฏธ์ง€๋ฅผ Container Registry๋กœ ๋ณต์‚ฌํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. gcrane ๋„๊ตฌ๋Š” ๋ณต์‚ฌ ํ”„๋กœ์„ธ์Šค๋ฅผ ์ง€์›ํ•ฉ๋‹ˆ๋‹ค.

  1. Cloud Shell์—์„œ ํ™•์ธํ•œ ์ด๋ฏธ์ง€์˜ ์ด๋ฆ„์œผ๋กœ ํ…์ŠคํŠธ ํŒŒ์ผ images.txt๋ฅผ ๋งŒ๋“ญ๋‹ˆ๋‹ค. ์˜ˆ๋ฅผ ๋“ค๋ฉด ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.

    ubuntu:18.04
    debian:buster
    hello-world:latest
    redis:buster
    jupyter/tensorflow-notebook
    
  2. gcrane์„ ๋‹ค์šด๋กœ๋“œํ•ฉ๋‹ˆ๋‹ค.

      GO111MODULE=on go get github.com/google/go-containerregistry/cmd/gcrane
    
  3. copy_images.sh๋ผ๋Š” ์Šคํฌ๋ฆฝํŠธ๋ฅผ ๋งŒ๋“ค์–ด ํŒŒ์ผ ๋ชฉ๋ก์„ ๋ณต์‚ฌํ•ฉ๋‹ˆ๋‹ค.

    #!/bin/bash
    
    images=$(cat images.txt)
    
    if [ -z "${GCR_PROJECT}" ]
    then
        echo ERROR: GCR_PROJECT must be set before running this
        exit 1
    fi
    
    for img in ${images}
    do
        gcrane cp ${img} gcr.io/${GCR_PROJECT}/${img}
    done
    

    ์Šคํฌ๋ฆฝํŠธ๋ฅผ ์‹คํ–‰ ๊ฐ€๋Šฅํ•˜๊ฒŒ ๋งŒ๋“ญ๋‹ˆ๋‹ค.

      chmod +x copy_images.sh
    
  4. ์Šคํฌ๋ฆฝํŠธ๋ฅผ ์‹คํ–‰ํ•˜์—ฌ ํŒŒ์ผ์„ ๋ณต์‚ฌํ•ฉ๋‹ˆ๋‹ค.

    GCR_PROJECT=${PROJECT}
    ./copy_images.sh
    

๊ถŒํ•œ ํ™•์ธ

๊ธฐ๋ณธ์ ์œผ๋กœ Google Cloud CI/CD ์„œ๋น„์Šค๋Š” ๋™์ผํ•œ Google Cloud ํ”„๋กœ์ ํŠธ์˜ Container Registry์— ์•ก์„ธ์Šคํ•ฉ๋‹ˆ๋‹ค.

  • Cloud Build๋Š” ์ด๋ฏธ์ง€๋ฅผ ๋‚ด๋ณด๋‚ด๊ณ  ๊ฐ€์ ธ์˜ฌ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
  • GKE, Cloud Run, App Engine ๊ฐ€๋ณ€ํ˜• ํ™˜๊ฒฝ, Compute Engine๊ณผ ๊ฐ™์€ ๋Ÿฐํƒ€์ž„ ํ™˜๊ฒฝ์—์„œ ์ด๋ฏธ์ง€๋ฅผ ๊ฐ€์ ธ์˜ฌ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

ํ”„๋กœ์ ํŠธ ๊ฐ„์— ์ด๋ฏธ์ง€๋ฅผ ํ‘ธ์‹œํ•˜๊ฑฐ๋‚˜ ๊ฐ€์ ธ์™€์•ผ ํ•˜๋Š” ๊ฒฝ์šฐ ๋˜๋Š” Container Registry์— ์•ก์„ธ์Šคํ•ด์•ผ ํ•˜๋Š” ํƒ€์‚ฌ ๋„๊ตฌ๋ฅผ ํŒŒ์ดํ”„๋ผ์ธ์—์„œ ์‚ฌ์šฉํ•˜๋Š” ๊ฒฝ์šฐ ์›Œํฌ๋กœ๋“œ๋ฅผ ์—…๋ฐ์ดํŠธํ•˜๊ณ  ์žฌ๋ฐฐํฌํ•˜๊ธฐ ์ „์— ๊ถŒํ•œ์ด ์˜ฌ๋ฐ”๋ฅด๊ฒŒ ๊ตฌ์„ฑ๋˜์–ด ์žˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

์ž์„ธํ•œ ๋‚ด์šฉ์€ ์•ก์„ธ์Šค ์ œ์–ด ๋ฌธ์„œ๋ฅผ ์ฐธ์กฐํ•˜์„ธ์š”.

Container Registry๋ฅผ ์ฐธ์กฐํ•˜๋„๋ก ๋งค๋‹ˆํŽ˜์ŠคํŠธ ์—…๋ฐ์ดํŠธ

ํƒ€์‚ฌ ๋ ˆ์ง€์ŠคํŠธ๋ฆฌ ๋Œ€์‹  Container Registry๋ฅผ ์ฐธ์กฐํ•˜๋„๋ก Dockerfile๊ณผ ๋งค๋‹ˆํŽ˜์ŠคํŠธ๋ฅผ ์—…๋ฐ์ดํŠธํ•ฉ๋‹ˆ๋‹ค.

๋‹ค์Œ ์˜ˆ์‹œ์—์„œ๋Š” ํƒ€์‚ฌ ๋ ˆ์ง€์ŠคํŠธ๋ฆฌ๋ฅผ ์ฐธ์กฐํ•˜๋Š” ๋งค๋‹ˆํŽ˜์ŠคํŠธ๋ฅผ ๋ณด์—ฌ์ค๋‹ˆ๋‹ค.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

๋งค๋‹ˆํŽ˜์ŠคํŠธ์˜ ์ด ์—…๋ฐ์ดํŠธ๋œ ๋ฒ„์ „์€ Container Registry์˜ ์ด๋ฏธ์ง€๋ฅผ ๊ฐ€๋ฆฌํ‚ต๋‹ˆ๋‹ค.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: gcr.io/<GCR_PROJECT>/nginx:1.14.2
        ports:
        - containerPort: 80

๋งค๋‹ˆํŽ˜์ŠคํŠธ๊ฐ€ ๋งŽ์€ ๊ฒฝ์šฐ ๋งŽ์€ ํ…์ŠคํŠธ ํŒŒ์ผ์—์„œ ์—…๋ฐ์ดํŠธ๋ฅผ ์ฒ˜๋ฆฌํ•  ์ˆ˜ ์žˆ๋Š” sed๋‚˜ ๋‹ค๋ฅธ ๋„๊ตฌ๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

์›Œํฌ๋กœ๋“œ ์žฌ๋ฐฐํฌ

์—…๋ฐ์ดํŠธ๋œ ๋งค๋‹ˆํŽ˜์ŠคํŠธ๋กœ ์›Œํฌ๋กœ๋“œ๋ฅผ ์žฌ๋ฐฐํฌํ•ฉ๋‹ˆ๋‹ค.

BigQuery ์ฝ˜์†”์—์„œ ๋‹ค์Œ ์ฟผ๋ฆฌ๋ฅผ ์‹คํ–‰ํ•˜์—ฌ ์ƒˆ ์ด๋ฏธ์ง€ ๊ฐ€์ ธ์˜ค๊ธฐ๋ฅผ ์ถ”์ ํ•ฉ๋‹ˆ๋‹ค.

SELECT`

FORMAT_TIMESTAMP("%D %R", timestamp) as timeOfImagePull,
REGEXP_EXTRACT(jsonPayload.message, r'"(.*?)"') AS imageName,
COUNT(*) AS numberOfPulls
FROM
  `image_pull_logs.events_*`
GROUP BY
  timeOfImagePull,
  imageName
ORDER BY
  timeOfImagePull DESC,
  numberOfPulls DESC

๋ชจ๋“  ์ƒˆ ์ด๋ฏธ์ง€ ๊ฐ€์ ธ์˜ค๊ธฐ๋Š” Container Registry์—์„œ ์ˆ˜ํ–‰๋˜์–ด์•ผ ํ•˜๊ณ  gcr.io ๋ฌธ์ž์—ด์„ ํฌํ•จํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

(์„ ํƒ์‚ฌํ•ญ) ํƒ€์‚ฌ ๋ ˆ์ง€์ŠคํŠธ๋ฆฌ์—์„œ ์ด๋ฏธ์ง€ ๊ฐ€์ ธ์˜ค๊ธฐ ์ฐจ๋‹จ

Binary Authorization์„ ์‚ฌ์šฉํ•˜๋Š” GKE ํด๋Ÿฌ์Šคํ„ฐ์˜ ๊ฒฝ์šฐ ์ •์˜๋œ ์ •์ฑ…์€ ์ž๋™์œผ๋กœ ์‹ ๋ขฐํ•  ์ˆ˜ ์—†๋Š” ์†Œ์Šค์—์„œ ๊ฐ€์ ธ์˜ค๊ธฐ๋ฅผ ์ฐจ๋‹จํ•ฉ๋‹ˆ๋‹ค. ๋งˆ์ด๊ทธ๋ ˆ์ด์…˜๋œ ์ด๋ฏธ์ง€๋ฅผ ์˜ˆ์™ธ ๋ชฉ๋ก์— ์ถ”๊ฐ€ํ•˜์—ฌ ์ด ์ด๋ฏธ์ง€๊ฐ€ ์ •์ฑ…์—์„œ ์ฐจ๋‹จ๋˜์ง€ ์•Š๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค. ์ด ์•ˆ๋‚ด์—์„œ๋Š” ํ”„๋กœ์ ํŠธ ๋‚ด Container Registry์— ์ €์žฅ๋œ ๋ชจ๋“  ์ด๋ฏธ์ง€์— ์˜ˆ์™ธ๋ฅผ ์ง€์ •ํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ์„ค๋ช…ํ•ฉ๋‹ˆ๋‹ค.

์ฒ˜์Œ์— ์ •์ฑ…์„ ์—…๋ฐ์ดํŠธํ•  ๋•Œ๋Š” ํ…Œ์ŠคํŠธ ์‹คํ–‰ ๋ชจ๋“œ๋ฅผ ์‚ฌ์šฉ ์„ค์ •ํ•˜๋Š” ๊ฒƒ์ด ์ข‹์Šต๋‹ˆ๋‹ค. Binary Authorization์€ ์ด๋ฏธ์ง€๋ฅผ ์ฐจ๋‹จํ•˜๋Š” ๋Œ€์‹  ๊ฐ์‚ฌ ๋กœ๊ทธ ํ•ญ๋ชฉ์„ ๋งŒ๋“ค์–ด Container Registry๋กœ ๋งˆ์ด๊ทธ๋ ˆ์ด์…˜ํ•ด์•ผ ํ•˜๋Š” ํƒ€์‚ฌ ๋ ˆ์ง€์ŠคํŠธ๋ฆฌ์—์„œ ์ฒ˜๋ฆฌ๋˜์ง€ ์•Š์€ ์ด๋ฏธ์ง€๋ฅผ ์‹๋ณ„ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋ฐฐํฌ ์ •์ฑ…์„ ๊ตฌ์„ฑํ•˜๋Š” ๋ฐฉ๋ฒ•์— ๋Œ€ํ•œ ์ž์„ธํ•œ ๋‚ด์šฉ์€ Binary Authorization ๋ฌธ์„œ๋ฅผ ์ฐธ์กฐํ•˜์„ธ์š”.

  1. Binary Authorization ํŽ˜์ด์ง€๋กœ ์ด๋™
  2. ์ •์ฑ… ์ˆ˜์ •์„ ํด๋ฆญํ•ฉ๋‹ˆ๋‹ค.
  3. ํ”„๋กœ์ ํŠธ ๊ธฐ๋ณธ ๊ทœ์น™์—์„œ ํ…Œ์ŠคํŠธ ์‹คํ–‰ ๋ชจ๋“œ๋ฅผ ์‚ฌ์šฉ ์„ค์ •ํ•ฉ๋‹ˆ๋‹ค.
  4. ๋ฐฐํฌ ๊ทœ์น™์—์„œ ์ œ์™ธ๋˜๋Š” ์ด๋ฏธ์ง€์—์„œ Google์ด ์ œ๊ณตํ•˜๋Š” ๋ชจ๋“  ์‹œ์Šคํ…œ ์ด๋ฏธ์ง€ ์‹ ๋ขฐ๋ฅผ ์„ ํƒ๋œ ์ƒํƒœ๋กœ ๋‘ก๋‹ˆ๋‹ค.
  5. ์ด๋ฏธ์ง€ ๊ฒฝ๋กœ๋ฅผ ํŽผ์นฉ๋‹ˆ๋‹ค.
  6. ์ด๋ฏธ์ง€ ๊ฒฝ๋กœ๋ฅผ ๊ธฐ๋ณธ ํ”„๋กœ์ ํŠธ ๊ทœ์น™์— ๋Œ€ํ•œ ์˜ˆ์™ธ๋กœ ์ถ”๊ฐ€ํ•ฉ๋‹ˆ๋‹ค.
    1. ์ด๋ฏธ์ง€ ๋ชฉ๋ก ํ•˜๋‹จ์— ์žˆ๋Š” ์ด๋ฏธ์ง€ ์ถ”๊ฐ€๋ฅผ ํด๋ฆญํ•ฉ๋‹ˆ๋‹ค.
    2. Google Cloud ํ”„๋กœ์ ํŠธ์˜ ์ด๋ฏธ์ง€ ๊ฒฝ๋กœ๋ฅผ ์ž…๋ ฅํ•ฉ๋‹ˆ๋‹ค. ์˜ˆ๋ฅผ ๋“ค์–ด gcr.io/my-project/*๋Š” my-project ํ”„๋กœ์ ํŠธ์˜ ๋ชจ๋“  ์ด๋ฏธ์ง€๋ฅผ ์˜ˆ์™ธ๋กœ ์ฒ˜๋ฆฌํ•ฉ๋‹ˆ๋‹ค.
  7. ๋ฐฐํฌํ•  ์ด๋ฏธ์ง€๊ฐ€ ํฌํ•จ๋œ ๋‹ค๋ฅธ ํ”„๋กœ์ ํŠธ์— ์ด์ „ ๋‹จ๊ณ„๋ฅผ ๋ฐ˜๋ณตํ•ฉ๋‹ˆ๋‹ค.

Logging์—์„œ ๋ฐฐํฌ์˜ ํ…Œ์ŠคํŠธ ์‹คํ–‰ ์ด๋ฒคํŠธ๋ฅผ ๊ฒ€ํ† ํ•ฉ๋‹ˆ๋‹ค. ํƒ€์‚ฌ ๋ ˆ์ง€์ŠคํŠธ๋ฆฌ์—์„œ ์ •๊ธฐ์ ์œผ๋กœ ๊ฐ€์ ธ์˜ค๋Š” ๋‚˜๋จธ์ง€ ์ด๋ฏธ์ง€๋ฅผ ๋งˆ์ด๊ทธ๋ ˆ์ด์…˜ํ•ฉ๋‹ˆ๋‹ค. ๋ชจ๋“  ์ด๋ฏธ์ง€๊ฐ€ ๋งˆ์ด๊ทธ๋ ˆ์ด์…˜๋˜๋ฉด ์ •์ฑ…์„ ์ˆ˜์ •ํ•˜์—ฌ ํ…Œ์ŠคํŠธ ์‹คํ–‰ ๋ชจ๋“œ๋ฅผ ์ค‘์ง€ํ•˜๊ณ  ์‹ ๋ขฐํ•  ์ˆ˜ ์—†๋Š” ์†Œ์Šค์˜ ์ด๋ฏธ์ง€๋ฅผ ์ฐจ๋‹จํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.