Cassandra ๋ฐฑ์—… ๋ฐ ๋ณต๊ตฌ

์ด ์„น์…˜์—์„œ๋Š” Apigee ํ•˜์ด๋ธŒ๋ฆฌ๋“œ ๋Ÿฐํƒ€์ž„ ์˜์—ญ์— ์„ค์น˜๋œ Apache Cassandra ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค ๋ง์˜ ๋ฐ์ดํ„ฐ ๋ฐฑ์—…๊ณผ ๋ณต๊ตฌ๋ฅผ ๊ตฌ์„ฑํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ์„ค๋ช…ํ•ฉ๋‹ˆ๋‹ค. Cassandra Datastore๋„ ์ฐธ์กฐํ•˜์„ธ์š”.

Cassandra ๋ฐฑ์—…์— ๋Œ€ํ•ด ์•Œ์•„์•ผ ํ•  ์‚ฌํ•ญ

Cassandra๋Š” ๊ฐ ๋ฆฌ์ „ ๋˜๋Š” ๋ฐ์ดํ„ฐ ์„ผํ„ฐ์— ๋ฐ์ดํ„ฐ ๋ณต์‚ฌ๋ณธ์ด ์ตœ์†Œ 3๊ฐœ ์ด์ƒ ์žˆ๋„๋ก ๊ตฌ์„ฑ๋œ ๋ณต์ œ๋œ ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค์ž…๋‹ˆ๋‹ค. Cassandra๋Š” ์ŠคํŠธ๋ฆฌ๋ฐ ๋ณต์ œ ๋ฐ ์ฝ๊ธฐ ๋ณต๊ตฌ๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ํŠน์ • ์‹œ์ ์— ๊ฐ ๋ฆฌ์ „ ๋˜๋Š” ๋ฐ์ดํ„ฐ ์„ผํ„ฐ์˜ ๋ฐ์ดํ„ฐ ๋ณต์ œ๋ณธ์„ ์œ ์ง€ํ•ฉ๋‹ˆ๋‹ค.

ํ•˜์ด๋ธŒ๋ฆฌ๋“œ์—์„œ Cassandra ๋ฐฑ์—…์€ ๊ธฐ๋ณธ์ ์œผ๋กœ ์‚ฌ์šฉ ์„ค์ •๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ํ•˜์ง€๋งŒ ์‹ค์ˆ˜๋กœ ๋ฐ์ดํ„ฐ๊ฐ€ ์‚ญ์ œ๋˜๋Š” ๊ฒฝ์šฐ๋ฅผ ๋Œ€๋น„ํ•˜์—ฌ Cassandra ๋ฐฑ์—…์„ ์‚ฌ์šฉ ์„ค์ •ํ•˜๋Š” ๊ฒƒ์ด ์ข‹์Šต๋‹ˆ๋‹ค.

๋ฐฑ์—…๋˜๋Š” ํ•ญ๋ชฉ

์ด ์ฃผ์ œ์—์„œ ์„ค๋ช…ํ•˜๋Š” ๋ฐฑ์—… ๊ตฌ์„ฑ์—์„œ๋Š” ๋‹ค์Œ ํ•ญ๋ชฉ์„ ๋ฐฑ์—…ํ•ฉ๋‹ˆ๋‹ค.

  • ์‚ฌ์šฉ์ž ์Šคํ‚ค๋งˆ๋ฅผ ํฌํ•จํ•œ Cassandra ์Šคํ‚ค๋งˆ(Apigee ํ‚ค์ŠคํŽ˜์ด์Šค ์ •์˜)
  • ๋…ธ๋“œ๋‹น Cassandra ํŒŒํ‹ฐ์…˜ ํ† ํฐ ์ •๋ณด
  • Cassandra ๋ฐ์ดํ„ฐ์˜ ์Šค๋ƒ…์ƒท

๋ฐฑ์—… ๋ฐ์ดํ„ฐ๊ฐ€ ์ €์žฅ๋˜๋Š” ์œ„์น˜

๋ฐฑ์—…๋œ ๋ฐ์ดํ„ฐ๋Š” Google Cloud Storage ๋ฒ„ํ‚ท์— ์ €์žฅ๋˜๊ณ  ๊ฐœ๋ฐœ์ž๊ฐ€ ์ด ๋ฒ„ํ‚ท์„ ๋งŒ๋“ค์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ์ด ์ฃผ์ œ์—์„œ๋Š” ๋ฒ„ํ‚ท ๋งŒ๋“ค๊ธฐ์™€ ๊ตฌ์„ฑ์„ ์„ค๋ช…ํ•ฉ๋‹ˆ๋‹ค.

Cassandra ๋ฐฑ์—… ์˜ˆ์•ฝ

๋ฐฑ์—…์€ ๋Ÿฐํƒ€์ž„ ์˜์—ญ์—์„œ cron ์ž‘์—…์œผ๋กœ ์˜ˆ์•ฝ๋ฉ๋‹ˆ๋‹ค. Cassandra ๋ฐฑ์—…์„ ์˜ˆ์•ฝํ•˜๋ ค๋ฉด ๋‹ค์Œ ์•ˆ๋‚ด๋ฅผ ๋”ฐ๋ฅด์„ธ์š”.

  1. ๋‹ค์Œ create-service-account ๋ช…๋ น์–ด๋ฅผ ์‹คํ–‰ํ•˜์—ฌ ํ‘œ์ค€ roles/storage.objectAdmin ์—ญํ• ์ด ์žˆ๋Š” Google Cloud ์„œ๋น„์Šค ๊ณ„์ •(SA)์„ ๋งŒ๋“ญ๋‹ˆ๋‹ค. ์ด SA ์—ญํ• ์„ ์‚ฌ์šฉํ•˜๋ฉด ๋ฐฑ์—… ๋ฐ์ดํ„ฐ๋ฅผ Cloud Storage์— ์“ธ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ํ•˜์ด๋ธŒ๋ฆฌ๋“œ ์„ค์น˜ ๋ฃจํŠธ ๋””๋ ‰ํ„ฐ๋ฆฌ์—์„œ ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค.
    ./tools/create-service-account apigee-cassandra OUTPUT_DIR
    ์˜ˆ๋ฅผ ๋“ค๋ฉด ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.
    ./tools/create-service-account apigee-cassandra ./service-accounts
    Google Cloud ์„œ๋น„์Šค ๊ณ„์ •์— ๋Œ€ํ•œ ์ž์„ธํ•œ ๋‚ด์šฉ์€ ์„œ๋น„์Šค ๊ณ„์ • ๋งŒ๋“ค๊ธฐ ๋ฐ ๊ด€๋ฆฌ๋ฅผ ์ฐธ์กฐํ•˜์„ธ์š”.
  2. create-service-account ๋ช…๋ น์–ด๋Š” ์„œ๋น„์Šค ๊ณ„์ • ๋น„๊ณต๊ฐœ ํ‚ค๊ฐ€ ํฌํ•จ๋œ JSON ํŒŒ์ผ์„ ์ €์žฅํ•ฉ๋‹ˆ๋‹ค. ํŒŒ์ผ์€ ๋ช…๋ น์–ด๊ฐ€ ์‹คํ–‰๋˜๋Š” ๋™์ผํ•œ ๋””๋ ‰ํ„ฐ๋ฆฌ์— ์ €์žฅ๋ฉ๋‹ˆ๋‹ค. ๋‹ค์Œ ๋‹จ๊ณ„์—์„œ๋Š” ์ด ํŒŒ์ผ์˜ ๊ฒฝ๋กœ๊ฐ€ ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค.
  3. Cloud Storage ๋ฒ„ํ‚ท ์ƒ์„ฑ ๋ฒ„ํ‚ท์— ์ ์ ˆํ•œ ๋ฐ์ดํ„ฐ ๋ณด๊ด€ ์ •์ฑ…์„ ์ง€์ •ํ•ฉ๋‹ˆ๋‹ค. ๋ฐ์ดํ„ฐ ๋ณด๊ด€ ์ •์ฑ…์„ 15์ผ๋กœ ์„ค์ •ํ•˜๋Š” ๊ฒƒ์ด ์ข‹์Šต๋‹ˆ๋‹ค.
  4. overrides.yaml ํŒŒ์ผ์„ ์—ฝ๋‹ˆ๋‹ค.
  5. ๋‹ค์Œ cassandra.backup ์†์„ฑ์„ ์ถ”๊ฐ€ํ•˜์—ฌ ๋ฐฑ์—…์„ ์‚ฌ์šฉ ์„ค์ •ํ•ฉ๋‹ˆ๋‹ค. ์ด๋ฏธ ๊ตฌ์„ฑ๋œ ์†์„ฑ์„ ์‚ญ์ œํ•˜์ง€ ๋งˆ์„ธ์š”.

    ๋งค๊ฐœ๋ณ€์ˆ˜

    cassandra:
      ...
    
      backup:
        enabled: true
        serviceAccountPath: SA_JSON_FILE_PATH
        dbStorageBucket: CLOUD_STORAGE_BUCKET_PATH
        schedule: BACKUP_SCHEDULE_CODE
    
      ...
      

    ์˜ˆ

    ...
    
    cassandra:
      storage:
        type: gcepd
        capacity: 50Gi
        gcepd:
          replicationType: regional-pd
      sslRootCAPath: "/Users/myhome/ssh/cassandra.crt"
      sslCertPath: "/Users/myhome/ssh/cassandra.crt"
      sslKeyPath: "/Users/myhome/ssh/cassandra.key"
      auth:
        default:
          password: "abc123"
        admin:
          password: "abc234"
        ddl:
          password: "abc345"
        dml:
          password: "abc456"
      nodeSelector:
        key: cloud.google.com/gke-nodepool
        value: apigee-data
      backup:
        enabled: true
        serviceAccountPath: "/Users/myhome/.ssh/my_cassandra_backup.json"
        dbStorageBucket: "gs://myname-cassandra-backup"
        schedule: "45 23 * * 6"
    
      ... 
  6. ๊ฐ ํ•ญ๋ชฉ์˜ ์˜๋ฏธ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.
    ์†์„ฑ ์„ค๋ช…
    backup:enabled ๋ฐฑ์—…์€ ๊ธฐ๋ณธ์ ์œผ๋กœ ์‚ฌ์šฉ ์ค‘์ง€๋˜์–ด ์žˆ์Šต๋‹ˆ๋‹ค. ์ด ์†์„ฑ์„ true๋กœ ์„ค์ •ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.
    backup:serviceAccountPath

    SA_JSON_FILE_PATH

    ./tools/create-service-account๋ฅผ ์‹คํ–‰ํ•  ๋•Œ ๋‹ค์šด๋กœ๋“œ๋œ ์„œ๋น„์Šค ๊ณ„์ • JSON ํŒŒ์ผ์˜ ํŒŒ์ผ ์‹œ์Šคํ…œ ๊ฒฝ๋กœ์ž…๋‹ˆ๋‹ค.

    backup:dbStorageBucket

    CLOUD_STORAGE_BUCKET_PATH

    gs://BUCKET_NAME ํ˜•์‹์˜ Cloud Storage ๋ฒ„ํ‚ท ๊ฒฝ๋กœ์ž…๋‹ˆ๋‹ค. gs://๋Š” ํ•„์ˆ˜ ํ•ญ๋ชฉ์ž…๋‹ˆ๋‹ค.

    backup:schedule

    BACKUP_SCHEDULE_CODE

    ๋ฐฑ์—…์ด ์‹œ์ž‘๋˜๋Š” ์‹œ๊ฐ„์ด๋ฉฐ ํ‘œ์ค€ crontab ๋ฌธ๋ฒ•์— ์ง€์ •๋˜์–ด ์žˆ์Šต๋‹ˆ๋‹ค. ๊ธฐ๋ณธ๊ฐ’: 0 2 * * *

  7. ๊ตฌ์„ฑ ๋ณ€๊ฒฝ์‚ฌํ•ญ์„ ์ƒˆ ํด๋Ÿฌ์Šคํ„ฐ์— ์ ์šฉํ•ฉ๋‹ˆ๋‹ค. ์˜ˆ๋ฅผ ๋“ค๋ฉด ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.
    ./apigeectl apply -f overrides.yaml

๋ฐฑ์—… ๋ณต์›

๋ณต์›์€ ๋ฐฑ์—… ์œ„์น˜์—์„œ ๋ฐ์ดํ„ฐ๋ฅผ ๊ฐ€์ ธ์™€ ๋…ธ๋“œ ์ˆ˜๊ฐ€ ๋™์ผํ•œ ์ƒˆ Cassandra ํด๋Ÿฌ์Šคํ„ฐ์— ๋ณต์›ํ•ฉ๋‹ˆ๋‹ค. ์ด์ „ Cassandra ํด๋Ÿฌ์Šคํ„ฐ์—์„œ๋Š” ๋ฐ์ดํ„ฐ๋ฅผ ๊ฐ€์ ธ์˜ค์ง€ ์•Š์Šต๋‹ˆ๋‹ค.

์•„๋ž˜์˜ ๋ณต์› ์•ˆ๋‚ด๋Š” ๋ฐฑ์—…์— Google Cloud Storage๋ฅผ ์‚ฌ์šฉํ•˜๋Š” ๋‹จ์ผ ๋ฆฌ์ „ ๋ฐฐํฌ์— ์ ์šฉ๋ฉ๋‹ˆ๋‹ค. ๋‹ค๋ฅธ ๋ฐฐํฌ์ธ ๊ฒฝ์šฐ ๋‹ค์Œ์„ ์ฐธ์กฐํ•˜์„ธ์š”.

Cassandra ๋ฐฑ์—…์„ ๋ณต์›ํ•˜๋ ค๋ฉด ๋‹ค์Œ ์•ˆ๋‚ด๋ฅผ ๋”ฐ๋ฅด์„ธ์š”.

  1. ํ•˜์ด๋ธŒ๋ฆฌ๋“œ ๋Ÿฐํƒ€์ž„ ๋ฐฐํฌ๋ฅผ ๋ณต์›ํ•˜๋Š” ๋ฐ ์‚ฌ์šฉ๋˜๋Š” ๊ธฐ์กด Kubernetes ํด๋Ÿฌ์Šคํ„ฐ ๋‚ด์— ์ƒˆ ๋„ค์ž„์ŠคํŽ˜์ด์Šค๋ฅผ ๋งŒ๋“ญ๋‹ˆ๋‹ค. ์ƒˆ ๋„ค์ž„์ŠคํŽ˜์ด์Šค์— ์›๋ณธ ๋„ค์ž„์ŠคํŽ˜์ด์Šค ์ด๋ฆ„์„ ์‚ฌ์šฉํ•˜์ง€ ๋งˆ์„ธ์š”. ๋ณต์›์— ์ด์ „ ๋„ค์ž„์ŠคํŽ˜์ด์Šค๋ฅผ ์‚ฌ์šฉํ•˜์ง€ ๋งˆ์„ธ์š”.
  2. ๋ฃจํŠธ ํ•˜์ด๋ธŒ๋ฆฌ๋“œ ์„ค์น˜ ๋””๋ ‰ํ„ฐ๋ฆฌ์— ์ƒˆ overrides-restore.yaml ํŒŒ์ผ์„ ๋งŒ๋“ญ๋‹ˆ๋‹ค.
  3. ์ „์ฒด Cassandra ๊ตฌ์„ฑ์„ ์›๋ณธ overrides.yaml ํŒŒ์ผ์—์„œ ์ƒˆ overrides-restore.yaml ํŒŒ์ผ๋กœ ๋ณต์‚ฌํ•ฉ๋‹ˆ๋‹ค. ์˜ˆ์‹œ๋Š” ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์ฐธ์กฐํ•˜์„ธ์š”.
    cp ./overrides.yaml ./overrides-restore.yaml
  4. ์ƒˆ overrides-restore.yaml ํŒŒ์ผ์— ๋„ค์ž„์ŠคํŽ˜์ด์Šค ์š”์†Œ๋ฅผ ์ถ”๊ฐ€ํ•ฉ๋‹ˆ๋‹ค. ์›๋ณธ ํด๋Ÿฌ์Šคํ„ฐ์— ์‚ฌ์šฉํ•œ ๋„ค์ž„์ŠคํŽ˜์ด์Šค์™€ ๋™์ผํ•œ ๋„ค์ž„์ŠคํŽ˜์ด์Šค๋ฅผ ์‚ฌ์šฉํ•˜์ง€ ๋งˆ์„ธ์š”.

    ๋งค๊ฐœ๋ณ€์ˆ˜

    namespace: YOUR_RESTORE_NAMESPACE
    cassandra:
      ...
      restore:
        enabled: true
        snapshotTimestamp: TIMESTAMP
        serviceAccountPath: SA_JSON_FILE_PATH
        dbStorageBucket: CLOUD_STORAGE_BUCKET_PATH
             image:
               pullPolicy: Always
      ...

    ์˜ˆ

    ...
        namespace: cassandra-restore
        cassandra:
          storage:
            type: gcepd
            capacity: 50Gi
            gcepd:
              replicationType: regional-pd
          sslRootCAPath: "/Users/myhome/ssh/cassandra.crt"
          sslCertPath: "/Users/myhome/ssh/cassandra.crt"
          sslKeyPath: "/Users/myhome/ssh/cassandra.key"
          auth:
            default:
              password: "abc123"
            admin:
              password: "abc234"
            ddl:
              password: "abc345"
            dml:
              password: "abc456"
          nodeSelector:
            key: cloud.google.com/gke-nodepool
            value: apigee-data
    
          restore:
            enabled: true
            snapshotTimestamp: "20210203213003"
            serviceAccountPath: "/Users/myhome/.ssh/my_cassandra_backup.json"
            dbStorageBucket: "gs://myname-cassandra-backup"
            image:
              pullPolicy: Always
        ...
    

    ๊ฐ ํ•ญ๋ชฉ์˜ ์˜๋ฏธ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.

    ์†์„ฑ ์„ค๋ช…
    namespace

    YOUR_RESTORE_NAMESPACE

    ์ƒˆ Cassandra ํด๋Ÿฌ์Šคํ„ฐ์˜ 1๋‹จ๊ณ„์—์„œ ๋งŒ๋“  ์ƒˆ ๋„ค์ž„์ŠคํŽ˜์ด์Šค์˜ ์ด๋ฆ„์ž…๋‹ˆ๋‹ค. ์›๋ž˜ ํด๋Ÿฌ์Šคํ„ฐ์— ์‚ฌ์šฉํ•œ ๋„ค์ž„์ŠคํŽ˜์ด์Šค์™€ ๋™์ผํ•œ ๋„ค์ž„ ์ŠคํŽ˜์ด์Šค๋ฅผ ์‚ฌ์šฉํ•˜์ง€ ๋งˆ์„ธ์š”.

    restore:enabled ๋ณต์›์€ ๊ธฐ๋ณธ์ ์œผ๋กœ ์‚ฌ์šฉ ์ค‘์ง€๋˜์–ด ์žˆ์Šต๋‹ˆ๋‹ค. ์ด ์†์„ฑ์„ true๋กœ ์„ค์ •ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.
    restore:snapshotTimestamp

    TIMESTAMP

    ๋ณต์›ํ•  ๋ฐฑ์—… ์Šค๋ƒ…์ƒท์˜ ํƒ€์ž„์Šคํƒฌํ”„์ž…๋‹ˆ๋‹ค. ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ๋Š” ํƒ€์ž„์Šคํƒฌํ”„๋ฅผ ํ™•์ธํ•˜๋ ค๋ฉด dbStorageBucket์œผ๋กœ ์ด๋™ํ•˜์—ฌ ๋ฒ„ํ‚ท์— ์žˆ๋Š” ํŒŒ์ผ์„ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค. ๊ฐ ํŒŒ์ผ ์ด๋ฆ„์—๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์€ ํƒ€์ž„์Šคํƒฌํ”„ ๊ฐ’์ด ํฌํ•จ๋ฉ๋‹ˆ๋‹ค.

    backup_20210203213003_apigee-cassandra-default-0.tgz

    ์—ฌ๊ธฐ์„œ 20210203213003์€ ํ•ด๋‹น ์‹œ์ ์— ์ƒ์„ฑ๋œ ๋ฐฑ์—…์„ ๋ณต์›ํ•˜๋ ค๋Š” ๊ฒฝ์šฐ์— ์‚ฌ์šฉํ•˜๋Š” snapshotTimestamp ๊ฐ’์ž…๋‹ˆ๋‹ค.

    restore:serviceAccountPath

    SA_JSON_FILE_PATH

    ๋ฐฑ์—…์šฉ์œผ๋กœ ๋งŒ๋“  ์„œ๋น„์Šค ๊ณ„์ •์˜ ํŒŒ์ผ ์‹œ์Šคํ…œ ๊ฒฝ๋กœ์ž…๋‹ˆ๋‹ค.

    restore:dbStorageBucket

    CLOUD_STORAGE_BUCKET_PATH

    ๋ฐฑ์—… ๋ฐ์ดํ„ฐ๊ฐ€ ๋‹ค์Œ ํ˜•์‹์œผ๋กœ ์ €์žฅ๋˜๋Š” Cloud Storage ๋ฒ„ํ‚ท ๊ฒฝ๋กœ์ž…๋‹ˆ๋‹ค.

    gs://BUCKET_NAME

    gs://๋Š” ํ•„์ˆ˜ ํ•ญ๋ชฉ์ž…๋‹ˆ๋‹ค.

  5. ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‹คํ–‰ํ•˜์—ฌ ์ด์ „ ๋„ค์ž„์ŠคํŽ˜์ด์Šค์˜ Cassandra ๋…ธ๋“œ์—์„œ app ๋ผ๋ฒจ์„ ๋ณ€๊ฒฝํ•ฉ๋‹ˆ๋‹ค.
    kubectl label pods --overwrite --namespace=OLD_NAMESPACE -l app=apigee-cassandra app=apigee-cassandra-old
    
  6. ์ƒˆ ํ•˜์ด๋ธŒ๋ฆฌ๋“œ ๋Ÿฐํƒ€์ž„ ๋ฐฐํฌ๋ฅผ ์ƒ์„ฑํ•ฉ๋‹ˆ๋‹ค. ๊ทธ๋Ÿฌ๋ฉด ์ƒˆ Cassandra ํด๋Ÿฌ์Šคํ„ฐ๊ฐ€ ์ƒ์„ฑ๋˜๊ณ  ๋ฐฑ์—… ๋ฐ์ดํ„ฐ๊ฐ€ ํด๋Ÿฌ์Šคํ„ฐ๋กœ ๋ณต์›๋˜๊ธฐ ์‹œ์ž‘ํ•ฉ๋‹ˆ๋‹ค.
    ./apigeectl init  -f ../overrides-restore.yaml
    
    ./apigeectl apply  -f ../overrides-restore.yaml
    
  7. ๋ณต์›์ด ์™„๋ฃŒ๋˜๋ฉด ์ƒˆ ๋„ค์ž„์ŠคํŽ˜์ด์Šค์—์„œ Cassandra ํด๋Ÿฌ์Šคํ„ฐ๋ฅผ ์‚ฌ์šฉํ•˜๋„๋ก ํŠธ๋ž˜ํ”ฝ์„ ์ „ํ™˜ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‹คํ–‰ํ•˜์—ฌ ํŠธ๋ž˜ํ”ฝ์„ ์ „ํ™˜ํ•ฉ๋‹ˆ๋‹ค.

    kubectl get rs -n OLD_NAMESPACE # look for the 'apigee-connect' replicaset
    
    kubectl patch rs -n OLD_NAMESPACE APIGEE_CONNECT_RS_NAME -p '{"spec":{"replicas" : 0}}'
    
  8. ํŠธ๋ž˜ํ”ฝ ์ „ํ™˜์ด ์™„๋ฃŒ๋˜๋ฉด restore ๊ตฌ์„ฑ์„ ์‚ญ์ œํ•˜๊ณ  backup ๊ตฌ์„ฑ์„ overrides-restore.yaml ํŒŒ์ผ์— ์ถ”๊ฐ€ํ•˜์—ฌ ๋ณต์›๋œ ํด๋Ÿฌ์Šคํ„ฐ์—์„œ ๋ฐฑ์—…์„ ๋‹ค์‹œ ๊ตฌ์„ฑํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. YOUR_RESTORE_NAMESPACE๋ฅผ 1๋‹จ๊ณ„์—์„œ ๋งŒ๋“  ์ƒˆ ๋„ค์ž„์ŠคํŽ˜์ด์Šค ์ด๋ฆ„์œผ๋กœ ๋ฐ”๊ฟ‰๋‹ˆ๋‹ค.
    namespace: YOUR_RESTORE_NAMESPACE
    cassandra:
      ...
       backup:
        enabled: true
        serviceAccountPath: SA_JSON_FILE_PATH
        dbStorageBucket: CLOUD_STORAGE_BUCKET_PATH
        schedule: BACKUP_SCHEDULE_CODE
      ...

    ๊ทธ๋Ÿฐ ํ›„ ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ backup ๊ตฌ์„ฑ์„ ์ ์šฉํ•ฉ๋‹ˆ๋‹ค.

    ./apigeectl apply  -f ../overrides-restore.yaml
    

๋ณต์› ๋กœ๊ทธ ๋ณด๊ธฐ

๋ณต์› ์ž‘์—… ๋กœ๊ทธ๋ฅผ ํ™•์ธํ•˜๊ณ  grep์„ ์‚ฌ์šฉํ•˜์—ฌ error๋ฅผ ํ™•์ธํ•˜์—ฌ ๋ณต์› ๋กœ๊ทธ์— ์˜ค๋ฅ˜๊ฐ€ ์—†๋Š”์ง€ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋ณต์›์ด ์™„๋ฃŒ๋˜์—ˆ๋Š”์ง€ ํ™•์ธ

๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ๋ณต์› ์ž‘์—…์ด ์™„๋ฃŒ๋˜์—ˆ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

kubectl get pods

์ถœ๋ ฅ์€ ๋‹ค์Œ๊ณผ ๋น„์Šทํ•ฉ๋‹ˆ๋‹ค.

NAME                           READY     STATUS      RESTARTS   AGE
apigee-cassandra-default-0     1/1       Running     0          1h
apigee-cassandra-default-1     1/1       Running     0          1h
apigee-cassandra-default-2     1/1       Running     0          59m
apigee-cassandra-restore-b4lgf 0/1       Completed   0          51m

๋ณต์› ๋กœ๊ทธ ๋ณด๊ธฐ

๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ๋ณต์› ๋กœ๊ทธ๋ฅผ ๋ด…๋‹ˆ๋‹ค.

kubectl logs -f apigee-cassandra-restore-b4lgf

์ถœ๋ ฅ์€ ๋‹ค์Œ๊ณผ ๋น„์Šทํ•ฉ๋‹ˆ๋‹ค.

Restore Logs:

Activated service account credentials for: [apigee-cassandra-backup-svc@gce-myusername.iam.gserviceaccount.com]
to download file gs://gce-myusername-apigee-cassandra-backup/apigeecluster/dc-1/backup_20190405011309_schema.tgz
INFO: download successfully extracted the backup files from gs://gce-myusername-apigee-cassandra-backup/apigeecluster/dc-1
finished downloading schema.cql
to create schema from 10.32.0.28

Warnings :
dclocal_read_repair_chance table option has been deprecated and will be removed in version 4.0

dclocal_read_repair_chance table option has been deprecated and will be removed in version 4.0


Warnings :
dclocal_read_repair_chance table option has been deprecated and will be removed in version 4.0

dclocal_read_repair_chance table option has been deprecated and will be removed in version 4.0

INFO: the schema has been restored
starting apigee-cassandra-default-0 in default
starting apigee-cassandra-default-1 in default
starting apigee-cassandra-default-2 in default
84 95 106
waiting on waiting nodes $pid to finish  84
Activated service account credentials for: [apigee-cassandra-backup-svc@gce-myusername.iam.gserviceaccount.com]
Activated service account credentials for: [apigee-cassandra-backup-svc@gce-myusername.iam.gserviceaccount.com]
Activated service account credentials for: [apigee-cassandra-backup-svc@gce-myusername.iam.gserviceaccount.com]
INFO: restore downloaded  tarball and extracted the file from  gs://gce-myusername-apigee-cassandra-backup/apigeecluster/dc-1
INFO: restore downloaded  tarball and extracted the file from  gs://gce-myusername-apigee-cassandra-backup/apigeecluster/dc-1
INFO: restore downloaded  tarball and extracted the file from  gs://gce-myusername-apigee-cassandra-backup/apigeecluster/dc-1
INFO  12:02:28 Configuration location: file:/etc/cassandra/cassandra.yaml
โ€ฆ...

INFO  12:02:41 [Stream #e013ee80-5863-11e9-8458-353e9e3cb7f9] All sessions completed

Summary statistics:
   Connections per host    : 3
   Total files transferred : 2
   Total bytes transferred : 0.378KiB
   Total duration          : 5048 ms
   Average transfer rate   : 0.074KiB/s
   Peak transfer rate      : 0.075KiB/s

progress: [/10.32.1.155]0:1/1 100% 1:1/1 100% [/10.32.0.28]1:1/1 100% 0:1/1 100% [/10.32.3.220]0:1/1 100% 1:1/1 100% total: 100% 0.000KiB/s (avg: 0.074KiB/s)
INFO  12:02:41 [Stream #e013ee80-5863-11e9-8458-353e9e3cb7f9] All sessions completed
progress: [/10.32.1.155]0:1/1 100% 1:1/1 100% [/10.32.0.28]1:1/1 100% 0:1/1 100% [/10.32.3.220]0:1/1 100% 1:1/1 100% total: 100% 0.000KiB/s (avg: 0.074KiB/s)
INFO  12:02:41 [Stream #e013ee80-5863-11e9-8458-353e9e3cb7f9] All sessions completed
INFO  12:02:41 [Stream #e013ee80-5863-11e9-8458-353e9e3cb7f9] All sessions completed
INFO: ./apigee/data/cassandra/data/ks1/user-9fbae960571411e99652c7b15b2db6cc restored successfully
INFO: Restore 20190405011309 completed
INFO: ./apigee/data/cassandra/data/ks1/user-9fbae960571411e99652c7b15b2db6cc restored successfully
INFO: Restore 20190405011309 completed
waiting on waiting nodes $pid to finish  106
Restore finished

๋ฐฑ์—… ์ž‘์—… ํ™•์ธ

๋˜ํ•œ ๋ฐฑ์—… cronjob์ด ์˜ˆ์•ฝ๋˜๋ฉด ๋ฐฑ์—… ์ž‘์—…์„ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. cronjob์ด ์˜ˆ์•ฝ๋˜๋ฉด ๋‹ค์Œ๊ณผ ๊ฐ™์€ ์ถœ๋ ฅ์ด ํ‘œ์‹œ๋ฉ๋‹ˆ๋‹ค.

kubectl get pods

์ถœ๋ ฅ์€ ๋‹ค์Œ๊ณผ ๋น„์Šทํ•ฉ๋‹ˆ๋‹ค.

NAME                                       READY     STATUS      RESTARTS   AGE
apigee-cassandra-default-0                 1/1       Running     0          2h
apigee-cassandra-default-1                 1/1       Running     0          2h
apigee-cassandra-default-2                 1/1       Running     0          2h
apigee-cassandra-backup-1554515580-pff6s   0/1       Running     0          54s

๋ฐฑ์—… ๋กœ๊ทธ ํ™•์ธ

๋ฐฑ์—… ์ž‘์—…์€ ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.

  • schema.cql ํŒŒ์ผ์„ ๋งŒ๋“ญ๋‹ˆ๋‹ค.
  • ์Šคํ† ๋ฆฌ์ง€ ๋ฒ„ํ‚ท์— ์—…๋กœ๋“œํ•ฉ๋‹ˆ๋‹ค.
  • ๋…ธ๋“œ๋ฅผ ์—์ฝ”ํ•˜์—ฌ ๋ฐ์ดํ„ฐ๋ฅผ ๋ฐฑ์—…ํ•˜๊ณ  ๋™์‹œ์— ์—…๋กœ๋“œํ•ฉ๋‹ˆ๋‹ค.
  • ๋ชจ๋“  ๋ฐ์ดํ„ฐ๊ฐ€ ์—…๋กœ๋“œ๋  ๋•Œ๊นŒ์ง€ ๊ธฐ๋‹ค๋ฆฝ๋‹ˆ๋‹ค.
kubectl logs -f apigee-cassandra-backup-1554515580-pff6s

์ถœ๋ ฅ์€ ๋‹ค์Œ๊ณผ ๋น„์Šทํ•ฉ๋‹ˆ๋‹ค.

myusername-macbookpro:cassandra-backup-utility myusername$ kubectl logs -f apigee-cassandra-backup-1554577680-f9sc4
starting apigee-cassandra-default-0 in default
starting apigee-cassandra-default-1 in default
starting apigee-cassandra-default-2 in default
35 46 57
waiting on process  35
Activated service account credentials for: [apigee-cassandra-backup-svc@gce-myusername.iam.gserviceaccount.com]
Activated service account credentials for: [apigee-cassandra-backup-svc@gce-myusername.iam.gserviceaccount.com]
Activated service account credentials for: [apigee-cassandra-backup-svc@gce-myusername.iam.gserviceaccount.com]
Requested creating snapshot(s) for [all keyspaces] with snapshot name [20190406190808] and options {skipFlush=false}
Snapshot directory: 20190406190808
INFO: backup created cassandra snapshot 20190406190808
tar: Removing leading `/' from member names
/apigee/data/cassandra/data/ks1/mytest3-37bc2df0587811e98e8d875b0ed64754/snapshots/
/apigee/data/cassandra/data/ks1/mytest3-37bc2df0587811e98e8d875b0ed64754/snapshots/20190406190808/
/apigee/data/cassandra/data/ks1/mytest3-37bc2df0587811e98e8d875b0ed64754/snapshots/20190406190808/mc-1-big-Data.db
Requested creating snapshot(s) for [all keyspaces] with snapshot name [20190406190808] and options {skipFlush=false}
Requested creating snapshot(s) for [all keyspaces] with snapshot name [20190406190808] and options {skipFlush=false}
Snapshot directory: 20190406190808
INFO: backup created cassandra snapshot 20190406190808
tar: Removing leading `/' from member names
/apigee/data/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/snapshots/
/apigee/data/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/snapshots/20190406190808/
/apigee/data/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/snapshots/20190406190808/manifest.json
/apigee/data/cassandra/data/system/prepared_statements-18a9c2576a0c3841ba718cd529849fef/snapshots/
/apigee/data/cassandra/data/system/prepared_statements-18a9c2576a0c3841ba718cd529849fef/snapshots/20190406190808/
/apigee/data/cassandra/data/system/prepared_statements-18a9c2576a0c3841ba718cd529849fef/snapshots/20190406190808/manifest.json
/apigee/data/cassandra/data/system/range_xfers-55d764384e553f8b9f6e676d4af3976d/snapshots/
/apigee/data/cassandra/data/system/range_xfers-55d764384e553f8b9f6e676d4af3976d/snapshots/20190406190808/
/apigee/data/cassandra/data/system/range_xfers-55d764384e553f8b9f6e676d4af3976d/snapshots/20190406190808/manifest.json
/apigee/data/cassandra/data/system/peer_events-59dfeaea8db2334191ef109974d81484/snapshots/
/apigee/data/cassandra/data/system/peer_events-59dfeaea8db2334191ef109974d81484/snapshots/20190406190808/
/apigee/data/cassandra/data/system/peer_events-59dfeaea8db2334191ef109974d81484/snapshots/20190406190808/manifest.json
/apigee/data/cassandra/data/system/built_views-4b3c50a9ea873d7691016dbc9c38494a/snapshots/
/apigee/data/cassandra/data/system/built_views-4b3c50a9ea873d7691016dbc9c38494a/snapshots/20190406190808/
/apigee/data/cassandra/data/system/built_views-4b3c50a9ea873d7691016dbc9c38494a/snapshots/20190406190808/manifest.json
โ€ฆโ€ฆ
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-3-big-Filter.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-2-big-CompressionInfo.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-2-big-Index.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-3-big-Statistics.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-2-big-Data.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-1-big-Index.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-1-big-Statistics.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-2-big-TOC.txt
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-2-big-Statistics.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-1-big-Summary.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-1-big-Filter.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-2-big-Summary.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-3-big-Index.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/manifest.json
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-2-big-Filter.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-2-big-Digest.crc32
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-3-big-Summary.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-3-big-Data.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-1-big-TOC.txt
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/schema.cql
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-3-big-CompressionInfo.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-1-big-Digest.crc32
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-3-big-TOC.txt
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-1-big-Data.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-3-big-Digest.crc32
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-1-big-CompressionInfo.db
โ€ฆโ€ฆ
/tmp/tokens.txt
/ [1 files][    0.0 B/    0.0 B]
Operation completed over 1 objects.
/ [1 files][    0.0 B/    0.0 B]
Operation completed over 1 objects.
INFO: backup created tarball and transferred the file to gs://gce-myusername-apigee-cassandra-backup/apigeecluster/dc-1
INFO: removing cassandra snapshot
INFO: backup created tarball and transferred the file to gs://gce-myusername-apigee-cassandra-backup/apigeecluster/dc-1
INFO: removing cassandra snapshot
Requested clearing snapshot(s) for [all keyspaces]
INFO: Backup 20190406190808 completed
waiting on process  46
Requested clearing snapshot(s) for [all keyspaces]
INFO: Backup 20190406190808 completed
Requested clearing snapshot(s) for [all keyspaces]
waiting on process  57
INFO: Backup 20190406190808 completed
waiting result
to get schema from 10.32.0.28
INFO: /tmp/schema.cql has been generated
Activated service account credentials for: [apigee-cassandra-backup-svc@gce-myusername.iam.gserviceaccount.com]
tar: removing leading '/' from member names
tmp/schema.cql
Copying from <TDIN>...
/ [1 files][    0.0 B/    0.0 B]
Operation completed over 1 objects.
INFO: backup created tarball and transferred the file to gs://gce-myusername-apigee-cassandra-backup/apigeecluster/dc-1
finished uploading schema.cql