์ œํ•œ ์‹œ๊ฐ„ ์˜ค๋ฅ˜๋กœ API ํ˜ธ์ถœ ์‹คํŒจ

Apigee ๋ฐ Apigee Hybrid ๋ฌธ์„œ์ž…๋‹ˆ๋‹ค.
Apigee Edge ๋ฌธ์„œ ๋ณด๊ธฐ

์ฆ์ƒ

๋‹ค์Œ ์ฆ์ƒ ์ค‘ ํ•˜๋‚˜๊ฐ€ ๊ด€์ฐฐ๋  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

  1. ํด๋ผ์ด์–ธํŠธ ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜์— Apigee Hybrid์— ๋Œ€ํ•œ API ํ˜ธ์ถœ์˜ ์‘๋‹ต์œผ๋กœ ์ œํ•œ ์‹œ๊ฐ„ ์˜ค๋ฅ˜๊ฐ€ ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค.
  2. ํ•˜์ด๋ธŒ๋ฆฌ๋“œ ์„ค์น˜ ์ค‘์— ํด๋Ÿฌ์Šคํ„ฐ์— ๊ตฌ์„ฑ(overrides.yaml)์„ ์ ์šฉํ•˜๋Š” ๋™์•ˆ Error from server (invalid) ๋˜๋Š” The Job "apigee-resources-install" is invalid์™€ ๊ฐ™์€ ์˜ค๋ฅ˜๊ฐ€ ๊ด€์ฐฐ๋ฉ๋‹ˆ๋‹ค.

์˜ค๋ฅ˜ ๋ฉ”์‹œ์ง€

๋‹ค์Œ ์˜ค๋ฅ˜ ์ค‘ ํ•˜๋‚˜๊ฐ€ ๊ด€์ฐฐ๋  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

API ํ˜ธ์ถœ์— ๋Œ€ํ•œ ์˜ค๋ฅ˜ ์‘๋‹ต

Apigee Hybrid์— ๋Œ€ํ•œ API ์š”์ฒญ์ด ๋‹ค์Œ ์˜ค๋ฅ˜ ๋ฉ”์‹œ์ง€์™€ ํ•จ๊ป˜ ์‹คํŒจํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

* Connection failed
* connect to 34.84.67.39 port 443 failed: Operation timed out
* Failed to connect to example.apis.com port 443: Operation timed out
* Closing connection 0
curl: (7) Failed to connect to example.apis.com port 443: Operation timed out

ํด๋Ÿฌ์Šคํ„ฐ์— ๊ตฌ์„ฑ(overrides.yaml)์„ ์ ์šฉํ•  ๋•Œ ๊ด€์ฐฐ๋˜๋Š” ์˜ค๋ฅ˜

์„ค์น˜ ์ค‘ ํด๋Ÿฌ์Šคํ„ฐ์— ๊ตฌ์„ฑ(overrides.yaml ํŒŒ์ผ)์„ ์ ์šฉํ•  ๋•Œ ๋‹ค์Œ ์˜ค๋ฅ˜ ์ค‘ ํ•˜๋‚˜๊ฐ€ ๊ด€์ฐฐ๋  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

์˜ค๋ฅ˜ 1๋ฒˆ

helm upgrade operator apigee-operator/ \
  --install \
  --create-namespace \
  --namespace APIGEE_NAMESPACE \
  --atomic \
  -f OVERRIDES_FILE

...
...
Error from server (Invalid): error when applying patch:
to:
Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job"
Name: "istio-init-crd-10-1.4.6", Namespace: "istio-system"
to:
Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job"
Name: "istio-init-crd-11-1.4.6", Namespace: "istio-system"
to:
Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job"
Name: "istio-init-crd-14-1.4.6", Namespace: "istio-system"

์˜ค๋ฅ˜ 2๋ฒˆ

helm upgrade operator apigee-operator/ \
--install \
--create-namespace \
--namespace APIGEE_NAMESPACE \
--atomic \
-f OVERRIDES_FILE

...
...
The Job "apigee-resources-install" is invalid: spec.template: Invalid value:
core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"apigee-resources-install",
GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"",
Generation:0,

๊ฐ€๋Šฅํ•œ ์›์ธ

์ด๋Ÿฌํ•œ ์˜ค๋ฅ˜๋Š” istio-ingressgateway ์„œ๋น„์Šค๊ฐ€ pending ์ƒํƒœ์— ์žˆ๊ณ  ์•„๋ž˜ ํ‘œ์‹œ๋œ ๊ฒƒ์ฒ˜๋Ÿผ ์™ธ๋ถ€ IP ์ฃผ์†Œ์— ๋ฐ”์ธ๋”ฉ๋  ์ˆ˜ ์—†์„ ๋•Œ ๋ฐœ์ƒํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

kubectl get services -n istio-system
NAME                      TYPE         CLUSTER-IP   EXTERNAL-IP  PORT(S)             AGE
istio-ingressgateway      LoadBalancer 10.198.5.104 <pending>    15020:31927/TCP,    12h
                                                                 80:31381/TCP,
                                                                 443:31391/TCP,
                                                                 31400:31401/TCP,
                                                                 15443:32623/TCP

istio-ingressgateway ์„œ๋น„์Šค๊ฐ€ pending ์ƒํƒœ๊ฐ€ ๋˜๋Š” ๊ฐ€๋Šฅํ•œ ์›์ธ์€ ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.

์›์ธ ์„ค๋ช…
istio-system ๋„ค์ž„์ŠคํŽ˜์ด์Šค์—์„œ erroneous/pending ์ƒํƒœ์˜ ์ž‘์—… istio-system ๋„ค์ž„์ŠคํŽ˜์ด์Šค์—์„œ incomplete/erroneous ์ž‘์—…์€ istio-ingressgateway ์„œ๋น„์Šค๋ฅผ ์˜๊ตฌ์ ์œผ๋กœ pending ์ƒํƒœ๋กœ ๋งŒ๋“ค๊ณ  ์™ธ๋ถ€ IP ์ฃผ์†Œ์— ๋ฐ”์ธ๋”ฉ๋  ์ˆ˜ ์—†๋„๋ก ๋งŒ๋“ค ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
apigee-system ๋„ค์ž„์ŠคํŽ˜์ด์Šค์—์„œ erroneous/pending ์ƒํƒœ์˜ apigee-resources-install ์ž‘์—… apigee-system ๋„ค์ž„์ŠคํŽ˜์ด์Šค์—์„œ incomplete ์ž‘์—…์€ istio- ingressgateway ์„œ๋น„์Šค๋ฅผ ์˜๊ตฌ์ ์œผ๋กœ pending ์ƒํƒœ๋กœ ๋งŒ๋“ค๊ณ  ์™ธ๋ถ€ IP ์ฃผ์†Œ์— ๋ฐ”์ธ๋”ฉ๋  ์ˆ˜ ์—†๋„๋ก ๋งŒ๋“ค ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
์™ธ๋ถ€ ๋ถ€ํ•˜ ๋ถ„์‚ฐ๊ธฐ์— ํ• ๋‹น๋œ ์ž˜๋ชป๋œ IP ์ฃผ์†Œ ๋ฒ”์œ„ ์ž˜๋ชป๋œ IP ์ฃผ์†Œ ๋ฒ”์œ„๊ฐ€ istio-operator.yaml ํŒŒ์ผ์— ๊ตฌ์„ฑ๋˜์–ด istio-ingressgateway ์„œ๋น„์Šค๋ฅผ pending ์ƒํƒœ๋กœ ๋งŒ๋“ค๊ณ  ์„ค์น˜ ์ค‘ ์™ธ๋ถ€ IP ์ฃผ์†Œ์— ๋ฐ”์ธ๋”ฉ๋  ์ˆ˜ ์—†๋„๋ก ๋งŒ๋“ค ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

์›์ธ: istio-system ๋„ค์ž„์ŠคํŽ˜์ด์Šค์—์„œ erroneous/pending ์ƒํƒœ์˜ ์ž‘์—…

์ง„๋‹จ

  1. ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ istio-system ๋„ค์ž„์ŠคํŽ˜์ด์Šค์—์„œ ์ž‘์—…์˜ ์ƒํƒœ๋ฅผ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.
    kubectl get jobs -n istio-system
    
  2. ์ž‘์—… ์ƒํƒœ๋Š” complete์—ฌ์•ผ ํ•ฉ๋‹ˆ๋‹ค. ์ž‘์—… ์ƒํƒœ๊ฐ€ erroneous/pending ์ƒํƒœ์ด๋ฉด ์ด๊ฒƒ์ด ๋ฌธ์ œ์˜ ์›์ธ์ž…๋‹ˆ๋‹ค.

ํ•ด๊ฒฐ ๋ฐฉ๋ฒ•

  1. ์ž‘์—…์ด pending ๋˜๋Š” erroneous ์ƒํƒœ์ด๋ฉด ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ์ด๋ฅผ ์‚ญ์ œํ•ฉ๋‹ˆ๋‹ค.
    kubectl -n istio-system delete job JOB_NAME_FROM_STEP_1
    
  2. overrides.yaml ํŒŒ์ผ์„ ์ ์šฉํ•˜์—ฌ ์„ค์น˜๋ฅผ ๋‹ค์‹œ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค.

    Helm์„ ์‚ฌ์šฉํ•˜์—ฌ apigee-serving-cert๋ฅผ ์—…๋ฐ์ดํŠธํ•ฉ๋‹ˆ๋‹ค.

    helm install operator apigee-operator/
    --namespace APIGEE_NAMESPACE \
    --atomic \
    -f OVERRIDES_FILE \
    --dry-run=server

    ์ž‘์—…์ด ์‹คํŒจ ์‹œ ๋กค๋ฐฑ๋˜๋„๋ก --atomic์„ ๋น„๋กฏํ•˜์—ฌ ํ‘œ์‹œ๋œ ๋ชจ๋“  ์„ค์ •์„ ํฌํ•จํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

    ์ฐจํŠธ๋ฅผ ์„ค์น˜ํ•ฉ๋‹ˆ๋‹ค.

    helm upgrade operator apigee-operator/
    --namespace APIGEE_NAMESPACE \
    --atomic \
    -f OVERRIDES_FILE

์›์ธ: apigee-system ๋„ค์ž„์ŠคํŽ˜์ด์Šค์˜ apigee-resources-install ์ž‘์—…์ด ์˜ค๋ฅ˜ ์ƒํƒœ์ผ ์ˆ˜ ์žˆ์Œ

์ง„๋‹จ

  1. ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ apigee-system ๋„ค์ž„์ŠคํŽ˜์ด์Šค์—์„œ ์ž‘์—…์˜ ์ƒํƒœ๋ฅผ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.
    kubectl get jobs -n apigee-system
    
  2. ์ž‘์—… ์ƒํƒœ๋Š” complete์—ฌ์•ผ ํ•ฉ๋‹ˆ๋‹ค. ์ž‘์—… ์ƒํƒœ๊ฐ€ erroneous/pending ์ƒํƒœ์ด๋ฉด ์ด๊ฒƒ์ด ๋ฌธ์ œ์˜ ์›์ธ์ž…๋‹ˆ๋‹ค. ๋‹ค์Œ ์ƒ˜ํ”Œ ์ถœ๋ ฅ์€ apigee-resources-install ์ž‘์—…์ด ์„ฑ๊ณต์ ์œผ๋กœ ์™„๋ฃŒ๋œ ๊ฒƒ์„ ๋ณด์—ฌ์ค๋‹ˆ๋‹ค.
    kubectl get jobs  -n apigee-system
    NAME                       COMPLETIONS   DURATION   AGE
    apigee-resources-install   1/1           23s        16d
    

ํ•ด๊ฒฐ ๋ฐฉ๋ฒ•

  1. ์ž‘์—…์ด pending ๋˜๋Š” erroneous ์ƒํƒœ์ด๋ฉด ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ์ด๋ฅผ ์‚ญ์ œํ•ฉ๋‹ˆ๋‹ค.
    kubectl -n apigee-system delete job JOB_NAME_FROM_STEP_1
    
  2. overrides.yaml ํŒŒ์ผ์„ ์ ์šฉํ•˜์—ฌ ์„ค์น˜๋ฅผ ๋‹ค์‹œ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค.
    apigeectl apply -f overrides.yaml
    

์›์ธ: ์™ธ๋ถ€ ๋ถ€ํ•˜ ๋ถ„์‚ฐ๊ธฐ์— ํ• ๋‹น๋œ ์ž˜๋ชป๋œ IP ์ฃผ์†Œ ๋ฒ”์œ„

์ง„๋‹จ

  1. istio- operator.yaml ํŒŒ์ผ์—์„œ ๋ถ€ํ•˜ ๋ถ„์‚ฐ๊ธฐ์— ๋Œ€ํ•ด ๊ตฌ์„ฑ๋œ IP ์ฃผ์†Œ๋ฅผ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค. ์˜ˆ๋ฅผ ๋“ค์–ด ๋‹ค์Œ ์Šค๋‹ˆํŽซ์€ istio-operator.yaml ํŒŒ์ผ์—์„œ IP ์ฃผ์†Œ๊ฐ€ ๊ตฌ์„ฑ๋œ ์œ„์น˜๋ฅผ ๋ณด์—ฌ์ค๋‹ˆ๋‹ค.
    -name: istio-ingressgateway
      enabled: true
      k8s:
        service:
          type: LoadBalancer
         loadBalancerIP: 10.195.24.23
  2. istio-ingressgateway ์„œ๋น„์Šค๋Š” istio-operator.yaml ํŒŒ์ผ์—์„œ ์œ ํ˜•๋ณ„๋กœ ํ‘œ์‹œ๋œ ๋ถ€ํ•˜ ๋ถ„์‚ฐ๊ธฐ๋กœ ๊ตฌ์„ฑ๋ฉ๋‹ˆ๋‹ค. ASM ์„ค์น˜ ์ค‘ ๋ถ€ํ•˜ ๋ถ„์‚ฐ๊ธฐ๋Š” ๊ตฌ์„ฑ๋œ IP ์ฃผ์†Œ๋กœ ์ƒ์„ฑ๋˜๊ณ  istio- ingressgateway ์„œ๋น„์Šค์™€ ํ†ต์‹ ํ•˜๋„๋ก ์—ฐ๊ฒฐ๋ฉ๋‹ˆ๋‹ค. ๋”ฐ๋ผ์„œ ๊ตฌ์„ฑ๋œ IP ์ฃผ์†Œ๊ฐ€ ์˜ฌ๋ฐ”๋ฅด๊ณ  ๋ถ€ํ•˜ ๋ถ„์‚ฐ๊ธฐ์— ๋Œ€ํ•ด ์˜ˆ์•ฝ๋˜์–ด ์žˆ์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.
  3. ๋„คํŠธ์›ŒํฌํŒ€์— ์—ฐ๋ฝํ•˜์—ฌ loadBalancerIP์— ๋Œ€ํ•ด ๊ตฌ์„ฑ๋œ IP ์ฃผ์†Œ๊ฐ€ ์˜ฌ๋ฐ”๋ฅธ์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค. ์˜ฌ๋ฐ”๋ฅด์ง€ ์•Š์œผ๋ฉด ๋ถ€ํ•˜ ๋ถ„์‚ฐ๊ธฐ ์„œ๋น„์Šค๊ฐ€ IP ์ฃผ์†Œ์— ๋ฐ”์ธ๋”ฉํ•  ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค. ์ด๋ ‡๊ฒŒ ํ•˜๋ฉด istio-ingressgateway ์„œ๋น„์Šค๊ฐ€ ์˜๊ตฌ์ ์œผ๋กœ pending ์ƒํƒœ๊ฐ€ ๋ฉ๋‹ˆ๋‹ค.

ํ•ด๊ฒฐ ๋ฐฉ๋ฒ•

  1. ๋„คํŠธ์›Œํฌ ํŒ€๊ณผ ํ˜‘๋ ฅํ•˜์—ฌ istio- operator.yaml ํŒŒ์ผ์—์„œ ์˜ฌ๋ฐ”๋ฅธ IP ์ฃผ์†Œ๋ฅผ ๊ตฌ์„ฑํ•ฉ๋‹ˆ๋‹ค.
  2. Apigee ์ธ๊ทธ๋ ˆ์Šค ๊ฒŒ์ดํŠธ์›จ์ด์˜ ์„ค์น˜๋ฅผ ๋‹ค์‹œ ์‹คํ–‰ํ•˜๊ณ  overrides.yaml ํŒŒ์ผ์„ ์ ์šฉํ•ฉ๋‹ˆ๋‹ค.
    helm upgrade $ORG_NAME apigee-org/ \
      --install \
      --namespace APIGEE_NAMESPACE \
      --atomic \
      -f OVERRIDES_FILE

์ง„๋‹จ ์ •๋ณด ์ˆ˜์ง‘ ํ•„์š”

์œ„ ์•ˆ๋‚ด๋ฅผ ๋”ฐ๋ฅธ ํ›„์—๋„ ๋ฌธ์ œ๊ฐ€ ์ง€์†๋˜๋ฉด ๋‹ค์Œ ์ง„๋‹จ ์ •๋ณด๋ฅผ ์ˆ˜์ง‘ํ•œ ํ›„ Google Cloud Customer Care์— ๋ฌธ์˜ํ•˜์„ธ์š”.

  1. Google Cloud ํ”„๋กœ์ ํŠธ ID
  2. Apigee Hybrid ์กฐ์ง์˜ ์ด๋ฆ„
  3. Kubernetes ํด๋Ÿฌ์Šคํ„ฐ ์ด๋ฆ„
  4. kubernetes ํด๋Ÿฌ์Šคํ„ฐ๊ฐ€ ๋‹ค๋ฅธ Google Cloud ํ”„๋กœ์ ํŠธ์— ์žˆ๋Š” ๊ฒฝ์šฐ Google Cloud ํ”„๋กœ์ ํŠธ ์ด๋ฆ„
  5. overrides.yaml ํŒŒ์ผ
  6. ASM ์„ค์น˜ ์ค‘์— ์‚ฌ์šฉ๋œ Istio-operator .yaml ํŒŒ์ผ
  7. istio-system ๋„ค์ž„์ŠคํŽ˜์ด์Šค์˜ ๊ฐ istio-ingressgateway Pod์—์„œ ๋กœ๊ทธ๋ฅผ ์ˆ˜์ง‘ํ•ฉ๋‹ˆ๋‹ค.
    kubectl logs NAME_OF_ISTIO_INGRESSGATEWAY_POD -n istio-system > /tmp/NAME_OF_ISTIO_INGRESSGATEWAY_POD.log
    
  8. istio-system ๋„ค์ž„์ŠคํŽ˜์ด์Šค์—์„œ ๊ฐ Pod์˜ ์„ค๋ช…์„ ์ˆ˜์ง‘ํ•ฉ๋‹ˆ๋‹ค.
    kubectl describe pod NAME_OF_ISTIO_INGRESSGATEWAY_POD -n istio-system > /tmp/NAME_OF_ISTIO_INGRESSGATEWAY_POD.yaml
    
  9. istio-system ๋„ค์ž„์ŠคํŽ˜์ด์Šค์—์„œ ์„œ๋น„์Šค ๋ชฉ๋ก์„ ์ˆ˜์ง‘ํ•ฉ๋‹ˆ๋‹ค.
    kubectl get svc -n istio-system