Cassandra TLS 인증서 확인 실패

Apigee 및 Apigee Hybrid 문서입니다.
Apigee Edge 문서 보기

증상

Cassandra 포드의 TLS 인증서 확인 프로세스가 다음과 유사한 오류와 함께 실패할 수 있습니다.

Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:456)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:323)
at sun.security.validator.Validator.validate(Validator.java:271)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:315)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:278)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:261)
at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:698)
...
Suppressed: javax.net.ssl.SSLHandshakeException: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1309)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1270)
...

가능한 원인

기본 요건

• Apigee 클러스터에 액세스하려면 kubectl을 설치하고 구성해야 합니다.
• JSON 콘텐츠의 형식을 지정하려면 jq가 필요합니다.
• TLS 인증서를 출력하려면 keytool이 필요합니다.
• Cert Manager를 사용하여 인증서를 재발급하려면 cmctl이 필요합니다.

원인: Apigee 클러스터 간에 Apigee CA 인증서가 일치하지 않습니다.

진단

1. 다음 명령어를 사용하여 apigee-ca 보안 비밀을 읽고 모든 클러스터의 Apigee CA 인증서를 출력합니다.

   kubectl -n cert-manager get secret apigee-ca -o json | jq -r '.data["tls.crt"]' | base64 -d | keytool -printcert | grep Version -B 10

   출력 예시는 다음과 같습니다.

   kubectl -n cert-manager get secret apigee-ca -o json | jq -r '.data["tls.crt"]' | base64 -d | keytool -printcert | grep Version -B 10
   Owner: CN=apigee-hybrid, O=apigee + O=cluster.local
   Issuer: CN=apigee-hybrid, O=apigee + O=cluster.local
   Serial number: afcc2ef957cebfd52b118b0b1622021
   Valid from: Wed Oct 30 03:09:23 UTC 2024 until: Sat Oct 28 03:09:23 UTC 2034
   Certificate fingerprints:
           SHA1: 32:D9:77:54:B1:FC:CB:6C:9E:28:C1:04:25:49:0D:F5:7C:88:A5:6C
           SHA256: 7C:97:31:3B:56:CD:A3:EF:88:A7:DC:17:AE:D5:F2:A7:6A:48:B3:FC:AA:04:F0:6B:9F:43:C4:5C:6E:15:DE:37
   Signature algorithm name: SHA256withECDSA
   Subject Public Key Algorithm: 256-bit EC (secp256r1) key
   Version: 3

2. apigee-cassandra-default-tls 보안 비밀을 읽고 Cassandra 인증서를 생성할 때 위의 Apigee CA 인증서가 사용되었는지 확인합니다. apigee-cassandra-default-tls 보안 비밀에는 ca.crt 아래에 Apigee CA 인증서가 포함되어 있습니다.

   kubectl -n apigee get secret apigee-cassandra-default-tls -o json | jq -r '.data["ca.crt"]' | base64 -d | keytool -printcert | grep Version -B 10

   출력 예시는 다음과 같습니다.

   kubectl -n apigee get secret apigee-cassandra-default-tls -o json | jq -r '.data["ca.crt"]' | base64 -d | keytool -printcert | grep Version -B 10
   Owner: CN=apigee-hybrid, O=apigee + O=cluster.local
   Issuer: CN=apigee-hybrid, O=apigee + O=cluster.local
   Serial number: afcc2ef957cebfd52b118b0b1622021
   Valid from: Wed Oct 30 03:09:23 UTC 2024 until: Sat Oct 28 03:09:23 UTC 2034
   Certificate fingerprints:
           SHA1: 32:D9:77:54:B1:FC:CB:6C:9E:28:C1:04:25:49:0D:F5:7C:88:A5:6C
           SHA256: 7C:97:31:3B:56:CD:A3:EF:88:A7:DC:17:AE:D5:F2:A7:6A:48:B3:FC:AA:04:F0:6B:9F:43:C4:5C:6E:15:DE:37
   Signature algorithm name: SHA256withECDSA
   Subject Public Key Algorithm: 256-bit EC (secp256r1) key
   Version: 3

3. 위 예에서 apigee-ca 보안 비밀에 있는 Apigee CA 인증서의 일련번호는 apigee-cassandra-default-tls 보안 비밀에 있는 Apigee CA 인증서(afcc2ef957cebfd52b118b0b1622021)와 일치합니다. 이를 통해 Cassandra 인증서가 동일한 Apigee CA 인증서로 서명되었음을 확인할 수 있습니다. 아래 단계에 따라 이를 추가로 확인할 수 있습니다.
4. Apigee CA 인증서 pem 파일을 추출합니다.

   kubectl -n cert-manager get secret apigee-ca -o json | jq -r '.data["tls.crt"]' | base64 -d > apigee-ca.crt

   출력 예시는 다음과 같습니다.

   kubectl -n cert-manager get secret apigee-ca -o json | jq -r '.data["tls.crt"]' | base64 -d > apigee-ca.crt
   cat apigee-ca.crt
   -----BEGIN CERTIFICATE-----
   MIIBvjCCAWSgAwIBAgIQCvzC75V86/1SsRiwsWIgITAKBggqhkjOPQQDAjA/MSUw
   DQYDVQQKEwZhcGlnZWUwFAYDVQQKEw1jbHVzdGVyLmxvY2FsMRYwFAYDVQQDEw1h
   cGlnZWUtaHlicmlkMB4XDTI0MTAzMDAzMDkyM1oXDTM0MTAyODAzMDkyM1owPzEl
   MA0GA1UEChMGYXBpZ2VlMBQGA1UEChMNY2x1c3Rlci5sb2NhbDEWMBQGA1UEAxMN
   YXBpZ2VlLWh5YnJpZDBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABNSow7pxNvjj
   R/jV66nY/w/tn22tu7oXyZS8tAFBnP7D2fFfIdk4tJub3gw/CsoyNa1cKXwAt7Tw
   SLp1iGJ3CY+jQjBAMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0G
   A1UdDgQWBBRSjN/cNNbg2kvmddskzdurglxuwTAKBggqhkjOPQQDAgNIADBFAiBp
   pCgNNC8TVEgF8jR5RK9dXZJRcNY39nFY4DqbH6bUJwIhAPdzx5gee3BIWYwlQAYX
   CgtCf4blLNq3KlBWTO993XoY
   -----END CERTIFICATE-----

5. Cassandra 인증서 pem 파일을 추출합니다.

   kubectl -n apigee get secrets apigee-cassandra-default-tls -o json | jq -r '.data["tls.crt"]' | base64 -d > apigee-cassandra-default-tls.crt

   출력 예시는 다음과 같습니다.

   kubectl -n apigee get secrets apigee-cassandra-default-tls -o json | jq -r '.data["tls.crt"]' | base64 -d > apigee-cassandra-default-tls.crt
   cat apigee-cassandra-default-tls.crt
   -----BEGIN CERTIFICATE-----
   MIIDSDCCAu6gAwIBAgIQZcYk/VOfGUQEzpLbAvyyNjAKBggqhkjOPQQDAjA/MSUw
   DQYDVQQKEwZhcGlnZWUwFAYDVQQKEw1jbHVzdGVyLmxvY2FsMRYwFAYDVQQDEw1h
   cGlnZWUtaHlicmlkMB4XDTI0MTAzMDAzMTAyMFoXDTM0MTAyODAzMTAyMFowPDE6
   MDgGA1UEAxMxYXBpZ2VlLWNhc3NhbmRyYS1kZWZhdWx0LmFwaWdlZS5zdmMuY2x1
   c3Rlci5sb2NhbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAM6k8YyB
   m/AV9cgexU8fZ4OFw8M72oxWEF44sFezZB7NpCqIFBxAM/7iL0tF2qU4S4gpcabD
   bn30fKKID8651Kytc7KHGT13Nlj9vQRjd0HJD8Qa8YtRcmGKtp+1fbQOcMPxvuNA
   CzaQyuPwieYKc6D9DpDDkPPCmjVwfaxHmNpdswrt0NQbSecg/xZPXbpzOZ6bUFha
   2vTvSTomiDKIPGhWrMnEMJDjFyjpdYND74HnYgw1XGnC4SQNts/kvXligbVmW+Rz
   oyV7n99eN6cE5J/FHDgiHrBRZUw8ujP2l/p7Y96NcMBnXCsQu6RsCDltXqX1f1pG
   sIjUAFAZZvM0pDECAwEAAaOCAQIwgf8wDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQW
   MBQGCCsGAQUFBwMCBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaA
   FFKM39w01uDaS+Z12yTN26uCXG7BMIGeBgNVHREEgZYwgZOCGGFwaWdlZS1jYXNz
   YW5kcmEtZGVmYXVsdIIfYXBpZ2VlLWNhc3NhbmRyYS1kZWZhdWx0LmFwaWdlZYIj
   YXBpZ2VlLWNhc3NhbmRyYS1kZWZhdWx0LmFwaWdlZS5zdmOCMWFwaWdlZS1jYXNz
   YW5kcmEtZGVmYXVsdC5hcGlnZWUuc3ZjLmNsdXN0ZXIubG9jYWwwCgYIKoZIzj0E
   AwIDSAAwRQIhANt7WYfSbS4a14Fvf3IXcG+/p3iEGg61suK8jOxtgJMyAiBG3z7Y
   kgR7SWNzSoom4Oznq9NSub7v75kfQJFKEtP0Mg==
   -----END CERTIFICATE-----

6. Apigee CA 인증서를 사용하여 Cassandra 인증서를 확인합니다.

   openssl verify -CAfile apigee-ca.crt apigee-cassandra-default-tls.crt

   성공적인 출력의 예는 다음과 같습니다.

   openssl verify -CAfile apigee-ca.crt apigee-cassandra-default-tls.crt
   apigee-cassandra-default-tls.crt: OK

   실패한 출력의 예는 다음과 같습니다.

   openssl verify -CAfile apigee-ca.crt apigee-cassandra-default-tls.crt
   CN = apigee-cassandra-default.apigee.svc.cluster.local
   error 20 at 0 depth lookup: unable to get local issuer certificate
   error apigee-cassandra-default-tls.crt: verification failed

해결 방법

1. 올바른 Apigee CA 인증서가 있는 Apigee 클러스터를 선택합니다.
2. 해당 클러스터에서 Apigee CA 인증서 보안 비밀을 파일로 내보냅니다.

   kubectl -n cert-manager get secret apigee-ca -o yaml > apigee-ca.yaml

3. 클러스터를 한 번에 하나씩 선택하여 위의 Apigee CA 인증서 보안 비밀을 다른 모든 클러스터에 적용하고 모든 클러스터에서 나머지 모든 단계를 수행합니다.

   kubectl -n cert-manager apply -f apigee-ca.yaml

4. apigee 네임스페이스에서 사용 가능한 모든 기존 인증서를 백업 파일로 내보냅니다.

   kubectl -n apigee get certificates --all -o yaml > all-certificates.yaml

5. 다음 cmctl 명령어를 실행하여 apigee 네임스페이스에 있는 모든 인증서를 재발급합니다.

   cmctl renew --namespace=apigee --all

   출력 예시는 다음과 같습니다.

   cmctl renew --namespace=apigee --all
   
     Manually triggered issuance of Certificate apigee/apigee-cassandra-default
     Manually triggered issuance of Certificate apigee/apigee-cassandra-schema-setup-demo-hybrid-de-5fdc6d2
     Manually triggered issuance of Certificate apigee/apigee-cassandra-schema-val-demo-hybrid-de-5fdc6d2
     Manually triggered issuance of Certificate apigee/apigee-cassandra-user-setup-demo-hybrid-de-5fdc6d2
     Manually triggered issuance of Certificate apigee/apigee-connect-agent-demo-hybrid-de-5fdc6d2
     Manually triggered issuance of Certificate apigee/apigee-datastore-guardrails-tls
     Manually triggered issuance of Certificate apigee/apigee-istiod
     Manually triggered issuance of Certificate apigee/apigee-mart-demo-hybrid-de-5fdc6d2
     Manually triggered issuance of Certificate apigee/apigee-metrics-adapter-apigee-telemetry
     Manually triggered issuance of Certificate apigee/apigee-redis-default
     Manually triggered issuance of Certificate apigee/apigee-redis-envoy-default
     Manually triggered issuance of Certificate apigee/apigee-runtime-demo-hybrid-de-dev-b276d3f
     Manually triggered issuance of Certificate apigee/apigee-serving-cert
     Manually triggered issuance of Certificate apigee/apigee-synchronizer-demo-hybrid-de-dev-b276d3f
     Manually triggered issuance of Certificate apigee/apigee-udca-demo-hybrid-de-5fdc6d2
     Manually triggered issuance of Certificate apigee/apigee-watcher-demo-hybrid-de-5fdc6d2

   이 단계에서는 새로 가져온 Apigee CA 인증서를 사용하여 모든 Apigee 런타임 인증서를 재발급하여 문제를 해결합니다.

6. 모든 인증서의 발급 날짜를 UTC 시간과 비교하여 재발급되었는지 확인합니다.

   kubectl get certificates -n apigee -o json | jq -r '.items[] | "\(.metadata.name): \(.status.notBefore)"'
   date -u

   출력 예시는 다음과 같습니다.

   kubectl get certificates -n apigee -o json | jq -r '.items[] | "\(.metadata.name): \(.status.notBefore)"'
   
     apigee-cassandra-default: 2024-12-16T04:19:58Z
     apigee-cassandra-schema-setup-demo-hybrid-de-5fdc6d2: 2024-12-16T04:19:58Z
     apigee-cassandra-schema-val-demo-hybrid-de-5fdc6d2: 2024-12-16T04:19:58Z
     apigee-cassandra-user-setup-demo-hybrid-de-5fdc6d2: 2024-12-16T04:19:59Z
     apigee-connect-agent-demo-hybrid-de-5fdc6d2: 2024-12-16T04:20:00Z
     apigee-datastore-guardrails-tls: 2024-12-16T04:20:01Z
     apigee-istiod: 2024-12-16T04:20:02Z
     apigee-mart-demo-hybrid-de-5fdc6d2: 2024-12-16T04:20:02Z
     apigee-metrics-adapter-apigee-telemetry: 2024-12-16T04:20:03Z
     apigee-redis-default: 2024-12-16T04:20:04Z
     apigee-redis-envoy-default: 2024-12-16T04:20:04Z
     apigee-runtime-demo-hybrid-de-dev-b276d3f: 2024-12-16T04:20:04Z
     apigee-serving-cert: 2024-12-16T04:20:04Z
     apigee-synchronizer-demo-hybrid-de-dev-b276d3f: 2024-12-16T04:20:05Z
     apigee-udca-demo-hybrid-de-5fdc6d2: 2024-12-16T04:20:06Z
     apigee-watcher-demo-hybrid-de-5fdc6d2: 2024-12-16T04:20:07Z
   
     date -u
     Mon Dec 16 04:23:45 AM UTC 2024

7. 모든 인증서의 만료일을 확인하고 적절하게 연장되었는지 확인합니다.

   kubectl get certificates -n apigee -o json | jq -r '.items[] | "\(.metadata.name): \(.status.notAfter)"'

   출력 예시는 다음과 같습니다.

   kubectl get certificates -n apigee -o json | jq -r '.items[] | "\(.metadata.name): \(.status.notAfter)"'
   
     apigee-cassandra-default: 2034-12-14T04:19:58Z
     apigee-cassandra-schema-setup-demo-hybrid-de-5fdc6d2: 2034-12-14T04:19:58Z
     apigee-cassandra-schema-val-demo-hybrid-de-5fdc6d2: 2034-12-14T04:19:58Z
     apigee-cassandra-user-setup-demo-hybrid-de-5fdc6d2: 2034-12-14T04:19:59Z
     apigee-connect-agent-demo-hybrid-de-5fdc6d2: 2034-12-14T04:20:00Z
     apigee-datastore-guardrails-tls: 2024-12-16T05:20:01Z
     apigee-istiod: 2024-12-18T04:20:02Z
     apigee-mart-demo-hybrid-de-5fdc6d2: 2034-12-14T04:20:02Z
     apigee-metrics-adapter-apigee-telemetry: 2034-12-14T04:20:03Z
     apigee-redis-default: 2034-12-14T04:20:04Z
     apigee-redis-envoy-default: 2034-12-14T04:20:04Z
     apigee-runtime-demo-hybrid-de-dev-b276d3f: 2034-12-14T04:20:04Z
     apigee-serving-cert: 2025-03-16T04:20:04Z
     apigee-synchronizer-demo-hybrid-de-dev-b276d3f: 2034-12-14T04:20:05Z
     apigee-udca-demo-hybrid-de-5fdc6d2: 2034-12-14T04:20:06Z
     apigee-watcher-demo-hybrid-de-5fdc6d2: 2034-12-14T04:20:07Z

진단 정보 수집 필요

위 안내를 따른 후에도 문제가 지속되면 다음 진단 정보를 수집한 후 Google Cloud Customer Care에 문의하세요.

1. Google Cloud 프로젝트 ID
2. Apigee Hybrid 조직
3. 민감한 정보를 마스킹하는 소스 및 새 리전의 overrides.yaml 파일
4. Apigee Hybrid 수집 필요의 명령어 출력
5. Apigee Hybrid Cassandra 수집 필요의 명령어 출력