NAME SEVERITY AGE URL
control-plane-connectivity-failure Error 1hr https://cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/no-network-connectivity
{"level":"error","ts":1675232549.5093117,"caller":"watcher/watcher.go:60","msg":"error during watch","name":"mpstatus","error":"INTERNAL: error sending mp status to mgmt: INTERNAL: failed to send runtime status Post \"https://apigee.googleapis.com/v1/organizations/example-hybrid-dev/instances/apigee-hybrid-australia-southeast1:reportStatus\": dial tcp 172.217.24.42:443: i/o timeout","stacktrace":"edge-internal.git.corp.google.com/apigee-watcher.git/watcher. (*Watcher).Start.func1.1\n\t/go/src/edge-internal/apigee-watcher/watcher/watcher.go:60"}
[[["์ดํดํ๊ธฐ ์ฌ์","easyToUnderstand","thumb-up"],["๋ฌธ์ ๊ฐ ํด๊ฒฐ๋จ","solvedMyProblem","thumb-up"],["๊ธฐํ","otherUp","thumb-up"]],[["์ดํดํ๊ธฐ ์ด๋ ค์","hardToUnderstand","thumb-down"],["์๋ชป๋ ์ ๋ณด ๋๋ ์ํ ์ฝ๋","incorrectInformationOrSampleCode","thumb-down"],["ํ์ํ ์ ๋ณด/์ํ์ด ์์","missingTheInformationSamplesINeed","thumb-down"],["๋ฒ์ญ ๋ฌธ์ ","translationIssue","thumb-down"],["๊ธฐํ","otherDown","thumb-down"]],["์ต์ข ์ ๋ฐ์ดํธ: 2025-09-05(UTC)"],[[["\u003cp\u003eThis document troubleshoots network connectivity issues between Apigee hybrid runtime plane components and the Apigee control plane, applicable only to Apigee hybrid users.\u003c/p\u003e\n"],["\u003cp\u003eSymptoms of network connectivity issues include non-populated data on the Apigee UI, failed API proxy deployments, and Management API request failures.\u003c/p\u003e\n"],["\u003cp\u003ePotential causes for these network issues include a lack of a network route to the internet, unallow-listed Apigee API endpoints, disabled Private Google Access on GKE, or other unknown network failures.\u003c/p\u003e\n"],["\u003cp\u003eResolutions involve establishing network routes, allow-listing Apigee API endpoints, enabling Private Google Access, or working with network administrators to resolve other network issues, depending on the identified root cause.\u003c/p\u003e\n"],["\u003cp\u003eIf issues persist, gather diagnostic information such as Google Cloud Project ID, organization name, network test outputs, \u003ccode\u003eoverrides.yaml\u003c/code\u003e, pod status, and cluster-info dumps, before contacting Google Cloud Customer Care.\u003c/p\u003e\n"]]],[],null,["# No network connectivity between runtime plane and control plane\n\n*You're viewing **Apigee** and **Apigee hybrid** documentation.\nThere is no equivalent\n[Apigee Edge](https://docs.apigee.com/api-platform/get-started/get-started) documentation for this topic.*\n| **Note:** This document is applicable for Apigee hybrid users only.\n\nSymptoms\n--------\n\n- API products, Developers, Apps do not get populated on the Apigee UI.\n- API proxy deployments do not complete.\n- Management API requests executed for reading and writing API products, Developers, and Apps fail.\n\nError messages\n--------------\n\n\nThis section describes the possible error messages displayed when there is\nno network connectivity.\n\n### ApigeeIssue\n\n\nIf this issue was reported as an ApigeeIssue, when the following command\nis run: \n\n```\nkubectl -n APIGEE_NAMESPACE get apigeeissues\n```\n\n\nwhere \u003cvar translate=\"no\"\u003eAPIGEE_NAMESPACE\u003c/var\u003e is the name for\na grouping of Kubernetes resources.\n\n\nthe following error code is displayed: \n\n```text\nNAME SEVERITY AGE URL\ncontrol-plane-connectivity-failure Error 1hr https://cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/no-network-connectivity\n```\n\n### API products, Developers, and Apps Apigee UI pages\n\n\nThe API products, Developers, and Apps Apigee UI pages display the\nfollowing error: \n\n```actionscript-3\nError: no connections available from the Apigee connect agent(s):\nrefer to documentation to triage further.\n```\n\n### apigee-synchronizer logs\n\n\nThe following error message can be seen on `apigee-synchronizer`\nlogs: \n\n```gdscript\n{\"level\":\"SEVERE\",\"thread\":\"Apigee-Timer-3\",\"mdc\":{\"action\":\"SYNC\",\"env\":\"dev\",\n \"org\":\"example-hybrid-dev\"},\"className\":\"com.apigee.httpclient.adaptor.RequestAdaptor\",\n \"method\":\"doSend\",\"severity\":\"SEVERE\",\"message\":\"request failed\n [CONTEXT ratelimit_period\\u003d\\\"1 MINUTES [skipped: 1]\\\" ]\",\n \"formattedDate\":\"2023-02-01T06:27:48.700Z\",\"logger\":\"SERVICES.HTTPCLIENTSERVICE\",\n \"exceptionStackTrace\":\"java.net.SocketTimeoutException: connect timed out\n... com.apigee.httpclient.adaptor.RequestAdaptor.send(RequestAdaptor.java:251)\nat com.apigee.hybrid.runtime.contract.sync.context.HttpContractDownloader.download(HttpContractDownloader.java:84)\nat com.apigee.hybrid.runtime.contract.sync.context.ControlPlaneReplicationContext.getContract(ControlPlaneReplicationContext.java:66)\n```\n\n### apigee-watcher logs\n\n\nThe following error message can be seen on `apigee-watcher` logs: \n\n```transact-sql\n{\"level\":\"error\",\"ts\":1675232549.5093117,\"caller\":\"watcher/watcher.go:60\",\n\"msg\":\"error during watch\",\"name\":\"mpstatus\",\"error\":\"INTERNAL: error sending\n mp status to mgmt: INTERNAL: failed to send runtime status\n Post \\\"https://apigee.googleapis.com/v1/organizations/example-hybrid-dev/instances/apigee-hybrid-australia-southeast1:reportStatus\\\":\n dial tcp 172.217.24.42:443: i/o timeout\",\"stacktrace\":\n \"edge-internal.git.corp.google.com/apigee-watcher.git/watcher.\n (*Watcher).Start.func1.1\\n\\t/go/src/edge-internal/apigee-watcher/watcher/watcher.go:60\"}\n```\n\nPossible Causes\n---------------\n\n### Cause: No network route available to the internet\n\n#### Diagnosis\n\n1. Depending on the platform used, verify whether there is a network route available from the cluster network to the internet. For example, for more information on networking on GKE, see [Network overview](/kubernetes-engine/docs/concepts/network-overview).\n2. Check with your infrastructure and networking team to see if the cluster network uses a forward proxy server for connecting to the internet.\n\n#### Resolution\n\n1. Check with the network administrator and add a network route to the internet if possible.\n2. If there is a forward proxy server used for communicating between the cluster network and internet, [configure forward proxy server settings](/apigee/docs/hybrid/latest/config-prop-ref#httpproxy) in Apigee using the `overrides.yaml` file and apply that change to the runtime plane using the [Helm command](/apigee/docs/hybrid/latest/helm-reference):\n\n Dry run: \n\n ```\n helm upgrade ENV_GROUP_RELEASE_NAME apigee-virtualhost/ \\\n --namespace APIGEE_NAMESPACE \\\n --atomic \\\n --set envgroup=ENV_GROUP \\\n -f OVERRIDES_FILE \\\n --dry-run=server\n ```\n\n Make sure to include all of the settings shown, including `--atomic`\n so that the action rolls back on failure.\n\n Your installation may or may not have been set up using an\n \u003cvar translate=\"no\"\u003eENV_GROUP_RELEASE_NAME\u003c/var\u003e that differs from \u003cvar translate=\"no\"\u003eENV_GROUP\u003c/var\u003e.\n For information on the settings, see\n [Install Apigee hybrid using Helm](/apigee/docs/hybrid/latest/install-helm-charts).\n\n Install the chart: \n\n ```\n helm upgrade ENV_GROUP_RELEASE_NAME apigee-virtualhost/ \\\n --namespace APIGEE_NAMESPACE \\\n --atomic \\\n --set envgroup=ENV_GROUP \\\n -f OVERRIDES_FILE\n ```\n\n### Cause: Apigee API endpoints are not allow-listed\n\n#### Diagnosis\n\nCheck with the network administrator and verify whether\n[the list of Apigee API endpoints](https://cloud.google.com/anthos/clusters/docs/on-prem/latest/how-to/firewall-rules) are allow-listed in the firewall on\nthe platform where Apigee is installed. For GKE, this could be\n[Cloud Next Generation Firewall](https://console.cloud.google.com/networking/firewalls/list).\n\n#### Resolution\n\n\nIf the above list of Apigee API endpoints are not allow-listed, contact the\nnetwork administrator and complete that requirement.\n\n### Cause: Private Google Access is not enabled\n\n#### Diagnosis\n\n1. If Apigee is deployed on a [private GKE cluster](/kubernetes-engine/docs/how-to/private-clusters) that does not have access to the internet, [Private Google Access](https://cloud.google.com/vpc/docs/private-google-access) needs to be enabled to allow the Apigee runtime plane components access to Google APIs internally.\n2. In the Google Cloud console, go to the **VPC networks** page.\n\n [Go to VPC networks](https://console.cloud.google.com/networking/networks/list)\n3. Click the name of a network.\n4. On the **Subnets** tab, In the **Private Google Access** column, verify that the relevant subnet is enabled. If it is not, that's the reason for this failure.\n\n#### Resolution\n\n1. In the Google Cloud console, go to the **VPC networks** page.\n\n [Go to VPC networks](https://console.cloud.google.com/networking/networks/list)\n2. Click the name of a network.\n3. Click the name of a subnet.\n4. Click edit **Edit**.\n5. Select **Private Google Access On**.\n6. Click **Save**. Once enabled, this issue will be resolved.\n\n### Cause: Unknown network failure\n\n#### Diagnosis\n\n\nContact your network administrator and verify whether there are any\nunknown failures in the cluster network.\n\n#### Resolution\n\n\nWork with your network administrator and resolve the issues in the cluster\nnetwork. Once the network issues are fixed this problem will get resolved.\n\nMust gather diagnostic information\n----------------------------------\n\n\nIf the problem persists even after following the above instructions, gather\nthe following diagnostic information and then contact [Google Cloud Customer Care](https://cloud.google.com/support-hub/):\n\n1. The Google Cloud Project ID.\n2. The name of the Apigee hybrid organization.\n3. The outputs of the following commands executed on all Kubernetes cluster nodes: \n\n ```\n ping apigee.googleapis.com\n ``` \n\n ```\n traceroute apigee.googleapis.com\n ``` \n\n ```\n telnet apigee.googleapis.com 443\n ```\n4. The `overrides.yaml` file, masking any sensitive information.\n5. The Kubernetes pod status in all namespaces: \n\n ```\n kubectl get pods -A \u003e kubectl-pod-status`date +%Y.%m.%d_%H.%M.%S`.txt\n ```\n6. The Kubernetes cluster-info dump: Generate Kubernetes cluster-info dump:\n\n ```\n kubectl cluster-info dump -A --output-directory=/tmp/kubectl-cluster-info-dump\n ```\n\n Zip Kubernetes cluster-info dump: \n\n ```\n zip -r kubectl-cluster-info-dump`date +%Y.%m.%d_%H.%M.%S`.zip /tmp/kubectl-cluster-info-dump/*\n ```"]]
