Perform failover tests for highly resilient environments

Cloud Composer 3 | Cloud Composer 2 | Cloud Composer 1

This page describes how to perform a database failover test for highly resilient (Highly Available) environments.

Failover tests for your environment simulate a complete outage of a zone in a data center. In such a scenario, a zonal outage of a database might happen. By performing a failover test, you can monitor how your highly resilient environment performs a failover and check how this affects your DAGs and tasks.

Before you begin

  • To perform failover tests, your Google Account must have the following roles and permissions:

  • In Cloud Composer 3, the environment's cluster is located in the tenant project of your environment and it's not possible to simulate a zonal outage for it.

Check that your environment is healthy

Make sure to perform failover tests only on healthy environments. To check that your environment is healthy:

  1. In the Google Cloud console, go to the Environments page.

    Go to Environments

  2. In the list of environments, click the name of your environment. The Environment details page opens.

  3. Go to the Monitoring tab.

  4. Make sure that all health metrics are green.

Perform a database failover test

You can perform a database failover test, which simulates a zonal outage, by triggering it with a Google Cloud CLI command. For example, you might want to do it to measure the amount of time it takes for your environment's database to switch to another zone.

To perform a database failover test for your environment:

  1. Make sure that your environment is healthy.

  2. Get the primary zone of your environment's database:

    gcloud composer environments fetch-database-properties \
    ENVIRONMENT_NAME \
    --location LOCATION

    Replace the following:

    • ENVIRONMENT_NAME: the name of your Cloud Composer environment.
    • LOCATION: the region where the environment is located.

    Example:

    gcloud composer environments fetch-database-properties \
    example-environment \
    --location us-central1

  3. Start the database failover test:

    gcloud composer environments database-failover \
    ENVIRONMENT_NAME \
    --location LOCATION

    Replace the following:

    • ENVIRONMENT_NAME: the name of your Cloud Composer environment.
    • LOCATION: the region where the environment is located.

    Example:

    gcloud composer environments database-failover \
    example-environment \
    --location us-central1

  4. Wait until the database failover test is finished. The process can take up to 3 minutes.

  5. Check that the primary zone of your environment's database is changed:

    gcloud composer environments fetch-database-properties \
    ENVIRONMENT_NAME \
    --location LOCATION

  6. Check your environment's health metrics to make sure that your environment is healthy.

  7. Your environment's database becomes ready for another failover when the Database available for failover (composer.googleapis.com/environment/database/available_for_failover) environment metric becomes True. For more information about viewing your environment's metrics in Cloud Monitoring, see Monitor environments.

What's next