Update SDKs for google provider package (#30067)

* Update SDK versions for Google provider

* Adjust google ads operators to v12

Changes:
- fix tests/system/providers/google/cloud/bigquery/example_bigquery_queries.py
- fix tests/system/providers/google/cloud/bigquery/example_bigquery_queries_async.py

* Fix GCS system tests

* Fix CloudBuild unit test

* Update BigTable operators to accomodate for new dependencies.

* Fix Cloud Tasks System tests

Tasks dag was quite flaky without the retry option in the run_task step,
but it's consistently green with the option set.

We also add a GCP_APP_ENGINE_LOCATION env variable since this depends on
the used GCP Project App Engine's location

* Add setup docstring to Tasks system tests.

* Update Vision operators to accommodate new dependencies.

Changes:
- fix methods for CloudVisionHook
- fix Vision Operators
- fix tests/providers/google/cloud/hooks/test_vision.py
- fix tests/providers/google/cloud/operators/test_vision.py
- fix tests/system/providers/google/cloud/vision/example_vision_annotate_image.py
- fix tests/system/providers/google/cloud/vision/example_vision_autogenerated.py
- fix tests/system/providers/google/cloud/vision/example_vision_explicit.py

* Update SpeechToText operators to accommodate new dependencies.

Changes:
- fix synthesize_speech method for CloudTextToSpeechHook
- fix CloudSpeechToTextRecognizeSpeechOperator
- fix tests/providers/google/cloud/operators/test_speech_to_text.py
- fix tests/providers/google/cloud/hooks/test_text_to_speech.py
- fix tests/providers/google/cloud/hooks/test_speech_to_text.py

* Update Translate Speech operators to accommodate new dependencies.

Changes:
- fix synthesize_speech method for CloudTextToSpeechHook
- fix CloudTranslateSpeechOperator
- tests/providers/google/cloud/operators/test_translate_speech.py

* Update VideoIntelligence operators to accommodate new dependencies.

Changes:
- fix annotate_video method for CloudVideoIntelligenceHook
- fix VideoIntelligence Operators
- fix tests/providers/google/cloud/hooks/test_video_intelligence.py
- fix tests/providers/google/cloud/operators/test_video_intelligence.py

* Update Compute Engine operators to accomodate for new dependencies.

Changes:
- added wait_for_operation_complete() method to check the execution flow
- added new attribute cmd_timeout for ComputeEngineSSHHook

* Fix Stackdriver system test

This test has not worked because of slack channel and credentials not
being setup. We now test the same operators by creating notification
channels and policy alerts against pubsub topics, which don't need to
exist before the test is ran, making the test self-contained.

* Update Natural Language operators to accommodate new dependencies.

Changes:
- fix airflow/providers/google/cloud/operators/natural_language.py
- fix airflow/providers/google/cloud/hooks/natural_language.py
- fix tests/providers/google/cloud/hooks/test_natural_language.py
- fix tests/providers/google/cloud/operators/test_natural_language.py
- fix tests/system/providers/google/cloud/natural_language/example_natural_language.py

* Update Composer system tests.

Fix environment id to contain underscores.

* Update AutoML operators to accommodate new dependencies.

Changes:
- add timeout parameter to all long-running operations for operators
- fix tests/system/providers/google/cloud/automl/example_automl_dataset.py
- fix tests/system/providers/google/cloud/automl/example_automl_model.py
- fix tests/system/providers/google/cloud/automl/example_automl_nl_text_extraction.py
- fix tests/system/providers/google/cloud/automl/example_automl_vision_classification.py

* Fix Cloud SQL delete operator

For some delete instance operations, the operation stops being available ~9 seconds after completion, so we need a shorter sleep time to make sure we don'tmiss the DONE status.

* Update VertexAI operators to accommodate new dependencies.

* Add SQL to Sheets Test instructions

* Update Dataproc Metastore operators to accommodate new dependencies.

* Update Dataproc operators to accommodate new dependencies.

* Update Dataflow sys tests to new sdk

* Update Dataproc on gke operators to accommodate new dependencies.

* Update MLEngine operators to accomodate new dependencies.

Changes:
- update train model that is used for prediction
- update version and runner for ApacheBeam in utils for MLEngine
- update connection inside async hook

* Update Dataprep operators to accommodate new dependencies.

Changes:
- fix tests/system/providers/google/cloud/dataprep/example_dataprep.py

* Add Dataflow Go system test

* Update providers.yaml for google

* fixup! Update providers.yaml for google

* Google SDK Fixes after rebase

- Secret Manager was missing updating to v2, now expects a request dict
- Compute ssh had a bug when no cmd_timeout was passed
- Cloud Build tests were improved/refactored in community, so deleting
old ones
- googleapiclient.errors.HttpError was incorrectly used in our tests, it
it didn´t matter before but a change in the class makes HttpError()
raise an error in initialization the way we were using it before
- fix static checks
* Fix Google providers type errors

---------

Co-authored-by: Lukasz Wyszomirski <wyszomirski@google.com>
Co-authored-by: Maksim Moiseenkov <maksim_moiseenkov@epam.com>
Co-authored-by: Eugene Kostieiev <kosteev@google.com>
Co-authored-by: Augusto Hidalgo <augustoh@google.com>
Co-authored-by: Beata Kossakowska <bkossakowska@google.com>
Co-authored-by: Ulada Zakharava <uladaz@google.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

Author: 8 people

airflow/providers/apache/beam/provider.yaml

@@ -43,7 +43,7 @@ versions:
 
 dependencies:
   - apache-airflow>=2.4.0
-  - apache-beam>=2.33.0
+  - apache-beam>=2.47.0
 
 integrations:
   - integration-name: Apache Beam

airflow/providers/google/ads/hooks/ads.py

@@ -21,26 +21,18 @@
 from tempfile import NamedTemporaryFile
 from typing import IO, Any
 
+from google.ads.googleads.client import GoogleAdsClient
+from google.ads.googleads.errors import GoogleAdsException
+from google.ads.googleads.v12.services.services.customer_service import CustomerServiceClient
+from google.ads.googleads.v12.services.services.google_ads_service import GoogleAdsServiceClient
+from google.ads.googleads.v12.services.types.google_ads_service import GoogleAdsRow
+from google.api_core.page_iterator import GRPCIterator
 from google.auth.exceptions import GoogleAuthError
 
 from airflow import AirflowException
 from airflow.compat.functools import cached_property
 from airflow.hooks.base import BaseHook
 from airflow.providers.google.common.hooks.base_google import get_field
-from airflow.providers.google_vendor.googleads.client import GoogleAdsClient
-from airflow.providers.google_vendor.googleads.errors import GoogleAdsException
-from airflow.providers.google_vendor.googleads.v12.services.services.customer_service import (
-    CustomerServiceClient,
-)
-from airflow.providers.google_vendor.googleads.v12.services.services.google_ads_service import (
-    GoogleAdsServiceClient,
-)
-from airflow.providers.google_vendor.googleads.v12.services.services.google_ads_service.pagers import (
-    SearchPager,
-)
-from airflow.providers.google_vendor.googleads.v12.services.types.google_ads_service import (
-    GoogleAdsRow,
-)
 
 
 class GoogleAdsHook(BaseHook):
@@ -238,7 +230,7 @@ def _search(
 
         return self._extract_rows(iterators)
 
-    def _extract_rows(self, iterators: list[SearchPager]) -> list[GoogleAdsRow]:
+    def _extract_rows(self, iterators: list[GRPCIterator]) -> list[GoogleAdsRow]:
         """
         Convert Google Page Iterator (GRPCIterator) objects to Google Ads Rows
 

airflow/providers/google/cloud/_internal_client/secret_manager_client.py

@@ -71,7 +71,7 @@ def get_secret(self, secret_id: str, project_id: str, secret_version: str = "lat
         """
         name = self.client.secret_version_path(project_id, secret_id, secret_version)
         try:
-            response = self.client.access_secret_version(name)
+            response = self.client.access_secret_version(request={"name": name})
             value = response.payload.data.decode("UTF-8")
             return value
         except NotFound:

airflow/providers/google/cloud/example_dags/example_compute.py

@@ -48,6 +48,7 @@
 )
 from google.protobuf.field_mask_pb2 import FieldMask
 
+from airflow import AirflowException
 from airflow.compat.functools import cached_property
 from airflow.providers.google.common.consts import CLIENT_INFO
 from airflow.providers.google.common.hooks.base_google import PROVIDE_PROJECT_ID, GoogleBaseHook
@@ -93,6 +94,14 @@ def get_conn(self) -> AutoMlClient:
             self._client = AutoMlClient(credentials=self.get_credentials(), client_info=CLIENT_INFO)
         return self._client
 
+    def wait_for_operation(self, operation: Operation, timeout: float | None = None):
+        """Waits for long-lasting operation to complete."""
+        try:
+            return operation.result(timeout=timeout)
+        except Exception:
+            error = operation.exception(timeout=timeout)
+            raise AirflowException(error)
+
     @cached_property
     def prediction_client(self) -> PredictionServiceClient:
         """