GCSToGCSOperator ignores replace parameter when there is no wildcard

[Yao-ATG]

Apache Airflow Provider(s)

google

Versions of Apache Airflow Providers

Latest

Apache Airflow version

2.2.5 (latest released)

Operating System

MacOS 12.2.1

Deployment

Composer

Deployment details

No response

What happened

Ran the same DAG twice with 'replace = False', in the second run files are overwritten anyway.
source_object does not include wildcard.

Not sure whether this incorrect behavior happens to "with wildcard" scenario, but from source code
https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/transfers/gcs_to_gcs.py
in line 346 (inside _copy_source_with_wildcard) we have
if not self.replace:
but in _copy_source_without_wildcard we don't check self.replace at all.

What you think should happen instead

When 'replace = False', the second run should skip copying files since they are already there.

How to reproduce

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

Can you add a PR to fix it please? That's the best way to make it happen quickly otherwise it will have to wait for somene to pick it up @Yao-ATG

[sahutd]

@potiuk i am interested in filing a fix for this[i might be facing a similar issue described here]

as i have not developed airflow in the past, it might take me some time. in the meantime, if someone wants to file a fix, go for it :)

[spatocode]

@potiuk i am interested in filing a fix for this[i might be facing a similar issue described here]

as i have not developed airflow in the past, it might take me some time. in the meantime, if someone wants to file a fix, go for it :)

I'll be picking this up, then.

[spatocode]

@potiuk I raised a PR to fix this.