Skip to content

Commit c7e5bce

Browse files
potiukpotiuk
authored
Prepare backport release candidate for 2020.6.23rc1 (#9370)
* Prepare backport release candidate for 2020.6.23rc1 * fixup! Prepare backport release candidate for 2020.6.23rc1 * fixup! fixup! Prepare backport release candidate for 2020.6.23rc1
1 parent 0cfa766 commit c7e5bce

File tree

120 files changed

+663
-564
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

120 files changed

+663
-564
lines changed

β€ŽREADME.md

Lines changed: 67 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The
4646
- [Beyond the Horizon](#beyond-the-horizon)
4747
- [Principles](#principles)
4848
- [User Interface](#user-interface)
49-
- [Using hooks and Operators from "master" in Airflow 1.10](#using-hooks-and-operators-from-master-in-airflow-110)
49+
- [Backport packages: Using providers from Airflow 2.0 in Airflow 1.10.*](#backport-packages-using-providers-from-airflow-20-in-airflow-110)
5050
- [Contributing](#contributing)
5151
- [Who uses Apache Airflow?](#who-uses-apache-airflow)
5252
- [Who Maintains Apache Airflow?](#who-maintains-apache-airflow)
@@ -167,31 +167,80 @@ unit of work and continuity.
167167
![](/docs/img/code.png)
168168

169169

170-
## Using hooks and Operators from "master" in Airflow 1.10
170+
## Backport packages: Using providers from Airflow 2.0 in Airflow 1.10.*
171171

172-
Currently, stable versions of Apache Airflow are released in 1.10.* series. We are working on the
173-
future, major version of Airflow from the 2.0.* series. It is going to be released in 2020. However, the exact time of release depends on many factors and is yet unknown.
174-
We have already a lot of changes in the hooks/operators/sensors for many external systems
175-
and they are not used because they are part of the master/2.0 release.
172+
Currently, stable Apache Airflow versions are from the 1.10.* series.
173+
We are working on the future, major version of Airflow from the 2.0.* series.
174+
It is going to be released in 2020. However, the exact time of release depends on many factors and is
175+
not yet confirmed.
176176

177-
In the Airflow 2.0 - following AIP-21 "change in import paths" all the non-core operators/hooks/sensors
178-
of Apache Airflow have been moved to the "airflow.providers" package. This opened a possibility to
179-
use the operators from Airflow 2.0 in Airflow 1.10 - with the constraint that those
180-
packages can only be used in python3.6+ environment.
177+
We have already a lot of changes in the operators, transfers, hooks, sensors, secrets for many external
178+
systems, but they are not used nor tested widely because they are part of the master/2.0 release.
181179

182-
Therefore we decided to prepare and release backport packages that can be installed
183-
for older Airflow versions. Those backport packages are released more frequently. Users do not
184-
have to upgrade their Airflow version to use those packages. There are a number of changes
185-
between Airflow 2.0 and 1.10.* - documented in [UPDATING.md](UPDATING.md). With backported
186-
providers package users can migrate their DAGs to the new providers package incrementally
180+
In the Airflow 2.0 - following AIP-21 "change in import paths" all the non-core interfaces to external
181+
systems of Apache Airflow have been moved to the "airflow.providers" package.
182+
183+
Thanks to that and automated backport effort we took, the operators from Airflow 2.0
184+
can be used in Airflow 1.10 as separately installable packages, with the constraint that
185+
those packages can only be used in python3.6+ environment.
186+
187+
We released backport packages that can be installed for older Airflow versions.
188+
Those backport packages are going to be released more frequently that main Airflow 1.10.& releases.
189+
190+
You will not have to upgrade your Airflow version to use those packages. You can find those packages in the
191+
[PyPI](https://pypi.org/search/?q=apache-airflow-backport-providers&o=) and install them separately for each
192+
provider.
193+
194+
Those packages are available now and can be used in the latest Airflow 1.10* version. Most of those
195+
packages are also installable and usable in most Airflow 1.10.* releases but there is no extensive testing
196+
done beyond the latest released version, so you might expect more problems in earlier Airflow versions.
197+
198+
Releasing the backport packages is also a way to ease migration to 2.0.
199+
200+
With backported providers package users can migrate their DAGs to the new providers package incrementally
187201
and once they convert to the new operators/sensors/hooks they can seamlessly migrate their
188-
environments to Airflow 2.0.
202+
environments to Airflow 2.0. The nice thing about providers backport packages is that you can use
203+
both old and new classes at the same time - even in the same DAG. So your migration can be gradual and smooth.
204+
Note that in Airflow 2.0 old classes raise deprecation warning and redirect to the new classes wherever
205+
it is possible. In some rare cases the new operators will not be fully backwards compatible - you will find
206+
information about those cases in [UPDATING.md](UPDATING.md) where we explained all such cases. Switching
207+
early to the Airflow 2.0 operators while still running Airflow 1.10 will make your migration much easier.
189208

190209
More information about the status and releases of the back-ported packages are available
191210
at [Backported providers package page](https://cwiki.apache.org/confluence/display/AIRFLOW/Backported+providers+packages+for+Airflow+1.10.*+series)
192211

193-
Dependencies between packages are stored in ``airflow/providers/dependencies.json``. See
194-
[CONTRIBUTING.rst](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#backport-providers-packages)
212+
Note that the backport packages might require extra dependencies. Pip installs the required dependencies
213+
automatically when it installs the backport package, but there are sometimes cross-dependencies between the
214+
backport packages. For example `google` package has cross-dependency with `amazon` package to allow
215+
transfers between those two cloud providers. You might need to install those packages in case you use
216+
cross-dependent packages. The easiest way to install them is to use "extras" when installing the package,
217+
for example the below will install both `google` and `amazon` backport packages:
218+
219+
```bash
220+
pip install apache-airflow-backport-providers-google[amazon]
221+
```
222+
223+
This is all documented in the PyPI description of the packages
224+
as well as in the README.md file available for each provider package. For example for google package
225+
you can find the readme in [README.md](airflow/providers/google/README.md). You will also find there
226+
the summary of both - new classes and moved classes as well as requirement information.
227+
228+
IMPORTANT NOTE!
229+
230+
Backport providers only work when they are installed in the same namespace as the 'apache-airflow' 1.10
231+
package. This is majority of cases when you simply run `pip install` - it installs all packages
232+
in the same folder (usually in `/usr/local/lib/pythonX.Y/site-packages`). But when you install
233+
the `apache-airflow` and `apache-airflow-backport-package-*` using different methods (for example using
234+
`pip install -e .` or `pip install --user` they might be installed in different namespaces.
235+
If that's the case, the provider packages will not be importable (the error in such case is
236+
`ModuleNotFoundError: No module named 'airflow.providers'`).
237+
238+
If you experience the problem, you can easily fix it by creating symbolic link
239+
in your installed "airflow" folder to the "providers" folder where you installed your backport packages.
240+
If you installed it with `-e`, this link should be created in your airflow
241+
sources, if you installed it with the `--user` flag it should be from the
242+
`~/.local/lib/pythonX.Y/site-packages/airflow/` folder,
243+
195244

196245
## Contributing
197246

β€Žairflow/providers/amazon/PROVIDERS_CHANGES_2020.05.20.md renamed to β€Žairflow/providers/amazon/PROVIDERS_CHANGES_2020-06-23.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,11 @@
11

22

3-
### Release 2020.5.20
3+
### Release 2020-06-23
44

55
| Commit | Committed | Subject |
66
|:-----------------------------------------------------------------------------------------------|:------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------|
7+
| [40bf8f28f](https://github.com/apache/airflow/commit/40bf8f28f97f17f40d993d207ea740eba54593ee) | 2020-06-18 | Detect automatically the lack of reference to the guide in the operator descriptions (#9290) |
8+
| [f6bd817a3](https://github.com/apache/airflow/commit/f6bd817a3aac0a16430fc2e3d59c1f17a69a15ac) | 2020-06-16 | Introduce 'transfers' packages (#9320) |
79
| [58a8ec0e4](https://github.com/apache/airflow/commit/58a8ec0e46f624ee0369dd156dd8fb4f81884a21) | 2020-06-16 | AWSBatchOperator <> ClientHook relation changed to composition (#9306) |
810
| [a80cd25e8](https://github.com/apache/airflow/commit/a80cd25e8eb7f8b5d89af26cdcd62a5bbe44d65c) | 2020-06-15 | Close/Flush byte stream in s3 hook load_string and load_bytes (#9211) |
911
| [ffb857403](https://github.com/apache/airflow/commit/ffb85740373f7adb70d28ec7d5a8886380170e5e) | 2020-06-14 | Decrypt secrets from SystemsManagerParameterStoreBackend (#9214) |

β€Žairflow/providers/amazon/README.md

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@
2020

2121
# Package apache-airflow-backport-providers-amazon
2222

23-
Release: 2020.5.20
23+
Release: 2020-06-23
2424

2525
**Table of contents**
2626

@@ -44,7 +44,7 @@ Release: 2020.5.20
4444
- [Secrets](#secrets)
4545
- [Moved secrets](#moved-secrets)
4646
- [Releases](#releases)
47-
- [Release 2020.5.20](#release-2020520)
47+
- [Release 2020-06-23](#release-2020-06-23)
4848

4949
## Backport package
5050

@@ -95,9 +95,11 @@ pip install apache-airflow-backport-providers-amazon[apache.hive]
9595
| [apache-airflow-backport-providers-postgres](https://github.com/apache/airflow/tree/master/airflow/providers/postgres) | postgres |
9696
| [apache-airflow-backport-providers-ssh](https://github.com/apache/airflow/tree/master/airflow/providers/ssh) | ssh |
9797

98-
# Provider class summary
98+
# Provider classes summary
9999

100-
All classes in Airflow 2.0 are in `airflow.providers.amazon` package.
100+
In Airflow 2.0, all operators, transfers, hooks, sensors, secrets for the `amazon` provider
101+
are in the `airflow.providers.amazon` package. You can read more about the naming conventions used
102+
in [Naming conventions for provider packages](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#naming-conventions-for-provider-packages)
101103

102104

103105
## Operators
@@ -253,10 +255,12 @@ All classes in Airflow 2.0 are in `airflow.providers.amazon` package.
253255

254256
## Releases
255257

256-
### Release 2020.5.20
258+
### Release 2020-06-23
257259

258260
| Commit | Committed | Subject |
259261
|:-----------------------------------------------------------------------------------------------|:------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------|
262+
| [40bf8f28f](https://github.com/apache/airflow/commit/40bf8f28f97f17f40d993d207ea740eba54593ee) | 2020-06-18 | Detect automatically the lack of reference to the guide in the operator descriptions (#9290) |
263+
| [f6bd817a3](https://github.com/apache/airflow/commit/f6bd817a3aac0a16430fc2e3d59c1f17a69a15ac) | 2020-06-16 | Introduce 'transfers' packages (#9320) |
260264
| [58a8ec0e4](https://github.com/apache/airflow/commit/58a8ec0e46f624ee0369dd156dd8fb4f81884a21) | 2020-06-16 | AWSBatchOperator <> ClientHook relation changed to composition (#9306) |
261265
| [a80cd25e8](https://github.com/apache/airflow/commit/a80cd25e8eb7f8b5d89af26cdcd62a5bbe44d65c) | 2020-06-15 | Close/Flush byte stream in s3 hook load_string and load_bytes (#9211) |
262266
| [ffb857403](https://github.com/apache/airflow/commit/ffb85740373f7adb70d28ec7d5a8886380170e5e) | 2020-06-14 | Decrypt secrets from SystemsManagerParameterStoreBackend (#9214) |

β€Žairflow/providers/apache/cassandra/PROVIDERS_CHANGES_2020.05.20.md renamed to β€Žairflow/providers/apache/cassandra/PROVIDERS_CHANGES_2020-06-23.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,10 @@
11

22

3-
### Release 2020.5.20
3+
### Release 2020-06-23
44

55
| Commit | Committed | Subject |
66
|:-----------------------------------------------------------------------------------------------|:------------|:---------------------------------------------------------------------------------|
7+
| [f6bd817a3](https://github.com/apache/airflow/commit/f6bd817a3aac0a16430fc2e3d59c1f17a69a15ac) | 2020-06-16 | Introduce 'transfers' packages (#9320) |
78
| [0b0e4f7a4](https://github.com/apache/airflow/commit/0b0e4f7a4cceff3efe15161fb40b984782760a34) | 2020-05-26 | Preparing for RC3 relase of backports (#9026) |
89
| [00642a46d](https://github.com/apache/airflow/commit/00642a46d019870c4decb3d0e47c01d6a25cb88c) | 2020-05-26 | Fixed name of 20 remaining wrongly named operators. (#8994) |
910
| [375d1ca22](https://github.com/apache/airflow/commit/375d1ca229464617780623c61c6e8a1bf570c87f) | 2020-05-19 | Release candidate 2 for backport packages 2020.05.20 (#8898) |

β€Žairflow/providers/apache/cassandra/README.md

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@
2020

2121
# Package apache-airflow-backport-providers-apache-cassandra
2222

23-
Release: 2020.5.20
23+
Release: 2020-06-23
2424

2525
**Table of contents**
2626

@@ -34,7 +34,7 @@ Release: 2020.5.20
3434
- [Hooks](#hooks)
3535
- [Moved hooks](#moved-hooks)
3636
- [Releases](#releases)
37-
- [Release 2020.5.20](#release-2020520)
37+
- [Release 2020-06-23](#release-2020-06-23)
3838

3939
## Backport package
4040

@@ -64,9 +64,11 @@ For full compatibility and test status of the backport packages check
6464
|:-----------------|:-------------------|
6565
| cassandra-driver | >=3.13.0,<3.21.0 |
6666

67-
# Provider class summary
67+
# Provider classes summary
6868

69-
All classes in Airflow 2.0 are in `airflow.providers.apache.cassandra` package.
69+
In Airflow 2.0, all operators, transfers, hooks, sensors, secrets for the `apache.cassandra` provider
70+
are in the `airflow.providers.apache.cassandra` package. You can read more about the naming conventions used
71+
in [Naming conventions for provider packages](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#naming-conventions-for-provider-packages)
7072

7173

7274

@@ -103,10 +105,11 @@ All classes in Airflow 2.0 are in `airflow.providers.apache.cassandra` package.
103105

104106
## Releases
105107

106-
### Release 2020.5.20
108+
### Release 2020-06-23
107109

108110
| Commit | Committed | Subject |
109111
|:-----------------------------------------------------------------------------------------------|:------------|:---------------------------------------------------------------------------------|
112+
| [f6bd817a3](https://github.com/apache/airflow/commit/f6bd817a3aac0a16430fc2e3d59c1f17a69a15ac) | 2020-06-16 | Introduce 'transfers' packages (#9320) |
110113
| [0b0e4f7a4](https://github.com/apache/airflow/commit/0b0e4f7a4cceff3efe15161fb40b984782760a34) | 2020-05-26 | Preparing for RC3 relase of backports (#9026) |
111114
| [00642a46d](https://github.com/apache/airflow/commit/00642a46d019870c4decb3d0e47c01d6a25cb88c) | 2020-05-26 | Fixed name of 20 remaining wrongly named operators. (#8994) |
112115
| [375d1ca22](https://github.com/apache/airflow/commit/375d1ca229464617780623c61c6e8a1bf570c87f) | 2020-05-19 | Release candidate 2 for backport packages 2020.05.20 (#8898) |

β€Žairflow/providers/apache/druid/PROVIDERS_CHANGES_2020.05.20.md renamed to β€Žairflow/providers/apache/druid/PROVIDERS_CHANGES_2020-06-23.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,10 @@
11

22

3-
### Release 2020.5.20
3+
### Release 2020-06-23
44

55
| Commit | Committed | Subject |
66
|:-----------------------------------------------------------------------------------------------|:------------|:--------------------------------------------------------------------------------------------|
7+
| [f6bd817a3](https://github.com/apache/airflow/commit/f6bd817a3aac0a16430fc2e3d59c1f17a69a15ac) | 2020-06-16 | Introduce 'transfers' packages (#9320) |
78
| [0b0e4f7a4](https://github.com/apache/airflow/commit/0b0e4f7a4cceff3efe15161fb40b984782760a34) | 2020-05-26 | Preparing for RC3 relase of backports (#9026) |
89
| [00642a46d](https://github.com/apache/airflow/commit/00642a46d019870c4decb3d0e47c01d6a25cb88c) | 2020-05-26 | Fixed name of 20 remaining wrongly named operators. (#8994) |
910
| [375d1ca22](https://github.com/apache/airflow/commit/375d1ca229464617780623c61c6e8a1bf570c87f) | 2020-05-19 | Release candidate 2 for backport packages 2020.05.20 (#8898) |

β€Žairflow/providers/apache/druid/README.md

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@
2020

2121
# Package apache-airflow-backport-providers-apache-druid
2222

23-
Release: 2020.5.20
23+
Release: 2020-06-23
2424

2525
**Table of contents**
2626

@@ -37,7 +37,7 @@ Release: 2020.5.20
3737
- [Hooks](#hooks)
3838
- [Moved hooks](#moved-hooks)
3939
- [Releases](#releases)
40-
- [Release 2020.5.20](#release-2020520)
40+
- [Release 2020-06-23](#release-2020-06-23)
4141

4242
## Backport package
4343

@@ -82,9 +82,11 @@ pip install apache-airflow-backport-providers-apache-druid[apache.hive]
8282
|:-----------------------------------------------------------------------------------------------------------------------------|:------------|
8383
| [apache-airflow-backport-providers-apache-hive](https://github.com/apache/airflow/tree/master/airflow/providers/apache/hive) | apache.hive |
8484

85-
# Provider class summary
85+
# Provider classes summary
8686

87-
All classes in Airflow 2.0 are in `airflow.providers.apache.druid` package.
87+
In Airflow 2.0, all operators, transfers, hooks, sensors, secrets for the `apache.druid` provider
88+
are in the `airflow.providers.apache.druid` package. You can read more about the naming conventions used
89+
in [Naming conventions for provider packages](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#naming-conventions-for-provider-packages)
8890

8991

9092
## Operators
@@ -133,10 +135,11 @@ All classes in Airflow 2.0 are in `airflow.providers.apache.druid` package.
133135

134136
## Releases
135137

136-
### Release 2020.5.20
138+
### Release 2020-06-23
137139

138140
| Commit | Committed | Subject |
139141
|:-----------------------------------------------------------------------------------------------|:------------|:--------------------------------------------------------------------------------------------|
142+
| [f6bd817a3](https://github.com/apache/airflow/commit/f6bd817a3aac0a16430fc2e3d59c1f17a69a15ac) | 2020-06-16 | Introduce 'transfers' packages (#9320) |
140143
| [0b0e4f7a4](https://github.com/apache/airflow/commit/0b0e4f7a4cceff3efe15161fb40b984782760a34) | 2020-05-26 | Preparing for RC3 relase of backports (#9026) |
141144
| [00642a46d](https://github.com/apache/airflow/commit/00642a46d019870c4decb3d0e47c01d6a25cb88c) | 2020-05-26 | Fixed name of 20 remaining wrongly named operators. (#8994) |
142145
| [375d1ca22](https://github.com/apache/airflow/commit/375d1ca229464617780623c61c6e8a1bf570c87f) | 2020-05-19 | Release candidate 2 for backport packages 2020.05.20 (#8898) |

0 commit comments

Comments
 (0)