Dataproc ํด๋Ÿฌ์Šคํ„ฐ ์ด๋ฏธ์ง€ ๋ฒ„์ „ ๋ชฉ๋ก

๐ŸŒ Dataproc ํด๋Ÿฌ์Šคํ„ฐ ์ด๋ฏธ์ง€ ๋ฒ„์ „ ๋ชฉ๋ก  |  Dataproc Documentation  |  Google Cloud"Ubuntu ๋ฐ Debian ๊ธฐ๋ฐ˜ ์ด๋ฏธ์ง€ ๋ฒ„์ „ ํด๋Ÿฌ์Šคํ„ฐ | Google Cloud" - cloud.google.com

Google Dataproc์€ Ubuntu, Debian, Rocky Linux ์ด๋ฏธ์ง€ ๋ฒ„์ „์„ ์‚ฌ์šฉํ•˜์—ฌ ์šด์˜์ฒด์ œ, ๋น…๋ฐ์ดํ„ฐ ๊ตฌ์„ฑ์š”์†Œ, Google Cloud ์ปค๋„ฅํ„ฐ๋ฅผ ํด๋Ÿฌ์Šคํ„ฐ์— ๋ฐฐํฌ๋˜๋Š” ํ•˜๋‚˜์˜ ํŒจํ‚ค์ง€๋กœ ๋ฒˆ๋“ค๋กœ ๋ฌถ์Šต๋‹ˆ๋‹ค. ์ž์„ธํ•œ ๋‚ด์šฉ์€ Dataproc ๋ฒ„์ „ ๊ด€๋ฆฌ๋ฅผ ์ฐธ์กฐํ•˜์„ธ์š”.

์ฐธ๊ณ :

  • Dataproc ์ด๋ฏธ์ง€ ๋ฒ„์ „์€ ์ตœ์ดˆ ์ถœ์‹œ ํ›„ 24๊ฐœ์›” ๋™์•ˆ ์ง€์›๋ฉ๋‹ˆ๋‹ค.
  • ์ผ๋ฐ˜์ ์œผ๋กœ Dataproc ์ด๋ฏธ์ง€ ๋ฒ„์ „์€ ์ง€์› ์ข…๋ฃŒ ๋‚ ์งœ ์ดํ›„ 24๊ฐœ์›” ๋™์•ˆ ์‚ฌ์šฉ ๊ฐ€๋Šฅํ•˜์ง€๋งŒ ์ด๋ฏธ์ง€ ์ง€์› ๋‚ ์งœ๊ฐ€ ์—ฐ์žฅ๋œ ๊ฒฝ์šฐ ์‚ฌ์šฉ ๊ฐ€๋Šฅํ•œ ๊ธฐ๊ฐ„์ด ๋‹จ์ถ•๋  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๊ธฐ๋ณธ Dataproc ์ด๋ฏธ์ง€ ๋ฒ„์ „

Dataproc์€ ์ •์‹ ๋ฒ„์ „(GA) ์ถœ์‹œ์ผ ์ดํ›„ ๊ธฐ๋ณธ ์ด๋ฏธ์ง€ ๋ฒ„์ „์„ ์ตœ์‹  ์ •์‹ ๋ฒ„์ „ Debian ๊ธฐ๋ฐ˜ Dataproc์œผ๋กœ ์—…๋ฐ์ดํŠธํ•ฉ๋‹ˆ๋‹ค.

์ง€์›๋˜๋Š” Dataproc ์ด๋ฏธ์ง€ ๋ฒ„์ „

Debian ์ด๋ฏธ์ง€

Dataproc ํด๋Ÿฌ์Šคํ„ฐ์—์„œ ์ง€์›๋˜๋Š” Debian ๊ธฐ๋ฐ˜ ์ด๋ฏธ์ง€ ๋ฒ„์ „์€ ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค. ์ƒˆ๋กœ์šด ํด๋Ÿฌ์Šคํ„ฐ๋Š” ์ถœ์‹œ ์ดํ›„ ๋ฒ„์ „์— ์ ์šฉ๋œ ๋ชจ๋“  ํ•˜์œ„ ์„ธ๋ถ€ ํŒจ์น˜๋ฅผ ํฌํ•จํ•˜๋„๋ก ์ƒ์„ฑ๋ฉ๋‹ˆ๋‹ค.

๋ฒ„์ „ ์ตœ์ข… ์—…๋ฐ์ดํŠธ ์ถœ์‹œ์ผ ์ง€์› ๋งŒ๋ฃŒ์ผ ์‚ฌ์šฉ ๊ฐ€๋Šฅ ๊ธฐํ•œ ์ฐธ๊ณ 
2.3-debian12 2025/08/19 2025/06/09 2027/06/09 2029/06/09 ์ •์‹ ์ถœ์‹œ ๋ฒ„์ „์ž…๋‹ˆ๋‹ค.
2.2-debian12 2025/08/19 2023/12/08 2027/03/31 2027/12/31 ์ •์‹ ์ถœ์‹œ ๋ฒ„์ „์ž…๋‹ˆ๋‹ค. ์ด๋ฏธ์ง€ ๋ฒ„์ „ 2.2๊ฐ€ 2024๋…„ 9์›” 13์ผ์— ๊ธฐ๋ณธ ์ด๋ฏธ์ง€ ๋ฒ„์ „์ด ๋ฉ๋‹ˆ๋‹ค.
2.1-debian11 2025/08/19 2022/12/12 2026/03/31 2026/12/31 ์ •์‹ ์ถœ์‹œ ๋ฒ„์ „์ž…๋‹ˆ๋‹ค.
2.0-debian10 2025/08/19 2021/01/22 2025/09/30 2026/07/31 ์ •์‹ ์ถœ์‹œ ๋ฒ„์ „์ž…๋‹ˆ๋‹ค.

Ubuntu ์ด๋ฏธ์ง€

Dataproc ํด๋Ÿฌ์Šคํ„ฐ์—์„œ ์ง€์›๋˜๋Š” Ubuntu LTS ๊ธฐ๋ฐ˜ ์ด๋ฏธ์ง€ ๋ฒ„์ „์€ ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค. ์ƒˆ๋กœ์šด ํด๋Ÿฌ์Šคํ„ฐ๋Š” ์ถœ์‹œ ์ดํ›„ ๋ฒ„์ „์— ์ ์šฉ๋œ ๋ชจ๋“  ํ•˜์œ„ ์„ธ๋ถ€ ํŒจ์น˜๋ฅผ ํฌํ•จํ•˜๋„๋ก ์ƒ์„ฑ๋ฉ๋‹ˆ๋‹ค.

๋ฒ„์ „ ์ตœ์ข… ์—…๋ฐ์ดํŠธ ์ถœ์‹œ์ผ ์ง€์› ๋งŒ๋ฃŒ์ผ ์‚ฌ์šฉ ๊ฐ€๋Šฅ ๊ธฐํ•œ ์ฐธ๊ณ 
2.3-ubuntu22 2025/08/19 2025/06/09 2027/06/09 2029/06/09 ์ •์‹ ์ถœ์‹œ ๋ฒ„์ „์ž…๋‹ˆ๋‹ค.
2.2-ubuntu22 2025/08/19 2023/12/08 2027/03/31 2027/12/31 ์ •์‹ ์ถœ์‹œ ๋ฒ„์ „์ž…๋‹ˆ๋‹ค.
2.1-ubuntu20 2025/08/19 2022/12/12 2026/03/31 2026/12/31 ์ •์‹ ์ถœ์‹œ ๋ฒ„์ „์ž…๋‹ˆ๋‹ค.
2.0-ubuntu18 2025/08/19 2021/01/22 2025/09/30 2026/07/31 ์ •์‹ ์ถœ์‹œ ๋ฒ„์ „์ž…๋‹ˆ๋‹ค.

Rocky Linux ์ด๋ฏธ์ง€

Dataproc ํด๋Ÿฌ์Šคํ„ฐ์—์„œ ์ง€์›๋˜๋Š” Rocky Linux ๊ธฐ๋ฐ˜ ์ด๋ฏธ์ง€ ๋ฒ„์ „์€ ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค. ์ƒˆ๋กœ์šด ํด๋Ÿฌ์Šคํ„ฐ๋Š” ์ถœ์‹œ ์ดํ›„ ๋ฒ„์ „์— ์ ์šฉ๋œ ๋ชจ๋“  ํ•˜์œ„ ์„ธ๋ถ€ ํŒจ์น˜๋ฅผ ํฌํ•จํ•˜๋„๋ก ์ƒ์„ฑ๋ฉ๋‹ˆ๋‹ค.

๋ฒ„์ „ ์ตœ์ข… ์—…๋ฐ์ดํŠธ ์ถœ์‹œ์ผ ์ง€์› ๋งŒ๋ฃŒ์ผ ์‚ฌ์šฉ ๊ฐ€๋Šฅ ๊ธฐํ•œ ์ฐธ๊ณ 
2.3-rocky9 2025/08/19 2025/06/09 2027/06/09 2029/06/09 ์ •์‹ ์ถœ์‹œ ๋ฒ„์ „์ž…๋‹ˆ๋‹ค.
2.2-rocky9 2025/08/19 2023/12/08 2027/03/31 2027/12/31 ์ •์‹ ์ถœ์‹œ ๋ฒ„์ „์ž…๋‹ˆ๋‹ค.
2.1-rocky8 2025/08/19 2022/12/12 2026/03/31 2026/12/31 ์ •์‹ ์ถœ์‹œ ๋ฒ„์ „์ž…๋‹ˆ๋‹ค.
2.0-rocky8 2025/08/19 2022/02/18 2025/09/30 2026/07/31 ์ •์‹ ์ถœ์‹œ ๋ฒ„์ „์ž…๋‹ˆ๋‹ค.

์ง€์›๋˜์ง€ ์•Š๋Š” Dataproc ์ด๋ฏธ์ง€ ๋ฒ„์ „

๋‹ค์Œ Dataproc ๋ฒ„์ „์€ ์ง€์›๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. Dataproc์€ ์ด๋Ÿฌํ•œ ๋ฒ„์ „์œผ๋กœ ์ƒ์„ฑ๋œ ํด๋Ÿฌ์Šคํ„ฐ์— ๋Œ€ํ•œ ์—…๋ฐ์ดํŠธ ๋ฐ ์ง€์›์„ ์ œ๊ณตํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ์ง€์›๋˜์ง€ ์•Š๋Š” ๋ฒ„์ „์œผ๋กœ ์ž‘์„ฑ๋œ ํด๋Ÿฌ์Šคํ„ฐ๋ฅผ ๊ณ„์† ์‹คํ–‰ํ•  ์ˆ˜ ์žˆ์ง€๋งŒ ์ง€์›๋˜๋Š” ๋ฒ„์ „์œผ๋กœ ์ž‘์„ฑ๋œ ์ƒˆ ํด๋Ÿฌ์Šคํ„ฐ๋กœ ๊ต์ฒดํ•˜๋Š” ๊ฒƒ์ด ์ข‹์Šต๋‹ˆ๋‹ค.

๋ฒ„์ „ ํฌํ•จ ์ถœ์‹œ ๋‚ ์งœ ์ตœ์ข… ์—…๋ฐ์ดํŠธ ์ฐธ๊ณ 
1.5-debian10/-ubuntu18/-rocky8 Apache Spark 2.4.8
Apache Hadoop 2.10.2
Apache Pig 0.17.0
Apache Hive 2.3.7
Cloud Storage ์ปค๋„ฅํ„ฐ 2.1.9-hadoop2
Python 3.7
Scala 2.12.10
Zookeeper 3.4.14
2020/03/25:debian10/ubuntu18
2022/02/18:rocky8
2023/04/28 2023/04/28 ํ˜„์žฌ ์ง€์›๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.
1.5.89-debian10/-ubuntu18/-rocky8์ด ์ตœ์ข… ์ถœ์‹œ ๋ฒ„์ „์ž…๋‹ˆ๋‹ค.
2.0-centos8 Apache Spark 3.1.2
Apache Hadoop 3.2.2
Apache Pig 0.18.0-SNAPSHOT
Apache Hive 3.1.2
Cloud Storage ์ปค๋„ฅํ„ฐ 2.2.4-hadoop3
Python 3.8
Scala 2.12.14
Zookeeper 3.4.14
2021/03/16 2022/02/01 2022/02/01 ํ˜„์žฌ ์ง€์›๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.
2.0.30-centos8์ด ์ตœ์ข… ์ถœ์‹œ ๋ฒ„์ „์ž…๋‹ˆ๋‹ค.
1.5-centos8 Apache Spark 2.4.8
Apache Hadoop 2.10.1
Apache Pig 0.17.0
Apache Hive 2.3.7
Cloud Storage ์ปค๋„ฅํ„ฐ 2.1.5-hadoop2
Python 3.7
Scala 2.12.10
Zookeeper 3.4.14
2020/12/14 2022/02/01 2022/02/01 ํ˜„์žฌ ์ง€์›๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.
1.5.56-centos8์ด ์ตœ์ข… ์ถœ์‹œ ๋ฒ„์ „์ž…๋‹ˆ๋‹ค.
1.4-debian10/-ubuntu18 Apache Spark 2.4.8
Apache Hadoop 2.9.2
Apache Pig 0.17.0
Apache Hive 2.3.7
Cloud Storage ์ปค๋„ฅํ„ฐ 1.9.18-hadoop2
Python 3.6
Scala 2.11.12
Zookeeper 3.4.14
2019/03/22 2022/02/01 2022/02/01๋ถ€ํ„ฐ ์ง€์›๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.
1.4.80-debian10/-ubuntu18์ด ์ตœ์ข… ์ถœ์‹œ ๋ฒ„์ „์ž…๋‹ˆ๋‹ค.
1.3-debian10/-ubuntu18 Apache Spark 2.3.4
Apache Hadoop 2.9.2
Apache Pig 0.17.0
Apache Hive 2.3.7
Cloud Storage ์ปค๋„ฅํ„ฐ 1.9.18-hadoop2
Python 2.7
Scala 2.11.8
Zookeeper 3.4.13
2018/06/29 2021/12/22 2021/08/01 ์ดํ›„๋กœ ์ง€์›๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.
1.3.95-debian10/-ubuntu18์€ log4j2 ์ทจ์•ฝ์ ์ด ํ•ด๊ฒฐ๋œ ์ตœ์ข… ์ถœ์‹œ ๋ฒ„์ „์ž…๋‹ˆ๋‹ค. ์ฐธ๊ณ : ์ด์ „์— ์ถœ์‹œ๋œ ๋ฒ„์ „์€ ์ทจ์•ฝํ•˜๋ฏ€๋กœ ์—…๊ทธ๋ ˆ์ด๋“œํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.
1.4-debian9 Apache Spark 2.4.5
Apache Hadoop 2.9.2
Apache Pig 0.17.0
Apache Hive 2.3.7
Cloud Storage ์ปค๋„ฅํ„ฐ 1.9.17-hadoop2
Python 3.6
Scala 2.11.12
Zookeeper 3.4.13
2019/03/22 2020/07/10 2020/07/10 ํ˜„์žฌ ์ง€์›๋˜์ง€ ์•Š์Œ.
1.4.33-debian9๊ฐ€ ์ตœ์ข… ์ถœ์‹œ ๋ฒ„์ „์ž…๋‹ˆ๋‹ค.
1.3-debian9 Apache Spark 2.3.4
Apache Hadoop 2.9.2
Apache Pig 0.17.0
Apache Hive 2.3.7
Cloud Storage ์ปค๋„ฅํ„ฐ 1.9.17-hadoop2
Python 2.7
Scala 2.11.8
Zookeeper 3.4.13
2018/06/29 2020/07/10 2020/07/10 ํ˜„์žฌ ์ง€์›๋˜์ง€ ์•Š์Œ.
1.3.62-debian9๊ฐ€ ์ตœ์ข… ์ถœ์‹œ ๋ฒ„์ „์ž…๋‹ˆ๋‹ค.
1.2-debian9 Apache Spark 2.2.3
Apache Hadoop 2.8.5
Apache Pig 0.16.0
Apache Hive 2.1.1
Cloud Storage ์ปค๋„ฅํ„ฐ 1.6.10-hadoop2
BigQuery ์ปค๋„ฅํ„ฐ 0.10.11-hadoop2
Python 2.7
Scala 2.11.8
Zookeeper 3.4.13
2017/07/21 2020/07/10 2020/07/10 ํ˜„์žฌ ์ง€์›๋˜์ง€ ์•Š์Œ.
1.2.102-debian9๊ฐ€ ์ตœ์ข… ์ถœ์‹œ ๋ฒ„์ „์ž…๋‹ˆ๋‹ค.
1.1-debian9 Apache Spark 2.0.2
Apache Hadoop 2.7.7
Apache Pig 0.16.0
Apache Hive 2.1.1
Cloud Storage ์ปค๋„ฅํ„ฐ 1.6.10-hadoop2
BigQuery ์ปค๋„ฅํ„ฐ 0.10.11-hadoop2
2016/08/08 2019/09/26 2019/10/01๋ถ€ํ„ฐ ์ง€์›๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.
1.1.121-debian9๊ฐ€ ์ตœ์ข… ์ถœ์‹œ ๋ฒ„์ „์ž…๋‹ˆ๋‹ค.
1.0-debian9 Apache Spark 1.6.2
Apache Hadoop 2.7.4
Apache Pig 0.15.0
Apache Hive 1.2.1
Cloud Storage ์ปค๋„ฅํ„ฐ 1.6.10-hadoop2
BigQuery ์ปค๋„ฅํ„ฐ 0.10.11-hadoop2
2016/02/22 2019/05/09 GA ์ด๋ฏธ์ง€์˜ ์ฒ˜์Œ ์ถœ์‹œ ๋ฒ„์ „.
2019/04/01 ํ˜„์žฌ ์ง€์›๋˜์ง€ ์•Š์Œ.
1.0.119-debian9๊ฐ€ ์ตœ์ข… ์ถœ์‹œ ๋ฒ„์ „์ž…๋‹ˆ๋‹ค.
0.2 Apache Spark 1.5.2
Apache Hadoop 2.7.1
Apache Pig 0.15.0
Apache Hive 1.2.1
Cloud Storage ์ปค๋„ฅํ„ฐ 1.5.1-hadoop2
BigQuery ์ปค๋„ฅํ„ฐ 0.7.7-hadoop2
2015/11/18 2016/08/02 ๋ฒ ํƒ€ ์ด๋ฏธ์ง€์˜ ๋‘ ๋ฒˆ์งธ ์ถœ์‹œ ๋ฒ„์ „์ž…๋‹ˆ๋‹ค.
0.1 Apache Spark 1.5.0
Apache Hadoop 2.7.1
Apache Pig 0.14.10
Apache Hive 1.0
Cloud Storage ์ปค๋„ฅํ„ฐ 1.5.1-hadoop2
BigQuery ์ปค๋„ฅํ„ฐ 0.7.7-hadoop2
2015/09/23 2016/08/02 Dataproc ๋ฒ ํƒ€ ์ถœ์‹œ.
Spark 1.5๋Š” Hive 1.2๋ฅผ ๋Œ€์ƒ์œผ๋กœ ์ปดํŒŒ์ผ๋˜์—ˆ์Šต๋‹ˆ๋‹ค.