Dataproc ์„ ํƒ์  Solr ๊ตฌ์„ฑ์š”์†Œ

์„ ํƒ์  ๊ตฌ์„ฑ์š”์†Œ ๊ธฐ๋Šฅ์„ ์‚ฌ์šฉํ•˜์—ฌ Dataproc ํด๋Ÿฌ์Šคํ„ฐ๋ฅผ ๋งŒ๋“ค ๋•Œ Solr๊ณผ ๊ฐ™์€ ์ถ”๊ฐ€ ๊ตฌ์„ฑ์š”์†Œ๋ฅผ ์„ค์น˜ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด ํŽ˜์ด์ง€์—์„œ๋Š” Solr ๊ตฌ์„ฑ์š”์†Œ๋ฅผ ์„ค๋ช…ํ•ฉ๋‹ˆ๋‹ค.

Apache Solr ๊ตฌ์„ฑ์š”์†Œ๋Š” ์˜คํ”ˆ์†Œ์Šค ๊ธฐ๋ฐ˜์˜ ์—”ํ„ฐํ”„๋ผ์ด์ฆˆ ๊ฒ€์ƒ‰ ํ”Œ๋žซํผ์ž…๋‹ˆ๋‹ค. Solr ์„œ๋ฒ„ ๋ฐ ์›น UI๋Š” ํด๋Ÿฌ์Šคํ„ฐ์˜ ๋งˆ์Šคํ„ฐ ๋…ธ๋“œ์— ์žˆ๋Š” ํฌํŠธ 8983์—์„œ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

Solr ํŒŒ์ผ ์œ ์ง€: ๊ธฐ๋ณธ์ ์œผ๋กœ Solr๋Š” HDFS์—์„œ ์ƒ‰์ธ ๋ฐ ํŠธ๋žœ์žญ์…˜ ๋กœ๊ทธ ํŒŒ์ผ์„ ์“ฐ๊ณ  ์ฝ์Šต๋‹ˆ๋‹ค. Solr ํŒŒ์ผ์„ ์œ ์ง€ํ•˜๋ ค๋ฉด ๊ตฌ์„ฑ์š”์†Œ๋ฅผ ์„ค์น˜ํ•  ๋•Œ dataproc:solr.gcs.path ํด๋Ÿฌ์Šคํ„ฐ ์†์„ฑ์„ ์„ค์ •ํ•˜์—ฌ Cloud Storage ๊ฒฝ๋กœ๋ฅผ Solr ํ™ˆ ๋””๋ ‰ํ„ฐ๋ฆฌ๋กœ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

๊ตฌ์„ฑ์š”์†Œ ์„ค์น˜

Dataproc ํด๋Ÿฌ์Šคํ„ฐ๋ฅผ ๋งŒ๋“ค ๋•Œ ๊ตฌ์„ฑ์š”์†Œ๋ฅผ ์„ค์น˜ํ•ฉ๋‹ˆ๋‹ค. ๊ตฌ์„ฑ์š”์†Œ๋Š” Dataproc ๋ฒ„์ „ 1.3 ์ด์ƒ์œผ๋กœ ๋งŒ๋“  ํด๋Ÿฌ์Šคํ„ฐ์— ์ถ”๊ฐ€ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๊ฐ Dataproc ์ด๋ฏธ์ง€ ์ถœ์‹œ์— ํฌํ•จ๋œ ๊ตฌ์„ฑ์š”์†Œ ๋ฒ„์ „์€ ์ง€์›๋˜๋Š” Dataproc ๋ฒ„์ „์„ ์ฐธ์กฐํ•˜์„ธ์š”.

gcloud ๋ช…๋ น์–ด

Solr ๊ตฌ์„ฑ์š”์†Œ๋ฅผ ํฌํ•จํ•˜๋Š” Dataproc ํด๋Ÿฌ์Šคํ„ฐ๋ฅผ ๋งŒ๋“ค๋ ค๋ฉด --optional-components ํ”Œ๋ž˜๊ทธ์™€ ํ•จ๊ป˜ gcloud dataproc clusters create cluster-name ๋ช…๋ น์–ด๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ์•„๋ž˜์˜ ์ƒ˜ํ”Œ ๋ช…๋ น์–ด๋Š” ์„ ํƒ์ ์ธ properties ํ”Œ๋ž˜๊ทธ๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ Cloud Storage ๊ฒฝ๋กœ๋ฅผ Solr ํ™ˆ ๋””๋ ‰ํ„ฐ๋ฆฌ๋กœ ์„ค์ •ํ•ฉ๋‹ˆ๋‹ค.

gcloud dataproc clusters create cluster-name \
    --region=region \
    --optional-components=SOLR \
    --enable-component-gateway \
    ... other flags
ํด๋Ÿฌ์Šคํ„ฐ ์†์„ฑ --properties="dataproc:solr.gcs.path=gs://bucket-name/"์„ gcloud dataproc clusters create๋ช…๋ น์–ด์— ์ถ”๊ฐ€ํ•˜์—ฌ Solr ๋ฌธ์„œ๊ฐ€ ์ €์žฅ๋  Cloud Storage ๋ฒ„ํ‚ท์„ ์„ค์ •ํ•ฉ๋‹ˆ๋‹ค(Solr ํ™ˆ ๋””๋ ‰ํ„ฐ๋ฆฌ)

REST API

Solr ๊ตฌ์„ฑ์š”์†Œ๋Š” clusters.create ์š”์ฒญ์˜ ์ผ๋ถ€๋กœ SoftwareConfig.Component๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ Dataproc API๋ฅผ ํ†ตํ•ด ์ง€์ •ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

Console

  1. ๊ตฌ์„ฑ์š”์†Œ ๋ฐ ๊ตฌ์„ฑ์š”์†Œ ๊ฒŒ์ดํŠธ์›จ์ด๋ฅผ ์‚ฌ์šฉ ์„ค์ •ํ•ฉ๋‹ˆ๋‹ค.
    • Google Cloud ์ฝ˜์†”์—์„œ Dataproc ํด๋Ÿฌ์Šคํ„ฐ ๋งŒ๋“ค๊ธฐ ํŽ˜์ด์ง€๋ฅผ ์—ฝ๋‹ˆ๋‹ค. ํด๋Ÿฌ์Šคํ„ฐ ์„ค์ • ํŒจ๋„์ด ์„ ํƒ๋˜์—ˆ์Šต๋‹ˆ๋‹ค.
    • ๊ตฌ์„ฑ์š”์†Œ ์„น์…˜์—์„œ ๋‹ค์Œ์„ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค.
      • ์„ ํƒ์  ๊ตฌ์„ฑ์š”์†Œ ์•„๋ž˜์—์„œ ํด๋Ÿฌ์Šคํ„ฐ์— ์„ค์น˜ํ•  Solr ๋ฐ ๊ธฐํƒ€ ์„ ํƒ์ ์ธ ๊ตฌ์„ฑ์š”์†Œ๋ฅผ ์„ ํƒํ•ฉ๋‹ˆ๋‹ค.
      • ๊ตฌ์„ฑ์š”์†Œ ๊ฒŒ์ดํŠธ์›จ์ด ์•„๋ž˜์—์„œ ๊ตฌ์„ฑ์š”์†Œ ๊ฒŒ์ดํŠธ์›จ์ด ์‚ฌ์šฉ ์„ค์ •์„ ์„ ํƒํ•ฉ๋‹ˆ๋‹ค(๊ตฌ์„ฑ์š”์†Œ ๊ฒŒ์ดํŠธ์›จ์ด URL ๋ณด๊ธฐ ๋ฐ ์•ก์„ธ์Šค ์ฐธ์กฐ).