ใๅฐ Pub/Sub ็ๅ งๅฎนๅณ่ผธๅฐ Splunk ็ฏๆฌใๆฏไธ็จฎไธฒๆต็ฎก้๏ผๅฏ่ฎๅ Pub/Sub ่จ้ฑ็่จๆฏ๏ผไธฆ้้ Splunk ็ HTTP Event Collector (HEC) ๅฐ่จๆฏ้ ฌ่ผๅฏซๅ ฅ Splunkใ้ๅ็ฏๆฌๆๅธธ่ฆ็็จ้ๆฏๅฐ่จ้ๅฏๅบ่ณ Splunkใๅฆ่ฆๆฅ็ๅบ็คๅทฅไฝๆต็จ็็ฏไพ๏ผ่ซๅ้ฑใไฝฟ็จ Dataflow ๅฐๅฏ็จๆผๅฏฆ้ๅทฅไฝ็่จ้ๆชๅฏๅบไฝๆฅญ้จ็ฝฒ่ณ Splunkใใ
ๆจไนๅฏไปฅๅจๅฏซๅ ฅ Splunk ๅ๏ผๅฐ JavaScript ไฝฟ็จ่ ๅฎ็พฉๅฝๅผๅฅ็จ่ณ่จๆฏ้ ฌ่ผใ็ณป็ตฑๆๅฐ็ผ็่็ๅ้ก็ๆๆ่จๆฏ่ฝ้่ณๆช่็็ Pub/Sub ไธป้ก๏ผไปฅไพฟ้ฒไธๆญฅๅท่ก็้ฃๆ่งฃๅ้ๆฐ่็ใ
็บไบๅขๅ ๅคไธๅฑคๅฎๅ จ้ฒ่ญท๏ผๆจไนๅฏไปฅๅณ้ Cloud KMS ้้ฐ๏ผไปฅๅไฝฟ็จ Cloud KMS ้้ฐๅ ๅฏ็ Base64 ็ทจ็ขผ HEC ๆฌๆๅๆธใๅฆ่ฆ้ฒไธๆญฅ็ญ่งฃๅฆไฝๅ ๅฏ HEC ๆฌๆๅๆธ๏ผ่ซๅ้ฑ Cloud KMS API ๅ ๅฏ็ซฏ้ปใ
็ฎก้็ธ้่ฆๅฎ
- ๅท่ก็ฎก้ไนๅ๏ผไพๆบ Pub/Sub ่จ้ฑ้ ็ฎๅฟ ้ ๅทฒ็ถๅญๅจใ
- ๅท่ก็ฎก้ไนๅ๏ผPub/Sub ๆช่็ไธป้กๅฟ ้ ๅทฒ็ถๅญๅจใ
- Splunk HEC ็ซฏ้ปๅฟ ้ ๅฏ้้ Dataflow ๅทฅไฝ็ซ็็ถฒ่ทฏๅญๅใ
- ๅฟ ้ ็ข็ Splunk HEC ๆฌๆไธ่ฉฒๆฌๆ้ ๅฏไพไฝฟ็จใ
็ฏๆฌๅๆธ
ๅฟ ่ฆๅๆธ
- inputSubscription๏ผ่ฆ่ฎๅ่ผธๅ
ฅๅ
งๅฎน็ Pub/Sub ่จ้ฑ้
็ฎใไพๅฆ๏ผ
projects/your-project-id/subscriptions/your-subscription-nameใ - url๏ผSplunk HEC ็ถฒๅใ็ถฒๅๅฟ
้ ๅฏๅพ็ฎก้ๅท่ก็่ๆฌ็งๆ้ฒ (VPC) ่ทฏ็ฑใไพๅฆ๏ผ
https://splunk-hec-host:8088ใ - outputDeadletterTopic๏ผ่ฆๅฐ็กๆณๅณ้็่จๆฏ่ฝ้่ณ็ Pub/Sub ไธป้กใไพๅฆ๏ผ
projects/<PROJECT_ID>/topics/<TOPIC_NAME>ใ
้ธ็จๅๆธ
- token๏ผSplunk HEC ้ฉ่ญๆฌๆใๅฆๆ
tokenSourceๅๆธ่จญ็บPLAINTEXTๆKMS๏ผๅๅฟ ้ ๆไพ้้ ๅฑฌๆงใ - batchCount๏ผๅฐๅคๅไบไปถๅณ้่ณ Splunk ็ๆนๆฌกๅคงๅฐใ้ ่จญๅผ็บ
1(ไธๅๆน)ใ - disableCertificateValidation๏ผๅ็จ SSL ๆ่ญ้ฉ่ญใ้ ่จญ
false(ๅทฒๅ็จ้ฉ่ญ)ใๅฆๆtrue๏ผ็ณป็ตฑไธๆ้ฉ่ญๆ่ญ (ๆๆๆ่ญ้ฝๅไฟกไปป)๏ผไธฆๆๅฟฝ็ฅrootCaCertificatePathๅๆธใ - ๅนณ่ก่็้๏ผๅนณ่ก่ฆๆฑๆธ้ไธ้ใ้ ่จญๅผ็บ
1(็กๅนณ่ก่็)ใ - includePubsubMessage๏ผๅจ้
ฌ่ผไธญๅ ๅ
ฅๅฎๆด็ Pub/Sub ่จๆฏใ้ ่จญ
false(้ ฌ่ผไธญๅชๆๅ ๅซ่ณๆๅ ็ด )ใ - tokenKMSEncryptionKey๏ผ็จไพ่งฃๅฏ HEC ๆฌๆๅญไธฒ็ Cloud KMS ้้ฐใๅฆๆ tokenSource ่จญ็บ KMS๏ผๅฐฑๅฟ
้ ๆไพ้้
ๅๆธใๅฆๆๆไพ Cloud KMS ้้ฐ๏ผๅๅฟ
้ ไปฅๅ ๅฏๆนๅผๅณ้ HEC ๆฌๆๅญไธฒใไพๅฆ๏ผ
projects/your-project-id/locations/global/keyRings/your-keyring/cryptoKeys/your-key-nameใ - tokenSecretId๏ผๆฌๆ็ Secret Manager ๅฏ้ฐ IDใๅฆๆ tokenSource ่จญ็บ
SECRET_MANAGER๏ผๅฐฑๅฟ ้ ๆไพ้้ ๅๆธใไพๅฆ๏ผprojects/your-project-id/secrets/your-secret/versions/your-secret-versionใ - tokenSource๏ผๆฌๆไพๆบใๅ
่จฑ็ๅผๅ
ๆฌ๏ผ
PLAINTEXTใKMSๅSECRET_MANAGERใไฝฟ็จ Secret Manager ๆ๏ผๆจๅฟ ้ ๆไพ้ๅๅๆธใๅฆๆtokenSource่จญ็บKMSใtokenKMSEncryptionKeyไธฆ็ถ้ๅ ๅฏ๏ผๅๅฟ ้ ๆไพtokenใๅฆๆtokenSource่จญ็บSECRET_MANAGER๏ผๅๅฟ ้ ๆไพtokenSecretIdใๅฆๆtokenSource่จญ็บPLAINTEXT๏ผๅๅฟ ้ ๆไพtokenใ - rootCaCertificatePath๏ผCloud Storage ไธญๆ น CA ๆ่ญ็ๅฎๆด็ถฒๅใCloud Storage ไธญๆไพ็ๆ่ญๅฟ
้ ๆก็จ DER ็ทจ็ขผ๏ผไธๅฏๆก็จไบ้ฒไฝๆๅฏๅๅฐ (Base64) ็ทจ็ขผใๅฆๆๆ่ญๆฏไปฅ Base64 ็ทจ็ขผๆไพ๏ผ้้ ญๅฟ
้ ็บใ-----BEGIN CERTIFICATE-----ใ๏ผ็ตๅฐพๅฟ
้ ็บใ-----END CERTIFICATE-----ใใๅฆๆๆไพ้ๅๅๆธ๏ผ็ณป็ตฑๆๆทๅ้ๅ็งๆ CA ๆ่ญๆชๆก๏ผไธฆๆฐๅข่ณ Dataflow ๅทฅไฝไบบๅก็ไฟกไปปๅญๆพๅ๏ผไปฅไพฟ้ฉ่ญ Splunk HEC ็ซฏ้ป็ SSL ๆ่ญใๅฆๆชๆไพ้ๅๅๆธ๏ผ็ณป็ตฑๆไฝฟ็จ้ ่จญ็ไฟกไปปๅฒๅญๅใไพๅฆ๏ผ
gs://mybucket/mycerts/privateCA.crtใ - enableBatchLogs๏ผๆๅฎๆฏๅฆๆ็บๅฏซๅ
ฅ Splunk ็ๆนๆฌกๅ็จ่จ้ใ้ ่จญๅผ๏ผ
trueใ - enableGzipHttpCompression๏ผๆๅฎๅณ้่ณ Splunk HEC ็ HTTP ่ฆๆฑๆฏๅฆๆๅฃ็ธฎ (gzip ๅ
งๅฎน็ทจ็ขผ)ใ้ ่จญๅผ๏ผ
trueใ - javascriptTextTransformGcsPath๏ผๅฎ็พฉ่ฆไฝฟ็จ็ JavaScript ไฝฟ็จ่
ๅฎ็พฉๅฝๅผ (UDF) ็ .js ๆชๆก Cloud Storage URIใไพๅฆ๏ผ
gs://my-bucket/my-udfs/my_file.jsใ - javascriptTextTransformFunctionName๏ผ่ฆไฝฟ็จ็ JavaScript ไฝฟ็จ่
ๅฎ็พฉๅฝๅผ (UDF) ๅ็จฑใ่ไพไพ่ชช๏ผๅฆๆๆจ็ JavaScript ๅฝๅผ็จๅผ็ขผๆฏ
myTransform(inJson) { /*...do stuff...*/ }๏ผๅๅฝๅผๅ็จฑๅฐฑๆฏmyTransformใๅฆ้ JavaScript UDF ็ฏไพ๏ผ่ซๅ้ฑ UDF ็ฏไพ (https://github.com/GoogleCloudPlatform/DataflowTemplates#udf-examples)ใ - javascriptTextTransformReloadIntervalMinutes๏ผๅฎ็พฉๅทฅไฝไบบๅกๆชขๆฅ JavaScript UDF ่ฎๆดไปฅ้ๆฐ่ผๅ ฅๆชๆก็้้ใ้ ่จญๅผ็บ 0ใ
ไฝฟ็จ่ ๅฎ็พฉๅฝๅผ
ๆจๅฏไปฅ่ฆ้่ฆๆฐๅฏซไฝฟ็จ่ ๅฎ็พฉๅฝๅผ (UDF) ไพๆดๅ ้ๅ็ฏๆฌใ็ฏๆฌๆ้ๅฐๆฏๅ่ผธๅ ฅๅ ็ด ๅผๅซ UDFใๅ ็ด ้ ฌ่ผๆๅบๅๅ็บ JSON ๅญไธฒใ่ฉณๆ ่ซๅ้ฑใ็บ Dataflow ็ฏๆฌๅปบ็ซไฝฟ็จ่ ๅฎ็พฉๅฝๅผใใ
ๅฝๅผ่ฆๆ ผ
UDF ็่ฆๆ ผๅฆไธ๏ผ
- ่ผธๅ ฅ๏ผPub/Sub ่จๆฏ่ณๆๆฌไฝ๏ผ ๅบๅๅ็บ JSON ๅญไธฒใ
- ่ผธๅบ๏ผ่ฆๅณ้่ณ Splunk HEC ไบไปถ็ซฏ้ป็ไบไปถ่ณๆใ่ผธๅบๅ งๅฎนๅฟ ้ ๆฏๅญไธฒๆๅญไธฒๅ็ JSON ็ฉไปถใ
ๅท่ก็ฏๆฌ
ๆงๅถๅฐ
- ๅๅพ Dataflow ็ใCreate job from templateใ(้้็ฏๆฌๅปบ็ซๅทฅไฝ) ้ ้ขใ ๅๅพใไพๆ็ฏๆฌๅปบ็ซๅทฅไฝใ
- ๅจใๅทฅไฝๅ็จฑใๆฌไฝไธญ๏ผ่ผธๅ ฅๅฐๅฑฌๅทฅไฝๅ็จฑใ
- ้ธ็จ๏ผๅฆ่ฆไฝฟ็จๅๅ็ซฏ้ป๏ผ่ซๅพไธๆๅผ้ธๅฎไธญ้ธๅๅผใ้ ่จญๅๅ็บ
us-central1ใๅฆ้ๅฏๅท่ก Dataflow ๅทฅไฝ็ๅฐๅๆธ ๅฎ๏ผ่ซๅ้ฑใDataflow ไฝ็ฝฎใใ
- ๅพใDataflow templateใ(Dataflow ็ฏๆฌ) ไธๆๅผ้ธๅฎไธญ้ธๅ the Pub/Sub to Splunk templateใ
- ๅจๆไพ็ๅๆธๆฌไฝไธญ่ผธๅ ฅๅๆธๅผใ
- ้ธ็จ๏ผๅฆ่ฆๅพใๅ ้่็ไธๆฌกใๅๆ่ณใ่ณๅฐไธๆฌกใไธฒๆตๆจกๅผ๏ผ่ซ้ธๅใ่ณๅฐไธๆฌกใใ
- ๆไธไธใRun Jobใ(ๅท่กๅทฅไฝ)ใ
gcloud
ๅจๆฎผๅฑคๆ็ต็ซฏๆฉไธญๅท่ก็ฏๆฌ๏ผ
gcloud dataflow jobs run JOB_NAME \ --gcs-location gs://dataflow-templates-REGION_NAME/VERSION/Cloud_PubSub_to_Splunk \ --region REGION_NAME \ --staging-location STAGING_LOCATION \ --parameters \ inputSubscription=projects/PROJECT_ID/subscriptions/INPUT_SUBSCRIPTION_NAME,\ token=TOKEN,\ url=URL,\ outputDeadletterTopic=projects/PROJECT_ID/topics/DEADLETTER_TOPIC_NAME,\ javascriptTextTransformGcsPath=PATH_TO_JAVASCRIPT_UDF_FILE,\ javascriptTextTransformFunctionName=JAVASCRIPT_FUNCTION,\ batchCount=BATCH_COUNT,\ parallelism=PARALLELISM,\ disableCertificateValidation=DISABLE_VALIDATION,\ rootCaCertificatePath=ROOT_CA_CERTIFICATE_PATH
ๆดๆนไธๅๅ งๅฎน๏ผ
JOB_NAME๏ผ ๆจ้ธๆ็ไธ้่คๅทฅไฝๅ็จฑREGION_NAME๏ผ ๆจ่ฆ้จ็ฝฒ Dataflow ๅทฅไฝ็ๅฐๅ๏ผไพๅฆus-central1VERSION๏ผ ๆจ่ฆไฝฟ็จ็็ฏๆฌ็ๆฌๆจๅฏไปฅไฝฟ็จไธๅๅผ๏ผ
latest๏ผไฝฟ็จ็ฏๆฌ็ๆๆฐ็ๆฌ๏ผ่ฉฒ็ๆฌไฝๆผๅผๅไธญ้ไพๆฅๆๅฝๅ็ไธๅฑค่ณๆๅคพ๏ผgs://dataflow-templates-REGION_NAME/latest/- ็ๆฌๅ็จฑ (ไพๅฆ
2023-09-12-00_RC00)๏ผ็จๆผๆๅฎ็ฏๆฌ็ๆฌ๏ผ่ฉฒ็ๆฌๆไปฅๅทข็็ตๆงๅญๆพๅจๅผๅไธญไพๆฅๆๅฝๅ็ไธๅฑค่ณๆๅคพไธญ๏ผgs://dataflow-templates-REGION_NAME/
STAGING_LOCATION๏ผ็จๆผๆซๅญๆฌๆฉๆชๆก็ไฝ็ฝฎ (ไพๅฆgs://your-bucket/staging)INPUT_SUBSCRIPTION_NAME๏ผPub/Sub ่จ้ฑ้ ็ฎๅ็จฑTOKEN๏ผSplunk ็ HTTP ไบไปถๆถ้ๅจๆฌๆURL๏ผSplunk HTTP ไบไปถๆถ้ๅจ็็ถฒๅ่ทฏๅพ (ไพๅฆhttps://splunk-hec-host:8088)DEADLETTER_TOPIC_NAME๏ผPub/Sub ไธป้กๅ็จฑJAVASCRIPT_FUNCTION๏ผ ๆจ่ฆไฝฟ็จ็ JavaScript ไฝฟ็จ่ ๅฎ็พฉๅฝๅผ (UDF) ๅ็จฑ่ไพไพ่ชช๏ผๅฆๆๆจ็ JavaScript ๅฝๅผ็จๅผ็ขผๆฏ
myTransform(inJson) { /*...do stuff...*/ }๏ผๅๅฝๅผๅ็จฑๅฐฑๆฏmyTransformใๅฆ้ JavaScript UDF ็ฏไพ๏ผ่ซๅ้ฑใUDF ็ฏไพใใPATH_TO_JAVASCRIPT_UDF_FILE๏ผ ๅฎ็พฉ่ฆไฝฟ็จ็ JavaScript ไฝฟ็จ่ ๅฎ็พฉๅฝๅผ (UDF) ็.jsๆชๆก Cloud Storage URI๏ผไพๅฆgs://my-bucket/my-udfs/my_file.jsBATCH_COUNT๏ผๅฐๅคๅไบไปถๅณ้่ณ Splunk ๆไฝฟ็จ็ๆนๆฌกๅคงๅฐPARALLELISM๏ผ็จๆผๅฐไบไปถๅณ้่ณ Splunk ็ๅนณ่ก่ฆๆฑๆธ้DISABLE_VALIDATION๏ผtrueๅฆๆ่ฆๅ็จ SSL ๆ่ญ้ฉ่ญROOT_CA_CERTIFICATE_PATH๏ผCloud Storage ไธญๆ น CA ๆ่ญ็่ทฏๅพ (ไพๅฆgs://your-bucket/privateCA.crt)
API
ๅฆ่ฆไฝฟ็จ REST API ๅท่ก็ฏๆฌ๏ผ่ซๅณ้ HTTP POST ่ฆๆฑใๅฆ่ฆ้ฒไธๆญฅ็ญ่งฃ API ๅๆๆฌ็ฏๅ๏ผ่ซๅ้ฑ projects.templates.launchใ
POST https://dataflow.googleapis.com/v1b3/projects/PROJECT_ID/locations/LOCATION/templates:launch?gcsPath=gs://dataflow-templates-LOCATION/VERSION/Cloud_PubSub_to_Splunk { "jobName": "JOB_NAME", "environment": { "ipConfiguration": "WORKER_IP_UNSPECIFIED", "additionalExperiments": [] }, "parameters": { "inputSubscription": "projects/PROJECT_ID/subscriptions/INPUT_SUBSCRIPTION_NAME", "token": "TOKEN", "url": "URL", "outputDeadletterTopic": "projects/PROJECT_ID/topics/DEADLETTER_TOPIC_NAME", "javascriptTextTransformGcsPath": "PATH_TO_JAVASCRIPT_UDF_FILE", "javascriptTextTransformFunctionName": "JAVASCRIPT_FUNCTION", "batchCount": "BATCH_COUNT", "parallelism": "PARALLELISM", "disableCertificateValidation": "DISABLE_VALIDATION", "rootCaCertificatePath": "ROOT_CA_CERTIFICATE_PATH" } }
ๆดๆนไธๅๅ งๅฎน๏ผ
PROJECT_ID๏ผ ๆจ่ฆๅท่ก Dataflow ๅทฅไฝ็ๅฐๆก ID Google CloudJOB_NAME๏ผ ๆจ้ธๆ็ไธ้่คๅทฅไฝๅ็จฑLOCATION๏ผ ๆจ่ฆ้จ็ฝฒ Dataflow ๅทฅไฝ็ๅฐๅ๏ผไพๅฆus-central1VERSION๏ผ ๆจ่ฆไฝฟ็จ็็ฏๆฌ็ๆฌๆจๅฏไปฅไฝฟ็จไธๅๅผ๏ผ
latest๏ผไฝฟ็จ็ฏๆฌ็ๆๆฐ็ๆฌ๏ผ่ฉฒ็ๆฌไฝๆผๅผๅไธญ้ไพๆฅๆๅฝๅ็ไธๅฑค่ณๆๅคพ๏ผgs://dataflow-templates-REGION_NAME/latest/- ็ๆฌๅ็จฑ (ไพๅฆ
2023-09-12-00_RC00)๏ผ็จๆผๆๅฎ็ฏๆฌ็ๆฌ๏ผ่ฉฒ็ๆฌๆไปฅๅทข็็ตๆงๅญๆพๅจๅผๅไธญไพๆฅๆๅฝๅ็ไธๅฑค่ณๆๅคพไธญ๏ผgs://dataflow-templates-REGION_NAME/
STAGING_LOCATION๏ผ็จๆผๆซๅญๆฌๆฉๆชๆก็ไฝ็ฝฎ (ไพๅฆgs://your-bucket/staging)INPUT_SUBSCRIPTION_NAME๏ผPub/Sub ่จ้ฑ้ ็ฎๅ็จฑTOKEN๏ผSplunk ็ HTTP ไบไปถๆถ้ๅจๆฌๆURL๏ผSplunk HTTP ไบไปถๆถ้ๅจ็็ถฒๅ่ทฏๅพ (ไพๅฆhttps://splunk-hec-host:8088)DEADLETTER_TOPIC_NAME๏ผPub/Sub ไธป้กๅ็จฑJAVASCRIPT_FUNCTION๏ผ ๆจ่ฆไฝฟ็จ็ JavaScript ไฝฟ็จ่ ๅฎ็พฉๅฝๅผ (UDF) ๅ็จฑ่ไพไพ่ชช๏ผๅฆๆๆจ็ JavaScript ๅฝๅผ็จๅผ็ขผๆฏ
myTransform(inJson) { /*...do stuff...*/ }๏ผๅๅฝๅผๅ็จฑๅฐฑๆฏmyTransformใๅฆ้ JavaScript UDF ็ฏไพ๏ผ่ซๅ้ฑใUDF ็ฏไพใใPATH_TO_JAVASCRIPT_UDF_FILE๏ผ ๅฎ็พฉ่ฆไฝฟ็จ็ JavaScript ไฝฟ็จ่ ๅฎ็พฉๅฝๅผ (UDF) ็.jsๆชๆก Cloud Storage URI๏ผไพๅฆgs://my-bucket/my-udfs/my_file.jsBATCH_COUNT๏ผๅฐๅคๅไบไปถๅณ้่ณ Splunk ๆไฝฟ็จ็ๆนๆฌกๅคงๅฐPARALLELISM๏ผ็จๆผๅฐไบไปถๅณ้่ณ Splunk ็ๅนณ่ก่ฆๆฑๆธ้DISABLE_VALIDATION๏ผtrueๅฆๆ่ฆๅ็จ SSL ๆ่ญ้ฉ่ญROOT_CA_CERTIFICATE_PATH๏ผCloud Storage ไธญๆ น CA ๆ่ญ็่ทฏๅพ (ไพๅฆgs://your-bucket/privateCA.crt)
ๅพ็บๆญฅ้ฉ
- ็ญ่งฃ Dataflow ็ฏๆฌใ
- ่ซๅ้ฑ Google ๆไพ็็ฏๆฌๆธ ๅฎใ