[Bug]: PubSubIO writes from bounded PCollection cannot be more than 100 bytes

[jonathan-lemos]

What happened?

Writing to Pub/Sub currently results in the following issue:

javax.naming.SizeLimitExceededException: Pubsub message of length 220 exceeds maximum of 100 bytes, when considering the payload and attributes. See https://cloud.google.com/pubsub/quotas#resource_limits

This happens because this call passes in the maximum number of messages allowed in a batch, when it should pass in the maximum number of bytes allowed in a batch.

This issue only seems to present when the PCollection is bounded.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[jonathan-lemos]

.take-issue

[bvolpato]

The issue was fixed and will be available in Beam 2.49.0.

The impact is specific to batch jobs that use PubsubIO, as the wrong validation is part of PubsubBoundedWriter.

Thanks for the fix @jonathan-lemos.

(I am repeating some details as we are going to link this at https://cloud.google.com/dataflow/docs/support/sdk-version-support-status)