[Bug]: Flaky Spanner Integration test due to "Could not initialize class com.google.spanner.v1.Session$LabelsDefault"

[Abacn]

Related Template(s)

Spanner templates

Template Version

N/A

What happened?

Spanner PR is flaky, integration test job failed launch. Example error:

2025-02-13 13:52:51.974 EST
java.lang.NoClassDefFoundError: Could not initialize class com.google.spanner.v1.Session$LabelsDefaultEntryHolder
2025-02-13 13:52:51.974 EST
at com.google.spanner.v1.Session.internalGetLabels(Session.java:147)
2025-02-13 13:52:51.974 EST
at com.google.spanner.v1.Session.getSerializedSize(Session.java:490)
2025-02-13 13:52:51.974 EST
at com.google.protobuf.CodedOutputStream.computeMessageSizeNoTag(CodedOutputStream.java:861)
2025-02-13 13:52:51.974 EST
at com.google.protobuf.CodedOutputStream.computeMessageSize(CodedOutputStream.java:641)
2025-02-13 13:52:51.974 EST
at com.google.spanner.v1.BatchCreateSessionsRequest.getSerializedSize(BatchCreateSessionsRequest.java:232)
2025-02-13 13:52:51.974 EST
at io.grpc.protobuf.lite.ProtoInputStream.available(ProtoInputStream.java:108)
2025-02-13 13:52:51.974 EST
at io.grpc.internal.MessageFramer.getKnownLength(MessageFramer.java:204)
2025-02-13 13:52:51.974 EST
at io.grpc.internal.MessageFramer.writePayload(MessageFramer.java:139)
2025-02-13 13:52:51.974 EST
at io.grpc.internal.AbstractStream.writeMessage(AbstractStream.java:66)
2025-02-13 13:52:51.974 EST
at io.grpc.internal.ForwardingClientStream.writeMessage(ForwardingClientStream.java:37)
2025-02-13 13:52:51.974 EST
at io.grpc.internal.DelayedStream$6.run(DelayedStream.java:282)
2025-02-13 13:52:51.974 EST
at io.grpc.internal.DelayedStream.drainPendingCalls(DelayedStream.java:182)
2025-02-13 13:52:51.975 EST
at io.grpc.internal.DelayedStream.access$100(DelayedStream.java:44)
2025-02-13 13:52:51.975 EST
at io.grpc.internal.DelayedStream$4.run(DelayedStream.java:148)
...
2025-02-13 13:52:51.978 EST
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.ExceptionInInitializerError [in thread "grpc-default-executor-0"]
2025-02-13 13:52:51.978 EST
at com.google.spanner.v1.Session$LabelsDefaultEntryHolder.<clinit>(Session.java:132)

different tests failed same reason each time. See #2177 for example.

Relevant log output

[darshan-sj]

NoClassDef found should happen everytime the same tests are run. I don't understand why this is flaky.

Is there any difference between the way we run the tests in the Java PR workflow and Spanner PR workflow?

The Spanner PR continues workflow has one flaky test SpannerToSourceDbCustomTransformationIT which we are working on.

Example: https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/13307357883/job/37161942486
https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/workflows/spanner-pr.yml?query=branch%3Amain

[Abacn]

Thanks for the comment.

NoClassDef found should happen everytime the same tests are run. I don't understand why this is flaky.

NoClassDef could also happen when the static initializer fails, in this case it is in com.google.spanner.v1.Session$LabelsDefaultEntryHolder, there is a static block

static {
      defaultEntry = MapEntry.newDefaultInstance(...);
    }

likely failed

[damccorm]

Looks like this has been fixed. There are other issues causing problems now, but I think they're being addressed

[Deep1998]

I have noticed multiple occurences of this recently.
These are documented in b/400992122