Describe the enhancement requested
Curent gzip decompress is calling 'infalte' until getting 'Z_STREAM_END ' or error is returned, but zccording to gzip (zlib) documentation, this might be not enough:
" inflate() will not automatically decode concatenated gzip members. inflate() will return Z_STREAM_END at the end of the gzip member. The state would need to be reset to continue decoding a subsequent gzip member. This must be done if there is more data after a gzip member, in order for the decompression to be compliant with the gzip standard (RFC 1952)." (https://www.zlib.net/manual.html)
This PR is for supporting reading parquet files that contains more than 1 gzip member. (example file attahced)
concatenated_gzip_members.zip
Component(s)
C++, Parquet
Describe the enhancement requested
Curent gzip decompress is calling 'infalte' until getting 'Z_STREAM_END ' or error is returned, but zccording to gzip (zlib) documentation, this might be not enough:
" inflate() will not automatically decode concatenated gzip members. inflate() will return Z_STREAM_END at the end of the gzip member. The state would need to be reset to continue decoding a subsequent gzip member. This must be done if there is more data after a gzip member, in order for the decompression to be compliant with the gzip standard (RFC 1952)." (https://www.zlib.net/manual.html)
This PR is for supporting reading parquet files that contains more than 1 gzip member. (example file attahced)
concatenated_gzip_members.zip
Component(s)
C++, Parquet