-
Notifications
You must be signed in to change notification settings - Fork 1.5k
HadoopPositionOutputStream.close() is not idempotent (i.e. calling it more than once may fail) #3205
Copy link
Copy link
Closed
Labels
Description
Describe the bug, including details regarding any error messages, version, and platform.
Describe the bug, including details regarding any error messages, version, and platform.
If HadoopPositionOutputStream.close() is called more than once, an exception may be thrown. Since HadoopPositionOutputStream implements Closable, its close method should have no effect if already closed.
Example exception:
Caused by: org.apache.parquet.util.AutoCloseables$ParquetCloseResourceException: Unable to close resource
at org.apache.parquet.util.AutoCloseables.uncheckedClose(AutoCloseables.java:85) ~[hive-exec-4.0.0-20-IBM.jar:4.0.0]
at org.apache.parquet.util.AutoCloseables.uncheckedClose(AutoCloseables.java:94) ~[hive-exec-4.0.0-20-IBM.jar:4.0.0]
at org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:144) ~[hive-exec-4.0.0-20-IBM.jar:4.0.0]
at org.apache.parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:223) ~[hive-exec-4.0.0-20-IBM.jar:4.0.0]
at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.close(ParquetRecordWriterWrapper.java:137) ~[hive-exec-4.0.0-20-IBM.jar:4.0.0]
...
Caused by: java.nio.channels.ClosedChannelException
at org.apache.hadoop.hdfs.ExceptionLastSeen.throwException4Close(ExceptionLastSeen.java:73) ~[hadoop-hdfs-client-3.4.1.jar:?]
at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:158) ~[hadoop-hdfs-client-3.4.1.jar:?]
at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:639) ~[hadoop-hdfs-client-3.4.1.jar:?]
at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:594) ~[hadoop-hdfs-client-3.4.1.jar:?]
at org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:136) ~[hadoop-common-3.4.1.jar:?]
at org.apache.parquet.hadoop.util.HadoopPositionOutputStream.close(HadoopPositionOutputStream.java:65) ~[hive-exec-4.0.0-20-IBM.jar:4.0.0]
at org.apache.parquet.hadoop.ParquetFileWriter.close(ParquetFileWriter.java:1663) ~[hive-exec-4.0.0-20-IBM.jar:4.0.0]
at org.apache.parquet.util.AutoCloseables.close(AutoCloseables.java:49) ~[hive-exec-4.0.0-20-IBM.jar:4.0.0]
at org.apache.parquet.util.AutoCloseables.uncheckedClose(AutoCloseables.java:83) ~[hive-exec-4.0.0-20-IBM.jar:4.0.0]
at org.apache.parquet.util.AutoCloseables.uncheckedClose(AutoCloseables.java:94) ~[hive-exec-4.0.0-20-IBM.jar:4.0.0]
at org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:144) ~[hive-exec-4.0.0-20-IBM.jar:4.0.0]
at org.apache.parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:223) ~[hive-exec-4.0.0-20-IBM.jar:4.0.0]
at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.close(ParquetRecordWriterWrapper.java:137) ~[hive-exec-4.0.0-20-IBM.jar:4.0.0]
Component(s)
parquet-hadoop
Component(s)
No response
Reactions are currently unavailable