Should I call flush before closing when using ZstdOutputStream?

[ConeyLiu]

Hopefully, someone could help on whether we should call flush before closing a ZstdOuputStream? Because this could produce different behavior.

    // compress without flush
    byte[] bytes1 = null;
    ByteArrayOutputStream outputStream1 = new ByteArrayOutputStream();
    try(ByteArrayOutputStream out = outputStream1;
        ZstdOutputStream zstd = new ZstdOutputStream(out, 3)) {
      zstd.setWorkers(0);

      zstd.write(0);
    }

    bytes1 = outputStream1.toByteArray();
    
    // compress with flush
    byte[] bytes2 = null;
    ByteArrayOutputStream outputStream2 = new ByteArrayOutputStream();
    try(ByteArrayOutputStream out = outputStream2;
        ZstdOutputStream zstd = new ZstdOutputStream(out, 3)) {
      zstd.setWorkers(0);

      zstd.write(0);

      zstd.flush(); // flush before close
    }

    bytes2 = outputStream2.toByteArray();

    Arrays.equals(bytes1, bytes2);

Call flush could produce extra 3 bytes:

[luben]

Closing the stream makes sure all the data is flushed downstream, so you don't need to flush it before that.

You need to call flush only if you have nor finished writing to the stream but want to make sure that all written data is propagated downstream.

[ConeyLiu]

@luben, thank you for your response. The reason for this is that Apache Parquet added a flush before closing the ZstdOutputStream (as seen in this pull request: apache/parquet-java#1309). We noticed that the file sizes changed after upgrading the Parquet version.

I have another question: Is it acceptable to have some extra bytes in the output? These should be frame metadata, correct?

[luben]

Yes, it will not hurt.

[ConeyLiu]

thanks @luben