feat(parquet/examples): enhance library examples

[milden6]

Rationale for this change

Enhancing the library examples

What changes are included in this PR?

2 new examples on how to use the arrow-go library:

• Writing parquet file
• Reading parquet file

Are these changes tested?

Not applicable

Are there any user-facing changes?

No, just enhancing examples

[zeroshade]

Instead of standalone examples like this, can we use Go testable examples that will show up in the docs and be tested to ensure they don't get out of date?

[milden6]

Instead of standalone examples like this, can we use Go testable examples that will show up in the docs and be tested to ensure they don't get out of date?

I did it like here, what the difference?
https://github.com/apache/arrow-go/tree/main/arrow/examples/table_creation

[zeroshade]

My preference would be to follow this pattern

Here's a good explanation of it: https://go.dev/blog/examples

[milden6]

okay, it should be in the path: arrow-go/parquet/example_writing_parquet_test.go?

[zeroshade]

That path works great

[zeroshade]

Thanks for this! It looks great, just a few nitpicks

[zeroshade]

Should we elaborate in the comment about the benefit to storing the Arrow schema in the metadata?

[milden6]

Yes, you are right, what about this comment?

// WithStoreSchema embeds the original Arrow schema into the Parquet file metadata,
// allowing it to be accurately restored when reading. This ensures correct handling
// of advanced types like dictionaries and preserves full type fidelity across write/read

[zeroshade]

that looks good! Perhaps also mention that some of the other Parquet libraries also support this so it helps ensure cross-language type consistency too during reading.

[zeroshade]

do we need to do the FieldIndices calls when we know what the schema is and the indices are since we created the schema? There might be benefits to keeping this as simple as possible

[milden6]

Yep, sure, I'll specify the indices explicitly for simplicity:

	intFieldBuilder := recordBuilder.Field(0).(*array.Int32Builder)
	stringFieldBuilder := recordBuilder.Field(1).(*array.StringBuilder)
	listFieldBuilder := recordBuilder.Field(2).(*array.ListBuilder)

[zeroshade]

log.Fatalf?

[milden6]

Thanks, I will fix it

[zeroshade]

Suggested change

	// Get the current record from the reader
	// Get a record reader from the file to iterate over

[milden6]

Thanks

[zeroshade]

We only need to call retain if we need to utilize the record beyond and outside of this loop. Since we only interact with the record in the loop and aren't storing it for use beyond the loop we don't need to call Retain and Release. Might be better to simplify this

[milden6]

Okay, thanks for clarification

[zeroshade]

Do we need to grab each column separately like this instead of just calling fmt.Println(record) ?

[milden6]

Are you sure? If I write it like this:

for recordReader.Next() {
	// Create a record
	record := recordReader.Record()
	fmt.Println(record)
}

The output will be like this:

record:
  schema:
  fields: 3
    - intField: type=int32
          metadata: ["PARQUET:field_id": "-1"]
    - stringField: type=utf8
             metadata: ["PARQUET:field_id": "-1"]
    - listField: type=list<element: float32, nullable>
           metadata: ["PARQUET:field_id": "-1"]
  rows: 2
  col[0][intField]: [38 13]
  col[1][stringField]: ["val1" "val2"]
  col[2][listField]: [[1 2 4 8] [1 2 4 8]]

record:
  schema:
  fields: 3
    - intField: type=int32
          metadata: ["PARQUET:field_id": "-1"]
    - stringField: type=utf8
             metadata: ["PARQUET:field_id": "-1"]
    - listField: type=list<element: float32, nullable>
           metadata: ["PARQUET:field_id": "-1"]
  rows: 2
  col[0][intField]: [53 93]
  col[1][stringField]: ["val3" "val4"]
  col[2][listField]: [[1 2 4 8] [1 2 4 8]]

record:
  schema:
  fields: 3
    - intField: type=int32
          metadata: ["PARQUET:field_id": "-1"]
    - stringField: type=utf8
             metadata: ["PARQUET:field_id": "-1"]
    - listField: type=list<element: float32, nullable>
           metadata: ["PARQUET:field_id": "-1"]
  rows: 1
  col[0][intField]: [66]
  col[1][stringField]: ["val5"]
  col[2][listField]: [[1 2 4 8]]

Instead of:

38  val1  [1 2 4 8]
13  val2  [1 2 4 8]
53  val3  [1 2 4 8]
93  val4  [1 2 4 8]
66  val5  [1 2 4 8]

Or did I misunderstand you?

[zeroshade]

Nope you didn't misunderstand, perhaps change the batch size to be 3 instead of 2 so that it only outputs two records?

[milden6]

I think it’s best to leave it as is, since it demonstrates how to access cell values for each row. Working with records isn’t always obvious for beginners.

Like here:

arrow-go/parquet/file/file_reader_test.go

Lines 687 to 696 in 084371e

	for rr.Next() {
	rec := rr.Record()
	for i := 0; i < int(rec.NumRows()); i++ {
	col := rec.Column(0).(*array.Int64)
	val := col.Value(i)
	require.Equal(t, val, int64(totalRows+i))
	}
	totalRows += int(rec.NumRows())
	}

What do you think?

[zeroshade]

fair enough, we can leave it as is.

[zeroshade]

Thanks so much for this. It will significantly help people with interacting with this library!