[C#] Expose method to compare 2 Tables

[andrewshandy]

Describe the enhancement requested

Motivation

I am working on integration tests where I am testing that a service returns an arrow table that equals to an expected arrow table.

Feature

Expose a method that compares 2 Tables

Considerations

I see that there's an ArrowReaderVerifier class that does have some similar functionality here to compare record batches -

arrow/csharp/test/Apache.Arrow.Tests/ArrowReaderVerifier.cs

Lines 52 to 65 in 1521148

	public static void CompareBatches(RecordBatch expectedBatch, RecordBatch actualBatch, bool strictCompare = true)
	{
	SchemaComparer.Compare(expectedBatch.Schema, actualBatch.Schema);
	Assert.Equal(expectedBatch.Length, actualBatch.Length);
	Assert.Equal(expectedBatch.ColumnCount, actualBatch.ColumnCount);
	for (int i = 0; i < expectedBatch.ColumnCount; i++)
	{
	IArrowArray expectedArray = expectedBatch.Arrays.ElementAt(i);
	IArrowArray actualArray = actualBatch.Arrays.ElementAt(i);
	CompareArrays(expectedArray, actualArray, strictCompare);
	}
	}

But as far as I understand Apache.Arrow.Tests namespace is not publicly available. Is that correct? Is it possible to expose some of the methods from it to be public?

Component(s)

C#

[amoeba]

Hi @andrewshandy, I think a PR adding this to the public API would be welcome.

From a code search and seeing its presence on https://github.com/apache/arrow/blob/main/csharp/README.md#not-implemented, it looks like comparison isn't (publicly) supported for Table as well as other types. If you do take this on, it might be good to (1) file a tracking issue for the overall feature and use sub-issues/PRs for the pieces and (2) review the semantics of similar operations in C++, Python, etc. and figure out if C# can match or if it needs to deviate for some reason.

cc @CurtHagenlocher @adamreeve in case they want to add any context

[adamreeve]

But as far as I understand Apache.Arrow.Tests namespace is not publicly available. Is that correct?

Yes that's correct, to expose this functionality publicly it would need to be moved into the Apache.Arrow project or a new project.

I don't imagine we'd want to expose the ArrowReaderVerifier or ArrayComparer classes directly, as once something is in the public API it's hard to change later, but adding new methods for comparing arrays, record batches and tables to the public API would be useful.

Comparing tables is a little bit more complicated than the existing functionality for comparing record batches, because two tables that contain the same data but are chunked differently should be considered equal. If you expect chunking to be the same, you might want to start with adding a way to compare record batches and compare tables by comparing all of their record batches.

[andrewshandy]

Great! Thank you both for the info! I will start working on this then, I will probably start from comparing record batches and see what else would be needed to implement it 🤗

[CurtHagenlocher]

Superseded by apache/arrow-dotnet#183.