Handle Avro fixed field type in AvroInputCodec#6603
Handle Avro fixed field type in AvroInputCodec#6603p1ck wants to merge 2 commits intoopensearch-project:mainfrom
Conversation
p1ck
requested review from
KarstenSchnitter,
dinujoh,
dlvenable,
engechas,
graytaylor0,
kkondaka,
oeyh,
san81,
sb2k16 and
srikanthjg
as code owners
Added a case in convertRecordToMap to handle GenericData.Fixed. Previously, the use of the fixed field type in Avro schemas caused an exception. Now it is converted to a byte array and added to the map as expected. Signed-off-by: Jack Pickett <jackpick@gmail.com>
|
Would love to see this fixed! |
| value = new String(utf8Bytes, "UTF-8"); | ||
| } | ||
|
|
||
| else if(value instanceof GenericData.Fixed){ |
There was a problem hiding this comment.
@p1ck , Thank you for the contribution. Please add a unit tests that fails without this change but works with it. This way we know it works and it won't fail again if somebody changes the code.
|
This is somewhat related to #4096 "Create a model for binary data". Even when I am able to read the bytes field, there are no functions or processors within data prepper to do anything with the contents except pass them on as a base64 encoded string. e.g. I read an Avro file with an IPv4 address stored as 4 bytes. I would like to convert it to dotted decimal notation to ingest into an opensearch |
|
@p1ck , Are you able to add a unit test that demonstrates the failure that the code change fixes? We want to be sure that the behavior doesn't regress by having a unit test. |
3 participants
Added a case in AvroInputCodec convertRecordToMap to handle GenericData.Fixed which comes from Avro "fixed" field type.
Description
Previously, the use of the "fixed" field type in Avro schemas caused an exception on read. Here it is converted to a byte array and added to the map as expected.
e.g.
An Avro file with the following valid schema results in an exception
{ "type": "record", "name": "TestRecord", "fields": [ { "name": "fixedField", "type": { "type": "fixed", "name": "FixedType", "size": 16 } } ] }Issues Resolved
Resolves #6602
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.