[SUPPORT]Duplicate data in MOR table Hudi

[xiagupqin]

Tips before filing an issue

• Have you gone through our FAQs?

• Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

• If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced
We run sparkstructed streaming application where we read kafka stream process the data and stores in Hudi.
We started seeing duplicates in our hudi dataset
Below are our Hudi configs

DataSourceWriteOptions.TABLE_TYPE.key() -> DataSourceWriteOptions.MOR_TABLE_TYPE_OPT_VAL, DataSourceWriteOptions.RECORDKEY_FIELD.key() -> "id", DataSourceWriteOptions.PARTITIONPATH_FIELD.key() -> "dt", DataSourceWriteOptions.PRECOMBINE_FIELD.key() -> "ts", HoodieCompactionConfig.INLINE_COMPACT.key() -> "true",
We only use upsert in our code
dataframe.write.format("org.apache.hudi") .option(HoodieWriteConfig.TABLE_NAME, hudiTableName) .option(DataSourceWriteOptions.OPERATION_OPT_KEY, DataSourceWriteOptions.UPSERT_OPERATION_OPT_VAL) .options(hudiOptions).mode(SaveMode.Append) .save(s3Location)
A clear and concise description of the problem.

To Reproduce

Steps to reproduce the behavior:

Expected behavior

A clear and concise description of what you expected to happen.

Environment Description

• Hudi version :0.12.0

• Spark version :3.3.0

• Hive version :

• Hadoop version :

• Storage (HDFS/S3/GCS..) :s3

• Running on Docker? (yes/no) :no

Additional context

Add any other context about the problem here.

Stacktrace

Add the stacktrace of the error.

[xiagupqin]

@nsivabalan this is dupilcate records
query-hive-10092.csv

[xiagupqin]

please help me look at this

[xiagupqin]

@nsivabalan

[nsivabalan]

There could be two reasons:

1. you may need to set hoodie.datasource.write.streaming.ignore.failed.batch = false. (default value is false).
2. we found a corner case w/ metadata table and timeline server that may have resulted in duplicates. can you disable metadata table and give it a try.

[xiagupqin]

hello nsivabalan Thank you very much for your advice can you tell me how to disable metadata table

[xiagupqin]

@nsivabalan

[xiagupqin]

@nsivabalan set hoodie.datasource.write.streaming.ignore.failed.batch = false i dont found this config can you tell me

[ad1happy2go]

@xiagupqin you can disable metadata table by setting this property as false - hoodie.metadata.enable

[xiagupqin]

@xiagupqin you can disable metadata table by setting this property as false - hoodie.metadata.enable

taanks i try it

[codope]

@xiagupqin Can you confirm that there are no duplicates after disabling metadata? Most likely, #8079 fixes this issue too.