History Engine: Transfer queue getting stuck due to invalid error from matching

[samarabbas]

I'm seeing history engine crash due to stuck task in transfer queue. Seems like matching engine is wrapping EventAlreadyStartedError within InternalServiceError causing transfer queue to keep on retrying a non transient error 'EventAlreadyStartedError'.

time="2017-08-12T02:11:55Z" level=error msg="helpers.go:48 Processor failed to process transfer task: 24119292, type: 0. Error: InternalServiceError({Message:EventAlreadyStartedError({Message:Decision task already started.})})" Service=cadence-history deployment-name="prod_dca1a" shard-id=540 task-id=24119292 task-type=0 wf-component=transfer-queue-processor wf-error="InternalServiceError({Message:EventAlreadyStartedError({Message:Decision task already started.})})" wf-event-id=2105
time="2017-08-12T02:12:34Z" level=error msg="helpers.go:48 Processor failed to process transfer task: 24119292, type: 0. Error: InternalServiceError({Message:EventAlreadyStartedError({Message:Decision task already started.})})" Service=cadence-history deployment-name="prod_dca1a" shard-id=540 task-id=24119292 task-type=0 wf-component=transfer-queue-processor wf-error="InternalServiceError({Message:EventAlreadyStartedError({Message:Decision task already started.})})" wf-event-id=2105
time="2017-08-12T02:13:13Z" level=error msg="helpers.go:48 Processor failed to process transfer task: 24119292, type: 0. Error: InternalServiceError({Message:EventAlreadyStartedError({Message:Decision task already started.})})" Service=cadence-history deployment-name="prod_dca1a" shard-id=540 task-id=24119292 task-type=0 wf-component=transfer-queue-processor wf-error="InternalServiceError({Message:EventAlreadyStartedError({Message:Decision task already started.})})" wf-event-id=2105
time="2017-08-12T02:13:53Z" level=error msg="helpers.go:48 Processor failed to process transfer task: 24119292, type: 0. Error: InternalServiceError({Message:EventAlreadyStartedError({Message:Decision task already started.})})" Service=cadence-history deployment-name="prod_dca1a" shard-id=540 task-id=24119292 task-type=0 wf-component=transfer-queue-processor wf-error="InternalServiceError({Message:EventAlreadyStartedError({Message:Decision task already started.})})" wf-event-id=2105
time="2017-08-12T02:14:33Z" level=error msg="helpers.go:48 Processor failed to process transfer task: 24119292, type: 0. Error: InternalServiceError({Message:EventAlreadyStartedError({Message:Decision task already started.})})" Service=cadence-history deployment-name="prod_dca1a" shard-id=540 task-id=24119292 task-type=0 wf-component=transfer-queue-processor wf-error="InternalServiceError({Message:EventAlreadyStartedError({Message:Decision task already started.})})" wf-event-id=2105
time="2017-08-12T02:15:12Z" level=error msg="helpers.go:48 Processor failed to process transfer task: 24119292, type: 0. Error: InternalServiceError({Message:EventAlreadyStartedError({Message:Decision task already started.})})" Service=cadence-history deployment-name="prod_dca1a" shard-id=540 task-id=24119292 task-type=0 wf-component=transfer-queue-processor wf-error="InternalServiceError({Message:EventAlreadyStartedError({Message:Decision task already started.})})" wf-event-id=2105
time="2017-08-12T02:15:52Z" level=error msg="helpers.go:48 Processor failed to process transfer task: 24119292, type: 0. Error: InternalServiceError({Message:EventAlreadyStartedError({Message:Decision task already started.})})" Service=cadence-history deployment-name="prod_dca1a" shard-id=540 task-id=24119292 task-type=0 wf-component=transfer-queue-processor wf-error="InternalServiceError({Message:EventAlreadyStartedError({Message:Decision task already started.})})" wf-event-id=2105
time="2017-08-12T02:16:31Z" level=error msg="helpers.go:48 Processor failed to process transfer task: 24119292, type: 0. Error: InternalServiceError({Message:EventAlreadyStartedError({Message:Decision task already started.})})" Service=cadence-history deployment-name="prod_dca1a" shard-id=540 task-id=24119292 task-type=0 wf-component=transfer-queue-processor wf-error="InternalServiceError({Message:EventAlreadyStartedError({Message:Decision task already started.})})" wf-event-id=2105
time="2017-08-12T02:17:11Z" level=error msg="helpers.go:48 Processor failed to process transfer task: 24119292, type: 0. Error: InternalServiceError({Message:EventAlreadyStartedError({Message:Decision task already started.})})" Service=cadence-history deployment-name="prod_dca1a" shard-id=540 task-id=24119292 task-type=0 wf-component=transfer-queue-processor wf-error="InternalServiceError({Message:EventAlreadyStartedError({Message:Decision task already started.})})" wf-event-id=2105
time="2017-08-12T02:17:51Z" level=error msg="helpers.go:48 Processor failed to process transfer task: 24119292, type: 0. Error: InternalServiceError({Message:EventAlreadyStartedError({Message:Decision task already started.})})" Service=cadence-history deployment-name="prod_dca1a" shard-id=540 task-id=24119292 task-type=0 wf-component=transfer-queue-processor wf-error="InternalServiceError({Message:EventAlreadyStartedError({Message:Decision task already started.})})" wf-event-id=2105
time="2017-08-12T02:18:03Z" level=info msg="discovery.go:64 ringpop requests for peers via wrapped discovery provider (try #1)"
time="2017-08-12T02:18:31Z" level=error msg="helpers.go:48 Processor failed to process transfer task: 24119292, type: 0. Error: InternalServiceError({Message:EventAlreadyStartedError({Message:Decision task already started.})})" Service=cadence-history deployment-name="prod_dca1a" shard-id=540 task-id=24119292 task-type=0 wf-component=transfer-queue-processor wf-error="InternalServiceError({Message:EventAlreadyStartedError({Message:Decision task already started.})})" wf-event-id=2105
time="2017-08-12T02:18:41Z" level=fatal msg="helpers.go:64 Retry count exceeded for transfer taskID: 24119292. Error: " Service=cadence-history deployment-name="prod_dca1a" shard-id=540 wf-component=transfer-queue-processor wf-error= wf-event-id=9001