[Bug]: internal/msgstream unittest panics randomly

[congqixia]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:535441d
- Deployment mode(standalone or cluster): both
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

msgstream package unit test panics randomly

Expected Behavior

Unit tests passes

Steps To Reproduce

Run unit test in msgstream package multiple times

Anything else?

--- FAIL: TestStream_PulsarMsgStream_InsertRepackFunc (0.17s)
panic: topic persistent://public/default/dGOcPubs, subscription FZhxFNgl: server error: MetadataError: Consumer not found [recovered]
	panic: topic persistent://public/default/dGOcPubs, subscription FZhxFNgl: server error: MetadataError: Consumer not found

goroutine 546 [running]:
testing.tRunner.func1.2(0x16cfec0, 0xc000895db0)
	/usr/local/go/src/testing/testing.go:1143 +0x332
testing.tRunner.func1(0xc00075db00)
	/usr/local/go/src/testing/testing.go:1146 +0x4b6
panic(0x16cfec0, 0xc000895db0)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/milvus-io/milvus/internal/util/mqclient.(*PulsarConsumer).Close.func1()
	/home/silverxia/workspace/milvus/internal/util/mqclient/pulsar_consumer.go:113 +0xc5
sync.(*Once).doSlow(0xc0008a39c8, 0xc0000d9b40)
	/usr/local/go/src/sync/once.go:68 +0xec
sync.(*Once).Do(...)
	/usr/local/go/src/sync/once.go:59
github.com/milvus-io/milvus/internal/util/mqclient.(*PulsarConsumer).Close(0xc0008a3980)
	/home/silverxia/workspace/milvus/internal/util/mqclient/pulsar_consumer.go:108 +0x65
github.com/milvus-io/milvus/internal/msgstream.(*mqMsgStream).Close(0xc000281d40)
	/home/silverxia/workspace/milvus/internal/msgstream/mq_msgstream.go:226 +0x173
github.com/milvus-io/milvus/internal/msgstream.TestStream_PulsarMsgStream_InsertRepackFunc(0xc00075db00)
	/home/silverxia/workspace/milvus/internal/msgstream/mq_msgstream_test.go:557 +0x858
testing.tRunner(0xc00075db00, 0x18e5ee8)
	/usr/local/go/src/testing/testing.go:1193 +0xef
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[xiaofan-luan]

hit the same issue here~

[xiaofan-luan]

what is the main reason? not sure if retry would help

[congqixia]

what is the main reason? not sure if retry would help

@xiaofan-luan hate the retry patch as well.
Since the error is returned from pulsar (error type was server error), a wild guess was that pulsar meta data was not synced in a very short period of time.

I tested with a shell script to run UT until it fails, with retry patch it keeps working for about half an hour until I manually stop the script.

[xiaofan-luan]

ok, we will mark it as a temp solution.