panic: runtime error: index out of range [7] with length 1

[jassentang]

Bigcache runs for about 1 day, crash！
How to solve this problem?
My bigcache configuration is:

master_config := bigcache.DefaultConfig(300 * time.Second)
master_config.MaxEntriesInWindow = 10
master_config.HardMaxCacheSize = 2048
master_config.OnRemoveWithReason = func(key string, entry []byte, reason bigcache.RemoveReason) {
log.Info().Msgf("master bigcache OnRemove key[%s]entry[%s]reason[%d]Expired(1)NoSpace(2)Deleted(3)", key, entry, reason)
}
var initErr error
g_bigcache, initErr = bigcache.NewBigCache(master_config)
if initErr != nil {
log.Error().Err(initErr).Msgf("NewBigCache g_bigcache initErr!")
}

I used the Set, Get, Iterator(), SetNext() functions of bigcache.
The log of crash is as follows:

2020/05/29 17:38:31 Allocated new queue in 84.16µs; Capacity: 2560000 6
2020/05/29 17:38:31 Allocated new queue in 130.649µs; Capacity: 2560000 6
2020/05/29 17:38:32 Allocated new queue in 197.943µs; Capacity: 4194304 9
2020/05/29 17:38:33 Allocated new queue in 130.092µs; Capacity: 4194304 9
panic: runtime error: index out of range [7] with length 1

goroutine 188 [running]:
panic(0x127da40, 0xc01a3417a0)
        /usr/local/go/src/runtime/panic.go:1064 +0x46d fp=0xc003264d88 sp=0xc003264cd0 pc=0x4461cd
runtime.goPanicIndex(0x7, 0x1)
        /usr/local/go/src/runtime/panic.go:88 +0xa3 fp=0xc003264dd0 sp=0xc003264d88 pc=0x443f73
encoding/binary.littleEndian.Uint64(...)
        /usr/local/go/src/encoding/binary/binary.go:77
github.com/allegro/bigcache/v2.readTimestampFromEntry(...)
        /root/go/pkg/mod/github.com/widelee/bigcache/v2@v2.2.1/encoding.go:43
github.com/allegro/bigcache/v2.(*cacheShard).onEvict(0xc0016777a0, 0xc00e23a860, 0x1, 0x3f37a0, 0x5ed0d923, 0xc003264e40, 0x0)
        /root/go/pkg/mod/github.com/widelee/bigcache/v2@v2.2.1/shard.go:247 +0x7d fp=0xc003264df8 sp=0xc003264dd0 pc=0xe754fd
github.com/allegro/bigcache/v2.(*cacheShard).cleanUp(0xc0016777a0, 0x5ed0d923)
        /root/go/pkg/mod/github.com/widelee/bigcache/v2@v2.2.1/shard.go:260 +0xb9 fp=0xc003264e60 sp=0xc003264df8 pc=0xe755c9
github.com/allegro/bigcache/v2.(*BigCache).cleanUp(0xc001010820, 0x5ed0d923)
        /root/go/pkg/mod/github.com/widelee/bigcache/v2@v2.2.1/bigcache.go:215 +0x53 fp=0xc003264e98 sp=0xc003264e60 pc=0xe71b73
github.com/allegro/bigcache/v2.newBigCache.func1(0x400, 0x45d964b800, 0x3b9aca00, 0xa, 0x1f4, 0x100, 0x1528580, 0x5e7db28, 0x1000, 0x0, ...)
        /root/go/pkg/mod/github.com/widelee/bigcache/v2@v2.2.1/bigcache.go:92 +0x9b fp=0xc003264f60 sp=0xc003264e98 pc=0xe7663b
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc003264f68 sp=0xc003264f60 pc=0x479481
created by github.com/allegro/bigcache/v2.newBigCache
        /root/go/pkg/mod/github.com/widelee/bigcache/v2@v2.2.1/bigcache.go:86 +0x46f

[cristaloleg]

(Oops. closed by missclick)

Can you show what you're passing to Set Get methods?

[jassentang]

Hi @cristaloleg , this is my code for using Set and Get

func Get_value_by_key(key string) ([]byte, error) {
	result_str, err := g_bigcache.Get(key)
	if nil != err {
		log.Error().Err(err).Msgf("g_bigcache.Get key faield!key[%s]\n", key)
		return result_str, err
	}
	return result_str, err
}

func Set_key_value(key string, value []byte) error {
	log.Info().Msgf("Set_key_value key[%s] value[%s]", key, value)
	err := g_bigcache.Set(key, value)
	if nil != err {
		log.Error().Err(err).Msgf("g_bigcache.Set key[%s] value[%s] faield!\n", key, value)
		return err
	}
	return nil
}

Below is the code using Iterator and SetNext，Synchronize bigcache data to redis every 5s

func init() {
	// Run SyncBigcacheWithRedis once every 5 seconds
	f := func() {
		SyncBigcacheWithRedis()
	}

	spec := "*/5 * * * * ?"
	c := cron.New(cron.WithSeconds())
	_, e := c.AddFunc(spec, f)
	if e != nil {
		log.Fatal().Err(e).Msg("start cron task fail")
	}

	c.Start()
}

// Synchronize bigcache data to redis
func SyncBigcacheWithRedis() {
	log.Info().Msgf("come into SyncBigcacheWithRedis")
	defer log.Info().Msgf("come out SyncBigcacheWithRedis")

	sync_now_time := time.Now().Unix()
	count := 0
	lStartTime := time.Now()
	it := g_bigcache.Iterator()
	for ; ; {
		if it.SetNext() {
			count++
			entry_info, err := it.Value()
			if nil != err {
				log.Error().Err(err).Msgf("sync_bigcache traverse failed! break")
				break
			}
			// sync entry_info.Key() and entry_info.Value() to redis
			setToRedis(entry_info.Key(), entry_info.Value())
		} else {
			diff := time.Now().Sub(lStartTime).Milliseconds()
			log.Info().Msgf("sync done!! close bigcache and break, count[%d] cost[%d]ms", count, diff)
			break
		}
	}
}

[PaulLiang1]

Have the same issue on production

| 2020-08-18 02:32:06.870 | github.com/allegro/bigcache/v2.(*cacheShard).removeOldestEntry(0xc000375680, 0x1, 0x0, 0xa0f04b)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| 2020-08-18 02:32:06.870 |  /go/pkg/mod/github.com/allegro/bigcache/v2@v2.2.3/shard.go:296 +0x170                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| 2020-08-18 02:32:06.870 | github.com/allegro/bigcache/v2.(*cacheShard).onEvict(0xc000375680, 0xc00001ca20, 0x9, 0x1520, 0x5f3b3da6, 0xc000639e20, 0x0)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| 2020-08-18 02:32:06.870 |  /go/pkg/mod/github.com/allegro/bigcache/v2@v2.2.3/shard.go:249 +0x55                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| 2020-08-18 02:32:06.870 | github.com/allegro/bigcache/v2.(*cacheShard).cleanUp(0xc000375680, 0x5f3b3da6)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| 2020-08-18 02:32:06.870 |  /go/pkg/mod/github.com/allegro/bigcache/v2@v2.2.3/shard.go:260 +0xb9                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| 2020-08-18 02:32:06.870 | github.com/allegro/bigcache/v2.(*BigCache).cleanUp(0xc00034cb60, 0x5f3b3da6)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| 2020-08-18 02:32:06.870 |  /go/pkg/mod/github.com/allegro/bigcache/v2@v2.2.3/bigcache.go:215 +0x53                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| 2020-08-18 02:32:06.870 | github.com/allegro/bigcache/v2.newBigCache.func1(0x8, 0x8bb2c97000, 0xdf8475800, 0x0, 0x64, 0x0, 0x1387660, 0x1ab9ee8, 0x40, 0x0, ...)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| 2020-08-18 02:32:06.870 |  /go/pkg/mod/github.com/allegro/bigcache/v2@v2.2.3/bigcache.go:92 +0x9e                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| 2020-08-18 02:32:06.870 | created by github.com/allegro/bigcache/v2.newBigCache                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| 2020-08-18 02:32:06.870 |  /go/pkg/mod/github.com/allegro/bigcache/v2@v2.2.3/bigcache.go:86 +0x46f                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| 2020-08-18 02:32:06.869 | panic: runtime error: index out of range [7] with length 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| 2020-08-18 02:32:06.869 | goroutine 77 [running]:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| 2020-08-18 02:32:06.869 | encoding/binary.littleEndian.Uint64(...)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| 2020-08-18 02:32:06.869 |  /usr/local/go/src/encoding/binary/binary.go:76                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| 2020-08-18 02:32:06.869 | github.com/allegro/bigcache/v2.readHashFromEntry(...)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| 2020-08-18 02:32:06.869 |  /go/pkg/mod/github.com/allegro/bigcache/v2@v2.2.3/encoding.go:57                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |

[PaulLiang1]

also it appears after big cache panics, the cache get method will hang without throwing errors.
we observed this after the panic above is thrown and recovered, then subsequent cache operation hanged and caused our application to timed out.

it's better not to recovery the panic and let the app crash / or downgrade the bigcache version

[fpessolano]

I am experiencing the same issue on a deployed product.
Any solution or work around?

[xpp1021]

@jassentang Is it resolved ? I also meet this problem

[janisz]

Unfortunately there is not known way to reproduce it so it's impossible to fix :(

[fpessolano]

@jassentang Is it resolved ? I also meet this problem

I re-thought why i was using BigCache and managed to remove it from my design. Only solution I found so far.