Segmentation fault, panic & memory spikes since v1.80.0

[hxyannay]

What version of gRPC are you using?

v1.80.0

What version of Go are you using (go version)?

v1.26.2

What operating system (Linux, Windows, …) and version?

Linux

What did you do?

Since upgrading to gRPC 1.80.0 during high loads we are getting various crashes, the main ones are: segmentation fault, panic on makeslice: len out of range & memory spikes.
Attaching crash logs of segmentation fault and panic:

unexpected fault address 0x36ffffffda
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x36ffffffda pc=0x495aed]

goroutine 1232 gp=0x39f63925eb40 m=5 mp=0x39f63517f008 [running]:
runtime.throw({0x1aa20d7?, 0x1?})
        /usr/local/go/src/runtime/panic.go:1229 +0x48 fp=0x39f63a0c1078 sp=0x39f63a0c1048 pc=0x48cd48
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:945 +0x285 fp=0x39f63a0c10d8 sp=0x39f63a0c1078 pc=0x48f4c5
runtime.memmove()
        /usr/local/go/src/runtime/memmove_amd64.s:414 +0x4ad fp=0x39f63a0c10e0 sp=0x39f63a0c10d8 pc=0x495aed
google.golang.org/protobuf/encoding/protowire.AppendString(...)
        /gomod-cache/google.golang.org/protobuf@v1.36.11/encoding/protowire/wire.go:479
google.golang.org/protobuf/internal/impl.appendStringPtrValidateUTF8({0x39f647e4e000?, 0x2499?, 0xa00005000?}, {0xf?}, 0x39f63d2fa630?, {0x0?})
        /gomod-cache/google.golang.org/protobuf@v1.36.11/internal/impl/codec_gen.go:5173 +0x91 fp=0x39f63a0c1148 sp=0x39f63a0c10e0 pc=0xe9c3b1
google.golang.org/protobuf/internal/impl.(*MessageInfo).marshalAppendPointer(0x39f6351353b8, {0x39f647e4e000?, 0x39f63cf7c820?, 0xa00002487?}, {0xeb22af?}, {0xb8?})
        /gomod-cache/google.golang.org/protobuf@v1.36.11/internal/impl/encode.go:205 +0x21c fp=0x39f63a0c11e8 sp=0x39f63a0c1148 pc=0xeb29bc
google.golang.org/protobuf/internal/impl.appendOpaqueMessageSlice({0x39f647e4e000?, 0x39f636f8db30?, 0x39f63a0c1300?}, {0xeb2545?}, 0x39f63a230cf0, {0xf0?})
        /gomod-cache/google.golang.org/protobuf@v1.36.11/internal/impl/codec_field_opaque.go:166 +0xfc fp=0x39f63a0c1278 sp=0x39f63a0c11e8 pc=0xe8747c
google.golang.org/protobuf/internal/impl.(*MessageInfo).marshalAppendPointer(0x39f635135520, {0x39f647e4e000?, 0x43a365?, 0x7f67e2e91fc0?}, {0x39f63a0c1360?}, {0xde?})
        /gomod-cache/google.golang.org/protobuf@v1.36.11/internal/impl/encode.go:215 +0x403 fp=0x39f63a0c1318 sp=0x39f63a0c1278 pc=0xeb2ba3
google.golang.org/protobuf/internal/impl.(*MessageInfo).marshal(0x39f647e4e000?, {{}, {0x1b61018, 0x39f638b29440}, {0x39f647e4e000, 0x0, 0xa00005000}, 0x2})
        /gomod-cache/google.golang.org/protobuf@v1.36.11/internal/impl/encode.go:142 +0x69 fp=0x39f63a0c1358 sp=0x39f63a0c1318 pc=0xeb2729
google.golang.org/protobuf/internal/impl.(*MessageInfo).marshal-fm({{}, {0x1b61018, 0x39f638b29440}, {0x39f647e4e000, 0x0, 0xa00005000}, 0x2})
        <autogenerated>:1 +0x4e fp=0x39f63a0c13a0 sp=0x39f63a0c1358 pc=0xedf8ce
google.golang.org/protobuf/proto.MarshalOptions.marshal({{}, 0x0?, 0x50?, 0x0?}, {0x39f647e4e000, 0x0, 0xa00005000}, {0x1b61018, 0x39f638b29440})
        /gomod-cache/google.golang.org/protobuf@v1.36.11/proto/encode.go:195 +0x38b fp=0x39f63a0c1470 sp=0x39f63a0c13a0 pc=0xc88bab
google.golang.org/protobuf/proto.MarshalOptions.MarshalAppend({{}, 0xa0?, 0xf7?, 0x96?}, {0x39f647e4e000, 0x0, 0xa00005000}, {0x1b35f80?, 0x39f638b29440?})
        /gomod-cache/google.golang.org/protobuf@v1.36.11/proto/encode.go:159 +0x73 fp=0x39f63a0c14b8 sp=0x39f63a0c1470 pc=0xc886d3
google.golang.org/grpc/encoding/proto.(*codecV2).Marshal(0x39f63806fdb0?, {0x196f7a0?, 0x39f638b29440?})
        /gomod-cache/google.golang.org/grpc@v1.80.0/encoding/proto/proto.go:75 +0x13f fp=0x39f63a0c1580 sp=0x39f63a0c14b8 pc=0xf6777f
google.golang.org/grpc.encode({0x7f67e3695230?, 0x2ad82a0?}, {0x196f7a0?, 0x39f638b29440?})
        /gomod-cache/google.golang.org/grpc@v1.80.0/rpc_util.go:804 +0x4a fp=0x39f63a0c1600 sp=0x39f63a0c1580 pc=0xf90caa
google.golang.org/grpc.prepareMsg({0x196f7a0?, 0x39f638b29440?}, {0x7f67e3695230?, 0x2ad82a0?}, {0x0, 0x0}, {0x1b49fd0, 0x39f634f8a910}, {0x1b463e8, 0x39f634fff7a0})
        /gomod-cache/google.golang.org/grpc@v1.80.0/stream.go:1892 +0xe5 fp=0x39f63a0c1698 sp=0x39f63a0c1600 pc=0xfab105
google.golang.org/grpc.(*clientStream).SendMsg(0x39f6356966c0, {0x196f7a0, 0x39f638b29440})
        /gomod-cache/google.golang.org/grpc@v1.80.0/stream.go:957 +0xf1 fp=0x39f63a0c1888 sp=0x39f63a0c1698 pc=0xfa3cb1
google.golang.org/grpc.invoke({0x1b4ba10?, 0x39f638b29410?}, {0x1afe82e?, 0x39f638251950?}, {0x196f7a0, 0x39f638b29440}, {0x19ba5e0, 0x39f636e08000}, 0x500f6382519a8?, {0x39f63518cce0, ...})
        /gomod-cache/google.golang.org/grpc@v1.80.0/call.go:70 +0x99 fp=0x39f63a0c18f0 sp=0x39f63a0c1888 pc=0xf82c79
google.golang.org/grpc.(*ClientConn).Invoke(0x39f6351a0408, {0x1b4ba10?, 0x39f638b29410?}, {0x1afe82e?, 0x44?}, {0x196f7a0?, 0x39f638b29440?}, {0x19ba5e0?, 0x39f636e08000?}, {0x39f6379245c0, ...})
        /gomod-cache/google.golang.org/grpc@v1.80.0/call.go:37 +0x232 fp=0x39f63a0c1988 sp=0x39f63a0c18f0 pc=0xf82b72
github.com/haka/go-shared-protos/generated/firewall_events_backend_pb.(*firewallEventsBackendClient).FlushInvocations(0x39f63516cd90, {0x1b4ba10, 0x39f638b29410}, 0x39f638b29440, {0x0, 0x0, 0x39f634e45140?})
        /go-shared-protos/generated/firewall_events_backend_pb/firewall_events_backend_grpc.pb.go:55 +0x167 fp=0x39f63a0c1a18 sp=0x39f63a0c1988 pc=0x1010d27
github.com/haka/firewall_service/internal/services/grpc_invocation_reporter.flushBatch({0x1b3d748, 0x39f63516cd90}, {0x1b4b9d8, 0x2ad82a0}, {0x39f634e45140, 0x8}, 0x39f6351df490, 0x39f636b42780, {0x39f635cd7130, 0x2, ...})
        /workspace/internal/services/grpc_invocation_reporter/invocation_reporter.go:155 +0xadd fp=0x39f63a0c1e20 sp=0x39f63a0c1a18 pc=0x15c693d
github.com/haka/firewall_service/internal/services/grpc_invocation_reporter.(*InvocationReporter).loopForTenant(0x39f635185c00, {0x39f634e45140, 0x8}, 0x39f63d6fb3b0)
        /workspace/internal/services/grpc_invocation_reporter/invocation_reporter.go:188 +0x339 fp=0x39f63a0c1fb0 sp=0x39f63a0c1e20 pc=0x15c7139
github.com/haka/firewall_service/internal/services/grpc_invocation_reporter.(*InvocationReporter).ReportEvent.gowrap1()
        /workspace/internal/services/grpc_invocation_reporter/invocation_reporter.go:397 +0x25 fp=0x39f63a0c1fe0 sp=0x39f63a0c1fb0 pc=0x15c8da5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1771 +0x1 fp=0x39f63a0c1fe8 sp=0x39f63a0c1fe0 pc=0x494ea1
created by github.com/haka/firewall_service/internal/services/grpc_invocation_reporter.(*InvocationReporter).ReportEvent in goroutine 1234
        /workspace/internal/services/grpc_invocation_reporter/invocation_reporter.go:397 +0xf4

panic: runtime error: makeslice: len out of range

goroutine 3006 [running]:
google.golang.org/grpc/internal/mem.(*simpleBufferPool).Get(0x2b798dbf2300, 0x6c757365725f9d7c)
        /gomod-cache/google.golang.org/grpc@v1.80.0/internal/mem/buffer_pool.go:332 +0xe7
google.golang.org/grpc/internal/mem.(*BinaryTieredBufferPool).Get(0x196f7a0?, 0x1b35f80?)
        /gomod-cache/google.golang.org/grpc@v1.80.0/internal/mem/buffer_pool.go:143 +0x75
google.golang.org/grpc/encoding/proto.(*codecV2).Marshal(0x2b7994222410?, {0x196f7a0?, 0x2b79941e44e0?})
        /gomod-cache/google.golang.org/grpc@v1.80.0/encoding/proto/proto.go:74 +0x119
google.golang.org/grpc.encode({0x7f0d74eaf230?, 0x2ad82a0?}, {0x196f7a0?, 0x2b79941e44e0?})
        /gomod-cache/google.golang.org/grpc@v1.80.0/rpc_util.go:804 +0x4a
google.golang.org/grpc.prepareMsg({0x196f7a0?, 0x2b79941e44e0?}, {0x7f0d74eaf230?, 0x2ad82a0?}, {0x0, 0x0}, {0x1b49fd0, 0x2b798db60910}, {0x1b463e8, 0x2b798dbcb7a0})
        /gomod-cache/google.golang.org/grpc@v1.80.0/stream.go:1892 +0xe5
google.golang.org/grpc.(*clientStream).SendMsg(0x2b79919f2fc0, {0x196f7a0, 0x2b79941e44e0})
        /gomod-cache/google.golang.org/grpc@v1.80.0/stream.go:957 +0xf1
google.golang.org/grpc.invoke({0x1b4ba10?, 0x2b79941e44b0?}, {0x1afe82e?, 0x2b79942c3950?}, {0x196f7a0, 0x2b79941e44e0}, {0x19ba5e0, 0x2b798e926b40}, 0x50079942c39a8?, {0x2b79928c28e0, ...})
        /gomod-cache/google.golang.org/grpc@v1.80.0/call.go:70 +0x99
google.golang.org/grpc.(*ClientConn).Invoke(0x2b798dd56408, {0x1b4ba10?, 0x2b79941e44b0?}, {0x1afe82e?, 0x44?}, {0x196f7a0?, 0x2b79941e44e0?}, {0x19ba5e0?, 0x2b798e926b40?}, {0x2b798fce20d0, ...})
        /gomod-cache/google.golang.org/grpc@v1.80.0/call.go:37 +0x232
github.com/haka/go-shared-protos/generated/firewall_events_backend_pb.(*firewallEventsBackendClient).FlushInvocations(0x2b798dd2edf0, {0x1b4ba10, 0x2b79941e44b0}, 0x2b79941e44e0, {0x0, 0x0, 0x2b798de00e90?})
        /go-shared-protos/generated/firewall_events_backend_pb/firewall_events_backend_grpc.pb.go:55 +0x167
github.com/haka/firewall_service/internal/services/grpc_invocation_reporter.flushBatch({0x1b3d748, 0x2b798dd2edf0}, {0x1b4b9d8, 0x2ad82a0}, {0x2b798de00e90, 0xa}, 0x2b79915d5810, 0x2b7990139ec0, {0x2b79948e4690, 0x1, ...})
        /workspace/internal/services/grpc_invocation_reporter/invocation_reporter.go:155 +0xadd
github.com/haka/firewall_service/internal/services/grpc_invocation_reporter.(*InvocationReporter).loopForTenant(0x2b798dd45c00, {0x2b798de00e90, 0xa}, 0x2b798dcdd500)
        /workspace/internal/services/grpc_invocation_reporter/invocation_reporter.go:188 +0x339
created by github.com/haka/firewall_service/internal/services/grpc_invocation_reporter.(*InvocationReporter).ReportEvent in goroutine 2934
        /workspace/internal/services/grpc_invocation_reporter/invocation_reporter.go:397 +0xf4

Also our memory started spiking at times and we had some OOMKills where before that out memory usage was never over 10% (we limit our pods at 2GiB memory).

I assume this is related to this change https://github.com/grpc/grpc-go/pull/8775/changes released in v1.80.0.
It was marked as experimental in the changelog but apparently it is the default pool.

[Pranjali-2501]

Hi @hxyannay ,

The panic makeslice: len out of range occurs in SimpleBufferPool.Get because it is trying to allocate a slice with a size larger than maxPoolCap.

google.golang.org/grpc/internal/mem.(*simpleBufferPool).Get(0x2b798dbf2300, 0x6c757365725f9d7c)
        /gomod-cache/google.golang.org/grpc@v1.80.0/internal/mem/buffer_pool.go:332 +0xe7
google.golang.org/grpc/internal/mem.(*BinaryTieredBufferPool).Get(0x196f7a0?, 0x1b35f80?)
        /gomod-cache/google.golang.org/grpc@v1.80.0/internal/mem/buffer_pool.go:143 +0x75
google.golang.org/grpc/encoding/proto.(*codecV2).Marshal(0x2b7994222410?, {0x196f7a0?, 0x2b79941e44e0?})
        /gomod-cache/google.golang.org/grpc@v1.80.0/encoding/proto/proto.go:74 +0x119
google.golang.org/grpc.encode({0x7f0d74eaf230?, 0x2ad82a0?}, {0x196f7a0?, 0x2b79941e44e0?})

Here is how execution reached that point :

• BinaryTieredBufferPool.Get was called during message marshaling.
• It called poolForGet(size) to determine which pool to use.
  

  grpc-go/internal/mem/buffer_pool.go

  Lines 146 to 161 in 4acacce

  	func (b *BinaryTieredBufferPool) poolForGet(size int) bufferPool {
  	if size == 0 || size > b.maxPoolCap {
  	return b.fallbackPool
  	}
  	// Calculate the exponent of the smallest power of 2 >= size.
  	// We subtract 1 from size to handle exact powers of 2 correctly.
  	//
  	// Examples:
  	// size=16 (0b10000) -> size-1=15 (0b01111) -> bits.Len=4 -> Pool for 2^4
  	// size=17 (0b10001) -> size-1=16 (0b10000) -> bits.Len=5 -> Pool for 2^5
  	querySize := uint(size - 1)
  	poolIdx := b.exponentToNextLargestPoolMap[bits.Len(querySize)]
  	return b.sizedPools[poolIdx]
  	}

  
  Since the requested size was massive, it exceeded b.maxPoolCap and returned b.fallbackPool.
• It then called .Get(size) on that fallback pool, leading to the panic when it tried to make a slice with that huge size.

The size argument passed to Get in your log: 0x6c757365725f9d7c is huge, it bypasses the logic added in #8775 and eventually call the existing simpleBufferPool.Get function.

Can you share the minimal repro for the panic you are facing from you application.

Thanks

[Pranjali-2501]

cc: @arjan-bal

[arjan-bal]

Hi @hxyannay, it looks like these crashes are being caused by data races when accessing protobuf objects.

• Crash 1: A string object is pointing to an invalid memory address. This happens if a goroutine updates a proto field exactly while gRPC is calling proto.Marshal.
• Crash 2: When gRPC calls proto.Size, it is getting a return value of 0x6c757365725f9d7c (which translates to luser_ in ASCII). A string is being read as an integer length, causing gRPC to attempt a massive memory allocation and crash. This is another symptom of the protobuf object being mutated during the proto.Size call.

Could you try building the binary with the Go race detector enabled (-race)? That should help us quickly track down the code that is concurrently writing to these proto objects.

[github-actions]

This issue is labeled as requiring an update from the reporter, and no update has been received after 6 days. If no update is provided in the next 7 days, this issue will be automatically closed.