Bad collision behavior after updating bullet physics

[xhighway999]

Hey,
after updating Bullet Physics to the latest version something how static trimesh collisions are handled must have changed.
My game uses the static trimesh for level collisions. That weird collision behavior was introduced after updating bullet to the newest master (of this git repo). Video as reference: https://youtu.be/9HOU5IksyBo
changing substeps/ fixed time step does not yield any difference.
[UPDATE: After testing different versions i can tell that this bug occurs starting with version 3.22]

[xhighway999]

It seems this commit causes this 28b951c . Any ideas how to fix this ?

[erwincoumans]

Thanks for tracking this down! Apparently the early out causes issues. Do you have some reproduction case? Some triangle mesh and the collision objects / ray tests against it?

Let's revert the logic and disable this early-out by default.

[xhighway999]

Well, looking at bullet's debug draw, it seems like not all collisions are registered. Th
e simplest mesh to reproduce this behavior can be found in the zip file. I guess I do something wonky when loading the mesh as I call addTriangle (from the btTriangleMesh class) and generate a mesh this way.
Another video demonstrating the problem with a plane trimesh and a cube: https://youtu.be/eoaDEF4w32Q
The file: plane.glb.zip

If you want me to I can probably write an C/C++ example with the same behavior when I have some spare time

[erwincoumans]

A small C++ example would be great! You could modify one of the existing Bullet examples (in the ExampleBrowser?)

[DVLP]

In the two comparisons dist > contact_threshold there may be abs needed on dists and/or square roots(or normalization if this is about angles not distances?) on proj_distPt and proj_distTr.

[DVLP]

In some circumstances triangle_normal_world can have 0,0,0 values after ( v1 - v0).cross(v2 - v0) and normalizing in the next line causes it to turn into NaNs. Maybe it's due to a somehow broken geometry I'm using but it might be part of the issue here.

[DVLP]

@xhighway999 do you think you could track down what are the values of m_triBodyWrap->getWorldTransform(), triangle, and resulting v0 v1 v2 when encountering the issue?

Also before triangle_normal_world.normalize() line you can add if (triangle_normal_world.x() + triangle_normal_world.y() + triangle_normal_world.z() == btScalar(0.)) printf("Invalid normal \n"); and see if this prints out when encountering the issue.

If this shows up I'll look further into the issue.