pyramid strategy fails for sparse optical flow

[ahsteven]

I am running the tracker on several videos. However sometimes the gmc sparse optical flow fails. I get this error:

File BoTSORT/tracker/gmc.py", line 72, in apply
return self.applySparseOptFlow(raw_frame, detections)
File "BoTSORT/tracker/gmc.py", line 268, in applySparseOptFlow
matchedKeypoints, status, err = cv2.calcOpticalFlowPyrLK(self.prevFrame, frame, self.prevKeyPoints, None)
cv2.error: OpenCV(4.7.0) /io/opencv/modules/video/src/lkpyramid.cpp:1394: error: (-215:Assertion failed) prevPyr[level * lvlStep1].size() == nextPyr[level * lvlStep2].size() in function 'calc'

The video input size was 1080,1920,3

if I cannot use sparse optical flow what other setting do you recommend for gmc? 'orb', 'ecc', 'sift'?

[SalenGit]

#90 #78
These issues mentioned above all stem from the same bug. Although some time has passed, I encountered the same problem and found an effective solution. I hope this helps the next person who comes across this.

The root cause is that the video scene you input has excessive changes, causing the optical flow algorithm to fail to find any matching keypoints between adjacent frames. Strangely, several other optical flow algorithms have already implemented effective handling for this scenario—specifically, assuming no pose change for frames where pose estimation fails. Only the default method applySparseOptFlow lacks this implementation. Therefore, the solution is straightforward: replace the following code:

matchedKeypoints, status, err = cv2.calcOpticalFlowPyrLK(self.prevFrame, frame, self.prevKeyPoints, None)

to:

try:
    matchedKeypoints, status, err = cv2.calcOpticalFlowPyrLK(self.prevFrame, frame, self.prevKeyPoints, None)
except:
    print('Warning: find transform failed. Set warp as identity')
    # Store to next iteration
    self.prevFrame = frame.copy()
    self.prevKeyPoints = copy.copy(keypoints)

    return H

This perfectly resolves the issue. It instructs the algorithm to use the predefined identity transformation H matrix for frames where matching fails due to significant mid-process changes. Simultaneously, it preserves this frame for continued matching downstream. This ensures the optical flow algorithm resumes normal operation immediately once scene changes subside, preventing persistent errors.

Ideally, you should now encounter only one or two printed warnings, with no further errors occurring.