Add a demo showing how to process video frames using GL before presenting them

[msilva-verimatrix]

Good afternoon,
What would be the recommended way to extract video frames from ExoPlayer during the playback of a network url, edit the frames (to apply YUV filtering, effects or watermarking) and provide the updated frames to the player? Is it intercepting MediaCodec.queueInputBuffer() the way to go? The related ByteBuffer seems to aggregate raw video frames, but how to iterate through these frames?
Can you provide a code snippet or point to an example?
Warm regards,

Moacir Silva

[iamradhakrishna]

As per my basic analysis,this is possible by listening public void onFrameAvailable(SurfaceTexture surfaceTexture) method while rendering the video to surface but i am also looking for best solution.
Please suggest on this.

Best regards,
RK

[msilva-verimatrix]

OK, thank you.
By configuring the video stream MediaCodec with a null surface in MediaCodecVideoRenderer.configureCodec(), it was possible to redirect the video frames to MediaCodecRenderer.drainOutputBuffer(), convert them to jpeg using YuvImage and then to Bitmaps using BitmapFactory. But I will also try the approach that you have suggested.
Are there plans to add support to more image formats in YuvImage in the near future? At the moment it seems to only support ImageFormat.NV21 and ImageFormat.YUY2...
Warm regards,

Moacir Silva

[andrewlewis]

Is the idea to do some postprocessing on video frames output by the decoder before showing them to the user?

If so, this can be done by adding a GLSurfaceView to your layout, passing the player a Surface created from a SurfaceTexture based on a GL texture, and doing the filtering/compositing in a GL shader in your GLSurfaceView.Renderer, using the texture as input. I will mark this as an enhancement to add a demo of this functionality as there is quite a bit of non-obvious code required to make it work!

[msilva-verimatrix]

Indeed, that would be the idea. And it would be great to have the demo!
Many thanks,

Moacir

[ojw28]

There is now a demo app you can take a look at. Clone the project, open in Android Studio, and look at the demo-gl demo module.

[msilva-verimatrix]

That is great!
Many thanks,

Moacir