Unable to get screen frames on ios 16

[chenhengjie123]

I forked this project, and change qvh module to code version of externalizeGST branch fixing gstream issue. While I'm using ws-qvh with ws-scrcpy with USE_QVH_SERVER enabled. I found it can get datas from ios 14 and ios 15, but not ios 16.

Versions:

Here are versions I'm using:

• quicktime_video_hack: https://github.com/chenhengjie123/quicktime_video_hack (based on https://github.com/danielpaulus/quicktime_video_hack/tree/externalizeGST, only add some logs for troubleshooting. )

• ws-qvh: https://github.com/chenhengjie123/ws-qvh (based on https://github.com/NetrisTV/ws-qvh. Only add logs for troubleshooting and change quicktime_video_hack to version above)

• ws-scrcpy: https://github.com/chenhengjie123/ws-scrcpy/tree/feat-20230302-testOnWsscrcpy (based on https://github.com/NetrisTV/ws-scrcpy, Only add logs to output ws-qvh stdout and stderr to console)

• mac os version: 12.1(21C52)

• go version: go version go1.17 darwin/amd64

• node version: v14.16.0

Steps to reproduce

1. Clone and build ws-qvh using command go build, then got ws-qvh binary file.
2. Add ws-qvh file to environment variable PATH
3. Connect ios device to mac.
4. Start ws-scrcpy using code version https://github.com/chenhengjie123/ws-scrcpy/tree/feat-20230302-testOnWsscrcpy using command npm install && npm start
5. Open http://127.0.0.1:8000/ , select the ios device and open its remote control module:
   

On ios 14, 15, the screen content can appear normally like below:

But on ios 16.0, 16.1.1, the screen content did not appear:

From ws messages in screenshot, it seems no data came from server side.

Logs from server side

On iOS 14 or 15, I can get screen frames successfully and quickly, here are logs(most of them are logs from ws-qvh):

logs-on-ios-15.log

While on ios 16, it stuck while reading lengthBuffer like below:

logs-on-ios-16.1.1.log

[drauggres]

Hi.
Sorry, I can't really help you.
I don't have mac and any device with ios 16.

Were you able to receive video from quicktime_video_hack with ios 16?

[chenhengjie123]

Hi. Sorry, I can't really help you. I don't have mac and any device with ios 16.

Were you able to receive video from quicktime_video_hack with ios 16?

Actually I don't know how to get video using https://github.com/danielpaulus/quicktime_video_hack/tree/externalizeGST. On this branch, the gstream is removed and I have no idea how to get screen datas.

Do you know how to get it?

[drauggres]

qvh record out.h264 out.wav should work without gstreamer, but I haven't tested it.

[chenhengjie123]

qvh record out.h264 out.wav should work without gstreamer, but I haven't tested it.

I have tested using command above, it works fine on both ios 15.6.1 and ios 16.0.0. It can successfully record out.h264 videos.

Here are Logs:

• ios 15.6.1

./quicktime_video_hack record out.h264 out.wav
{"level":"info","msg":"no usbSerial specified, using '00008020000A1CD80A46002E\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000'","time":"2023-03-28T11:26:13+08:00"}
{"level":"info","msg":"found iPhone 15.6.1 for udid 00008020-000A1CD80A46002E","time":"2023-03-28T11:26:13+08:00"}
{"level":"warning","msg":"failed closing contextContext.Close called while 1 Devices are still open, Close may be called only after all previously opened devices were successfuly closed","time":"2023-03-28T11:26:13+08:00"}
{"level":"info","msg":"Device '00008020000A1CD80A46002E\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000' USB connection ready, waiting for ping..","time":"2023-03-28T11:26:14+08:00"}
{"level":"info","msg":"PING received","time":"2023-03-28T11:26:14+08:00"}
^C{"level":"info","msg":"Telling device to stop streaming..","time":"2023-03-28T11:26:20+08:00"}
{"level":"info","msg":"Waiting for device to tell us to stop..","time":"2023-03-28T11:26:20+08:00"}
{"level":"info","msg":"OK. Ready to release USB Device.","time":"2023-03-28T11:26:20+08:00"}
{"level":"info","msg":"Closing usb stream","time":"2023-03-28T11:26:20+08:00"}
{"level":"info","msg":"Closing usb interface","time":"2023-03-28T11:26:20+08:00"}
{"level":"fatal","msg":"Error closing usb context: \u0026{0x139804080 0x140001102a0 {} {0 0} map[0x14000140580:true]}","time":"2023-03-28T11:26:20+08:00"}

• ios 16.0.0

$ ./quicktime_video_hack record out.h264 out.wav
{"level":"info","msg":"no usbSerial specified, using '00008101000E5DD43C92001E\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000'","time":"2023-03-28T11:26:49+08:00"}
{"level":"info","msg":"found qiuzilin的iPhone 16.0 for udid 00008101-000E5DD43C92001E","time":"2023-03-28T11:26:49+08:00"}
{"level":"info","msg":"Device '00008101000E5DD43C92001E\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000' USB connection ready, waiting for ping..","time":"2023-03-28T11:26:49+08:00"}
{"level":"info","msg":"PING received","time":"2023-03-28T11:26:49+08:00"}
^C{"level":"info","msg":"Telling device to stop streaming..","time":"2023-03-28T11:26:54+08:00"}
{"level":"info","msg":"Waiting for device to tell us to stop..","time":"2023-03-28T11:26:54+08:00"}
{"level":"info","msg":"OK. Ready to release USB Device.","time":"2023-03-28T11:26:54+08:00"}
{"level":"info","msg":"Closing usb stream","time":"2023-03-28T11:26:54+08:00"}
{"level":"info","msg":"Closing usb interface","time":"2023-03-28T11:26:54+08:00"}
{"level":"fatal","msg":"Error closing usb context: \u0026{0x12d704080 0x140002036e0 {} {0 0} map[0x14000320040:true]}","time":"2023-03-28T11:26:54+08:00"}

It looks like not an issue from quicktime_video_hack project.

[LaiZhou]

@chenhengjie123 is there a workround? I need screen casting on IOS 16.0+