Breakpoint in RenderRequest.cpp while taking Scene Images

[Gastastrophe]

Bug report

• AirSim Version/#commit: v1.5.0-windows (also seen in 1.6.0)
• UE/Unity version: Unreal 4.25 (also seen in 4.26)
• autopilot version: SimpleFlight (Also seen with PX4 SITL)
• OS Version: Windows version 1809

What's the issue you encountered?

As I brought up in Issue #3291, when taking lots of Scene images as one would do for Reinforcement Learning, Unreal Engine generates a breakpoint inside of RenderRequest.cpp and crashes the simulator.

Settings

Default settings

How can the issue be reproduced?

1. Download and Build AirSim and the default Blocks environment.
2. Change the image request in the init function of drone_env.py from self.image_request = airsim.ImageRequest( 3, airsim.ImageType.DepthPerspective, True, False ) to self.image_request = airsim.ImageRequest( 3, airsim.ImageType.Scene, True, False )
3. Inside of transform_obs, change img2d = np.reshape(img1d, (responses[0].height, responses[0].width)) to img2d = np.reshape(img1d, (responses[0].height, responses[0].width, 3))
4. Inside of transform_obs, change return im_final.reshape([84, 84, 1]) to return im_final.reshape([84, 84, 3])
5. Start Unreal Engine and run dqn_drone.py. Wait for a minute or two for the images to be taken and processed.

Include full error message in text form

UE4Editor-Win64-DebugGame.exe has triggered a breakpoint.

[cpatel245]

@Gastastrophe
Facing same issue. Have you found the solution ?

[Gastastrophe]

@crypto-g245
No. I think it has something to do with how Unreal flushes these kinds of images and the computational expense of that. I think the closest thing to a workaround available is to write your own function to take screenshots on a separate thread, but this fails to generalize to real drones and is very slow.

Right now I am just using Depth images instead, though that means I am unable to move forward with my particular project until this is fixed.

[Gastastrophe]

@crypto-g245

This isn't a fix, but a simple workaround is to use the version of the function that returns a bitmap. This means you call self.image_request = airsim.ImageRequest( 3, airsim.ImageType.Scene, False, False )
and then do something like

`img1d_rgb = np.frombuffer(responses[0].image_data_uint8, dtype=np.uint8)'

'img3d_rgb = img1d_rgb.reshape(responses[0].height, responses[0].width, 3)'

'from PIL import Image'

'image = Image.fromarray(img3d_rgb)'

'im_final = np.array(image.resize((84, 84)).convert("L"))'

'return im_final.reshape([84, 84, 1])`
in the transform_obs function. I don't know why this one works, but it doesn't slow things down much because we are converting to an image with PIL for resizing anyway. RenderRequest.cpp has a comment saying '//below is undocumented method that avoids flushing, but it seems to segfault every 2000 or so calls' for this version, so someone must have addressed part of this at some point. I'd still appreciate it if the pixels_as_float version worked, but this will do as a temporary workaround

EDIT: I'm not sure why the code thing isn't working, but hopefully it is readable enough

[Gastastrophe]

I noticed people in past issues talking about the image APIs like they return RGBA images, but the bitmap is sized in such a way that implies it is a RGB image. Maybe Unreal changed this at some point and Airsim still needs to reflect this?

[zimmy87]

This is occurring due to a mismatch between the pixel format of UTextureRenderTarget2D instances inside scene capture components for each image type and the pixel format expected by the FD3D11DynamicRHI::RHIReadSurfaceFloatData API. I've put out PR #4321 to update the pixel formats. Please retest with my PR applied and let me know if you see any issues.

[Gastastrophe]

@zimmy87 Thank you for the fix! This'll be a great help