Segmentation Fault after registeration process in Reconstruction pipeline

[zainmehdi]

I am trying to run reconstruction pipeline with the python scripts provided. I recorded my data using realsense d435i. When I run the script it is able to register the fragment but just after this phase it gives me segmentation fault.

To Reproduce
I used the sample dataset sequence 16 as provided in the tutorial. If you try to run the pipeline it should give segmentation fault. The below screenshot is however of my custom data but the results were same in both cases.

**python run_system.py config/realsense.json --make --register --refine --integrate
**
Expected behavior
Segmentation fault

Screenshots
If applicable, add screenshots to help explain your problem.

I am running in in conda base environment

• Operating system: (Ubuntu 18.04)
• Python version: (e.g. Python 2.7)
• Open3D version:0.9
• Is this remote workstation?: no
• How did you install Open3D?: (build from source)
• Compiler version (if built from source): (e.g. gcc 7.5)

Additional context

[griegler]

@theNded can you have a look at this please.

[zainmehdi]

Ok so there is a definite memory leak somewhere. I have a core i7 2.6 Ghz with 16GB of RAM and its consuming all the memory and CPU usage is very high as well. Here is the screen shot

[theNded]

There are some issues with python multithread regarding memory usage. Can you try to turn it off in the config file and try again?
Another potential issue can be the failure of global registration. In global registration I can see there are 19 nodes and 18 edges, which means no valid loop closure is detected. This may cause incorrect integration into an unreasonably large space.

[zainmehdi]

@theNded Thanks for the reply. I tried turning multi threading option false and even then it causes segmentation fault if the voxel size is smaller than 5cm and depth scale in image is beyond 3m.

[theNded]

Does 5cm and 3m work? In fact voxel size in config file is a little bit confusing -- the real voxel size is 5cm / 512 (see https://github.com/intel-isl/Open3D/blob/master/examples/python/ReconstructionSystem/integrate_scene.py#L24). So it can be good to check the result of this default configuration.

Another suggestion can be checking the reconstruction of fragments in the fragment folder using https://github.com/intel-isl/Open3D/blob/master/examples/python/ReconstructionSystem/debug/visualize_fragments.py.

[zainmehdi]

Yes 5cm and 3m work. And I checked the results of the fragments. It produces ply files. Here are few more interesting observations.

• If I try to run Fragmentation part with multi-threading it crashes. Running it without multi threading works with the above mentioned voxel and depth values. Else it fails even in single threading mode.
• The registration part doesnt work with single thread no matter what the voxel and the depth values. If I change the config to multi-threading it works.
• Refine and integrate works fine on both the modes (single and multi thread)

[theNded]

• If I try to run Fragmentation part with multi-threading it crashes. Running it without multi threading works with the above mentioned voxel and depth values. Else it fails even in single threading mode.

For multithread you may want to tune MAX_THREAD here: https://github.com/intel-isl/Open3D/blob/master/examples/python/ReconstructionSystem/make_fragments.py#L181

It makes sense if the resolution is very high, otherwise it will be interesting to look at the log.

• The registration part doesnt work with single thread no matter what the voxel and the depth values. If I change the config to multi-threading it works.

This is weird, can you put the log here?

[zainmehdi]

Here is the log for the second case. For the first case I will do more test and share some logs with you.
Configuration

                                    name : Captured frames using Realsense
                            path_dataset : dataset/realsense/
                          path_intrinsic : dataset/realsense/camera_intrinsic.json
                               max_depth : 3.0
                              voxel_size : 0.1
                          max_depth_diff : 0.07
        preference_loop_closure_odometry : 0.1
    preference_loop_closure_registration : 5.0
                         tsdf_cubic_size : 3.0
                              icp_method : color
                     global_registration : ransac
                  python_multi_threading : False
                          depth_map_type : redwood
                   n_frames_per_fragment : 100
                 n_keyframes_per_n_frame : 5
                               min_depth : 0.3
                         folder_fragment : fragments/
             template_fragment_posegraph : fragments/fragment_%03d.json
   template_fragment_posegraph_optimized : fragments/fragment_optimized_%03d.json
            template_fragment_pointcloud : fragments/fragment_%03d.ply
                            folder_scene : scene/
               template_global_posegraph : scene/global_registration.json
     template_global_posegraph_optimized : scene/global_registration_optimized.json
              template_refined_posegraph : scene/refined_registration.json
    template_refined_posegraph_optimized : scene/refined_registration_optimized.json
                    template_global_mesh : scene/integrated.ply
                    template_global_traj : scene/trajectory.log
                              debug_mode : False
Filament library loaded.
register fragments.
reading dataset/realsense/fragments/fragment_000.ply ...
Segmentation fault (core dumped)

[theNded]

It would be nice if you can share your data somewhere, I think I have to take a look closely...

[zainmehdi]

https://drive.google.com/file/d/1O0CRAu7GJaY1J19l7Qk_Gnh9qfzfmxcG/view?usp=sharing
Here is the data

[theNded]

Sorry for coming back late.
There are some problems with the data:

1. You are not using the default intrinsic matrix -- you need to modify path_intrinsic and point it to your specific intrinsic config file. the config seems correct, but somehow the fragments look distorted...
2. More than 50% of the scene are glasses -- this is very challenging for RGB-D sensors. Your sensor will either return invalid value or 2x distances (caused by the mirror reflection).

[zainmehdi]

Thanks alot. I will look into it. I tried with Azure Kinect and results were better.

[devernay]

0.11.0 has this same memory leak issue with the reconstruction_system tutorial (using the data from 016.zip), so I don't think the data is faulty here.
It fails on a machine with 256Gb RAM (keeping the default config parameters). open3d.pipelines.integration.ScalableTSDFVolume.integrate seems to be responsible for this leak, as suggested by the crash report above, which is similar to the one I got, with a more explicit error message (I checked using htop, and it fills up the 256Gb+swap pretty quickly):

...
Fragment 002 / 013 :: integrate rgbd frame 287 (88 of 100).
Traceback (most recent call last):
  File "run_system.py", line 68, in <module>
    make_fragments.run(config)
  File ".../Open3D/examples/python/reconstruction_system/make_fragments.py", line 182, in run
    Parallel(n_jobs=MAX_THREAD)(delayed(process_single_fragment)(
  File ".../lib/python3.8/site-packages/joblib/parallel.py", line 1061, in __call__
    self.retrieve()
  File ".../lib/python3.8/site-packages/joblib/parallel.py", line 940, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File ".../lib/python3.8/site-packages/joblib/_parallel_backends.py", line 542, in wrap_future_result
    return future.result(timeout=timeout)
  File ".../lib/python3.8/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File ".../lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.

The exit codes of the workers are {SIGKILL(-9)}

Tested with 0.10.0: works fine. Anyone encountering this problem should downgrade Open3D for now.