How to use agnocast to get zero copy

[allstarry]

I am trying to integrate Agnocast in my code. According to some document pieces, I finally got it working with seeing agnocast in dmesg, but there are 2 main issues.

1. The listener subscription callback function cannot be triggered.
2. The program crashed with exit code -11 after working for a while, and it won't crash after commenting out agnocast code, but the receiving one works fine.
   After that, I used autoware_agnocast_wrapper and got the same.
   Here's some of my code with agnocast.

Talker

AUTOWARE_PUBLISHER_PTR(custom_msgs::msg::Rgb3) multi_rgb_pub_agn;
AUTOWARE_MESSAGE_UNIQUE_PTR(custom_msgs::msg::Rgb3) ipc_msg_ptr;
ipc_msg_ptr = ALLOCATE_OUTPUT_MESSAGE_UNIQUE(multi_rgb_pub_agn);
...
multi_rgb_pub_agn->publish(std::move(ipc_msg_ptr));

Listenner

AUTOWARE_SUBSCRIPTION_PTR(custom_msgs::msg::Rgb3) subscription;
agnocast::SubscriptionOptions sub_options{};
callback_group_ = create_callback_group(rclcpp::CallbackGroupType::MutuallyExclusive);
  sub_options.callback_group = callback_group_;
  subscription_ = AUTOWARE_CREATE_SUBSCRIPTION(
    custom_msgs::msg::Rgb3,
    "/multi_cam_rgb3",
    rclcpp::SensorDataQoS{}.keep_last(1),
    std::bind(&IpcTestNode::jpg_cb, this, std::placeholders::_1),
    sub_options);

void jpg_cb(AUTOWARE_MESSAGE_UNIQUE_PTR(custom_msgs::msg::Rgb3) msg)
{...}

[Koichi98]

@allstarry
Thank you for the question. Let me answer one by one.

Issue 1: Subscription Callback Not Being Triggered

The root cause might likely be that Agnocast callbacks require a specialized executor that integrates with the kernel module's message queue system. Standard ROS 2 executors (rclcpp::spin(), SingleThreadedExecutor, etc.) only handle regular ROS 2 subscriptions and are unaware of Agnocast's epoll-based message delivery mechanism.

Solution:

If you're running nodes directly:

Replace your standard executor with an Agnocast-aware executor:

// Instead of standard executors:
  auto executor = std::make_shared<agnocast::SingleThreadedAgnocastExecutor>();
  // OR for multi-threaded:
  auto executor = std::make_shared<agnocast::MultiThreadedAgnocastExecutor>();

  executor->add_node(node);
  executor->spin();

If you're using a component container:

You must use the Agnocast-specific component containers instead of the standard component_container or component_container_mt:

• For single-threaded: agnocast_component_container
• For multi-threaded: agnocast_component_container_mt

These containers internally use the Agnocast executors to ensure callbacks are properly triggered.

In addition, please also check that you've specified libagnocat_heaphook.so to LD_PRELOAD as explained here.

Note:
We apologize that the documentation regarding the Agnocast executor requirement is still insufficient. We plan to add comprehensive documentation about executor usage and component container integration in the near future to prevent this common integration issue.

[Koichi98]

@allstarry
Issue 2: Segfault (-11) After Running for a While

To help diagnose the segmentation fault, could you please confirm whether you are using MultiThreadedExecutor (or component_container_mt) in your setup?

If you are using MultiThreadedExecutor: There is a known limitation where ROS 2 callbacks and Agnocast callbacks require manual synchronization (mutex guards) when they share data. This is documented in two places:

1. From docs/autoware_integration.md:55: "Until the subscription callback thread is integrated into ROS 2 executor, all callback functions should be guarded with mutex lock."
2. From src/agnocastlib/src/agnocast_multi_threaded_executor.cpp:64-71: "To prevent performance degradation, MultiThreadedAgnocastExecutor prohibits the agnocast callback and the ros2 callback from belonging to the same MutuallyExclusive callback group. If mutual exclusion between callbacks is not required, consider using Reentrant callback group. If mutual exclusion is required, separate them into different callback groups and use a mutex or other synchronization mechanism."

In MultiThreadedAgnocastExecutor, ROS 2 callbacks run in separate threads from Agnocast callbacks. If your callbacks access shared data without proper mutex guards, this can lead to race conditions and segmentation faults.

Troubleshooting Step: Switch to SingleThreadedAgnocastExecutor

To isolate the issue, please first try using the single-threaded executor:

auto executor = std::make_shared<agnocast::SingleThreadedAgnocastExecutor>();
executor->add_node(node);
executor->spin();

Or if using component containers, use agnocast_component_container instead of agnocast_component_container_mt.

Other likely cause: Insufficient Memory Pool Size

Another common cause of segfaults might be insufficient shared memory pool size. Please ensure you have set a sufficient AGNOCAST_MEMPOOL_SIZE as explained here.

Please feel free to ask any further questions!

[allstarry]

@allstarry Thank you for the question. Let me answer one by one.

Issue 1: Subscription Callback Not Being Triggered

The root cause might likely be that Agnocast callbacks require a specialized executor that integrates with the kernel module's message queue system. Standard ROS 2 executors (rclcpp::spin(), SingleThreadedExecutor, etc.) only handle regular ROS 2 subscriptions and are unaware of Agnocast's epoll-based message delivery mechanism.

Solution:

If you're running nodes directly:

Replace your standard executor with an Agnocast-aware executor:

// Instead of standard executors:
  auto executor = std::make_shared<agnocast::SingleThreadedAgnocastExecutor>();
  // OR for multi-threaded:
  auto executor = std::make_shared<agnocast::MultiThreadedAgnocastExecutor>();

  executor->add_node(node);
  executor->spin();

If you're using a component container:

You must use the Agnocast-specific component containers instead of the standard component_container or component_container_mt:

* For single-threaded: agnocast_component_container

* For multi-threaded: agnocast_component_container_mt

These containers internally use the Agnocast executors to ensure callbacks are properly triggered.

In addition, please also check that you've specified libagnocat_heaphook.so to LD_PRELOAD as explained here.

Note: We apologize that the documentation regarding the Agnocast executor requirement is still insufficient. We plan to add comprehensive documentation about executor usage and component container integration in the near future to prevent this common integration issue.

Thanks for your reply.
I finally solved this problem after finding the example code in this repository, but a complete document would be really helpful.

[allstarry]

@allstarry Issue 2: Segfault (-11) After Running for a While

To help diagnose the segmentation fault, could you please confirm whether you are using MultiThreadedExecutor (or component_container_mt) in your setup?

If you are using MultiThreadedExecutor: There is a known limitation where ROS 2 callbacks and Agnocast callbacks require manual synchronization (mutex guards) when they share data. This is documented in two places:

1. From docs/autoware_integration.md:55: ["Until the subscription callback thread is integrated into ROS 2 executor, all callback functions should be guarded with mutex lock."](https://github.com/tier4/agnocast/blob/main/docs/autoware_integration.md#:~:text=Until%20the%20subscription%20callback%20thread%20is%20integrated%20into%20ROS%202%20executor%2C%20all%20callback%20functions%20should%20be%20guarded%20with%20mutex%20lock.)

2. From src/agnocastlib/src/agnocast_multi_threaded_executor.cpp:64-71: ["To prevent performance degradation, MultiThreadedAgnocastExecutor prohibits the agnocast callback and the ros2 callback from belonging to the same MutuallyExclusive callback group. If mutual exclusion between callbacks is not required, consider using Reentrant callback group. If mutual exclusion is required, separate them into different callback groups and use a mutex or other synchronization mechanism."](https://github.com/tier4/agnocast/blob/b0b27e394d8cbdb9b176dccc236dcd7f186f132c/src/agnocastlib/src/agnocast_multi_threaded_executor.cpp#L67)

In MultiThreadedAgnocastExecutor, ROS 2 callbacks run in separate threads from Agnocast callbacks. If your callbacks access shared data without proper mutex guards, this can lead to race conditions and segmentation faults.

Troubleshooting Step: Switch to SingleThreadedAgnocastExecutor

To isolate the issue, please first try using the single-threaded executor:

auto executor = std::make_shared<agnocast::SingleThreadedAgnocastExecutor>();
executor->add_node(node);
executor->spin();

Or if using component containers, use agnocast_component_container instead of agnocast_component_container_mt.

Other likely cause: Insufficient Memory Pool Size

Another common cause of segfaults might be insufficient shared memory pool size. Please ensure you have set a sufficient AGNOCAST_MEMPOOL_SIZE as explained here.

Please feel free to ask any further questions!

Actually I have done all the tips you mentioned and after many test the final solution is to change ros log level from Debug to Info.
Here is my launch.py

import launch
from launch import LaunchDescription
from launch.actions import DeclareLaunchArgument, LogInfo
from launch.substitutions import LaunchConfiguration
from launch_ros.actions import Node
from ament_index_python.packages import get_package_share_directory
from launch.actions import SetEnvironmentVariable, ExecuteProcess
import os

def generate_launch_description():
    # Find the absolute path to the config file
    config_file_path = os.path.join(
        get_package_share_directory('multi_cam'), 'config', 'multi_cam_config.yaml'
    )
    
    return LaunchDescription([
        # Declare a launch argument for the config file, defaulting to the absolute path
        #SetEnvironmentVariable(name='LD_PRELOAD', value=f"libagnocast_heaphook.so:{os.getenv('LD_PRELOAD', '')}"),
        SetEnvironmentVariable(name='LD_PRELOAD', value="libagnocast_heaphook.so"),
        SetEnvironmentVariable(name='AGNOCAST_MEMPOOL_SIZE', value='671088640'),
        DeclareLaunchArgument(
            'config_file', default_value=config_file_path, description='Path to configuration file'
        ),
        # Log the configuration file path to check if it's correct
        LogInfo(
            msg=LaunchConfiguration('config_file')
        ),
        
        # Node for multi_cam_node
        Node(
            package='multi_cam',
            executable='multi_cam_node',
            #name='multi_cam_node',
            output='screen',
            parameters=[LaunchConfiguration('config_file')],
            # arguments=['--ros-args', '--log-level', 'DEBUG'],
            arguments=['--ros-args', '--log-level', 'INFO'],
            remappings=[],
            namespace='',
        ),
    ]
)

@Koichi98
I originally used ['--ros-args', '--log-level', 'DEBUG'] and the program crashed, after changing it to ['--ros-args', '--log-level', 'INFO'] the program worked well. I don't know why the log level will affect agnocast so I assumed it is a bug?

[Koichi98]

@allstarry
That sounds highly likely to be a timing bug... I know it might not be consistently reproducible, but if you are developing this with open source code, I would really appreciate it if you could share it. If possible, could you also provide the simple steps to reproduce it? I'll try it out myself as well.

[allstarry]

@allstarry That sounds highly likely to be a timing bug... I know it might not be consistently reproducible, but if you are developing this with open source code, I would really appreciate it if you could share it. If possible, could you also provide the simple steps to reproduce it? I'll try it out myself as well.

I'm sorry for responding so late. I have been busy recently and it took me some time to delete the code which relys on my hardware to make a simple test demo.
Here's the situation:
I have 3 workspace, they are agnocast, autoware_dependency and demo_ws. Agnocast is where I build agnocast code, and autoware_dependency is where I build autoware_agnocast_wrapper and its dependencies. Finally, demo_ws is where I build my demo code.

My environment:

~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.5 LTS
Release:	22.04
Codename:	jammy

$ uname -r
6.8.0-87-generic

Ros distro is humble and it was installed by apt

To reproduce the problem please follow the steps:
Download the attachment which is a compressed file and extract it under ~/ directory
agnocast_demo.zip

In ~/.bashrc, add

source /opt/ros/humble/setup.bash
export RMW_IMPLEMENTATION=rmw_fastrtps_cpp

build agnocast

cd ~/agnocast_demo/agnocast/agnocast
bash scripts/setup
colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release --parallel-workers 1
source install/setup.bash

build autoware_agnocast_wrapper

cd ~/agnocast_demo/autoware_dependency
colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release --parallel-workers 1
source install/setup.bash

build demo

export ENABLE_AGNOCAST=1
cd ~/agnocast_demo/demo_ws
colcon build --packages-select custom_msgs
source install/setup.bash
colcon build
source install/setup.bash

run demo

sudo modprobe agnocast
ros2 launch multi_cam multi_cam.launch.py

Normally, after running for less than 1 min, you may see the program crash.

[Koichi98]

@allstarry
Sorry for the late reply. It looks like the agnocast included in the agnocast_demo.zip you sent isn't tied to a specific tag, but is just the latest version from that time. Is that correct?

For agnocast, the kernel module and heaphook versions must match the API library version. Since scripts/setup installs the latest release, we need to make sure that we are using a specific tagged version of the agnocast API library as well.

In your case, you should get the agnocast API library as follows since you've installed agnocast-heaphook-v2.1.2 and agnocast-kmod-v2.1.2 through scripts/setup.

git clone -b 2.1.2 git@github.com:tier4/agnocast.git

Could you check it the problem still exists after you've done above?

[allstarry]

@allstarry Sorry for the late reply. It looks like the agnocast included in the agnocast_demo.zip you sent isn't tied to a specific tag, but is just the latest version from that time. Is that correct?

Correct.

For agnocast, the kernel module and heaphook versions must match the API library version. Since scripts/setup installs the latest release, we need to make sure that we are using a specific tagged version of the agnocast API library as well.

In your case, you should get the agnocast API library as follows since you've installed agnocast-heaphook-v2.1.2 and agnocast-kmod-v2.1.2 through scripts/setup.

git clone -b 2.1.2 git@github.com:tier4/agnocast.git

Could you check it the problem still exists after you've done above?

Problem still exists.
I just removed the whole agnocast folder and do
git clone -b 2.1.2 https://github.com/tier4/agnocast.git
Then I repeated the reproduction steps I listed above, and the program crashed again.