Question about SpikeRecorder

[neworderofjamie]

This definitely sounds like a bug - could you share your code with me? Either here or with J.C.Knight@sussex.ac.uk (although this email account, helpfully, blocks Python attachments so you'll need to rename the script to .txt or something).

Discussed in https://github.com/orgs/genn-team/discussions/668

^{Originally posted by OOOOOmoi April 8, 2025}
Hello, I'm using GeNN to simulate a variant of the Potjans model containing three types of inhibitory neurons, with more diverse weights between each neuronal population. When I try to use the GeNNModel.pull_recording_buffers_from_device() method to record neuronal spikes in the network like this:

model.load(num_recording_timesteps=args.buffer_size)
spike_data = {n: [] for n in neuron_populations.keys()}
while model.t < duration:
    # Advance simulation
    print(model.timestep)
    model.step_time()
    # Indicate every 10%
    if not model.timestep % args.buffer_size:
        model.pull_recording_buffers_from_device()
        for n, pop in neuron_populations.items():
            spike_times, spike_ids = pop.spike_recording_data[0]
            spike_data[n].append(np.column_stack((spike_times, spike_ids)))
    if (model.timestep % ten_percent_timestep) == 0:
        print("%u%%" % (model.timestep / 100))

I found the program couldn't run properly and would always get stuck at pull_recording_buffers_from_device(), I've already enabled spike recording when creating neurons using neuron_pop.spike_recording_enabled = True. I also tried reducing the number of neurons and synapses, but that didn't work. What's puzzling is that when I don't create any synapses, set all synaptic weights to zero, or make the weights between different neuronal populations follow the same mean and variance, the program runs normally. Where do you think the problem might lie?
By the way, I encountered this same issue when using GeNN 4.X before. At that time, the program also got stuck at pull_recording_buffers_from_device(), but I didn't pay much attention to it - I directly modified the generated C++ code and makefile to bypass the SpikeRecorder.

[OOOOOmoi]

You can clone my code from https://github.com/OOOOOmoi/PyGeNNProject.git. The main script is located at /SingleColumn/SingleColumn.py, which is written following the PyGeNN documentation at https://genn-team.github.io/genn/documentation/5/userproject/potjans_microcircuit.html.

In lines 178–189, I defined three different synaptic weight distributions. You can selectively comment them out—currently, I have set the weights to zero. The default recording buffer size is 100 timesteps, but you can adjust it by passing the --buffer-size argument when running the code.

If you set the weights to specialized (non-zero) distributions, the code will halt when it reaches the same number of timesteps as the recording buffer size.

Unfortunately, I've already deleted the C++ version since it was too outdated. I only recalled this issue when working with the Python version.

[saadkhi]

Hey @neworderofjamie , I want to work on this can you brief me so that i can follow up that.

[neworderofjamie]

I have done some initial investigating and I think this is actually the same issue as https://github.com/orgs/genn-team/discussions/671. The reason you are seeing the freeze here is only because this is the first CUDA synchronisation point rather than because it's anything to do with spike recording. If you turn on kernel profiling (which synchronises more frequently), you can see this the freeze happens here:

i.e. at the first synchronisation point after the sparse initialisation kernel.

[neworderofjamie]

Then, inspecting what the CUDA threads are doing while the host is waiting using the CUDA debugger:

so this thread is initialising a weight with NormalClipped and re-sampling until it's negative. However, the group->meang is 2.63809061 and group->sdg is 0.06595226 which is pretty unlike to produce a negative number. I think your problem is just that your synaptic weight distributions are not being initialised correctly as e.g.

 Connection 'E5' to 'H1': numConnections=9218, meanWeight=1.028855, weightSD=0.102886, meanDelay=0.750000, delaySD=0.3750

is treated as an inhibitory connection. I think the fixes are simple:

1. On line 10, make G positive i.e. G = 4 so it doesn't make inhibitory weights positive
2. Changing your tests for excitatory connections on lines 193 and 215 to if src_pop.startswith("E")

After making these fixes I get:

[OOOOOmoi]

Well, I think these are actually two different issues. Sorry I forgot to mention earlier - the parameter G in the code was designed for the second and third weight configurations. If you're using the first weight configuration, you need to set G=1, like this:

Because the synapseWeight.txt file already contains negative weights. Moreover, the code isn't getting stuck in initializeSparse() - the project can successfully complete initializeSparse(). The default buffer-size is 100, and I've made sure to output the current timestep before every step_time() call:

But the code consistently freezes when reaching the 99th timestep. The same issue occurs if I set buffer-size to other values.

[neworderofjamie]

If you turn on kernel profiling, the freeze will occur earlier though - I fully reproduced and debugged this so pretty confident here! Can you try my second fix of correcting your excitatory population test?

[OOOOOmoi]

OK，I understand the question, thanks.