Closed
Conversation
It's cool feature that deserves to be documented!
Title. * Fixes conda-forge/rerun-sdk-feedstock#92 (comment) (or so I hope)
…12288) Co-authored-by: Gijs de Jong <14833076+oxkitsune@users.noreply.github.com>
### Related * Fixes small regression from #12246 ### What Fixes depth image visualizer view spawn heuristics having no preferred view kind set by explicitly setting `SpatialViewKind::TwoD`, so they are correctly classified as 2D again.
Should solve #12295 Related to #12208 Looks like it doesn't break #12193 I believe our updated rounding logic avoids the accumulation of floating point errors over time which was the original issue. It looks like forcing the new plot x was enforcing the absolute time instead of encoding all our different view variants. We could probably keep the float offset for precision and have a match statement but I don't know this area well enough without doing more digging. https://github.com/user-attachments/assets/3c26b8e9-57ec-49bb-8598-194b91107aa6
When we introduced #12220 the recording-stream cleanup logic had unexpected side-effects for some users of grpc_server sink. See: #12301 Although this change in behavior is subtle, it's correct given the current architecture of the RecordingStream and Sink ownership model. I've created a future issue for making it possible to avoid this by changing that ownership model rather than introducing new edge-cases in the cleanup code-paths: #12313
## Why?
We are facing two distinct problems when dealing with sliced chunks:
1. Their size on disk is much larger than it should be, owing to the
fact that data that is not part of the slice still gets serialized along
with it.
1. Their reported physical in-memory size is also much larger than it
should be, leading to the chunk splitter in the `ChunkStore` splitting
chunks way more than it ever should.
The two issues, while distinct, are very related and together form a
feedback loop that effectively turns the chunk splitter into a zip-bomb
(well, an RRD-bomb).
Both of these issues vary in intensity depending on the datatype and
nested-ness of the data (because it all has to do with offsets in the
end).
### Issue 1: too much data on disk
The core of issue number 1 is that `Array::slice` is shallow, by which
we really mean that it is guaranteed to run in O(1)-ish time. It
achieves this by never actually physically slicing data, because doing
so would require rewriting all the offsets that point at that data in
the first place.
And since it doesn't patch the offsets, it must also serialize the
entirety of the original data when writing to disk, not just the sliced
part, else the offsets wouldn't make sense anymore.
Looking at the low-level slicing implementation in `ArrayData` gives a
clear picture:
```rust
pub fn slice(&self, offset: usize, length: usize) -> ArrayData {
if let DataType::Struct(_) = self.data_type() {
let new_offset = self.offset + offset;
ArrayData {
data_type: self.data_type().clone(),
len: length,
offset: new_offset,
buffers: self.buffers.clone(),
child_data: self
.child_data()
.iter()
.map(|data| data.slice(offset, length))
.collect(),
nulls: self.nulls.as_ref().map(|x| x.slice(offset, length)),
}
} else {
let mut new_data = self.clone();
new_data.len = length;
new_data.offset = offset + self.offset;
new_data.nulls = self.nulls.as_ref().map(|x| x.slice(offset, length));
new_data
}
}
```
Note how all the buffers are cloned as-is while the offsets, although
sliced, are left untouched.
To make matters more confusing, there seem to exist a few datatype
specific exceptions. Consider a pretty simple `list[i32]` for example:
```
array[0..]: len= 100000 buf_size= 800004 data.slice_size= 800000 / IPC=825544
slice[25000..75000]: len= 50000 buf_size= 800004 data.slice_size= 600000 / IPC=413000
deep[25000..75000]: len= 50000 buf_size= 400064 data.slice_size= 400000 / IPC=413000
```
Note how the reported in-memory sliced size (`data.slice_size`) for the
shallow slice says `600000`, and yet somehow it ends up at `413000` once
serialized to IPC. It seems that, in this specific case, the IPC writer
is rewriting the offsets just-in-time in order to properly pack the
data.
This is not always the case though, consider e.g. a
`union#dense{u8,f32,i64}`:
```
array[0..]: len= 100000 buf_size= 1000008 data.slice_size= 1000008 / IPC=1021832
slice[25000..75000]: len= 50000 buf_size= 1000008 data.slice_size= 750008 / IPC=771848
deep[25000..75000]: len= 50000 buf_size= 900096 data.slice_size= 466670 / IPC=473864
```
Here we can see that the in-memory slice size reported for the shallow
slice does match what eventually gets serialized to disk, except it's
way more than it should be. Presumably because neither the shallow
slicer nor the IPC writer make any attempt at removing unnecessary
values from the underlying dense union arrays. Which makes sense,
re-packing the underlying union arrays is a pretty heavy operation.
Anyway, what's important is that is this more of a fundamental design
limitation rather than a bug: if you're not willing to hot patch
offsets, then you cannot possibly serialize just the data you need.
Therefore, we need a proper deep-slice implementation no matter what.
### Issue 2: reported physical in-memory size is completely off
The second issue, I believe, has more to do with bugs than fundamental
design decisions. Indeed, I cannot think of any reason why
[`ArrayData::get_slice_memory_size`](https://docs.rs/arrow/latest/arrow/array/struct.ArrayData.html#method.get_slice_memory_size)
couldn't be made to work properly with all kinds of slicing setups.
Today, though, it seems to struggle a lot with anything that makes use
of offsets (which is basically everything besides basic primitive
arrays).
It is trivial to end up with extreme discrepancies by stacking multiple
layers of offsets on top of each other. Here's a `list[list[i32]]`:
```
array[0..]: len= 100000 buf_size= 1200008 data.slice_size= 1200000 / IPC=1238280
slice[ 5000.. 5005]: len= 5 buf_size= 1200008 data.slice_size= 800020 / IPC= 904
deep[ 5000.. 5005]: len= 5 buf_size= 192 data.slice_size= 60 / IPC= 904
```
Note how far off the reported slice size is for the shallow case.
## What
This PR implements a slow but reliable `deep_slice_array` routine.
It works similar to `concat` in that it just allocates and copies what
it needs. While technically much slower that it should be, this is A)
likely not gonna matter in practice (`concat()` performs fine, after
all) and, more importantly, B) this will serve as a baseline if and when
we decide to optimize.
The entire implementation fits in a few lines, which literally just
recreates an array from the ground up with just the desired slice of
data, packed as much as possible.
```rust
/// Deep-slicing operation for Arrow arrays.
///
/// The data, offsets, bitmaps and any other buffers required will be reallocated, copied around, and patched
/// as much as required so that the resulting physical data becomes as packed as possible for the desired slice.
pub fn deep_slice_array<T: Array + From<ArrayData>>(array: &T, offset: usize, length: usize) -> T {
let data = array.to_data();
let use_null_optimization = false;
let mut data_sliced =
arrow::array::MutableArrayData::new(vec![&data], use_null_optimization, length);
data_sliced.extend(0, offset, offset + length);
T::from(data_sliced.freeze())
}
```
For now, this puts us in a stable state so that we can ship a patch
release and breathe a little.
This routine is exposed via a new `Chunk::row_sliced_deep` method. The
old `Chunk::row_sliced` has been renamed to `Chunk::row_sliced_shallow`:
```rust
/// Shallow-slices the [`Chunk`] vertically.
///
/// The result is a new [`Chunk`] with the same columns and (potentially) less rows.
///
/// This cannot fail nor panic: `index` and `len` will be capped so that they cannot
/// run out of bounds.
/// This can result in an empty [`Chunk`] being returned if the slice is completely OOB.
///
/// WARNING: the returned chunk has the same old [`crate::ChunkId`]! Change it with [`Self::with_id`].
///
/// ## When to use shallow vs. deep slicing?
///
/// This operation is shallow and therefore always O(1), which implicitly means that it cannot
/// ever modify the values of the offsets themselves.
/// Since the offsets are left untouched, the original unsliced data must always be kept around
/// too, _even if the sliced data were to be written to disk_.
/// Similarly, the sizes reported might not always make intuitive sense, and should be used
/// very carefully.
///
/// For these reasons, shallow slicing should only be used in the context of short-term, in-memory storage
/// (e.g. when slicing the results of a query).
/// When slicing data for long-term storage, whether in-memory or on disk, see [`Self::row_sliced_deep`] instead.
fn row_sliced_shallow(&self, index: usize, len: usize) -> Self;
/// Deep-slices the [`Chunk`] vertically.
///
/// The result is a new [`Chunk`] with the same columns and (potentially) less rows.
///
/// This cannot fail nor panic: `index` and `len` will be capped so that they cannot
/// run out of bounds.
/// This can result in an empty [`Chunk`] being returned if the slice is completely OOB.
///
/// WARNING: the returned chunk has the same old [`crate::ChunkId`]! Change it with [`Self::with_id`].
///
/// ## When to use shallow vs. deep slicing?
///
/// This operation is deep and therefore always O(N).
///
/// The underlying data, offsets, bitmaps and other buffers required will be reallocated, copied around,
/// and patched as much as required so that the resulting physical data becomes as packed as possible for
/// the desired slice.
/// Similarly, the reported sizes would always match intuitive expectations.
///
/// These characteristics make deep slicing very useful for longer term data, whether it's
/// stored in-memory (e.g. in a `ChunkStore`), or on disk.
/// When slicing data for short-term needs (e.g. slicing the results of a query), whether in-memory or on
/// disk, prefer [`Self::row_sliced_shallow`] instead.
#[must_use]
fn row_sliced_deep(&self, index: usize, len: usize) -> Self
```
## What's next
There are definitely ways to make deep-slicing faster (we should only
ever need to copy/reallocate/patch offsets), and I believe
`get_slice_memory_size` could also be implemented in a way that works
with all possible slicing setups.
In fact, I wonder if deep-slicing could be avoided entirely by
hot-patching everything as needed during serialization. Again, hard to
no whether there are design limitations preventing that without digging
first.
Because these things are tricky, take time to get right and even more
time to test properly, this PR instead focuses on providing a provably
correct baseline that will always return the correct size and never
writes one too many byte on disk. Optimizations can follow later.
---
* #12306 is an example of a PR
that implements such optimizations, but [further
testing](#12306 (comment))
has shown that its results differ in some cases at the moment. We shall
come back to it.
* Fixes #12304
* Fixes #12305
---
```python
import rerun as rr
import numpy as np
import time
rr.init("testing")
rr.save("wtf.rrd")
for i in range(5000):
rr.set_time("local_ts_us", timestamp=np.datetime64(int(time.time() * 1e6), "us"))
x = np.random.randint(0, 50000, size=1024)
rr.log("test/tensors", rr.Tensor(x))
```
Now compacts as expected:
```
pixi run rerun rrd compact wtf.rrd > /dev/null
✨ Pixi task (rerun in default): cargo run --package rerun-cli --no-default-features --features release_no_web_viewer -- rrd compact wtf.rrd
Finished `dev` profile [optimized] target(s) in 0.54s
Running `target/debug/rerun rrd compact wtf.rrd`
[2026-01-05T16:35:22Z DEBUG re_analytics::native::pipeline] Analytics disabled in debug builds
[2026-01-05T16:35:22Z DEBUG rerun::commands::entrypoint] Detected 32 cores. Using 30 compute threads.
[2026-01-05T16:35:22Z INFO rerun::commands::rrd::merge_compact] merge/compaction started max_rows=4 096 max_rows_if_unsorted=1 024 max_bytes=384 KiB srcs=["wtf.rrd"]
[2026-01-05T16:35:22Z INFO rerun::commands::rrd::merge_compact] processing input…
[2026-01-05T16:35:22Z DEBUG re_sorbet::migrations] Encountered batch without 'sorbet:version' metadata.
[2026-01-05T16:35:22Z DEBUG re_sorbet::migrations] Performing migrations from 0.0.1…
[2026-01-05T16:35:22Z DEBUG re_sorbet::migrations] Migrating record batch from Sorbet 'v0.0.1' to 'v0.0.2'.
[2026-01-05T16:35:22Z DEBUG re_sorbet::migrations] Migrating record batch from Sorbet 'v0.0.1' to 'v0.1.0'.
[2026-01-05T16:35:22Z DEBUG re_sorbet::migrations] Migrating record batch from Sorbet 'v0.0.1' to 'v0.1.1'.
[2026-01-05T16:35:22Z DEBUG re_sorbet::migrations] Migrating record batch from Sorbet 'v0.0.1' to 'v0.1.2'.
[2026-01-05T16:35:22Z INFO rerun::commands::rrd::merge_compact] running extra compaction pass… pass=0
[2026-01-05T16:35:22Z INFO rerun::commands::rrd::merge_compact] extra compaction pass completed pass=0 num_chunks_before=122 num_chunks_after=122 num_chunks_reduction="-0.000%" time=798.954µs
[2026-01-05T16:35:22Z INFO rerun::commands::rrd::merge_compact] cannot possibly improve further, stopping early pass=0 time=808.384µs
[2026-01-05T16:35:22Z INFO rerun::commands::rrd::merge_compact] preparing output…
[2026-01-05T16:35:22Z INFO rerun::commands::rrd::merge_compact] encoding…
[2026-01-05T16:35:22Z INFO rerun::commands::rrd::merge_compact] merge/compaction finished srcs=["wtf.rrd"] time=153.255563ms num_chunks_before=42 num_chunks_after=122 num_chunks_reduction="--190.476%" srcs_size_bytes=20.0 MiB dst_size_bytes=20.2 MiB size_reduction="--0.754%"
```
---------
Co-authored-by: ntjohnson1 <24689722+ntjohnson1@users.noreply.github.com>
### Related * Closes #5251 ### What This implements a mesh loader for Collada (`.dae`) scenes, commonly used with URDF.  Things still left: - [x] Load materials - [x] Load texcoords and use textures - [x] Evaluate `collada-rs` crate over `dae-parser`, I picked `dae-parser` as it seems more feature complete. - [x] Test, test, test - [x] ~~Make sure `up_axis` tag is respected~~ #12335
rerun-bot
added
⛴ release
* Picked from: #12292 Title.
### What Fixes the `animated_urdf` Python example spawning the viewer on CI
grtlr
force-pushed
the
prepare-release-0.28.2
branch
from
06ded74 to
cfff106
Compare
Fixed a small typo in the documentation code example. The parameter name was wrong which might confuse new users.
Try to fix nightly link checker errors.
Collaborator
Author
Requested by @nikolausWest Here is the new option: <img width="315" height="263" alt="image" src="https://github.com/user-attachments/assets/83599e28-beed-4f0d-b96f-2937f8b1e3f0" /> Tested. Works.
This reverts commit e058250.
…ets" This reverts commit 1b60109.
Collaborator
Author
grtlr
force-pushed
the
prepare-release-0.28.2
branch
from
e84b7ff to
38a92e2
Compare
grtlr
force-pushed
the
prepare-release-0.28.2
branch
from
e3fbc35 to
660c08f
Compare
Collaborator
Author
Collaborator
Author
|
GitHub release draft: 0.28.2 Add a description, changelog, and a nice header video/picture, then click 'Publish release'. |
Member
|
I've cherry picked all relevant changes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Next steps
Test the release
For alpha releases:
DISCLAIMER: This is an unreviewed, automatically generated changelog. We only provide fully reviewed changelogs for final releases.
main, but not the version bump one (it would introduce bad links).For non-alpha releases:
main, then run the release workflow in 'release' modeSet as latest releaseis checkedPublish releaseTests