yield_on_full + small output buffer: false NoProgress loses data

[lilith]

Summary

When yield_on_full_buffer(true) is used with an output buffer smaller than a decoded code value, the Classic decoder returns NoProgress with consumed_in=0, consumed_out=0 while the internal buffer and bit buffer still contain decodable data. The caller interprets this as "done" and stops, silently losing the remaining output.

Reproduction

use weezl::{decode::{Configuration, TableStrategy}, encode, BitOrder, LzwStatus};

let data = vec![42u8; 8];
let encoded = encode::Encoder::new(BitOrder::Lsb, 8).encode(&data).unwrap();

let mut dec = Configuration::new(BitOrder::Lsb, 8)
    .with_yield_on_full_buffer(true)
    .with_table_strategy(TableStrategy::Classic)
    .build();

let mut result = Vec::new();
let mut inp = encoded.as_slice();
loop {
    let mut tmp = [0u8; 1]; // 1-byte output buffer
    let r = dec.decode_bytes(inp, &mut tmp);
    inp = &inp[r.consumed_in..];
    result.extend_from_slice(&tmp[..r.consumed_out]);
    match r.status {
        Ok(LzwStatus::Done) => break,
        Ok(LzwStatus::NoProgress) if r.consumed_in == 0 && r.consumed_out == 0 => break,
        Ok(_) => {}
        Err(_) => break,
    }
}
assert_eq!(result.len(), 8); // FAILS: result.len() == 1

Chunked has the same bug. KwKwK-heavy data (repeated single byte) triggers it most easily because every code is next_code, forcing the burst loop to exit after one code and route through the internal buffer.

Root cause

In DecodeState::advance(), when the internal buffer is empty at the start of a call, line 957 unconditionally sets status = NoProgress:

} else if remain.is_empty() {
    status = Ok(LzwStatus::NoProgress);
    have_yet_to_decode_data = true;
}

The burst loop then extracts a code from the bit buffer (which still has codes from an earlier greedy refill_bits) and writes it to the internal buffer (is_in_buffer = true). This produces consumed_out = 0 because data went to the internal buffer, not to out. At the end of advance(), the guard that corrects NoProgress:

if o_in > inp.len() {
    if let Ok(LzwStatus::NoProgress) = status {
        status = Ok(LzwStatus::Ok);
    }
}

...doesn't fire because o_in == inp.len() (no new input was consumed — the bits were already in the bit buffer from a previous call's greedy refill). So the false NoProgress propagates to the caller.

Suggested fixes

Option A: Fix the guard (correct for all buffer sizes)

At the end of advance(), also check whether data was written to the internal buffer:

let made_progress = o_in > inp.len()
    || !self.buffer.buffer().is_empty();
if made_progress {
    if let Ok(LzwStatus::NoProgress) = status {
        status = Ok(LzwStatus::Ok);
    }
}

This is safe because if the buffer has data, the next advance() call will drain it and produce consumed_out > 0.

Option B: Document minimum buffer size + debug_assert

LZW code values can be up to 3839 bytes long (KwKwK worst case with MAX_ENTRIES=4096). If the output buffer is always ≥ 4096 bytes, every code's output fits directly in out and the is_in_buffer path never triggers with yield_on_full.

This can be documented on with_yield_on_full_buffer:

/// # Buffer size requirement
///
/// When this option is enabled, the output buffer passed to
/// `decode_bytes` should be at least 4096 bytes. LZW codes can
/// decode to up to 3839 bytes; if the output buffer is smaller
/// than a code's decoded value, the Classic decoder may return
/// a spurious `NoProgress`.

Plus a debug assertion in advance():

debug_assert!(
    !CgC::YIELD_ON_FULL || out.len() >= MAX_ENTRIES,
    "yield_on_full requires output buffer >= {} bytes, got {}",
    MAX_ENTRIES, out.len()
);

image-tiff always passes strip-sized buffers (typically 8 KiB–1 MiB), so this constraint is already met in practice.

Note on Streaming strategy

The TableStrategy::Streaming decoder (#69) handles this correctly for all buffer sizes via its pending buffer — no minimum buffer requirement, no data loss. This bug only affects Classic and Chunked.

Impact

• TIFF is unaffected (strips are always large buffers).
• GIF is unaffected in practice (decoders use reasonably sized buffers).
• Any caller using yield_on_full_buffer(true) with small or 1-byte output buffers will silently truncate output. This includes custom Read::read adapters that forward small caller-provided buffers without internal buffering.

[lilith]

Fuzz reproducer found

While fuzzing the roundtrip_all target (encode → decode with all strategies → compare), we found a concrete reproducer for this issue:

Input: size=7, MSB, non-TIFF, yield_on_full=true, 111 bytes of repeating 0x4A with 0x7F runs.

Behavior: Both ByteLink and Streaming produce incorrect output. Bytes 99-108 of the decoded stream should be 74 (0x4A) but instead contain 127 (0x7F) — stale data from the preceding run of 0x7F values that wasn't properly flushed when the decoder yielded.

Reproduction: The failing test is in tests/streaming_parity.rs::regression_yield_on_full_fuzz_crash on the combined-strategies branch (commit 0567370). Fuzz regression seed saved to fuzz/regression/yield_on_full_bytelink_divergence_min.bin.

This is a pre-existing bug in the Classic/ByteLink burst decoder's yield_on_full path — it reproduces on upstream master as well. The reconstruct_simple overwrite optimization from the wuffs branch adds a second concern (8-byte aligned writes can clobber past code length), but the root cause is the burst loop's buffer management when yield_on_full causes early returns.

[lilith]

Performance measurement — no regression from the fix

Back-to-back best-of-3-passes bench on the wuffs branch, before vs after the buffer-drain fix. Same machine (Ryzen 9 7950X), load average ~10 (noisy but consistent between runs). Strategy_compare bench with fitted synthetic generators covering documents, photos, and flat-UI.

Workload	Before fix	After fix	Delta
tiff/scanned-email	2.51 GiB/s	2.55 GiB/s	+1.6%
tiff/scanned-blank	5.19 GiB/s	5.39 GiB/s	+3.9%
tiff/scanned-dense	1.01 GiB/s	1.03 GiB/s	+2.0%
tiff/scanned-mono-old	702 MiB/s	720 MiB/s	+2.6%
tiff/solid-kwkwk	16.8 GiB/s	16.5 GiB/s	-1.8%
tiff/photo-gray	536 MiB/s	559 MiB/s	+4.3%
tiff/photo-color	192 MiB/s	197 MiB/s	+2.6%
gif/scanned-email	2.58 GiB/s	2.65 GiB/s	+2.7%
gif/flat-ui	3.21 GiB/s	3.16 GiB/s	-1.6%

All deltas within ±4% (system noise). The fix touches only the per-decode_bytes() buffer drain path, not the inner burst loop. The reconstruct_simple slack guard (&& slack >= 8) adds one subtraction + compare per non-literal burst code — unmeasurable.

PRs:

• test: failing regression tests for yield_on_full data loss (#68) #71 — fix + tests on master (3/3 tests pass)
• fix: yield_on_full correctness on wuffs branch (#68) #72 — fix + tests on wuffs (3/3 tests pass, also fixes a debug_assert panic in TIFF mode with small buffers)