Regression [BUG] LiteDB.LiteException: ReadFull must read PAGE_SIZE bytes [{0}]

[michelmoorlag]

Version
On 5.0.20 everything works fine but when I upgrade to 5.0.21 I can't save anything

Describe the bug
When I try this code:

var collection = _liteDatabaseAsync.GetCollection<T>();
return collection.UpsertAsync(entity);

it will throw this exception: {LiteDB.Async.LiteAsyncException: LiteDB encounter an error. Details in the inner exception. ---> LiteDB.LiteException: ReadFull must read PAGE_SIZE bytes [{0}]

Code to Reproduce
I have a repositry that implements this code that used to run fine:

public Task SaveAsync<T>(T entity) where T : DataObject
{
    try
    {
        var collection = _liteDatabaseAsync.GetCollection<T>();
        return collection.UpsertAsync(entity);
    }
    catch (Exception ex)
    {
        Debug.WriteLine(ex);
        throw;
    }
}

where _liteDatabaseAsync is of type: ILiteDatabaseAsync created like this
_liteDatabaseAsync = new LiteDatabaseAsync(_connectionString);

Expected behavior
I expect that the new versions are backwards comaptible and that above code inserts or updates the given entity without exception.

Screenshots/Stacktrace

{LiteDB.Async.LiteAsyncException: LiteDB encounter an error. Details in the inner exception. ---> LiteDB.LiteException: ReadFull must read PAGE_SIZE bytes [{0}]
  at LiteDB.Constants.ENSURE (System.Boolean conditional, System.String format, System.Object[] args) [0x00022] in <74012b958fcc45cfa90d6855b44c81c5>:0 
  at LiteDB.Engine.DiskService+<ReadFull>d__23.MoveNext () [0x000b5] in <74012b958fcc45cfa90d6855b44c81c5>:0 
  at LiteDB.Engine.WalIndexService+<>c__DisplayClass19_0+<<CheckpointInternal>g__source|0>d.MoveNext () [0x000d5] in <74012b958fcc45cfa90d6855b44c81c5>:0 
  at LiteDB.Engine.DiskService.WriteDataDisk (System.Collections.Generic.IEnumerable`1[T] pages) [0x0006e] in <74012b958fcc45cfa90d6855b44c81c5>:0 
  at LiteDB.Engine.WalIndexService.CheckpointInternal () [0x00020] in <74012b958fcc45cfa90d6855b44c81c5>:0 
  at LiteDB.Engine.WalIndexService.TryCheckpoint () [0x0002f] in <74012b958fcc45cfa90d6855b44c81c5>:0 
  at LiteDB.Engine.LiteEngine.CommitAndReleaseTransaction (LiteDB.Engine.TransactionService transaction) [0x0004a] in <74012b958fcc45cfa90d6855b44c81c5>:0 
  at LiteDB.Engine.LiteEngine.AutoTransaction[T] (System.Func`2[T,TResult] fn) [0x00050] in <74012b958fcc45cfa90d6855b44c81c5>:0 
  at LiteDB.Engine.LiteEngine.Upsert (System.String collection, System.Collections.Generic.IEnumerable`1[T] docs, LiteDB.BsonAutoId autoId) [0x0004d] in <74012b958fcc45cfa90d6855b44c81c5>:0 
  at LiteDB.LiteCollection`1[T].Upsert (System.Collections.Generic.IEnumerable`1[T] entities) [0x00021] in <74012b958fcc45cfa90d6855b44c81c5>:0 
  at LiteDB.LiteCollection`1[T].Upsert (T entity) [0x00013] in <74012b958fcc45cfa90d6855b44c81c5>:0 
  at LiteDB.Async.LiteCollectionAsync`1+<>c__DisplayClass55_0[T].<UpsertAsync>b__0 () [0x0000b] in <c9f9109ef3444de596197a6ad09e3854>:0 
  at LiteDB.Async.LiteDatabaseAsync+<>c__DisplayClass37_0`1[T].<EnqueueAsync>g__Function|0 () [0x00000] in <c9f9109ef3444de596197a6ad09e3854>:0 
   --- End of inner exception stack trace ---
  at PSExampleApp.Core.Services.MeasurementService.SaveMeasurements () [0x00045] in C:\Repos\mux16app\PSExampleApp.Core\Services\MeasurementService.cs:303 }

[a38782615]

mark

[NongBenz]

Same 5.0.20 works but 5.0.21 gives this error.

[fabiorme]

I also have the same problem, I updated from version 5.0.17 to 5.0.21 and I encounter several problems with deleting and updating collections.

var col = db.GetCollection();
col.DeleteMany(x => x.Stato == StatoTavolo.Libero);

var document = BsonMapper.Global.ToDocument(item);
var col = db.GetCollection(type.Name);
col.Upsert(document);

Error:
LiteDB.LiteException: 'ReadFull must read PAGE_SIZE bytes [{0}]'

[Blue101black]

I can consistently get the error by spamming the following, two FindById calls on two seperate LiteDb files and then an UpsertCall on one of the LiteDb files.

Doesn't happen in 5.0.20

5.0.21 seems to have a dependency on System.Buffer, maybe the issue?

[JKamsker]

@Blue101black Can you please provide a repro?

[Blue101black]

@JKamsker I cannot provide a repo sorry, but I can give some details.

Using Xamarin Forms (4.8.0.1687) as a hybrid app with a webview that runs Vue.js. Only build/run the app on Android (We should definitely upgrade to .Net Android)

I communicate back to C# land from Vue via Javascript interface on the WebView. I have a lock in place to make sure only one request can take place at a time.

Spamming a UI button that sends a request and does the below, interacting with two LiteDb instances.
Here is an example of the code (renaming things randomly for privacy):

public Item AddItem(Guid parentId)
{
    var newItem = new Item()
    {
        Id = Guid.NewGuid(),
        ParentId = parentId,
        Quantity = 1,
        RecordedAt = _recordTimeService.GetCurrentDateTimeOffset()
    };

    _database.Upsert<Item>(newItem);

    return newItem;
}

GetOrDefault just a wrapper around FindById

public T GetOrDefault<T>(Guid id) where T : class, IEntity, new()
{
    return GetCollection<T>().FindById(id);
}

public DateTimeOffset GetCurrentDateTimeOffset()
{
    var timeItem = _database.GetOrDefault<TimeItem>(_state.CurrentRecordId);
    if (timeItem == null || !timeItem.ShiftId.HasValue || !timeItem.StartAt.HasValue)
    {
        return _timeService.DateTimeOffsetNow;
    }

    var shift = _database2.GetOrDefault<Shift>(timeItem.ShiftId.Value);
    if (shift == null || !shift.StartTime.HasValue || !shift.EndTime.HasValue)
    {
        return _timeService.DateTimeOffsetNow;
    }

    var timeDate = (DateTimeOffset)timeItem.StartAt;
    var currentDate = _timeService.DateTimeOffsetNow;
    var time = new TimeSpan(currentDate.Hour, currentDate.Minute, currentDate.Second);

    // Shift spans over midnight
    if (shift.StartTime > shift.EndTime)
    {
        // Time is outside shift bounds.
        if (time < shift.StartTime && time > shift.EndTime)
        {
            time = (TimeSpan)shift.StartTime;
            return new DateTimeOffset(timeDate.Year, timeDate.Month, timeDate.Day, time.Hours, time.Minutes, time.Seconds, timeDate.Offset);
        }
        // Time is between midnight and endTime
        else if (time < shift.EndTime && time > new TimeSpan(0, 0, 0))
        {
            var newDate = timeDate.AddDays(1);
            return new DateTimeOffset(newDate.Year, newDate.Month, newDate.Day, time.Hours, time.Minutes, time.Seconds, newDate.Offset);
        }
    }
    // Time is outside shift bounds.
    else if (time < shift.StartTime || time > shift.EndTime)
    {
        time = (TimeSpan)shift.StartTime;
        return new DateTimeOffset(timeDate.Year, timeDate.Month, timeDate.Day, time.Hours, time.Minutes, time.Seconds, timeDate.Offset);
    }

    // Time is within shift bounds.
    return new DateTimeOffset(timeDate.Year, timeDate.Month, timeDate.Day, time.Hours, time.Minutes, time.Seconds, timeDate.Offset);
}

[nexnmoon]

DiskService.cs file...
public int WriteLogDisk(IEnumerable pages)

Shouldn't there be a stream.FlushToDisk(); line before the function returns?

[XLittleLeft]

same

[timothyparez]

Same here.

I had a working solution on 5.0.11 and upgraded to 5.0.21.
This resulted in LIteDB ReadFull must read PAGE_SIZE bytes

I then downgraded to 5.0.20 and everything worked again.

From this it looks as if it was fixed
in 5.0.17, 5.0.18, 5.0.19 or 5.0.20 and re-introduced in 5.0.21.

[Terricide]

I asked Claude about this issue and it suggested this fix

The issue is in the error check. When reading from a stream, there's no guarantee that Read will fill the entire buffer in one call, especially at the end of the file.
Here's the proposed fix:

public IEnumerable ReadFull(FileOrigin origin)
{
// do not use MemoryCache factory - reuse same buffer array (one page per time)
// do not use BufferPool because header page can't be shared (byte[] is used inside page return)
var buffer = new byte[PAGE_SIZE];

var pool = origin == FileOrigin.Log ? _logPool : _dataPool;
var stream = pool.Rent();

try
{
    // get length before starts (avoid grow during loop)
    var length = this.GetFileLength(origin);

    stream.Position = 0;

    while (stream.Position < length)
    {
        var position = stream.Position;
        var totalBytesRead = 0;
        
        // Read until we've filled the buffer or reached EOF
        while (totalBytesRead < PAGE_SIZE)
        {
            var bytesRead = stream.Read(buffer, totalBytesRead, PAGE_SIZE - totalBytesRead);
            
            // If no bytes were read, we've reached EOF
            if (bytesRead == 0) break;
            
            totalBytesRead += bytesRead;
        }
        
        // Check if we read a full page
        ENSURE(totalBytesRead == PAGE_SIZE, "ReadFull must read PAGE_SIZE bytes [{0}]", totalBytesRead);

        yield return new PageBuffer(buffer, 0, 0)
        {
            Position = position,
            Origin = origin,
            ShareCounter = 0
        };
    }
}
finally
{
    pool.Return(stream);
}

}
This change ensures that multiple Read operations are attempted to fill the buffer completely before throwing an error. This is a common pattern when dealing with stream reads, as the Stream.Read method doesn't guarantee that it will read the requested number of bytes in a single call.

[hackerzhuli]

Same, any progress? It happens when I use bool ILiteCollection<T>.Update(T) to update a document. But the exception doesn't happen all the time, sometimes it does, sometimes it doesn't. It seems to have to do with how many times I run update. I am generating and saving data to a database, it will update the document in the database once in a few minutes. It seems to error frequently after I run my generators for about a few hours.

[timothyparez]

If this doesn't get fixed I'm migrating back to sqlite-net

[isbdnt1]

This PR fixes the bug.
#2717