uBlock appears to be unable to update filters in Firefox's private mode

[UltramarineBlue0]

Prerequisites

• I verified that this is not a filter issue
  • Filter issues MUST be reported at filter issue tracker
• This is not a support issue or a question
  • Support issues and questions are handled at /r/uBlockOrigin
• I performed a cursory search of the issue tracker to avoid opening a duplicate issue
  • Your issue may already be reported.
• I tried to reproduce the issue when...
  • uBlock Origin is the only extension
  • uBlock Origin with default lists/settings
  • using a new, unmodified browser profile
• I am running the latest version of uBlock Origin
• I checked the documentation to understand that the issue I report is not a normal behavior

Description

While uBO can't access indexedDB in Firefox's permanent private mode is a known issue, it appears that it can't update the filters in private mode as well. Updating using the "Update now" button doesn't change the filter count. (I'm not sure if it's a display issue or if no filters are actually loaded)

A specific URL where the issue occurs

[A specific URL is MANDATORY for issue happening on a web page, even if it happens "everywhere"]

Steps to Reproduce

1. Start Firefox normally (not in private mode) and make sure the filters are up to date
2. Close Firefox then restart Firefox with the command line options: '-no-remote -private'
3. Go to uBlock Origin Setting and click "Update now"

Expected behavior:

The filters are initially all out of date. After updating the filters, the yellow triangles disappear and the filter count changes because newer version of the filters are loaded. The filter count should be roughly equal the filter count in Firefox normal mode (before the restart with extra command line options)

Actual behavior:

The filters are initially all out of date. After updating the filters, the yellow triangles disappear. However neither the network filter count nor the cosmetic filter count change, giving the appearance as if none of the filters are updated. The filter count is also significantly lower than in Firefox normal mode (before the restart with extra command line options)

Your environment

• uBlock Origin version: 1.18.2 (however this is happening for at least a few months)
• Browser Name and version: Firefox 60.5.0 ESR on Win10 && Firefox 64 on Linux
• Operating System and version: Windows 10 Update 1803 && Lubuntu 18.04.1

[uBlock-user]

Duplicate of gorhill/uBlock#3071

[UltramarineBlue0]

I'm aware of that issue but I might have misunderstood it. I thought it was only about the fact that new versions of filter cannot be saved and as a result ublock is forced to load the packaged filters when starting firefox in permanent private mode. My issue is that even after clicking "Update now" the filter count doesn't change. Is this the expected behavior?

[uBlock-user]

Is this the expected behavior?

Yes, part of the issue amoung others(like storage appearing 0 bytes.) Don't use private browsing mode until the main issue is resolved by Firefox devs.

[gorhill]

uBO is not allowed to save the lists it downloads, and as a result always ends up loading the list from the package when it tries to reload all the filter lists after the lists have been "updated".

[UltramarineBlue0]

Is it reasonable / feasible to add a workaround for private mode that only loads the updated filters into memory without trying to save them when you click "Update now"?

[gorhill]

Caching in memory sounds abusive.

Another idea -- which I entertained when I fixed #399 -- is to allow uBO to fallback using browser.storage.local on Firefox for caching purpose, just like it is with Chromium.

There is a new hidden setting which allows a user to force a specific storage implementation for cache purpose, currently looked up only on non-Firefox browsers. I could add support for a webext value, to force uBO to use browser.storage.local. My understanding is that this storage is not affected by private mode.

An issue on Firefox though is that it does not declare the unlimitedStorage in its manifest, and I wonder if there will be enough storage space for uBO to save all its cached data.

[gorhill]

Of course, it turns out it's not that simple. uBO needs to look-up which storage API to use early as its JS is executed. To do so it stores the cacheStorageAPI value in localStorage, which can synchronously return the set value. However, in private mode, localStorage is always cleared, meaning settting cacheStorageAPI to browser.storage.local accomplishes nothing in the end. The only way to fix this is for the cache storage API to be become available only in an asynchronous manner, which is a non-trivial change to the code.