Expose (kind of) `InternalServer` in the public API ?

[mgautierfr]

The server feature in libkiwix does several things:

• Listen for http request (using libmicrohttpd) and convert it to a RequestContext
• Parse the request to understand what user want.
• Find the correct books/Archive/Searcher in the library and "resolve" the user request.
• Build a Response
• Convert the Response to a libmicrohttpd's response and send back the data to the user.

The first and last step is pretty tied to a http server but the other points are pretty common and it is somehow what is made in all applications.
In kiwix-desktop, we span some kind of internal server and answer to QWebEngine requests (https://github.com/kiwix/kiwix-desktop/blob/master/src/urlschemehandler.cpp)
This is probably what is done in kiwix-android or macos application.

It would be a nice thing to move all the handling in InternalServer as a first class citizen (libmicrohttpd neutral) in libkiwix API.
Then we could :

• Unit test the "request handler" without launching a temporary server
• Create the server on top of this "request handler" (and keep the current testing system here)
• Use the "request handler" in kiwix-desktop

Some open questions or points to keep in mind:

• Should we enforce a url schema or be configurable (kiwix-desktop uses a different schema than kiwix-serve)
• I'm convinced (but you can counter-argument) that we should move to architecture where we serve "raw" content in a zim file and we add chrome/gui around using different technologies (Qt Widgets in kiwix-desktop, iframe/js in kiwix-serve, ....) With this idea, the "request handler" should handle "raw" content (I include search and co) and the "http server" should add the Chrome.

At your comments :)

[kelson42]

First impression on my side: sounds a good idea, but I have a really superficial understanding of all what this concretly means. Forcing URL schema does not seem a bad idea either. Probably the feedback from @veloman-yunkan here is far more valuable than mine :)

[mossroy]

This looks interesting, but would most probably not be usable in kiwix-js.

In kiwix-js, we do such thing in a ServiceWorker (which traps the user HTTP requests). But a ServiceWorker is very isolated, and notably can not access the ZIM file.
So we would not be able to use it as a one-chunk emscripten-compiled C code, that handles the requests from A to Z.

[mgautierfr]

But a ServiceWorker is very isolated, and notably can not access the ZIM file.

How do you respond to the user request then ? You "transfer" the request to something that can access the ZIM file ?

The idea of this issue is to separate the front request handling (parsing the incoming request to know what to do) from the back (doing what is requested).
If you transfer the request for the SW (the front) to something that can access zim files (the back), you could transfer something like a kiwix::Request (created in the SW by parsing the HTTP request).

[mossroy]

How do you respond to the user request then ? You "transfer" the request to something that can access the ZIM file ?

Exactly. We transfer it through a MessageChannel to the main thread, and transfer the result back to the ServiceWorker.

The idea of this issue is to separate the front request handling (parsing the incoming request to know what to do) from the back (doing what is requested). If you transfer the request for the SW (the front) to something that can access zim files (the back), you could transfer something like a kiwix::Request (created in the SW by parsing the HTTP request).

Unfortunately not. There are big restrictions on what can be transferred: https://developer.mozilla.org/en-US/docs/Glossary/Transferable_objects

On the other hand, I discover that recent browsers now allow to transfer a https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream , which could be obtained from the File object we have (representing the ZIM file): https://developer.mozilla.org/en-US/docs/Web/API/Blob/stream
That might allow us to read the ZIM file from the ServiceWorker (I did not test), but only in recent browsers.

[mossroy]

On kiwix-js, the protocol can not be zim, and the zim_id can not be the host.
The links are relative links, and use the same protocol/host as the app itself: http://localhost, moz-extension://extensionid, chrome-extension://extensionid etc.

We currently put the zim_id as the first part of the path. Example (under Chromium extension): chrome-extension://donaljnlmapmngakoipdmehbfcioahhk/wikipedia_fr_all_novid_2018-03.zim/A/Logiciel.html (here, donaljnlmapmngakoipdmehbfcioahhk is the id of the extension)

[mgautierfr]

We currently put the zim_id as the first part of the path. Example (under Chromium extension): chrome-extension://donaljnlmapmngakoipdmehbfcioahhk/wikipedia_fr_all_novid_2018-03.zim/A/Logiciel.html (here, donaljnlmapmngakoipdmehbfcioahhk is the id of the extension)

This is pretty close from what is used in kiwix-serve.

Unfortunately not. There are big restrictions on what can be transferred: developer.mozilla.org/en-US/docs/Glossary/Transferable_objects

You don't have to transfert a kiwix::Request. You can parse the url yourself (or serialize kiwix::Request) and transfert a dictionnary (and it seems it is already what you do)

The main thread would handled the request by :

• convert you object/json to kiwix::Request)
• use the new api to get the content -> which return a zim::Entry or a error (to be defined)
• transfer the content back (the same you already do it)

You will have to handle the request yourself (and you keep the hand on the schema you want to use) but you give to libkiwix all the work to locate the zim file, read the content and so.

It would handle code like:

• https://github.com/kiwix/kiwix-desktop/blob/master/src/urlschemehandler.cpp#L26-L48 in kiwix-desktop
• https://github.com/kiwix/libkiwix/blob/master/src/server/internalServer.cpp#L864-L881 in kiwix-serve
• https://github.com/kiwix/kiwix-js/blob/39c27fda9923c06f25ca1eee11cf9b86a4eab40f/www/js/app.js#L1503-L1532 in kiwix-js

(And it is only for the "get content" part, libkiwix would also handle random, main page, search for all applications)

---

This way, application developpers concentrate on the application code (UI, understand what user want, display information to the user, ...) and let libkiwix handle the zims and access to content (and internal cache).
It would be the same "API", whatever you are reading one or several zims, getting the content of a specific article or searching in zim files, or get a random article, .... Only the content of the request would change