Add the ability to modify the resource path from lua

[CodeSandwich]

I need to dynamically load markdown files referencing resources relatively. E.g. docs/03/doc.md references docs/03/MEDIA/image.png as ![](MEDIA/image.png). If I run pandoc --resource-path=docs/03 docs/03/doc.md ..., they are found just fine, but if I load the markdown file dynamically, with pandoc.read there doesn't seem to be any way to perform pandoc.mediabag.fetch that would include docs/03 as the search path unless I declare this path with --resource-path when starting pandoc. There's the PANDOC_STATE.resource_path list, but it's read-only, modifying it has no effect.

Describe your proposed improvement and the problem it solves.

Being able to modify PANDOC_STATE.resource_path inside the filter could be the right solution.

Describe alternatives you've considered.

I'm not aware of any alternatives, this use case is simply unsupported.

[tarleb]

My first attempt at this can be found in the pandoc.init branch. The approach here is to allow initializing the common state in Lua, but only if it hasn't been initialized before, as to prevent too much confusion. The approach mostly works, but I stopped working on it because it's a very fragile solution and not very flexible.

However, I still like the idea of keeping the common state read-only after it's been initialized

My preferred solution would therefore be to support the creation of new Lua-threads, which could be initialized with any PANDOC_STATE. That would allow to run certain parts of the code with an arbitrary state, and it could open up a path towards multi-threading, which some heavy Lua users have been asking about.

Ideally we'd use Lua coroutines, as to get an idiomatic interface, but those are a bit tricky to work with.

[CodeSandwich]

Now that I'm thinking about it, my use case could be solved by just passing the extra resource paths to pandoc.mediabag.fetch and .fill. It would only affect the specific call, so it would be explicitly scoped and in it wouldn't alter the common state. I'm not aware of uses of PANDOC_STATE.resource_path other than during mediabag filling, are there any?

[tarleb]

The resource path currently used during citeproc processing and in the Roff and LaTeX parsers, plus every time that the Haskell function fetchItem is called, which happens in a lot of writers and in the RST reader.

[CodeSandwich]

I'm not sure, but it could mean that for the readers and parsers the dynamic resource path may be already resolved. I don't understand and can't find on the internet what the "ersatz" filesystem is, but some paths can be passed to pandoc.read as read_env, it looks like it may be used to override the resources path? If this is the case, then the mediabag is the last place where the resource path is used but can't be overridden.

[tarleb]

No, the resource path cannot be overridden in any Lua function right now. Consistently adding an optional parameter for that would require changes to, at least, pandoc.mediabag.fetch, pandoc.mediabag.fill, pandoc.utils.citeproc, pandoc.read, and pandoc.write. I have no appetite for that.

[CodeSandwich]

Makes sense. I'm probably a poor judge here, but adding an optional parameter to 5 functions doesn't sound that bad. It's a simple solution that is easy to grasp for the users. The other approach you described involving threads to override global variables sounds like something difficult to understand and use. I'm stepping into a wild guesses territory, but wrapping things into threads is usually a tricky thing to implement with long lasting consequences on the architecture and a large class of hard bugs. Passing around an additional argument seems less clever and more arduous, but usually easier and less disruptive. It's just my 5 cents, I probably won't be implementing any of this and I'll be happy if I'm ever able to fulfill my use case.