texMathToInlines in custom writers

[refashioned]

Is it possible to access texMathToInlines from a Lua custom writer?

Lots of built-in writers call texMathToInlines to render uncomplicated math (that only requires basic markup like italics and superscripts): for example html, man, and rtf.

For example, this Markdown document:

A formula: $a^2 + b^2$.

$$\frac{\sqrt 2}{2}$$

renders to the below HTML. The first formula is simple enough that it can be rendered with HTML elements like em and sup. The second formula cannot: so texMathToInlines falls back to the TeX representation. (Note the warning that is emitted.)

$ pandoc -f markdown -t html --wrap=none demo.md
[WARNING] Could not convert TeX math \frac{\sqrt 2}{2}, rendering as TeX
<p>A formula: <span class="math inline"><em>a</em><sup>2</sup> + <em>b</em><sup>2</sup></span>.</p>
<p><span class="math display">$$\frac{\sqrt 2}{2}$$</span></p>

It's a neat feature. Is it possible to access it from a custom writer?

Below is a hack to call texMathToInlines in a custom writer, indirectly, by roundtripping through the HTML writer. This is its output on the above input file:

$ pandoc -f markdown -t demo.lua --wrap=none demo.md
[WARNING] Could not convert TeX math \frac{\sqrt 2}{2}, rendering as TeX
(Para "A" " " "formula:" " " (Span (Emph "a") (Superscript "2") " + " (Emph "b") (Superscript "2")) ".")

(Para (Code "$$\\frac{\\sqrt 2}{2}$$"))

The output format is a made-up S-expression thing; that's not the important part. The important part is Writer.Inline.Math and the tex_math_to_inlines function. It calls pandoc.write to generate a string of HTML, taking advantage of the fact that the HTML writer calls texMathToInlines internally. It calls pandoc.read to parse the HTML string back into a pandoc.Pandoc, then extracts the necessary inlines from it.

This hack actually seems pretty reasonable, except for the error checking. See, texMathToInlines doesn't exactly return an error when the math cannot be represented by inlines, it logs "[WARNING] Could not convert TeX math" and returns a fallback (the input text surrounded by $ or $$). The implementation below takes the approach of checking the length of PANDOC_STATE.log before and after writing the HTML, to see if a warning log message was emitted. The approach seems fragile. Alternative approaches might be to check whether the return value is a Span or a Str, or whether it starts with $. But these too depend on implementation details. Is there a better way? I'm wondering if there is an API I have overlooked.

Writer = pandoc.scaffolding.Writer

local function sexp_inlines(tag, content)
	local rendered = {}
	for i, el in ipairs(content) do
		rendered[i] = Writer.Inline(el)
	end
	return pandoc.layout.parens(tag + pandoc.layout.concat(rendered, pandoc.layout.space))
end

local function sexp_text(tag, text)
	return sexp_inlines(tag, {pandoc.Str(text)})
end

local function quote(s) return string.format("%q", s) end

function Writer.Block.Para(el, opts)         return sexp_inlines(el.tag, el.content) end
function Writer.Inline.Emph(el, opts)        return sexp_inlines(el.tag, el.content) end
function Writer.Inline.Span(el, opts)        return sexp_inlines(el.tag, el.content) end
function Writer.Inline.Superscript(el, opts) return sexp_inlines(el.tag, el.content) end
function Writer.Inline.Code(el, opts)        return sexp_text(el.tag, el.text) end
function Writer.Inline.Str(el, opts)         return quote(el.text) end
function Writer.Inline.Space(el, opts)       return quote(" ") end

local function tex_math_to_inlines(mathtype, inp)
	-- Create a new pandoc.Pandoc containing just the desired Math.
	local math_doc = pandoc.Pandoc({pandoc.Plain({pandoc.Math(mathtype, inp)})})
	-- Count the number of log messages. We will check this again after the
	-- conversion to see if a warning log was written, indicating that the
	-- conversion failed.
	local log_len_before = #PANDOC_STATE.log
	-- Render the Math doc to HTML, taking advantage of the fact that the
	-- HTML writer calls texMathToInlines.
	local html = pandoc.write(math_doc, "html", {html_math_method = "plain", wrap_text = "wrap-none"})
	-- Count the number of log messages again.
	local log_len_after = #PANDOC_STATE.log
	local num_logs = log_len_after - log_len_before
	if num_logs == 0 then
		-- No new log messages were written: the conversion succeeded.
		-- Parse the HTML back into a pandoc.Pandoc and extract the
		-- inlines containing the rendered formula.
		return pandoc.utils.blocks_to_inlines(pandoc.read(html, "html").blocks)
	elseif num_logs == 1 then
		-- The conversion failed. A log message was written, presumably
		-- 	[WARNING] Could not convert TeX math ..., rendering as TeX
		-- Return nil to signal error.
		return nil
	else
		error("unexpected %d new log messages", num_logs)
	end
end

function Writer.Inline.Math(el, opts)
	local plain = tex_math_to_inlines(el.mathtype, el.text)
	if plain then
		-- Managed to represent the math as plain inlines.
		return Writer.Inlines(plain)
	else
		-- Couldn't render the math as inlines, output the TeX math.
		local delim = assert(({InlineMath = "$", DisplayMath = "$$"})[el.mathtype])
		return Writer.Inline(pandoc.Code(delim .. el.text .. delim))
	end
end

[jgm]

To simplify a bit, you could use the plain writer instead of the HTML writer, and check for an initial $ for failure.

[refashioned]

I don't understand. The plain writer does call texMathToInlines, but it doesn't return the result in a form that can be re-parsed to recover the AST.

If I switch to the plain writer, the inlines (such as they are) returned by tex_math_to_inlines are just a flat string containing an approximate rendering of the formula:

[Str "a\178\8197+\8197b\178"]

(Where \178 is a superscript ² and \8197 is a space character.) Whereas the html writer uses HTML elements like <em> and <sup> that I can then re-parse into the rich inlines I want:

[Span ("",["math","inline"],[]) [Emph [Str "a"],Superscript [Str "2"],Str "\8197+\8197",Emph [Str "b"],Superscript [Str "2"]]]

Maybe plain was a poor choice of variable name in Writer.Inline.Math, but I'm not looking for just a plain-text version of the input formula. Suppose my custom writer is for a rich-text format that can support italics, nested superscripts, etc., and that I want to be able to retain those features in the output. It doesn't necessarily have to be html, but the intermediate format has to be something with markup that preserves the structure of the output of texMathToInlines.

But I suppose what you could do is render to the plain format in addition to html format: using plain only for an error check (looking for initial $ as you suggest); then proceeding with the html roundtrip in case there is no error. That way does seem more robust than the approach of counting log lines.

I guess though, from your answer, that I'm not overlooking anything, that texMathToInlines is not directly available to Lua writers.

[refashioned]

Here's a version of tex_math_to_inlines that uses the plain writer for an error check.

-- Convert a pandoc.Math element into plain Pandoc inlines, if possible. This
-- only works for math that is not too fancy. Returns the inlines if the
-- conversion is possible; otherwise returns nil. (In this respect it differs
-- from the Pandoc function texMathToInlines, which returns a fallback in case
-- of error.)
local function tex_math_to_inlines(mathtype, inp)
	-- We don't have access to Pandoc's texMathToInlines function, so we
	-- employ a hack to access indirectly. We create a new Pandoc document
	-- containing just the Math we are interested in and convert it to an
	-- HTML string, taking advantage of the fact that the HTML writer uses
	-- texMathToInlines when html_math_method = "plain" in WriterOptions.
	-- Then we parse the HTML string back into an AST to recover the
	-- inlines. We don't get an explicit error when the conversion fails
	-- inside the HTML, so in order to check for error, we do a preliminary
	-- conversion using the plain writer and check for the output starting
	-- with '$', which is true of the plain writer's fallback rendering.
	-- https://github.com/jgm/pandoc/discussions/11399

	-- Create a new pandoc.Pandoc containing just the desired Math element.
	local doc = pandoc.Pandoc({pandoc.Plain({pandoc.Math(mathtype, inp)})})
	-- Write to plain format to check if conversion is possible.
	local plain = pandoc.write(doc, "plain", {html_math_method = "plain", wrap_text = "wrap-none"})
	if string.match(plain, "^%$") then
		-- The output starts with '$': conversion failed.
		return nil
	end
	-- If the conversion succeeded for the plain writer, assume it will
	-- also succeed for the HTML writer.
	local html = pandoc.write(doc, "html", {html_math_method = "plain", wrap_text = "wrap-none"})
	-- Parse the HTML string back into a pandoc.Pandoc and extract the inlines.
	return pandoc.utils.blocks_to_inlines(pandoc.read(html, "html").blocks)
end

[jgm]

If you want to avoid the plain writer step, you could just check the HTML writer output for the string >$.

[refashioned]

you could just check the HTML writer output for the string >$.

I'm worried that test will be fooled by notation like A^\$, such as might arise in cryptography:

<span class="math inline"><em>A</em><sup>$</sup></span>

See, for example, Section 1.4 of these lecture notes.

Checking for $ only at the beginning could be fooled, too, but I still think it's overall better than counting log messages (more likely to keep working as Pandoc changes). As long as it's a sufficient test for conversion error, may as well make it as narrow as possible.

[jgm]

Could check for "math inline">$ or "math display">$.