Emojis failing in spell check dictionary.txt

[avahoffman]

Emojis in the dictionary.txt are ignored.

GDSCN Repo might be helpful - we used unicode UTF-8 so not sure if that's helpful
https://github.com/fhdsl/GDSCN_BioDIGS_Book/blob/729af6996ce29373bedfa7b82b47f9042cb1821d/child/_child_research_team.Rmd#L20C52-L27

[acoffman]

Potentially useful from inside the actual ottrpal check:

stop("Spell check did not work. Check that your dictionary is formatted correctly. You cannot have special characters (non-ASCII) in the dictionary.txt file. You need to use HTML formatting (e.g., Diné) for these.")

I believe emoji are unicode only so we may have to investigate a different method of handling them.

[avahoffman]

@acoffman I think I added that error handling 😅 Would be great to supersede this (special characters AND emoji) somehow with the check itself. Perhaps somewhere in the R code rather than the workflow.

[acoffman]

Good news and bad news here. The bad news is that I can confirm emojis are ignored and it's due to a dependency of a dependency we're using. The spelling library that spell_check_files comes from itself relies on a library called hunspell and that library does not seem to support emoji. In fact, it just drops them which matches the behavior reported here.

As an example:

> hunspell::hunspell_parse("b😀a")
[[1]]
[1] "b" "a"

This seems to be due to the fact that hunspell tokenizes and then stems words in order to compare them to a dictionary along with conjugation and pluralization rules. It's not just doing an equality comparison between a list of words and the words in the file. Without switching to a different spellcheck library, I don't think there's a direct fix for this. There is a related outstanding PR to the spelling library that pre-filters words before passing them into hunspell:

ropensci/spelling#67

Likewise, other people are seeing similar issues with punctuation, numbers, etc: ropensci/spelling#45

We could propose a PR upstream that works off the existing one; I don't think it'd be all that complicated. Being that it's been open for 3 years at this point though I'm not super optimistic about getting it merged. If we wanted to do something kludgy to support this on our end we could basically copy the file to a tmp location, removing any instances of words in our dictionary during the copy, and then run the spell check library on the new file. As long as we don't remove any \n characters, the line numbers should still line up in the report. Essentially our own version of a "pre filter" like that PR, but outside the context of the spell check library.

If that's something you think is worth pursuing, I'm happy to put together a PR, it should be fairly straightforward.

The good news is that I do not seem to have any issues with unicode characters. I don't know if the library has been updated since the example linked above but one of the characters that had to be escaped in the example is ñ.

With the following dictionary.txt:

foo
bar
ña

and the following test file:

foo fooo bar baz hello world añ ña

Running

dict <- readLines("spelling.txt")
sp_errors <- spelling::spell_check_files("spell-test.txt", ignore = dict)

Reports spelling errors of

añ
baz
fooo

Which I believe is the expected behavior, so it seems likely that we do not have to escape unicode chars going forward (at least for the purposes of the spell checker. There may be some other part of the build pipeline I'm unaware of that requires escaping).