0.18.0: broken HTML file (charset declaration *wrong*) - override possibility??

[andim2]

Hello,

as reported at lynx-dev:
[BUG] [DOCS] broken HTML file (charset declaration wrong) - override possibility?? (ASSUME_CHARSET areas etc.),
I am having an HTML file (Microsoft-originating data; "Microsoft Word 15") with wrong charset declaration (iso-8859-1), where
the document body contains UTF-8 code units (as can be directly seen via
the transport-side quoted-printable encoding:

<p class=3D"MsoNormal" style=3D"margin-bottom:0cm;line-height:normal"><span=
 style=3D"color:#003C74;mso-fareast-language:DE">Viele Gr=C3=BC=C3=9Fe,
<o:p></o:p></span></p>

).

links -dump test.mre.html
(at least version 0.18.0, i.e. older than HEAD)

will display glorious Mojibake-laden

Hallo Herr Mustermann,
 
vielen Dank für Ihre Meldung. Hiermit bestätigt.

output.

I then frantically tried to
override things, via

links -dump -dump-charset UTF-8 test.mre.html

This did not work.

(-dump-charset option does seem to be initially considered, since
e.g. ATF-8 will properly cause a
ELinks: Cannot parse option ATF-8: Read error
error report).

Thus, I am suspecting that
[e]links is having
the same kind of support weakness that
lynx has (overriding of a b0rken encoding declaration not possible).

...or it might just be that
-dump-charset option is intended to
handle this, yet that implementation simply is
broken, currently.

This issue should be easily verifiable in an alternative manner, by
modifying a properly UTF-8 HTML file (containing
extended i.e. non-ASCII-range characters, umlauts etc.) to
declare iso-8859-1 charset.

Thank you!!

[rkd77]

elinks has options "document.codepage.assume" and "document.codepage.force_assumed".
I would try setting these first. libdom can cause some problems, so if it does not work, build elinks without js and libcss. If these fail either copy files, and serve them by some http server, with proper encoding. AFAIR charset set by server has precedence. You can also set pre_format_hook in your favorite language and fix encoding in script.

[andim2]

Thank you for your fast aid!

I now tried

$ links -config-file ~/.etc/elinks.conf -eval 'set document.codepage.assume = "UTF-8" set document.codepage.force_assumed = 1' test.mre.html|less

and that made it work (both settings required):
Mojibake gone for good ("völlig hübsch!').

HOWEVER, relying on -config-file alone did not work - I needed to resort to custom config (-eval), else things did not end up correctly in -config-dump. So maybe for some weird reason links is ignoring [evaluation of] my local config file?? strace to the rescue, maybe?
Indeed:

openat(AT_FDCWD, "/etc//home/USER/.etc/elinks.conf", O_RDONLY|O_NOCTTY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/./home/USER/.etc/elinks.conf", O_RDONLY|O_NOCTTY) = -1 ENOENT (No such file or directory)

Turns out that
[e]links implements things as a -config-dir / -config-file split (d'oh! Missed that one...).
"Kaum macht man's richtig, schon geht's..."

Also, I still have a nagging question why a simple -dump-charset option is not enough to make it work (though perhaps
-dump-charset has a slightly different behaviour/intention/targeting beyond expectations).

so if it does not work, build elinks without js and libcss.

Surely very helpful extra advice (for some certain populace), but less so for me:
am in a pre-made environment (currently less inclined to do some customization/bending) - I just wanted to
have this issue raised/discussed ;-)

(and I have to admit that
I had initiated this issue with lynx, thus
did most of the testing there, and less so on the [e]links side so far)

So, this discussion seems to somehow hint that
[e]links currently is more flexible than lynx (especially since
it also offers hooking functionality, via pre_format_hook).

With current issue status, one could consider it to be closed I guess (unless
there's some related documentation usability waiting to be improved, perhaps).

[rkd77]

@andim2 links and elinks are not the same browsers. In your code samples was called links.

[andim2]

links and elinks are not the same browsers. In your code samples was called links.

Whoopsie.

Color me a bit confused though:

$ which links
/usr/bin/links

$ ls -l /usr/bin/links
lrwxrwxrwx. 1 root root 23 Jan 11  2020 /usr/bin/links -> /etc/alternatives/links

$ ls -l /etc/alternatives/links
lrwxrwxrwx. 1 root root 15 Jan 11  2020 /etc/alternatives/links -> /usr/bin/elinks

So this is an "elinks" thingy (and possibly adopts "links" behaviour/"personality" via
usual symlink pinpointing qualification stuff...?).

So pray tell me I managed to hit the entirely wrong universe here? :->
(let me just try to insinuate that I haven't created any issue # activity whatsoever here... "you haven't even seen me" :-))

[rkd77]

Sorry, I did not know that distributions do such symlinks. It is kind of premature optimisation IMO.

[andim2]

@rkd77

Sorry, I did not know that distributions do such symlinks. It is kind of premature optimisation IMO.

Not quite sure what you're alluding to here.

Intrestingly, on Debian-based systems (well, Ubuntu) - as opposed to AFAIK RedHat-based -, there's no symlinking taking place - instead,
they're separate packages (and links already was installed, and elinks installed now).

$ dpkg -S /usr/bin/links
links: /usr/bin/links
$ dpkg -S /usr/bin/elinks
elinks: /usr/bin/elinks
$ ls -l /usr/bin/links
-rwxr-xr-x 1 root root 2329784 Nov 29  2021 /usr/bin/links
$ ls -l /usr/bin/elinks
-rwxr-xr-x 1 root root 1767600 Feb  6  2022 /usr/bin/elinks
$ dpkg -s links
Package: links
Status: install ok installed
Priority: optional
Section: web
Installed-Size: 2313
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: amd64
Multi-Arch: foreign
Source: links2
Version: 2.25-1build1
Provides: www-browser
Depends: libbrotli1 (>= 0.6.0), libbz2-1.0, libc6 (>= 2.34), libgpm2 (>= 1.20.7), liblz1 (>= 1.12), liblzma5 (>= 5.1.1alpha+20120614), libssl3 (>= 3.0.0~~alpha1), libzstd1 (>= 1.4.0), zlib1g (>= 1:1.1.4)
Recommends: ca-certificates
Suggests: mc
Description: Web browser running in text mode
 Links is a text mode WWW browser, similar to Lynx. It displays tables,
 frames, downloads on background, uses HTTP/1.1 keepalive connections.
 .
 This version is compiled without graphics mode. See the package
 links2 for a version with graphics support compiled in.
Original-Maintainer: Axel Beckert <abe@debian.org>
Built-Using: publicsuffix (= 20211109.1735-1)
Homepage: http://links.twibright.com/
$ dpkg -s elinks
Package: elinks
Status: install ok installed
Priority: optional
Section: web
Installed-Size: 1773
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: amd64
Version: 0.13.2-1build3
Provides: www-browser
Depends: libbz2-1.0, libc6 (>= 2.34), libev4 (>= 1:4.15), libexpat1 (>= 2.0.1), libfsplib0 (>= 0.9), libgcrypt20 (>= 1.9.0), libgnutls30 (>= 3.7.2), libgpm2 (>= 1.20.7), libgssapi-krb5-2 (>= 1.17), libidn12 (>= 1.13), liblua5.1-0, liblzma5 (>= 5.1.1alpha+20120614), libperl5.34 (>= 5.34.0), libtinfo6 (>= 6), libtre5, elinks-data (= 0.13.2-1build3)
Suggests: elinks-doc
Description: advanced text-mode WWW browser
 ELinks is a feature-rich program for browsing the web in text mode.  It is
 like enhanced Lynx and Links.  The most noteworthy features of ELinks are:
 .
  * Lots of protocols (local files, finger, HTTP(S), FTP, IPv4/6 etc.)
  * Internationalized domain names
  * Persistent cookies, HTTP authentication and proxy authentication
  * Tabbed browsing, good looking menus and dialogs, and key-binding manager
  * History browsing and typeahead searches
  * Forms history and completion, and history in commonly used input dialogs
  * CSS support and support for browser scripting (Perl, Lua, Guile etc.)
  * Tables and frames rendering, and configurable color support
  * Compressed and background (non-blocking) downloads, and download resuming
 .
 This package is based on felinks fork, since original elinks seems to be no
 longer maintained.
Homepage: http://elinks.cz/
Original-Maintainer: أحمد المحمودي (Ahmed El-Mahmoudy) <aelmahmoudy@users.sourceforge.net>

Thanks!

[rkd77]

set document.codepage.assume = "UTF-8"
set document.codepage.force_assumed = 1'

These settings are for document codepage. --dump-charset is equivalent of the terminal codepage.
There can be many combinations of these settings.
I don't know whether to add some aliases or other cmdline options.
@andim2 Is not eval sufficient for them?