Upgrade some npm packages so this can build for linux aarch64 (arm64)

[stuaxo]

I tried installing from npm on aarch64 on Ubuntu, but got stuck because the first version of sentry-cli (which is used here by sentry webpack) is 1.63.1, but this package has an older version of sentry-webpack which pulls in an older sentry-cli.

getsentry/sentry-cli#676

[stuaxo]

Using a bit of a guess, I pushed webpack-plugin to the following version:

"@sentry/webpack-plugin": "^1.14.2",

(The since we are already on 1.14.0, I thought this may not have any API changes)

Probably unrelated, but this didn't work:

sh: 1: npm-run-all: Permission denied
❯ sudo npm run start

> start
> npm-run-all --parallel --print-label start:server start:web

sh: 1: npm-run-all: Permission denied

I tried

❯ npm run start:web

And can get the banner saying things take longer than usual.

[stuaxo]

OT: But I had to go to the webserver using 0.0.0.0:8080 since the local subdomain isn't by default setup on Linux, is that something the dev has on their computer ?

[pimterry]

Hi @stuaxo! Thanks for reporting this. I don't have an aarch64 Linux test device, but I would love to get this working!

I'm very happy to provide some pointers if you have a test device and you're interested in helping fix this. Just in case you're not aware, HTTP Toolkit Pro is totally free for contributors.

Using a bit of a guess, I pushed webpack-plugin to the following version

I think this is a red herring. The sentry webpack plugin might need updating, sure, but that's only used when building production releases - it's not used for local launch at all.

The install might complain if there are native components here that don't build (you can just uninstall it completely, if you want to quickly solve that) but it shouldn't affect local development at all.

npm-run-all: Permission denied

This sounds quite unusual! Npm run all is a JS package, which just runs the other npm scripts listed here in a convenient cross-platform way. As far as I'm aware there's no native code involved at all except node itself, so I don't know why this wouldn't work on aarch64.

I would focus on working this out first. What should happen is that this package installs a node script at node_modules/.bin/npm-run-all, with a shebang line of /usr/bin/env node (so it's run using your current env's node version), and then everything from then on is just node & JavaScript.

Maybe that script is incorrectly not marked as executable, or /usr/bin/env node doesn't successfully run node on your machine?

I tried npm run start:web` and can get the banner saying things take longer than usual.

That's expected. When you run start:web, you're starting only the web UI, without a server. You need to run a server too.

npm run start will do that for you, or you can run the server separately by checking that out and launching it separately. That lives here: https://github.com/httptoolkit/httptoolkit-server/.

---

Stepping back slightly, I would expect there's a few possible issues here:

• The server might not install correctly, because the automatic install (implemented here and run by npm run server:setup) doesn't handle architectures correctly.
• The server might not run as expected after installing (I think we've solved this already recently for Mac ARM64, but maybe not yet for Linux)
• There might be some UI dependencies that have native components which don't easily build on aarch64, and so npm install will fail.
• There might be some native components we actually use at runtime, but to be honest I'd be surprised - unlike the server, which does use quite a few native components, this UI is very much just a JS project.

If we can work out which of those bits are failing, I can help explain what's required to get each bit of that working locally. Anything beyond that is probably unrelated to the architecture I think, and might relate to some other system configuration issue.

[stuaxo]

OK, so going back to the stepping back part - I reset the repo and pulled.. and tried npm run server:setup

❯ npm run server:setup

> server:setup
> ts-node -P ./automation/tsconfig.json ./automation/setup-server.ts

sh: 1: ts-node: Permission denied
  ~/p/e/httptoolkit-projects/httptoolkit-ui   main ─────  local  18:48:05
❯ 

Let's see what state npm-run-all is in:

❯ ls -l node_modules/.bin/npm-run-all
lrwxrwxrwx 1 stu stu 39 Sep 17 12:09 node_modules/.bin/npm-run-all -> ../npm-run-all/bin/npm-run-all/index.js

OK, it's a symlink and it points to:

❯ ls -l $(realpath node_modules/.bin/npm-run-all)
-rw-rw-r-- 1 stu stu 397 Sep 17 12:14 /home/stu/projects/external/httptoolkit-projects/httptoolkit-ui/node_modules/npm-run-all/bin/npm-run-all/index.js

I tried giving that executable status with chmod +x but that didn't help.

I verified I can run node:

❯ /usr/bin/env node --version
v16.20.2

ts-node is mentioned, is that something I need to have installed somewhere ?

[pimterry]

ts-node is mentioned, is that something I need to have installed somewhere ?

You do - but that's a dependency of the project that should be installed automatically by npm install, and npm run should automatically include node_modules/.bin in its PATH to use them. For most systems you shouldn't need to manually install anything other than Node itself.

❯ ls -l $(realpath node_modules/.bin/npm-run-all)
-rw-rw-r-- 1 stu stu 397 Sep 17 12:14 /home/stu/projects/external/httptoolkit-projects/httptoolkit-ui/node_modules/npm-run-all/bin/npm-run-all/index.js

I tried giving that executable status with chmod +x but that didn't help.

This is definitely weird. On my machine, with a fresh npm install using node v16.18.1, I see:

$ ls -l "$(realpath node_modules/.bin/npm-run-all)"
-rwxrwxr-x 1 tim tim 397 oct  6 18:14 '[...]/node_modules/npm-run-all/bin/npm-run-all/index.js'

I.e. it's executable by default. The same is true for node_modules/.bin/ts-node, and I can run that directly to get a TS-node REPL.

When you say changing that didn't help - presumably you mean changing npm-run-all's permissions still shows that same ts-node error? What happens if you make ts-node executable too?

My best guess would be that for some reason (env vars, system configuration, filesystem flags, who knows) your npm install isn't correctly marking JS scripts that get installed as executable, which then breaks all the scripts that use them.

[stuaxo]

You're right npm dropped the executable bit on all those files.

I found some old bugs in npm from 2017 covering this for npm 5.4.0 but my install has 8.x

I fixed the permissions with this script:

#!/bin/bash
cd node_modules/.bin
for file in *; do
    if [[ -L "${file}" ]]; then
        target=("$(readlink "${file}")")
        echo ${file}
        chmod +x ${target}
    fi
done

Then was able to get further with setup, where it fails because it can't find the right download.

❯ npm run server:setup

> server:setup
> ts-node -P ./automation/tsconfig.json ./automation/setup-server.ts

Downloading latest httptoolkit-server...
Error: No server download available matching /httptoolkit-server-v[\d\.]+-linux-arm64\.tar\.gz/
    at /home/stu/projects/external/httptoolkit-projects/httptoolkit-ui/automation/setup-server.ts:79:15
    at Generator.next (<anonymous>)
    at /home/stu/projects/external/httptoolkit-projects/httptoolkit-ui/automation/setup-server.ts:31:71
    at new Promise (<anonymous>)
    at __awaiter (/home/stu/projects/external/httptoolkit-projects/httptoolkit-ui/automation/setup-server.ts:27:12)
    at downloadServer (/home/stu/projects/external/httptoolkit-projects/httptoolkit-ui/automation/setup-server.ts:90:12)
    at /home/stu/projects/external/httptoolkit-projects/httptoolkit-ui/automation/setup-server.ts:37:15
    at Generator.next (<anonymous>)
    at fulfilled (/home/stu/projects/external/httptoolkit-projects/httptoolkit-ui/automation/setup-server.ts:28:58)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

[stuaxo]

Re the name: the convention on on seems to be arm64, on Linux I see a lot of aarch64.

I have no idea what the convention for js projects on arm64/aarch64 on Linux is but probably worth checking before creating a release.

[pimterry]

I have no idea what the convention for js projects on arm64/aarch64 on Linux is but probably worth checking before creating a release.

I don't know the full implications of the options, but I think in practice we'll want to use arm64, as that's the arch specifier used by Node.js everywhere, and I think it's reasonably widely used elsewhere too so shouldn't be too problematic.

Then was able to get further with setup, where it fails because it can't find the right download.

Ok, that output is more promising. The server error shown there is an interesting problem, but that's something we'll need to solve in httptoolkit-server - it's not really related to the UI.

Can you try running npm run start:web? If that works for you, and localhost:8080 then loads (it should load a page, but show an 'HTTP Toolkit' progress bar that never completes, because it needs a running server) then the UI part is working correctly. If that works for you, I think we can close this issue and call the UI workable.

I've opened a separate issue for the server work over here. There's some details there on exactly what's required for this. If you have a minute it'd be really helpful to just try checking out that repo, and doing an npm install and npm start, and sharing the output, so we can start work on the first step there.

Let's keep further discussion of this over in that repo though, and just track UI issues here, just to keep things organised.

[stuaxo]

Confirming that npm run start:web correctly starts the banner on localhost:8080

[pimterry]

Ok, great! Glad the UI setup works then, although do let me know if you find any more clues about the default npm permissions issue you're seeing.

I'll close this for now, and we can continue the server side portion on the other issue.