System unavailable: trss.adoptopenjdk.net

[sxa]

• Please describe the issue:
  System is unresponsive. There was an issue with the TRSS server yesterday but it was able to be fixed on the machine. Today the system was competely unresponsive and not contact could be made with it (Initially reported at ~3am GMT)

I have managed to restart the host on the provider (AWS) but after 20 minutes it is not responding on the ssh port (although it is pingable)

[sxa]

Server now responding to the ssh port. Unfortunately the backend Node.js process appears to be repeatedly crashing and restarting so the service is not yet responsive.

[sxa]

From the backend logs -

error: Forever detected script exited with code: 0
error: Script restart attempt #182
12:08:31 PM - warn: Cannot find the config file:  --configFile=/dev/mongodb/credentials/trssConf.json
12:08:32 PM - error: Exception in database query:  message=Cannot read property 'collection' of undefined, stack=TypeError: Cannot read property 'collection' of undefined
    at new TestResultsDB (/home/jenkins/openjdk-test-tools/TestResultSummaryService/Database.js:257:23)
    at EventHandler.processBuild (/home/jenkins/openjdk-test-tools/TestResultSummaryService/EventHandler.js:19:37)
    at Timeout._onTimeout (/home/jenkins/openjdk-test-tools/TestResultSummaryService/backend.js:11:13)
    at listOnTimeout (internal/timers.js:531:17)
    at processTimers (internal/timers.js:475:7)
12:08:32 PM - error: Exception in database query:  message=Cannot read property 'collection' of undefined, stack=TypeError: Cannot read property 'collection' of undefined
    at new BuildListDB (/home/jenkins/openjdk-test-tools/TestResultSummaryService/Database.js:278:23)
    at EventHandler.monitorBuild (/home/jenkins/openjdk-test-tools/TestResultSummaryService/EventHandler.js:57:37)
    at Timeout._onTimeout (/home/jenkins/openjdk-test-tools/TestResultSummaryService/backend.js:12:13)
    at listOnTimeout (internal/timers.js:531:17)
    at processTimers (internal/timers.js:475:7)
(node:3779) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'collection' of undefined
    at new AuditLogsDB (/home/jenkins/openjdk-test-tools/TestResultSummaryService/Database.js:285:23)
    at EventHandler.processBuild (/home/jenkins/openjdk-test-tools/TestResultSummaryService/EventHandler.js:42:23)
    at Timeout._onTimeout (/home/jenkins/openjdk-test-tools/TestResultSummaryService/backend.js:11:13)
    at listOnTimeout (internal/timers.js:531:17)
    at processTimers (internal/timers.js:475:7)
(node:3779) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:3779) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:3779) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'collection' of undefined
    at new AuditLogsDB (/home/jenkins/openjdk-test-tools/TestResultSummaryService/Database.js:285:23)
    at EventHandler.monitorBuild (/home/jenkins/openjdk-test-tools/TestResultSummaryService/EventHandler.js:78:23)
    at Timeout._onTimeout (/home/jenkins/openjdk-test-tools/TestResultSummaryService/backend.js:12:13)
    at listOnTimeout (internal/timers.js:531:17)
    at processTimers (internal/timers.js:475:7)
(node:3779) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)

[sxa]

Machine became unresponsive at 02:29:28 (base on the kernel messages) with an Out of memory situation.
I am trying to recover the system but it looks like there may have been configuration files etc. stored in the dynamic /dev filesystem (as per the snippet in the previous comment which referneces --configFile=/dev/mongodb/credentials/trssConf.json - I can see entries in the shell history for things like mkdir -p /dev/mongodb/data which slightly worries me since it suggests that's not a dynamically created area that will get regenerated somehow (After reboot there is no /dev/mongodb on the machine)

There is no information on restarting the mongodb service on https://github.com/AdoptOpenJDK/openjdk-test-tools/tree/master/TestResultSummaryService

I have tried starting mongo - it first had a problem with /dev/mongodb/log not existing, then not being owned by the correct user. After resolving that, systemctl start mongod appears to have got it working, but I'm still at a loss as to where the /dev/mongodb/credentials is supposed to come from.

Current status: mongodb, TRSSBackend and TRSSFrontend services are showing as active (running) but connections to the server (SSH or the nginx on 443) are not possible. Reason currently unknown (external firewall?) - I was lucky to have been able to get in while connections were allowed.

[sxa]

@llxia Need your input on the /dev/mongodb directory and also likely some doc updates if mongodb has to be started manually separately from the front and back end services.

[sxa]

Looks like all the database stuff had been stored on a ramdrive and is therefore lost and will need to be rebuilt.

[sxa]

Machine now has 16Gb of swap (equals the amount of RAM) and a 160Gb /data partition that we can use for the results database.

[sxa]

AWS moved the IP address on the host. After it rebooted there was still a log entry with the pold IP address but it subsequently switched. PR in for inventory change.

https://trss.adoptopenjdk.net address now pointing to the new IP address

MongoDB is now running on a persistent filesystem (The new /data) so we should be back in action ...

[sxa]

@llxia Can you update the documentation to cover the setup of MongoDB and how to restart it etc.

[llxia]

We use standard cmd to install and to restart systemctl restart mongod. The only thing is that if there is user/password for DB access, then TRSS needs to know (in trssConf.json). In the above case, MongoDB started correctly. TRSS cannot find trssConf.json, so it cannot connect to MongoDB.

I will update the readme.

[karianna]

Can this be closed now (nice job on the rescue BTW)?

[sxa]

Can this be closed now (nice job on the rescue BTW)?

I was holding off until we have the documentation updated with details of what goes into the trssConf.json on the production server so we don't hit so many problems next time (keeping this open stops us from forgetting about it...)