QPC: All platforms are unstable

[Willsparker]

Ref: https://ci.adoptopenjdk.net/job/QEMUPlaybookCheck/229/

With the latest QPC run, all platforms have failed to some degree. 4 have failed due calling buildJDK.sh in an incorrect way (incorrect as of #1962 ).

The risc-v platform is still blocked by #1483

And the arm32 platform seems to be running out of space during the playbook execution.

[Willsparker]

Changes made to qemuPlaybookCheck.sh have been made to use the correct arguments (willsparker/2123_1). Testing on QPC at: https://ci.adoptopenjdk.net/job/QEMUPlaybookCheck/230/

[Willsparker]

The above PR should fix the buildJDK.sh issues. Once this has been merged, I'll look at running the arm32 box, to determine if there's still issues with the space.

[Willsparker]

The PR to fix the arguments has been merged, and it appears to be working on all platforms. For the S390x and ppc64le architectures, the linux.sh script fails for JDK8, as it is unable to find JDK7 on the machines (as Zulu-7 is not installed). I thought that they were meant to default then to using JDK8, but apparently not. This is not the case for JDK11, where I have had the build start on both machines (though currently, I've only managed a QPC run that had ppc64le fail (maybe due to the wrong version of gcc..?), and s390x that core dumped (maybe due to a bad jdk-10 install..?))

I'm going to look into that platform-specific-configuration script, to see if it's meant to set the boot jdk to the build jdk in cases wehre the boot jdk can't be found. If it isn't, I'll look into install zulu-7 on those platforms.

[sxa]

It can always be overriden with JDK7_BOOT_DIR if needed, but I know we've tried to let the autodetection work.
My preference is for it not to fall back if possible, since you then run the risk of being unsure whether from one build to the next it's used a JDK7 or JDK8 as the boot JDK. Our "real" CentOS/RHEL7 ppc64le and s390x build machines all use a true JDK7.

[Willsparker]

Okay! Thanks for the guidance. Looks like I'll have to adapt the Zulu-7 ansible task to include those platforms :-)

EDIT: Turns out no, this can't be done (see: https://adoptium.slack.com/archives/C53GHCXL4/p1619777906294700). Best case is having those platforms use JDK-8 to build

[Willsparker]

With #2176 (comment) , I think the last part of fixing the non RISC-V platforms is to figure out how to extend the build images, as they run out of space halfway through a build.

[Willsparker]

QEMU qcow2 images can be resized easily enough with
qemu-img resize $IMG +10G (if you wanted to add 10GB to it).
However, the partitions still need to be extended, which will be specific to the platform, and is likely to be a massive pain. Helpfully, I think all of the platforms that fail due to disk issues will fail in the build. Therefore, we could mount it as a new partition and just build on that one if it's that difficult.

EDIT: AH hang on, I've already documented all of this. Okay, just going to extend them by 10GB, and create a new partition on /home/linux that will have that extra 10GB. This method should be fine for all images that don't have a swap partitions. I'll create backups of the current images, just in case :-)

[Willsparker]

List of what I did to extend each of the images. After extending partitions, I'll re-compress the image with xz -z, and move the images into /home/jenkins/qemu_base_images/resized_images/, that way we still have backups of the old images

• Debian10.aarch64.dsk :

• Image is qcow2, so extend by running qemu img resize $IMAGE +10G

When logging into the machine, it automatically extended the root partition, which was lovely to see. So that has 25GB on it now.

• Debian11.riscv.dsk:

• qcow2 image, so same command as above.

• This one has 10GB mounted on the root partition, and 5GB on /home/linux. So, I'll delete the /home/linux partition and create a new one in the same place, of size 15GB:

root@debian:~# lsblk
NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda    254:0    0  25G  0 disk 
|-vda1 254:1    0  10G  0 part /
`-vda2 254:2    0   5G  0 part /home/linux
root@debian:~# fdisk /dev/vda
Command (m for help): d
Partition number (1,2, default 2): 2

Partition 2 has been deleted.

Command (m for help): n
Partition number (2-128, default 2): 2
First sector (20971393-52428766, default 20971520): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (20971520-52428766, default 52428766): 

Created a new partition 2 of type 'Linux filesystem' and of size 15 GiB.
Partition #2 contains a ext4 signature.

Do you want to remove the signature? [Y]es/[N]o: Y

The signature will be removed by a write command.
Command (m for help): w
The partition table has been altered.
Syncing disks.

root@debian:~# lsblk
NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda    254:0    0  25G  0 disk 
|-vda1 254:1    0  10G  0 part /
`-vda2 254:2    0  15G  0 part /home/linux

• Debian8.Arm32.dsk:

• Still a qcow2 image, so, as above

This ones a bit different, there's a lot more partitions:

$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda    254:0    0   30G  0 disk 
├─vda1 254:1    0  243M  0 part /boot
├─vda2 254:2    0  5.3G  0 part /
├─vda3 254:3    0    1K  0 part 
├─vda5 254:5    0  502M  0 part [SWAP]
└─vda6 254:6    0   14G  0 part /home

Unfortunately, I can't seem to find a way of extending the root partition, without the following error / issue on startup

[  155.371104] FS-Cache: Netfs 'nfs' registered for caching
Welcome to emergency mode! After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" to try again
to boot into default mode.
[  155.698568] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
Give root password for maintenance
(or type Control-D to continue): 

I tried removing partitions 2-6, and recreating the / and /home partition, but the same issue occurs. I tried just removing the second and third partition, and it caused the rest of them to disappear as well. I'm going to skip this one for now, in the hopes that somebody else knows how to do this ( @sxa ...? 👀 )

[Willsparker]

• Ubuntu1804.arm64.dsk:

The disk format is raw. I was able to extend it with qemu-img resize <> +10G though, and it is showing up in lsblk:

NAME                             MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda                              252:0    0   30G  0 disk 
├─vda1                           252:1    0  512M  0 part /boot/efi
└─vda2                           252:2    0 19.5G  0 part 
  ├─ubuntu--18--arm64--vg-root   253:0    0 18.6G  0 lvm  /
  └─ubuntu--18--arm64--vg-swap_1 253:1    0  976M  0 lvm  [SWAP]

However, whenever I run any fdisk commands, I get
GPT PMBR size mismatch (41943039 != 62914559) will be corrected by w(rite)., which fails when I exit, with fdisk: failed to write disklabel: Invalid argument. For some reason, fdisk can't fix this, but parted can. Running parted -l will prompt a fix/ignore check. After this, I followed the instructions from this link - created a new partition with fdisk (called /dev/vda3) with the remaining space, created the physical volume pvcreate /dev/vda3, extended the volume group with vgextend /dev/ubuntu-18-arm64-vg /dev/vda3. From there, I extended the LV that was needed lvextend +2568 /dev/ubuntu-18-arm64-vg/root and resized it to the file system : resize2fs /dev/ubuntu-18-arm64-vg/root.
The result is:

root@ubuntu-18-arm64:~# lsblk
NAME                             MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda                              252:0    0   30G  0 disk 
├─vda1                           252:1    0  512M  0 part /boot/efi
├─vda2                           252:2    0 19.5G  0 part 
│ ├─ubuntu--18--arm64--vg-root   253:0    0 28.6G  0 lvm  /
│ └─ubuntu--18--arm64--vg-swap_1 253:1    0  976M  0 lvm  [SWAP]
└─vda3                           252:3    0   10G  0 part 
  └─ubuntu--18--arm64--vg-root   253:0    0 28.6G  0 lvm  /

Which looks alright to me 👍

• Ubuntu18.ppc64le.dsk:

Also uses LVM, and a raw disk image, so I'll do the same process as before, hopefully. Initial lsblk shows (after extending the disk):

NAME                 MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                    8:0    0   25G  0 disk 
├─sda1                 8:1    0    7M  0 part 
└─sda2                 8:2    0   15G  0 part 
  ├─linux--vg-root   253:0    0 14.3G  0 lvm  /
  └─linux--vg-swap_1 253:1    0  676M  0 lvm  [SWAP]
sr0                   11:0    1 1024M  0 rom 

I wonder if sr0 there is going to cause issues ..
Nope, didn't seem to. Having followed the same instructions as before (and instructions on how to extend the LVM here):

root@linux:~# lsblk
NAME                 MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                    8:0    0   25G  0 disk 
├─sda1                 8:1    0    7M  0 part 
├─sda2                 8:2    0   15G  0 part 
│ ├─linux--vg-root   253:0    0 24.3G  0 lvm  /
│ └─linux--vg-swap_1 253:1    0  676M  0 lvm  [SWAP]
└─sda3                 8:3    0   10G  0 part 
  └─linux--vg-root   253:0    0 24.3G  0 lvm  /
sr0                   11:0    1 1024M  0 rom 

• Ubuntu18.S390x.dsk :

This one is a QCOW2 image, so same command as all of them. This one is also LVM, but for some reason, didn't require parted -l.

root@ubuntu:~# lsblk
NAME                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda                   252:0    0   25G  0 disk 
└─vda1                252:1    0   15G  0 part 
  ├─ubuntu--vg-root   253:0    0   14G  0 lvm  /
  └─ubuntu--vg-swap_1 253:1    0  964M  0 lvm  [SWAP]

After the same above process:

NAME                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda                   252:0    0   25G  0 disk 
├─vda1                252:1    0   15G  0 part 
│ ├─ubuntu--vg-root   253:0    0 24.1G  0 lvm  /
│ └─ubuntu--vg-swap_1 253:1    0  964M  0 lvm  [SWAP]
└─vda2                252:2    0   10G  0 part 
  └─ubuntu--vg-root   253:0    0 24.1G  0 lvm  /

I'm glad I/whoever setup the Ubuntu Images, had the foresight to make the partitions LVM :-)