AWX playbook deployment machine audit

[Haroon-Khel]

Using this issue to document the successful, failing and unreachable playbook deployments of the weekly playbook deployments in AWX

[Haroon-Khel]

Unix

Unreachable:
build-digitalocean-centos69-x64-2 - unpingable, might be decommissioned, last build on the machine was 1.5 years ago
build-osuosl-centos74-ppc64le-2 - No longer exists
build-osuosl-ubuntu2204-aarch64-1 - No longer exists, removing from inventory and awx inventory
test-rise-ubuntu2404-riscv64-6 - Missing bastillion key, added ✅
test-marist-sles15-s390x-3 - Scott added the bastillion key onto it, should fix it ✅

Fails:
build-osuosl-centos74-ppc64le-1 - python interpreter - going to decommission, no longer needed
test-aws-rhel76-armv8-1 - python interpreter, fixed by building python3 ✅
test-marist-sles15-s390x-2 - changed python3 path to /root/src/Python-3.8.19/python, fails at Common : Add Devel-Tools repository (SLES15)
test-osuosl-centos74-ppc64le-1 - python interpreter, no longer an issue ✅
test-osuosl-centos74-ppc64le-2 - python interpreter, no longer an issue ✅
test-azure-rhel10-x64-1 - Import EPEL GPG key - gpg: WARNING: no command supplied. likely diff version of gpg
test-aws-rhel8-x64-1 - Enable CodeReady Builder repos on Rhel8Plus - This system has no repositories available through subscriptions.
test-marist-rhel7-s390x-2 - YUM upgrade all packages ( Prior To RHEL8 ) - Could not detect which major revision of dnf is in use
build-marist-rhel8-s390x-1 - YUM upgrade all packages ( RHEL8+ ) - Problem: package podman-4:4.9.4-30.module+el8.10.0+24081+a04d358a.s390x
test-marist-rhel8-s390x-2 - YUM upgrade all packages ( RHEL8+ ) - package podman-4:4.9.4-30.module+el8.10.0+24081+a04d358a.s390x
test-ibmcloud-rhel8-x64-1 - Set Default JRE (RHEL 7 and later) - The task includes an option with an undefined variable.. 'dict object' has no attribute 'lnk_source'
test-ibmcloud-rhel9-x64-1 - Set Default JRE (RHEL 7 and later) - The task includes an option with an undefined variable.. 'dict object' has no attribute 'lnk_source'
test-ibmcloud-ubuntu2204-x64-1 - Run apt-get upgrade - Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 939792 - Killed locked apt process ✅

Success:
test-azure-ubuntu2404-x64-1
test-marist-ubuntu2204-s390x-1
test-marist-ubuntu2404-s390x-1
test-osuosl-ubuntu2404-aarch64-1
test-osuosl-ubuntu2404-ppc64le-1
test-osuosl-ubuntu2404-ppc64le-2
test-osuosl-ubuntu2404-ppc64le-3
test-osuosl-ubuntu2404-ppc64le-4
test-osuosl-ubuntu2404-ppc64le-5
test-rise-ubuntu2404-riscv64-1
test-rise-ubuntu2404-riscv64-2
test-rise-ubuntu2404-riscv64-3
test-rise-ubuntu2404-riscv64-4
test-rise-ubuntu2404-riscv64-5
test-rise-ubuntu2404-riscv64-7
test-rise-ubuntu2404-riscv64-8
test-rise-ubuntu2404-riscv64-9
test-skytap-ubuntu2404-ppc64le-1

Windows

Playbooks are failing to deploy on ANY machine
skipping: no hosts matched

build*win*,test*win* to the limit variable should work ✅

AIX
Unreachable:
build-osuosl-aix72-ppc64-1 - bastillion key and awx is on the machine, likely the same ssh issues we have been seeing when connecting from our own workstations
build-osuosl-aix72-ppc64-2 - Same as -1
test-osuosl-aix72-ppc64-3 - Same as -1
test-osuosl-aix72-ppc64-4 - Same as -1, could try adding an ansible variable into the AWX inventory config of each machine
test-osuosl-aix73-ppc64-1 - was missing awx key ✅

Fails:
test-osuosl-aix72-ppc64-2 - error with python interpreter, installed python3 ✅
test-osuosl-aix72-ppc64-6 - error with python interpreter, installed python3 ✅
test-osuosl-aix72-ppc64-5 - fails at dnf : Remove yum-utils before dnf update
build-osuosl-aix72-ppc64-3 - fails at dnf : Remove yum-utils before dnf update

Success
build-osuosl-aix72-ppc64-4
test-osuosl-aix72-ppc64-1

MacOS
Deploys on test-macincloud-macos1201-x64-1 but fails at

TASK [Common : Upgrade installed packages] *************************************
fatal: [test-macincloud-macos1201-x64-1]: FAILED! => {"changed": false, "msg": "Error: Unexpected method 'appcast' called on Cask adoptopenjdk10.\nFollow the instructions here:\n  https://github.com/Homebrew/homebrew-cask#reporting-bugs"}

Unreachable on test-macincloud-macos1201-x64-2 and test-macstadium-macos1014-x64-1, these machines are offline/decommissioned

Solaris
No longer support Solaris, machines should stop running playbook, clean up inventory of Solaris machines maybe

[Haroon-Khel]

Many of the centos7/8 and rhel7/8 failures are to do with not being able to install python3. It isnt available in the yum/dnf repos

[Haroon-Khel]

For the machines which have a python version that isnt compatible with ansible, i will build python 3.15 from source, store the binary on the machine only to be used for ansible

[Haroon-Khel]

I built python3 (with openssl3) on test-osuosl-centos74-ppc64le-1, lets see if that fixes things

UPDATE: it fixed the python interpreter error, will use this solution for the other nodes which are experiencing the same issue