Build error on Raspberry Pi 4

[xuancong84]

This is a very useful tool because it is not easy to rebuild the kernel without kernel config parameters. All you need to do is to apt install Linux kernel header then build this module.

However, on Raspberry Pi 4, the build process fails with the following error:

root@raspberrypi:~/kexec-mod/kernel# make KDIR=/usr/src/linux-headers-5.10.63-v8+/
make -C /usr/src/linux-headers-5.10.63-v8+/ M=/root/kexec-mod/kernel modules
make[1]: Entering directory '/usr/src/linux-headers-5.10.63-v8+'
  CC [M]  /root/kexec-mod/kernel/arch/arm64/idmap.o
/root/kexec-mod/kernel/arch/arm64/idmap.c: In function ‘kexec_pa_symbol’:
/root/kexec-mod/kernel/arch/arm64/idmap.c:118:19: error: passing argument 1 of ‘pud_offset’ from incompatible pointer type [-Werror=incompatible-pointer-types]
  pud = pud_offset(pgd , va);
                   ^~~
In file included from ./arch/arm64/include/asm/pgtable-types.h:50,
                 from ./arch/arm64/include/asm/page.h:16,
                 from ./arch/arm64/include/asm/proc-fns.h:14,
                 from ./arch/arm64/include/asm/pgtable.h:9,
                 from /root/kexec-mod/kernel/arch/arm64/idmap.c:15:
./include/asm-generic/pgtable-nopud.h:42:40: note: expected ‘p4d_t *’ {aka ‘struct <anonymous> *’} but argument is of type ‘pgd_t *’ {aka ‘struct <anonymous> *’}
 static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address)
                                 ~~~~~~~^~~
cc1: some warnings being treated as errors
make[3]: *** [scripts/Makefile.build:280: /root/kexec-mod/kernel/arch/arm64/idmap.o] Error 1
make[2]: *** [scripts/Makefile.build:497: /root/kexec-mod/kernel/arch/arm64] Error 2
make[1]: *** [Makefile:1825: /root/kexec-mod/kernel] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-5.10.63-v8+'
make: *** [Makefile:15: module] Error 2

[fabianishere]

Hi,

I have only tested the module to work for Linux kernels up to 4.18. It seems that in more newer versions, the kernel developers have unified the page directory representation, requiring changes to my page walk implementation.

Could you check whether the following patch resolves the issue:

diff --git a/kernel/arch/arm64/idmap.c b/kernel/arch/arm64/idmap.c
index c644914..df87920 100644
--- a/kernel/arch/arm64/idmap.c
+++ b/kernel/arch/arm64/idmap.c
@@ -105,6 +105,7 @@ phys_addr_t kexec_pa_symbol(void *ptr)
 	unsigned long va = (unsigned long) ptr;
 	unsigned long page_offset;
 	pgd_t *pgd;
+	p4d_t* p4d;
 	pud_t *pud;
 	pmd_t *pmd;
 	pte_t *ptep, pte;
@@ -115,7 +116,12 @@ phys_addr_t kexec_pa_symbol(void *ptr)
 		return 0;
 	}
 
-	pud = pud_offset(pgd , va);
+	p4d = p4d_offset(pgd, vmpage);
+	if (p4d_none(*p4d) || p4d_bad(*p4d)) {
+		return 0;
+	}
+
+	pud = pud_offset(p4d , va);
 	if (pud_none(*pud) || pud_bad(*pud)) {
 		return 0;
 	}

I recommend first testing the module in QEMU though, since it does some pretty hacky things and I cannot guarantee it still working correctly in Linux 5.10.

[xuancong84]

Thanks for the quick reply! In addition to this error, there are still tones of other errors preventing a successful compilation:

1. Deprecated symbols ENTRY() and ENDPROC()

/home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/arch/arm64/cpu-reset.S: Assembler messages:
/home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/arch/arm64/cpu-reset.S:34: Error: unknown mnemonic `entry' -- `entry(__cpu_soft_restart)'
/home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/arch/arm64/cpu-reset.S:57: Error: unknown mnemonic `endproc' -- `endproc(__cpu_soft_restart)'
make[3]: *** [scripts/Makefile.build:361: /home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/arch/arm64/cpu-reset.o] Error 1
make[2]: *** [scripts/Makefile.build:497: /home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/arch/arm64] Error 2
make[1]: *** [Makefile:1825: /home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel] Error 2

This one is easy to resolve, just replace by SYM_FUNC_START and SYM_FUNC_END

2. A few missing links

/home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/arch/arm64/machine_kexec.c:2:10: fatal error: orig/machine_kexec.c: No such file or directory
/home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/kexec.c:8:10: fatal error: orig/kexec.c: No such file or directory

This one also easy to solve, just download the full kernel source and make the correct soft link

3. Tons of symbol not found

ERROR: modpost: "crashk_res" [/home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/arch/arm64/kexec_mod_arm64.ko] undefined!
ERROR: modpost: "idmap_pg_dir" [/home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/arch/arm64/kexec_mod_arm64.ko] undefined!
ERROR: modpost: "reserved_pg_dir" [/home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/arch/arm64/kexec_mod_arm64.ko] undefined!
ERROR: modpost: "kallsyms_lookup_name" [/home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/arch/arm64/kexec_mod_arm64.ko] undefined!
ERROR: modpost: "swapper_pg_dir" [/home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/arch/arm64/kexec_mod_arm64.ko] undefined!
ERROR: modpost: "kimage_file_post_load_cleanup" [/home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/kexec_mod.ko] undefined!
ERROR: modpost: "kallsyms_lookup_name" [/home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/kexec_mod.ko] undefined!

4. Not sure.

All these making the project hopeless for the new kernel -:(

[xuancong84]

diff --git a/kernel/arch/arm64/idmap.c b/kernel/arch/arm64/idmap.c
index c644914..df87920 100644
--- a/kernel/arch/arm64/idmap.c
+++ b/kernel/arch/arm64/idmap.c
@@ -105,6 +105,7 @@ phys_addr_t kexec_pa_symbol(void *ptr)
 	unsigned long va = (unsigned long) ptr;
 	unsigned long page_offset;
 	pgd_t *pgd;
+	p4d_t* p4d;
 	pud_t *pud;
 	pmd_t *pmd;
 	pte_t *ptep, pte;
@@ -115,7 +116,12 @@ phys_addr_t kexec_pa_symbol(void *ptr)
 		return 0;
 	}
 
-	pud = pud_offset(pgd , va);
+	p4d = p4d_offset(pgd, vmpage);
+	if (p4d_none(*p4d) || p4d_bad(*p4d)) {
+		return 0;
+	}
+
+	pud = pud_offset(p4d , va);
 	if (pud_none(*pud) || pud_bad(*pud)) {
 		return 0;
 	}

Btw, your patch is also problematic because:
kexec-mod/kernel/arch/arm64/idmap.c:119:24: error: ‘vmpage’ undeclared (first use in this function); did you mean ‘page’?

[fabianishere]

Some of these issues are indeed pretty easy to fix. However, this big issue is that kallsyms_lookup_name is not exported in newer kernels.

[JoelLinn]

Doesn't work with 5.4 indeed (not rpi but same issues)

make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- KDIR=~/Documents/dev/kernel-android-5.4 
make -C /home/joel/Documents/dev/kernel-android-5.4 M=/home/joel/Documents/dev/kexec-mod/kernel modules
make[1]: Entering directory '/home/joel/Documents/dev/kernel-android-5.4'
  Building modules, stage 2.
  MODPOST 2 modules
ERROR: "idmap_pg_dir" [/home/joel/Documents/dev/kexec-mod/kernel/arch/arm64/kexec_mod_arm64.ko] undefined!
ERROR: "reserved_pg_dir" [/home/joel/Documents/dev/kexec-mod/kernel/arch/arm64/kexec_mod_arm64.ko] undefined!
ERROR: "kallsyms_lookup_name" [/home/joel/Documents/dev/kexec-mod/kernel/arch/arm64/kexec_mod_arm64.ko] undefined!
ERROR: "swapper_pg_dir" [/home/joel/Documents/dev/kexec-mod/kernel/arch/arm64/kexec_mod_arm64.ko] undefined!
ERROR: "kallsyms_lookup_name" [/home/joel/Documents/dev/kexec-mod/kernel/kexec_mod.ko] undefined!
make[2]: *** [scripts/Makefile.modpost:114: __modpost] Error 1
make[1]: *** [Makefile:1729: modules] Error 2
make[1]: Leaving directory '/home/joel/Documents/dev/kernel-android-5.4'
make: *** [Makefile:15: module] Error 2

[thesaadmemon]

@fabianishere is it confirmed to work for 4.18? Thank you!

[fabianishere]

@thesaadmemon It has worked with 4.18 ARM64 kernels, but easiest would be to try building the module against your kernel configuration and checking if it runs.