GPUs without DRM are not detected in Linux

[mlorenzofr]

Nvidia AI cards (Tesla, A100, A40, A2) are not detected as GPU elements. The reason is that these controllers are 3D controller devices (0x0302) and don't have DRM interface.

For example:

$ lspci -nnvs 0b:00.0
0b:00.0 VGA compatible controller [0300]: Matrox Electronics Systems Ltd. G200eR2 [102b:0534] (rev 01) (prog-if 00 [VGA controller])
        DeviceName: Embedded Video
        Subsystem: Dell Device [1028:0600]
        Flags: bus master, medium devsel, latency 0, IRQ 19, NUMA node 0, IOMMU group 30
        Memory at 90000000 (32-bit, prefetchable) [size=16M]
        Memory at 93000000 (32-bit, non-prefetchable) [size=16K]
        Memory at 92800000 (32-bit, non-prefetchable) [size=8M]
        Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
        Capabilities: [dc] Power Management version 1
        Kernel driver in use: mgag200
        Kernel modules: mgag200

$ lspci -nnvs 04:00.0
04:00.0 3D controller [0302]: NVIDIA Corporation GA100 [A100 PCIe 40GB] [10de:20f1] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:145f]
        Flags: bus master, fast devsel, latency 0, IRQ 162, NUMA node 0, IOMMU group 28
        Memory at 91000000 (32-bit, non-prefetchable) [size=16M]
        Memory at 39000000000 (64-bit, prefetchable) [size=64G]
        Memory at 3b020000000 (64-bit, prefetchable) [size=32M]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] Null
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [c8] MSI-X: Enable+ Count=6 Masked-
        Capabilities: [100] Virtual Channel
        Capabilities: [258] L1 PM Substates
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [420] Advanced Error Reporting
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] Secondary PCI Express
        Capabilities: [bb0] Physical Resizable BAR
        Capabilities: [bcc] Single Root I/O Virtualization (SR-IOV)
        Capabilities: [c14] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [c1c] Physical Layer 16.0 GT/s <?>
        Capabilities: [d00] Lane Margining at the Receiver <?>
        Capabilities: [e00] Data Link Feature <?>
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau

$ ls -l /sys/class/drm
total 0
lrwxrwxrwx. 1 root root    0 Aug  5 11:10 card0 -> ../../devices/pci0000:00/0000:00:1c.7/0000:08:00.0/0000:09:00.0/0000:0a:00.0/0000:0b:00.0/drm/card0
lrwxrwxrwx. 1 root root    0 Aug  5 11:10 card0-VGA-1 -> ../../devices/pci0000:00/0000:00:1c.7/0000:08:00.0/0000:09:00.0/0000:0a:00.0/0000:0b:00.0/drm/card0/card0-VGA-1
-r--r--r--. 1 root root 4096 Sep 16 07:23 version

In this case we should add a patch (e.g. detect GPUs by PCI base class or PCI subclass) to detect these cards as GPUs, or add different function/parameters for these scenarios.

Going futher, Intel Gaudi cards and other vendor classified their hardware as processing accelerators (PCI class 0x12). Maybe we can discuss if it should detected as a GPU (and then added to the fix) or detected in a different hardware category.

[jaypipes]

@mlorenzofr Hi! Thanks so much for reporting this issue! I will look into this ASAP.

[jaypipes]

@mlorenzofr may we ask you to do us a favour for debugging purposes? Can you take a ghw snapshot of the machine you are seeing this issue on and either attach the resulting file to this GH issue or email me at jaypipes at gmail dot com?

You can create a ghw snapshot by git cloning the ghw repo and running the following command:

go run cmd/ghw-snapshot/main.go -o mlorenzofr.tar.gz 

[jaypipes]

@ffromani and I were chatting a bit about this on Slack... I'm thinking a more generic pkg/accelerator might be needed. We can address these non-DRM things like the Tesla GPUs and FPGAs in a more generic pkg/accelerator.

[ffromani]

random initial braindump:

• a GPU is an accelerator
• an accelerator is not necessarily a GPU (could be a FPGA)
• I think overlaps between pkg/accelerator and pkg/gpu is not a big deal, kinda like having overlap between pkg/gpu and pkg/pci

[mlorenzofr]

@mlorenzofr may we ask you to do us a favour for debugging purposes? Can you take a ghw snapshot of the machine you are seeing this issue on and either attach the resulting file to this GH issue or email me at jaypipes at gmail dot com?

You can create a ghw snapshot by git cloning the ghw repo and running the following command:

go run cmd/ghw-snapshot/main.go -o mlorenzofr.tar.gz 

Sure, here is the snapshot:
mlorenzofr.tar.gz

[mlorenzofr]

random initial braindump:

* a GPU is an accelerator

* an accelerator is not necessarily a GPU (could be a FPGA)

* I _think_ overlaps between `pkg/accelerator` and `pkg/gpu` is not a big deal, kinda like having overlap between `pkg/gpu` and `pkg/pci`

That's a good proposal.
How should these devices be detected?
May we use the PCI class/subclass?

[ffromani]

random initial braindump:

* a GPU is an accelerator

* an accelerator is not necessarily a GPU (could be a FPGA)

* I _think_ overlaps between `pkg/accelerator` and `pkg/gpu` is not a big deal, kinda like having overlap between `pkg/gpu` and `pkg/pci`

That's a good proposal. How should these devices be detected? May we use the PCI class/subclass?

we need to figure out in this issue :)
the simplest approach is to have a general filtering over known PCI devices plus perhaps an overridable filter.
ghw can:

• list all PCI devices which it's certain they are accelerators. This list has to be very selective to avoid false negatives. IOW ghw will return devices it's totally positive they are accelerators, but it can miss something
• accept a filter to inject custom user-provided filters. If a device which ghw doesn't consider an accelerator passes the filtering, it will added to the returned list. The rationale is that ghw doing so acknowledges the user knows best about their specific domain and thus which accelerator needs, but still takes care of the boring mundane details (listing PCIs and so forth)

[mlorenzofr]

According to the kernel documentation accelerator devices will be exposed to userspace under /sys/class/accel/accel*/. This is true for Intel Gaudi devices, but not for Nvidia.

I will try to summarize it in a table:

model	vendor	PCI base class	PCI subclass	DRM subsystem	accel subsystem
20f1 (A100)	10de (Nvidia)	03 (display controller)	02 (3D controller)	no	no
0118 (tesla P100)	10de (Nvidia)	03 (display controller)	02 (3D controller)	no	no
1020 (Gaudi2)	1da3 (Habana Labs)	12 (processing accelerators)	00 (processing accelerators)	no	yes
1060 (Gaudi3)	1da3 (Habana Labs)	12 (processing accelerators)	00 (processing accelerators)	no	yes
74a1 (Instinct MI300X)	1002 (AMD/ATI)	12 (processing accelerators)	00 (processing accelerators)	?	?
740f (Instinct MI210)	1002 (AMD/ATI)	03 (display controller)	80 (display controller)	?	?

[mrlake]

I just wanted to add this is true of a Tesla P100 GPU and adding support for headless GPU's would be great. I took a snapshot of my system as per the process above (of an LXD vm with a pci passthrough for a tesla p100) incase it provides any additional insight.
nunet-tesla-gpu-test.tar.gz

[jaypipes]

Hey, thanks @mrlake! Much appreciated :) I'll try to get to this in the next few weeks.

[ffromani]

it seems there's consensus about this feature (= new accel pkg in ghw) being worthy. Great! @mlorenzofr are you interested in taking a stab at the implementation?

[mlorenzofr]

it seems there's consensus about this feature (= new accel pkg in ghw) being worthy. Great! @mlorenzofr are you interested in taking a stab at the implementation?

Yes, I'd be happy to do so, although reviews will be appreciated 😅
I've been a bit busy lately, are you hurry with the implementation?

[jaypipes]

@mlorenzofr no rush, no :) feel free to take your time and rely on @ffromani and myself for reviews/feedback.