Home » News & Events » IMMERSION COOLING SOLUTION » Intergated Extension Box

IOMMU Group is not viable

Mar 04, 2024

Hi, I am encountered a issue when i try to attach my pcie device to vfio-pci driver in Jetson Xavier NX development kit.

labuser@lab69:~$ lspci -v

0004:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad1 (rev a1) (prog-if 00 [Normal decode])

Flags: bus master, fast devsel, latency 0, IRQ 24

Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0

I/O behind bridge: 00001000-00001fff [size=4K]

Memory behind bridge: 40000000-400fffff [size=1M]

Prefetchable memory behind bridge: [disabled]

Capabilities:

Kernel driver in use: pcieport

0005:01:00.0 Processing accelerators: Device 1e58:0001 (rev 01)

Subsystem: Allied Telesis, Inc (Wrong ID) Device 0010

Flags: fast devsel, IRQ 255

Memory at 1f40080000 (64-bit, non-prefetchable) [disabled] [size=16K]

Memory at 1f40000000 (64-bit, non-prefetchable) [disabled] [size=512K]

Memory at 1c00000000 (64-bit, prefetchable) [disabled] [size=1M]

Capabilities:

I want to attach pcie device with 1e58:0001 id to vfio-pci driver.

I implemented by following steps:

labuser@lab69:~$ sudo -s

root@lab69:/home/labuser# modprobe vfio-pci

root@lab69:/home/labuser# echo 1e58 0001 > /sys/bus/pci/drivers/vfio-pci/new_id

root@lab69:/home/labuser# lspci -v

0004:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad1 (rev a1) (prog-if 00 [Normal decode])

Flags: bus master, fast devsel, latency 0, IRQ 24

Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0

I/O behind bridge: 00001000-00001fff [size=4K]

Memory behind bridge: 40000000-400fffff [size=1M]

Prefetchable memory behind bridge: [disabled]

Capabilities: [40] Power Management version 3

Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+

Capabilities: [70] Express Root Port (Slot-), MSI 00

Capabilities: [b0] MSI-X: Enable- Count=8 Masked-

Capabilities: [100] Advanced Error Reporting

Capabilities: [148] Secondary PCI Express

Capabilities: [168] Physical Layer 16.0 GT/s
Capabilities: [18c] Lane Margining at the Receiver

Capabilities: [1ac] L1 PM Substates

Capabilities: [1bc] Vendor Specific Information: ID=0002 Rev=4 Len=100
Capabilities: [2bc] Vendor Specific Information: ID=0001 Rev=1 Len=038

Capabilities: [2f4] Data Link Feature
Capabilities: [300] Precision Time Measurement
Capabilities: [30c] Vendor Specific Information: ID=0004 Rev=1 Len=054

Kernel driver in use: pcieport

0004:01:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8822CE 802.11ac PCIe Wireless Network Adapter

Subsystem: AzureWave RTL8822CE 802.11ac PCIe Wireless Network Adapter

Flags: bus master, fast devsel, latency 0, IRQ 270

I/O ports at 1000 [size=256]

Memory at 1740000000 (64-bit, non-prefetchable) [size=64K]

Capabilities: [40] Power Management version 3

Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+

Capabilities: [70] Express Endpoint, MSI 00

Capabilities: [100] Advanced Error Reporting

Capabilities: [148] Device Serial Number 00-e0-4c-ff-fe-c8-22-01

Capabilities: [158] Latency Tolerance Reporting

Capabilities: [160] L1 PM Substates

Kernel driver in use: rtl88x2ce

Kernel modules: rtl8822ce

0005:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad0 (rev a1) (prog-if 00 [Normal decode])

Flags: bus master, fast devsel, latency 0, IRQ 26

Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0

I/O behind bridge: [disabled]

Memory behind bridge: 40000000-400fffff [size=1M]

Prefetchable memory behind bridge: 0000001c00000000-0000001c000fffff [size=1M]

Capabilities: [40] Power Management version 3

Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+

Capabilities: [70] Express Root Port (Slot-), MSI 00

Capabilities: [b0] MSI-X: Enable- Count=8 Masked-

Capabilities: [100] Advanced Error Reporting

Capabilities: [148] Secondary PCI Express

Capabilities: [168] Physical Layer 16.0 GT/s
Capabilities: [190] Lane Margining at the Receiver

Capabilities: [1c0] L1 PM Substates

Capabilities: [1d0] Vendor Specific Information: ID=0002 Rev=4 Len=100
Capabilities: [2d0] Vendor Specific Information: ID=0001 Rev=1 Len=038

Capabilities: [308] Data Link Feature
Capabilities: [314] Precision Time Measurement
Capabilities: [320] Vendor Specific Information: ID=0004 Rev=1 Len=054

Kernel driver in use: pcieport

0005:01:00.0 Processing accelerators: Device 1e58:0001 (rev 01)

Subsystem: Allied Telesis, Inc (Wrong ID) Device 0010

Flags: fast devsel, IRQ 26

Memory at 1f40080000 (64-bit, non-prefetchable) [disabled] [size=16K]

Memory at 1f40000000 (64-bit, non-prefetchable) [disabled] [size=512K]

Memory at 1c00000000 (64-bit, prefetchable) [disabled] [size=1M]

Capabilities: [40] Power Management version 3

Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+

Capabilities: [70] Express Endpoint, MSI 00

Capabilities: [100] Advanced Error Reporting

Capabilities: [148] Secondary PCI Express

Capabilities: [168] Latency Tolerance Reporting

Capabilities: [170] L1 PM Substates

Capabilities: [180] Dynamic Power Allocation

Kernel driver in use: vfio-pci

I attached my device to vfio-pci driver

root@lab69:/home/labuser/usha/vfio_bins# ./a.out

Container opened

open IOMMU group

ERROR: group not viable 0

This is the error i am facing. I checked iommu groups i found 2 pci bridges in that folder.

root@lab69:/home/labuser/usha/vfio_bins# ls /sys/kernel/iommu_groups/7/devices/

0005:00:00.0 0005:01:00.0 141a0000.pcie

What should i do now? Do i need to isolate my device?