Jetson Orin PCIE-NTB function with DMA failed

Hi,

I’m using 2*Jetson Orin Developer Kit and a Microchip Switchtec PCIE Switch to test PCIE NTB function with ntb_perf.c and ntb_hw_switchtec.c(Switchtec PCIE driver) provided by kernel.

My BSP version is JetPack 5.0.1.

For the case of NTB without DMA, it works.

However if I try to test NTB with DMA:

sudo modprobe ntb_perf chunk_order=20 total_order=25 use_dma=1

Then when I started transfer, i got EMEM address decode error like below:

d@d-desktop:/sys/kernel/debug/ntb_perf/0005:01:00.0$ sudo echo 0 > run

[  150.540694] mc-err: (255) csw_axisw: EMEM address decode error

[  150.540991] mc-err:   status = 0x2001008d; hi_addr_reg = 0x00000027 addr = 0x2740400400

[  150.541396] mc-err:   secure: no, access-type: write

[  150.617862] mc-err: (255) csw_axisw: EMEM address decode error

[  150.619921] mc-err:   status = 0x2001008d; hi_addr_reg = 0x00000027 addr = 0x2740400600

[  150.620362] mc-err:   secure: no, access-type: write

[  150.626044] irq 14: nobody cared (try booting with the "irqpoll" option)

[  150.626288] handlers:

[  150.626355] [<0000000068f128fa>] tegra_mcerr_hard_irq threaded [<00000000123ec76b>] tegra_mcerr_thread

[  150.626598] Disabling IRQ #14

[  150.626706] mc-err: (255) csw_axisw: EMEM address decode error

[  150.626862] mc-err:   status = 0x2001008d; hi_addr_reg = 0x00000027 addr = 0x2740400000

[  150.627074] mc-err:   secure: no, access-type: write

[  150.627216] mc-err: (255) csw_axisw: EMEM address decode error

[  150.627372] mc-err:   status = 0x2001008d; hi_addr_reg = 0x00000027 addr = 0x2740400200

[  150.627579] mc-err:   secure: no, access-type: write

[  150.728606] mc-err: Too many MC errors; throttling prints

^C[  179.420942] WARNING: CPU: 1 PID: 125 at drivers/iommu/io-pgtable-arm.c:593 __arm_lpae_unmap+0x380/0x490

[  179.421486] ---[ end trace 4ee791a0b2d23248 ]---

[  179.421668] WARNING: CPU: 1 PID: 125 at drivers/iommu/dma-iommu.c:507 __iommu_dma_unmap+0xf8/0x110

[  179.422028] ---[ end trace 4ee791a0b2d23249 ]---

[  179.422514] WARNING: CPU: 3 PID: 7 at drivers/iommu/io-pgtable-arm.c:593 __arm_lpae_unmap+0x380/0x490

[  179.423004] ---[ end trace 4ee791a0b2d2324a ]---

[  179.423145] WARNING: CPU: 3 PID: 7 at drivers/iommu/dma-iommu.c:507 __iommu_dma_unmap+0xf8/0x110

[  179.423515] ---[ end trace 4ee791a0b2d2324b ]---

[  179.423786] WARNING: CPU: 10 PID: 561 at drivers/iommu/io-pgtable-arm.c:593 __arm_lpae_unmap+0x380/0x490

[  179.424216] ---[ end trace 4ee791a0b2d2324c ]---

[  179.424374] WARNING: CPU: 10 PID: 561 at drivers/iommu/dma-iommu.c:507 __iommu_dma_unmap+0xf8/0x110

[  179.424715] ---[ end trace 4ee791a0b2d2324d ]---

[  179.424896] WARNING: CPU: 10 PID: 614 at drivers/iommu/io-pgtable-arm.c:593 __arm_lpae_unmap+0x380/0x490

[  179.426872] ---[ end trace 4ee791a0b2d2324e ]---

[  179.427078] WARNING: CPU: 10 PID: 614 at drivers/iommu/dma-iommu.c:507 __iommu_dma_unmap+0xf8/0x110

[  179.436111] ---[ end trace 4ee791a0b2d2324f ]---

[  179.441455] WARNING: CPU: 5 PID: 1834 at drivers/iommu/io-pgtable-arm.c:593 __arm_lpae_unmap+0x380/0x490

[  179.450438] ---[ end trace 4ee791a0b2d23250 ]---

[  179.454958] WARNING: CPU: 5 PID: 1834 at drivers/iommu/dma-iommu.c:507 __iommu_dma_unmap+0xf8/0x110

[  179.464068] ---[ end trace 4ee791a0b2d23251 ]---

[  179.468845] WARNING: CPU: 6 PID: 1835 at drivers/iommu/io-pgtable-arm.c:593 __arm_lpae_unmap+0x380/0x490

[  179.478324] ---[ end trace 4ee791a0b2d23252 ]---

[  179.482872] WARNING: CPU: 6 PID: 1835 at drivers/iommu/dma-iommu.c:507 __iommu_dma_unmap+0xf8/0x110

[  179.491944] ---[ end trace 4ee791a0b2d23253 ]---

[  179.497313] WARNING: CPU: 4 PID: 1836 at drivers/iommu/io-pgtable-arm.c:593 __arm_lpae_unmap+0x380/0x490

[  179.506245] ---[ end trace 4ee791a0b2d23254 ]---

[  179.510794] WARNING: CPU: 4 PID: 1836 at drivers/iommu/dma-iommu.c:507 __iommu_dma_unmap+0xf8/0x110

[  179.519860] ---[ end trace 4ee791a0b2d23255 ]---

It seems the initialization of my NTB drivers with DMA worked fine on BOTH side:

d@d-desktop:/sys/kernel/debug/ntb_perf/0005:01:00.0$ sudo cat info

    Performance measuring tool info:



Local port 0, Global index 0

Test status: idle

Port 0 (0), Global index 0:

        Link status: up

        Out buffer addr 0xffff800022400000

        Out buff phys addr 0x0000002740400000[p]

        Out buffer size 0x0000000000400000

        Out buffer xlat 0x00000000ff800000[p]

        In buffer addr 0xffff8000209ff000

        In buffer size 0x0000000000400000

        In buffer xlat 0x00000000ff800000[p]

And all of the source codes(including JetPack 5.0.1 and all drivers) are exactly the same on both side.