Home » News & Events » GPU AI

When disable PCIe SMMU, happened error when use function: dma_map_single(xx,xx,xx,xx)

Mar 04, 2024

Dear Sir/Madam:

Question description:
1. We made a custom carrier board with FPGA just like Tx2 used before.
2. When used Tx2 with FPGA, we disable PCIe’s mmu with DMA, so here I also disable PCIe’s(pcie@14160000) mmu. what i modify is remove 4 lines from device tree, modify code is below list:

/* C4 X4 */

pcie_c4_rp: pcie@14160000 {

compatible = “nvidia,tegra234-pcie”, “snps,dw-pcie”;

…

//del iommus = <&smmu_niso0 TEGRA_SID_NISO0_PCIE4>;

//del iommu-map = <0x0 &smmu_niso0 TEGRA_SID_NISO0_PCIE4 0x1000>;

msi-parent = <&gic_v2m TEGRA_SID_NISO0_PCIE4>;

msi-map = <0x0 &gic_v2m TEGRA_SID_NISO0_PCIE4 0x1000>;

//del dma-coherent;

//del iommu-map-mask = <0x0>;

…

nvidia,dl-feature-cap = <0x2f8>;

nvidia,ptm-cap-off = <0x304>;

};

3. Our driver can work well with TX2 device, install driver modified above with orin NX, when call function dma_map_single in kernel will happen a kernel crash(log is added below && sysfs cannot find 14160000.pcie).

4. Please help to check if any more modifications needed, thanks

btw: cache_inv_range only call function dma_map_single(xx,xx,xx,xx)

void cache_inv_range(void *start,int len) {

dma_map_single(NULL, start, len, DMA_FROM_DEVICE);

}

[ 1006.649782] arm-smmu 12000000.iommu: Unexpected global fault, this could be serious

[ 1006.664249] arm-smmu 12000000.iommu: GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00001013, GFSYNR2 0x00000000

[ 1006.676516] arm-smmu 12000000.iommu: Unexpected global fault, this could be serious

[ 1006.684391] arm-smmu 12000000.iommu: GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00001013, GFSYNR2 0x00000000

[ 1006.696645] arm-smmu 12000000.iommu: Unexpected global fault, this could be serious

[ 1006.704511] arm-smmu 12000000.iommu: GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00001013, GFSYNR2 0x00000000

[ 1006.716753] arm-smmu 12000000.iommu: Unexpected global fault, this could be serious

[ 1006.724621] arm-smmu 12000000.iommu: GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00001013, GFSYNR2 0x00000000

[ 1006.736853] arm-smmu 12000000.iommu: Unexpected global fault, this could be serious

[ 1006.744723] arm-smmu 12000000.iommu: GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00001013, GFSYNR2 0x00000000

[ 1006.756968] arm-smmu 12000000.iommu: Unexpected global fault, this could be serious

[ 1006.764834] arm-smmu 12000000.iommu: GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00001013, GFSYNR2 0x00000000

[ 1006.777079] arm-smmu 12000000.iommu: Unexpected global fault, this could be serious

[ 1006.784947] arm-smmu 12000000.iommu: GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00001013, GFSYNR2 0x00000000

[ 1006.797189] arm-smmu 12000000.iommu: Unexpected global fault, this could be serious

[ 1006.805064] arm-smmu 12000000.iommu: GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00001013, GFSYNR2 0x00000000

[ 1006.817309] arm-smmu 12000000.iommu: Unexpected global fault, this could be serious

[ 1006.825184] arm-smmu 12000000.iommu: GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00001013, GFSYNR2 0x00000000

[ 1006.837423] arm-smmu 12000000.iommu: Unexpected global fault, this could be serious

[ 1006.845291] arm-smmu 12000000.iommu: GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00001013, GFSYNR2 0x00000000

[ 1007.204834] mc-err: vpr base=0:0, size=0, ctrl=1, override:(201803c6, b9ee11c1, 1, 0)

[ 1007.490292] .

[ 1007.491078] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000258

[ 1007.491081] Mem abort info:

[ 1007.491082] ESR = 0x96000006

[ 1007.491085] EC = 0x25: DABT (current EL), IL = 32 bits

[ 1007.491087] SET = 0, FnV = 0

[ 1007.491088] EA = 0, S1PTW = 0

[ 1007.491088] Data abort info:

[ 1007.491090] ISV = 0, ISS = 0x00000006

[ 1007.491090] CM = 0, WnR = 0

[ 1007.491094] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000110f0e000

[ 1007.491096] [0000000000000258] pgd=0000000147efa003, p4d=0000000147efa003, pud=0000000147ebc003, pmd=0000000000000000

[ 1007.491105] Internal error: Oops: 96000006 [#1] PREEMPT SMP

[ 1007.491110] Modules linked in: OrinNxFpgaDrv(OE) nvidia_modeset(O) fuse md4 md5 cifs libdes lzo_rle lzo_compress zram ramoops reed_solomon loop snd_soc_tegra186_asrc snd_soc_tegra210_ope snd_soc_tegra186_arad snd_soc_tegra210_iqc snd_soc_tegra186_dspk snd_soc_tegra210_mvc snd_soc_tegra210_dmic snd_soc_tegra210_afc snd_soc_tegra210_adx snd_soc_tegra210_amx snd_soc_tegra210_mixer snd_soc_tegra210_i2s snd_soc_tegra210_sfc snd_soc_tegra210_admaif snd_soc_tegra_pcm aes_ce_blk crypto_simd cryptd aes_ce_cipher ghash_ce sha2_ce sha256_arm64 sha1_ce snd_soc_spdif_tx snd_soc_tegra_machine_driver snd_soc_tegra210_adsp snd_soc_tegra210_ahub userspace_alert snd_soc_tegra_utils r8168 snd_soc_simple_card_utils snd_hda_codec_hdmi nvadsp tegra_bpmp_thermal tegra210_adma snd_hda_tegra snd_hda_codec snd_hda_core r8169 realtek spi_tegra114 nvidia(O) binfmt_misc ina3221 pwm_fan nvgpu nvmap ip_tables x_tables [last unloaded: fpgaConfigIo]

[ 1007.491218] CPU: 1 PID: 2268 Comm: hwTest Tainted: G OE 5.10.104-tegra #1

[ 1007.491220] Hardware name: NVIDIA Orin NX Developer Kit (DT)

[ 1007.491223] pstate: 60400009 (nZCv daif +PAN -UAO -TCO BTYPE=–)

[ 1007.491239] pc : dma_map_page_attrs+0x54/0x240

[ 1007.491240] lr : dma_map_page_attrs+0x40/0x240

[ 1007.491242] sp : ffff800013ffbca0

[ 1007.491243] x29: ffff800013ffbca0 x28: ffff7b87c4984880

[ 1007.491245] x27: 0000000000000000 x26: 0000000000000000

[ 1007.491247] x25: 0000000000208000 x24: ffffffee1be10000

[ 1007.491249] x23: 0000000000000002 x22: 0000000000000000

[ 1007.491251] x21: ffff7b87c4984880 x20: 0000000000000000

[ 1007.491253] x19: 0000000000000000 x18: 0000000000000000

[ 1007.491255] x17: 0000000000000000 x16: ffffd75786aa1bd0

[ 1007.491257] x15: 0000000000000000 x14: 0000000000000000

[ 1007.491259] x13: 0000000000000000 x12: 0000000000000000

[ 1007.491261] x11: 0000000000000000 x10: 0000000000000000

[ 1007.491263] x9 : 0000000000000000 x8 : 0000000000000000

[ 1007.491265] x7 : 0000000000000000 x6 : 0000000000000002

[ 1007.491267] x5 : 0000000000000000 x4 : 0000000000000002

[ 1007.491269] x3 : 0000000000208000 x2 : ffffd75786b2e170

[ 1007.491271] x1 : 0000000000000000 x0 : ffffd75786b2e170

[ 1007.491274] Call trace:

[ 1007.491276] dma_map_page_attrs+0x54/0x240

[ 1007.491292] cache_inv_range+0x60/0xb0 [OrinNxFpgaDrv]

[ 1007.491297] acq_read+0xb4/0x1b0 [OrinNxFpgaDrv]

[ 1007.491306] vfs_read+0xb4/0x1c0

[ 1007.491309] ksys_read+0x7c/0x110

[ 1007.491310] __arm64_sys_read+0x28/0x40

[ 1007.491321] el0_svc_common.constprop.0+0x80/0x1d0

[ 1007.491324] do_el0_svc+0x38/0xb0

[ 1007.491333] el0_svc+0x1c/0x30

[ 1007.491335] el0_sync_handler+0xa8/0xb0

[ 1007.491339] el0_sync+0x16c/0x180

[ 1007.491344] Code: f9002fe1 d2800001 71000aff 54000ac8 (f9412e62)

[ 1007.491361] —[ end trace 7f91c0ad1e1bf9b1 ]—

[ 1007.496198] Kernel panic - not syncing: Oops: Fatal exception

[ 1007.771752] SMP: stopping secondary CPUs

[ 1007.776889] mc-err: (255) csw_pcie4w: MC request violates VPR requirements

[ 1007.778467] Kernel Offset: 0x575776970000 from 0xffff800010000000

[ 1007.778468] PHYS_OFFSET: 0xffff847980000000

[ 1007.778470] CPU features: 0x0040006,4a80aa38

[ 1007.778472] Memory Limit: none

[ 1007.856382] —[ end Kernel panic - not syncing: Oops: Fatal exception ]—

iommu value: