The ethernet interface over PCIe(endpoint mode) not work
Mar 04, 2024
Hello,
I used two Jetson AGX Xavier DevKit and connected by PCIe, and follow the guide below:
https://docs.nvidia.com/jetson/archives/r35.2.1/DeveloperGuide/text/SD/Communications/PcieEndpointMode.html
I tried to bring up the ethernet interface over PCIe, and the eth interface has appeared both on the ep and rc sides. But they can’t communicate by the eth interface, even the static IP address assigned.
Jetson Linux : 35.2.1
---- EP log ----
root@tegra-ubuntu:/sys/kernel/config/pci_ep# dmesg | grep -i pci
[ 0.739560] PCI: CLS 0 bytes, default 64
[ 2.772446] ohci-pci: OHCI PCI platform driver
[ 4.425390] tegra194-pcie 141a0000.pcie_ep: Adding to iommu group 7
[ 4.432117] tegra194-pcie 141a0000.pcie_ep: Failed to get PERST GPIO: -517
[ 4.432131] tegra194-pcie 141a0000.pcie_ep: Failed to parse device tree: -517
[ 4.432679] tegra194-pcie 14180000.pcie: Adding to iommu group 8
[ 4.441612] tegra194-pcie 14180000.pcie: host bridge /pcie@14180000 ranges:
[ 4.443151] tegra194-pcie 14180000.pcie: IO 0x0038100000..0x00381fffff -> 0x0038100000
[ 4.451865] tegra194-pcie 14180000.pcie: MEM 0x1800000000..0x1b3fffffff -> 0x1800000000
[ 4.460116] tegra194-pcie 14180000.pcie: MEM 0x1b40000000..0x1bffffffff -> 0x0040000000
[ 5.567821] tegra194-pcie 14180000.pcie: Phy link never came up
[ 5.568360] tegra194-pcie 14180000.pcie: PCI host bridge to bus 0000:00
[ 5.568574] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 5.568787] pci_bus 0000:00: root bus resource [io 0x0000-0xfffff] (bus address [0x38100000-0x381fffff])
[ 5.569149] pci_bus 0000:00: root bus resource [mem 0x1800000000-0x1b3fffffff pref]
[ 5.569388] pci_bus 0000:00: root bus resource [mem 0x1b40000000-0x1bffffffff] (bus address [0x40000000-0xffffffff])
[ 5.569914] pci 0000:00:00.0: [10de:1ad0] type 01 class 0x060400
[ 5.570386] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
[ 5.597770] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[ 5.598121] pci 0000:00:00.0: Max Payload Size set to 256/ 256 (was 256), Max Read Rq 512
[ 5.600490] pcieport 0000:00:00.0: Adding to iommu group 8
[ 5.602276] pcieport 0000:00:00.0: PME: Signaling with IRQ 25
[ 5.603628] pcieport 0000:00:00.0: AER: enabled with IRQ 25
[ 5.605418] pci_bus 0000:01: busn_res: [bus 01-ff] is released
[ 5.606399] pci 0000:00:00.0: Removing from iommu group 8
[ 5.606717] pci_bus 0000:00: busn_res: [bus 00-ff] is released
[ 5.611223] tegra194-pcie 14100000.pcie: Adding to iommu group 9
[ 5.616312] tegra194-pcie 14100000.pcie: host bridge /pcie@14100000 ranges:
[ 5.616686] tegra194-pcie 14100000.pcie: IO 0x0030100000..0x00301fffff -> 0x0030100000
[ 5.617080] tegra194-pcie 14100000.pcie: MEM 0x1200000000..0x122fffffff -> 0x1200000000
[ 5.617459] tegra194-pcie 14100000.pcie: MEM 0x1230000000..0x123fffffff -> 0x0040000000
[ 5.731821] tegra194-pcie 14100000.pcie: Link up
[ 5.743472] tegra194-pcie 14100000.pcie: PCI host bridge to bus 0001:00
[ 5.743693] pci_bus 0001:00: root bus resource [bus 00-ff]
[ 5.743921] pci_bus 0001:00: root bus resource [io 0x100000-0x1fffff] (bus address [0x30100000-0x301fffff])
[ 5.744229] pci_bus 0001:00: root bus resource [mem 0x1200000000-0x122fffffff pref]
[ 5.744485] pci_bus 0001:00: root bus resource [mem 0x1230000000-0x123fffffff] (bus address [0x40000000-0x4fffffff])
[ 5.744901] pci 0001:00:00.0: [10de:1ad2] type 01 class 0x060400
[ 5.745361] pci 0001:00:00.0: PME# supported from D0 D3hot D3cold
[ 5.758074] pci 0001:01:00.0: [1b4b:9171] type 00 class 0x010601
[ 5.758490] pci 0001:01:00.0: reg 0x10: [io 0x8000-0x8007]
[ 5.758795] pci 0001:01:00.0: reg 0x14: [io 0x8040-0x8043]
[ 5.759064] pci 0001:01:00.0: reg 0x18: [io 0x8100-0x8107]
[ 5.759341] pci 0001:01:00.0: reg 0x1c: [io 0x8140-0x8143]
[ 5.759610] pci 0001:01:00.0: reg 0x20: [io 0x800000-0x80000f]
[ 5.759983] pci 0001:01:00.0: reg 0x24: [mem 0x00900000-0x009001ff]
[ 5.760280] pci 0001:01:00.0: reg 0x30: [mem 0xd0000000-0xd000ffff pref]
[ 5.761093] pci 0001:01:00.0: PME# supported from D3hot
[ 5.783190] pci 0001:00:00.0: BAR 14: assigned [mem 0x1230000000-0x12300fffff]
[ 5.783567] pci 0001:00:00.0: BAR 13: assigned [io 0x100000-0x100fff]
[ 5.783863] pci 0001:01:00.0: BAR 6: assigned [mem 0x1230000000-0x123000ffff pref]
[ 5.784115] pci 0001:01:00.0: BAR 5: assigned [mem 0x1230010000-0x12300101ff]
[ 5.786058] pci 0001:01:00.0: BAR 4: assigned [io 0x100000-0x10000f]
[ 5.792583] pci 0001:01:00.0: BAR 0: assigned [io 0x100010-0x100017]
[ 5.799154] pci 0001:01:00.0: BAR 2: assigned [io 0x100018-0x10001f]
[ 5.805403] pci 0001:01:00.0: BAR 1: assigned [io 0x100020-0x100023]
[ 5.811787] pci 0001:01:00.0: BAR 3: assigned [io 0x100024-0x100027]
[ 5.818311] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[ 5.823418] pci 0001:00:00.0: bridge window [io 0x100000-0x100fff]
[ 5.830233] pci 0001:00:00.0: bridge window [mem 0x1230000000-0x12300fffff]
[ 5.837565] pci 0001:00:00.0: Max Payload Size set to 256/ 256 (was 256), Max Read Rq 512
[ 5.845997] pci 0001:01:00.0: Max Payload Size set to 256/ 512 (was 128), Max Read Rq 512
[ 5.855667] pcieport 0001:00:00.0: Adding to iommu group 9
[ 5.860656] pcieport 0001:00:00.0: PME: Signaling with IRQ 27
[ 5.866407] pcieport 0001:00:00.0: AER: enabled with IRQ 27
[ 5.873162] tegra194-pcie 14140000.pcie: Adding to iommu group 10
[ 5.880330] tegra194-pcie 14140000.pcie: host bridge /pcie@14140000 ranges:
[ 5.884542] tegra194-pcie 14140000.pcie: IO 0x0034100000..0x00341fffff -> 0x0034100000
[ 5.893093] tegra194-pcie 14140000.pcie: MEM 0x1280000000..0x12afffffff -> 0x1280000000
[ 5.901650] tegra194-pcie 14140000.pcie: MEM 0x12b0000000..0x12bfffffff -> 0x0040000000
[ 7.003886] tegra194-pcie 14140000.pcie: Phy link never came up
[ 7.004345] tegra194-pcie 14140000.pcie: PCI host bridge to bus 0003:00
[ 7.004577] pci_bus 0003:00: root bus resource [bus 00-ff]
[ 7.004740] pci_bus 0003:00: root bus resource [io 0x200000-0x2fffff] (bus address [0x34100000-0x341fffff])
[ 7.005088] pci_bus 0003:00: root bus resource [mem 0x1280000000-0x12afffffff pref]
[ 7.005315] pci_bus 0003:00: root bus resource [mem 0x12b0000000-0x12bfffffff] (bus address [0x40000000-0x4fffffff])
[ 7.005790] pci 0003:00:00.0: [10de:1ad2] type 01 class 0x060400
[ 7.006290] pci 0003:00:00.0: PME# supported from D0 D3hot D3cold
[ 7.028735] pci 0003:00:00.0: PCI bridge to [bus 01-ff]
[ 7.028984] pci 0003:00:00.0: Max Payload Size set to 256/ 256 (was 256), Max Read Rq 512
[ 7.030214] pcieport 0003:00:00.0: Adding to iommu group 10
[ 7.031177] pcieport 0003:00:00.0: PME: Signaling with IRQ 29
[ 7.032201] pcieport 0003:00:00.0: AER: enabled with IRQ 29
[ 7.033484] pci_bus 0003:01: busn_res: [bus 01-ff] is released
[ 7.034248] pci 0003:00:00.0: Removing from iommu group 10
[ 7.034514] pci_bus 0003:00: busn_res: [bus 00-ff] is released
[ 32.037061] pci_epf_tvnet pci_epf_tvnet.0: tvnet_ep_open: PCIe link is not up
[ 32.165174] pci_epf_tvnet pci_epf_tvnet.0: tvnet_ep_open: PCIe link is not up
[ 32.288201] pci_epf_tvnet pci_epf_tvnet.0: tvnet_ep_open: PCIe link is not up
[ 32.396980] pci_epf_tvnet pci_epf_tvnet.0: tvnet_ep_open: PCIe link is not up
[ 32.413002] pci_epf_tvnet pci_epf_tvnet.0: tvnet_ep_open: PCIe link is not up
[ 39.904020] vdd-3v3-pcie: disabling
[ 39.904049] vdd-12v-pcie: disabling
[ 41.508833] tegra194-pcie 141a0000.pcie_ep: LTSSM state: 0x2018 timeout: -110
root@tegra-ubuntu:/sys/kernel/config/pci_ep#
---- RC log ----
root@tegra-ubuntu:/home/denso# dmesg | grep -i pci
[ 0.730468] PCI: CLS 0 bytes, default 64
[ 2.885237] ohci-pci: OHCI PCI platform driver
[ 4.401785] tegra194-pcie 14180000.pcie: Adding to iommu group 7
[ 4.413108] tegra194-pcie 14180000.pcie: host bridge /pcie@14180000 ranges:
[ 4.414029] tegra194-pcie 14180000.pcie: IO 0x0038100000..0x00381fffff -> 0x0038100000
[ 4.422787] tegra194-pcie 14180000.pcie: MEM 0x1800000000..0x1b3fffffff -> 0x1800000000
[ 4.430912] tegra194-pcie 14180000.pcie: MEM 0x1b40000000..0x1bffffffff -> 0x0040000000
[ 5.546824] tegra194-pcie 14180000.pcie: Phy link never came up
[ 5.547340] tegra194-pcie 14180000.pcie: PCI host bridge to bus 0000:00
[ 5.547618] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 5.547787] pci_bus 0000:00: root bus resource [io 0x0000-0xfffff] (bus address [0x38100000-0x381fffff])
[ 5.548084] pci_bus 0000:00: root bus resource [mem 0x1800000000-0x1b3fffffff pref]
[ 5.548380] pci_bus 0000:00: root bus resource [mem 0x1b40000000-0x1bffffffff] (bus address [0x40000000-0xffffffff])
[ 5.548856] pci 0000:00:00.0: [10de:1ad0] type 01 class 0x060400
[ 5.549352] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
[ 5.583231] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[ 5.583497] pci 0000:00:00.0: Max Payload Size set to 256/ 256 (was 256), Max Read Rq 512
[ 5.585666] pcieport 0000:00:00.0: Adding to iommu group 7
[ 5.587457] pcieport 0000:00:00.0: PME: Signaling with IRQ 24
[ 5.589531] pcieport 0000:00:00.0: AER: enabled with IRQ 24
[ 5.591591] pci_bus 0000:01: busn_res: [bus 01-ff] is released
[ 5.592575] pci 0000:00:00.0: Removing from iommu group 7
[ 5.592894] pci_bus 0000:00: busn_res: [bus 00-ff] is released
[ 5.596984] tegra194-pcie 14100000.pcie: Adding to iommu group 8
[ 5.602824] tegra194-pcie 14100000.pcie: host bridge /pcie@14100000 ranges:
[ 5.603193] tegra194-pcie 14100000.pcie: IO 0x0030100000..0x00301fffff -> 0x0030100000
[ 5.603547] tegra194-pcie 14100000.pcie: MEM 0x1200000000..0x122fffffff -> 0x1200000000
[ 5.603895] tegra194-pcie 14100000.pcie: MEM 0x1230000000..0x123fffffff -> 0x0040000000
[ 5.709981] tegra194-pcie 14100000.pcie: Link up
[ 5.721950] tegra194-pcie 14100000.pcie: PCI host bridge to bus 0001:00
[ 5.722154] pci_bus 0001:00: root bus resource [bus 00-ff]
[ 5.722323] pci_bus 0001:00: root bus resource [io 0x100000-0x1fffff] (bus address [0x30100000-0x301fffff])
[ 5.722603] pci_bus 0001:00: root bus resource [mem 0x1200000000-0x122fffffff pref]
[ 5.722827] pci_bus 0001:00: root bus resource [mem 0x1230000000-0x123fffffff] (bus address [0x40000000-0x4fffffff])
[ 5.723214] pci 0001:00:00.0: [10de:1ad2] type 01 class 0x060400
[ 5.723613] pci 0001:00:00.0: PME# supported from D0 D3hot D3cold
[ 5.737284] pci 0001:01:00.0: [1b4b:9171] type 00 class 0x010601
[ 5.737704] pci 0001:01:00.0: reg 0x10: [io 0x8000-0x8007]
[ 5.738187] pci 0001:01:00.0: reg 0x14: [io 0x8040-0x8043]
[ 5.738451] pci 0001:01:00.0: reg 0x18: [io 0x8100-0x8107]
[ 5.738721] pci 0001:01:00.0: reg 0x1c: [io 0x8140-0x8143]
[ 5.739028] pci 0001:01:00.0: reg 0x20: [io 0x800000-0x80000f]
[ 5.739341] pci 0001:01:00.0: reg 0x24: [mem 0x00900000-0x009001ff]
[ 5.739621] pci 0001:01:00.0: reg 0x30: [mem 0xd0000000-0xd000ffff pref]
[ 5.740348] pci 0001:01:00.0: PME# supported from D3hot
[ 5.763463] pci 0001:00:00.0: BAR 14: assigned [mem 0x1230000000-0x12300fffff]
[ 5.763802] pci 0001:00:00.0: BAR 13: assigned [io 0x100000-0x100fff]
[ 5.764067] pci 0001:01:00.0: BAR 6: assigned [mem 0x1230000000-0x123000ffff pref]
[ 5.764365] pci 0001:01:00.0: BAR 5: assigned [mem 0x1230010000-0x12300101ff]
[ 5.764626] pci 0001:01:00.0: BAR 4: assigned [io 0x100000-0x10000f]
[ 5.771054] pci 0001:01:00.0: BAR 0: assigned [io 0x100010-0x100017]
[ 5.777347] pci 0001:01:00.0: BAR 2: assigned [io 0x100018-0x10001f]
[ 5.783884] pci 0001:01:00.0: BAR 1: assigned [io 0x100020-0x100023]
[ 5.790447] pci 0001:01:00.0: BAR 3: assigned [io 0x100024-0x100027]
[ 5.796923] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[ 5.802185] pci 0001:00:00.0: bridge window [io 0x100000-0x100fff]
[ 5.808463] pci 0001:00:00.0: bridge window [mem 0x1230000000-0x12300fffff]
[ 5.815778] pci 0001:00:00.0: Max Payload Size set to 256/ 256 (was 256), Max Read Rq 512
[ 5.824698] pci 0001:01:00.0: Max Payload Size set to 256/ 512 (was 128), Max Read Rq 512
[ 5.834086] pcieport 0001:00:00.0: Adding to iommu group 8
[ 5.839130] pcieport 0001:00:00.0: PME: Signaling with IRQ 26
[ 5.845332] pcieport 0001:00:00.0: AER: enabled with IRQ 26
[ 5.851682] tegra194-pcie 14140000.pcie: Adding to iommu group 9
[ 5.859402] tegra194-pcie 14140000.pcie: host bridge /pcie@14140000 ranges:
[ 5.863035] tegra194-pcie 14140000.pcie: IO 0x0034100000..0x00341fffff -> 0x0034100000
[ 5.871676] tegra194-pcie 14140000.pcie: MEM 0x1280000000..0x12afffffff -> 0x1280000000
[ 5.880175] tegra194-pcie 14140000.pcie: MEM 0x12b0000000..0x12bfffffff -> 0x0040000000
[ 6.994400] tegra194-pcie 14140000.pcie: Phy link never came up
[ 6.994890] tegra194-pcie 14140000.pcie: PCI host bridge to bus 0003:00
[ 6.995065] pci_bus 0003:00: root bus resource [bus 00-ff]
[ 6.995186] pci_bus 0003:00: root bus resource [io 0x200000-0x2fffff] (bus address [0x34100000-0x341fffff])
[ 6.995434] pci_bus 0003:00: root bus resource [mem 0x1280000000-0x12afffffff pref]
[ 6.995601] pci_bus 0003:00: root bus resource [mem 0x12b0000000-0x12bfffffff] (bus address [0x40000000-0x4fffffff])
[ 6.995920] pci 0003:00:00.0: [10de:1ad2] type 01 class 0x060400
[ 6.996249] pci 0003:00:00.0: PME# supported from D0 D3hot D3cold
[ 7.012908] pci 0003:00:00.0: PCI bridge to [bus 01-ff]
[ 7.013119] pci 0003:00:00.0: Max Payload Size set to 256/ 256 (was 256), Max Read Rq 512
[ 7.014463] pcieport 0003:00:00.0: Adding to iommu group 9
[ 7.015403] pcieport 0003:00:00.0: PME: Signaling with IRQ 28
[ 7.016442] pcieport 0003:00:00.0: AER: enabled with IRQ 28
[ 7.017385] pci_bus 0003:01: busn_res: [bus 01-ff] is released
[ 7.018096] pci 0003:00:00.0: Removing from iommu group 9
[ 7.018301] pci_bus 0003:00: busn_res: [bus 00-ff] is released
[ 7.020284] tegra194-pcie 141a0000.pcie: Adding to iommu group 10
[ 7.021702] tegra194-pcie 141a0000.pcie: Failed to get slot regulators: -517
[ 7.667526] tegra194-pcie 141a0000.pcie: Failed to get slot regulators: -517
[ 8.102274] tegra194-pcie 141a0000.pcie: host bridge /pcie@141a0000 ranges:
[ 8.102304] tegra194-pcie 141a0000.pcie: IO 0x003a100000..0x003a1fffff -> 0x003a100000
[ 8.102340] tegra194-pcie 141a0000.pcie: MEM 0x1c00000000..0x1f3fffffff -> 0x1c00000000
[ 8.102349] tegra194-pcie 141a0000.pcie: MEM 0x1f40000000..0x1fffffffff -> 0x0040000000
[ 8.209931] tegra194-pcie 141a0000.pcie: Link up
[ 8.216340] tegra194-pcie 141a0000.pcie: PCI host bridge to bus 0005:00
[ 8.227179] pci_bus 0005:00: root bus resource [bus 00-ff]
[ 8.227187] pci_bus 0005:00: root bus resource [io 0x300000-0x3fffff] (bus address [0x3a100000-0x3a1fffff])
[ 8.227191] pci_bus 0005:00: root bus resource [mem 0x1c00000000-0x1f3fffffff pref]
[ 8.227213] pci_bus 0005:00: root bus resource [mem 0x1f40000000-0x1fffffffff] (bus address [0x40000000-0xffffffff])
[ 8.242452] pci 0005:00:00.0: [10de:1ad0] type 01 class 0x060400
[ 8.255170] pci 0005:00:00.0: PME# supported from D0 D3hot D3cold
[ 8.281071] pci 0005:01:00.0: [10de:2296] type 00 class 0x028000
[ 8.315722] pci 0005:01:00.0: reg 0x10: [mem 0x00000000-0x003fffff]
[ 8.321620] pci 0005:01:00.0: reg 0x18: [mem 0x00000000-0x0001ffff 64bit pref]
[ 8.329184] pci 0005:01:00.0: reg 0x20: [mem 0x00000000-0x000fffff 64bit]
[ 8.336314] pci 0005:01:00.0: PME# supported from D0 D3hot D3cold
[ 8.342482] pci 0005:01:00.0: 63.008 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x8 link at 0005:00:00.0 (capable of 126.024 Gb/s with 16.0 GT/s PCIe x8 link)
[ 8.363680] pci 0005:00:00.0: BAR 14: assigned [mem 0x1f40000000-0x1f405fffff]
[ 8.364379] pci 0005:00:00.0: BAR 15: assigned [mem 0x1c00000000-0x1c000fffff 64bit pref]
[ 8.372703] pci 0005:01:00.0: BAR 0: assigned [mem 0x1f40000000-0x1f403fffff]
[ 8.380063] pci 0005:01:00.0: BAR 4: assigned [mem 0x1f40400000-0x1f404fffff 64bit]
[ 8.387784] pci 0005:01:00.0: BAR 2: assigned [mem 0x1c00000000-0x1c0001ffff 64bit pref]
[ 8.395595] pci 0005:00:00.0: PCI bridge to [bus 01-ff]
[ 8.401047] pci 0005:00:00.0: bridge window [mem 0x1f40000000-0x1f405fffff]
[ 8.408204] pci 0005:00:00.0: bridge window [mem 0x1c00000000-0x1c000fffff 64bit pref]
[ 8.416350] pci 0005:00:00.0: Max Payload Size set to 256/ 256 (was 256), Max Read Rq 512
[ 8.424612] pci 0005:01:00.0: Max Payload Size set to 256/ 256 (was 256), Max Read Rq 512
[ 8.433981] pcieport 0005:00:00.0: Adding to iommu group 10
[ 8.439163] pcieport 0005:00:00.0: PME: Signaling with IRQ 30
[ 8.444777] pcieport 0005:00:00.0: AER: enabled with IRQ 30
root@tegra-ubuntu:/home/denso#
thanks,
Hermes Wu