What is the actual maximum speed of Jetson AGX Xavier PCIE Ethernet?
Mar 04, 2024
Dear Nvidia Team
We are using PCIe to communicate between two Xaviers, According to << Jetson AGX Xavier PCIe Endpoint Mode>>, but have some doubts in actual use.
Version information about the systems we use: JetPack4.6 kernel version:4.9.253-tegra Device:Jetson AGX Xavier
The tests I have performed are as follows:
1.Xavier B starts iperf3 server
# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.66.6, port 40890
[ 5] local 192.168.66.7 port 5201 connected to 192.168.66.6 port 40892
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 359 MBytes 3.01 Gbits/sec
[ 5] 1.00-2.00 sec 478 MBytes 4.01 Gbits/sec
[ 5] 2.00-3.00 sec 423 MBytes 3.55 Gbits/sec
[ 5] 3.00-4.00 sec 464 MBytes 3.89 Gbits/sec
[ 5] 4.00-5.00 sec 484 MBytes 4.06 Gbits/sec
[ 5] 5.00-6.00 sec 553 MBytes 4.64 Gbits/sec
[ 5] 6.00-7.00 sec 551 MBytes 4.63 Gbits/sec
[ 5] 7.00-8.00 sec 521 MBytes 4.37 Gbits/sec
[ 5] 8.00-9.00 sec 167 MBytes 1.40 Gbits/sec
[ 5] 9.00-10.00 sec 172 MBytes 1.44 Gbits/sec
[ 5] 10.00-11.00 sec 168 MBytes 1.41 Gbits/sec
[ 5] 11.00-12.00 sec 172 MBytes 1.44 Gbits/sec
[ 5] 12.00-13.00 sec 161 MBytes 1.35 Gbits/sec
[ 5] 13.00-14.00 sec 358 MBytes 3.01 Gbits/sec
[ 5] 14.00-14.16 sec 90.2 MBytes 4.63 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-14.16 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-14.16 sec 5.00 GBytes 3.03 Gbits/sec receiver
2.Xavier A starts iperf3 client,5G data volume test using 10000M bandwidth
# iperf3 -c 192.168.66.7 -b 10000M -n 5G
Connecting to host 192.168.66.7, port 5201
[ 4] local 192.168.66.6 port 40892 connected to 192.168.66.7 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 379 MBytes 3.18 Gbits/sec 0 1.11 MBytes
[ 4] 1.00-2.00 sec 472 MBytes 3.95 Gbits/sec 0 1.11 MBytes
[ 4] 2.00-3.00 sec 431 MBytes 3.62 Gbits/sec 0 1.11 MBytes
[ 4] 3.00-4.00 sec 464 MBytes 3.89 Gbits/sec 0 1.11 MBytes
[ 4] 4.00-5.00 sec 483 MBytes 4.06 Gbits/sec 0 1.11 MBytes
[ 4] 5.00-6.00 sec 554 MBytes 4.64 Gbits/sec 0 1.54 MBytes
[ 4] 6.00-7.00 sec 551 MBytes 4.62 Gbits/sec 0 1.54 MBytes
[ 4] 7.00-8.00 sec 507 MBytes 4.25 Gbits/sec 0 3.93 MBytes
[ 4] 8.00-9.00 sec 169 MBytes 1.42 Gbits/sec 0 3.93 MBytes
[ 4] 9.00-10.00 sec 171 MBytes 1.43 Gbits/sec 0 3.93 MBytes
[ 4] 10.00-11.00 sec 167 MBytes 1.40 Gbits/sec 0 3.93 MBytes
[ 4] 11.00-12.00 sec 173 MBytes 1.45 Gbits/sec 0 3.93 MBytes
[ 4] 12.00-13.00 sec 160 MBytes 1.35 Gbits/sec 0 3.93 MBytes
[ 4] 13.00-14.00 sec 371 MBytes 3.12 Gbits/sec 0 3.93 MBytes
[ 4] 14.00-14.12 sec 68.1 MBytes 4.63 Gbits/sec 0 3.93 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-14.12 sec 5.00 GBytes 3.04 Gbits/sec 0 sender
[ 4] 0.00-14.12 sec 5.00 GBytes 3.04 Gbits/sec receiver
3.View the actual physical rate of the system PCIE
# lspci
0000:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad0 (rev a1)
0000:01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a809
0005:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad0 (rev a1)
0005:01:00.0 Network controller: NVIDIA Corporation Device 2296
root@neolix-xavier-rt:/home/nvidia# 318 lspci -n | grep -i 0005:01:00.0
# lspci -n | grep -i 0005:01:00.0
0005:01:00.0 0280: 10de:2296
# lspci -n | grep -i 0005:01:00.0
0005:01:00.0 0280: 10de:2296
root@neolix-xavier-rt:/home/nvidia# lspci -n -d 10de:2296 -vvv | grep --color Width
LnkCap: Port #0, Speed 16GT/s, Width x8, ASPM not supported, Exit Latency L0s <1us, L1 <64us
LnkSta: Speed 16GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
After carrying out the above tests, it was found that there are several problems in use：
- Using lspci, we can see that the actual PCIE rate is 16GT/s, Width x8, and the throughput is about 15.754GB/s, which is 126Gb/s, but the performance measurement using iperf3 shows that the average rate is only about 3.04 Gbits/sec, which is similar to The actual physical PCIE rate is very different, why is this?
- From the test results of iperf3, it can be seen that the rate of pcie ethernet fluctuates between 1.35 Gbits/sec and 4.64 Gbits/sec. The results of many long-term tests are also the same. What is the reason for the unstable rate?
- What is the maximum actual speed of the official Jetson AGX Xavier using PCIE Ethernet?
Thank you for your support.
Best regards