PCI MSI interrupt generation issue in Jetson
Hello experts,
We have an FPGA connected as endpoint device to Nvidia Jetson Xavier NX. And we have developed the PCIe driver for Jetson.
We are using an MSI interrupt for generating an interrupt. This interrupt is sent by FPGA to Jetson when FPGA is done transferring data to Jetson.
In our driver code, we have registered an interrupt vector in probe function like shown below (for loop is for assigning multiple vectors, but currently only using one):
static int fpga_probe(struct pci_dev *pdev, const struct pci_device_id *id)
{
...
...
/* PCI interrupt handling */
ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
if (ret < 0)
{
dev_err(drvdata->dev, "pci_alloc_irq_vectors failed\n");
goto pci_err_int;
}
else
{
dev_info(drvdata->dev, "Number of vectors assigned: %d\n", ret);
}
irq = pci_irq_vector(pdev, 0);
for (i = 0; i < 1; i++)
{
ret = request_irq((irq + i), irq_handler, IRQF_SHARED, "fpga", drvdata);
if (ret < 0)
{
dev_err(drvdata->dev, "request_irq failed\n");
goto pci_err_int;
}
else
{
dev_info(drvdata->dev, "IRQ: %d\n", (irq + i));
}
}
...
...
}
1 vector is successfully assigned.
The interrupt is registered successfully as we can see them in /proc/interrupts file:
811: 0 0 0 0 PCI-MSI 0 Edge fpga
and the registered IRQ handler which will be called when MSI interrupt is raised from endpoint (FPGA) is defined as shown below:
static irqreturn_t irq_handler(int irq, void *cookie)
{
pr_info("====> IRQ Handled: IRQ number %d\n", irq);
return IRQ_HANDLED;
}
lspci’s verbose output is shown below:
root@user:/home/ubuntu# lspci -vvv
0005:01:00.0 Unassigned class [ff00]: Altera Corporation Device e001
Subsystem: Altera Corporation Device e001
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-
Interrupt: pin A routed to IRQ 811
Region 0: Memory at 1c00000000 (64-bit, prefetchable) [size=32K]
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fffff000 Data: 0000
Capabilities: [78] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [80] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #1, Speed 5GT/s, Width x2, ASPM not supported, Exit Latency L0s <4us, L1 <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x2, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [200 v1] Vendor Specific Information: ID=1172 Rev=0 Len=044
Kernel driver in use: fpga
The config space:
root@user:/home/ubuntu# lspci -xxx
0005:01:00.0 Unassigned class [ff00]: Altera Corporation Device e001
00: 72 11 01 e0 06 04 10 00 00 00 00 ff 00 00 00 00
10: 0c 00 00 00 1c 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 72 11 01 e0
30: 00 00 00 00 50 00 00 00 00 00 00 00 00 01 00 00
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 05 78 81 00 00 f0 ff ff 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 11 78 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 01 80 03 00 00 00 00 00
80: 10 00 02 00 01 80 00 00 30 28 00 00 22 60 40 01
90: 00 00 22 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 10 00 10 00 00 00 00 00 06 00 00 00
b0: 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
When FPGA is done transferring the data to DMA addresses allocated by Jetson, it writes data (0x0000) to the MSI address (0x00000000fffff000) which is displayed in lspci’s output as well as in config space.
But Jetson is somehow not able to receive the interrupt. Counter values in /proc/interrupts files stays 0 and irq_handler is also not executed.
We think issue is probably related to Jetson since we can see in FPGA’s software that it is writing data (0x0000) to the MSI address (0x00000000fffff000).
Is there anything missing in driver code or kernel related to MSI interrupt handling?