GPUDirect RDMA PCIe Topology

I am working on a hardware design with a CPU connected to a PCIe 3.0 switch over 4 PCIe lanes (x4) and an FPGA and GPU connected to the same PCIe switch over 16 lanes (x16). When performing a GPUDirect RDMA to transfer data between the FPGA and GPU, will the two devices use all 16 lanes, or will the CPU connected to the PCIe switch with only 4 lanes effect the transfer speed or number of lanes used between the FPGA and GPU?