Getting NVIDIA ConnectX NICs working

Accurate as of QuantumBNG 1.2.0. Sourced from the 1.2.0 release notes and the NIC compatibility reference. ConnectX cards sit in Tier 2 — the driver family is supported, but we have not yet validated these cards end to end ourselves.

The short version

QuantumBNG does not bundle MLNX_OFED or Nvidia firmware. The installer loads kernel modules your operating system already has; it ships no vendor driver stack for anything. For a ConnectX card that means the driver and firmware are yours to install first, and the wizard will not see the ports until you have.

This applies to ConnectX-5 (25/40/100 GbE), ConnectX-6 Dx and Lx (25/100 GbE), and ConnectX-7 (200/400 GbE). All use mlx5_core.

Getting a card working

  1. Install MLNX_OFED for your kernel, from Nvidia’s download centre. Match your Ubuntu 24.04 and kernel 6.8 version exactly — an OFED package built for a different kernel is the most common reason a card never appears.

  2. Flash matching firmware if the card is older than the OFED package expects.

    mlxfwmanager --query                    # what is on the card now
mlxfwmanager --online -u -d <pci>        # update from Nvidia's catalogue

  3. Reboot, so the new driver and firmware load cleanly together.

  4. Re-run the setup wizard’s DPDK Bind step. The ConnectX ports should now appear in the Kernel column, draggable across to VPP.

The rough edge worth knowing about

Mellanox cards work best with PCI passthrough rather than vfio-pci. VPP’s DPDK PMD handles the mlx5 bifurcated driver correctly, but you may need to leave the card bound to mlx5_core and let VPP claim it directly through startup.conf:

dpdk {
    dev <pci>
}

…rather than moving it across in the wizard’s rebind flow. This is a genuine rough edge in this release, not a misconfiguration on your side. If the wizard’s rebind does not stick, this is the workaround.

If the card still does not appear

If a card shows in neither the Kernel nor the VPP column after the driver is loaded, open a support ticket and attach all three of these:

sudo lspci -nnk -s <pci>
sudo dpdk-devbind.py --status
sudo journalctl -u vpp -n 200

The first tells us which driver claimed the card, the second what DPDK can see, and the third what VPP did when it tried. Together they usually identify the problem without a second round trip.

A note on firmware reporting

Once a NIC is bound to a userspace driver, kernel tools can no longer read its firmware or DDP state, so GET /api/bng/drivers reports those fields empty and explains why in binding_note. Separately, DDP and serial fields are only ever populated for the Intel ice, i40e and iavf drivers — a ConnectX card reporting empty strings there is expected, not a fault.

See also: Intel NIC compatibility matrix.