Skip to main content

Network traffic performance on AMD processors

This page details performance improvements on AMD processors and how to deploy the fixes on your pool.

What's the "AMD EPYC performance" story?​

This issue was first identified on EPYC processors in 2024, which is why it was first called the "AMD EPYC performance issue". However, it affects all AMD processors, not just the EPYC product line.

The performance issue on AMD processors was characterized by low bandwidth on internal VM-to-VM traffic (inside the same host) and VM-to-host-network traffic, especially when compared with Intel processors. Network traffic is efficient for many workloads, but not the most demanding ones.

As with most performance issues, this case is complicated. The issue can stem from multiple causes, but at least one bottleneck has been eliminated.

Fix 1 (significant): uncached grant-tables​

Technical description of the issue and its fix​

Due to differences between processor architectures and Xen's isolation, grant-tables in Xen were not cached the same way on Intel and AMD CPUs. This patch fixes the location of the grant-tables mapping: it moves caching from Uncached mode to Writeback mode.

Finally, with help from the XenServer team, we realized that a specific data structure wasn't cached, which severely reduces performance.

However, this fix is not useful on current Intel processors. You can enable it on Intel processors anyway, since the changes are not known to cause any regression.

How to enable the fix​

The fix is enabled by default on up-to-date XCP-ng 8.3 hosts (xapi >= 25.27.0-2.1.xcpng8.3): grant-tables are mapped as writeback instead of uncacheable, and no configuration is needed.

  • Stop and start VMs that were started before the update, so that they benefit from the fix during their new boot (a simple reboot is not enough).
  • Guests need their kernel to support the feature that enables the fix (see the list below). Windows and BSD guests were not affected by the performance problem solved by this change.
tip

The fix was first available as an opt-in setting with xapi 25.6.0-1.11 and xen 4.17.5-15.2. If you created /etc/xenopsd.conf.d/amd-gnttab-wb.conf at that time, it's now redundant and you can remove it.

To revert to the previous behavior (should side effects occur in edge cases), create on each host a file in /etc/xenopsd.conf.d/ containing xen-platform-pci-bar-uc=true, restart the toolstack, then stop and start the VMs.

There's more work to be done to improve performance on AMD EPYC servers, but it is significant progress!

How to validate the fix on Linux guests​

Use the check_grant_table_cacheability.py script:

How to validate the fix on Linux guests
# wget https://raw.githubusercontent.com/xcp-ng/xcp/refs/heads/master/scripts/amd_perf/check_grant_table_cacheability.py
$ sudo python3 ./check_grant_table_cacheability.py
'xen-platform-pci' PCI IO mem address is 0xFB000000
Grant table cacheability fix is ACTIVE.

OS support​

Ready: Linux distributions with recent enough kernels or distros that apply fixes from the mainline LTS kernels are OK.

Not ready:

Older distros.

Some currently supported LTS distros don't have the patch yet. Specifically: RHEL 8 and 9 and their derivatives.

Will never be ready:

End-of-Life distros will never be updated and therefore, will never benefit from this improvement.

Guest kernel version determines whether the performance fix is supported or not. All kernel versions equal to or newer than 5.19 support it. LTS kernels include 5.15.189+.

  • Debian:

    • Debian 11 "Bullseye" (5.10 -> EOL)
    • Debian 12 "Bookworm" (6.1 -> OK)

  • Ubuntu:

    • 20.04 LTS "Focal Fossa" (5.4: EOL)
    • 22.04 LTS "Jammy Jellyfish" (5.15.0-154+: OK, HWE 6.8: OK)
    • 24.04 LTS "Noble Numbat" (6.8 & HWE 6.14: OK)

  • openSUSE Leap:

    • 15.5 (5.14: EOL)
    • 15.6 (6.4: OK)

  • SUSE Enterprise (LTSS):

    • SLE15 SP3 - LTSS (5.3: Not upstream)
    • SLE15 SP4/5 - LTSS (5.14: Not upstream)
    • SLE15 SP6+ (OK)

  • RHEL (+derivatives):

    • 8 (4.18: Not upstream)
    • 9 (5.14: Not upstream)
    • 10 (6.12: OK)

  • Fedora: All supported: OK (37+)

  • Alpine Linux: All supported: OK (v3.18+)

  • EOL = distro is End-of-Life

  • Not upstream = not covered by Linux stable project

Fix 2: spurious interrupts​

Technical description of the issue and its fix​

Spurious interrupts are interrupts that arise for no reason —there is no work to do. To avoid denial-of-service, xen devices front-ends implement security measures to avoid spurious interrupts from the backends. To do that, front-ends put interrupt handlers on hold for a short period of time. With this delay, front-ends can hopefully mitigate the flood of spurious interruptions caused by the backend.

Sadly, the spurious interrupts weren't caused by the backend. They were caused by another part of the code, which consumes packets outside of the interrupt handler.

The real cause was fixed upstream. Don't hesitate to read the full commit message for more technical details.

How to enable the fix​

This issue occurred only on the front-end side. This means that the only way to improve network bandwidth is to upgrade the guest kernel, which limits spurious interrupts.

Fix 1 (on grant-tables) and fix 2 work independently from each other. You can apply them both. The fixes don't guarantee better performance for every processor model (See Future improvements below) but we didn't detect any regression when applying them. The fix is not specific to AMD —it can improve internal network performance on Intel processors too.

OS support​

The fix is upstream in the Linux kernel (v6.17) and in the LTS kernels listed below. Debian 12 and 13, Ubuntu 22.04 (kernel 5.15.0-163 or later) and Ubuntu 24.04 (kernel 6.8.0-98 or later) ship it. As of October 2026, RHEL 8, 9 and 10 and their derivatives don't include it yet.

Here are the LTS kernels that have been fixed:

  • v5.10.241
  • v5.15.190
  • v6.1.149
  • v6.6.103
  • v6.12.43
  • v6.15.11
  • v6.16.2

Future improvements and leads​

The last improvement does not work on every platform, so the story may not be over yet. At least one bottleneck remains.

The investigation goes on.