Optimizing Linux for AMD EPYC™ 9004 Series Processors with SUSE Linux Enterprise 15 SP4

The ability to use local memory where possible and remote memory if necessary is valuable. But there are cases where it is imperative that local memory always be used. If this is the case, the first priority is to bind the task to that node. If that is not possible, then the command sysctl vm.zone_reclaim_mode=1 can be used to aggressively reclaim memory if local memory is not available.

Note: Potential Hazard with vm.zone_reclaim_mode

While this option is good from a locality perspective, it can incur high costs because of stalls related to reclaim and the possibility that data from the task will be reclaimed. Treat this option with a high degree of caution and testing.

There are three major hazards to consider with CPU binding.

The first to watch for is remote memory nodes being used when the process is not allowed to run on CPUs local to that node. The scenarios when this can occur are outside the scope of this paper. However, a common reason is an IO-bound thread communicating with a kernel IO thread on a remote node bound to the IO controller. In such a setup, the data buffers managed by the application are stored in remote memory incurring an access cost for the IO.

While tasks may be bound to CPUs, the resources they are accessing, such as network or storage devices, may not have interrupts routed locally. irqbalance generally makes good decisions. But in cases where the network or IO is extremely high-performance or the application has very low latency requirements, it may be necessary to disable irqbalance using systemctl. When that is done, the IRQs for the target device need to be routed manually to CPUs local to the target workload for optimal performance.

The second is that guides about CPU binding tend to focus on binding to a single CPU. This is not always optimal when the task communicates with other threads, as fixed bindings potentially miss an opportunity for the processes to use idle CPUs sharing a common cache. This is particularly true when dispatching IO, be it to disk or a network interface, where a task may benefit from being able to migrate close to the related threads. It also applies to pipeline-based communicating threads for a computational workload. Hence, focus initially on binding to CPUs sharing L3 cache. Then consider whether to bind based on an L1/L2 cache or a single CPU using the primary metric of the workload to establish whether the tuning is appropriate.

The final hazard is similar: if many tasks are bound to a smaller set of CPUs, then the subset of CPUs could be oversaturated even though there is spare CPU capacity available.

CPUsets are ideal when multiple workloads must be isolated on a machine in a predictable fashion. CPUsets allow a machine to be partitioned into subsets. These sets may overlap, and in that case they suffer from similar problems as CPU affinities. In the event there is no overlap, they can be switched to “exclusive” mode which treats them completely in isolation with relatively little overhead. Similarly, they are well suited when a primary workload must be protected from interference because of low-priority tasks. In such cases the low-priority tasks can be placed in a CPUset. The caveat with CPUsets is that the overhead is higher than using scheduler and memory policies. Ordinarily, the accounting code for CPUsets is completely disabled. But when a single CPUset is created, there is a second layer of checks against scheduler and memory policies.

Similarly, memcg can be used to limit the amount of memory that can be used by a set of processes. When the limits are exceeded, the memory will be reclaimed by tasks within memcg directly without interfering with any other tasks. This is ideal for ensuring there is no inference between two or more sets of tasks. Similar to CPUsets, there is some management overhead incurred. This means, if tasks can simply be isolated on a NUMA boundary, then this is preferred from a performance perspective. The major hazard is that, if the limits are exceeded, then the processes directly stall to reclaim the memory which can incur significant latencies.

Note

Without memcg, when memory gets low, the global reclaim daemon does work in the background and if it reclaims quickly enough, no stalls are incurred. When using memcg, observe the allocstall counter in /proc/vmstat as this can detect early if stalling is a problem.

Decisions on whether to bind a workload to a subset of CPUs require that the CPU utilization and any saturation risk is known. Both the ps and pidstat commands can be used to sample the number of threads in a system. Typically, pidstat yields more useful information with the important exception of the run state. A system may have many threads, but if they are idle, they are not contributing to utilization. The mpstat command can report the utilization of each CPU in the system.

High utilization of a small subset of CPUs may be indicative of a single-threaded workload that is pushing the CPU to the limits and may indicate a bottleneck. Conversely, low utilization may indicate a task that is not CPU-bound, is idling frequently or is migrating excessively. While each workload is different, load utilization of CPUs may show a workload that can run on a subset of CPUs to reduce latencies because of either migrations or remote accesses. When utilization is high, it is important to determine if the system could be saturated. The vmstat tool reports the number of runnable tasks waiting for a CPU in the “r” column where any value over 1 indicates that wakeup latencies may be incurred. While the exact wakeup latency can be calculated using trace points, knowing that there are tasks queued is an important step. If a system is saturated, it may be possible to tune the workload to use fewer threads.

Overall, the initial intent should be to use CPUs from as few NUMA nodes as possible to reduce access latency. However, there are exceptions. The AMD EPYC 9004 Series Processor has a large number of high-speed memory channels to main memory, so consider the workload thread activity. If they are cooperating threads or sharing data, isolate them on as few nodes as possible to minimize cross-node memory accesses. If the threads are completely independent with no shared data, it may be best to isolate them on a subset of CPUs from each node. This is to maximize the number of available memory channels and throughput to main memory. For some computational workloads, it may be possible to use hybrid models such as MPI for parallelization across nodes and OpenMP for threads within nodes.

Note: Updating tuning for AMD EPYC 9004 Series Processor

It is expected that tuning based on the AMD EPYC 7003 Series Processor will also usually perform optimally on AMD EPYC 9004 series. The main consideration is to account for potential differences in L3 cache sizes because of AMD V-Cache if workloads are tuned specifically for cache size. Also, keep in mind that CPU bindings based on caches may potentially be relaxed on SUSE Linux Enterprise 15 SP4.

Huge pages are a feature that can improve performance in many cases. This is achieved by reducing the number of page faults, the cost of translating virtual addresses to physical addresses because of fewer layers in the page table and being able to cache translations for a larger portion of memory. Transparent Huge Pages (THP) is supported for private anonymous memory that automatically backs mappings with huge pages where anonymous memory could be allocated as heap, malloc(), mmap(MAP_ANONYMOUS), etc. There is also support for using THP pages backed by tmpfs which can be configured at mount time using the huge= mount option. While the THP feature has existed for a long time, it has evolved significantly.

Many tuning guides recommend disabling THP because of problems with early implementations. Specifically, when the machine was running for long enough, the use of THP could incur severe latencies and could aggressively reclaim memory in certain circumstances. These problems were resolved by the time SUSE Linux Enterprise Server 15 SP2 was released, and this is still the case for SUSE Linux Enterprise Server 15 SP4. This means there are no good grounds for automatically disabling THP because of severe latency issues without measuring the impact. However, there are exceptions that are worth considering for specific workloads.

Some high-end in-memory databases and other applications aggressively use mprotect() to ensure that unprivileged data is never leaked. If these protections are at the base page granularity, then there may be many THP splits and rebuilds that incur overhead. It can be identified if this is a potential problem by using strace or perf trace to detect the frequency and granularity of the system call. If they are high-frequency, consider disabling THP. It can also be sometimes inferred from observing the thp_split and thp_collapse_alloc counters in /proc/vmstat.

Workloads that sparsely address large mappings may have a higher memory footprint when using THP. This could result in premature reclaim or fallback to remote nodes. An example would be HPC workloads operating on large sparse matrices. If memory usage is much higher than expected, compare memory usage with and without THP to decide if the trade-off is not worthwhile. This may be critical on AMD EPYC 7003 and 9004 Series Processor given that any spillover will congest the Infinity links and potentially cause cores to run at a lower frequency.

Note: Sparsely addressed memory

This is specific to sparsely addressed memory. A secondary hint for this case may be that the application primarily uses large mappings with a much higher Virtual Size (VSZ, see Section 8.4, “Memory utilization and saturation”) than Resident Set Size (RSS). Applications which densely address memory benefit from the use of THP by achieving greater bandwidth to memory.

Parallelized workloads that operate on shared buffers with thread counts exceeding the number of available CPUs on a single node may experience a slowdown with THP if the granularity of partitioning is not aligned to the huge page. The problem is that if a large shared buffer is partitioned on a 4K boundary, then false sharing may occur whereby one thread accesses a huge page locally and other threads access it remotely. If this situation is encountered, the granularity of sharing should be increased to the THP size. But if that is not possible, disabling THP is an option.

Applications that are extremely latency-sensitive or must always perform in a deterministic fashion can be hindered by THP. While there are fewer faults, the time for each fault is higher as memory must be allocated and cleared before being visible. The increase in fault times may be in the microsecond granularity. Ensure this is a relevant problem as it typically only applies to extremely latency-sensitive applications. The secondary problem is that a kernel daemon periodically scans a process looking for contiguous regions that can be backed by huge pages. When creating a huge page, there is a window during which that memory cannot be accessed by the application and new mappings cannot be created until the operation is complete. This can be identified as a problem with thread-intensive applications that frequently allocate memory. In this case, consider effectively disabling khugepaged by setting a large value in /sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs. This will still allow THP to be used opportunistically while avoiding stalls when calling malloc() or mmap().

THP can be disabled. To do so, specify transparent_hugepage=disable on the kernel command line, at runtime via /sys/kernel/mm/transparent_hugepage/enabled or on a per-process basis by using a wrapper to execute the workload that calls prctl(PR_SET_THP_DISABLE).

Assuming an application is mostly CPU- or memory-bound, it is useful to determine if the footprint is primarily in user space or kernel space. This gives a hint where tuning should be focused. The percentage of CPU time can be measured on a coarse-grained fashion using vmstat or a fine-grained fashion using mpstat. If an application is mostly spending time in user space, then the focus should be on tuning the application itself. If the application is spending time in the kernel, then it should be determined which subsystem dominates. The strace or perf trace commands can measure the type, frequency and duration of system calls as they are the primary reasons an application spends time within the kernel. In some cases, an application may be tuned or modified to reduce the frequency and duration of system calls. In other cases, a profile is required to identify which portions of the kernel are most relevant as a target for tuning.

The traditional means of measuring memory utilization of a workload is to examine the Virtual Size (VSZ) and Resident Set Size (RSS). This can be done by using either the ps or pidstat tool. This is a reasonable first step but is potentially misleading when shared memory is used and multiple processes are examined. VSZ is simply a measure of memory space reservation and is not necessarily used. RSS may be double accounted if it is a shared segment between multiple processes. The file /proc/pid/maps can be used to identify all segments used and whether they are private or shared. The file /proc/pid/smaps and /proc/pid/smaps_rollup reveals more detailed information including the Proportional Set Size (PSS). PSS is an estimate of RSS except it is divided between the number of processes mapping that segment, which can give a more accurate estimate of utilization. Note that the smaps and smaps_rollup files are very expensive to read and should not be monitored at a high frequency. This is especially the case if workloads are using large amounts of address space, many threads or both. Finally, the Working Set Size (WSS) is the amount of active memory required to complete computations during an arbitrary phase of a program's execution. It is not a value that can be trivially measured. But conceptually it is useful as the interaction between WSS relative to available memory affects memory residency and page fault rates.

On NUMA systems, the first saturation point is a node overflow when the “local” policy is in effect. Given no binding of memory, when a node is filled, a remote node’s memory will be used transparently and background reclaim will take place on the local node. Two consequences of this are that remote access penalties will be used and old memory from the local node will be reclaimed. If the WSS of the application exceeds the size of a local node, then paging and re-faults may be incurred.

The first item to identify is whether a remote node overflow occurred, which is accounted for in /proc/vmstat as the numa_hit, numa_miss, numa_foreign, numa_interleave, numa_local and numa_other counters:

For the local memory policy, the numa_hit and numa_miss counters are the most important to pay attention to. An application that is allocating memory that starts incrementing the numa_miss implies that the first level of saturation has been reached. If monitoring the proc is undesirable, then the numastat provides the same information. If this is observed on the AMD EPYC 9004 Series Processor, it may be valuable to bind the application to nodes that represent dies on a single socket. If the ratio of hits to misses is close to 1, consider an evaluation of the interleave policy to avoid unnecessary reclaim.

Note: NUMA statistics

These NUMA statistics only apply at the time a physical page is allocated and are not related to the reference behavior of the workload. For example, if a task running on node 0 allocates memory local to node 0, then it will be accounted for as a node_hit in the statistics. However, if the memory is shared with a task running on node 1, all the accesses may be remote, which is a miss from the perspective of the hardware but not accounted for in /proc/vmstat. Detecting remote and local accesses at the hardware level requires using the hardware's Performance Management Unit to detect.

When the first saturation point is reached, reclaim will be active. This can be observed by monitoring the pgscan_kswapd and pgsteal_kswapd counters in /proc/vmstat. If this is matched with an increase in major faults or minor faults, then it may be indicative of severe thrashing. In this case, the interleave policy should be considered. An ideal tuning option is to identify if shared memory is the source of the usage. If this is the case, then interleave the shared memory segments. This can be done in some circumstances using numactl or by modifying the application directly.

More severe saturation is observed if the pgscan_direct and pgsteal_direct counters are also increasing. These counters indicate that the application is stalling while memory is being reclaimed. If the application was bound to individual nodes, increasing the number of available nodes will alleviate the pressure. If the application is unbound, it indicates that the WSS of the workload exceeds all available memory. It can only be alleviated by tuning the application to use less memory or increasing the amount of RAM available.

A more generalized view of resource pressure for CPU, memory and IO can be measured using the kernel Pressure Stall Information feature enabled with the command line psi=1. When enabled, proc files under /proc/pressure show if some or all active tasks were stalled recently contending on a resource. This information is not always available in production. But if the information is available, the memory pressure information may be used to guide whether a deeper analysis is necessary and which resource is the bottleneck.

As before, whether to use memory nodes from one socket or two sockets depends on the application. If the individual processes are independent, either socket can be used. Where possible, keep communicating processes on the same socket to maximize memory throughput while minimizing the socket interconnect traffic.

The analysis of other resources is outside the scope of this paper. However, a common scenario is that an application is IO-bound. A superficial check can be made using the vmstat tool. This tool checks what percentage of CPU time is spent idle combined with the number of processes that are blocked and the values in the bi and bo columns. Similarly, if PSI is enabled, then the IO pressure file will show whether some or all active tasks are losing time because of lack of resources. Further analysis is required to determine if an application is IO rather than CPU- or memory-bound. However, this is a sufficient check to start with.

The following sections will demonstrate how an OpenMP and MPI workload can be configured and tuned on an AMD EPYC 9004 Series Processor reference platform. The system has two processors, each with 96 cores and SMT enabled for a total of 192 cores (384 logical CPUs). The peak bandwidth available to the machine depends on the type of DIMMs installed and how the DIMM slots are populated. Note that the peak theoretical transfer speed is rarely reached in practice, given that it can be affected by the mix of reads/writes and the location and temporal proximity of memory locations accessed.

STREAM is a memory bandwidth benchmark created by Dr. John D. McCalpin from the University of Virginia (for more information, see https://www.cs.virginia.edu/stream/). It can be used to measure bandwidth of each cache level and bandwidth to main memory while calculating four basic vector operations. Each operation can exhibit different throughputs to main memory depending on the locations and type of access.

The benchmark was configured to run both single-threaded and parallelized with OpenMP to take advantage of multiple memory controllers. The array of elements for the benchmark was set at 268,435,456 elements at compile time so that each array was 2048MB in size for a total memory footprint of approximately 6144 MB. The size was selected in line with the recommendation from STREAM that the array sizes be at least 4 times the total size of L3 cache available in the system. Pay special attention to the exact size of the L3 cache if V-Cache is present. An array-size offset was used so that the separate arrays for each parallelized thread would not share a Transparent Huge Page. The reason is that NUMA balancing may choose to migrate shared pages leading to some distortion of the results.

The march=znver3 is a reflection of the compiler available in SUSE Linux Enterprise 15 SP4 at the time of writing. It should be checked if a later GCC version is available in the Developer Tools Module that supported march=znver4. The number of openMP threads was selected to have at least one thread running for every memory channel by having one thread per L3 cache available. The OMP_PROC_BIND parameter spreads the threads such that one thread is bound to each available dedicated L3 cache to maximize available bandwidth. This can be verified using perf, as illustrated below with slight editing for formatting and clarity.

epyc:~ # perf record -e sched:sched_migrate_task ./stream
epyc:~ # perf script
...
          stream-nnn x: sched:sched_migrate_task: comm=stream pid=494780 prio=120 orig_cpu=0 dest_cpu=8
          stream-nnn x: sched:sched_migrate_task: comm=stream pid=494781 prio=120 orig_cpu=0 dest_cpu=16
          stream-nnn x: sched:sched_migrate_task: comm=stream pid=494782 prio=120 orig_cpu=0 dest_cpu=24
          stream-nnn x: sched:sched_migrate_task: comm=stream pid=494783 prio=120 orig_cpu=0 dest_cpu=32
          stream-nnn x: sched:sched_migrate_task: comm=stream pid=494784 prio=120 orig_cpu=0 dest_cpu=40
          stream-nnn x: sched:sched_migrate_task: comm=stream pid=494785 prio=120 orig_cpu=0 dest_cpu=48
          stream-nnn x: sched:sched_migrate_task: comm=stream pid=494786 prio=120 orig_cpu=0 dest_cpu=56
...

Several options were considered for the test system that were unnecessary for STREAM running on the AMD EPYC 9004 Series Processor but may be useful in other situations. STREAM performance can be limited if a load/store instruction stalls to fetch data. CPUs may automatically prefetch data based on historical behavior but it is not guaranteed. In limited cases, depending on the CPU and workload, this may be addressed by specifying -fprefetch-loop-arrays and depending on whether the workload is store-intensive, -mprefetchwt1. In the test system using AMD EPYC 9004 Series Processors, explicit prefetching did not help and was omitted. This is because an explicitly scheduled prefetch may disable a CPU's predictive algorithms and degrade performance. Similarly, for some workloads branch mispredictions can be a major problem, and in some cases breach mispredictions can be offset against I-Cache pressure by specifying -funroll-loops. In the case of STREAM on the test system, the CPU accurately predicted the branches rendering the unrolling of loops unnecessary. For math-intensive workloads it can be beneficial to link the application with -lmvec depending on the application. In the case of STREAM, the workload did not use significant math-based operations and so this option was not used. Some styles of code blocks and loops can also be optimized to use vectored operations by specifying -ftree-vectorize and explicitly adding support for CPU features such as -mavx2. In all cases, STREAM does not benefit as its operations are very basic. But it should be considered on an application-by-application basis and when building support libraries such as numerical libraries. In all cases, experimentation is recommended but caution advised. This holds particularly true when considering options like prefetch that may have been advisable on much older CPUs or completely different workloads. Such options are not universally beneficial or always suitable for modern CPUs such as the AMD EPYC 9004 Series Processors.

In the case of STREAM running on the AMD EPYC 9004 Series Processor, it was sufficient to enable -Ofast. This includes the -O3 optimizations to enable vectorization. In addition, it gives some leeway for optimizations that increase the code size with additional options for fast math that may not be standards-compliant.

For OpenMP, the SPREAD option was used to spread the load across L3 caches. OpenMP has a variety of different placement options if manually tuning placement. But careful attention should be paid to OMP_PLACES, given the importance of the L3 Cache topology in AMD EPYC 9004 Series Processors, if the operating system does not automatically place tasks appropriately. At the time of writing, it is not possible to specify l3cache as a place similar to what MPI has. In this case, the topology will need to be examined either with library support such as hwloc, directly via the sysfs or manually. While it is possible to guess via the CPUID, it is not recommended as CPUs may be offlined or the enumeration may vary between platforms because of BIOS implementations. An example specification of places based on L3 cache for the test system is:

{0:8,192:8},   {8:8,200:8},   {16:8,208:8},  {24:8,216:8},  {32:8,224:8},  {40:8,232:8},
{48:8,240:8},  {56:8,248:8},  {64:8,256:8},  {72:8,264:8},  {80:8,272:8},  {88:8,280:8},
{96:8,288:8},  {104:8,296:8}, {112:8,304:8}, {120:8,312:8}, {128:8,320:8}, {136:8,328:8},
{144:8,336:8}, {152:8,344:8}, {160:8,352:8}, {168:8,360:8}, {176:8,368:8}, {184:8,376:8}

Figure 2, “STREAM Bandwidth, Single Threaded and Parallelized” shows the reported bandwidth for the single and parallelized case. The single-threaded bandwidth for the basic Copy vector operation on a single core was 50.3 GB/sec. This is higher than the theoretical maximum of a single DIMM, but the IO die may interleave accesses, and caching effects and prefetch still apply. The total throughput for each parallel operation ranged from 1259 GB/sec to 1587 GB/sec depending on the type of operation and how efficiently memory bandwidth was used. This is very roughly scaling with the number of memory channels available on the machine with caching effects accounting for results exceeding theoretical maximums.

Note: STREAM scores

Higher STREAM scores can be reported by reducing the array sizes so that cache is partially used with the maximum score requiring that each threads memory footprint fits inside the L1 cache. Additionally, it is possible to achieve results closer to the theoretical maximum by manual optimization of the STREAM benchmark using vectored instructions and explicit scheduling of loads and stores. The purpose of this configuration was to illustrate the impact of properly binding a workload that can be fully parallelized with data-independent threads.

Figure 2: STREAM Bandwidth, Single Threaded and Parallelized #

  

NASA Parallel Benchmark (NPB) is a small set of compute-intensive kernels designed to evaluate the performance of supercomputers. They are small compute kernels derived from Computational Fluid Dynamics (CFD) applications. The problem size can be adjusted for different memory sizes. Reference implementations exist for both MPI and OpenMP. This setup will focus on the MPI reference implementation.

While each application behaves differently, one common characteristic is that the workload is very context-switch intensive, barriers frequently yield the CPU to other tasks and the lifetime of individual processes can be very short-lived. The following paragraphs detail the tuning selected for this workload.

The most basic step is setting the CPU governor to “performance” although it is not mandatory. This can address issues with short-lived or mobile tasks failing to run long enough for a higher P-State to be selected even though the workload is very throughput-sensitive. The migration cost parameter is set to reduce the frequency in which the load balancer will move an individual task. The minimum granularity is adjusted to reduce overscheduling effects.

Depending on the computational kernel used, the workload may require a power-of-two number or a square number of processes to be used. However, note that using all available CPUs can mean that the application can contend with itself for CPU time. Furthermore, as IO may be issued to shared memory backed by disk, there are system threads that also need CPU time. Finally, if there is CPU contention, MPI tasks can be stalled waiting on an available CPU and OpenMPI may yield tasks prematurely if it detects there are more MPI tasks than CPUs available. These factors should be carefully considered when tuning for parallelized workloads in general and MPI workloads in particular.

In the specific case of testing NPB on the System Under Test, there was usually a limited advantage to limiting the number of CPUs used. For the Embarrassingly Parallel (EP) load in particular, it benefits from using all available CPUs. Hence, the default configuration used all available CPUs (384) which is both a power-of-two and square number of CPUs because it was a sensible starting point. However, this is not universally true. Using perf, it was found that some workloads are memory-bound and do not benefit from a high degree of parallelization. In addition, for the final configuration, some workloads were parallelized to have one task per L3 cache in the system to maximize cache usage. The exception was the Scalar Pentadiagonal (SP) workload which was both memory-bound and benefited from using all available CPUs. As the number of cores can vary between chips and the number of populated memory channels, the tuning parameters used for this test may not be universally true for all AMD EPYC platforms. This highlights that there is no universal good choice for optimizing a workload for a platform. Thus, experimentation and validation of tuning parameters is vital.

The basic compilation flags simply turned on all safe optimizations. The tuned flags used -Ofast which can be unsafe for some mathematical workloads but generated the correct output for NPB. The other flags used the optimal instructions available on the distribution compiler and vectorized some operations. GCC 11 is more strict in terms of matching types in Fortran. Depending on the version of NPB used, it may be necessary to specify the -fallow-argument-mismatch or -fallow-invalid-boz to compile unless the source code is modified.

As NPB uses shared files, an XFS partition was used for the temporary files. It is, however, only used for mapping shared files and is not a critical path for the benchmark and no IO tuning is necessary. In some cases, with MPI applications, it will be possible to use a tmpfs partition for OpenMPI. This avoids unnecessary IO assuming the increased physical memory usage does not cause the application to be paged out.

Figure 3, “NAS MPI Results” shows the time, as reported by the benchmark, for each of the kernels to complete.

Figure 3: NAS MPI Results #

  

The gcc-7-default test used the system compiler, all available CPUs, basic compilation options and the performance governor. The second test gcc-11-default used an alternative compiler. gcc-11-tuned used additional compilation options, and bound tasks to L3 caches gaining between 1.89% and 43.4% performance on average relative to gcc-7-default. Even taking into account that ep.D is an outlier because of the fact it is an embarrassingly parallel workload, the range of improvements is between 1.89% and 11.65%. The final test selective used processes that either used all CPUs, avoided heavy overloaded or limited processes to one per L3 cache, showing additional gains between 0.7% and 15.56% depending on the computational kernel.

In some cases, it will be necessary to compile an application that can run on different CPUs. In such cases, -march=znver3 may not be suitable if it generates binaries that are incompatible with other vendors. In such cases, it is possible to specify the ISA-specific options that are cross-compatible with many x86-based CPUs such as -mavx2, -mfma, -msse2 or msse4a while favoring optimal code generation for AMD EPYC 9004 Series Processors with -mtune=znver3. This can be used to strike a balance between excellent performance on a single CPU and great performance on multiple CPUs.

Even if the main purpose of a server is “limited” to running VMs, some activities will be carried out on the host OS. In fact, in both Xen and KVM, the host OS is at least responsible for helping with the IO of the VMs. It may, therefore, be necessary to make sure that the host OS has some resources (namely, CPUs and memory) exclusively assigned to itself.

Note: Host OS on KVM and on Xen

On KVM, the host OS is the SUSE Linux Enterprise distribution installed on the server, which then loads the hypervisor kernel modules. On Xen, the host OS still is a SUSE Linux Enterprise distribution, but it runs inside what is to all the effect an (although special) Virtual Machine (called Domain 0, or Dom0).

In the absence of any specific requirements involving host resources, a good rule of thumb suggests that 5 to 10 percent of the physical RAM should be reserved to the host OS. On KVM, increase that quota in case the plan is to run many (for example hundreds) of VMs. On Xen, it is okay to always give dom0 not more than a few GigaBytes of memory. This is especially the case when planning to take advantage of disaggregation (see https://wiki.xenproject.org/wiki/Dom0_Disaggregation).

In terms of CPUs, depending on the workload, it may be fine to use all the physical CPUs for the VMs (and this is in fact how the benchmarks in the experimental section of this guide have been conducted). On the other hand, if it is necessary to keep some CPUs for the host Os / dom0, we advise to reserve at least one physical core on each NUMA node. This is especially true for a system like the one show in Figure 1, “AMD EPYC 9004 Series Classic Processor Topology”. In fact, host OS activities are mostly related to performing IO for VMs and it is beneficial for performance if the kernel threads that handle devices can run on the nodes to which the devices themselves are attached, which is both NUMA nodes, in our case.

System administrators need to be able to reach out and login to the system, to monitor, manage and troubleshoot it. Therefore, it is possible that even more resources needs to be assigned to the host OS. This would be for making sure that management tools (for example, the SSH daemon) can be reached, and that the hypervisor's toolstack (for example, libvirt) can run without too much contention.

dom0_mem=65536M,max:65536M dom0_max_vcpus=48

xl vcpu-pin 0 <vcpu-ID> <pcpu-list>

virsh vcpupin 0 -\-vcpu <vcpu-ID> -\-cpulist <pcpu-list>

For virtualization workloads, rather than using Transparent Huge Pages (THP) on the host, it is recommended that 1 GB huge pages are used for the memory of the VMs. This sensibly reduces both the page table management overhead and the level of resource contention that the system faces when VMs update their page tables. Moreover, if the host is entirely devoted to running VMs, Huge Pages are likely not required for host OS processes. Actually, in this case, having them active on the host may even negatively affect performance, as the THP service daemon (which is there for merging “regular” pages and form Huge Pages) can interfere with VMs' vCPUs execution. For disabling Huge Pages for the host OS, in a KVM setup, add the following host kernel command-line option:

transparent_hugepage=never

Or execute this command, at runtime:

echo never > /sys/kernel/mm/transparent_hugepage/enabled

To use 1 GB Huge Pages as backing memory of KVM guests, such pages need to be allocated at the host OS level. It is best to make that happen by adding the following boot time parameter:

default_hugepagesz=1GB hugepagesz=1GB hugepages=<number of hugepages>

The value <number of hugepages> can be computed by taking the amount of memory we want to devoted to VMs, and dividing it by the page size (1 GB). For example on our host with 754 GB of RAM, creating 672 Huge Pages means we can accommodate up to GB of VMs' RAM, and leave plenty (~80GB) to the host OS.

On Xen, none of the above is necessary. In fact, Dom0 is a paravirtualized guest, for which Huge Pages support is not present. On the other hand, for the memory used for the hypervisor and for the memory allocated by the hypervisor for HVM VMs, Huge Pages are always used as much as possible. So no explicit tuning is needed.

On KVM, NUMAB can be useful in dynamic virtualization scenarios, where VMs are created, destroyed and re-created relatively quickly. Or, in general, it can be useful in cases where it is not possible or desirable to statically partition the host resources and assign them explicitly to VMs. This, however, comes at the price of some latency being introduced. Plus, NUMAB's own operation can interfere with VMs' execution and cause further performance degradation. In any case, this document focuses on achieving the best possible performance for VMs, through specific tuning and static resource allocation, therefore it is recommended to leave NUMAB turned off. This can be done by adding the following parameter to the host kernel command line:

numa_balancing=disable

If anything changes and the system is repurposed to achieve different goals, NUMAB can be enabled on-line with the command:

echo 0 > /proc/sys/kernel/numa_balancing

On Xen, Automatic NUMA Balancing (NUMAB) in the host OS should be disabled. Dom0 is, currently, a paravirtualized guest without a (virtual) NUMA topology and, thus, NUMAB would be totally useless.

Service daemons have been discussed already, in the first part of the document. Most of the consideration done there, applies here as well.

For example, tuned should either not be used, or the profile should be set to one that does not implicitly put the CPUs in polling mode. Both throughput-performance and virtual-host profiles from SUSE Linux Enterprise Server 15 SP4 are okay, as neither of them touches /dev/cpu_dma_latency.

irqbalance can be a source of latency, for no significant performance improvement. The suggestion is again to disable it for latency sensitive workloads. This means, though, that IRQs may need to be manually bound to the appropriate CPUs, considering the IO topology.

As far as power management is concerned, the cpufreq governor can either be kept as it is by default, or switched to performance, depending on the nature of the workloads of interest.

Note: Power management

For anything that concerns power management on KVM, changing the tuned profile or using cpupower, from the host OS will have the effects one can expect, and described already in the first half of the document. On Xen, CPU frequency scaling is enacted by the hypervisor. It can be controlled from within Dom0, but the tool that needs to be used is different. For example, for setting governor to performance, we need the following:

xenpm set-scaling-governor performance

Secure Encrypted Virtualization (SEV) and SEV with Encrypted State (SEV-ES) technologies, are available on the 5th Generation AMD EPYC Processors when SUSE Linux Enterprise Server 15 SP4 is used both as a host and guest OS. They allow the memory of the VMs to be encrypted, enabling a high level of confidentiality. SEV-ES is considered superior to plain SEV, as also CPU registers are encrypted when they are saved into the host memory. For using SEV-ES for a VM, we need to enable it in the VM's own configuration file, but there are preparation steps that needs to occur at the host level.

The libvirt documentation contains all the necessary steps required for enabling SEV-ES on a host that supports it. It has been enough to add the following boot parameters to the host kernel command line, in the boot loader:

mem_encrypt=on kvm_amd.sev=1

For further details, refer to Libvirt documentation: Enabling SEV on the host.

It should be noted that, at least as far as the workload analyzed in this document, enabling SEV-ES on the host has not caused any noticeable performance degradation. In fact, running CPU and memory intensive workloads, both on the host and in VMs, with or without the parameters above added to the host kernel boot command line, resulted in indistinguishable results. Therefore, enabling SEV-ES at the host level can be considered safe, from the point of view of not hurting performance of VMs and workloads that will not take advantage of VM memory encryption.

On the other hand, what happens when SEV and SEV-ES are used for encrypting VMs' memory is described later in the guide.

Note: SEV on Xen

In SUSE Linux Enterprise Server 15 SP4, SEV-ES is not available on Xen.

When a VM is created, memory is allocated on the host to act as its virtual RAM. Moving this memory, for example on a different NUMA node from the one where it is first allocated, incurs (when possible) in a significant performance impact. Therefore, it is of paramount importance that the initial placement of the VMs is as optimal as possible. Both Xen and KVM can make “educated guesses” on what a good placement might be. For the purpose of this document, however, we are interested in what is the absolute best possible initial placement of the VMs, taking into account the specific characteristics of AMD EPYC 9004 Series Processor systems. And this can only be achieved by manually doing the initial placement. Of course, this comes at the price of reduced flexibility, and is only possible when there is no oversubscription.

Deciding on what pCPUs the vCPUs of a VM run, can be done at creation time, but also easily changed along the VM lifecycle. It is, however, still recommended to start the VMs with good vCPUs placement. This is particularly important on Xen, where vCPU placement is what actually drives and controls memory placement.

Placement of VM memory on host NUMA nodes happens by means of the <numatune> XML element:

<numatune>
  <memory mode='strict' nodeset='0-1'/>
  <memnode cellid='0' mode='strict' nodeset='0'/>
  <memnode cellid='1' mode='strict' nodeset='1'/>
  ...
</numatune>

The 'strict' guarantees that the all memory for the VM will come only from the (set of) NUMA node(s) specified in nodeset. A cell is, in fact, a virtual NUMA node, with cellid being its ID and nodeset telling exactly from what host physical NUMA node(s) the memory for this virtual node should be allocated.

The correctness of this kind of tuning can be verified checking on which NUMA node(s) the memory of the QEMU processes representing the VMs have been allocated on, by using the numastat tool (on the host) like this:

host:~ # numastat -p qemu-system-x86_64
Per-node process memory usage (in MBs) for PID 53153 (qemu-system-x86)
                           Node 0          Node 1           Total
                  --------------- --------------- ---------------
Huge                    344064.00       344064.00       688128.00
Heap                         0.00          150.57          150.57
Stack                        0.00            0.13            0.13
Private                    149.59         4522.84         4672.43
----------------  --------------- --------------- ---------------
Total                   344213.59       348737.54       692951.12

In fact, we see that there is only 1 QEMU process (as there is only 1 VM running) and its memory footprint has been equally split between the two NUMA nodes.

A NUMA-unaware VM can still include this element in its configuration file. It will have only one <memnode> element, and the output of numastat will look like this:

host:~ # numastat -p qemu-system-x86_64
Per-node process memory usage (in MBs)
PID                               Node 0          Node 1           Total
-----------------------  --------------- --------------- ---------------
184717 (qemu-system-x86)       346040.98           19.30       346060.28
184980 (qemu-system-x86)           19.50       346227.04       346246.54
-----------------------  --------------- --------------- ---------------
Total                          346060.48       346246.34       692306.82

In this case there are 2 VMs running and their memory have been pretty much entirely allocated on a single NUMA node: Node 0 for the VM associated to the QEMU process with PID 184717, and Node 1 for the other one.

Placement of vCPUs happens via the <cputune> element, as in the example below:

<vcpu placement='static'>288</vcpu>
<cputune>
  <vcpupin vcpu='0' cpuset='0'/>
  <vcpupin vcpu='1' cpuset='192'/>
  <vcpupin vcpu='2' cpuset='1'/>
  <vcpupin vcpu='3' cpuset='193'/>
  ...
</cputune>

The value 288 means that the VM will have 288 vCPUs. The static guarantees that each vCPU will stay on the pCPU(s) on which it is “pinned” to. The various <vcpupin> elements are where the mapping between vCPUs and pCPUs is established (vcpu being the vCPU ID and cpuset being either one or a list of pCPU IDs).

To be able to create VMs with more than 255 vCPUs, the following element should be added in the <device> section:

<features>
  ...
  <ioapic driver="qemu"/>
</features>
...
<devices>
  ...
  <iommu model='intel'>
    <driver intremap='on'/>
  </iommu>
  ...
</devices>

When pinning vCPUs to pCPUs, adjacent vCPU IDs (like vCPU ID 0 and vCPU ID 1) must be assigned to host SMT siblings (like pCPU 4 and pCPU 196, on our test server). In fact, QEMU uses a static SMT sibling CPU ID assignment. This is how we guarantee that virtual SMT siblings will always execute on actual physical SMT siblings.

The following sections give more specific and precise details about placement of vCPUs and memory for VM of varying sizes, on the reference system for this guide (see Figure 1, “AMD EPYC 9004 Series Classic Processor Topology”.

Note that, despite the fact that the server has 384 pCPUs, a VM in SUSE Linux Enterprise 15 SP4 cannot have more than 288 vCPUs (the value will be higher in future versions of SUSE Linux Enterprise Server). Of course, we can still use all the pCPUs of the server, when creating multiple VMs.

vm1:~ # numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143
node 0 size: 354661 MB
node 0 free: 352996 MB
node 1 cpus: 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287
node 1 size: 321929 MB
node 1 free: 320767 MB
node distances:
node   0   1
  0:  10  32
  1:  32  10

vm1:~ # numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191
node 0 size: 676778 MB
node 0 free: 675046 MB
node distances:
node   0
  0:  10

17.1.4 Placement of many small VMs #

  

If small VMs are the target, VMs with either 8, 4 or 2 vCPUs each can be created and efficiently allocated on AMD EPYC 9004 Series Processors servers.

VMs with 8 vCPUs “occupies” half a die, and we can have 48 of them. VMs with either 4 or 2 vCPUs will be given 2 and 1 host cores and we can have 96 and 192 of them, respectively. And this is all possible without the need for any of the VMs to span more than one host NUMA node.

Of course, in all these cases, VMs are sharing dies, which means they will interfere with each other via the L3 caches. This may or may not be a problem, but there is no way around it, as soon as more VMs than the number of available dies are necessary. The performance impact of such sharing should be tolerable, in most cases, for these configurations, but this needs to be assessed with tests and benchmarks. If that is not the case, then the recommendation is to not go above 24 VMs. More clever (but also more complex) tuning strategies can be designed, but they would require further a-priori knowledge of the workload and/or the load profile of the various VMs, which may not be always available (or can change during the lifecycle of the VMs).

Note: Hundreds of VMs with limited RAM size and disk sapce

When the number of VM increases so much, the limiting factor will become the memory or the disk. With ~700 GB of RAM available for VMs on our server, we can give only 3 GB of RAM to each of the 192 VMs of the last example above. Of course, this limitation is only an issue of the system used as reference for this guide. The AMD EPIC 9004 Series Processor architecture itself can accommodate much more RAM.

In cases when it is enough or desirable to have VMs with only 1 vCPUs, it is recommended to still not go beyond 192, and assign no less than one full core to each VM. It should be avoided to have vCPUs from different VMs running on two siblings threads of the same core. Such a solution, although functional, is not ideal for all the cases where good and consistent performance within VMs is a goal (as well as for security concerns, but that is out of the scope of this guide). What can be done for each VM and for each core is to use one of the two threads for the actual vCPU, and the other for its IO thread(s), on the host. This solution allows to take advantage of the full 4th Generation AMD EPYC processing power. However, we advise to check and verify whether this is a good setup and if it brings performance benefits for the specific workloads of interest, before committing to it.

IO for the VMs is carried out by device emulators running on the host, or by the so-called back-ends of paravirtualized drivers (also running on the host). Both the IO threads of the emulators and the back-ends are seen as (either user or kernel) threads by the host OS. As such, they can be configured to run on specific subsets of the CPUs assigned to the host OS (Dom0’s virtual CPUs, in the case of Xen). And their placement on such CPUs may have an impact on performance of VMs, especially on IO bound workloads.

For example, a 1 vCPU VM can have its vCPU bound to the one SMT thread of a core, while the other thread can be assigned to the host, and the IO thread(s) of the VM pinned to it. The idea behind this is that a common execution pattern for the vCPU is to be idle when the IO threads are busy. Hence this setup potentially maximizes the exploitation of hardware resources.

On Xen, there is also the possibility of setting up driver domains. These are special VMs which act as back-ends of a particular IO device, for one or more VMs. In case they are used, make sure that they run “close” enough to both the hardware they are providing their abstraction for, and the VMs that they are servicing.

Oversubscription happens when the demand for some resource is higher than the resource is physically available. In virtualization, this can happen for CPU and memory.

Note: Not covering oversubscribed scenarios

This guide does not cover in great details oversubscribed scenarios. However, given the large number of CPUs available on an AMD EPYC 9004 Series Processor system and the huge amount of memory the architecture supports, this is not considered a limitation.

CPU oversubscription happens when the total cumulative number of vCPUs from all VMs becomes higher than 384. Such a situation inevitably introduces latencies, resulting in lower performance compared to when host resources are sufficient. It is, however, impossible to tell a priori by what extent this happens, at least not without a detailed knowledge of the actual workload.

If we knew in advance that the load on each vCPU will always stay below 50%, we could assume that even oversubscribing the host by a factor of 2 (like with ~768 vCPUs in total, in our case!) will work well. On the other hand, if we knew that the load that each vCPU tries to impose on the system is always 100%, creating even 1 vCPUs more than the host has pCPUs can be considered a misconfiguration.

When oversubscription is necessary, exclusive 1-to-1 vCPU to pCPU assignment may not the best approach and we need to trust the hypervisor scheduler to handle the situation well. What can be done, though, is providing such scheduler at least with some “suggestions”, if we have enough knowledge about the workload. For example, we can try to help the scheduler avoiding that all the VMs running CPU intensive workloads end on the same nodes, or avoiding cross-node VM migrations happening too frequently.

This grouping of VMs on (a set of) nodes can be done with some less strict forms of vCPU pinning (often called “affinity”), or with other mechanisms. For example, cgroups can be leveraged on a KVM host, or cpupools on a Xen host. Xen also contains a feature, called soft vCPU affinity, which can be used together with “traditional” vCPU affinity (also called hard vCPU affinity) as a finer grained and more powerful way of controlling resource allocation in oversubscribed scenario.

Memory oversubscription happens when the total cumulative amount of memory used by all VMs is more than the RAM available on the host. In this case, some of such memory is kept outside of the RAM (swapped out) when the VMs using it are not running. It is put back in RAM (swapped in) when these VMs run again. This also has (potentially severe) performance implications, and is not analyzed in details in here.

On KVM, swapping is handled by Linux virtual memory management and paging code and mechanism, exactly as it happens for host/bare metal processes. On Xen, this is not supported unless special technologies (xenpaging or transcendent memory) are employed.

Then there are page sharing and memory ballooning.

Page sharing relies on the principle that, when VMs use memory pages which are identical, only one copy of them needs to be kept in physical RAM, at least for most of the time. This is supported on KVM, via a mechanism called Kernel Samepage Merging (KSM). On Xen, this again require special tools and actions.

Ballooning relies on the idea that VMs do not always use all the memory they are configured to be able to use. This is supported in both Xen and KVM. However, using it is not ideal for performance, thus it is not analyzed in any further details in this document.

This is always considered a misconfiguration. In fact, in some more details:

The term “enlightenment” refers to letting the guest OS know as many details as possible about the virtual topology of the VM. This is only useful and brings actual performance improvements only if such topology is properly and effectively mapped on host resources, and if such mapping is stable and does not change during the life of the VM.

  <cpu mode="host-model" check="none">
    <topology sockets="2" dies="9" cores="8" threads="2"/>
    <numa>
      <cell id="0" cpus="0-143" memory="369098752" unit="KiB">
        <distances>
          <sibling id="0" value="10"/>
          <sibling id="1" value="32"/>
        </distances>
      </cell>
      <cell id="1" cpus="144-287" memory="369098752" unit="KiB">
        <distances>
          <sibling id="0" value="32"/>
          <sibling id="1" value="10"/>
        </distances>
      </cell>
    </numa>
  </cpu>

vm1:~ # lscpu
...
CPU(s):                  288
  On-line CPU(s) list:   0-287
Vendor ID:               AuthenticAMD
  Model name:            AMD EPYC 9654 96-Core Processor
    CPU family:          25
    Model:               17
    Thread(s) per core:  1
    Core(s) per socket:  288
    Socket(s):           1
...
Caches (sum of all):
  L1d:                   18 MiB (288 instances)
  L1i:                   18 MiB (288 instances)
  L2:                    144 MiB (288 instances)
  L3:                    4.5 GiB (288 instances)
...

vm1:~ # cat /sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list
0
vm1:~ # cat /sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list
0
vm1:~ # cat /sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list
0
vm1:~ # cat /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list
0-143

vm1:~ # lscpu
...
CPU(s):                  288
  On-line CPU(s) list:   0-287
Vendor ID:               AuthenticAMD
  Model name:            AMD EPYC-Milan Processor
    CPU family:          25
    Model:               1
    Thread(s) per core:  2
    Core(s) per socket:  72
    Socket(s):           2
...
Caches (sum of all):
  L1d:                   4.5 MiB (144 instances)
  L1i:                   4.5 MiB (144 instances)
  L2:                    72 MiB (144 instances)
  L3:                    576 MiB (18 instances)
...

vm1:~ # cat /sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list
0-1
vm1:~ # cat /sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list
0-1
vm1:~ # cat /sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list
0-1
vm1:~ # cat /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list
0-15

<topology sockets="2" dies="9" cores="8" threads="2"/>

vm1:~ # lscpu
...
CPU(s):                  288
  On-line CPU(s) list:   0-287
Vendor ID:               AuthenticAMD
  Model name:            AMD EPYC-Milan Processor
    CPU family:          25
    Model:               1
    Thread(s) per core:  2
    Core(s) per socket:  72
    Socket(s):           2
...
Caches (sum of all):
  L1d:                   4.5 MiB (144 instances)
  L1i:                   4.5 MiB (144 instances)
  L2:                    72 MiB (144 instances)
  L3:                    64 MiB (2 instances)
...
vm1:~ # cat /sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list
0-1
vm1:~ # cat /sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list
0-1
vm1:~ # cat /sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list
0-1
vm1:~ # cat /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list
0-143
...

<topology sockets="1" dies="12" cores="8" threads="2"/>

vm1:~ # lscpu
...
CPU(s):                  192
  On-line CPU(s) list:   0-191
Vendor ID:               AuthenticAMD
  Model name:            AMD EPYC-Milan Processor
    CPU family:          25
    Model:               1
    Thread(s) per core:  2
    Core(s) per socket:  96
    Socket(s):           1
...
Caches (sum of all):
  L1d:                   3 MiB (96 instances)
  L1i:                   3 MiB (96 instances)
  L2:                    48 MiB (96 instances)
  L3:                    384 MiB (12 instances)
...

vm1:~ # cat /sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list
0-1
vm1:~ # cat /sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list
0-1
vm1:~ # cat /sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list
0-1
vm1:~ # cat /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list
0-15

vm1:~ # modprobe cpuidle-haltpoll

17.4.2 Memory backing #

  

Memory wise, the VMs must be told to use the Huge Pages that were reserved for them. To effectively use 1 GB huge pages, the amount of memory each VM is given must be a multiple of 1 GB. Also, KSM should be disabled and we also must ensure that VMs' memory is never going to be swapped out. This is all achieved as follows:

<memory unit='KiB'>704643072</memory>
<memoryBacking>
  <hugepages>
    <page size='1' unit='GiB'/>
  </hugepages>
  <nosharepages/>
</memoryBacking>

To verify that the appropriate type of memory is being used by the VMs, one can check the content of /proc/meminfo, with the VMs running, and observe that all the pre-allocated Huge Pages are actually occupied.

host:~ # cat /proc/meminfo | grep Huge
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:     672
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB
Hugetlb:        704643072 kB

Note: Huge Pages, shared pages and locked pages on Xen

On Xen, for HVM guests, huge pages are used by default and there is neither any page sharing nor swapping (at least not in SUSE Linux Enterprise Server 15 SP4). Therefore, the entire <memoryBacking> element is technically not necessary.

<memory unit='KiB'>704643072</memory>
<currentMemory unit='KiB'>704643072</currentMemory>

The following documents

Explain how to configure a VM to use AMD SEV-ES (on an AMD SEV-ES capable and properly configured host) is not in the scope of this document. Refer to the linked materials for the details.

It is possible to check if the host is properly configured to run SEV-ES VMs with the following command:

dmesg |grep SEV
[   32.668559] ccp 0000:02:00.5: SEV API:1.52 build:18
[   39.219320] SEV supported: 495 ASIDs
[   39.225083] SEV-ES supported: 511 ASIDs

Once inside of the VM, this is how one can check whether it is only the memory that is being encrypted (with SEV):

dmesg | grep SEV
[    0.069436] AMD Memory Encryption Features active: SEV

Or if we are also fully encrypting the VM state (with SEV-ES):

dmesg | grep SEV
[    0.069436] AMD Memory Encryption Features active: SEV SEV-ES

Note that it is not possible to provide a virtual IOMMU device to an encrypted VM. This means that, for having a functional SEV or SEV-ES VM, we need to remove the iommu element, in the <device> section of the VM configuration. And since that element is necessary for having more than 255 vCPUs in the VM, all the experiments conducted with SEV enabled have been done in VMs with at most 192 vCPUs.

Figure 4, “STREAM Bandwidth - Bare metal compared with one VM” shows the bandwidth achieved by STREAM when the benchmark is built in single thread mode ('single') or parallelized with OpenMP ('ompenmp'). For both cases, we compare the results achieved on the host ('BM') and inside one VM ('VM').

Figure 4: STREAM Bandwidth - Bare metal compared with one VM #

  

The single thread results identical between baremetal (blue rectangles) and inside of the VM (orange rectangles), for all the operations (Copy, Scale, Add and Triadd) of the benchmark.

About the parallel case, remember that the VM is slightly "smaller"" than the host, in terms of number of CPUs. Therefore, we cannot run the parallel version of STREAM with as many thread as on the host. In fact, a very good result is reached, on the host, with twice as many threads as there are Least Level Caches (LLCs), that is 48. If we do the same inside of the VM, there it means 36 threads. We can see, however, that the performance reached inside of the VM, even with 36 threads (dark red rectangles, ~616 GBytes/sec for the Copy operation) is close enough to the values achieved on the host with 48 threads (yellow rectangles, ~640 GBytes/sec for the Copy operation). For completeness, we also run the benchmark on the host with 36 threads (green rectangles, ~605 Gbytes/sec for the Copy operation); and as we could have expectd, the results of such baremetal runs and of the VM runs are very close.

This clearly shows how proper tuning allows a single VM running on an AMD EPYC 7004 Series Processor server to achieve a memory bandwidth performance that matches the one that we can reach directly on the host.

Note

Inside of the VM, the STREAM benchmark was configured almost identically to what has been shown already in Section 13.2, “Test workload: STREAM”.

Figure 5, “STREAM Bandwidth - Single thread in one VM with different CPU models” and Figure 6, “STREAM Bandwidth - OpenMP in one VM with different CPU models” show the effect of using different CPU models for the virtual machine. We see that, as long as a single thread is used, the CPU model choosen for the VM is completely irrelevant.

On the other hand, the OpenMP run clearly shows some problems when the host-passthrough CPU model is selected. In fact, since using that model builds a VM with only 2 LLCs (and also to other problems with the cache topology), running STREAM with twice as many threads as there are LLCs in the system, results in the benchmark spawning only 4 of them (yellow and green rectangles). And this, of course, dramatically reduces the performance. We can also see that, if we instead manually set the number of threads to the value that we know to be the best for this VM (that is 36, dark red and cyan rectangles) performance are restored to how good we know things can be from Figure 4, “STREAM Bandwidth - Bare metal compared with one VM” (and from the cpumodel results, see the blue and orange rectangles).

Actually, the cyan rectangles represent the absolute best result (~620 GBytes/sec for the Copy operation), probably thanks to the fact that the CPUIdle haltpoll governor is the most effective when coupled with cpupassthrough. However, the orange rectangles follow very closely (~614 Gbytes/sec for the Copy operation). And it is probably not worthwhile to tolerate a completely inaccurate cache hierarchy inside of the VM, for such a small gain. In fact, sticking to cpupassthrough would likely mean having to manually tweak most applications, and this could be much more complicated it has been here for the STREAM benchmark or, sometimes, even impossible. For this reason, this document recommends cpumodel as the best compromise between convenience and absolute best performance.

Figure 5: STREAM Bandwidth - Single thread in one VM with different CPU models #

  

Figure 6: STREAM Bandwidth - OpenMP in one VM with different CPU models #

  

This situation also confirms the need to always run benchmarks for assessing the performance of the relevant workloads, on a given platform. In fact, because of specific characteristics of some components of the virtualization stack available in SUSE Linux Enterprise Server 15 SP4, the EPYC-Milan (or, equivalently, the host-model) CPU model might be preferable to what it would have appeared to be the most obvious choice (host-passthrough).

Finally, about the CPUIdle haltpoll governor, it is recommended to enable and make use of it, although for STREAM (and we can guess also for other, similar, memory intensive benchmarks) it introduces no significant performance difference.

Figure 7, “STREAM Bandwidth - Single, average of the bandwidth achieved among all VMs of each group” and Figure 8, “STREAM Bandwidth - Single, sum of the bandwidth achieved within all VMs of each group” show what happens when 1, 2, 4, 6, 12 and 24 VMs are used. The former reports the average bandwidth achieved in each experiments, considering all the VMs involved. For example, the yellow blue bar (4 VMS) is relative to an experiment where 4 VMs were concurrently running the STREAM benchmark (in single thread mode) and represents the average of the 4 different memory bandwidth values, one for each of such VMs. The latter shows something similar, but the bars represents the cumulative STREAM bandwidth achieved in each experiments, considering all the VMs involved. For example, the cyan bar (24 VMs) is relative to an experiment where 24 VMs were concurrently running the STREAM benchmark (in single thread mode) and represents the sum of the 24 different memory bandwidth values, one of each of such VMs.

Figure 7: STREAM Bandwidth - Single, average of the bandwidth achieved among all VMs of each group #

  

Figure 8: STREAM Bandwidth - Single, sum of the bandwidth achieved within all VMs of each group #

  

We see in Figure 7, “STREAM Bandwidth - Single, average of the bandwidth achieved among all VMs of each group” how the single STREAM bandwidth stays flat until (and including when) 6 VMs are used. Then it starts to decline a bit, as more VMs are packed on the NUMA nodes and, hence, compete for the bandwith of the memory controllers. However, Figure 8, “STREAM Bandwidth - Single, sum of the bandwidth achieved within all VMs of each group” reminds us how the total bandwidth achieved, if we consider all the VMs involved in each experiments, actually goes up, until it reaches the same level that we know (for example, from Figure 4, “STREAM Bandwidth - Bare metal compared with one VM”) it can touch.

Figure 9, “STREAM Bandwidth - OpenMP, average of the bandwidth achieved among all VMs of each group” and Figure 10, “STREAM Bandwidth - OpenMP, sum of the bandwidth achieved within all VMs of each group” show the same, but when the parallel (via OpenMP) version of STREAM is run, inside of each VM of each experiment.

Figure 9: STREAM Bandwidth - OpenMP, average of the bandwidth achieved among all VMs of each group #

  

Figure 10: STREAM Bandwidth - OpenMP, sum of the bandwidth achieved within all VMs of each group #

  

In Figure 9, “STREAM Bandwidth - OpenMP, average of the bandwidth achieved among all VMs of each group” we see that performance scales down in a close to linear fashion with the increase of the number of VMs involved (although, not perfectly linearly for all the type of STREAM operations). At the same time, Figure 10, “STREAM Bandwidth - OpenMP, sum of the bandwidth achieved within all VMs of each group” shows how the total cumulative bandwidth of each experiments stays high, as one would expect. Actually, it even seems that using many small VMs may make it possible to even better exploit the large memory bandwidth provided by the memory controllers of the EPYC 9004 processors, as compared to what happens when few large VMs are running.

The goal for this test was evaluating the impact on the performance of encrypting the VMs with SEV and with SEV-ES, when running a memory intensive workload like STREAM.

Considering the case when 2 VMs are concurrently running on the host and are executing the STREAM benchmark, Figure 11, “STREAM Bandwidth - Single, average bandwidth in 2 VMs when unencrypted, encrypted with SEV and with SEV-ES” shows the average bandwidth of the single thread execution of the benchmark in both such VMs when memory is not encrypted (blue bars), when memory is encrypted with SEV (orange bar) and with both memory and state are encrypted with SEV-ES (yellow bar).

Figure 11: STREAM Bandwidth - Single, average bandwidth in 2 VMs when unencrypted, encrypted with SEV and with SEV-ES #

  

Figure 12: STREAM Bandwidth - OpenMP, average bandwidth in 2 VMs when unencrypted, encrypted with SEV and with SEV-ES #

  

It is quite evident how both SEV and SEV-ES are have only a very limited impact on the performance for this workload. In fact, even when SEV-ES is employed, the drop in performance is limited to ~2%.

Figure 13, “NAS Completion Time - Baremetal and in one VM” shows the results of running the various NPB workloads on the host and in one VMs. In both cases, the tuning introduced above was applied (in the latter case, of course, inside of the VM itself).

Figure 13: NAS Completion Time - Baremetal and in one VM #

  

The VM appears slightly solwer than the host, but this should have been expected, considering that the VM can only have 288 CPUs.

Figure 14, “NAS Completion Time - Baremetal and 2 to 12 VMs” depicts the same as Figure 13, “NAS Completion Time - Baremetal and in one VM”, but for a few more cases, in terms of number of concurrently running VMs.

Figure 14: NAS Completion Time - Baremetal and 2 to 12 VMs #

  

We can see how the slowdown is almost always linear with the number of the VMs, which makes sense considering that when we increase the number of VMs, each one of them can also have fewer CPUs (the only exceptions being lu.D and --but only to some extent sp.D)-- especially with 12 VMs). This testifies the effectivens of the suggested tuning measures, when it comes to guerantee a good level of isolation of multiple concurrently running VMs, on the EPYC 9004 processors.

Similarly to what has been done with STREAM, we evaluate here the overhead introduced by SEV and SEV-ES, this time for CPU-bound workloads, and we do that by running the NPB benchmarks it inside two VMs concurrently that are all either unencrypted, encrypted with SEV or encrypted with SEV-ES (Figure 15, “NAS Completion Time - Tuned configuration in 2 VM, with and without SEV”).

Figure 15: NAS Completion Time - Tuned configuration in 2 VM, with and without SEV #

  

As for the memory-intensive case the overhead introduced by either of the Confidential Computing technologies is found to be limited. In fact, the worst case is reached in the SEV-ES case (as it could have be expected), during the bt.D benchmark, but is well within 5% of the unencrypted run.