People working on Linux PCs with Tensorflow 2 [TF2] and CUDA may be confronted with warnings complaining a lack of an assignment of their Nvidia graphics card to a NUMA node. This is somewhat enervating as depending on the TF2 version a default entry of “-1” for the NUMA on consumer systems may clatter some of your Jupyter notebook cells with warnings. In this post I first will briefly turn to the question what NUMA is good for on sever systems with multiple CPUs and GPUs. Afterward I describe two “quick fixes” to get rid of the warnings in your notebooks on a standard Linux PC.
NUMA
NUMA stands for “Non-Uniform Memory Access”. See the following Wikipedia article for an elementary introduction:
Non-uniform memory access. A first description can also be found at the following link at the Nvidia Enterprise Support Portal: Understanding NUMA Node for Performance Benchmarks
NUMA basically refers to server architectures and boards with multiple CPU sockets and bus systems. It supports optimal CPU access to memory. A NUMA node describes an available range of memory and the respective “distance” to a CPU by a numerical value. A Linux administrator would assign the right “local” memory (with lowest distance) to a CPU to enhance performance. The assignment also has an indirect impact on data transfer to devices sitting on the same bus. Certain devices on a PCI-bus can be associated with a NUMA node, too. A wrong NUMA assignment of a device can reduce performance. Regarding GPUs the following paper of 2011 by K. Spafford, J. S. Meredith J. S. Vetter describes such consequences in an understandable way:
Quantifying NUMA and Contention Effects in Multi-GPU Systems
NUMA on Linux PCs with a single CPU socket?
In contrast to servers with multiple CPU sockets, multiple buses, multiple RAM banks and multiple GPUs, the whole topic of memory access may in not be important on standard PC mainboards with just one CPU socket and one GPU. There – if properly configured – you find just one NUMA node. This typically has the number 0.
How can you find out about this? A very good introduction to NUMA-related Linux commands is given at the following Baeldung site: Checking for NUMA Capabilities. Let us apply 2 of the commands mentioned there:
mytux:~ # cat /boot/config-$(uname -r) | grep NUMA
CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y
CONFIG_NUMA_BALANCING=y
CONFIG_NUMA_BALANCING_DEFAULT_ENABLED=y
# CONFIG_X86_NUMACHIP is not set
CONFIG_NUMA=y
CONFIG_AMD_NUMA=y
CONFIG_X86_64_ACPI_NUMA=y
CONFIG_NUMA_EMU=y
CONFIG_USE_PERCPU_NUMA_NODE_ID=y
CONFIG_ACPI_NUMA=y
CONFIG_NUMA_KEEP_MEMINFO=y
# CONFIG_DMA_PERNUMA_CMA is not set
mytux:~ # numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 64241 MB
node 0 free: 52727 MB
node distances:
node 0
0: 10
You see that all cores of the CPU are assigned to the very same NUMA node with number 0. So far there is no problem. The Linux kernel supports NUMA, but – not too surprisingly – detects just one node on my consumer ASRock board.
The problem with consumer cards and the Nvidia drivers
The problem starts when we tray to find out to which node our Nvidia card is assigned to. We would expect that the Nvidia drivers in cooperation with the vendor does something to set it to NUMA node 0 on a PC. But, no … we get a default value of “-1”.
mytux:~ # cat /sys/module/nvidia/drivers/pci:nvidia/*/numa_node
-1
This is something both TF2 and also CUDA react allergic to. Here is a Jupyter cell preparing the Nvidia card:
b_tf_CPU_only = False # we need to work on a GPU
tf_limit_CPU_cores = 4
tf_limit_GPU_RAM = 9400
if b_tf_CPU_only:
tf.config.set_visible_devices([], 'GPU') # No GPU, only CPU
#tf.config.set_visible_devices([], 'CPU') # No CPU - DO NOT USE
# Restrict number of CPU cores
tf.config.threading.set_intra_op_parallelism_threads(tf_limit_CPU_cores)
tf.config.threading.set_inter_op_parallelism_threads(tf_limit_CPU_cores)
else:
gpus = tf.config.experimental.list_physical_devices('GPU')
tf.config.experimental.set_virtual_device_configuration(gpus[0],
[tf.config.experimental.VirtualDeviceConfiguration(memory_limit = tf_limit_GPU_RAM)])
# JiT optimizer
tf.config.optimizer.set_jit(True)
tf.config.experimental.list_physical_devices()
It creates the following output:
2024-05-08 15:05:29.276730: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-05-08 15:05:29.295193: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-05-08 15:05:29.295382: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
Similar warnings may later occur during execution of certain commands related to model compilation and training.
Quick Fix
As a quick fix TF2 warnings can be avoided by executing the following command as root:
mytux:~ # echo 0 | tee /sys/module/nvidia/drivers/pci:nvidia/*/numa_node
0
For references see this question in the “Ask Ubuntu forum” and further links therein:
How to set the NUMA node for an (NVIDIA) GPU persistently?
Regarding a more permanent fix, you also find a recipe there to add a respective task to crontab, for performing the quoted command at reboot. Unfortunately, even after having set the NUMA node the Nvidia Linux driver still does not use it:
rux:~ # echo 0 | tee /sys/module/nvidia/drivers/pci:nvidia/*/numa_node
0
rux:~ # nvidia-smi topo -m
GPU0 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X 0-7 0 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
Also interesting is a question at “Stackoverflow”; read also the comments there:
MemoryError in TensorFlow; and “successful NUMA node read from SysFS had negative value (-1)” with xen
Control the level of TF2 warnings
As another option to get rid of the annoying warnings you may simply set an environment variable “TF_CPP_MIN_LOG_LEVEL”
for the shell of your Juypter notebook or Jupyter lab. E.g. in a Juypter cell :
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
This helps to control the severity level for which messages are displayed. The different levels are
Level | Level for Humans | Level Description
-------|------------------|------------------------------------
0 | DEBUG | [Default] Print all messages
1 | INFO | Filter out INFO messages
2 | WARNING | Filter out INFO & WARNING messages
3 | ERROR | Filter out all messages
Links
People who are interested of how the Linux kernel handles NUMA can find some information at kernel.org:
The Linux Kernel – NUMA Memory Policy