numad

#13 numad freezes during multiple access (numa -w issued by libvirtd)
Opened by elcoyote. Modified

Open
Close Issue

Copy from: https://issues.redhat.com/browse/RHELPLAN-139824

Description of problem:
numad -w will occasionally freeze if invoked concurrently with the numad daemon running

Version-Release number of selected component (if applicable):
0.5-26.20150602git

How reproducible:
Always

Steps to Reproduce:
1.Execute numad -w 4:4096 from multiple concurrent sessions. A good command line is: while true; do numad -w 4:4096; numad -w 4:4096; numad -w 4:4096; done

2.Wait for some time. Perhaps restart the command sequence to create a suitable timing skew.
3.

Actual results:
You will see one of the sessions freezing after a while

Expected results:
The sequence should run smoothly

Additional info:

This problem appeared during concurrent migration of several (n>3) VM:s in a KVM cluster. The VM:s had a numatune section looking like this:

8


libvirt tried to use numad -w 8:4096 to figure out the placement and the freeze of numad would also freeze libvirtd, which led to pcs rebooting the node.

Looking at the source of numad it seems like the potential problem is that each instance of numad will run init_msg_queue() at invocation. As this also does a flush_msg_queue() the queue will be emptied at every invocation. If this happens in the middle of the send/rcv-sequence of another invocation the other process will never get an answer and hang.

I've successfully tested a small patch that resolves the freeze for me on RHEL9.4 EUS:

[code]$ git diff -c
diff --git a/numad.c b/numad.c
index 25363a2..8862b3e 100644
--- a/numad.c
+++ b/numad.c
@@ -2508,7 +2508,9 @@ int main(int argc, char argv[]) {
}
}
open_log_file();
- init_msg_queue();
+ if (!w_flag) {
+ init_msg_queue();
+ }
num_cpus = get_num_cpus();
page_size_in_bytes = sysconf(_SC_PAGESIZE);
huge_page_size_in_bytes = get_huge_page_size_in_bytes();
@@ -2578,8 +2580,6 @@ int main(int argc, char argv[]) {
if (w_flag) {
// Get pre-placement NUMA advice without starting daemon
update_nodes();
- sleep(2);
- update_nodes();
numad_log(LOG_NOTICE, "Getting NUMA pre-placement advice for %d CPUs and %d MBs\n", requested_cpus, requested_mbs);
id_list_p node_list_p = pick_numa_nodes(-1, requested_cpus, requested_mbs, 0);
char buf[BUF_SIZE];
[/code]
Edited by elcoyote
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.