A data race happens, or could happen, when two threads access a
shared memory location without using suitable locks or other
synchronisation to ensure single-threaded access. Such missing
locking can cause obscure timing dependent bugs. Ensuring programs
are race-free is one of the central difficulties of threaded
programming.
Reliably detecting races is a difficult problem, and most
of Helgrind's internals are devoted to dealing with it.
We begin with a simple example.
About the simplest possible example of a race is as follows. In
this program, it is impossible to know what the value
of var
is at the end of the program.
Is it 2 ? Or 1 ?
#include <pthread.h>
int var = 0;
void* child_fn ( void* arg ) {
var++; /* Unprotected relative to parent */ /* this is line 6 */
return NULL;
}
int main ( void ) {
pthread_t child;
pthread_create(&child, NULL, child_fn, NULL);
var++; /* Unprotected relative to child */ /* this is line 13 */
pthread_join(child, NULL);
return 0;
}
The problem is there is nothing to
stop var
being updated simultaneously
by both threads. A correct program would
protect var
with a lock of type
pthread_mutex_t
, which is acquired
before each access and released afterwards. Helgrind's output for
this program is:
Thread #1 is the program's root thread
Thread #2 was created
at 0x511C08E: clone (in /lib64/libc-2.8.so)
by 0x4E333A4: do_clone (in /lib64/libpthread-2.8.so)
by 0x4E33A30: pthread_create@@GLIBC_2.2.5 (in /lib64/libpthread-2.8.so)
by 0x4C299D4: pthread_create@* (hg_intercepts.c:214)
by 0x400605: main (simple_race.c:12)
Possible data race during read of size 4 at 0x601038 by thread #1
Locks held: none
at 0x400606: main (simple_race.c:13)
This conflicts with a previous write of size 4 by thread #2
Locks held: none
at 0x4005DC: child_fn (simple_race.c:6)
by 0x4C29AFF: mythread_wrapper (hg_intercepts.c:194)
by 0x4E3403F: start_thread (in /lib64/libpthread-2.8.so)
by 0x511C0CC: clone (in /lib64/libc-2.8.so)
Location 0x601038 is 0 bytes inside global var "var"
declared at simple_race.c:3
This is quite a lot of detail for an apparently simple error.
The last clause is the main error message. It says there is a race as
a result of a read of size 4 (bytes), at 0x601038, which is the
address of var
, happening in
function main
at line 13 in the
program.
Two important parts of the message are:
-
Helgrind shows two stack traces for the error, not one. By
definition, a race involves two different threads accessing the
same location in such a way that the result depends on the relative
speeds of the two threads.
The first stack trace follows the text "Possible
data race during read of size 4 ...
" and the
second trace follows the text "This conflicts with
a previous write of size 4 ...
". Helgrind is
usually able to show both accesses involved in a race. At least
one of these will be a write (since two concurrent, unsynchronised
reads are harmless), and they will of course be from different
threads.
By examining your program at the two locations, you should be
able to get at least some idea of what the root cause of the
problem is. For each location, Helgrind shows the set of locks
held at the time of the access. This often makes it clear which
thread, if any, failed to take a required lock. In this example
neither thread holds a lock during the access.
-
For races which occur on global or stack variables, Helgrind
tries to identify the name and defining point of the variable.
Hence the text "Location 0x601038 is 0 bytes inside
global var "var" declared at simple_race.c:3
".
Showing names of stack and global variables carries no
run-time overhead once Helgrind has your program up and running.
However, it does require Helgrind to spend considerable extra time
and memory at program startup to read the relevant debug info.
Hence this facility is disabled by default. To enable it, you need
to give the --read-var-info=yes
option to
Helgrind.
The following section explains Helgrind's race detection
algorithm in more detail.
7.4.2.Helgrind's Race Detection Algorithm
Most programmers think about threaded programming in terms of
the basic functionality provided by the threading library (POSIX
Pthreads): thread creation, thread joining, locks, condition
variables, semaphores and barriers.
The effect of using these functions is to impose
constraints upon the order in which memory accesses can
happen. This implied ordering is generally known as the
"happens-before relation". Once you understand the happens-before
relation, it is easy to see how Helgrind finds races in your code.
Fortunately, the happens-before relation is itself easy to understand,
and is by itself a useful tool for reasoning about the behaviour of
parallel programs. We now introduce it using a simple example.
Consider first the following buggy program:
Parent thread: Child thread:
int var;
// create child thread
pthread_create(...)
var = 20; var = 10;
exit
// wait for child
pthread_join(...)
printf("%d\n", var);
The parent thread creates a child. Both then write different
values to some variable var
, and the
parent then waits for the child to exit.
What is the value of var
at the
end of the program, 10 or 20? We don't know. The program is
considered buggy (it has a race) because the final value
of var
depends on the relative rates
of progress of the parent and child threads. If the parent is fast
and the child is slow, then the child's assignment may happen later,
so the final value will be 10; and vice versa if the child is faster
than the parent.
The relative rates of progress of parent vs child is not something
the programmer can control, and will often change from run to run.
It depends on factors such as the load on the machine, what else is
running, the kernel's scheduling strategy, and many other factors.
The obvious fix is to use a lock to
protect var
. It is however
instructive to consider a somewhat more abstract solution, which is to
send a message from one thread to the other:
Parent thread: Child thread:
int var;
// create child thread
pthread_create(...)
var = 20;
// send message to child
// wait for message to arrive
var = 10;
exit
// wait for child
pthread_join(...)
printf("%d\n", var);
Now the program reliably prints "10", regardless of the speed of
the threads. Why? Because the child's assignment cannot happen until
after it receives the message. And the message is not sent until
after the parent's assignment is done.
The message transmission creates a "happens-before" dependency
between the two assignments: var = 20;
must now happen-before var = 10;
.
And so there is no longer a race
on var
.
Note that it's not significant that the parent sends a message
to the child. Sending a message from the child (after its assignment)
to the parent (before its assignment) would also fix the problem, causing
the program to reliably print "20".
Helgrind's algorithm is (conceptually) very simple. It monitors all
accesses to memory locations. If a location -- in this example,
var
,
is accessed by two different threads, Helgrind checks to see if the
two accesses are ordered by the happens-before relation. If so,
that's fine; if not, it reports a race.
It is important to understand that the happens-before relation
creates only a partial ordering, not a total ordering. An example of
a total ordering is comparison of numbers: for any two numbers
x
and
y
, either
x
is less than, equal to, or greater
than
y
. A partial ordering is like a
total ordering, but it can also express the concept that two elements
are neither equal, less or greater, but merely unordered with respect
to each other.
In the fixed example above, we say that
var = 20;
"happens-before"
var = 10;
. But in the original
version, they are unordered: we cannot say that either happens-before
the other.
What does it mean to say that two accesses from different
threads are ordered by the happens-before relation? It means that
there is some chain of inter-thread synchronisation operations which
cause those accesses to happen in a particular order, irrespective of
the actual rates of progress of the individual threads. This is a
required property for a reliable threaded program, which is why
Helgrind checks for it.
The happens-before relations created by standard threading
primitives are as follows:
When a mutex is unlocked by thread T1 and later (or
immediately) locked by thread T2, then the memory accesses in T1
prior to the unlock must happen-before those in T2 after it acquires
the lock.
The same idea applies to reader-writer locks,
although with some complication so as to allow correct handling of
reads vs writes.
When a condition variable (CV) is signalled on by
thread T1 and some other thread T2 is thereby released from a wait
on the same CV, then the memory accesses in T1 prior to the
signalling must happen-before those in T2 after it returns from the
wait. If no thread was waiting on the CV then there is no
effect.
If instead T1 broadcasts on a CV, then all of the
waiting threads, rather than just one of them, acquire a
happens-before dependency on the broadcasting thread at the point it
did the broadcast.
A thread T2 that continues after completing sem_wait
on a semaphore that thread T1 posts on, acquires a happens-before
dependence on the posting thread, a bit like dependencies caused
mutex unlock-lock pairs. However, since a semaphore can be posted
on many times, it is unspecified from which of the post calls the
wait call gets its happens-before dependency.
For a group of threads T1 .. Tn which arrive at a
barrier and then move on, each thread after the call has a
happens-after dependency from all threads before the
barrier.
A newly-created child thread acquires an initial
happens-after dependency on the point where its parent created it.
That is, all memory accesses performed by the parent prior to
creating the child are regarded as happening-before all the accesses
of the child.
Similarly, when an exiting thread is reaped via a
call to pthread_join
, once the call returns, the
reaping thread acquires a happens-after dependency relative to all memory
accesses made by the exiting thread.
In summary: Helgrind intercepts the above listed events, and builds a
directed acyclic graph represented the collective happens-before
dependencies. It also monitors all memory accesses.
If a location is accessed by two different threads, but Helgrind
cannot find any path through the happens-before graph from one access
to the other, then it reports a race.
There are a couple of caveats:
Helgrind doesn't check for a race in the case where
both accesses are reads. That would be silly, since concurrent
reads are harmless.
Two accesses are considered to be ordered by the
happens-before dependency even through arbitrarily long chains of
synchronisation events. For example, if T1 accesses some location
L, and then pthread_cond_signals
T2, which later
pthread_cond_signals
T3, which then accesses L, then
a suitable happens-before dependency exists between the first and second
accesses, even though it involves two different inter-thread
synchronisation events.
7.4.3.Interpreting Race Error Messages
Helgrind's race detection algorithm collects a lot of
information, and tries to present it in a helpful way when a race is
detected. Here's an example:
Thread #2 was created
at 0x511C08E: clone (in /lib64/libc-2.8.so)
by 0x4E333A4: do_clone (in /lib64/libpthread-2.8.so)
by 0x4E33A30: pthread_create@@GLIBC_2.2.5 (in /lib64/libpthread-2.8.so)
by 0x4C299D4: pthread_create@* (hg_intercepts.c:214)
by 0x4008F2: main (tc21_pthonce.c:86)
Thread #3 was created
at 0x511C08E: clone (in /lib64/libc-2.8.so)
by 0x4E333A4: do_clone (in /lib64/libpthread-2.8.so)
by 0x4E33A30: pthread_create@@GLIBC_2.2.5 (in /lib64/libpthread-2.8.so)
by 0x4C299D4: pthread_create@* (hg_intercepts.c:214)
by 0x4008F2: main (tc21_pthonce.c:86)
Possible data race during read of size 4 at 0x601070 by thread #3
Locks held: none
at 0x40087A: child (tc21_pthonce.c:74)
by 0x4C29AFF: mythread_wrapper (hg_intercepts.c:194)
by 0x4E3403F: start_thread (in /lib64/libpthread-2.8.so)
by 0x511C0CC: clone (in /lib64/libc-2.8.so)
This conflicts with a previous write of size 4 by thread #2
Locks held: none
at 0x400883: child (tc21_pthonce.c:74)
by 0x4C29AFF: mythread_wrapper (hg_intercepts.c:194)
by 0x4E3403F: start_thread (in /lib64/libpthread-2.8.so)
by 0x511C0CC: clone (in /lib64/libc-2.8.so)
Location 0x601070 is 0 bytes inside local var "unprotected2"
declared at tc21_pthonce.c:51, in frame #0 of thread 3
Helgrind first announces the creation points of any threads
referenced in the error message. This is so it can speak concisely
about threads without repeatedly printing their creation point call
stacks. Each thread is only ever announced once, the first time it
appears in any Helgrind error message.
The main error message begins at the text
"Possible data race during read
". At
the start is information you would expect to see -- address and size
of the racing access, whether a read or a write, and the call stack at
the point it was detected.
A second call stack is presented starting at the text
"This conflicts with a previous
write
". This shows a previous access which also
accessed the stated address, and which is believed to be racing
against the access in the first call stack. Note that this second
call stack is limited to a maximum of 8 entries to limit the
memory usage.
Finally, Helgrind may attempt to give a description of the
raced-on address in source level terms. In this example, it
identifies it as a local variable, shows its name, declaration point,
and in which frame (of the first call stack) it lives. Note that this
information is only shown when --read-var-info=yes
is specified on the command line. That's because reading the DWARF3
debug information in enough detail to capture variable type and
location information makes Helgrind much slower at startup, and also
requires considerable amounts of memory, for large programs.
Once you have your two call stacks, how do you find the root
cause of the race?
The first thing to do is examine the source locations referred
to by each call stack. They should both show an access to the same
location, or variable.
Now figure out how how that location should have been made
thread-safe:
Perhaps the location was intended to be protected by
a mutex? If so, you need to lock and unlock the mutex at both
access points, even if one of the accesses is reported to be a read.
Did you perhaps forget the locking at one or other of the accesses?
To help you do this, Helgrind shows the set of locks held by each
threads at the time they accessed the raced-on location.
-
Alternatively, perhaps you intended to use a some
other scheme to make it safe, such as signalling on a condition
variable. In all such cases, try to find a synchronisation event
(or a chain thereof) which separates the earlier-observed access (as
shown in the second call stack) from the later-observed access (as
shown in the first call stack). In other words, try to find
evidence that the earlier access "happens-before" the later access.
See the previous subsection for an explanation of the happens-before
relation.
The fact that Helgrind is reporting a race means it did not observe
any happens-before relation between the two accesses. If
Helgrind is working correctly, it should also be the case that you
also cannot find any such relation, even on detailed inspection
of the source code. Hopefully, though, your inspection of the code
will show where the missing synchronisation operation(s) should have
been.