LTP Test Writing Guidelines
===========================
This document describes LTP guidelines and LTP test interface and is intended
for anybody who want to write or modify a LTP testcase. It's not a definitive
guide and it's not, by any means, a substitute for common sense.
1. General Rules
----------------
1.1 Simplicity
~~~~~~~~~~~~~~
For all it's worth keep the testcases simple or better as simple as possible.
The kernel and libc are tricky beasts and the complexity imposed by their
interfaces is quite high. Concentrate on the interface you want to test and
follow the UNIX philosophy. It's a good idea to make the test as
self-contained as possible too (it should not depend on tools or libraries
that are not widely available).
Do not reinvent the wheel!
* Use LTP standard interface
* Do not add custom PASS/FAIL reporting functions
* Do not write Makefiles from scratch,
use LTP build system instead, etc.
* ...
1.2 Code duplication
~~~~~~~~~~~~~~~~~~~~
Copy & paste is a good servant but very poor master. If you are about to copy a
large part of the code from one testcase to another, think what would happen if
you find bug in the code that has been copied all around the tree. What about
moving it to a library instead?
The same goes for short but complicated parts, whenever you are about to copy &
paste a syscall wrapper that packs arguments accordingly to machine
architecture or similarly complicated code, put it into a header instead.
1.3 Coding style
~~~~~~~~~~~~~~~~
1.3.1 C coding style
^^^^^^^^^^^^^^^^^^^^
LTP adopted Linux kernel coding style. If you aren't familiar with its rules
locate 'linux/Documentation/CodingStyle' in the kernel sources and read it,
it's a well written introduction.
There is also a checkpatch (see 'linux/scripts/checkpatch.pl') script that can
be used to check your patches before the submission.
NOTE: If checkpatch does not report any problems, the code still may be wrong
as the tool only looks for common mistakes.
1.3.2 Shell coding style
^^^^^^^^^^^^^^^^^^^^^^^^
When writing testcases in shell write in *portable shell* only, it's a good
idea to try to run the test using alternative shell (alternative to bash, for
example dash) too.
*Portable shell* means Shell Command Language as defined by POSIX with a
exception of few widely used extensions, namely 'local' keyword used inside of
functions and '-o' and '-a' test parameters (that are marked as obsolete in
POSIX).
You can either try to run the testcases on Debian which has '/bin/sh' pointing
to 'dash' by default or install 'dash' on your favorite distribution and use
it to run the tests. If your distribution lacks 'dash' package you can always
compile it from http://gondor.apana.org.au/~herbert/dash/files/[source].
Debian also has nice devscript
https://salsa.debian.org/debian/devscripts/raw/master/scripts/checkbashisms.pl[checkbashism.pl]
that can be used to check for non-portable shell code.
Here are some common sense style rules for shell
* Keep lines under 80 chars
* Use tabs for indentation
* Keep things simple, avoid unnecessary subshells
* Don't do confusing things (i.e. don't name your functions like common shell
commands, etc.)
* Quote variables
* Be consistent
1.4 Commenting code
~~~~~~~~~~~~~~~~~~~
Comments can sometimes save you day but they can easily do more harm than
good. There has been several cases where comments and actual implementation
were drifting slowly apart which yielded into API misuses and hard to find
bugs. Remember there is only one thing worse than no documentation, wrong
documentation.
Generally everybody should write code that is obvious (which unfortunately
isn't always possible). If there is a code that needs to be commented keep it
short and to the point. Never ever comment the obvious.
In case of LTP testcases it's customary to add a paragraph with highlevel test
description somewhere at the beginning of the file (usually right under the GPL
header). This helps other people to understand the overall goal of the test
before they dive into the technical details.
1.5 Backwards compatibility
~~~~~~~~~~~~~~~~~~~~~~~~~~~
LTP test should be as backward compatible as possible. Think of an enterprise
distributions with long term support (more than five years since the initial
release) or of an embedded platform that needs to use several years old
toolchain supplied by the manufacturer.
Therefore LTP test for more current features should be able to cope with older
systems. It should at least compile fine and if it's not appropriate for the
configuration it should return 'TCONF' (see test interface description below).
There are several types of checks we use:
The *configure script* is usually used to detect availability of a function
declarations in system headers. It's used to disable tests at compile time.
We also have runtime kernel version detection that can be used to disable
tests at runtime.
Checking the *errno* value is another type of runtime check. Most of the
syscalls returns either 'EINVAL' or 'ENOSYS' when syscall was not implemented
or was disabled upon kernel compilation.
Sometimes it also makes sense to define a few macros instead of creating
configure test. One example are Linux specific POSIX clock ids in
'include/lapi/posix_clocks.h'.
1.6 Dealing with messed up legacy code
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LTP contains a lot of old and messy code and we are cleaning it up as fast as
we can but despite the efforts there is still a lot. If you start modifying
old or a messed up testcase and your changes are more complicated than simple
typo fixes you should do a cleanup first (in a separate patch). It's easier to
review the changes if you separate the formatting fixes from the changes that
affects the test behavior.
The same goes for moving files. If you need a rename or move file do it in a
separate patch.
1.7 License
~~~~~~~~~~~
Code contributed to LTP should be licensed under GPLv2+ (GNU GPL version 2 or
any later version).
2. Writing a testcase
---------------------
2.1 LTP Structure
~~~~~~~~~~~~~~~~~
The structure of LTP is quite simple. Each test is a binary written either in
portable shell or C. The test gets a configuration via environment variables
and/or command line parameters, it prints additional information into the
stdout and reports overall success/failure via the exit value.
Tests are generally placed under the 'testcases/' directory. Everything that
is a syscall or (slightly confusingly) libc syscall wrapper goes under
'testcases/kernel/syscalls/'. Then there is 'testcases/open_posix_testsuite'
which is a well maintained fork of the upstream project that has been dead
since 2005 and also a number of directories with tests for more specific
features.
2.1.1 Runtest Files
^^^^^^^^^^^^^^^^^^^
The list of tests to be executed is stored in runtest files under the
'runtest/' directory. The default set of runtest files to be executed is
stored in 'scenario_groups/default'. When you add a test you should add
corresponding entries into some runtest file(s) as well.
For syscall tests (these placed under 'testcases/kernel/syscalls/') use
'runtest/syscalls' file, for kernel related tests for memory management we
have 'runtest/mm', etc.
IMPORTANT: The runtest files should have one entry per a test. Creating a
wrapper that runs all your tests and adding it as a single test
into runtest file is strongly discouraged.
2.1.2 Datafiles
^^^^^^^^^^^^^^^
If your test needs datafiles to work, these should be put into a subdirectory
named 'datafiles' and installed into the 'testcases/data/$TCID' directory (to
do that you have to add 'INSTALL_DIR := testcases/data/TCID' into the
'datafiles/Makefile').
You can obtain path to datafiles via $TST_DATAROOT provided by test.sh
'$TST_DATAROOT/...'
or via C function 'tst_dataroot()' provided by libltp:
[source,c]
-------------------------------------------------------------------------------
const char *dataroot = tst_dataroot();
-------------------------------------------------------------------------------
Datafiles can also be accessed as '$LTPROOT/testcases/data/$TCID/...',
but '$TST_DATAROOT' and 'tst_dataroot()' are preferred as these can be used
when running testcases directly in git tree as well as from install
location.
The path is constructed according to these rules:
1. if '$LTPROOT' is set, return '$LTPROOT/testcases/data/$TCID'
2. else if 'tst_tmpdir()' was called return '$STARTWD/datafiles'
(where '$STARTWD' is initial working directory as recorded by 'tst_tmpdir()')
3. else return '$CWD/datafiles'
See 'testcases/commands/file/' for example.
2.1.3 Subexecutables
^^^^^^^^^^^^^^^^^^^^
If you test needs to execute a binary, place it in the same directory as the
testcase and name the file starting with '${test_binary_name}_'. Once the
test is executed by the framework, the path to the directory with all LTP
binaries is added to the '$PATH' and you can execute it just by its name.
TIP: If you need to execute such test from the LTP tree, you can add path to
current directory to '$PATH' manually with: 'PATH="$PATH:$PWD" ./foo01'.
2.2 Writing a test in C
~~~~~~~~~~~~~~~~~~~~~~~
2.2.1 Basic test structure
^^^^^^^^^^^^^^^^^^^^^^^^^^
Let's start with an example, following code is a simple test for a 'getenv()'.
[source,c]
-------------------------------------------------------------------------------
/*
* This is test for basic functionality of getenv().
*
* - create an env variable and verify that getenv() can get get it
* - call getenv() with nonexisting variable name, check that it returns NULL
*/
#include "tst_test.h"
#define ENV1 "LTP_TEST_ENV"
#define ENV2 "LTP_TEST_THIS_DOES_NOT_EXIST"
#define ENV_VAL "val"
static void setup(void)
{
if (setenv(ENV1, ENV_VAL, 1))
tst_brk(TBROK | TERRNO, "setenv() failed");
}
static void test(void)
{
char *ret;
ret = getenv(ENV1);
if (!ret) {
tst_res(TFAIL, "getenv(" ENV1 ") = NULL");
goto next;
}
if (!strcmp(ret, ENV_VAL)) {
tst_res(TPASS, "getenv(" ENV1 ") = '"ENV_VAL "'");
} else {
tst_res(TFAIL, "getenv(" ENV1 ") = '%s', expected '"
ENV_VAL "'", ret);
}
next:
ret = getenv(ENV2);
if (ret)
tst_res(TFAIL, "getenv(" ENV2 ") = '%s'", ret);
else
tst_res(TPASS, "getenv(" ENV2 ") = NULL");
}
static struct tst_test test = {
.test_all = test,
.setup = setup,
};
-------------------------------------------------------------------------------
Each test includes the 'tst_test.h' header and must define the 'struct
tst_test test' structure.
The overall test initialization is done in the 'setup()' function.
The overall cleanup is done in a 'cleanup()' function. Here 'cleanup()' is
omitted as the test does not have anything to clean up. If cleanup is set in
the test structure it's called on test exit just before the test library
cleanup. That especially means that cleanup can be called at any point in a
test execution. For example even when a test setup step has failed, therefore
the 'cleanup()' function must be able to cope with unfinished initialization,
and so on.
The test itself is done in the 'test()' function. The test function must work
fine if called in a loop.
There are two types of a test function pointers in the test structure. The
first one is a '.test_all' pointer that is used when test is implemented as a
single function. Then there is a '.test' function along with the number of
tests '.tcnt' that allows for more detailed result reporting. If the '.test'
pointer is set the function is called '.tcnt' times with an integer parameter
in range of [0, '.tcnt' - 1].
IMPORTANT: Only one of '.test' and '.test_all' can be set at a time.
Each test has a default timeout set to 300s. The default timeout can be
overriden by setting '.timeout' in the test structure or by calling
'tst_set_timeout()' in the test 'setup()'. There are a few testcases whose run
time may vary arbitrarily, for these timeout can be disabled by setting it to
-1.
Test can find out how much time (in seconds) is remaining to timeout,
by calling 'tst_timeout_remaining()'.
A word about the cleanup() callback
+++++++++++++++++++++++++++++++++++
There are a few rules that needs to be followed in order to write correct
cleanup() callback.
1. Free only resources that were initialized. Keep in mind that callback can
be executed at any point in the test run.
2. Make sure to free resources in the reverse order they were
initialized. (Some of the steps may not depend on others and everything
will work if there were swapped but let's keep it in order.)
The first rule may seem complicated at first however, on the contrary, it's
quite easy. All you have to do is to keep track of what was already
initialized. For example file descriptors needs to be closed only if they were
assigned a valid file descriptor. For most of the things you need to create
extra flag that is set right after successful initialization though. Consider,
for example, test setup below.
[source,c]
-------------------------------------------------------------------------------
static int fd0, fd1, mount_flag;
#define MNTPOINT "mntpoint"
#define FILE1 "mntpoint/file1"
#define FILE2 "mntpoint/file2"
static void setup(void)
{
SAFE_MKDIR(MNTPOINT, 0777);
SAFE_MKFS(tst_device->dev, tst_device->fs_type, NULL, NULL);
SAFE_MOUNT(tst_device->dev, MNTPOINT, tst_device->fs_type, 0, 0);
mount_flag = 1;
fd0 = SAFE_OPEN(cleanup, FILE1, O_CREAT | O_RDWR, 0666);
fd1 = SAFE_OPEN(cleanup, FILE2, O_CREAT | O_RDWR, 0666);
}
-------------------------------------------------------------------------------
In this case the 'cleanup()' function may be invoked when any of the 'SAFE_*'
macros has failed and therefore must be able to work with unfinished
initialization as well. Since global variables are initialized to zero we can
just check that fd > 0 before we attempt to close it. The mount function
requires extra flag to be set after device was successfully mounted.
[source,c]
-------------------------------------------------------------------------------
static void cleanup(void)
{
if (fd1 > 0)
SAFE_CLOSE(fd1);
if (fd0 > 0)
SAFE_CLOSE(fd0);
if (mount_flag && tst_umouont(MNTPOINT))
tst_res(TWARN | TERRNO, "umount(%s)", MNTPOINT);
}
-------------------------------------------------------------------------------
IMPORTANT: 'SAFE_MACROS()' used in cleanup *do not* exit the test. Failure
only produces a warning and the 'cleanup()' carries on. This is
intentional as we want to execute as much 'cleanup()' as possible.
WARNING: Calling tst_brk() in test 'cleanup()' does not exit the test as well
and 'TBROK' is converted to 'TWARN'.
NOTE: Creation and removal of the test temporary directory is handled in
the test library and the directory is removed recursively. Therefore
we do not have to remove files and directories in the test cleanup.
2.2.2 Basic test interface
^^^^^^^^^^^^^^^^^^^^^^^^^^
[source,c]
-------------------------------------------------------------------------------
void tst_res(int ttype, char *arg_fmt, ...);
-------------------------------------------------------------------------------
Printf-like function to report test result, it's mostly used with ttype:
|==============================
| 'TPASS' | Test has passed.
| 'TFAIL' | Test has failed.
| 'TINFO' | General message.
|==============================
The 'ttype' can be combined bitwise with 'TERRNO' or 'TTERRNO' to print
'errno', 'TST_ERR' respectively.
[source,c]
-------------------------------------------------------------------------------
void tst_brk(int ttype, char *arg_fmt, ...);
-------------------------------------------------------------------------------
Printf-like function to report error and exit the test, it can be used with ttype:
|============================================================
| 'TBROK' | Something has failed in test preparation phase.
| 'TCONF' | Test is not appropriate for current configuration
(syscall not implemented, unsupported arch, ...)
|============================================================
The 'ttype' can be combined bitwise with 'TERRNO' or 'TTERRNO' to print
'errno', 'TST_ERR' respectively.
[source,c]
-------------------------------------------------------------------------------
const char *tst_strsig(int sig);
-------------------------------------------------------------------------------
Return the given signal number's corresponding string.
[source,c]
-------------------------------------------------------------------------------
const char *tst_strerrno(int err);
-------------------------------------------------------------------------------
Return the given errno number's corresponding string. Using this function to
translate 'errno' values to strings is preferred. You should not use the
'strerror()' function in the testcases.
[source,c]
-------------------------------------------------------------------------------
const char *tst_strstatus(int status);
-------------------------------------------------------------------------------
Returns string describing the status as returned by 'wait()'.
WARNING: This function is not thread safe.
[source,c]
-------------------------------------------------------------------------------
void tst_set_timeout(unsigned int timeout);
-------------------------------------------------------------------------------
Allows for setting timeout per test iteration dymanically in the test setup(),
the timeout is specified in seconds. There are a few testcases whose runtime
can vary arbitrarily, these can disable timeouts by setting it to -1.
[source,c]
-------------------------------------------------------------------------------
void tst_flush(void);
-------------------------------------------------------------------------------
Flush output streams, handling errors appropriately.
This function is rarely needed when you have to flush the output streams
before calling 'fork()' or 'clone()'. Note that the 'SAFE_FORK()' calls this
function automatically. See 3.4 FILE buffers and fork() for explanation why is
this needed.
2.2.3 Test temporary directory
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If '.needs_tmpdir' is set to '1' in the 'struct tst_test' unique test
temporary is created and it's set as the test working directory. Tests *MUST
NOT* create temporary files outside that directory.
IMPORTANT: Close all file descriptors (that point to files in test temporary
directory, even the unlinked ones) either in the 'test()' function
or in the test 'cleanup()' otherwise the test may break temporary
directory removal on NFS (look for "NFS silly rename").
2.2.4 Safe macros
^^^^^^^^^^^^^^^^^
Safe macros aim to simplify error checking in test preparation. Instead of
calling system API functions, checking for their return value and aborting the
test if the operation has failed, you just use corresponding safe macro.
Use them whenever it's possible.
Instead of writing:
[source,c]
-------------------------------------------------------------------------------
fd = open("/dev/null", O_RDONLY);
if (fd < 0)
tst_brk(TBROK | TERRNO, "opening /dev/null failed");
-------------------------------------------------------------------------------
You write just:
[source,c]
-------------------------------------------------------------------------------
fd = SAFE_OPEN("/dev/null", O_RDONLY);
-------------------------------------------------------------------------------
IMPORTANT: The SAFE_CLOSE() function also sets the passed file descriptor to -1
after it's successfully closed.
They can also simplify reading and writing of sysfs files, you can, for
example, do:
[source,c]
-------------------------------------------------------------------------------
SAFE_FILE_SCANF("/proc/sys/kernel/pid_max", "%lu", &pid_max);
-------------------------------------------------------------------------------
See 'include/tst_safe_macros.h', 'include/tst_safe_stdio.h' and
'include/tst_safe_file_ops.h' and 'include/tst_safe_net.h' for a complete list.
2.2.5 Test specific command line options
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[source,c]
-------------------------------------------------------------------------------
struct tst_option {
char *optstr;
char **arg;
char *help;
};
-------------------------------------------------------------------------------
Test specific command line parameters can be passed with the 'NULL'-terminated
array of 'struct tst_option'. The 'optstr' is the command line option i.e. "o"
or "o:" if option has a parameter. Only short options are supported. The 'arg'
is where 'optarg' is stored upon match. If option has no parameter it's set to
non-'NULL' value if option was present. The 'help' is a short help string.
NOTE: The test parameters must not collide with common test parameters defined
in the library the currently used ones are +-i+, +-I+, +-C+, and +-h+.
[source,c]
-------------------------------------------------------------------------------
int tst_parse_int(const char *str, int *val, int min, int max);
int tst_parse_float(const char *str, float *val, float min, float max);
-------------------------------------------------------------------------------
Helpers for parsing the the strings returned in the 'struct tst_option'.
Both return zero on success and 'errno', mostly 'EINVAL' or 'ERANGE', on
failure.
Both functions are no-op if 'str' is 'NULL'.
The valid range for result includes both 'min' and 'max'.
.Example Usage
[source,c]
-------------------------------------------------------------------------------
#include <limits.h>
#include "tst_test.h"
static char *str_threads;
static int threads = 10;
static struct tst_option options[] = {
{"t:", &str_threads, "Number of threads (default 10)"},
...
{NULL, NULL, NULL}
};
static void setup(void)
{
if (tst_parse_int(str_threads, &threads, 1, INT_MAX))
tst_brk(TBROK, "Invalid number of threads '%s'", str_threads);
...
}
static void test_threads(void)
{
...
for (i = 0; i < threads; i++) {
...
}
...
}
static struct tst_test test = {
...
.options = options,
...
};
-------------------------------------------------------------------------------
2.2.6 Runtime kernel version detection
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Testcases for newly added kernel functionality require kernel newer than a
certain version to run. All you need to skip a test on older kernels is to
set the '.min_kver' string in the 'struct tst_test' to a minimal required
kernel version, e.g. '.min_kver = "2.6.30"'.
For more complicated operations such as skipping a test for a certain range
of kernel versions, following functions could be used:
[source,c]
-------------------------------------------------------------------------------
int tst_kvercmp(int r1, int r2, int r3);
struct tst_kern_exv {
char *dist_name;
char *extra_ver;
};
int tst_kvercmp2(int r1, int r2, int r3, struct tst_kern_exv *vers);
-------------------------------------------------------------------------------
These two functions are intended for runtime kernel version detection. They
parse the output from 'uname()' and compare it to the passed values.
The return value is similar to the 'strcmp()' function, i.e. zero means equal,
negative value means that the kernel is older than than the expected value and
positive means that it's newer.
The second function 'tst_kvercmp2()' allows for specifying per-vendor table of
kernel versions as vendors typically backport fixes to their kernels and the
test may be relevant even if the kernel version does not suggests so. See
'testcases/kernel/syscalls/inotify/inotify04.c' for example usage.
WARNING: The shell 'tst_kvercmp' maps the result into unsigned integer - the
process exit value.
2.2.7 Fork()-ing
^^^^^^^^^^^^^^^^
Be wary that if the test forks and there were messages printed by the
'tst_*()' interfaces, the data may still be in libc/kernel buffers and these
*ARE NOT* flushed automatically.
This happens when 'stdout' gets redirected to a file. In this case, the
'stdout' is not line buffered, but block buffered. Hence after a fork content
of the buffers will be printed by the parent and each of the children.
To avoid that you should use 'SAFE_FORK()'.
IMPORTANT: You have to set the '.forks_child' flag in the test structure
if your testcase forks.
2.2.8 Doing the test in the child process
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Results reported by 'tst_res()' are propagated to the parent test process via
block of shared memory.
Calling 'tst_brk()' causes child process to exit with non-zero exit value.
Which means that it's safe to use 'SAFE_*()' macros in the child processes as
well.
Children that outlive the 'test()' function execution are waited for in the
test library. Unclean child exit (killed by signal, non-zero exit value, etc.)
will cause the main test process to exit with 'tst_brk()', which especially
means that 'TBROK' propagated from a child process will cause the whole test
to exit with 'TBROK'.
If a test needs a child that segfaults or does anything else that cause it to
exit uncleanly all you need to do is to wait for such children from the
'test()' function so that it's reaped before the main test exits the 'test()'
function.
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
void tst_reap_children(void);
-------------------------------------------------------------------------------
The 'tst_reap_children()' function makes the process wait for all of its
children and exits with 'tst_brk(TBROK, ...)' if any of them returned
a non zero exit code.
.Using tst_res() from binaries started by exec()
[source,c]
-------------------------------------------------------------------------------
/* test.c */
#define _GNU_SOURCE
#include <unistd.h>
#include "tst_test.h"
static void do_test(void)
{
char *const argv[] = {"test_exec_child", NULL};
char path[4096];
if (tst_get_path("test_exec_child", path, sizeof(path)))
tst_brk(TCONF, "Couldn't find test_exec_child in $PATH");
execve(path, argv, environ);
tst_res(TBROK | TERRNO, "EXEC!");
}
static struct tst_test test = {
.test_all = do_test,
.child_needs_reinit = 1,
};
/* test_exec_child.c */
#define TST_NO_DEFAULT_MAIN
#include "tst_test.h"
int main(void)
{
tst_reinit();
tst_res(TPASS, "Child passed!");
return 0;
}
-------------------------------------------------------------------------------
The 'tst_res()' function can be also used from binaries started by 'exec()',
the parent test process has to set the '.child_needs_reinit' flag so that the
library prepares for it and has to make sure the 'LTP_IPC_PATH' environment
vairiable is passed down, then the very fist thing the program has to call in
'main()' is 'tst_reinit()' that sets up the IPC.
2.2.9 Fork() and Parent-child synchronization
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
As LTP tests are written for Linux, most of the tests involve fork()-ing and
parent-child process synchronization. LTP includes a checkpoint library that
provides wait/wake futex based functions.
In order to use checkpoints the '.needs_checkpoints' flag in the 'struct
tst_test' must be set to '1', this causes the test library to initialize
checkpoints before the 'test()' function is called.
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
TST_CHECKPOINT_WAIT(id)
TST_CHECKPOINT_WAIT2(id, msec_timeout)
TST_CHECKPOINT_WAKE(id)
TST_CHECKPOINT_WAKE2(id, nr_wake)
TST_CHECKPOINT_WAKE_AND_WAIT(id)
-------------------------------------------------------------------------------
The checkpoint interface provides pair of wake and wait functions. The 'id' is
unsigned integer which specifies checkpoint to wake/wait for. As a matter of
fact it's an index to an array stored in a shared memory, so it starts on
'0' and there should be enough room for at least of hundred of them.
The 'TST_CHECKPOINT_WAIT()' and 'TST_CHECKPOINT_WAIT2()' suspends process
execution until it's woken up or until timeout is reached.
The 'TST_CHECKPOINT_WAKE()' wakes one process waiting on the checkpoint.
If no process is waiting the function retries until it success or until
timeout is reached.
If timeout has been reached process exits with appropriate error message (uses
'tst_brk()').
The 'TST_CHECKPOINT_WAKE2()' does the same as 'TST_CHECKPOINT_WAKE()' but can
be used to wake precisely 'nr_wake' processes.
The 'TST_CHECKPOINT_WAKE_AND_WAIT()' is a shorthand for doing wake and then
immediately waiting on the same checkpoint.
Child processes created via 'SAFE_FORK()' are ready to use the checkpoint
synchronization functions, as they inherited the mapped page automatically.
Child processes started via 'exec()', or any other processes not forked from
the test process must initialize the checkpoint by calling 'tst_reinit()'.
For the details of the interface, look into the 'include/tst_checkpoint.h'.
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
/*
* Waits for process state change.
*
* The state is one of the following:
*
* R - process is running
* S - process is sleeping
* D - process sleeping uninterruptibly
* Z - zombie process
* T - process is traced
*/
TST_PROCESS_STATE_WAIT(pid, state)
-------------------------------------------------------------------------------
The 'TST_PROCESS_STATE_WAIT()' waits until process 'pid' is in requested
'state'. The call polls +/proc/pid/stat+ to get this information.
It's mostly used with state 'S' which means that process is sleeping in kernel
for example in 'pause()' or any other blocking syscall.
2.2.10 Signal handlers
^^^^^^^^^^^^^^^^^^^^^^
If you need to use signal handlers, keep the code short and simple. Don't
forget that the signal handler is called asynchronously and can interrupt the
code execution at any place.
This means that problems arise when global state is changed both from the test
code and signal handler, which will occasionally lead to:
* Data corruption (data gets into inconsistent state), this may happen, for
example, for any operations on 'FILE' objects.
* Deadlock, this happens, for example, if you call 'malloc(2)', 'free(2)',
etc. from both the test code and the signal handler at the same time since
'malloc' has global lock for it's internal data structures. (Be wary that
'malloc(2)' is used by the libc functions internally too.)
* Any other unreproducible and unexpected behavior.
Quite common mistake is to call 'exit(3)' from a signal handler. Note that this
function is not signal-async-safe as it flushes buffers, etc. If you need to
exit a test immediately from a signal handler use '_exit(2)' instead.
TIP: See 'man 7 signal' for the list of signal-async-safe functions.
If a signal handler sets a variable, its declaration must be 'volatile',
otherwise compiler may misoptimize the code. This is because the variable may
not be changed in the compiler code flow analysis. There is 'sig_atomic_t'
type defined in C99 but this one *DOES NOT* imply 'volatile' (it's just a
'typedef' to 'int'). So the correct type for a flag that is changed from a
signal handler is either 'volatile int' or 'volatile sig_atomic_t'.
2.2.11 Kernel Modules
^^^^^^^^^^^^^^^^^^^^^
There are certain cases where the test needs a kernel part and userspace part,
happily, LTP can build a kernel module and then insert it to the kernel on test
start for you. See 'testcases/kernel/device-drivers/block' for details.
2.2.11 Useful macros
^^^^^^^^^^^^^^^^^^^^^
[source,c]
-------------------------------------------------------------------------------
ARRAY_SIZE(arr)
-------------------------------------------------------------------------------
Returns the size of statically defined array, i.e.
'(sizeof(arr) / sizeof(*arr))'
[source,c]
-------------------------------------------------------------------------------
LTP_ALIGN(x, a)
-------------------------------------------------------------------------------
Aligns the x to be next multiple of a. The a must be power of 2.
2.2.12 Filesystem type detection
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Some tests are known to fail on certain filesystems (you cannot swap on TMPFS,
there are unimplemented 'fcntl()' etc.).
If your test needs to be skipped on certain filesystems, use the interface
below:
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
/*
* Unsupported only on NFS.
*/
if (tst_fs_type(".") == TST_NFS_MAGIC)
tst_brk(TCONF, "Test not supported on NFS filesystem");
/*
* Unsupported on NFS, TMPFS and RAMFS
*/
long type;
switch ((type = tst_fs_type("."))) {
case TST_NFS_MAGIC:
case TST_TMPFS_MAGIC:
case TST_RAMFS_MAGIC:
tst_brk(TCONF, "Test not supported on %s filesystem",
tst_fs_type_name(type));
break;
}
-------------------------------------------------------------------------------
2.2.13 Thread-safety in the LTP library
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
It is safe to use library 'tst_res()' function in multi-threaded tests.
Only the main thread must return from the 'test()' function to the test
library and that must be done only after all threads that may call any library
function has been terminated. That especially means that threads that may call
'tst_brk()' must terminate before the execution of the 'test()' function
returns to the library. This is usually done by the main thread joining all
worker threads at the end of the 'test()' function. Note that the main thread
will never get to the library code in a case that 'tst_brk()' was called from
one of the threads since it will sleep at least in 'pthread_join()' on the
thread that called the 'tst_brk()' till 'exit()' is called by 'tst_brk()'.
The test-supplied cleanup function runs *concurrently* to the rest of the
threads in a case that cleanup was entered from 'tst_brk()'. Subsequent
threads entering 'tst_brk()' must be suspended or terminated at the start of
the the user supplied cleanup function. It may be necessary to stop or exit
the rest of the threads before the test cleans up as well. For example threads
that create new files should be stopped before temporary directory is be
removed.
Following code example shows thread safe cleanup function example using atomic
increment as a guard. The library calls its cleanup after the execution returns
from the user supplied cleanup and expects that only one thread returns from
the user supplied cleanup to the test library.
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
static void cleanup(void)
{
static int flag;
if (tst_atomic_inc(&flag) != 1)
pthread_exit(NULL);
/* if needed stop the rest of the threads here */
...
/* then do cleanup work */
...
/* only one thread returns to the library */
}
-------------------------------------------------------------------------------
2.2.14 Testing with a block device
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Some tests needs a block device (inotify tests, syscall 'EROFS' failures,
etc.). LTP library contains a code to prepare a testing device.
If '.needs_device' flag in the 'struct tst_test' is set the the 'tst_device'
structure is initialized with a path to a test device and default filesystem
to be used.
You can also request minimal device size in megabytes by setting
'.dev_min_size' the device is guaranteed to have at least the requested size
then.
If '.format_device' flag is set the device is formatted with a filesystem as
well. You can use '.dev_fs_type' to override the default filesystem type if
needed and pass additional options to mkfs via '.dev_fs_opts' and
'.dev_extra_opts' pointers. Note that '.format_device' implies '.needs_device'
there is no need to set both.
If '.mount_device' is set, the device is mounted at '.mntpoint' which is used
to pass a directory name that will be created and used as mount destination.
You can pass additional flags and data to the mount command via '.mnt_flags'
and '.mnt_data' pointers. Note that '.mount_device' implies '.needs_device'
and '.format_device' so there is no need to set the later two.
If '.needs_rofs' is set, read-only filesystem is mounted at '.mntpoint' this
one is supposed to be used for 'EROFS' tests.
If '.all_filesystems' is set the test function is executed for all supported
filesystems. Supported filesystems are detected based on existence of the
'mkfs.$fs' helper and on kernel support to mount it. For each supported
filesystem the 'tst_device.fs_type' is set to the currently tested fs type, if
'.format_device' is set the device is formatted as well, if '.mount_device' is
set it's mounted at '.mntpoint'. Also the test timeout is reset for each
execution of the test fuction. This flag is expected to be used for filesystem
related syscalls that are at least partly implemented in the filesystem
specific code e.g. fallocate().
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
struct tst_device {
const char *dev;
const char *fs_type;
};
extern struct tst_device *tst_device;
int tst_umount(const char *path);
-------------------------------------------------------------------------------
In case that 'LTP_DEV' is passed to the test in an environment, the library
checks that the file exists and that it's a block device, if
'.device_min_size' is set the device size is checked as well. If 'LTP_DEV'
wasn't set or if size requirements were not met a temporary file is created
and attached to a free loop device.
If there is no usable device and loop device couldn't be initialized the test
exits with 'TCONF'.
The 'tst_umount()' function works exactly as 'umount(2)' but retries several
times on 'EBUSY'. This is because various desktop daemons (gvfsd-trash is known
for that) may be stupid enough to probe all newly mounted filesystem which
results in 'umount(2)' failing with 'EBUSY'.
IMPORTANT: All testcases should use 'tst_umount()' instead of 'umount(2)' to
umount filesystems.
2.2.15 Formatting a device with a filesystem
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
static void setup(void)
{
...
SAFE_MKFS(tst_device->dev, tst_device->fs_type, NULL, NULL);
...
}
-------------------------------------------------------------------------------
This function takes a path to a device, filesystem type and an array of extra
options passed to mkfs.
The fs options 'fs_opts' should either be 'NULL' if there are none, or a
'NULL' terminated array of strings such as:
+const char *const opts[] = {"-b", "1024", NULL}+.
The extra options 'extra_opts' should either be 'NULL' if there are none, or a
'NULL' terminated array of strings such as +{"102400", NULL}+; 'extra_opts'
will be passed after device name. e.g: +mkfs -t ext4 -b 1024 /dev/sda1 102400+
in this case.
2.2.16 Verifying a filesystem's free space
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Some tests have size requirements for the filesystem's free space. If these
requirements are not satisfied, the tests should be skipped.
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
int tst_fs_has_free(const char *path, unsigned int size, unsigned int mult);
-------------------------------------------------------------------------------
The 'tst_fs_has_free()' function returns 1 if there is enough space and 0 if
there is not.
The 'path' is the pathname of any directory/file within a filesystem.
The 'mult' is a multiplier, one of 'TST_BYTES', 'TST_KB', 'TST_MB' or 'TST_GB'.
The required free space is calculated by 'size * mult', e.g.
'tst_fs_has_free("/tmp/testfile", 64, TST_MB)' will return 1 if the
filesystem, which '"/tmp/testfile"' is in, has 64MB free space at least, and 0
if not.
2.2.17 Files, directories and fs limits
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Some tests need to know the maximum count of links to a regular file or
directory, such as 'rename(2)' or 'linkat(2)' to test 'EMLINK' error.
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
int tst_fs_fill_hardlinks(const char *dir);
-------------------------------------------------------------------------------
Try to get maximum count of hard links to a regular file inside the 'dir'.
NOTE: This number depends on the filesystem 'dir' is on.
This function uses 'link(2)' to create hard links to a single file until it
gets 'EMLINK' or creates 65535 links. If the limit is hit, the maximum number of
hardlinks is returned and the 'dir' is filled with hardlinks in format
"testfile%i", where i belongs to [0, limit) interval. If no limit is hit or if
'link(2)' failed with 'ENOSPC' or 'EDQUOT', zero is returned and previously
created files are removed.
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
int tst_fs_fill_subdirs(const char *dir);
-------------------------------------------------------------------------------
Try to get maximum number of subdirectories in directory.
NOTE: This number depends on the filesystem 'dir' is on. For current kernel,
subdir limit is not available for all filesystems (available for ext2, ext3,
minix, sysv and more). If the test runs on some other filesystems, like ramfs,
tmpfs, it will not even try to reach the limit and return 0.
This function uses 'mkdir(2)' to create directories in 'dir' until it gets
'EMLINK' or creates 65535 directories. If the limit is hit, the maximum number
of subdirectories is returned and the 'dir' is filled with subdirectories in
format "testdir%i", where i belongs to [0, limit - 2) interval (because each
newly created dir has two links already - the '.' and the link from parent
dir). If no limit is hit or if 'mkdir(2)' failed with 'ENOSPC' or 'EDQUOT',
zero is returned and previously created directories are removed.
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
int tst_dir_is_empty(const char *dir, int verbose);
-------------------------------------------------------------------------------
Returns non-zero if directory is empty and zero otherwise.
Directory is considered empty if it contains only '.' and '..'.
2.2.18 Getting an unused PID number
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Some tests require a 'PID', which is not used by the OS (does not belong to
any process within it). For example, kill(2) should set errno to 'ESRCH' if
it's passed such 'PID'.
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
pid_t tst_get_unused_pid(void);
-------------------------------------------------------------------------------
Return a 'PID' value not used by the OS or any process within it.
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
int tst_get_free_pids(void);
-------------------------------------------------------------------------------
Returns number of unused pids in the system. Note that this number may be
different once the call returns and should be used only for rough estimates.
2.2.20 Running executables
^^^^^^^^^^^^^^^^^^^^^^^^^^
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
int tst_run_cmd(const char *const argv[],
const char *stdout_path,
const char *stderr_path,
int pass_exit_val);
-------------------------------------------------------------------------------
'tst_run_cmd' is a wrapper for 'vfork() + execvp()' which provides a way
to execute an external program.
'argv[]' is a NULL-terminated array of strings starting with the program name
which is followed by optional arguments.
A non-zero 'pass_exit_val' makes 'tst_run_cmd' return the program exit code to
the caller. A zero for 'pass_exit_val' makes 'tst_run_cmd' exit the tests
on failure.
In case that 'execvp()' has failed and the 'pass_exit_val' flag was set, the
return value is '255' if 'execvp()' failed with 'ENOENT' and '254' otherwise.
'stdout_path' and 'stderr_path' determine where to redirect the program
stdout and stderr I/O streams.
.Example
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
const char *const cmd[] = { "ls", "-l", NULL };
...
/* Store output of 'ls -l' into log.txt */
tst_run_cmd(cmd, "log.txt", NULL, 0);
...
-------------------------------------------------------------------------------
2.2.21 Measuring elapsed time and helper functions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[source,c]
-------------------------------------------------------------------------------
#include "tst_timer.h"
void tst_timer_check(clockid_t clk_id);
void tst_timer_start(clockid_t clk_id);
void tst_timer_stop(void);
struct timespec tst_timer_elapsed(void);
long long tst_timer_elapsed_ms(void);
long long tst_timer_elapsed_us(void);
int tst_timer_expired_ms(long long ms);
-------------------------------------------------------------------------------
The 'tst_timer_check()' function checks if specified 'clk_id' is suppored and
exits the test with 'TCONF' otherwise. It's expected to be used in test
'setup()' before any resources that needs to be cleaned up are initialized,
hence it does not include a cleanup function parameter.
The 'tst_timer_start()' marks start time and stores the 'clk_id' for further
use.
The 'tst_timer_stop()' marks the stop time using the same 'clk_id' as last
call to 'tst_timer_start()'.
The 'tst_timer_elapsed*()' returns time difference between the timer start and
last timer stop in several formats and units.
The 'tst_timer_expired_ms()' function checks if the timer started by
'tst_timer_start()' has been running longer than ms miliseconds. The function
returns non-zero if timer has expired and zero otherwise.
IMPORTANT: The timer functions use 'clock_gettime()' internally which needs to
be linked with '-lrt' on older glibc. Please do not forget to add
'LDLIBS+=-lrt' in Makefile.
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
#include "tst_timer.h"
static void setup(void)
{
...
tst_timer_check(CLOCK_MONOTONIC);
...
}
static void run(void)
{
...
tst_timer_start(CLOCK_MONOTONIC);
...
while (!tst_timer_expired_ms(5000)) {
...
}
...
}
sturct tst_test test = {
...
.setup = setup,
.test_all = run,
...
};
-------------------------------------------------------------------------------
Expiration timer example usage.
[source,c]
-------------------------------------------------------------------------------
long long tst_timespec_to_us(struct timespec t);
long long tst_timespec_to_ms(struct timespec t);
struct timeval tst_us_to_timeval(long long us);
struct timeval tst_ms_to_timeval(long long ms);
int tst_timespec_lt(struct timespec t1, struct timespec t2);
struct timespec tst_timespec_add_us(struct timespec t, long long us);
struct timespec tst_timespec_diff(struct timespec t1, struct timespec t2);
long long tst_timespec_diff_us(struct timespec t1, struct timespec t2);
long long tst_timespec_diff_ms(struct timespec t1, struct timespec t2);
struct timespec tst_timespec_abs_diff(struct timespec t1, struct timespec t2);
long long tst_timespec_abs_diff_us(struct timespec t1, struct timespec t2);
long long tst_timespec_abs_diff_ms(struct timespec t1, struct timespec t2);
-------------------------------------------------------------------------------
The first four functions are simple inline conversion functions.
The 'tst_timespec_lt()' function returns non-zero if 't1' is earlier than
't2'.
The 'tst_timespec_add_us()' function adds 'us' microseconds to the timespec
't'. The 'us' is expected to be positive.
The 'tst_timespec_diff*()' functions returns difference between two times, the
't1' is expected to be later than 't2'.
The 'tst_timespec_abs_diff*()' functions returns absolute value of difference
between two times.
NOTE: All conversions to ms and us rounds the value.
2.2.22 Datafiles
^^^^^^^^^^^^^^^^
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
static const char *const res_files[] = {
"foo",
"bar",
NULL
};
static struct tst_test test = {
...
.resource_files = res_files,
...
}
-------------------------------------------------------------------------------
If the test needs additional files to be copied to the test temporary
directory all you need to do is to list their filenames in the
'NULL'-terminated array '.resource_files' in the tst_test structure.
When resource files is set test temporary directory is created automatically,
there is need to set '.needs_tmpdir' as well.
The test library looks for datafiles first, these are either stored in a
directory called +datafiles+ in the +$PWD+ at the start of the test or in
+$LTPROOT/testcases/data/${test_binary_name}+. If the file is not found the
library looks into +$LTPROOT/testcases/bin/+ and to +$PWD+ at the start of the
test. This ensures that the testcases can copy the file(s) effortlessly both
when test is started from the directory it was compiled in as well as when LTP
was installed.
The file(s) are copied to the newly created test temporary directory which is
set as the test working directory when the 'test()' functions is executed.
2.2.23 Code path tracing
^^^^^^^^^^^^^^^^^^^^^^^^
'tst_res' is a macro, so on when you define a function in one file:
[source,c]
-------------------------------------------------------------------------------
int do_action(int arg)
{
...
if (ok) {
tst_res(TPASS, "check passed");
return 0;
} else {
tst_res(TFAIL, "check failed");
return -1;
}
}
-------------------------------------------------------------------------------
and call it from another file, the file and line reported by 'tst_res' in this
function will be from the former file.
'TST_TRACE' can make the analysis of such situations easier. It's a macro which
inserts a call to 'tst_res(TINFO, ...)' in case its argument evaluates to
non-zero. In this call to 'tst_res(TINFO, ...)' the file and line will be
expanded using the actual location of 'TST_TRACE'.
For example, if this another file contains:
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
if (TST_TRACE(do_action(arg))) {
...
}
-------------------------------------------------------------------------------
the generated output may look similar to:
-------------------------------------------------------------------------------
common.h:9: FAIL: check failed
test.c:8: INFO: do_action(arg) failed
-------------------------------------------------------------------------------
2.2.24 Tainted kernels
^^^^^^^^^^^^^^^^^^^^^^
If you need to detect, if a testcase triggers a kernel warning, bug or oops,
the following can be used to detect TAINT_W or TAINT_D:
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
#include "tst_taint.h"
void setup(void)
{
...
tst_taint_init(TST_TAINT_W | TST_TAINT_D);
...
}
...
void run(void)
{
...
if (tst_taint_check() == 0)
tst_res(TPASS, "kernel is not tainted");
else
tst_res(TFAIL, "kernel is tainted");
}
-------------------------------------------------------------------------------
You have to call tst_taint_init() with non-zero flags first, preferably during
setup(). The function will generate a TCONF if the requested flags are not
fully supported on the running kernel, and TBROK if either a zero mask was
supplied or if the kernel is already tainted before executing the test.
Then you can call tst_taint_check() during run(), which returns 0 or the
tainted flags set in /proc/sys/kernel/tainted as specified earlier.
Depending on your kernel version, not all tainted-flags will be supported.
For reference to tainted kernels, see kernel documentation:
Documentation/admin-guide/tainted-kernels.rst or
https://www.kernel.org/doc/html/latest/admin-guide/tainted-kernels.html
2.2.25 Checksums
^^^^^^^^^^^^^^^^
CRC32c checksum generation is supported by LTP. In order to use it, the
test should include "tst_checksum.h" header, then can call tst_crc32c().
2.2.26 Checking kernel for the driver support
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Some tests may need specific kernel drivers, either compiled in, or built
as a module. If .need_drivers points to a NULL-terminated array of kernel
module names these are all checked and the test exits with TCONF on the
first missing driver.
Since it relies on modprobe command, the check will be skipped if the command
itself is not available on the system.
2.2.27 Saving & restoring /proc|sys values
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
LTP library can be instructed to save and restore value of specified
(/proc|sys) files. This is achieved by initialized tst_test struct
field 'save_restore'. It is a NULL terminated array of strings where
each string represents a file, whose value is saved at the beginning
and restored at the end of the test. Only first line of a specified
file is saved and restored.
Pathnames can be optionally prefixed to specify how strictly (during
'store') are handled files that don't exist:
(no prefix) - test ends with TCONF
'?' - test prints info message and continues
'!' - test ends with TBROK
'restore' is always strict and will TWARN if it encounters any error.
Example:
static const char *save_restore[] = {
"/proc/sys/kernel/core_pattern",
NULL,
};
static void setup(void)
{
FILE_PRINTF("/proc/sys/kernel/core_pattern", "/mypath");
}
static struct tst_test test = {
...
.setup = setup,
.save_restore = save_restore,
};
2.3 Writing a testcase in shell
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LTP supports testcases to be written in a portable shell too.
There is a shell library modeled closely to the C interface at
'testcases/lib/tst_test.sh'.
WARNING: All identifiers starting with TST_ or tst_ are reserved for the
test library.
2.3.1 Basic test interface
^^^^^^^^^^^^^^^^^^^^^^^^^^
[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh
#
# This is a basic test for true shell buildin
#
TST_TESTFUNC=do_test
. tst_test.sh
do_test()
{
true
ret=$?
if [ $ret -eq 0 ]; then
tst_res TPASS "true returned 0"
else
tst_res TFAIL "true returned $ret"
fi
}
tst_run
-------------------------------------------------------------------------------
TIP: To execute this test the 'tst_test.sh' library must be in '$PATH'. If you
are executing the test from a git checkout you can run it as
'PATH="$PATH:../../lib" ./foo01.sh'
The shell library expects test setup, cleanup and the test function executing
the test in the '$TST_SETUP', '$TST_CLEANUP' and '$TST_TESTFUNC' variables.
Both '$TST_SETUP' and '$TST_CLEANUP' are optional.
The '$TST_TESTFUNC' may be called several times if more than one test
iteration was requested by passing right command line options to the test.
The '$TST_CLEANUP' may be called even in the middle of the setup and must be
able to clean up correctly even in this situation. The easiest solution for
this is to keep track of what was initialized and act accordingly in the
cleanup.
WARNING: Similar to the C library, calling tst_brk() in the $TST_CLEANUP does
not exit the test and 'TBROK' is converted to 'TWARN'.
Notice also the 'tst_run' function called at the end of the test that actually
starts the test.
[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh
#
# Example test with tests in separate functions
#
TST_TESTFUNC=test
TST_CNT=2
. tst_test.sh
test1()
{
tst_res TPASS "Test $1 passed"
}
test2()
{
tst_res TPASS "Test $1 passed"
}
tst_run
# output:
# foo 1 TPASS: Test 1 passed
# foo 2 TPASS: Test 2 passed
-------------------------------------------------------------------------------
If '$TST_CNT' is set, the test library looks if there are functions named
'$\{TST_TESTFUNC\}1', ..., '$\{TST_TESTFUNC\}$\{TST_CNT\}' and if these are
found they are executed one by one. The test number is passed to it in the '$1'.
[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh
#
# Example test with tests in a single function
#
TST_TESTFUNC=do_test
TST_CNT=2
. tst_test.sh
do_test()
{
case $1 in
1) tst_res TPASS "Test $1 passed";;
2) tst_res TPASS "Test $1 passed";;
esac
}
tst_run
# output:
# foo 1 TPASS: Test 1 passed
# foo 2 TPASS: Test 2 passed
-------------------------------------------------------------------------------
Otherwise, if '$TST_CNT' is set but there is no '$\{TST_TESTFUNC\}1', etc.,
the '$TST_TESTFUNC' is executed '$TST_CNT' times and the test number is passed
to it in the '$1'.
[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh
#
# Example test with tests in a single function, using $TST_TEST_DATA and
# $TST_TEST_DATA_IFS
#
TST_TESTFUNC=do_test
TST_TEST_DATA="foo:bar:d dd"
TST_TEST_DATA_IFS=":"
. tst_test.sh
do_test()
{
tst_res TPASS "Test $1 passed with data '$2'"
}
tst_run
# output:
# foo 1 TPASS: Test 1 passed with data 'foo'
# foo 2 TPASS: Test 1 passed with data 'bar'
# foo 3 TPASS: Test 1 passed with data 'd dd'
-------------------------------------------------------------------------------
It's possible to pass data for function with '$TST_TEST_DATA'. Optional
'$TST_TEST_DATA_IFS' is used for splitting, default value is space.
[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh
#
# Example test with tests in a single function, using $TST_TEST_DATA and $TST_CNT
#
TST_TESTFUNC=do_test
TST_CNT=2
TST_TEST_DATA="foo bar"
. tst_test.sh
do_test()
{
case $1 in
1) tst_res TPASS "Test $1 passed with data '$2'";;
2) tst_res TPASS "Test $1 passed with data '$2'";;
esac
}
tst_run
# output:
# foo 1 TPASS: Test 1 passed with data 'foo'
# foo 2 TPASS: Test 2 passed with data 'foo'
# foo 3 TPASS: Test 1 passed with data 'bar'
# foo 4 TPASS: Test 2 passed with data 'bar'
-------------------------------------------------------------------------------
'$TST_TEST_DATA' can be used with '$TST_CNT'. If '$TST_TEST_DATA_IFS' not specified,
space as default value is used. Of course, it's possible to use separate functions.
2.3.2 Library variables
^^^^^^^^^^^^^^^^^^^^^^^
Similarily to the C library various checks and preparations can be requested
simply by setting right '$TST_NEEDS_FOO'.
[options="header"]
|=============================================================================
| Variable name | Action done
| 'TST_NEEDS_ROOT' | Exit the test with 'TCONF' unless executed under root
| 'TST_NEEDS_TMPDIR' | Create test temporary directory and cd into it.
| 'TST_NEEDS_DEVICE' | Prepare test temporary device, the path to testing
device is stored in '$TST_DEVICE' variable.
| 'TST_NEEDS_CMDS' | String with command names that has to be present for
the test (see below).
| 'TST_NEEDS_MODULE' | Test module name needed for the test (see below).
| 'TST_NEEDS_DRIVERS'| Checks kernel drivers support for the test.
|=============================================================================
Checking for presence of commands
+++++++++++++++++++++++++++++++++
[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh
...
TST_NEEDS_CMDS="modinfo modprobe"
. tst_test.sh
...
-------------------------------------------------------------------------------
Setting '$TST_NEEDS_CMDS' to a string listing required commands will check for
existence each of them and exits the test with 'TCONF' on first misssing.
Alternatively the 'tst_test_cmds()' function can be used to do the same on
runtime, since sometimes we need to the check at runtime too.
'tst_check_cmds()' can be used for requirements just for a particular test
as it doesn't exit (it issues 'tst_res TCONF'). Expected usage is:
...
TST_TESTFUNC=do_test
. tst_test.sh
do_test()
{
tst_check_cmds cmd || return
cmd --foo
...
}
tst_run
...
Locating kernel modules
+++++++++++++++++++++++
The LTP build system can build kernel modules as well, setting
'$TST_NEEDS_MODULE' to module name will cause to library to look for the
module in a few possible paths.
If module was found the path to it will be stored into '$TST_MODPATH'
variable, if module wasn't found the test will exit with 'TCONF'.
2.3.3 Optional command line parameters
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh
#
# Optional test command line parameters
#
TST_OPTS="af:"
TST_USAGE=usage
TST_PARSE_ARGS=parse_args
TST_TESTFUNC=do_test
. tst_test.sh
ALTERNATIVE=0
MODE="foo"
usage()
{
cat << EOF
usage: $0 [-a] [-f <foo|bar>]
OPTIONS
-a Enable support for alternative foo
-f Specify foo or bar mode
EOF
}
parse_args()
{
case $1 in
a) ALTERNATIVE=1;;
f) MODE="$2";;
esac
}
do_test()
{
...
}
tst_run
-------------------------------------------------------------------------------
The 'getopts' string for optional parameters is passed in the '$TST_OPTS'
variable. There are a few default parameters that cannot be used by a test,
these can be listed with passing help '-h' option to any test.
The function that prints the usage is passed in '$TST_USAGE', the help for
the options implemented in the library is appended when usage is printed.
Lastly the fucntion '$PARSE_ARGS' is called with the option name in the '$1'
and, if option has argument, its value in the '$2'.
[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh
#
# Optional test positional parameters
#
TST_POS_ARGS=3
TST_USAGE=usage
TST_TESTFUNC=do_test
. tst_test.sh
usage()
{
cat << EOF
usage: $0 [min] [max] [size]
EOF
}
min="$1"
max="$2"
size="$3"
do_test()
{
...
}
tst_run
-------------------------------------------------------------------------------
You can also request a number of positional parameters by setting the
'$TST_POS_ARGS' variable. If you do, these will be available as they were
passed directly to the script in '$1', '$2', ..., '$n'.
2.3.4 Usefull library functions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Retrieving configuration variables
++++++++++++++++++++++++++++++++++
You may need to retrieve configuration values such as PAGESIZE, there is
'getconf' but as some system may not have it, you are advised to use
'tst_getconf' instead. Note that it implements subset of 'getconf'
system variables used by the testcases only.
[source,sh]
-------------------------------------------------------------------------------
# retrieve PAGESIZE
pagesize=`tst_getconf PAGESIZE`
-------------------------------------------------------------------------------
Sleeping for subsecond intervals
++++++++++++++++++++++++++++++++
Albeit there is a sleep command available basically everywhere not all
implementations can support sleeping for less than one second. And most of the
time sleeping for a second is too much. Therefore LTP includes 'tst_sleep'
that can sleep for defined amount of seconds, milliseconds or microseconds.
[source,sh]
-------------------------------------------------------------------------------
# sleep for 100 milliseconds
tst_sleep 100ms
-------------------------------------------------------------------------------
Retry a function in limited time
++++++++++++++++++++++++++++++++
Sometimes LTP test needs retrying a function for many times to get success.
This achievement makes that possible via keeping it retrying if the return
value of the function is NOT as we expected. After exceeding a limited time,
test will break from the retries immediately.
[source,c]
-------------------------------------------------------------------------------
# retry function in 1 second
TST_RETRY_FUNC(FUNC, EXPECTED_RET)
# retry function in N second
TST_RETRY_FN_EXP_BACKOFF(FUNC, EXPECTED_RET, N)
-------------------------------------------------------------------------------
[source,sh]
-------------------------------------------------------------------------------
# retry function in 1 second
TST_RETRY_FUNC "FUNC arg1 arg2 ..." "EXPECTED_RET"
# retry function in N second
TST_RETRY_FN_EXP_BACKOFF "FUNC arg1 arg2 ..." "EXPECTED_RET" "N"
-------------------------------------------------------------------------------
Checking for integers
+++++++++++++++++++++
[source,sh]
-------------------------------------------------------------------------------
# returns zero if passed an integer parameter, non-zero otherwise
tst_is_int "$FOO"
-------------------------------------------------------------------------------
Obtaining random numbers
++++++++++++++++++++++++
There is no '$RANDOM' in portable shell, use 'tst_random' instead.
[source,sh]
-------------------------------------------------------------------------------
# get random integer between 0 and 1000 (including 0 and 1000)
tst_random 0 1000
-------------------------------------------------------------------------------
Formatting device with a filesystem
+++++++++++++++++++++++++++++++++++
The 'tst_mkfs' helper will format device with the filesystem.
[source,sh]
-------------------------------------------------------------------------------
# format test device with ext2
tst_mkfs ext2 $TST_DEVICE
-------------------------------------------------------------------------------
Umounting filesystems
+++++++++++++++++++++
The 'tst_umount' helper is a safe way to umount a filesystem.
If the path passed to the function is not mounted (present in '/proc/mounts')
it's noop.
Otherwise it retries to umount the filesystem a few times on a failure, which
is a workaround since there are a daemons dumb enough to probe all newly
mounted filesystems, which prevents them from umounting shortly after they
were mounted.
Running commands as different user with 'su'
++++++++++++++++++++++++++++++++++++++++++++
While some distributions retain paths added to +$PATH+ when doing
+su user -c "command"+ this does not work at least in Debian. If you want to
run LTP binaries as a different user you must use 'tst_su' instead which sets
up +$PATH+ and the runs the command.
.Run test child binary as a test user
[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh
TCID=foo01
. test.sh
tst_su testusr foo01_child
if [ $? -ne 0 ]; then
tst_resm TFAIL "foo failed"
else
tst_resm TPASS "foo passed"
fi
-------------------------------------------------------------------------------
ROD and ROD_SILENT
++++++++++++++++++
These functions supply the 'SAFE_MACROS' used in C although they work and are
named differently.
[source,sh]
-------------------------------------------------------------------------------
ROD_SILENT command arg1 arg2 ...
# is shorthand for:
command arg1 arg2 ... > /dev/null 2>&1
if [ $? -ne 0 ]; then
tst_brkm TBROK "..."
fi
ROD command arg1 arg2 ...
# is shorthand for:
ROD arg1 arg2 ...
if [ $? -ne 0 ]; then
tst_brkm TBROK "..."
fi
-------------------------------------------------------------------------------
WARNING: Keep in mind that output redirection (to a file) happens in the
caller rather than in the ROD function and cannot be checked for
write errors by the ROD function.
As a matter of a fact doing +ROD echo a > /proc/cpuinfo+ would work just fine
since the 'ROD' function will only get the +echo a+ part that will run just
fine.
[source,sh]
-------------------------------------------------------------------------------
# Redirect output to a file with ROD
ROD echo foo \> bar
-------------------------------------------------------------------------------
Note the '>' is escaped with '\', this causes that the '>' and filename are
passed to the 'ROD' function as parameters and the 'ROD' function contains
code to split '$@' on '>' and redirects the output to the file.
EXPECT_PASS and EXPECT_FAIL
+++++++++++++++++++++++++++
[source,sh]
-------------------------------------------------------------------------------
EXPECT_PASS command arg1 arg2 ... [ \> file ]
EXPECT_FAIL command arg1 arg2 ... [ \> file ]
-------------------------------------------------------------------------------
'EXPECT_PASS' calls 'tst_resm TPASS' if the command exited with 0 exit code,
and 'tst_resm TFAIL' otherwise. 'EXPECT_FAIL' does vice versa.
Output redirection rules are the same as for the 'ROD' function. In addition
to that, 'EXPECT_FAIL' always redirects the command's stderr to '/dev/null'.
tst_kvcmp
+++++++++
This command compares the currently running kernel version given conditions
with syntax similar to the shell test command.
[source,sh]
-------------------------------------------------------------------------------
# Exit the test if kernel version is older or equal to 2.6.8
if tst_kvcmp -le 2.6.8; then
tst_brk TCONF "Kernel newer than 2.6.8 is needed"
fi
# Exit the test if kernel is newer than 3.8 and older than 4.0.1
if tst_kvcmp -gt 3.8 -a -lt 4.0.1; then
tst_brk TCONF "Kernel must be older than 3.8 or newer than 4.0.1"
fi
-------------------------------------------------------------------------------
[options="header"]
|=======================================================================
| expression | description
| -eq kver | Returns true if kernel version is equal
| -ne kver | Returns true if kernel version is not equal
| -gt kver | Returns true if kernel version is greater
| -ge kver | Returns true if kernel version is greater or equal
| -lt kver | Returns true if kernel version is lesser
| -le kver | Returns true if kernel version is lesser or equal
| -a | Does logical and between two expressions
| -o | Does logical or between two expressions
|=======================================================================
The format for kernel version has to either be with one dot e.g. '2.6' or with
two dots e.g. '4.8.1'.
.tst_fs_has_free
[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh
...
# whether current directory has 100MB free space at least.
if ! tst_fs_has_free . 100MB; then
tst_brkm TCONF "Not enough free space"
fi
...
-------------------------------------------------------------------------------
The 'tst_fs_has_free' shell interface returns 0 if the specified free space is
satisfied, 1 if not, and 2 on error.
The second argument supports suffixes kB, MB and GB, the default unit is Byte.
.tst_retry
[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh
...
# Retry ping command three times
tst_retry "ping -c 1 127.0.0.1"
if [ $? -ne 0 ]; then
tst_resm TFAIL "Failed to ping 127.0.0.1"
else
tst_resm TPASS "Successfully pinged 127.0.0.1"
fi
...
-------------------------------------------------------------------------------
The 'tst_retry' function allows you to retry a command after waiting small
amount of time until it succeeds or until given amount of retries has been
reached (default is three attempts).
2.3.5 Restarting daemons
^^^^^^^^^^^^^^^^^^^^^^^^
Restarting system daemons is a complicated task for two reasons.
* There are different init systems
(SysV init, systemd, etc...)
* Daemon names are not unified between distributions
(apache vs httpd, cron vs crond, various syslog variations)
To solve these problems LTP has 'testcases/lib/daemonlib.sh' library that
provides functions to start/stop/query daemons as well as variables that store
correct daemon name.
.Supported operations
|==============================================================================
| start_daemon() | Starts daemon, name is passed as first parameter.
| stop_daemon() | Stops daemon, name is passed as first parameter.
| restart_daemon() | Restarts daemon, name is passed as first parameter.
| status_daemon() | Detect daemon status (exit code: 0: running, 1: not running).
|==============================================================================
.Variables with detected names
|==============================================================================
| CROND_DAEMON | Cron daemon name (cron, crond).
| SYSLOG_DAEMON | Syslog daemon name (syslog, syslog-ng, rsyslog).
|==============================================================================
.Cron daemon restart example
[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh
#
# Cron daemon restart example
#
TCID=cron01
TST_COUNT=1
. test.sh
. daemonlib.sh
...
restart_daemon $CROND_DAEMON
...
tst_exit
-------------------------------------------------------------------------------
2.3.6 Access to the checkpoint interface
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The shell library provides an implementation of the checkpoint interface
compatible with the C version. All TST_CHECKPOINT_* functions are available.
In order to initialize checkpoints '$TST_NEEDS_CHECKPOINTS' must be set to '1'
before the inclusion of 'test.sh':
[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh
TST_NEEDS_CHECKPOINTS=1
. test.sh
-------------------------------------------------------------------------------
Since both the implementations are compatible, it's also possible to start
a child binary process from a shell test and synchronize with it. This process
must have checkpoints initialized by calling tst_reinit()'.
3. Common problems
------------------
This chapter describes common problems/misuses and less obvious design patters
(quirks) in UNIX interfaces. Read it carefully :)
3.1 umask()
~~~~~~~~~~~
I've been hit by this one several times already... When you create files
with 'open()' or 'creat()' etc, the mode specified as the last parameter *is
not* the mode the file is created with. The mode depends on current 'umask()'
settings which may clear some of the bits. If your test depends on specific
file permissions you need either to change umask to 0 or 'chmod()' the file
afterwards or use SAFE_TOUCH() that does the 'chmod()' for you.
3.2 access()
~~~~~~~~~~~
If 'access(some_file, W_OK)' is executed by root, it will return success even
if the file doesn't have write permission bits set (the same holds for R_OK
too). For sysfs files you can use 'open()' as a workaround to check file
read/write permissions. It might not work for other filesystems, for these you
have to use 'stat()', 'lstat()' or 'fstat()'.
3.3 umount() EBUSY
~~~~~~~~~~~~~~~~~~
Various desktop daemons (gvfsd-trash is known for that) may be stupid enough
to probe all newly mounted filesystem which results in 'umount(2)' failing
with 'EBUSY'; use 'tst_umount()' described in 2.2.19 that retries in this case
instead of plain 'umount(2)'.
3.4 FILE buffers and fork()
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Be vary that if a process calls 'fork(2)' the child process inherits open
descriptors as well as copy of the parent memory so especially if there are
any open 'FILE' buffers with a data in them they may be written both by the
parent and children resulting in corrupted/duplicated data in the resulting
files.
Also open 'FILE' streams are flushed and closed at 'exit(3)' so if your
program works with 'FILE' streams, does 'fork(2)', and the child may end up
calling 'exit(3)' you will likely end up with corrupted files.
The solution to this problem is either simply call 'fflush(NULL)' that flushes
all open output 'FILE' streams just before doing 'fork(2)'. You may also use
'_exit(2)' in child processes which does not flush 'FILE' buffers and also
skips 'atexit(3)' callbacks.
4. Test Contribution Checklist
------------------------------
1. Test compiles and runs fine (check with -i 10 too)
2. Checkpatch does not report any errors
3. The runtest entires are in place
4. Test files are added into corresponding .gitignore files
5. Patches apply over the latest git
4.1 About .gitignore files
~~~~~~~~~~~~~~~~~~~~~~~~~~
There are numerous '.gitignore' files in the LTP tree. Usually there is a
'.gitignore' file per a group of tests. The reason for this setup is simple.
It's easier to maintain a '.gitignore' file per directory with tests, rather
than having single file in the project root directory. This way, we don't have
to update all the gitignore files when moving directories, and they get deleted
automatically when a directory with tests is removed.