os.mkdir() is not suitable to create arbitrary directory path (can
create only a subdir of an existing dir, will error out if already
exists), os.makedirs() should be always used in such cases.
Signed-off-by: Paul Sokolovsky <paul.sokolovsky@linaro.org>
A complete overhaul of the sanitycheck script and how we build and run
tests. This new version of sanitycheck uses python for job distribution
and drop use of Make.
In addition to the move to python threading library, the following has
been changed:
- All handlers now run in parallel, meaning that any simulator will run
in parallel and when testing on multiple devices (using
--device-testing) the tests are run in parallel.
- Lexicial filtering (using the filter keyword in yaml files) is now
evaluated at runtime and is no long being pre-processed. This will allow
us to immediately start executing tests and skip the wait time that was
needed for filtering.
- Device testing now supports multiple devices connected at the same
time and is managed using a hardware map that needs to be generated and
maintained for every test environment. (using --generate-hardware-map
option).
- Reports are not long stored in the Zephyr tree and instead stored in
the output directory where all build artifacts are generated.
- Each tested target now has a junit report in the output directory.
- Recording option for performance data and other metrics is now
available. This will allow us to record the output from the console and
store the data for later processing. For example benchmark data can be
captured and uploaded to a tracking server.
- Test configurations (or instances) are no longer being sorted, this
will help with balancing the load when we run sanitycheck on multiple
hosts (as we do in CI).
And many other cleanups and improvements...
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Currently, string values comming from "common" test definition section
and from test-specific section are just concatenated. Suppose, we want
to define some common filter condition, and also per-test additional
criteria. Currently, that leads to following syntax:
common:
filter: TOOLCHAIN_FOO == 1
tests:
sample.net.sockets.http_get.posix:
filter: and not CONFIG_BAR
That's arguable quite adhoc, and the only way to figure it out for
most people will be to add debug logging.
This patch proposes to use the expected syntax (i.e.
"filter: not CONFIG_BAR"), and combine conditions properly based on
their semantic meaning (which also includes parans for proper
evaluation order).
Signed-off-by: Paul Sokolovsky <paul.sokolovsky@linaro.org>
Removed redundant linker flag --coverage from native posix
EXTRA_LDFLAGS.
The correct linker flags will be set by defining the Kconfig flag
CONFIG_COVERAGE, either by sanitcheck it self or via the test/sample
prj.conf.
Signed-off-by: Jan Van Winkel <jan.van_winkel@dxplore.eu>
If we are doing a nightly build we can utilize a large (over 50G) of
disk space just to generate the list of tests to build. We need to
optimize this so as we finish building the initial pass we clean up
as we go and only keep around the files we need (like .config,
generated_dts_board.conf, CMakeCache.txt, etc).
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
This board and SoC was discontinued some time ago and is currently not
maintained in the zephyr tree.
Remove all associated configurations and variants from the tree.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In a3bea8872b (PR #16352)
_bt_br_channels_area was added to the code but not the
sanitycheck sections whitelist => Add it
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
In c20ff1150f
and 5f19c8160a
(PR #16897)
_bt_settings_area was replaced with _settings_handlers_area
Update sanitycheck sections whitelist accordingly
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
sanitycheck --help mentions that west-flash requires device-testing
to be enabled (and indeed it does because DeviceHandler will never
be called where west_flash option is used. Let's generate an error
if west-flash is used w/o specifying device-testing.
Also cleanup help text which looks odd in both sanitycheck --help
and in the file itself.
Signed-off-by: Michael Scott <mike@foundries.io>
Several boards have multiple runners setup. We need a way to specify
which runner to use with sanitycheck. Introduce --west-runner option.
Signed-off-by: Michael Scott <mike@foundries.io>
sanitycheck takes any "extra_config" list found in the testcase.yaml
file and generates an "overlay" file from it. This file is placed in
the per-test build directory and passed to cmake/kconfig.cmake through a
-DOVERLAY_CONF= option set in the (also) generated sanity-out/Makefile.
This commit moves this generated config overlay to a subdirectory one
level down from the build directory, otherwise kconfig.cmake picks it
up *twice*: once from the -DOVERLAY_CONF= option already mentioned above
and a second time because kconfig.cmake scans the build directory and
blindly picks up ALL files ending with .conf[*]. The second pickup is
problematic because kconfig.cmake currently gives it the top precedence,
even higher than anything the user espressed with --extra-args=CONFIG_*
Here's a quick and simple demonstration of the issue fixed by this
commit:
cd $ZEPHYR_BASE/samples/net/sockets/net_mgmt/
sanitycheck -T. -p qemu_x86 -b -v # --extra-args=CONFIG_USERSPACE=y|n
grep CONFIG_USERSPACE $(find sanity-out/ -name .config)
.net_mgmt.kernelmode/zephyr/.config: # CONFIG_USERSPACE is not set
.net_mgmt.usermode/zephyr/.config: CONFIG_USERSPACE=y
grep 'Merged configuration' $(find sanity-out/ -name build.log)
Without this commit, attemps to override anything with
--extra-args=CONFIG_ are silently dropped on the floor.
For more background this issue was found while using the recipe in
commit message 4afcc0f8af
[*] picking up all .conf files is debatable but a much bigger debate
with backward compatibility implications. This small fix makes
absolutely zero difference to anyone or anything not using sanitycheck.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
There are a few reasons why sanitycheck will only build a test and not
run it: list them in the developer guide. Also lists the --options that
provide that information and update their --help message.
A couple other --help fixes.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
The --jobs default was recently changed in commit 9f4f57eed3, update
its help message.
Add the hopefully last missing verbose("Spawning...") statement.
Fix comment updated in commit 095b82a301.
Replace two tags with whitespace.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
sanitycheck is opening an insane number of file descriptors
simultaneously as it opens up communication pipes with
every test that supports emulation, on every emulated
board target.
Increase the resource limit on open files until this code
can be properly refactored.
Workaround for: #17239
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
- The --gcov-tool argument now has a reasonable default
if the Zephyr SDK is in use.
- --coverage-platform, if unspecified, defaults to what
was passed to --platform
- --coverage implies --enable-slow, so that tests with
the 'slow' tag are built and run.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
We have a number of timing sensitive tests which run
correctly on a much more frequent basis if the system
is not so heavily loaded. Instead of squeezing a few
more crumbs of performance by doubling the CPU count,
just use the number of CPUs reported by the system.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
If GCOV coverage is enabled, the coverage dump happens after
"PROJECT EXECUTION SUCCESSFUL" is printed. In some cases,
the additional time added was not enough to capture all the
GCOV output on a heavily loaded system before the emulator
gets killed.
Ideally, the decision to kill the emulator needs to be smarter
and less race-prone, but that can wait for a future
enhancement.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
So the time used to run boards which use the BinaryHandler can be
reported, record the time used from spawning the process until
it finnishes or is killed.
The BinaryHandler is used by the "native" boards, unit tests,
nsim and Renode.
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
We already have the info so let's show it. This helps spots intermittent
issues[*], gives an indication of the time --build-only saves, can help
spot an overloaded test system, highlights the most time-consuming tests
which may need a longer timeout in their config, shows the effective
timeout value when one occurs... all this for a dirt cheap screen estate
price and two extra lines of code.
Sample -v output:
32/81 board123 tests/testme PASSED (qemu 2.049s)
33/81 board456 samples/hello PASSED (build)
34/81 qemu_x3 tests/kernel.stack.usage FAILED: timeout (qemu 60.029s)
see: sanity-out/qemu_x3/tests/kernel.stack.usage/handler.log
35/81 board456 tests/testme PASSED (build)
36/81 qemu_x5 tests/kernel.queue FAILED: failed (qemu 2.191s)
see: sanity-out/qemu_x5/tests/kernel.queue/handler.log
[*] running qemu in heavily packed cloud virtual machines comes to mind,
also see #12553, #14173 etc.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Found a few annoying typos and figured I better run script and
fix anything it can find, here are the results...
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
There is a case where using startswith to determine if a path is a
subdirectory of another path can erroneously match. When using a
testcase root outside of ZEPHYR_BASE, an erroneous match will cause the
relative path containing ".." to get prepended to the test output
directory.
Example:
$HOME/zephyr/zephyr # ZEPHYR_BASE
$HOME/zephyr/zephyr-rust/tests # testcase root
The relative path prepended to the testcase name is ../zephyr-rust/tests
and an example test output dir is
./sanity-out/qemu_x86/../zephyr-rust/tests/rust/rust.main
In this case, the build directory escapes the board directory and is no
longer unique. Parallel tests then clobber each other.
Use pathlib instead of string matching to cover this case.
Signed-off-by: Tyler Hall <tylerwhall@gmail.com>
Add an option that only invokes the cmake phase of sanitycheck. This
can be useful for any testing that only needs to initial generation
phase of cmake, for example device tree. Also useful if we want to
just generate compile_commands.json files from cmake via:
./sanitycheck -xCMAKE_EXPORT_COMPILE_COMMANDS=1 --cmake-only
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
Add new option --report-excluded to list all those tests with bad
filtering that never build or run. This option produces accurate results
with --all but can be used with default sanitycheck options to see what
does not run/build in CI for example. (limited coverage).
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This changes the declaration of fixed channels to be statically defined
with use of BT_L2CAP_CHANNEL_DEFINE since fixed channels are never
unregistered.
Signed-off-by: Luiz Augusto von Dentz <luiz.von.dentz@intel.com>
Execute the test binary from the output directory instead of directory
where sanitycheck was started.
This will ensure that any artifact created with a relative path by the
test binary will be placed in the output directory instead of creating
the artifact in the directory where sanitycheck was executed and prevent
any possible conflicts.
Signed-off-by: Jan Van Winkel <jan.van_winkel@dxplore.eu>
When Zephyr crashes immediately QEMU reports an error immediately. This
is immediately reported by "make run". Then sanitycheck points the user
at the output of "make run". However the error message(s) are in QEMU's
output which is in a different .log file.
To address this situation point the error message at handler.log
instead of run.log if and only if handler.log is not empty.
To reproduce here's an artificial but very simple crash:
sanitycheck --extra-args=CONFIG_TEST_USERSPACE=n \
-p qemu_x86 -T tests/kernel/mem_protect/stackprot/
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Fix issue where sanitycheck wrongly assumed tests inside ZEPHYR_BASE
to be outside ZEPHYR_BASE and dropped the prefix in their name. This
happened when:
- ZEPHYR_BASE contains symbolic link(s), and
- relative --testcase-root argument(s) are passed
To generate unique names, TestCase.get_unique(testcase_root) first
checks whether "testcase_root" starts with ZEPHYR_BASE. Either may or
may not include symbolic links so both must be canonicalized before
comparison. While fixing this method, replace explicit forward slash
"/" and string replace with os.path.relpath() and make a couple other
simplifications and minor pydoc fixes.
Add new canonical_zephyr_base = os.path.realpath(ZEPHYR_BASE) constant
and corresponding comments and guidelines.
The most visible effect of this mismatch was sanitycheck dropping the
--testcase-root prefix from the unique name of tests inside
ZEPHYR_BASE. This means some test names could be not unique anymore
and silently overwrite each other's results, example:
bash# cd zephyr_dir_with_symlink; export ZEPHYR_BASE=$(pwd)
./scripts/sanitycheck -T samples/portability/cmsis_rtos_v1 \
-T samples/portability/cmsis_rtos_v2
The more systematic and practical consequence (and how I actually
found this) was test outputs landing in unexpected locations.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Notably fix the wrong comment I added in commit 6f011c95c477: when
testing with QEmu sanitycheck does _not_ spawn QEmu; it relies on "make
run" instead.
Searching the code for "Spawning" now cycles directly to all the places
starting processes and threads.
- Sample -v -v verbose output (lines wrapped for commit message check)
Spawning QEMUHandler Thread for \
qemu_x86/samples/hello_world/sample.helloworld 'make run'
- native_posix example:
Spawning process /home/.../sanity-out/native_posix/\
/samples/hello_world/sample.helloworld/zephyr/zephyr.exe
Spawning BinaryHandler Thread for native_posix/\
samples/hello_world/sample.helloworld
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
This reintroduces support for static service in the form of a new API,
BT_GATT_SERVICE_DEFINE, and changes the internal services (GAP/GATT)
to be defined as const as they are never register/unregistered.
Internal service needed to be renamed in order to keep the same order
as before since the section elements are sorted by name.
The result is the following (make ram_report):
before:
gatt.c 572 0.66%
cf_cfg 32 0.04%
db 8 0.01%
db_hash 16 0.02%
db_hash_work 32 0.04%
gap_attrs 180 0.21%
gap_svc 12 0.01%
gatt_attrs 160 0.18%
gatt_sc 80 0.09%
gatt_svc 12 0.01%
sc_ccc_cfg 32 0.04%
subscriptions 8 0.01%
after:
gatt.c 210 0.24%
cf_cfg 32 0.04%
db 8 0.01%
db_hash 16 0.02%
db_hash_work 32 0.04%
gatt_sc 80 0.09%
last_static_handle 2 0.00%
sc_ccc_cfg 32 0.04%
subscriptions 8 0.01%
Signed-off-by: Luiz Augusto von Dentz <luiz.von.dentz@intel.com>
The function error expects only one parameter. The excpetion handler in
scan_path was calling this function with multiple parameters instead of
formatting the string.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Using the new option --timestamps, any output from sanitycheck will have
a timestamp to help identify bottle necks and monitor execution time.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
One of the first things needed when comparing builds of tests across
different environments/systems is to make sure the same (sub)tests were
selected and run in the first place. For that purpose sort the output of
--testcase-report and --discard-report as they were in random order.
Actually make the entire class TestInstance sortable by adding a
standard __lt__() method comparing unique instance names; it could be
useful again.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
After the testcase configs are built, there is a step to
filter all the test case information to determine the set
of tests to run.
As this step takes a nontrivial amount of time, add an
informational message about it.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Update the files which contain no license information with the
'Apache-2.0' SPDX license identifier. Many source files in the tree are
missing licensing information, which makes it harder for compliance
tools to determine the correct license.
By default all files without license information are under the default
license of Zephyr, which is Apache version 2.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some logging text and colors were not escaped correctly, make sure we
generate well formed XML reports.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We have not been counting samples in reports. This change lists tests
associated with sample code which in many cases is just verifying output
from the sample and counts as 1 test.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Fix --help message. Also rename run_report() to save_tests() as it's
used only once by --save-tests and nowhere else. Maybe the code was
shared with some --other-report feature in the past but not any more.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
