Fix --help message. Also rename run_report() to save_tests() as it's
used only once by --save-tests and nowhere else. Maybe the code was
shared with some --other-report feature in the past but not any more.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Also add a "generated by sanitycheck" header to indicate origin and a
warning about the dependency required to actually run the test.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
- Fix wrong --test help message
- Provide more examples to clarify naming hierarchy
- Document that --sub-test runs its entire --test
- Point out that save/load options use their own format
- Document that --list-tests is flattened
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
They're effectively mutually exclusive already because
options.sub_test in main() immediately discards any --test
argument(s). This commit preempts user confusion thanks to this new
message:
sanitycheck: error: argument --sub-test: not allowed with
argument -s/--test
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
$ sanitycheck -h # is re-ordered like this:
< ... all other options ... >
-C, --coverage Generate coverage reports. Implies
--enable_coverage
--coverage-platform COVERAGE_PLATFORM
Plarforms to run coverage reports on. This
option may be used multiple times.
Test case selection:
-f, --only-failed Run only those tests that failed the previous
sanity check invocation.
-s TEST, --test TEST Run only the specified test cases. These are
named by <path to test project relative to
--testcase-root>/<testcase.yaml section name>
--sub-test SUB_TEST Run only the specified sub-test cases and its
parent. These are named by test case name
appended by test function, i.e.
kernel.mutex.mutex_lock_unlock.
--list-tests list all tests.
-F FILENAME, --load-tests FILENAME
Load list of tests to be run from file.
-E FILENAME, --save-tests FILENAME
Save list of tests to be run to file.
-T TESTCASE_ROOT, --testcase-root TESTCASE_ROOT
Base directory to recursively search for test
cases. All testcase.yaml files under here will
be processed. May be called multiple times.
Defaults to the 'samples' and 'tests' directories
in the Zephyr tree.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Commit 45a7e5d076 removed scripts/sysgen
and _k_task_list/_k_event_list sections are no longer being used.
So remove them.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Commit 73cb9586ce renamed linker
section from _k_memory_pool to _k_mem_pool, but the references
in linker scripts are still there. So remove them now.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When re-running failed tests, do not go through any filtering, just load
the failed tests directly and execute them. The filtering was done based
on default platforms and any tests that were failing on non-default
platforms were ignored.
Fixes#13956
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We have been overriding OVERLAY_CONFIG coming from tests. Concat all
files and create one variable that we pass to kconfig as list.
Fixes#13320
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Device testing has various problems:
- the main thread doesn't halt the serial monitoring thread when an
error occurs while flashing, meaning flash errors can look like
timeouts
- serial_line is unbound if the ser.readline() call raises an
exception, so the following "if serial_line" would raise NameError
in that case
- the serial monitoring thread is not a daemon, so exceptions or
errors in the handle() thread that cause thread exit will leave the
process sitting around until the serial monitoring thread has a
chance to exit
Fix these issues:
- Rewrite the serial monitoring thread as a select loop over the
serial port file descriptor and a pipe shared by the main thread and
serial monitoring thread. If flashing fails, the main thread uses
the pipe to signal to the serial monitoring thread that it should
exit.
- Make sure serial_line is always bound when read.
- Mark the serial thread daemonic.
Signed-off-by: Marti Bolivar <marti@foundries.io>
If the user has requested verbose output, then use check_call() when
running the flash target so that the build system output goes to the
controlling terminal's stdout instead of being swallowed.
This makes it easier to debug flashing failures.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Parse CMakeCache.txt and filter based on variables defined in CMake.
The CMakeCache.txt parsing is borrowed from west.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In some cases sanitycheck handles a build error that started with 'make
run' as a handler crash when it actually failed during building. Right
now we try to attach the handler.log to the XML output even if it did
not exist. Check for the file and if it was not found, go back to the
build.log
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When calling genhtml to generate a coverage report directly from
sanitycheck (-C option), call it with the "--ignore-errors source"
option, so it does not exit when it meets a missing file, but
instead it warns the user and continues.
Fixes#13014
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
There are no longer per-partition initialization functions.
Instead, we iterate over all of them at boot to set up the
derived k_mem_partitions properly.
Some ARC-specific hacks that should never have been applied
have been removed from the userspace test.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Make sure we capture data from gcov and do not timeout before all the
data has been captured. Also report on incomplete data capture.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When building with older versions of the toolchain we need to use gcov
from the same tool set, otherwise data formar might not be compatible.
Added an option to point to the gcov tool from the same toolchain used
to build the code. By default the host gcov is used.
Relates to #12571
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Running sanitycheck from Windows is not supported yet. Unfortunately,
the error message is obscure when a user is unaware of this and runs
it from Windows anyway.
This patch gives an easy to understand error message explaining this
issue.
Signed-off-by: Sebastian Bøe <sebastian.boe@nordicsemi.no>
While trying to catch crashes after test completes, do not keep
extending the timeout which might lead to and endless loop and not
getting out based on set timeout.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We want to run coverage on select platforms even if a few do support
that now. Specify which platforms should be enabled using the new option
--coverage-platform
By default we have the posix platform configured. Any additional
platforms can be specified using the option above.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The change adds an optional capability to kill process by PID in the
BinaryHandler. At the moment proc.terminate only closed the make
process, also closing the stdin/stdout, but this is impossible to detect
in .NET.
Signed-off-by: Dawid Wojciechowski <dwojciechowski@internships.antmicro.com>
Signed-off-by: Jan Kowalewski <jkowalewski@antmicro.com>
Signed-off-by: Piotr Zierhoffer <pzierhoffer@antmicro.com>
Extend the "slow" flag on tests to inhibit building too. The original
assumption was that building would be fast but running slow, but now
We have tests using a component (OpenThread) that wants to pull and
build software from github for every app.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
The segment size computation has value, but it's actually a very slow
process that parallizes poorly. It seems to be bound by the Python
GIL doing the parsing, so never sees more than about 150% of CPU in
use even on wildly parallel systems. And it takes about 75-80
seconds, which is 15-20% of the entire runtime of the test on that
box!
And the only "failure" case this can detect (unexpected sections in
the output file) is now a duplicate of the orphan section warning
we've since enalbled at the linker level.
This defaults to enabling the test to preserve behavior (as I don't
know where all the existing users of the size report are to change
them), but long term we might consider making "disabled" the default
and switching this to an --enable flag.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
When running valgrind in sanitycheck, use the suppression file
and dump the log in its own file.
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
If the test process returns an error (return code != 0),
it should be considered a failure, even if the handler
considers the test passed.
But this is not done when we are killing (SIGTERM) the processes,
as then the return code is not necessarily sensible.
Without this, for example, when running with valgrind, all valgrind
errors are missed.
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
In addition to Kconfig values from generated .config, also parse and
filter based on configs generated by device-tree.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
There's no current need for this and it makes work items
declared with K_WORK_DEFINE() inaccessible to user mode.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Fix false passed on localized error message in make invocation.
Fixes#8348
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We should leave ZEPHYR_BASE alone and create the test output wherever
we are running.
This will support running sanitycheck on test roots other than zephyr's
main git repo.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
right now we assume we are running the tests inside the Zephyr tree and
things do not work well when someone tries to use sanitycheck on tests
outside the tree with a different test root. Cleanup the name handling
and support running outside of Zephyr.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Set COVERAGE option using an overlay instead of piping it through CMAKE
which breaks dependecies, i.e. if someone wants to disable that option
on the application level.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Support specifying a test fixture attached to a board to filter in/out
tests that require this fixture to be able to run a test.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Allow the user to disable the "unrecognized section" test. I can see
multiple use-cases for disabling the test.
If orphan sections exist and are dynamically or unpredictably named
the unrecognized section test will fail.
If out-of-tree sections exist, one might want to temporarily disable
the "unrecognized section" test until one has made it recognized.
The test is disabled through a CLI flag.
Signed-off-by: Sebastian Bøe <sebastian.boe@nordicsemi.no>
We now place the linker directives for the SW ISR table
in the common linker scripts, instead of repeating it
everywhere.
The table will be placed in RAM if dynamic interrupts are
enabled.
A dedicated section is used, as this data must not move
in between build phases.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
New Qemu is emitting binary charachters which is disturbing generation
of XML, so make sure we only log printable charachters.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This puts the priviledged stack at the end of RAM.
This combines PR #10507 and #10542.
Fixes#10473Fixes#10474Fixes#10515
Signed-off-by: Adithya Baglody <adithya.nagaraj.baglody@intel.com>
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Some boards depend on environment variables so we want to make sure we
do not attempt to build boards requiring additional setup.
Add the section below into the board YAML file, sanitycheck will check
the environment and will only run tests on that board if the variables
are defined.
env:
- VAR1
- VAR2
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
To be able to collect all coverage reports from the "native"
applications in the first CI instance, let's also place the
nrf52_bsim testcases first. The order now is:
unit_testing
native_posix
*_bsim
anything else
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
run tests based on sub-testcases, for example:
sanitycheck --sub-test net.app.app_tcp4_client_hostname_fail
will run all the tests in tests/net/app/net.app.
Useful for re-running tests that are reported in testrail and in reports
generated by sanitycheck.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
