When generating coverage report:
* Also generate branch coverage
* If the report is properly generated tell user where to find it
* Handle case in which no ztests are run (it would crash before)
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
Aligned sanitycheck command line options in doc
with the actual ones.
+
Minor fix in --coverage description.
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
sanitycheck: Compile unit tests with coverage enabled always
+ run also first unit tests together with native_posix
shippable: also include unit_testing coverage into report to
codecov
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
This patch adds app_pad and priv_stacks to the rw section list so that
tests which validate the sections found in the binary pass when run on
systems which contain MPUs.
Signed-off-by: Andy Gross <andy.gross@linaro.org>
Rather than having handler generate differently named log files,
generate the same for all handlers, so qemu, native and unit will now
generate handler.log.
This will simplify things and reporting will be consistent. Also, this
fixes a bug where we only included qemu.log in generated reports.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When building the testcase path, replace symlinks in path
with the realpath both in the testcase root and in ZEPHYR_BASE
For more info see issue #5772Fixes: #5772
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
Changed the default behavior for when we run sanitycheck to match the -R
flag that turns on assertions. Introduced a --disable-asserts option if
we desire to explicitly turn of asserts.
This matches behavior that our CI builds have been doing and addresses
part of #5726.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
Add 2 classes, one to handle the current TestCase scenario, and one more
for handling generic Console with regex matching.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some boards are supported natively by qemu. This option will allow us to
run tests using those platforms directly without having to go via a
dedicated qemu board definition.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
if we are using command line platform filter, no need to list every
other platform as excluded, we know that already. Show only the
discards that apply to the selected platforms on the command line
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When we load tests from a file, we do not have the discarded list, this
would have been done already when saving the test file.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When checking if a testcase passed or failed, allow there
to be prefixes or postfixes in the line, around
PROJECT EXECUTION SUCCESSFUL/FAILED
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
CONFIG_ASSERT is being set by cmake, so it is not possible to filter
using the generated config, add this as a standalone filter.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Generating coverage data is split over two CI jobs which means the
service will need to merge results and reports wrong coverage data when
only 1 job is finished. This puts the native_posix board first making
sure we run on the first job and generate data in one place.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Extend exception handling to cover not just YAML loading, but any
error while accessing parsed data too. That may catch e.g. schema
mismatch errors (for folks who don't have pykwalify installed, which
is optional). So, now error will be logged, but processing of other
tests will continue.
For example, I had a local, uncommitted test which wasn't converted
per 23f81eeb42 and caused:
Traceback (most recent call last):
File "./scripts/sanitycheck", line 2456, in <module>
main()
File "./scripts/sanitycheck", line 2324, in main
options.outdir, options.coverage)
File "./scripts/sanitycheck", line 1445, in __init__
for name in parsed_data.tests.keys():
AttributeError: 'list' object has no attribute 'keys'
Signed-off-by: Paul Sokolovsky <paul.sokolovsky@linaro.org>
We have been passing around options from one function to the next making
it very difficult to add a new option easily and requiring changes to
man function prototypes.
This declated the parsed command line options global and renames args to
options. args is being used elsewhere and this was confusing.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This keyword would mean that a special harness is needed to run the
tests sucessfully. This can be as simple as a loopback wiring or a
complete hardware test setup for sensor and IO testing. It is free form
initially and would be changed to be an enum once we have more values in
place.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We have now different runners/handlers, so avoid using qemu terminology
for the generic classes and for generic usage.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Passing ARCH during the build process is something from the past and
samples/tests should not do that, remove it here to catch any
violations.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Since the move to YAML format and the change in how we define default
platforms this is no longer needed as we are able to set multiple
default platforms per architecture and not using a list based on
priority anymore.
Fixes#4445
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The section terminology was relevant with the ini syntax, with yaml we
can call this a test and avoid confusion and make the code more
readable.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Simplify parsing of yaml structures and remove usage of cp which was for
the ConfigParser used for ini files.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
It is supported to add give extra flags to the linker from the
commandline like this:
cmake -DEXTRA_LDFLAGS=-Lmy_dir path
But unfortunately this was broken during the CMake
migration. Interestingly, the reason that it was broken is that KBuild
was also partially broken. KBuild would pass on EXTRA_LDFLAGS when
object files were linked together into built-in.o files, but it would
not use EXTRA_LDFLAGS for the final link into an elf file.
This patch fixes EXTRA_LDFLAGS.
Signed-off-by: Sebastian Bøe <sebastian.boe@nordicsemi.no>
This can be used by other handlers and is defined in the main Handler
class. Qemu is just an implementer.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This makes piped output work as the user expects. And looking at the
piped output is the only way to use sanitycheck normally because
of #4603.
Signed-off-by: Paul Sokolovsky <paul.sokolovsky@linaro.org>
sanitycheck was incorrectly documenting that --extra-args would pass
on it's input unchanged to Make.
In reality --extra-args acts as a way to define extra CMake cache
entries. The key-value entries will be prefixed with -D before being
passed to CMake.
E.g
"sanitycheck -x=USE_CCACHE=0"
will translate to
"cmake -DUSE_CCACHE=0"
Signed-off-by: Sebastian Boe <sebastian.boe@nordicsemi.no>
We only had a few hundred tests run when sanitycheck was first written,
and printing out the reasoning why tests were skipped seemed reasonable
at the time. Now that we are running tens of thousands of tests, this
is too much information.
The dump of what tests were skipped and why now requires two instances
of --verbose on the command line.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
It's possible to declare static threads that start up as K_USER,
but these threads can't do much since they start with permissions on
no kernel objects other than their own thread object.
Rather than do some run-time synchronization to have some other thread
grant the necessary permissions, we introduce macros
to conveniently assign object permissions to these threads when they
are brought up at boot by the kernel. The tables generated here
are constant and live in ROM when possible.
Example usage:
K_THREAD_DEFINE(my_thread, STACK_SIZE, my_thread_entry,
NULL, NULL, NULL, 0, K_USER, K_NO_WAIT);
K_THREAD_ACCESS_GRANT(my_thread, &my_sem, &my_mutex, &my_pipe);
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
