If a test instance from .yaml file was skipped due to being on
a filtered list then there are no entries about it in the
sanitycheck_report.xml for a given platform. This commit fills
instance.results with skipped test cases also for skipped instances so
those test cases are visible when the full report is generated. Tested
for ./tests/kernel/mem_protect/userspace on nrf52dk_nrf52832 where
userspace.gap_filling is filtered out.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
It is now possible to add a list of fixture a platform supports which is
matched to testcases requesting fixtures to be avaiable to be able to
run.
For example:
- available: true
connected: true
id: 0240000026334e450015400f5e0e000b4eb1000097969900
platform: frdm_k64f
product: DAPLink CMSIS-DAP
runner: pyocd
serial: /dev/ttyACM9
fixtures:
- gpio_loopback
Fixes#24943
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Cleanup fixture processing and allow ztest testcases to support
harness_config with fixture definition.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This fix creates class variables SAMPLE_FILENAME & TESTCASE_FILENAME
in Class TestSuite and remove the hardcoded sample.yaml & testcase.yaml
from add_testcases function. This makes testing for sanitylib script
easier so that shippable do not detect the test_data for sanitylib
as actual zephyr testcases.
Signed-off-by: Aastha Grover <aastha.grover@intel.com>
Add cpu time for QEMUHandler because the guest virtual time
in QEMU icount mode isn't host time and it's maintained by
counting guest instructions, we use QEMU process CPU time to
mostly simulate the time of guest OS.
Signed-off-by: Wentong Wu <wentong.wu@intel.com>
Sometimes QEMU can't handle SIGTERM signal correctly and just
run as normal, in that case kill -9 QEMU process directly and
leave sanitycheck judge the testing result by console output.
For failures caused by other reasons, terminate all releated
processes to avoid CI waits 2 hours and reports timeout.
Signed-off-by: Wentong Wu <wentong.wu@intel.com>
We don't need sanitycheck's use of EDT to report warnings, we'll get
them from the build system. So the warnings are just duplication and
noise, thus lets always suppress them.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
Abort if we find tests that are being skipped incorrectly. Tests should
be skipped using ztest_test_skip().
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
While testing sanitycheck classes, found some wrongly initialisized
variables which caused issues, fixed those..
All Test* classes should now be ignored by pytest and should not be
treated as pytest code.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In some cases terminate (SIGTERM) is not enough and we have to kill the
proces (SIGKILL), otherwise the process is stuck forever and taking
hours to complete and finally timeout in CI.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In some cases we were missing the reason and reporting 'Unknown',
check for failure and set the reason.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add option --report-suffix to append custom string to all generated
files. This is going to be useful for generating results for a specific
version, i.e. --report-suffix zephyr-v2.2.0-1814-ge737761d23
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This option prefers serial device names which are stable
across device plug/un-plug on platforms that support it (currently
just Linux, via /dev/serial/by-id).
This feature is opt-in as not all manufacturers include the
appropriate metadata for udev to generate unique names for their
devices.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Create support for muxed UARTs which are attached to a real
UART and which use GSM 07.10 muxing protocol to create virtual
channels that can be run on top of the real UART.
Signed-off-by: Jukka Rissanen <jukka.rissanen@linux.intel.com>
When retrying failed tests, make sure we keep old results and only
update those tests that were retried.
Also, remove duplication of code for creating the reports and make the
report function a bit more generic.
sanitycheck.xml is now listing or tests, not only the test application.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The "net stacks" shell command support was just removed, but
the net_stacks linker section was left around.
Signed-off-by: Jukka Rissanen <jukka.rissanen@linux.intel.com>
We are not propagting those options all the way to the device handler
now, fix this by using the suite member in the handler which has those
already.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Do not declare VERBOSE as global, instead pass verbosity as argument.
Also get rid of options as global and fix coverage class to not use
global option variable.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We do not use those arg files anymore, so remove them to avoid
confusion and for the sake of cleanup.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We rename the nRF51 Dev Kit board target (nrf51_pca10028)
to nrf51dk_nrf51422. We update all associated references
in the supportive documentation and all nRF51-related
cofigurations and overlay files in the samples and tests
in the tree.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
We were parsing random FAIL messages from the output of test runs ad
testcases and capturing them in the xml output. Now we only parse the
name if it starts with test_.
Fixes#21162
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add the following functions to allow filtering based on device tree
dt_compat_enabled(compat) - Returns true if a device tree node
compatible matches 'compat' and the node is enabled.
dt_alias_exists(alias) - Returns true if a device tree node exists with
'alias' and the node is enabled.
dt_compat_enabled_with_alias - Returns true if a device tree node
compatible matches 'compat' and the node has 'alias' and the node is
enabled.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
A complete overhaul of the sanitycheck script and how we build and run
tests. This new version of sanitycheck uses python for job distribution
and drop use of Make.
In addition to the move to python threading library, the following has
been changed:
- All handlers now run in parallel, meaning that any simulator will run
in parallel and when testing on multiple devices (using
--device-testing) the tests are run in parallel.
- Lexicial filtering (using the filter keyword in yaml files) is now
evaluated at runtime and is no long being pre-processed. This will allow
us to immediately start executing tests and skip the wait time that was
needed for filtering.
- Device testing now supports multiple devices connected at the same
time and is managed using a hardware map that needs to be generated and
maintained for every test environment. (using --generate-hardware-map
option).
- Reports are not long stored in the Zephyr tree and instead stored in
the output directory where all build artifacts are generated.
- Each tested target now has a junit report in the output directory.
- Recording option for performance data and other metrics is now
available. This will allow us to record the output from the console and
store the data for later processing. For example benchmark data can be
captured and uploaded to a tracking server.
- Test configurations (or instances) are no longer being sorted, this
will help with balancing the load when we run sanitycheck on multiple
hosts (as we do in CI).
And many other cleanups and improvements...
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Promote a handy and often-overlooked sys.exit() feature: Passing it a
string (or any other non-int object) prints it to stderr and exits with
status 1.
See the documentation at
https://docs.python.org/3/library/sys.html#sys.exit.
This indirectly prints some errors to stderr that previously went to
stdout.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Fixes these pylint warnings:
scripts/sanity_chk/ini2yaml.py:25:22: R1719: The if expression can
be replaced with 'test' (simplifiable-if-expression)
scripts/sanity_chk/expr_parser.py:208:15: R1719: The if expression
can be replaced with 'bool(test)' (simplifiable-if-expression)
scripts/sanity_chk/expr_parser.py:210:15: R1719: The if expression
can be replaced with 'bool(test)' (simplifiable-if-expression)
Also replace a redundant re.compile().match() with re.match(). compile()
doesn't help when re-compiling the regular expression each time through.
(compile() often doesn't help much in general, because the 're' module
caches compiled regexes.)
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
