This commit adds posibility to save zephyr version in the xml reports.
It adds 'properties' attribute to testsuite attribute. 'properties'
can contain multiple 'property' attributes. Zephyr version is added
as such 'property'.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
If we get a build failure with the -M option we get the following:
TypeError: 'NoneType' object is not iterable
This was due to having the default value of additional_keep in
cleanup_artifacts being None instead of an empty array.
Fixes#29376
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
To make the artifacts we keep around for --device-testing -M re-usable
we need to sanitize any aboslute build paths that might exist in
CMakeCache.txt or zephyr/runners.yaml.
This allows us to build the tests to run on one CI system and run the
tests on another system that has the board(s) connected to it.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
If we specify --device-testing and -M make it so that the resulting
artifacts are usable to run with device-testing again. This means
keeping around the zephyr binary images (zephyr.{bin, hex, elf}) and
a few files so 'west' can function for flashing (CMakeCache.txt and
zephyr/runners.yaml).
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
If we run with --cmake-only we get the following:
`ERROR - Unknown status None`
Fix this by treating no status set as "passed" for the --cmake-only
case.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
The legacy macros were first deprecated in Zephyr v2.3. Now that
Zephyr v2.4 has been released, that makes two releases where these
macros have been deprecated, so it's OK to remove them.
This leaves support for legacy binding syntax in place. Removing that
is left to future work.
We need to update various pieces of documentation related to flash
partitions that never got updated when the new API was introduced.
Consolidate this information in the flash_map.h API reference page,
since that's really where users will run into it. This also gives us
the opportunity to improve this documentation.
Adjust a couple of kconfigfunctions.py and sanitycheck bits to use
non-legacy edtlib APIs.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This patch fixes the ZeroDivisionError which can
occur during percantage of test execution reporting.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
So far 'tests' and 'samples' directories were matched only when they
were first directories in path hierarchy. This doesn't work when running
sanitycheck from directory other than Zephyr source code root.
Match any directory in path hierarchy, similar how file is matched
currently. That way sanitycheck can be executed from outside of Zephyr
source code directory.
Signed-off-by: Marcin Niestroj <m.niestroj@grinn-global.com>
Use --filter=runnable for example to limit the tests being built to
those which actually can run on a device or a emulation platform.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
build_only was being used in different contexts and was confusing, so
clean this up and in some places just set if a test is runnable, a
testcase should always be buildable if the filters are matching.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Only create the dir and file if we have actual "content" that will
exist in testcase_extra.conf. This is to reduce a bit of
noise/footprint in the sanitycheck output dir.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
If we kill qemu because of a timeout, capture this as the reason instead
of reporting the exit code.
Fixes#28040
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
By default, subprocess.Popen commands are supplied as a list of strings.
Using split() allows to use command with arguments, for example it is
possible to use following:
sanitycheck ... \
--device-serial-pty="set_ambient read_terminal.py" \
...
Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
Fix the case where we have no ztest test_* testcases and reports were
missed. This now makes sure we do not report the parent testcase when
there are individual test_* results.
Fixes#27765
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
add argument --pre-script to specify a pre script. This will be
executed before device handler open serial port and invoke runner.
Currently, pre_script can only get from hardware.map file. just like:
./scripts/sanitycheck --hardware-map xxx.map
and set pre_script in xxx.map file.
Signed-off-by: Watson Zeng <zhiwei@synopsys.com>
In some cases it might be a good idea to reset the board for real to
make sure it is completely recovered from some failed state (simple
re-loading of the application binary or even Elf file contents doesn't
affect most of internal CPU states so doesn't help in recovery,
see https://github.com/zephyrproject-rtos/zephyr/issues/25022 &
https://github.com/zephyrproject-rtos/zephyr/issues/26665).
And so we may want to utilize some external utility which triggers the
hard reset (in case of ARC boards it is
https://github.com/foss-for-synopsys-dwc-arc-processors/rff-ftdi-reset).
So we need to have a way to execute an external command before each and
every test.
Now given we already have quite some call-backs we try to use them
before re-inventing the wheel. And pre_script seem to be a good option
with just on minor note - it is called after serial port gets open.
And while in some cases it might be OK if serial port on the board is
not affected by the board's reset, if it is affected we'll be losing
connection on reset (and that's the case with ARC boards BTW as the
FTDI USB-to-Serial IC is also wired to the reset signal on most of
the boards). That said we just move invocation of pre_script before
opening the serial port and everything should be good now.
Signed-off-by: Watson Zeng <zhiwei@synopsys.com>
By default show reports based on last release. Fix a few other issues
where we had 0 values and were dividing by zero.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This will only build/run on emulation platforms.
The decision is made based on the value of the 'simulation' key in the
platform yaml file.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Adds some code to allow QEMU to crash without failing the test.
This is required for testing coredump code as it will certainly
cause QEMU to crash.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
There is a potential that QEMUHandler.handle() returns before
the testing harness finishes processing all the output from QEMU.
Simply wait for the harness thread to finish before returning.
Also, fix the return code in the debug message as it should be
the return code from Popen().
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Commit 8d7bb8ffd8 refactored
device structures which changed some of the linker sections:
"devconfig" was removed, and "devices" was added. However,
the list in sanitycheck's size calculator was not updated,
which results in sanitycheck complaining about unrecognized
sections when doing footprint.
Also, a few sections have been renamed (with added suffix
"_area") due to introduction of Z_ITERABLE_SECTION_RAM/ROM
macros. There are also some missing section names.
Fixes the issue by adding the missing sections names, and
updating existing ones.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This patch should improve couting and reporting of the number
of passed/skipped tests.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
In the junit output the attribute containing the number of skipped tests
must be named "skipped" instead of "skip".
See e.g. https://github.com/junit-team/junit5/blob/main/platform-tests/
src/test/resources/jenkins-junit.xsd#L95
Signed-off-by: Christian Taedcke <christian.taedcke@lemonbeat.com>
Currently '--coverage-tool gcovr' results in using lcov. Fix that to use
gcovr, as requested by user.
Fixes: f6462a3a8c ("sanitycheck: get rid of global VERBOSE")
Signed-off-by: Marcin Niestroj <m.niestroj@grinn-global.com>
Sanitycheck discards test instances if a set of given coditions
is not fulfilled. This leads to empty test results for these
instances. This can introduce ambiguity with results that are
missing due to some bugs in the framework. This commit fills
the results for skipped tests with 'skipped' states and provides
the reason for filtering them out in the msg field. The commit
also fixes the way sanitycheck counts and reports tests/test cases
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
When passing a using --hardware-map, sanitycheck would ignore extra
args passed via --west-flash, such as --west-flash="--erase".
This is because the command with the extra args was overwritten by the
command with the runner info from the hardware map.
This patch merges those code paths so they are aware of each other and
of --west-runner.
Signed-off-by: Øyvind Rønningstad <oyvind.ronningstad@nordicsemi.no>
This commit limits the removal of `test_` from test case name
to only the first occurance. There are test cases with `test_`
also in the middle of their names and removing it couses mismatches
between extracted testcases and the names obtained when passing
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
We have been doing an AND comparison instead of an OR. AND does exclude
way to many testcases where multiple tags are being used.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Use --integration testing for CI to limit execution and builds on
platforms that actually provide most of the coverage instead of blindly
building/running on all available platforms.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
instances that do not run will have no results (beside the fact they
built successfully), so log those as such.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This is a cargo-culted attempt to make an error observed in poorly
understood CI circumstances go away when loading edt.pickle, by making
edtlib visible in sys.modules before loading the pickle file.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Consolidate creation of edtlib.EDT objects from a build directory's
devicetree into one place by loading it from build/zephyr/edt.pickle
everywhere. A previous commit creates edt.pickle from gen_defines.py.
In addition to probably speeding things up slightly by not reparsing
the devicetree, the main benefit of this approach is creating a single
point of truth for the bindings directories and warnings
configuration, meaning we don't have to worry about them getting out
of sync while being passed around between devicetree creation and
usage time.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
serial-pty is an extension that is not supported in the hardware map, so
we need to work around the fact that it is not part of the map when
dealing with normal serial.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add option to provide to sanitycheck argument for creating
pseudoterminal. Used with hardware without serial console connected.
A pseudoterminal is used to make a sanitycheck believe that it
interacts with a terminal although it actually interacts with the
script.
E.g "sanitycheck --device-testing --device-serial-pty <script>"
Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
Support running/building only specific tags and ignoring everything
else.
Adding this to a platform yaml file will enable tests with one of of
tags listed to be executed on the platform.
This is useful for special platform configurations used for testing
specific features for example.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
--warnings-as-errors was basically doing nothing, it is the default.
Replace this with an option to disable erroring on warning:
-W, --disable-warnings-as-errors
Fixes#26910
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We still have cases where qemu crashes are not being correctly captured
in the on-screen summary, however they are captured in the final
generated report.
Fixes#26679
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
restore how --only-failed works by allow rebuilds in case of build
errors, however, do not rebuild when --retry-failed is used, which is a
CI usecase and nothing would change in the second iteration.
Fixes#26685
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
According to our compliance checks, we have 'useless-suppression'
and 'unused-import' on our hands:
W0611: Unused CLoader imported from yaml as Loader
W0611: Unused Loader imported from yaml
I0021: Useless suppression of 'unsubscriptable-object'
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Use the C LibYAML parser if available, rather than the Python parser. It
is much faster.
This is a clean and rebased version of PR #20210 by
Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Co-authored-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
