Add a new option that will enable treating RAM/SRAM overflows as errors
instead of skipping them, which is the default.
Fixes#27583
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We were setting all devices as connected, now do that based on serial
availability.
Rename internal variables and make them shorter:
connected_hardware -> duts
ConnectedDevice = DUT
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The available key is a runtime variable, it does not need to be in the
hardware map. Make it optional to keep the file format compatible for
now.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Try and keep terminilogy consistent, a test application with multiple
test cases is now a 'test suite'.
Also fix accounting when retrying failed.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
If we provide arch on command line, do not check for all platforms of
architectures not specified.
Fixes#30099
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Show in debug mode what files are being written. In some cases this
takes a while and it is good to see what is going on.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
With python threading we have multiple issues with performance and
leakage. Use the python Process and implement locking using
multiprocessing library. This change improves performance and fixes
various issues with logging, concurrency and reliability of the output.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Running sanitycheck for multiple platforms but only request coverage
on a single platform fails, example:
sanitycheck --enable-coverage --coverage-platform nrf52840dk_nrf52840
-p nrf52840dk_nrf52840 -p nrf52dk_nrf52832 -T <sample>
This happens because `testcase_extra.conf` will only be created if the
current platform is included in the list of coverage platforms.
The error in the example above that would be seen is:
File not found:
<...>/sanitycheck/testcase_extra.conf
This commit now uses the existence of `testcase_extra.conf` which is
created in the method `create_overlay()` before appending
the conf file to the list of overlay files.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Get list of connected hardware and filter tests to only be run on
connected hardware in cases where no platform was specified when runn
the tests. If the platform does not exist in the hardware map, just
skip it.
Fixes#29166
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Generating JSON report format without changing
the existing report formats.
JSON report is divided into three sections
environment tests are run
test cases basic info and discarded tests
after applying filters
Added the complete debug log to the report
Fixes: #26443
Signed-off-by: Spoorthy Priya Yerabolu <spoorthy.priya.yerabolu@intel.com>
When running sanitycheck without any platform filters (-p option), we
take the default platforms and run on those only. When the -G options is
used for integration testing, the platforms listed as integration
platforms are not being added on top of default platforms.
This change adds integration platforms to the list an treats
integration_platforms as a positive filter, so that
integration_platforms are always being considered.
Fixes#29829
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
TSIM3 is a LEON processor and system simulator.
This commit allows using TSIM sanitycheck and the make run target.
TSIM parameters can be further specified in the board configuration
using TSIM and TSIM_SYS.
Signed-off-by: Martin Åberg <martin.aberg@gaisler.com>
Add missing infrastructure for simulation run on mdb-nsim.
When the nSIM via mdb simulation support was added to Zephyr
scripts the handler call was lost. So nSIM via mdb simulation
was broken from the beginning.
Fix that bu adding missing handler call fro mdb-nsim.
Note that we use BinaryHandler which call west flash. It's
required in case of nSIM via mdb as we create mdb.pid file with
mdb-nsim runner help.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
ARC nSIM boards (starting with nsim_ prefix) allow to run
tests in simulator (nSIM). However with proper west runner
the tests can be run on real HW (FPGA based).
So check device_testing flag firstly and rollback to nSIM
simulation run in case of device_testing is not set.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
mdb runner is quite special as it can be used to run Zephyr on
both simulator (nSIM) and real hardware.
However it is really misleading as same command (west flash)
will run Zephyr in simulation for one board and try to run it
on HW for another board. Things are getting worse for boards
supporting both runs in simulation and on real hardware.
Let's split mdb runner for mdb-hw (for runs on HW) and mdb-nsim
(for runs in simulation) runners.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
skips due to ram/rom overflow were captured as failures. Fix this and
count them correctly as skips.
Fixes#29412
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This commit adds posibility to save zephyr version in the xml reports.
It adds 'properties' attribute to testsuite attribute. 'properties'
can contain multiple 'property' attributes. Zephyr version is added
as such 'property'.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
If we get a build failure with the -M option we get the following:
TypeError: 'NoneType' object is not iterable
This was due to having the default value of additional_keep in
cleanup_artifacts being None instead of an empty array.
Fixes#29376
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
To make the artifacts we keep around for --device-testing -M re-usable
we need to sanitize any aboslute build paths that might exist in
CMakeCache.txt or zephyr/runners.yaml.
This allows us to build the tests to run on one CI system and run the
tests on another system that has the board(s) connected to it.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
If we specify --device-testing and -M make it so that the resulting
artifacts are usable to run with device-testing again. This means
keeping around the zephyr binary images (zephyr.{bin, hex, elf}) and
a few files so 'west' can function for flashing (CMakeCache.txt and
zephyr/runners.yaml).
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
If we run with --cmake-only we get the following:
`ERROR - Unknown status None`
Fix this by treating no status set as "passed" for the --cmake-only
case.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
The legacy macros were first deprecated in Zephyr v2.3. Now that
Zephyr v2.4 has been released, that makes two releases where these
macros have been deprecated, so it's OK to remove them.
This leaves support for legacy binding syntax in place. Removing that
is left to future work.
We need to update various pieces of documentation related to flash
partitions that never got updated when the new API was introduced.
Consolidate this information in the flash_map.h API reference page,
since that's really where users will run into it. This also gives us
the opportunity to improve this documentation.
Adjust a couple of kconfigfunctions.py and sanitycheck bits to use
non-legacy edtlib APIs.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This patch fixes the ZeroDivisionError which can
occur during percantage of test execution reporting.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
So far 'tests' and 'samples' directories were matched only when they
were first directories in path hierarchy. This doesn't work when running
sanitycheck from directory other than Zephyr source code root.
Match any directory in path hierarchy, similar how file is matched
currently. That way sanitycheck can be executed from outside of Zephyr
source code directory.
Signed-off-by: Marcin Niestroj <m.niestroj@grinn-global.com>
Use --filter=runnable for example to limit the tests being built to
those which actually can run on a device or a emulation platform.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
build_only was being used in different contexts and was confusing, so
clean this up and in some places just set if a test is runnable, a
testcase should always be buildable if the filters are matching.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Only create the dir and file if we have actual "content" that will
exist in testcase_extra.conf. This is to reduce a bit of
noise/footprint in the sanitycheck output dir.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
If we kill qemu because of a timeout, capture this as the reason instead
of reporting the exit code.
Fixes#28040
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
By default, subprocess.Popen commands are supplied as a list of strings.
Using split() allows to use command with arguments, for example it is
possible to use following:
sanitycheck ... \
--device-serial-pty="set_ambient read_terminal.py" \
...
Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
Fix the case where we have no ztest test_* testcases and reports were
missed. This now makes sure we do not report the parent testcase when
there are individual test_* results.
Fixes#27765
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
add argument --pre-script to specify a pre script. This will be
executed before device handler open serial port and invoke runner.
Currently, pre_script can only get from hardware.map file. just like:
./scripts/sanitycheck --hardware-map xxx.map
and set pre_script in xxx.map file.
Signed-off-by: Watson Zeng <zhiwei@synopsys.com>
In some cases it might be a good idea to reset the board for real to
make sure it is completely recovered from some failed state (simple
re-loading of the application binary or even Elf file contents doesn't
affect most of internal CPU states so doesn't help in recovery,
see https://github.com/zephyrproject-rtos/zephyr/issues/25022 &
https://github.com/zephyrproject-rtos/zephyr/issues/26665).
And so we may want to utilize some external utility which triggers the
hard reset (in case of ARC boards it is
https://github.com/foss-for-synopsys-dwc-arc-processors/rff-ftdi-reset).
So we need to have a way to execute an external command before each and
every test.
Now given we already have quite some call-backs we try to use them
before re-inventing the wheel. And pre_script seem to be a good option
with just on minor note - it is called after serial port gets open.
And while in some cases it might be OK if serial port on the board is
not affected by the board's reset, if it is affected we'll be losing
connection on reset (and that's the case with ARC boards BTW as the
FTDI USB-to-Serial IC is also wired to the reset signal on most of
the boards). That said we just move invocation of pre_script before
opening the serial port and everything should be good now.
Signed-off-by: Watson Zeng <zhiwei@synopsys.com>
By default show reports based on last release. Fix a few other issues
where we had 0 values and were dividing by zero.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This will only build/run on emulation platforms.
The decision is made based on the value of the 'simulation' key in the
platform yaml file.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Adds some code to allow QEMU to crash without failing the test.
This is required for testing coredump code as it will certainly
cause QEMU to crash.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
There is a potential that QEMUHandler.handle() returns before
the testing harness finishes processing all the output from QEMU.
Simply wait for the harness thread to finish before returning.
Also, fix the return code in the debug message as it should be
the return code from Popen().
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Commit 8d7bb8ffd8 refactored
device structures which changed some of the linker sections:
"devconfig" was removed, and "devices" was added. However,
the list in sanitycheck's size calculator was not updated,
which results in sanitycheck complaining about unrecognized
sections when doing footprint.
Also, a few sections have been renamed (with added suffix
"_area") due to introduction of Z_ITERABLE_SECTION_RAM/ROM
macros. There are also some missing section names.
Fixes the issue by adding the missing sections names, and
updating existing ones.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This patch should improve couting and reporting of the number
of passed/skipped tests.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>