Commit Graph

355 Commits

Author SHA1 Message Date
Anas Nashif
f68146f74e sanitycheck: add option to treat overflows as errors
Add a new option that will enable treating RAM/SRAM overflows as errors
instead of skipping them, which is the default.

Fixes #27583

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-12-01 15:36:19 -05:00
Anas Nashif
8305d1b6ef sanitycheck: fix hardware map defaults
We were setting all devices as connected, now do that based on serial
availability.
Rename internal variables and make them shorter:

connected_hardware -> duts
ConnectedDevice = DUT

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-12-01 15:36:19 -05:00
Anas Nashif
ad70a69f85 sanitycheck: cleanup hardware map handling
Shorten class method names and print hardware map as before after
scanning.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-12-01 15:36:19 -05:00
Anas Nashif
d8e4a86788 sanitycheck: schema: 'available' is a runtime key
The available key is a runtime variable, it does not need to be in the
hardware map. Make it optional to keep the file format compatible for
now.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-12-01 15:36:19 -05:00
Anas Nashif
3b939da42c sanitycheck: unify terminology and fix accounting
Try and keep terminilogy consistent, a test application with multiple
test cases is now a 'test suite'.

Also fix accounting when retrying failed.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-12-01 15:36:19 -05:00
Anas Nashif
657b1ac354 sanitycheck: fix arch filtering
If we provide arch on command line, do not check for all platforms of
architectures not specified.

Fixes #30099

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-12-01 15:36:19 -05:00
Anas Nashif
f86bc058ce sanitycheck: reduce verbosity
Reduce debug messages while scanning directories for tests.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-12-01 15:36:19 -05:00
Anas Nashif
f909e99f04 sanitycheck: add some debugging during report phase
Show in debug mode what files are being written. In some cases this
takes a while and it is good to see what is going on.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-12-01 15:36:19 -05:00
Anas Nashif
696476e24e sanitycheck: report test name if no status
Also catch cases where we try to test on a device when no device is
connected.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-12-01 15:36:19 -05:00
Anas Nashif
bb42795644 sanitycheck: do not report about execution if --build-only
When running with --build-only, do not print messages about tests being
executed.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-12-01 15:36:19 -05:00
Anas Nashif
531fe89e80 sanitycheck: use multiprcoessing instead of threads
With python threading we have multiple issues with performance and
leakage. Use the python Process and implement locking using
multiprocessing library. This change improves performance and fixes
various issues with logging, concurrency and reliability of the output.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-12-01 15:36:19 -05:00
Anas Nashif
fe07d57e80 sanitycheck: error out if we do not find any tests
If we find no tests, error out instead of exiting silently.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-12-01 15:36:19 -05:00
Anas Nashif
743594faae sanitycheck: print pid of qemu process when debugging
For debugging, print the PID of the qemu running.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-12-01 15:36:19 -05:00
Torsten Rasmussen
ca08cc0d02 sanitycheck: check existence of testcase_extra.conf
Running sanitycheck for multiple platforms but only request coverage
on a single platform fails, example:
  sanitycheck --enable-coverage --coverage-platform nrf52840dk_nrf52840
  -p nrf52840dk_nrf52840 -p nrf52dk_nrf52832 -T <sample>

This happens because `testcase_extra.conf` will only be created if the
current platform is included in the list of coverage platforms.

The error in the example above that would be seen is:
  File not found:
      <...>/sanitycheck/testcase_extra.conf
This commit now uses the existence of `testcase_extra.conf` which is
created in the method `create_overlay()` before appending
the conf file to the list of overlay files.

Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
2020-11-28 07:19:15 -05:00
Anas Nashif
82a6a467ec sanitycheck: do not try platforms not in hardware map
Get list of connected hardware and filter tests to only be run on
connected hardware in cases where no platform was specified when runn
the tests.  If the platform does not exist in the hardware map, just
skip it.

Fixes #29166

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-11-27 09:29:59 -05:00
Spoorthy Priya Yerabolu
f8f220a9aa scripts: sanitycheck: Generate json report format
Generating JSON report format without changing
the existing report formats.
JSON report is divided into three sections
environment tests are run
test cases basic info and discarded tests
after applying filters
Added the complete debug log to the report

Fixes: #26443

Signed-off-by: Spoorthy Priya Yerabolu <spoorthy.priya.yerabolu@intel.com>
2020-11-20 11:44:29 -05:00
Anas Nashif
87e3ec4417 sanitycheck: test on integration platforms on top of default platforms
When running sanitycheck without any platform filters (-p option), we
take the default platforms and run on those only. When the -G options is
used for integration testing, the platforms listed as integration
platforms are not being added on top of default platforms.

This change adds integration platforms to the list an treats
integration_platforms as a positive filter, so that
integration_platforms are always being considered.

Fixes #29829

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-11-17 11:31:39 +01:00
Martin Åberg
c107714214 sanitycheck: Add TSIM3 support
TSIM3 is a LEON processor and system simulator.

This commit allows using TSIM sanitycheck and the make run target.

TSIM parameters can be further specified in the board configuration
using TSIM and TSIM_SYS.

Signed-off-by: Martin Åberg <martin.aberg@gaisler.com>
2020-11-13 14:53:55 -08:00
Eugeniy Paltsev
16032b67e8 sanitycheck: ARC: mdb-nsim: add missing stuff for simulation run
Add missing infrastructure for simulation run on mdb-nsim.
When the nSIM via mdb simulation support was added to Zephyr
scripts the handler call was lost. So nSIM via mdb simulation
was broken from the beginning.

Fix that bu adding missing handler call fro mdb-nsim.
Note that we use BinaryHandler which call west flash. It's
required in case of nSIM via mdb as we create mdb.pid file with
mdb-nsim runner help.

Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
2020-11-11 08:18:38 -05:00
Eugeniy Paltsev
9f2006c48b sanitycheck: ARC: nsim: allow to run tests on HW as well
ARC nSIM boards (starting with nsim_ prefix) allow to run
tests in simulator (nSIM). However with proper west runner
the tests can be run on real HW (FPGA based).

So check device_testing flag firstly and rollback to nSIM
simulation run in case of device_testing is not set.

Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
2020-11-11 08:18:38 -05:00
Eugeniy Paltsev
91d7ec5a35 ARC: west: split mdb runner for mdb-hw & mdb-nsim runners
mdb runner is quite special as it can be used to run Zephyr on
both simulator (nSIM) and real hardware.
However it is really misleading as same command (west flash)
will run Zephyr in simulation for one board and try to run it
on HW for another board. Things are getting worse for boards
supporting both runs in simulation and on real hardware.

Let's split mdb runner for mdb-hw (for runs on HW) and mdb-nsim
(for runs in simulation) runners.

Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
2020-11-11 08:18:38 -05:00
Anas Nashif
be1025a736 sanitycheck: handle overflow skips
skips due to ram/rom overflow were captured as failures. Fix this and
count them correctly as skips.

Fixes #29412

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-11-02 14:21:55 -05:00
Anas Nashif
5a6e64f448 sanitycheck: count skipped tests due to overflow
Count tests that were skipped due to SRAM/RAM overdlow as skipped.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-11-02 14:21:55 -05:00
Maciej Perkowski
060e00d9c5 sanitycheck: Add checking of zephyr version
Adds function checking the version of zephyr, used for reporting

Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
2020-10-26 14:19:34 -04:00
Maciej Perkowski
725d19b79e sanitycheck: Add info about zephyr version to the xml reports
This commit adds posibility to save zephyr version in the xml reports.
It adds 'properties' attribute to testsuite attribute. 'properties'
can contain multiple 'property' attributes. Zephyr version is added
as such 'property'.

Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
2020-10-26 14:19:34 -04:00
Kumar Gala
eb2e89aecf sanitycheck: Fix bug with -M option
If we get a build failure with the -M option we get the following:

TypeError: 'NoneType' object is not iterable

This was due to having the default value of additional_keep in
cleanup_artifacts being None instead of an empty array.

Fixes #29376

Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
2020-10-20 19:30:43 -05:00
Kumar Gala
b2c07e439e sanitycheck: sanitize paths for device testing artifacts
To make the artifacts we keep around for --device-testing -M re-usable
we need to sanitize any aboslute build paths that might exist in
CMakeCache.txt or zephyr/runners.yaml.

This allows us to build the tests to run on one CI system and run the
tests on another system that has the board(s) connected to it.

Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
2020-10-20 07:24:30 -05:00
Kumar Gala
1285c1f80a sanitycheck: make artifact cleanup for device-testing usable
If we specify --device-testing and -M make it so that the resulting
artifacts are usable to run with device-testing again.  This means
keeping around the zephyr binary images (zephyr.{bin, hex, elf}) and
a few files so 'west' can function for flashing (CMakeCache.txt and
zephyr/runners.yaml).

Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
2020-10-20 07:24:30 -05:00
Kumar Gala
659b24bdcf sanitycheck: Fix results for --cmake-only
If we run with --cmake-only we get the following:

`ERROR   - Unknown status None`

Fix this by treating no status set as "passed" for the --cmake-only
case.

Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
2020-10-13 13:13:21 +02:00
Martí Bolívar
8165008f44 dts: remove legacy macro support
The legacy macros were first deprecated in Zephyr v2.3. Now that
Zephyr v2.4 has been released, that makes two releases where these
macros have been deprecated, so it's OK to remove them.

This leaves support for legacy binding syntax in place. Removing that
is left to future work.

We need to update various pieces of documentation related to flash
partitions that never got updated when the new API was introduced.
Consolidate this information in the flash_map.h API reference page,
since that's really where users will run into it. This also gives us
the opportunity to improve this documentation.

Adjust a couple of kconfigfunctions.py and sanitycheck bits to use
non-legacy edtlib APIs.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-10-09 08:45:38 -05:00
Maciej Perkowski
2ec5ec2153 sanitycheck: Fix ZeroDivisionError during reporting
This patch fixes the ZeroDivisionError which can
occur during percantage of test execution reporting.

Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
2020-10-02 07:56:17 -04:00
Marcin Niestroj
fc67409291 sanitycheck: ignore coverage directories by matching any in path hierarchy
So far 'tests' and 'samples' directories were matched only when they
were first directories in path hierarchy. This doesn't work when running
sanitycheck from directory other than Zephyr source code root.

Match any directory in path hierarchy, similar how file is matched
currently. That way sanitycheck can be executed from outside of Zephyr
source code directory.

Signed-off-by: Marcin Niestroj <m.niestroj@grinn-global.com>
2020-10-01 17:11:34 -04:00
Anas Nashif
405f1b62f3 sanitycheck: support --filter to limit number of tests built
Use --filter=runnable for example to limit the tests being built to
those which actually can run on a device or a emulation platform.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-10-01 07:28:01 -05:00
Anas Nashif
4ca0b95956 sanitycheck: build_only ambiguity cleanup
build_only was being used in different contexts and was confusing, so
clean this up and in some places just set if a test is runnable, a
testcase should always be buildable if the filters are matching.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-10-01 07:28:01 -05:00
Kumar Gala
51d6931b16 sanitycheck: Only create testcase_extra.conf if needed
Only create the dir and file if we have actual "content" that will
exist in testcase_extra.conf.  This is to reduce a bit of
noise/footprint in the sanitycheck output dir.

Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
2020-09-28 12:10:34 +02:00
Anas Nashif
c1c10993c7 sanitycheck: comment cleanup and style fixes
Minor comment and style fixes.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-09-18 13:24:36 -05:00
Anas Nashif
fdc02b651c sanitycheck: capture timeout as reason in cases we kill qemu
If we kill qemu because of a timeout, capture this as the reason instead
of reporting the exit code.

Fixes #28040

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-09-18 13:24:36 -05:00
Anas Nashif
61c4a511ac sanitycheck: error on duplicate board identifier
Error when we have duplicate identifiers in board definition files.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-09-15 13:17:08 -04:00
Andrei Emeltchenko
9f7a90332b sanitycheck: Allow to pass command with argument to ser_pty
By default, subprocess.Popen commands are supplied as a list of strings.
Using split() allows to use command with arguments, for example it is
possible to use following:

sanitycheck ... \
--device-serial-pty="set_ambient read_terminal.py" \
...

Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
2020-09-04 07:55:52 -04:00
Anas Nashif
0f831b1e15 sanitycheck: mark arc simulation smp systems as mdb
This changes back smp simulation systems to require mdb.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-09-03 15:10:07 -07:00
Anas Nashif
8b425fa7d8 sanitycheck: do not report parent test
Fix the case where we have no ztest test_* testcases and reports were
missed. This now makes sure we do not report the parent testcase when
there are individual test_* results.

Fixes #27765

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-09-01 15:02:22 -04:00
Watson Zeng
0079cecc19 sanitycheck: add argument --pre-script
add argument --pre-script to specify a pre script. This will be
executed before device handler open serial port and invoke runner.

Currently, pre_script can only get from hardware.map file. just like:
./scripts/sanitycheck  --hardware-map xxx.map
and set pre_script in xxx.map file.

Signed-off-by: Watson Zeng <zhiwei@synopsys.com>
2020-08-31 15:54:32 +02:00
Watson Zeng
3b43d946fd sanitycheck: run custom pre script before open the serial port
In some cases it might be a good idea to reset the board for real to
make sure it is completely recovered from some failed state (simple
re-loading of the application binary or even Elf file contents doesn't
affect most of internal CPU states so doesn't help in recovery,
see https://github.com/zephyrproject-rtos/zephyr/issues/25022 &
https://github.com/zephyrproject-rtos/zephyr/issues/26665).
And so we may want to utilize some external utility which triggers the
hard reset (in case of ARC boards it is
https://github.com/foss-for-synopsys-dwc-arc-processors/rff-ftdi-reset).
So we need to have a way to execute an external command before each and
every test.

Now given we already have quite some call-backs we try to use them
before re-inventing the wheel. And pre_script seem to be a good option
with just on minor note - it is called after serial port gets open.
And while in some cases it might be OK if serial port on the board is
not affected by the board's reset, if it is affected we'll be losing
connection on reset (and that's the case with ARC boards BTW as the
FTDI USB-to-Serial IC is also wired to the reset signal on most of
the boards). That said we just move invocation of pre_script before
opening the serial port and everything should be good now.

Signed-off-by: Watson Zeng <zhiwei@synopsys.com>
2020-08-31 15:54:32 +02:00
Anas Nashif
ad44bed754 sanitycheck: fix footprint reports
By default show reports based on last release. Fix a few other issues
where we had 0 values and were dividing by zero.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-08-31 09:20:52 -04:00
Anas Nashif
9eb9c4cce9 sanitycheck: add option --emulation-only
This will only build/run on emulation platforms.
The decision is made based on the value of the 'simulation' key in the
platform yaml file.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-08-28 14:22:07 -07:00
Anas Nashif
dca317c730 sanitycheck: inclusive language
change whitelist -> allow.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-08-27 07:04:07 -04:00
Daniel Leung
faae15d838 sanitycheck: allow QEMU to crash without failing
Adds some code to allow QEMU to crash without failing the test.
This is required for testing coredump code as it will certainly
cause QEMU to crash.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2020-08-24 20:28:24 -04:00
Daniel Leung
5b1b4a3755 sanitycheck: need to wait for harness to process QEMU outputs
There is a potential that QEMUHandler.handle() returns before
the testing harness finishes processing all the output from QEMU.
Simply wait for the harness thread to finish before returning.

Also, fix the return code in the debug message as it should be
the return code from Popen().

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2020-08-24 20:28:24 -04:00
Daniel Leung
203556cd8c sanitycheck: update section names in size calculator
Commit 8d7bb8ffd8 refactored
device structures which changed some of the linker sections:
"devconfig" was removed, and "devices" was added. However,
the list in sanitycheck's size calculator was not updated,
which results in sanitycheck complaining about unrecognized
sections when doing footprint.

Also, a few sections have been renamed (with added suffix
"_area") due to introduction of Z_ITERABLE_SECTION_RAM/ROM
macros. There are also some missing section names.

Fixes the issue by adding the missing sections names, and
updating existing ones.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2020-08-24 20:18:06 -04:00
Maciej Perkowski
c67a0cd098 sanitycheck: Improve tests counting
This patch should improve couting and reporting of the number
of passed/skipped tests.

Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
2020-08-19 12:49:42 -04:00