add argument --pre-script to specify a pre script. This will be
executed before device handler open serial port and invoke runner.
Currently, pre_script can only get from hardware.map file. just like:
./scripts/sanitycheck --hardware-map xxx.map
and set pre_script in xxx.map file.
Signed-off-by: Watson Zeng <zhiwei@synopsys.com>
In some cases it might be a good idea to reset the board for real to
make sure it is completely recovered from some failed state (simple
re-loading of the application binary or even Elf file contents doesn't
affect most of internal CPU states so doesn't help in recovery,
see https://github.com/zephyrproject-rtos/zephyr/issues/25022 &
https://github.com/zephyrproject-rtos/zephyr/issues/26665).
And so we may want to utilize some external utility which triggers the
hard reset (in case of ARC boards it is
https://github.com/foss-for-synopsys-dwc-arc-processors/rff-ftdi-reset).
So we need to have a way to execute an external command before each and
every test.
Now given we already have quite some call-backs we try to use them
before re-inventing the wheel. And pre_script seem to be a good option
with just on minor note - it is called after serial port gets open.
And while in some cases it might be OK if serial port on the board is
not affected by the board's reset, if it is affected we'll be losing
connection on reset (and that's the case with ARC boards BTW as the
FTDI USB-to-Serial IC is also wired to the reset signal on most of
the boards). That said we just move invocation of pre_script before
opening the serial port and everything should be good now.
Signed-off-by: Watson Zeng <zhiwei@synopsys.com>
By default show reports based on last release. Fix a few other issues
where we had 0 values and were dividing by zero.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This will only build/run on emulation platforms.
The decision is made based on the value of the 'simulation' key in the
platform yaml file.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Adds some code to allow QEMU to crash without failing the test.
This is required for testing coredump code as it will certainly
cause QEMU to crash.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
There is a potential that QEMUHandler.handle() returns before
the testing harness finishes processing all the output from QEMU.
Simply wait for the harness thread to finish before returning.
Also, fix the return code in the debug message as it should be
the return code from Popen().
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Commit 8d7bb8ffd8 refactored
device structures which changed some of the linker sections:
"devconfig" was removed, and "devices" was added. However,
the list in sanitycheck's size calculator was not updated,
which results in sanitycheck complaining about unrecognized
sections when doing footprint.
Also, a few sections have been renamed (with added suffix
"_area") due to introduction of Z_ITERABLE_SECTION_RAM/ROM
macros. There are also some missing section names.
Fixes the issue by adding the missing sections names, and
updating existing ones.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This patch should improve couting and reporting of the number
of passed/skipped tests.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
In the junit output the attribute containing the number of skipped tests
must be named "skipped" instead of "skip".
See e.g. https://github.com/junit-team/junit5/blob/main/platform-tests/
src/test/resources/jenkins-junit.xsd#L95
Signed-off-by: Christian Taedcke <christian.taedcke@lemonbeat.com>
Currently '--coverage-tool gcovr' results in using lcov. Fix that to use
gcovr, as requested by user.
Fixes: f6462a3a8c ("sanitycheck: get rid of global VERBOSE")
Signed-off-by: Marcin Niestroj <m.niestroj@grinn-global.com>
Sanitycheck discards test instances if a set of given coditions
is not fulfilled. This leads to empty test results for these
instances. This can introduce ambiguity with results that are
missing due to some bugs in the framework. This commit fills
the results for skipped tests with 'skipped' states and provides
the reason for filtering them out in the msg field. The commit
also fixes the way sanitycheck counts and reports tests/test cases
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
When passing a using --hardware-map, sanitycheck would ignore extra
args passed via --west-flash, such as --west-flash="--erase".
This is because the command with the extra args was overwritten by the
command with the runner info from the hardware map.
This patch merges those code paths so they are aware of each other and
of --west-runner.
Signed-off-by: Øyvind Rønningstad <oyvind.ronningstad@nordicsemi.no>
This commit limits the removal of `test_` from test case name
to only the first occurance. There are test cases with `test_`
also in the middle of their names and removing it couses mismatches
between extracted testcases and the names obtained when passing
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
We have been doing an AND comparison instead of an OR. AND does exclude
way to many testcases where multiple tags are being used.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Use --integration testing for CI to limit execution and builds on
platforms that actually provide most of the coverage instead of blindly
building/running on all available platforms.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
instances that do not run will have no results (beside the fact they
built successfully), so log those as such.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This is a cargo-culted attempt to make an error observed in poorly
understood CI circumstances go away when loading edt.pickle, by making
edtlib visible in sys.modules before loading the pickle file.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Consolidate creation of edtlib.EDT objects from a build directory's
devicetree into one place by loading it from build/zephyr/edt.pickle
everywhere. A previous commit creates edt.pickle from gen_defines.py.
In addition to probably speeding things up slightly by not reparsing
the devicetree, the main benefit of this approach is creating a single
point of truth for the bindings directories and warnings
configuration, meaning we don't have to worry about them getting out
of sync while being passed around between devicetree creation and
usage time.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
serial-pty is an extension that is not supported in the hardware map, so
we need to work around the fact that it is not part of the map when
dealing with normal serial.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add option to provide to sanitycheck argument for creating
pseudoterminal. Used with hardware without serial console connected.
A pseudoterminal is used to make a sanitycheck believe that it
interacts with a terminal although it actually interacts with the
script.
E.g "sanitycheck --device-testing --device-serial-pty <script>"
Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
Support running/building only specific tags and ignoring everything
else.
Adding this to a platform yaml file will enable tests with one of of
tags listed to be executed on the platform.
This is useful for special platform configurations used for testing
specific features for example.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
--warnings-as-errors was basically doing nothing, it is the default.
Replace this with an option to disable erroring on warning:
-W, --disable-warnings-as-errors
Fixes#26910
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We still have cases where qemu crashes are not being correctly captured
in the on-screen summary, however they are captured in the final
generated report.
Fixes#26679
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
restore how --only-failed works by allow rebuilds in case of build
errors, however, do not rebuild when --retry-failed is used, which is a
CI usecase and nothing would change in the second iteration.
Fixes#26685
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
According to our compliance checks, we have 'useless-suppression'
and 'unused-import' on our hands:
W0611: Unused CLoader imported from yaml as Loader
W0611: Unused Loader imported from yaml
I0021: Useless suppression of 'unsubscriptable-object'
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Use the C LibYAML parser if available, rather than the Python parser. It
is much faster.
This is a clean and rebased version of PR #20210 by
Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Co-authored-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Deal with qemu crashes and do not register pass status as the default.
In case we do not have a status, report unknown.
This now captures different states from qemu that were not captured
before.
Fixes#26679
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Recent ST boards embed the new ST-Link probe V3.
It is advertised as "STLINK-V3", update sanitylmib to take it
into account.
In handle function, it is proposed to treat it separately as current
"STM32 STLink" as processing might differ in next future (hla_serial
deprecation).
Signed-off-by: Erwan Gouriou <erwan.gouriou@linaro.org>
Some devices have multiple serial ports, which show as duplicates in the
hardware map. This doesn't work well when using --generate-hardware-map
to regenerate an existing map. E.g. nrf5340pdk_nrf5340_cpuapp by default
prints to the 3rd of 3 devices. If it shows up on /dev/ttyACM0,
/dev/ttyACM1, and /dev/ttyACM2, printout will come on /dev/ttyACM2.
nrf9160dk_nrf9160 also provides 3 devices, but prints to the 1st.
This patch sorts the devices by the serial port and matches multiple
duplicate entries one-to-one to retain the ordering. This way, the
correct device can be given the platform name and the others can be kept
as "unknown" so that --device-testing understands correctly
1) that there is only one device (not 3)
2) where the serial output will come.
Signed-off-by: Øyvind Rønningstad <oyvind.ronningstad@nordicsemi.no>
Modifying discard_report function to fix the check for
self.discards dictionary in try block. self.discards
is None is dead code which didnot check if the dictionary was
empty. Changed it to if not self.discards.
Signed-off-by: Aastha Grover <aastha.grover@intel.com>
- Report build errors as errors, not test failures
- Do not try and build/run tests with build failures
- Fix issue with empty reports when running --only-failed
- Report build errors in the detailed and target reports
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
While testing the warnings in scan_file() format is not the same
everytime we run tests
Added sorted with reverse flag so that warnings are same for
every run
Signed-off-by: Spoorthy Priya Yerabolu <spoorthy.priya.yerabolu@intel.com>
If we pass --west-flash with no additional argument that was getting
treated the same as if it wasn't passed at all.
Change the check to be based on 'None' to handle the bare --west-flash
case properly.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
classname in the junit report was not being used correctly since the
beginning. We had the targetname in the classname and put all results in
1 single testsuite.
The new output add 1 single testsuite for each platform and simplifies
the classname so that tests of the same classname can be grouped in
output, producing readable reports.
The current output can be seen here:
https://testing.zephyrproject.org/daily_tests/zephyr-v2.2.0-2533-gdbbf834605/report/index.html
This should much better once we have this change in.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In some cases we were not handling return code correctly and evaluating
timeouts as a pass. Report failure correctly.
Fixes#26065
Signed-off-by: Wentong Wu <wentong.wu@intel.com>
For iterable areas defined with Z_STRUCT_SECTION_ITERABLE(),
the corresponding output section in the linker script is just
boilerplate. Add macros to make these definitions simpler.
Unfortunately, we have a fair number of iterable sections not
defined with Z_STRUCT_SECTION_ITERABLE(), this patch does not
address this.
The output sections are all named <struct name>_area, update
sanitylib.py with this.
sys_sem with no userspace, and k_lifo/k_fifo are special cases
where different data types that are all equivalent need to be
put in the same iterable area. Add
Z_STRUCT_SECTION_ITERABLE_ALTERNATE() for this special case.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Prior to addition of schema validation running sanitycheck
--generate-hw-map would preserve the value of any field in an existing
map that it didn't update. These fields now cause a parsing error.
Add notes as an optional key where information relevant to the board
can be added to provide useful context (such as which serial port
corresponds to the console, or why a non-standard runner is required).
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
probe_id is used as an optional key that overrides id as the source
of board_id when configured to use a runner that is not the default.
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
If a test instance from .yaml file was skipped due to being on
a filtered list then there are no entries about it in the
sanitycheck_report.xml for a given platform. This commit fills
instance.results with skipped test cases also for skipped instances so
those test cases are visible when the full report is generated. Tested
for ./tests/kernel/mem_protect/userspace on nrf52dk_nrf52832 where
userspace.gap_filling is filtered out.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
It is now possible to add a list of fixture a platform supports which is
matched to testcases requesting fixtures to be avaiable to be able to
run.
For example:
- available: true
connected: true
id: 0240000026334e450015400f5e0e000b4eb1000097969900
platform: frdm_k64f
product: DAPLink CMSIS-DAP
runner: pyocd
serial: /dev/ttyACM9
fixtures:
- gpio_loopback
Fixes#24943
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
