Add named exceptions instead of a generic one. Should help point to the
right issue when something goes wrong.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We used the column 'passed' as a boolean to signify pass or fail,
however we do have other states that need to be tracked.
Remove the boolean and use a text field instead that has the status as a
string.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When running 1 specific test, counting was off. Combine functions adding
tests into one and optimize filtering.
Fixes#22270
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
add lstrip function to delete the '\r' char in front of the
serial output from serail readline, because this '\r'
will influence the result match function in harness.py.
Signed-off-by: peng1 chen <peng1.chen@intel.com>
This commit updates the section names for memory size calculation in
the sanitycheck script as follows:
1. Remove `_TEXT_SECTION_NAME_2` section, which no longer exists, from
the `rw_sections`.
2. Add `rom_start` section, which mostly contains read-only data such
as the exception vector table, to the `ro_sections`.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
We only need pyserial python module if we are doing device testing.
Treat it similar to how we treat tabulate module.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
The SDK has a new path for x86 gcov, point to the new binary. Also, do
not clean artifacts when running coverage.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Since we do interger division to determine how many items are in each
set, its possible we can have bad rounding error and the last set having
a much larger amount of items compared to all other sets.
For example, total = 3740, sets = 300:
per_set = 3740 / 300 = 12.466 = 12
last_set = 3740 - 299 * 12 = 152
So instead of doing simple division, we add on extra item to the early
sets and we've exhausted the rounding error:
num_extra_sets = total - per_set * sets
140 = 3740 - 12 * 300
So the first 140 subsets will have 13 per_set, and the last 160 will
have 12 per_set.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
Overlay files were concatenated into single string with no whitespace
between them. If '--extra-args OVERLAY_CONFIG="overlay-1.conf"
--extra-args OVERLAY_CONFIG="overlay-2.conf"' was used, then the result
was OVERLAY_CONFIG="overlay-1.confoverlay-2.conf".
Another thing was that overlay extra args were not properly removed from
the list of regular arguments. As a result we had incorrect list of
overlays and incorrect list of other arguments.
Rework code to extract overlays in loop in a safe manner. Use for that a
list of strings instead of string directly. Join those strings and form
a single OVERLAY_CONFIG argument just before running cmake.
Tested with following testcase.yaml line:
extra_args: ARG1 OVERLAY_CONFIG="overlay-1.conf"
ARG2 OVERLAY_CONFIG="overlay-2.conf"
Before this patch we got:
args = ['OVERLAY_CONFIG="overlay-1.conf"',
'OVERLAY_CONFIG="overlay-2.conf"']
After this patch we get:
args = ['ARG1', 'ARG2',
'OVERLAY_CONFIG="overlay-1.conf overlay-2.conf"']
While at it, fix also regex pattern by removing requirement of double
quotes around value. Match any option value instead and strip both
single and double quotes when match is positive. Tested with:
$ ./scripts/sanitycheck -T samples/hello_world/ -p qemu_x86 \
--extra-args OVERLAY_CONFIG=overlay1.conf '
--extra-args OVERLAY_CONFIG=\"overlay2.conf\" '
--extra-args OVERLAY_CONFIG=\'overlay3.conf\'
Signed-off-by: Marcin Niestroj <m.niestroj@grinn-global.com>
Only dump data when we are interested in the analysing coverage. By
default just collect the data.
CONFIG_COVERAGE_DUMP is used to control this behaviour.
This will help speed up sanitycheck and will avoid lots of noise in the
log when some tests with coverage enabled failed. Dumping data to
console is also suspected to be one of the reason why qemu hangs in CI.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The terminal configuration should not be reset on exit if
sanitycheck's output is being directed to something that mediates the
terminal control, such as a pager. Only do the reset if stdout is
interactive.
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
Sort by testcase name and path and not by platforms. This way we do not
have the same platform executing on CI system causing failures to
overload.
Right now we have almost all qemu targets executing on 1 or 2 nodes,
causing those systems to be overloaded. Instead of taking the default
sorting by platforms, we now sort by testcases, so the distribution is
more "random", but still reproducible.
Remove the ordering function that would put native_posix in the first
set, it is not being used anymore.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Outer if is checking if the variable is in a list with two options
which means that we can just do an if / else and remove an assignment.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Timeouts were not reported correctly in the XML file as failures,
causing some confusion in the shippable results.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
It is often needed to run sanitycheck and exclude a specific platform or
multiple platforms. Add an option to provide this using
--exclude-platform. The option can be used multiple times to exclude
multiple platforms at the same time.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Using --runtime-artifact-cleanup, the script will now remove all
artifacts of passing tests. This is useful for running sanitycheck on
systems with little disk space.
Fixes#21922
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Show how many tests passed out of those that were actually
performed, not including skipped tests (which we also report
a count of)
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
stderr from the binary handler (native_posix for example) was redirected
to the logger as errors, this is confusing the console and users, so
remove this.
Fixes#21784
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Make sure we close the pipe file handles after we are done, otherwise we
will end up with too many open file descriptors and crash...
Fixes#20974Fixes#21637
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Makes it much easier to inspect them in a different terminal
from where the test was run. These paths tend to be long
anyway, even if relative.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Sanitycheck used to delete the output directory if -n
wasn't used, but now it renamed it. Add a new flag to
enable the old behavior.
Do not use logging for these early messages, logger
hasn't been set up yet.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
The code has a lot of sys.exit() calls, but internally these
just raise a special exception. Add a try...finally block
to ensure 'stty sane' is run before leaving the script.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
This is needed when a board needs to be reset using an external commands
or tools that are not part of the flash command. For example, power
reset or by poking a GPIO header on the board using external wiring.
add post_flash/pre_flash to the platform section in the hardware map.
For example:
- available: true
connected: true
id: OSHW000032254e4500128002ab98002784d1000097969900
platform: reel_board
post_script: /tmp/post_flash.sh
pre_script: /tmp/pre_flash.sh
product: DAPLink CMSIS-DAP
runner: pyocd
serial: /dev/ttyACM11
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add a static method for dumping hardware map and reuse it across the
script reducing duplicated code.
Fixes#21475
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Added support of the Microchip with FTDI serial devices to be able
create a hardware map for them. In the future that hardware map
for the Microchip board will let run automatic Sanitycheck tests on it.
Signed-off-by: Maksim Masalski <maksim.masalski@intel.com>
timeouts were not reported correctly and we were getting the default
'N/A' in case of a timeout.
Fixes#21438
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When testing with one device and without a hardware map, make sure we
load the temporary map into the suite.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
More changes moving away from using global options and instead using
class variables and parameters.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
