Use the pytest test framework in the dtlib.py and edtlib.py test
suites (testdtlib.py and testedtlib.py respectively).
The goal here is not to change what is being tested. The existing test
suite is excellent and very thorough.
However, it is made up of executable scripts where all of the tests
are run using a hand-rolled framework in a single function per file.
This is a bit all-or-nothing and prevents various nice features
available in the de-facto standard pytest test framework from being
used.
In particular, pytest can:
- drop into a debugger (pdb) when there is a problem
- accept a pattern which specifies a subset of tests to run
- print very detailed error messages about the actual and expected
results in various traceback formats from brief to very verbose
- gather coverage data for the python scripts being tested (via plugin)
- run tests in parallel (via plugin)
- It's easy in pytest to run tests with temporary directories
using the tmp_path and other fixtures. This us avoid
temporarily dirtying the working tree as is done now.
Moving to pytest lets us leverage all of these things without any loss
in ease of use (in fact, some things are nicer in pytest):
- Any function that starts with "test_" is automatically picked up and
run. No need for rolling up lists of functions into a test suite.
- Tests are written using ordinary Python 'assert'
statements.
- Pytest magic unpacks the AST of failed asserts to print details on
what went wrong in really nice ways. For example, it will show you
exactly what parts of two strings that are expected to be equal
differ.
For the most part, this is a pretty mechanical conversion:
- extract helpers and test cases into separate functions
- insert temporary paths and adjust tests accordingly to not match
file names exactly
- use 'assert CONDITION' instead of 'if not CONDITION: fail()'
There are a few cases where making this happen required slightly
larger changes than that, but they are limited.
Move the checks from check_compliance.py to a new GitHub workflow,
removing hacks that are no longer needed.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
So far 'tests' and 'samples' directories were matched only when they
were first directories in path hierarchy. This doesn't work when running
sanitycheck from directory other than Zephyr source code root.
Match any directory in path hierarchy, similar how file is matched
currently. That way sanitycheck can be executed from outside of Zephyr
source code directory.
Signed-off-by: Marcin Niestroj <m.niestroj@grinn-global.com>
Add '--coverage-basedir BASEDIR' argument, which can be used to specify
source code base directory other than default Zephyr root directory.
This is mainly useful for projects built on top of Zephyr, where
sanitycheck is used for unit testing application code.
Signed-off-by: Marcin Niestroj <m.niestroj@grinn-global.com>
Use --filter=runnable for example to limit the tests being built to
those which actually can run on a device or a emulation platform.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
build_only was being used in different contexts and was confusing, so
clean this up and in some places just set if a test is runnable, a
testcase should always be buildable if the filters are matching.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Inspect the hex file with intelhex, and fail if the hex file has any
contents in the UICR area(s).
family == 'NRF52' still always does --sectoranduicrerase, but this
option is not available on other families.
Add --force command line option to proceed with flashing instead of
failing.
Signed-off-by: Øyvind Rønningstad <oyvind.ronningstad@nordicsemi.no>
Only create the dir and file if we have actual "content" that will
exist in testcase_extra.conf. This is to reduce a bit of
noise/footprint in the sanitycheck output dir.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
If we kill qemu because of a timeout, capture this as the reason instead
of reporting the exit code.
Fixes#28040
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Breathe 4.21.0 brings a few interesting features that improve the
generated documentation:
* A new `separate_member_pages` config option that patches issues in the
Doxygen XML generated when SEPARATE_MEMBER_PAGES option is YES.
* A new rST verbatim mode that allows producing inline elements.
Signed-off-by: Fabio Utzig <fabio.utzig@nordicsemi.no>
This undo's commit dd65f7c38a that changed how we sorted tests to get
better distribution from a qemu running point of view. However we now
have some tests that take a long time to build and we end up loading
build machines we all the long running tests.
Lets try going back to sorting by platform and see how that behaves with
the new CI environment.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
Commit 3124c02 ("cmake: flash/debug: refactor runner configuration")
changed the way the hex_file and elf_file inputs in the RunnerConfig
object are created. In particular, they are now host-style paths.
This breaks flashing with openocd on Windows, which doesn't handle that
properly. Fix that by "casting" the internal hex_file and elf_file
attributes to POSIX paths.
Fixes: #28138
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This code had one purpose only, feed timing information into a test and
was not used by anything else. The custom trace points unfortunatly were
not accurate and this test was delivering informatin that conflicted
with other tests we have due to placement of such trace points in the
architecture and kernel code.
For such measurements we are planning to use the tracing functionality
in a special mode that would be used for metrics without polluting the
architecture and kernel code with additional tracing and timing code.
Furthermore, much of the assembly code used had issues.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
MWDT toolchain adds additional suffix to sections name in case of
ffunction-sections / fdata-sections are enabled.
As proposed by Andy Ross let's pick a single set of rules
and syntax that work.
Suggested-by: Andy Ross <andy@plausible.org>
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
I've repeatedly seen that people are not aware of the existence of
this configuration option.
I've been using build.pristine=auto daily for years and never had any
problems. I've also asked around on Slack a couple of times over
various points to see if anybody objects to making this change. Nobody
has, so let's just turn it on by default.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Tell the EDT instance that properties of the /zephyr,user node should
be generated based on the binding types inferred from the property
content.
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
Clean up of devicetree tooling removed generation of information
present in devicetree in nodes that have no compatible, or for extra
properties not defined by a binding. Discussion proposed that these
properties should be allowed, but only in a defined node /zephyr,user.
For that node infer bindings based on the presence of properties.
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
By default, subprocess.Popen commands are supplied as a list of strings.
Using split() allows to use command with arguments, for example it is
possible to use following:
sanitycheck ... \
--device-serial-pty="set_ambient read_terminal.py" \
...
Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
This commit message is a bit of a novel mostly:
- because the issues involved are longstanding
- as evidence this is not a capricious refactoring
The runners.core.RunnerConfig Python class holds common configuration
values used by multiple runners, such as the location of the build
outputs and board directory.
The runners code, first written in 2017-ish, replaced various shell
scripts that got this information from the environment. Avoiding
environment variables was a requirement, however. It's ghastly to set
environment variables for a single command invocation on Windows, and
the whole thing was part of a larger push to make Zephyr development
on Windows better.
I had a hammer (the argparse module). Finding a replacement naturally
looked like a nail, so the information that ends up in RunnerConfig
got shunted from the build system to Python in the form of 'west
flash' / 'west debug' command line options like '--board-dir',
'--elf-file', etc.
I initially stored the options and their values in the CMake cache.
This was chosen in hopes the build system maintainer would like
the strategy (which worked).
I knew the command line arguments approach was a bit hacky (this
wasn't a nail), but I also honestly didn't have a better idea at the
time.
It did indeed cause issues:
- users don't know that just because they specify --bin-file on the
command line doesn't mean that their runner respects the option, and
have gotten confused trying to flash alternate files, usually for
chain-loading by MCUboot (for example, see #15961)
- common options weren't possible to pass via board.cmake files
(#22563, fixed partly via introduction of runners.yaml and the west
flash/debug commands no longer relying on the cache)
- it is confusing that "west flash --help" prints information about
openocd related options even when the user's board has no openocd
support. The same could be said about gdb in potential future use
cases where debugging occurs via some other tool.
Over time, they've caused enough users enough problems that
improvements are a priority.
To work towards this, put these values into runners.yaml using a new
'config: ...' key/value instead of command line options.
For example, instead of this in the generated runners.yaml file:
args:
common:
- --hex-file=.../zephyr.hex
we now have:
config:
hex_file: zephyr.hex
and similarly for other values.
In Python, we still support the command line options, but they are not
generated by the build system for any in-tree boards. Further work is
needed to deprecate the confusing ones (like --hex-file) and move the
runner-specific host tool related options (like --openocd) to the
runners that need them.
Individual board.cmake files should now influence these values by
overriding the relevant target properties of the
runners_yaml_props_target.
For example, instead of:
board_runner_args(foo "--hex-file=bar.hex")
Do this:
set_target_properties(runners_yaml_props_target PROPERTIES
hex_file bar.hex)
This change additionally allows us to stitch cmake/mcuboot.cmake and
the runners together easily by having mcuboot.cmake override the
properties that set the hex or bin file to flash. (The command line
arguments are still supported as-is.)
Combined with 98e0c95d91ae16f14e4997fb64ccdf0956595712 ("build:
auto-generate signed mcuboot binaries"), this will allow users to
build and flash images to be chain loaded by mcuboot in a way that
avoids calling 'west sign' and passing 'west flash' its output files
entirely.
While we are here, rename runner_yml_write to runners_yaml_append().
This function doesn't actually write anything, and we're here
refactoring this file anyway, so we might as well improve the
situation while we're at it.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The 'runner_config' variable name is particularly misleading because
there is a class called RunnerConfig, and that variable does not
contain one.
Rename it to 'runners_yaml' since it contains the parsed contents of
the runners.yaml file. Rename the variable that refers to the path
itself to 'runners_yaml_path'. No functional changes expected.
This is prep work for redoing how actual RunnerConfig objects get
made.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Before venturing off into refactoring land, let's make sure we have a
type checker running on the main APIs used by the various runners so
we don't miss anything obvious.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Use mypy to type check the runners package.
The test procedure is now annoying enough to replicate locally that
I'm going to wrap it in a script. Do this for both UNIX and Windows
environments by writing that script in Python.
Keep the GitHub workflow up to date so we now get mypy results in CI.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Colorama needs to be initialized, and for colors to display correctly on
Windows one needs to use the `USES_TERMINAL` option in CMake.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
Certain symbols do not contain a valid path, instead showing
`<built-in>` in the filename attribute. This leads to the resolve() call
failing on Windows, since the `<>` characters are not allowed in
filenames there. Fix this by catching the exception and skipping the
call in that case.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
2 scripts are provided.
- find_functions.cocci (name probably sucks...)
- find_dev_usage.cocci (ditto...)
find_functions.cocci can patch files where device instance are not
const.
Then it is used to generate the function database:
./scripts/coccicheck --mode=report --jobs=1 \
--cocci=scripts/coccinelle/find_functions.cocci \
--sp-flag="--include-headers" ./
Then, find_dev_usage.cocci will check if the const qualifier is, or
might be lost in a function call.
For instance:
./scripts/coccicheck --mode=report --jobs=1 \
--cocci=scripts/coccinelle/find_dev_usage.cocci \
--sp-flag="--include-headers" drivers/i2c
Which will output a WARNING on non-zephyr functions calls with a device:
./drivers/i2c/i2c_cc13xx_cc26xx.c:393:5-8: WARNING: in i2c_cc13xx_cc26xx_pm_control calling cb param with dev, check if const qualifier is not lost
./drivers/i2c/i2c_mcux_lpi2c.c:205:40-43: WARNING: in mcux_lpi2c_init calling LPI2C_MasterTransferCreateHandle param with dev, check if const qualifier is not lost
./drivers/i2c/i2c_nrfx_twi.c:258:5-8: WARNING: in twi_nrfx_pm_control calling cb param with dev, check if const qualifier is not lost
./drivers/i2c/i2c_nrfx_twi.c:202:22-25: WARNING: in init_twi calling nrfx_twi_init param with dev, check if const qualifier is not lost
./drivers/i2c/i2c_mcux.c:187:38-41: WARNING: in i2c_mcux_init calling I2C_MasterTransferCreateHandle param with dev, check if const qualifier is not lost
./drivers/i2c/i2c_mcux_flexcomm.c:184:43-46: WARNING: in mcux_flexcomm_init calling I2C_MasterTransferCreateHandle param with dev, check if const qualifier is not lost
./drivers/i2c/i2c_nrfx_twim.c:232:5-8: WARNING: in twim_nrfx_pm_control calling cb param with dev, check if const qualifier is not lost
./drivers/i2c/i2c_nrfx_twim.c:174:8-11: WARNING: in init_twim calling nrfx_twim_init param with dev, check if const qualifier is not lost
./drivers/i2c/i2c_rv32m1_lpi2c.c:246:6-9: WARNING: in rv32m1_lpi2c_init calling LPI2C_MasterTransferCreateHandle param with dev, check if const qualifier is not lost
Or:
./scripts/coccicheck --mode=report --jobs=1 \
--cocci=scripts/coccinelle/find_dev_usage.cocci \
--sp-flag="--include-headers" drivers/ieee802154
Which will output an ERROR on using a zephyr function that looses the
const qualifier:
drivers/ieee802154/ieee802154_rf2xx.c:778:3-6: ERROR: in rf2xx_init calling k_thread_create param with dev, loosing const qualifier, please wrap
drivers/ieee802154/ieee802154_nrf5.c:477:19-22: ERROR: in nrf5_init calling k_thread_create param with dev, loosing const qualifier, please wrap
drivers/ieee802154/ieee802154_cc1200.c:819:3-6: ERROR: in cc1200_init calling k_thread_create param with dev, loosing const qualifier, please wrap
drivers/ieee802154/ieee802154_mcr20a.c:1443:3-6: ERROR: in mcr20a_init calling k_thread_create param with dev, loosing const qualifier, please wrap
drivers/ieee802154/ieee802154_cc2520.c:1116:3-6: ERROR: in cc2520_init calling k_thread_create param with dev, loosing const qualifier, please wrap
drivers/ieee802154/ieee802154_cc13xx_cc26xx.c:439:32-35: ERROR: in ieee802154_cc13xx_cc26xx_data_init calling k_thread_create param with dev, loosing const qualifier, please wrap
ISSUE:
- Is it possible to run a set of rules first on all the code, and then
another set, both sets being in the same .cocci file?
Would be nice to have all at once.
Fixes#27399
Signed-off-by: Tomasz Bursztyka <tomasz.bursztyka@linux.intel.com>
The k_object API associates mutable state structures with known kernel
objects to support userspace. The kernel objects themselves are not
modified by the API, and in some cases (e.g. device structures) may be
const-qualified. Update the API so that pointers to these const
kernel objects can be passed without casting away the const qualifier.
Fixes#27399
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
Fix the case where we have no ztest test_* testcases and reports were
missed. This now makes sure we do not report the parent testcase when
there are individual test_* results.
Fixes#27765
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
add argument --pre-script to specify a pre script. This will be
executed before device handler open serial port and invoke runner.
Currently, pre_script can only get from hardware.map file. just like:
./scripts/sanitycheck --hardware-map xxx.map
and set pre_script in xxx.map file.
Signed-off-by: Watson Zeng <zhiwei@synopsys.com>
In some cases it might be a good idea to reset the board for real to
make sure it is completely recovered from some failed state (simple
re-loading of the application binary or even Elf file contents doesn't
affect most of internal CPU states so doesn't help in recovery,
see https://github.com/zephyrproject-rtos/zephyr/issues/25022 &
https://github.com/zephyrproject-rtos/zephyr/issues/26665).
And so we may want to utilize some external utility which triggers the
hard reset (in case of ARC boards it is
https://github.com/foss-for-synopsys-dwc-arc-processors/rff-ftdi-reset).
So we need to have a way to execute an external command before each and
every test.
Now given we already have quite some call-backs we try to use them
before re-inventing the wheel. And pre_script seem to be a good option
with just on minor note - it is called after serial port gets open.
And while in some cases it might be OK if serial port on the board is
not affected by the board's reset, if it is affected we'll be losing
connection on reset (and that's the case with ARC boards BTW as the
FTDI USB-to-Serial IC is also wired to the reset signal on most of
the boards). That said we just move invocation of pre_script before
opening the serial port and everything should be good now.
Signed-off-by: Watson Zeng <zhiwei@synopsys.com>
By default show reports based on last release. Fix a few other issues
where we had 0 values and were dividing by zero.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This commit improves Zephyr modules support in CMake.
All Zephyr modules will now have a corresponding
`ZEPHYR_${module_name}_MODULE_DIR` set, regardless of whether they
define a cmake folder in module.yml.
This results in:
`ZEPHYR_${module_name}_MODULE_DIR` defined for all modules and refers to
the root of the modle.
`ZEPHYR_${module_name}_CMAKE_DIR` defined for modules that specifies
cmake in module.yml, or is having both a CMakeLists.txt and Kconfig file
in the zephyr folder.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
This commit has no functional changes but is simply a cleanup so that
flake8 will pass without warnings.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
This will only build/run on emulation platforms.
The decision is made based on the value of the 'simulation' key in the
platform yaml file.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This is needed for sanitycheck hardware maps which take the serial
number directly from USB metadata.
Signed-off-by: Øyvind Rønningstad <oyvind.ronningstad@nordicsemi.no>
Adds some code to allow QEMU to crash without failing the test.
This is required for testing coredump code as it will certainly
cause QEMU to crash.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
