Commit Graph

1063 Commits

Author SHA1 Message Date
Carlo Caione
daa94e5e59 aarch64: Remove redundant init_stack_frame
The init_stack_frame is the same as the the ESF. No need to have two
separate structs. Consolidate everything into one single struct and make
register entries explicit.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-17 18:59:23 -05:00
Carlo Caione
a7d94b003e aarch64: Use absolute symbols for the callee saved registers
Use GEN_OFFSET_SYM macro to genarate absolute symbols for the
_callee_saved struct and use these new symbols in the assembly code.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-17 18:59:23 -05:00
Carlo Caione
a054e424e4 aarch64: error: Beautify error printing
Make the printing of errors a bit more descriptive and print the FAR_ELn
register only when strictly required.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-17 18:52:45 -05:00
Johan Hedberg
9e4dfd8f4e drivers: pcie: Add support for IRQ allocation management
There are x86 platforms where the IRQ configuration register for PCIe
is not pre-populated and the OS needs to assign a number dynamically
by writing to the register.

In order to allocate interrupts we have to know which ones have been
hard-coded in device tree. We accomplish this by collecting these
values through the IRQ_CONNECT() macro and placing them in a dedicated
linker section (in ROM).

The full set of allocated interrupts are managed through a bitmap, and
the pre-allocated values (from the linker section) are inserted into
this upon initial runtime access.

This patch introduces a new pcie_alloc_irq() API that drivers can use
to allocate interrupt line numbers. The two in-tree drivers that were
using this API (I2C and UART) are converted to use the new API.

Signed-off-by: Johan Hedberg <johan.hedberg@intel.com>
2020-11-14 11:13:00 +02:00
Martin Åberg
feae3249b2 sparc: add support for thread local storage
Adds the necessary bits to initialize TLS in the stack
area and sets up CPU registers during context switch. Register g7 is
used to point to the thread data. Thread data is accessed with negative
offsets from g7.

Signed-off-by: Martin Åberg <martin.aberg@gaisler.com>
2020-11-13 14:53:55 -08:00
Martin Åberg
07160fa153 arch: Add SPARC processor architecture
SPARC is an open and royalty free processor architecture.

This commit provides SPARC architecture support to Zephyr. It is
compatible with the SPARC V8 specification and the SPARC ABI and is
independent of processor implementation.

Functionality specific to SPRAC processor implementations should
go in soc/sparc. One example is the LEON3 SOC which is part of this
patch set.

The architecture port is fully SPARC ABI compatible, including trap
handlers and interrupt context.

Number of implemented register windows can be configured.

Some SPARC V8 processors borrow the CASA (compare-and-swap) atomic
instructions from SPARC V9. An option has been defined in the
architecture port to forward the corresponding code-generation option
to the compiler.

Stack size related config options have been defined in sparc/Kconfig
to match the SPARC ABI.

Co-authored-by: Nikolaus Huber <nikolaus.huber.melk@gmail.com>
Signed-off-by: Martin Åberg <martin.aberg@gaisler.com>
2020-11-13 14:53:55 -08:00
Daniel Leung
c7704d8c66 arc: enable thread local storage
This adds the necessary bits to support thread local storage
in ARC.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2020-11-11 13:25:29 +01:00
Yuguo Zou
3826eb302c arch: arc: add support of ARConnect inter-core debug unit
The Inter-core Debug Unit provides additional debug assist features in
multi-core scenarios.This commit allows ARConnect to conditionally
halt cores during debugging.

Signed-off-by: Yuguo Zou <yuguo.zou@synopsys.com>
2020-11-09 15:52:15 -06:00
Nicolas Royer
f5ac11ae32 arch: riscv: linker: add support for userspace
- Move kobject list after .bss
  The previous order shift kernel object address defined in prebuild
  file (which is use as reference in running time). So it was impossible
  for zephyr to check if a kernel object address was granted because
  address changed during build.

- Add support for memory shared.

- Move sdata2 section in ROM because it contains constants.

Signed-off-by: Nicolas Royer <nroyer@baylibre.com>
2020-11-09 15:37:11 -05:00
Alexandre Mergnat
542a7fa25d arch: riscv: add memory protection support
The IRQ handler has had a major changes to manage syscall, reschedule
and interrupt from user thread and stack guard.

Add userspace support:
- Use a global variable to know if the current execution is user or
  machine. The location of this variable is read only for all user
  thread and read/write for kernel thread.
- Memory shared is supported.
- Use dynamic allocation to optimize PMP slot usage. If the area size
  is a power of 2, only one PMP slot is used, else 2 are used.

Add stack guard support:
- Use MPRV bit to force PMP rules to machine mode execution.
- IRQ stack have a locked stack guard to avoid re-write PMP
  configuration registers for each interruption and then win some
  cycle.
- The IRQ stack is used as "temporary" stack at the beginning of IRQ
  handler to save current ESF. That avoid to trigger write fault on
  thread stack during store ESF which that call IRQ handler to
  infinity.
- A stack guard is also setup for privileged stack of a user thread.

Thread:
- A PMP setup is specific to each thread. PMP setup are saved in each
  thread structure to improve reschedule performance.

Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com>
Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-11-09 15:37:11 -05:00
Alexandre Mergnat
18962e4ab8 arch: riscv: add pmp support
- Set some helper function to write/clear/print PMP config registers.
- Add support for different PMP slot size function to core/board.

Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com>
2020-11-09 15:37:11 -05:00
Andrew Boie
d2a72273b7 x86: add support for common page tables
We provide an option for low-memory systems to use a single set
of page tables for all threads. This is only supported if
KPTI and SMP are disabled. This configuration saves a considerable
amount of RAM, especially if multiple memory domains are used,
at a cost of context switching overhead.

Some caching techniques are used to reduce the amount of context
switch updates; the page tables aren't updated if switching to
a supervisor thread, and the page table configuration of the last
user thread switched in is cached.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-11-05 09:33:40 -05:00
Andrew Boie
a15be58019 x86: move page table reservation macros
We don't need this for stacks any more and only use this
for pre-calculating the boot page tables size. Move to C
code, this doesn't need to be in headers anywhere.

Names adjusted for conciseness.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-11-05 09:33:40 -05:00
Andrew Boie
1367c4a4b7 x86: don't reserve room for page tables in stack
These are handled at the memory domain level now.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-11-05 09:33:40 -05:00
Andrew Boie
b8242bff64 x86: move page tables to the memory domain level
- z_x86_userspace_enter() for both 32-bit and 64-bit now
  call into C code to clear the stack buffer and set the
  US bits in the page tables for the memory range.

- Page tables are now associated with memory domains,
  instead of having separate page tables per thread.
  A spinlock protects write access to these page tables,
  and read/write access to the list of active page
  tables.

- arch_mem_domain_init() implemented, allocating and
  copying page tables from the boot page tables.

- struct arch_mem_domain defined for x86. It has
  a page table link and also a list node for iterating
  over them.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-11-05 09:33:40 -05:00
Carlo Caione
7b7c328f7a aarch64: mmu: Enable support for unprivileged EL0
The current MMU code is assuming that both kernel and threads are both
running in EL1, not supporting EL0. Extend the support to EL0 by adding
the missing attribute to mirror the access / execute permissions to EL0.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-04 13:58:19 -08:00
Ioannis Glaropoulos
47e87d8459 arch: arm: cortex_m: implement functionality for ARCH core regs init
Implement the functionality for configuring the
architecture core registers to their warm reset
values upon system initialization. We enable the
support of the feature in the Cortex-M architecture.

Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
2020-11-02 15:02:24 +01:00
Carlo Caione
b3ff89bd51 arch: arm64: Remove _BIT suffix
This is redundant and not coherent with the rest of the file. Thus
remove the _BIT suffix from the bit field names.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-02 12:04:35 +01:00
Carlo Caione
8941f9a00c x86: mmustructs: Fix define typo
Fix typo s/Z_X96_MMU_RW/Z_X86_MMU_RW/

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-10-29 16:44:22 -04:00
Daniel Leung
2c8625ea7e xtensa: remove errno_var from strcut _thread_arch
The errno_var copy in Xtensa's struct is not being used at all
for errno (as there is already one in struct k_thread).
So remove it.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2020-10-24 10:52:00 -07:00
Daniel Leung
8a79ce1428 riscv: add support for thread local storage
Adds the necessary bits to initialize TLS in the stack
area and sets up CPU registers during context switch.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2020-10-24 10:52:00 -07:00
Daniel Leung
388725870f arm: cortex_m: add support for thread local storage
Adds the necessary bits to initialize TLS in the stack
area and sets up CPU registers during context switch.

Note that since Cortex-M does not have the thread ID or
process ID register needed to store TLS pointer at runtime
for toolchain to access thread data, a global variable is
used instead.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2020-10-24 10:52:00 -07:00
Daniel Leung
778c996831 arm: cortex_r: add support for thread local storage
Adds the necessary bits to initialize TLS in the stack
area and sets up CPU registers during context switch.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2020-10-24 10:52:00 -07:00
Daniel Leung
df77e2af8b arm64: add support for thread local storage
Adds the necessary bits to initialize TLS in the stack
area and sets up CPU registers during context switch.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2020-10-24 10:52:00 -07:00
Daniel Leung
4b38392ded x86: add support for thread local storage
Adds the necessary bits to initialize TLS in the stack
area and sets up CPU registers during context switch.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2020-10-24 10:52:00 -07:00
Daniel Leung
53ac1ee6fa x86_64: add support for thread local storage
Adds the necessary bits to initialize TLS in the stack
area and sets up CPU registers during context switch.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2020-10-24 10:52:00 -07:00
Andy Ross
0e83961b21 arch/xtensa: soc/xtensa/intel_adsp: Enable KERNEL_COHERENCE
Implement the kernel "coherence" API on top of the linker
cached/uncached mapping work.

Add Xtensa handling for the stack coherence API.

Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2020-10-21 06:38:53 -04:00
Carlo Caione
2f3962534a arch: arm64: Remove new thread entry wrapper
Instead of having some special stack frame when first scheduling new
thread and a new thread entry wrapper to pull out the needed data, we
can reuse the context restore code by adapting the initial stack frame.

This reduces the lines of code and simplify the code at the expense of a
slightly bigger initial stack frame.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-10-06 10:25:56 -04:00
Eugeniy Paltsev
9b0ef4f19a ARC: MWDT: drop redundant stack checking
MWDT toolchain has Stackcheck_alloca option enabled by default.
So it adds stack checking in addition to Zephyr's stack checking.

As it is completely redundant let's drop it.

Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
2020-10-02 11:32:12 +02:00
Yuguo Zou
df4b7803a1 arch: arc: unify different versions of MPU registers
Previously MPU registers macros are only defined within its own header
files and could not be used by other part of program. This commit unify
them together.

Signed-off-by: Yuguo Zou <yuguo.zou@synopsys.com>
2020-10-02 11:31:34 +02:00
Aastha Grover
83b9f69755 code-guideline: Fixing code violation 10.4 Rule
Both operands of an operator in the arithmetic conversions
performed shall have the same essential type category.

Changes are related to converting the integer constants to the
unsigned integer constants

Signed-off-by: Aastha Grover <aastha.grover@intel.com>
2020-10-01 17:13:29 -04:00
Tomasz Bursztyka
d98f7b1895 arch/x86: Optimize ACPI RSDP lookup
As well as normalizing its signature declaration through header.

Signed-off-by: Tomasz Bursztyka <tomasz.bursztyka@linux.intel.com>
2020-10-01 11:16:40 -07:00
Tomasz Bursztyka
4ff1885f69 arch/x86: Move ACPI structures to header file
Let's have all specified ACPI structures in the central header.

Signed-off-by: Tomasz Bursztyka <tomasz.bursztyka@linux.intel.com>
2020-10-01 11:16:40 -07:00
Tomasz Bursztyka
c7787c623e arch/x86: Cleanup ACPI structure attributes names
No need to mix super short version of names with other structures
having full name. Let's follow a more relevant naming where each and
every attribute name is self-documenting then. (such as s/id/apic_id
etc...)

Also make CONFIG_ACPI usable through IS_ENABLED by enclosing exposed
functions with ifdef CONFIG_ACPI.

Signed-off-by: Tomasz Bursztyka <tomasz.bursztyka@linux.intel.com>
2020-10-01 11:16:40 -07:00
Luke Starrett
169e7c5e75 arch: arm64: Fix arm64 crash dump output
- x0/x1 register printing is reversed
- The error stack frame struct (z_arch_esf_t) had the SPSR and ELR in
  the wrong position, inconsistent with the order these regs are pushed
  to the stack in z_arm64_svc.  This caused all register printing to be
  skewed by two.
- Verified by writing known values (abcd0000 -> abcd000f) to x0 - x15
  and then forcing a data abort.

Signed-off-by: Luke Starrett <luke.starrett@gmail.com>
2020-10-01 07:29:27 -04:00
Andrew Boie
391935bea1 x86_64: add dedicated MEMORY area for locore
This had been hacked into the linker script, define
a proper MEMORY region for it.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-09-30 14:14:07 -07:00
Andrew Boie
7f4901b8a6 x86: 32-bit: remove mmu region list
This related to old infrastructure which has been removed
from Zephyr.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-09-30 14:14:07 -07:00
Andrew Boie
3807c51e4e x86: add common memory.ld
We need the same logic for each SOC, instead of copypasting
things just put this in a common file. This approach still
leaves the door open for custom memory layouts if desired.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-09-30 14:14:07 -07:00
Andrew Boie
27e00f497a x86: 32-bit: set _image_text_start properly
Strictly speaking, should bee a virtual, not physical address.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-09-30 14:14:07 -07:00
Andrew Boie
c3c7f6c6d3 x86: don't define _image_rom_* unless XIP
Meaningless if we are not a XIP system and are running
from RAM.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-09-30 14:14:07 -07:00
Eugeniy Paltsev
8e1f40a632 ARC: linker: merge GNU and MWDT linker scripts
As discussed in #22668, there is additional risk ascociated with
splitting linker files, as one may update one script and not be
aware of the other. Especially related to updating GNU ld, and
not mwdt could break code for mwdt unnoticed, as mwdt is not
part of CI.

Let's create a single entry point for linker template.

Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
2020-09-18 09:49:09 -04:00
Ioannis Glaropoulos
d19d6a76a7 arch: arm: aarch64: remove non-applicable linker section
The non-secure callable functions' section is only applicable
to Cortex-M with TrustZone-M extension. Remove it from AARCH64
linker script. (CONFIG_ARM_FIRMWARE_HAS_SECURE_ENTRY_FUNCTIONS
is only enabled for Cortex-M so this is a no-op, but still, it
is a useful cleanup.)

Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
2020-09-14 19:17:04 -05:00
Anas Nashif
6e27478c3d benchmarking: remove execution benchmarking code
This code had one purpose only, feed timing information into a test and
was not used by anything else. The custom trace points unfortunatly were
not accurate and this test was delivering informatin that conflicted
with other tests we have due to placement of such trace points in the
architecture and kernel code.

For such measurements we are planning to use the tracing functionality
in a special mode that would be used for metrics without polluting the
architecture and kernel code with additional tracing and timing code.

Furthermore, much of the assembly code used had issues.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2020-09-05 13:28:38 -05:00
Daniel Leung
80fb6538b3 x86: use =A as output for RDTSC on x86-32
The timing_info benchmark on qemu_x86 shows this is a bit faster.

Before:
  START - Time Measurement
  Timing results: Clock frequency: 1000 MHz
  Context switch                               : 896 cycles ,   895 ns
  Interrupt latency                            : 768 cycles ,   767 ns
  Tick overhead                                :14912 cycles , 14911 ns
  Thread creation                              :18688 cycles , 18687 ns
  Thread abort (non-running)                   :49216 cycles , 49215 ns
  Thread abort (_current)                      :55616 cycles , 55615 ns
  Thread suspend                               :11072 cycles , 11071 ns
  Thread resume                                :10272 cycles , 10271 ns
  Thread yield                                 :12213 cycles , 12212 ns
  Thread sleep                                 :17984 cycles , 17983 ns
  Heap malloc                                  :21702 cycles , 21701 ns
  Heap free                                    :15176 cycles , 15175 ns
  Semaphore take with context switch           :19168 cycles , 19167 ns
  Semaphore give with context switch           :18400 cycles , 18399 ns
  Semaphore take without context switch        :2208 cycles ,  2207 ns
  Semaphore give without context switch        :4704 cycles ,  4703 ns
  Mutex lock                                   :1952 cycles ,  1951 ns
  Mutex unlock                                 :7936 cycles ,  7935 ns
  Message queue put with context switch        :20320 cycles , 20319 ns
  Message queue put without context switch     :5792 cycles ,  5791 ns
  Message queue get with context switch        :22112 cycles , 22111 ns
  Message queue get without context switch     :5312 cycles ,  5311 ns
  Mailbox synchronous put                      :27936 cycles , 27935 ns
  Mailbox synchronous get                      :23392 cycles , 23391 ns
  Mailbox asynchronous put                     :11808 cycles , 11807 ns
  Mailbox get without context switch           :20416 cycles , 20415 ns
  Drop to user mode                            :643712 cycles , 643711 ns
  User thread creation                         :652096 cycles , 652095 ns
  Syscall overhead                             :2720 cycles ,  2719 ns
  Validation overhead k_object init            :4256 cycles ,  4255 ns
  Validation overhead k_object permission      :4224 cycles ,  4223 ns
  Time Measurement finished

After:
  START - Time Measurement
  Timing results: Clock frequency: 1000 MHz
  Context switch                               : 896 cycles ,   895 ns
  Interrupt latency                            : 768 cycles ,   767 ns
  Tick overhead                                :14752 cycles , 14751 ns
  Thread creation                              :18464 cycles , 18463 ns
  Thread abort (non-running)                   :48992 cycles , 48991 ns
  Thread abort (_current)                      :55552 cycles , 55551 ns
  Thread suspend                               :10848 cycles , 10847 ns
  Thread resume                                :10048 cycles , 10047 ns
  Thread yield                                 :12213 cycles , 12212 ns
  Thread sleep                                 :17984 cycles , 17983 ns
  Heap malloc                                  :21702 cycles , 21701 ns
  Heap free                                    :15176 cycles , 15175 ns
  Semaphore take with context switch           :19104 cycles , 19103 ns
  Semaphore give with context switch           :18368 cycles , 18367 ns
  Semaphore take without context switch        :1984 cycles ,  1983 ns
  Semaphore give without context switch        :4480 cycles ,  4479 ns
  Mutex lock                                   :1728 cycles ,  1727 ns
  Mutex unlock                                 :7712 cycles ,  7711 ns
  Message queue put with context switch        :20224 cycles , 20223 ns
  Message queue put without context switch     :5568 cycles ,  5567 ns
  Message queue get with context switch        :22016 cycles , 22015 ns
  Message queue get without context switch     :5088 cycles ,  5087 ns
  Mailbox synchronous put                      :27840 cycles , 27839 ns
  Mailbox synchronous get                      :23296 cycles , 23295 ns
  Mailbox asynchronous put                     :11584 cycles , 11583 ns
  Mailbox get without context switch           :20192 cycles , 20191 ns
  Drop to user mode                            :643616 cycles , 643615 ns
  User thread creation                         :651872 cycles , 651871 ns
  Syscall overhead                             :2464 cycles ,  2463 ns
  Validation overhead k_object init            :4032 cycles ,  4031 ns
  Validation overhead k_object permission      :4000 cycles ,  3999 ns
  Time Measurement finished

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2020-09-05 13:28:38 -05:00
Daniel Leung
c234821861 x86: use LFENCE instead of CPUID before reading TSC for x86_64
According to Intel 64 and IA-32 Architectures Software
Developer’s Manual, volume 3, chapter 8.2.5, LFENCE provides
a more efficient method of controlling memory ordering than
the CPUID instruction. So use LFENCE here, as all 64-bit
CPUs have LFENCE.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2020-09-05 13:28:38 -05:00
Eugeniy Paltsev
05b6468a73 ARC: linker: add more place for optimization
Do not force linker to place text sections after each other
to have more freedom to optimize.

Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
2020-09-05 10:22:56 -05:00
Wayne Ren
cc897a5198 ARC: add linker script template for metware toolchain
Add linker script template for MWDT toolchain (linker-mwdt.ld)
Move linker.ld to linker-gnu.ld (without changes)

The "linker.ld" is wraper now.

Signed-off-by: Wayne Ren <wei.ren@synopsys.com>
2020-09-05 10:22:56 -05:00
Eugeniy Paltsev
0227056066 ARC: change direct IRQ declaration for metaware toolchain
* change direct IRQ declaration for metaware toolchain
* drop unused irq-related definitions

Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
2020-09-05 10:22:56 -05:00
Eugeniy Paltsev
932e178007 ARC: use MWDT intrinsics to access aux regs in case of MWDT toolchain
Metaware doesn't support gcc's builtins so use corresponding intrinsics
instead.

Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
2020-09-05 10:22:56 -05:00
Wayne Ren
ef224ce1cd ARC: make the assembly codes compatible
Make the assembly codes compatible with both GNU
and Metaware toolchain.

* replace ".balign" with ".align"
  ".align" assembler directive is supposed by all
  ARC toolchains and it is implemented in a same
  way across ARC toolchains.
* replace "mov_s __certain_reg" with "mov __certain_reg"
  Even though GCC encodes those mnemonics and even real
  HW executes them according to PRM these are restricted
  ones for mov_s and CCAC rightfully refuses to accept
  such mnemonics. So for compatibility and clarity sake
  we switch to 32-bit mov instruction which allows use
  of all those instructions.
* Add "%%" prefix while accessing registers from inline
  ASM as it is required by MWDT.
* Drop "@" prefix while accessing symbols (defined in C
  code) from ASM code as it is required by MWDT.

Signed-off-by: Wayne Ren <wei.ren@synopsys.com>

/#
2020-09-05 10:22:56 -05:00