Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.38%. Comparing base (27544a2) to head (6160677).
⚠️ Report is 43 commits behind head on master.

@@           Coverage Diff           @@
##           master   #18424   +/-   ##
=======================================
  Coverage   98.38%   98.38%           
=======================================
  Files         171      171           
  Lines       22294    22299    +5     
=======================================
+ Hits        21933    21938    +5     
  Misses        361      361           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Code size report:

Reference:  esp32/boards: Add Silicognition ManT1S board definition. [27544a2]
Comparison: stm32: Add common init linker script. [merge of 1daa79e]
  mpy-cross:    +0 +0.000% 
   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

Hi @iabdalkader I've thought of ways to achieve very similar goals a few times, wanting to consolidate more of the main files and the bring-up / soft-reset / etc of ports!

The linker solution is elegant; I'd been thinking more along the lines of a python script similar to the module / root-pointer registrations. I actually started playing with this idea (just on the DEINIT function initially) here https://github.com/andrewleech/micropython/commits/consolidate_mp_deinit/

The main extra contstraints I thought of was dependencies between modules and ensuring these can be handled cleanly; I didn't want each module to need to know the overall order it should be run in (unless it's at a specific phase eg pre- / post- boot.py ) but more just that a module could register that it has an explicit dependency on another one.

I've thought of ways to achieve very similar goals a few times, wanting to consolidate more of the main files and the bring-up / soft-reset / etc of ports!

Yes, I've been thinking about fixing this too for a very long time: #3639 (comment) I wish I had sent a fix earlier, but there weren’t as many init/deinit functions and ports as there are now

The linker solution is elegant; I'd been thinking more along the lines of a python script

It's pretty much the standard way to do this, for example the Linux kernel's init calls, and Zephyr's init functions are linker-script-based. I prefer to use standard tools when possible, the ld does a pretty good job at this. That said, it doesn't matter if it achieves the same results, but it might be slow to scan all those C files looking for init functions.

The main extra contstraints I thought of was dependencies between modules

Honestly, I only care about fixing this init/deinit mess, for dependency tracking you probably need something like kconfig. For init/deinit, most of them have no dependencies (like all init0 functions), most of the deinit similarly have no deps, but you could still add a few levels or order them within a level.

only care about fixing this init/deinit

I was referring to init/deinit - I'm pretty sure some peripheral / module inits do have explicit ordering, I'm thinking things like network subsystem vs network adaptors, or ble vs radio.
Maybe an early/late init approach would be enough for simplicity with a comment at registration noting the modules it needs registered first, but I'd prefer to make these kind of ordering dependencies explicit, rather than just based on a first/second/third/etc pass.

While yes I can see the linker is a standard approach here, I generally find it very hard to debug when things don't behave! The python parser approach is used for other things in micropython that feel similar to me so it's still my vote.

fwiw I've also got a WIP / POC of a kconfig overlay for the whole project, but that's certainly a much larger change!

, I'm thinking things like network subsystem vs network adaptors, or ble vs radio.

Those easily fit into DRIVERS and SERVICES, and we could add more levels if needed.

but I'd prefer to make these kind of ordering dependencies explicit, rather than just based on a first/second/third/etc pass

Actually, on second thought, it's probably better to work towards decoupling these init functions as much as possible. If it's possible to init the stacks and drivers in any order, that's much better than forcing some dependency. Note things usually don't start immediately, not until everything has been initialized.

, I generally find it very hard to debug when things don't behave! The python parser approach is used for other things in micropython that feel similar to me so it's still my vote.

I'd argue that this is Not true especially in this case. The levels produced by this are self-documenting, you can just read/parse the map and get a nice, ordered, list of levels and the functions in them, compared to looking for a generated header in the build files. There's also the critical issue of performance with parsing all C files. Note those are init/deinit functions, and don't necessarily live in Python C user modules, so I think we'll need to scan all C/C++ files in the project.

Anyway, if it gets the job done, it's fine with me. I'm just not good at adding these tools to MicroPython, so it will have to be your work.

fwiw I've also got a WIP / POC of a kconfig overlay for the whole project, but that's certainly a much larger change!

Awesome! I believe MicroPython is going to need all that, plus the memory management I proposed, if it's ever to continue scaling in the future.

There's also the critical issue of performance with parsing all C files. Note those are init/deinit functions, and don't necessarily live in Python C user modules, so I think we'll need to scan all C/C++ files in the project.

I must admit I do think the compilation performance benefit of this linker approach is a real benefit. It's impressive how little code is actually needed here to implement it too, the efficiency is quite nice.

That being said I think all C code is already parsed just once (via gcc preproc) to find and handle ROOT_POINTER and other similar registrations etc so this would just be an additional couple of patterns in the existing parser script.
That script is not particularly simple... but it is flexible.

So I come back to worrying the linker approach might not be flexible enough, or maybe having some constraints is a good thing to enforce consistency.

The performance of the linker approach is good and having a similar approach to other projects helps people work on all of them (though I often don't want to copy linux and zephyr as I feel many of their design patterns have a very steep learning curve too).

Long story short I think there's pros and cons to both, but both approaches can certainly achieve the same goal and it's a worthy goal!

... and yes, aiming to decouple separate drivers / services would certainly make more sense!

That being said I think all C code is already parsed just once (via gcc preproc) to find and handle ROOT_POINTER

I don't think so, I think only USER_C_MODULES are parsed looking for root pointers. Can you try adding a root pointer in main.c?

Yes, you're right about that.