RFC: KOS Project Organization And Multi-architecture Considerations For 2.3.0/3.0.0 Release
===========================================================
As we approach the 2.3.0/3.0.0 release of KallistiOS (KOS), it's essential to revisit the project's organization and multi-architecture considerations. This document outlines various opinions and suggestions for improving the overall structure and functionality of KOS.
KOS Root
Directory Structure
The current directory structure for KOS is /opt/toolchains/dc/kos, /opt/toolchains/gc/kos, etc. However, this approach may not be the most efficient or user-friendly. Instead, we propose a single KOS installation that supports all platforms, with a directory structure like /opt/toolchains/kos.
Standardized Directory for Toolchains
The current convention of /opt/toolchains/dc/sh-elf is no longer suitable, as many users maintain multiple toolchains. A more standardized approach would be to use /opt/kallistios for the overall KallistiOS environment, /opt/kallistios/kos for the $KOS_BASE, and /opt/kallistios/toolchains/${KOS_ARCH}/${PROFILE_NAME} for toolchains themselves.
Environment Scripts
With a single KOS installation, the environment scripts will need to be reorganized. environ.sh could toggle which archs/subarchs are enabled for building and contain KOS arch-agnostic settings. Platform-specific environment settings would be contained in scripts like environ_dreamcast.sh, environ_gamecube.sh, etc.
Moving Build-Related Files
Files like loadable, utils/build_wrappers, utils/cmake, utils/ldscripts, Makefile.prefab, and Makefile.rules could be moved out of the project root and into a subdir related to build scripts.
Utils
Installing Binaries
Currently, utils binaries are not installed in the $DC_TOOLS_BASE dir, which is already added to $PATH. We propose installing all utils binaries into this tools dir and stopping the hardcoding of their paths.
Organizing Utils
utils should be organized into separate dirs for common tools and per-arch tools. This would prevent Dreamcast-only users from building GameCube-only utils, and vice versa.
Toolchain Building System
We have two options for the toolchain building system: creating a gc-chain tool specifically for the GameCube toolchain or modifying dc-chain to kos-chain and making it a more robust toolchain building system. The latter option might also be justified as a separate repository or submodule.
Examples
Architecture-Agnostic Examples
While we have examples organized per-arch, we should have a common dir for examples expected to build on all archs. This could also help us establish some kind of high-level, architecture-agnostic APIs for things like controllers.
Addons and Ports
Enhancing Addons
addons is a great mechanism for allowing users to install external libraries directly into KallistiOS and integrate them into our build system. However, it is underutilized and underpowered compared to the build system support we have in kos-ports. We propose moving of the build system support from kos-ports to KOS proper.
Integrating kos-ports
We discussed putting the kos-ports dir into ${KOS_BASE}/ports and making it a git submodule. This would help couple KOS and kos-ports installations together, making it easier for maintainers and end users to manage their builds.
Parallelizing kos-ports Builds
It would be nice if we could parallelize kos-ports builds. Several repos cannot be built with more than 1 job for upstream reasons beyond our control, but we could at least make it so multiple ports were building at once. We could also design the makefile system to check if a kos-port is installed and more gracefully skip its compilation or automatically build it.
Conclusion
In conclusion, we propose a single KOS installation that supports all platforms, with a standardized directory structure and reorganized environment scripts. We also suggest enhancing the addons mechanism, integrating kos-ports, and parallelizing kos-ports builds. These changes would improve the overall structure and functionality of KOS, making it more user-friendly and efficient.
===========================================================
As we approach the 2.3.0/3.0.0 release of KallistiOS (KOS), we've received many questions and concerns about the proposed changes to the project's organization and multi-architecture considerations. In this article, we'll address some of the most frequently asked questions and provide additional clarification on the proposed changes.
Q: Why do we need to change the directory structure?
A: The current directory structure, with separate directories for each platform, can lead to confusion and make it difficult for users to manage their builds. A single KOS installation that supports all platforms would simplify the directory structure and make it easier for users to work with.
Q: How would the new directory structure work?
A: The proposed directory structure would be /opt/kallistios for the overall KallistiOS environment, /opt/kallistios/kos for the $KOS_BASE, and /opt/kallistios/toolchains/${KOS_ARCH}/${PROFILE_NAME} for toolchains themselves. This would allow users to easily switch between different toolchains and platforms.
Q: What about the environment scripts? How would they change?
A: The environment scripts would need to be reorganized to accommodate the new directory structure. environ.sh would toggle which archs/subarchs are enabled for building and contain KOS arch-agnostic settings. Platform-specific environment settings would be contained in scripts like environ_dreamcast.sh, environ_gamecube.sh, etc.
Q: What about the build-related files? Why move them out of the project root?
A: The build-related files, such as loadable, utils/build_wrappers, utils/cmake, utils/ldscripts, Makefile.prefab, and Makefile.rules, are not directly related to the KOS build process. Moving them out of the project root would simplify the directory structure and make it easier to manage the build process.
Q: How would the utils directory be organized?
A: The utils directory would be organized into separate dirs for common tools and per-arch tools. This would prevent Dreamcast-only users from building GameCube-only utils, and vice versa.
Q: What about the toolchain building system? Why not create a gc-chain tool?
A: Creating a gc-chain tool specifically for the GameCube toolchain would be redundant, as the dc-chain tool could be modified to support multiple toolchains. This would simplify the toolchain building system and make it easier to manage.
Q: How would the addons mechanism be enhanced?
A: The addons mechanism would be enhanced by moving the build system support from kos-ports to KOS proper. This would allow users to easily install external libraries directly into KallistiOS and integrate them into the build system.
Q: What about the kos-ports dir? Why make it a git submodule?
A: Making the kos-ports dir a git submodule would help couple KOS and kos-ports installations together, making it easier for maintainers and end users to manage their builds.
Q: How would the makefile system be designed to parallelize kos-ports builds?
A: The makefile system would be designed to if a kos-port is installed and more gracefully skip its compilation or automatically build it. This would allow users to parallelize kos-ports builds and simplify the build process.
Q: What about the high-level, architecture-agnostic APIs for things like controllers?
A: The high-level, architecture-agnostic APIs for things like controllers would be established by creating a common dir for examples expected to build on all archs. This would simplify the development process and make it easier to create architecture-agnostic APIs.
Q: What's the timeline for implementing these changes?
A: The timeline for implementing these changes would depend on the community's feedback and the complexity of the changes. We propose a phased implementation, with the first phase focusing on the directory structure and environment scripts, and subsequent phases addressing the build-related files, utils directory, toolchain building system, and kos-ports dir.
Q: How can I contribute to the discussion and implementation of these changes?
A: You can contribute to the discussion and implementation of these changes by joining the KOS community, providing feedback on the proposed changes, and participating in the development process. We welcome all contributions and look forward to working together to improve the KOS project.