Merge branch 'Revirt-post-1' into 'master'

🚀 Revirt posts (2 posts)

See merge request !262

content/news/bochs-qemu-2.md

@@ -16,12 +16,13 @@ For those of you who don't know me (or my role in Redox), I'd like to introduce
 
 This post brings you some updates on the progress I've made on that front (QEMU on Redox) and another juicy undertaking of mine - **It's BOCHS on Redox y'all!** 😀
 
-I'm writing this post as an UPDATE to my previous post (at [this link](https://redox-os.org/news/rsoc-2021-qemu-1/)) that I wrote last year, as part of RSoC 2021, where I promised an update (which is long due). This post is going to be more straightforward and technical than the last one, so if you aren't fully aware of QEMU, OSes, emulators, userspace program porting, etc, you should definitely read that post. I explain _from the basic_ in simple language, before getting more technical.  
+I'm writing this post as an UPDATE to my previous post ([refer post - RSoC 2021: QEMU on Redox OS - Part 1](https://redox-os.org/news/rsoc-2021-qemu-1/)) that I wrote last year, as part of RSoC 2021, where I promised an update (which is long due). This post is going to be more straightforward and technical than the last one, so if you aren't fully aware of QEMU, OSes, emulators, userspace program porting, etc, you should definitely read that post. I explain _from the basic_ in simple language, before getting more technical.  
 You should also read it if you're interested in how I started out porting QEMU to Redox OS.
 
 It has been one heck of a year and despite my preoccupation with my undergraduate degree (BTech - CSE, 2023), I was able to put in some amount of time after the duration of RSoC 2021, to further develop the project I was working on (QEMU, and later Bochs). I worked on the stuff mentioned in this post a while back, but am writing this post only now (procrastination). This post is not a part of either RSoC 2021 or RSoC 2022, just an update to spice things up.
 
-BTW, I'm currently working on █████████████ as part of RSoC 2022. (I'll first write the post and then reveal what I'm working on, because it's fancy-shmancy 😉).
+BTW, I'm currently working on **Virtualization on Redox OS** as part of RSoC 2022.  
+Yep! Virtualization support is coming to Redox OS _soon_! The feature is called **Revirt** and the project plan and implementation details are documented in the two posts - ["Revirt - Virtualization on Redox OS"](https://redox-os.org/news/revirt-1/) AND ["RSoC 2022: Revirt-U - Part 1"](https://redox-os.org/news/rsoc-2022-revirt-u-1/).
 
 <br/>
 
@@ -29,19 +30,25 @@ BTW, I'm currently working on █████████████ as part of
 
 If I make any updates to this post (which is sometimes helpful for people reading it in the future), I shall make note of it here and st the same time, ~~scratch out~~ older content and replace it with something like "UPDATE1: ...".
 
+<br/>
 <br/>
 
 # Emulators on Redox
 
-Emulators are applications that provide ISA-level virtualization (one among other types of virtualization). Having emulators like QEMU and Bochs successfully run on Redox OS, helps realize the support (tooling and ecosystem) required for self-hosting, and at the same time enabling support for specific use cases for Redox OS.
+_Emulators are applications that provide **ISA-level virtualization** (one among other types of virtualization). If an emulator emulates an ISA which is the same as the ISA of the hardware the emulator is running on, it is also known as **Hardware-level Virtualization** (example: Bochs)_  
+_Though, do note that 'Hardware-level virtualization' is not the same as 'Hardware-assisted virtualization'. The latter is a special type of the former, where 'hardware-assisted' means ISA-support (hardwired) for virtualization at the 'hardware-level'_
+
+<br/>
+
+Having emulators like QEMU and Bochs successfully run on Redox OS, helps realize the support (tooling and ecosystem) required for self-hosting, and at the same time enabling support for specific use cases for Redox OS.
 
-Self-hosting isn't a long way off on Redox, and tooling around a new software/hardware that makes it easier to work with pretty much decides how successful the software/hardware will be. (as suggested by Linux Torvalds, where he said that he loves `x86` more than `arm` because of the amazing ecosystem around x86 that has been built over the past decades).
+Self-hosting isn't a long way off on Redox, and tooling around a new software/hardware that makes it easier to work with pretty much decides how successful the software/hardware will be. (as suggested by Linus Torvalds, where he said that he loves `x86` more than `arm` because of the amazing ecosystem around x86 that has been built over the past decades).
 
 While I was attempting to port QEMU, I thought of looking into porting Bochs too, given that its codebase looked much simpler (given that it _only_ emulates x86 and some peripherals). Here, I write about both those attempts of mine (and their results).
 
 ## QEMU
 
-You can find all the work I did at [this GitLab Repo - `qemu_for_redox` branch](https://gitlab.redox-os.org/enygmator/qemu/-/tree/qemu_for_redox).
+You can find all the work I did at [`enygmator/qemu` GitLab Repo - `qemu_for_redox` branch](https://gitlab.redox-os.org/enygmator/qemu/-/tree/qemu_for_redox).
 
 QEMU has its own build system based on `MAKEFILES` and a meta-build system based on `meson` that is capable of compiling to Linux, MacOS and Windows targets. I created a new target called `redox`, which disables all features of QEMU by default and also the `tests` in `meson.build`. Then I manually enable only a subset of the features (for `qemu-system-x86_64`) to run.  
 
@@ -64,28 +71,28 @@ I'll have an update on QEMU when `qemu-system-x86_64` runs for the first time.
 
 ### TLDR; Working on the code
 
-The `recipe.sh` file is [here - on the `qemu_on_redox` branch of `cookbook` repo in MY GitLab](https://gitlab.redox-os.org/enygmator/cookbook/-/blob/qemu_on_redox/recipes/qemu/recipe.sh).
+The `recipe.sh` file is [here - on the `qemu_on_redox` branch of `enygmator/cookbook` repo](https://gitlab.redox-os.org/enygmator/cookbook/-/blob/qemu_on_redox/recipes/qemu/recipe.sh).
 
 Feature configuration is done in the `recipe.sh` file, which is used to build the QEMU package for Redox, which can later be installed on Redox. The `recipe.sh` file currently depends on the QEMU source being located at `try_redox/redox_apps/qemu-7.0.0` (where the Redox OS repo is located at `try_redox/redox`). This is used to run the VPATH build (done by commands in `recipe.sh`).
 
-If you want complete instructions on compiling `qemu-7.0.0` on Redox OS, you'll find them in the `README.md`, [here](https://gitlab.redox-os.org/enygmator/cookbook/-/tree/qemu_on_redox/recipes/qemu). DO contact me if you have questions, suggestions or face issues. I may be able to help. (or you can file an issue in one of MY repo links - `cookbook` or `qemu`)
+If you want complete instructions on compiling `qemu-7.0.0` on Redox OS, you'll find them in the `README.md`, [here - on the `qemu_on_redox` branch of `enygmator/cookbook` repo](https://gitlab.redox-os.org/enygmator/cookbook/-/tree/qemu_on_redox/recipes/qemu). DO contact me if you have questions, suggestions or face issues. I may be able to help. (or you can file an issue in one of repo links - [`cookbook` - GitLab Issues](https://gitlab.redox-os.org/enygmator/cookbook/-/issues) or [`qemu` - GitLab Issues](https://gitlab.redox-os.org/enygmator/qemu/-/issues))
 
 
 ## Bochs
 
-[Bochs 2.7](https://bochs.sourceforge.io/) seemed like a good emulator to port, as it is so much more lighter than QEMU, due to the fact that it only emulates x86 ISA with a certain number of peripherals, and doesn't support KVM either. So, I started to work on porting it [in this repo](https://gitlab.redox-os.org/enygmator/bochs/-/tree/bochs_on_redox).
+[Bochs 2.7](https://bochs.sourceforge.io/) seemed like a good emulator to port, as it is so much more lighter than QEMU, due to the fact that it only emulates x86 ISA with a certain number of peripherals, and doesn't support KVM either. So, I started to work on porting it [here - on the `bochs_on_redox` branch of `enygmator/bochs` on GitLab](https://gitlab.redox-os.org/enygmator/bochs/-/tree/bochs_on_redox).
 
 Bochs uses `Makefile` for it's build system, and similar to QEMU, it uses the VPATH build method (which ensures that all build files are worked on in a separate directory that you can manually specify, so that the source repo isn't touched in any manner at all).
 
 My experience with porting QEMU (incompletely) came in very handy, as within 6 hours I actually was able to get Bochs to statically compile! albeit with some 'extra' features turned off. There were some `#include` changes with respect to the `SDL` and `SDL2` libraries and some 'tricky' method to force bochs to compile statically (since it's build system was not doing it correctly). You can view the changes in the repo link above.
 
-I then configured the `make` procedure and `install` sections of the `recipe` and booted into Redox and ran bochs with a super minimal "bootloader + kernel" that I had written back in 2020 (my first "OS project" ☺️) loaded onto a 'hard drive file' and configured bochs emulator using a `bochs.src` _bochs configuration file_.... and it ran! You can see "stage 2" of my kernel in the below image:
+I then configured the `make` procedure and `install` sections of the `recipe` and booted into Redox and ran bochs with a super minimal "bootloader + kernel" that I had written back in 2020 (my first "OS project" 😊) loaded onto a 'hard drive file' and configured bochs emulator using a `bochs.src` _bochs configuration file_.... and it ran! You can see "stage 2" of my kernel in the below image:
 
 <img class="img-responsive" src="/img/screenshot/bochs_redox_demo_1.png" alt="Bochs GUI running on Redox" />
 
 <br/>
 
-At a later point, I'll enable and work on more features to make bochs work better. Though, the priority will always be QEMU as it is **so much more** important in terms of common/industrial usage and the functionality it offers, and a priority for emulation and ██████████████.
+At a later point, I'll enable and work on more features to make bochs work better. Though, the priority will always be QEMU as it is **so much more** important in terms of common/industrial usage and the functionality it offers, and a priority for emulation and Hardware-assisted Virtualization (which I'm working on as part of RSoC 2022).
 
 <br/>
 
@@ -98,13 +105,14 @@ At a later point, I'll enable and work on more features to make bochs work bette
 
 ### TLDR; Working on the code
 
-The `recipe.sh` file is [here - on the `bochs_on_redox` branch of `cookbook` repo in MY GitLab](https://gitlab.redox-os.org/enygmator/cookbook/-/blob/bochs_on_redox/recipes/bochs/recipe.sh).
+The `recipe.sh` file is [here - on the `bochs_on_redox` branch of `enygmator/cookbook` on GitLab](https://gitlab.redox-os.org/enygmator/cookbook/-/blob/bochs_on_redox/recipes/bochs/recipe.sh).
 
 Feature configuration is done in the `recipe.sh` file, which is used to build the Bochs package for Redox, which can later be installed on Redox. The `recipe.sh` file currently depends on the Bochs source being located at `try_redox/redox_apps/bochs` (where the Redox OS repo is located at `try_redox/redox`). This is used to run the VPATH build (done by commands in `recipe.sh`).
 
-If you want complete instructions on compiling `bochs` on Redox OS, you'll find them in the `README.md`, [here](https://gitlab.redox-os.org/enygmator/bochs/-/blob/bochs_on_redox/README.md). DO contact me if you have questions, suggestions or face issues. I may be able to help. (or you can file an issue in one of MY repo links - `cookbook` or `bochs`)
+If you want complete instructions on compiling `bochs` on Redox OS, you'll find them in the `README.md`, [here - on the `bochs_on_redox` branch of `enygmator/bochs` on GitLab](https://gitlab.redox-os.org/enygmator/bochs/-/blob/bochs_on_redox/README.md). DO contact me if you have questions, suggestions or face issues. I may be able to help. (or you can file an issue in one of repo links - [`enygmator/cookbook` - GitLab Issues](https://gitlab.redox-os.org/enygmator/cookbook/-/issues) or [`enygmator/bochs` - GitLab Issues](https://gitlab.redox-os.org/enygmator/bochs/-/issues))
 
 
+<br/>
 <br/>
 
 # Epilogue
@@ -115,17 +123,20 @@ Contributing to Redox OS's future is a lot of fun. I learnt about the different
 
 I shall soon write my next post (as an RSoC 2022 student), so hang on tight people!
 
+
 ## Until later!
 
-If you've reached till this point, thank you for reading. I shall be coming back another time with more exciting updates on this undertaking.
+If you've reached till this point, thank you for reading. I shall be coming back sometime ~~next month~~(UPDATE1) with more exciting updates!
 
-Until then, do keep safe, hale and hearty.  
-Bye! ❤❤❤
+Until then, keep well.  
+Bye! 💕💕💕
+
+<br/>
 
-You can find me on:
-- [Redox OS - GitLab](https://gitlab.redox-os.org/enygmator/) 
-- Redox OS - Mattermost chat - username: `enygmator`
-- [GitHub](http://github.com/enygmator/)
-- [LinkedIn](https://www.linkedin.com/in/ttarunaditya/)
-- [Twitter](https://twitter.com/Enygmator)
-- [Encrypted chat on Matrix (like _Element chat_)](https://matrix.to/#/@enygmator:matrix.org)
+You can find me on:  
+- [Redox OS - GitLab](https://gitlab.redox-os.org/enygmator/)  
+- Redox OS - Mattermost chat - username: `enygmator`  
+- [GitHub](http://github.com/enygmator/)  
+- [LinkedIn](https://www.linkedin.com/in/ttarunaditya/)  
+- [Twitter](https://twitter.com/Enygmator)  
+- [Encrypted chat on Matrix (like _Element chat_)](https://matrix.to/#/@enygmator:matrix.org)  

content/news/rsoc-2021-qemu-1.md

@@ -12,33 +12,46 @@ This is my first time writing on the Redox OS blog! In fact, it's my first time
 
 I am one of the **RSoC (Redox Summer of Code) participants (students) this year (2021)**. As part of RSoC, I have been working on **porting QEMU to Redox OS** for the past one month. This is my first post detailing my project in order to give you an insight into it and what the future might hold.
 
-## Updates
+<br/>
+
+### Updates
 
 If I make any updates to this post (which is sometimes helpful for people reading it in the future), I shall make note of it here and st the same time, ~~scratch out~~ older content and replace it with something like "UPDATE1: ...".
 
 - UPDATE1: 10 Jul 2022
+- UPDATE2: 18 Jul 2022
 
 ## About the project
 
 QEMU (Quick emulator), runs on many OSes from various Linux distributions to macOS and Windows.
 
-> _QEMU: It is an emulator that is capable of emulating various hardware (like Intel or ARM processors) within a software environment (your OS) in order to help run software, that runs only on the above-mentioned hardware. It is capable of emulating entire machines, so that you can test your software in the emulator rather than using the actual machine._
+<br/>  
+
+_QEMU: It is an emulator that is capable of emulating various hardware (like Intel or ARM processors) within a software environment (your OS) in order to help run software, that runs only on the above-mentioned hardware. It is capable of emulating entire machines, so that you can test your software in the emulator rather than using the actual machine._
+
+<br/>
 
 I am porting it so that it is capable of running on Redox OS. 
 
+<br/>
+
 ### "Why?", you ask?
 
 It's because, I feel that every OS developer has the dream of making their OS **"self-host" capable**. What I mean by that is, currently, the Redox OS toolchain uses a number of supported Linux distributions to compile the OS code into a binary, after which it can either be run on QEMU or on one of the supported machines. What you CANNOT do (at the moment), is compile and run Redox OS on QEMU **ON** Redox OS. **UPDATE1: The ability to compile Redox ON Redox seems to be close at hand**  
 
-But making it self-host capable _and usable_ requires the tooling around it to be well designed as well. In fact, tooling around a new software/hardware that makes it easier to work with pretty much decides how successful the software/hardware will be. (as suggested by Linux Torvalds, where he said that he loves `x86` more than `arm` because of the amazing ecosystem around x86 that has been built over the past decades).
+But making it self-host capable _and usable_ requires the tooling around it to be well designed as well. In fact, tooling around a new software/hardware that makes it easier to work with pretty much decides how successful the software/hardware will be. (as suggested by Linus Torvalds, where he said that he loves `x86` more than `arm` because of the amazing ecosystem around x86 that has been built over the past decades).
 
 So, once QEMU is able to run on Redox OS, it becomes an important tool in the ecosystem for emulator-based-development. And even if it isn't practical, one **could** run Redox OS on QEMU (on Redox). And if we combine that with the future ability to compile Redox OS on Redox OS, then we can theoretically eliminate the need for another operating system, as **development, compilation and testing/running Redox OS can be accomplished on Redox OS itself**.
 
 The process of manipulating the QEMU codebase in order to get to run on another OS/System is called porting.
 
+<br/>
+<br/>
 
 # Porting QEMU
 
+This section talks about the attempt to rework the QEMU codebase (and build procedure) to get it to compile and work on Redox OS.
+
 ## The basics in Layman's terms
 
 When you write code, it gets built by the build system (i.e. "converted") into a file that contains a specific format of 1s and 0s which is called a **binary** file. The binary generated is different for different OSes, since the features/capabilities and the way the OSes _execute_ your code differs. This also means that the code of complex programs varies from OS to OS, which requires you to write slightly different code (for the same program) for each OS you want to run your program on.
@@ -54,15 +67,25 @@ The advantage here is that, even though Redox OS is **not fully POSIX compatible
 The thing about QEMU is that it has a **HUGE** number of features that do all sorts of things from emulating peripheral devices to just making the emulated OS run faster. And many of these features heavily depend upon underlying OS support.  
 So, my first move is to just get a part of QEMU running on Redox OS. By "part", I mean that I am **not trying to get QEMU running with all its features enabled**, which would require me to heavily modify Redox OS by introducing new features (which might take weeks or months). Instead, I am **disabling all the features that the "core" of QEMU doesn't need** while it's emulating a simple `x86` machine with a `512 byte` bootloader so that I can enable the rest of the features once I get the core running, making sure that **something** is usable at all times, instead of trying to get it all running at once.
 
+<br/>
+
 ### The Process
 
 First, I got myself acquainted with the entire Redox OS codebase, since I will not only be working with QEMU source code, but also with `relibc` (Redox OS's `libc` implementation) and potentially even kernel code.
 
-> I want to write more about how the entire `Redox OS` code base is organized (in order to help newcomers make sense of it), but I guess it would be more appropriate to make it part of the documentation to be done on a future date. This will encourage more contributors.
+<br/>
+
+_I want to write more about how the entire `Redox OS` code base is organized (in order to help newcomers make sense of it), but I guess it would be more appropriate to make it part of the documentation to be done on a future date. This will encourage more contributors._
+
+<br/>
 
 I first took time to get a `recipe.sh` recipe ready, in order to instruct the Redox OS build system to build QEMU using the QEMU build system, by passing in the required parameters as input (which include all the correct environment variables like those that point to the cross-compiler, as we are compiling FROM Linux to run ON Redox). Currently, the location of the QEMU repo is fixed at the same level as the `redox` folder inside `try-redox` (refer Redox OS documentation).
 
-> _NOTE: `recipe.sh` is deprecated and will be soon replaced by me with a `recipe.toml`._
+<br/>
+
+_NOTE: `recipe.sh` is deprecated and will be soon replaced by me with a `recipe.toml`._
+
+<br/>
 
 I am using ~~**QEMU `6.0.0`**~~ **(UPDATE1: QEMU `7.0.0`)** as the base QEMU code which I will be modifying to support Redox OS. I initially thought of just letting a switch called `linux` remain on inside of QEMU's `configure` since `Redox OS` was similar to Linux and though this didn't prove to be a lot of trouble, it soon became evident to me as to the relatively bigger than anticipated feature gap that exist between Linux and Redox. But it isn't surprising as **Redox is NOT a Linux clone**.
 
@@ -74,16 +97,22 @@ Since I cannot immediately verify the successful execution of the code I modifie
 
 > _~~For the list of features that I enabled/disabled or for code that I replaced and removed, **refer [this GitLab Wiki](https://gitlab.redox-os.org/enygmator/qemu/-/wikis/Features-of-QEMU-on-Redox)**. Note that I haven't documented every change; just the major ones.~~ **UPDATE1: this link hasn't fully updated with all the purported info**_
 
+<br/>
+
 ### The progress
 
 At the moment, I still haven't run QEMU on Redox as each time I compile, there prop up errors that need to be rectified ~~(which I detailed on the GitLab Wiki)~~(**UPDATE1**) and then I recompile to find some more errors. This may not be the best method, but I feel it's the most efficient one I have in my hand. And since I don't know exactly how many incompatibilities are present, it is difficult to accurately estimate the progress of the project. I am hoping to discover a method that would help me resolve problems faster. (Like using `strict` analysis using `VS Code` that can reveal missing functions beforehand. I have already created a `VS Code` config that uses the `Redox OS` cross-compiler)
 
+<br/>
+
 ### Some more technical stuff
 
-1. One of the major missing features in QEMU is KVM as it needs virtualization support on Redox. 
-2. ~~I am not sure about this, but `sdl` seems to be a supported library on `Redox OS`, so I'm guessing QEMU will work just fine with `GUI` as it supports `sdl` rendering~~ **UPDATE1: Redox DOES have static linking with `sdl`, with some known issues that are documented in the post, ["Bochs on Redox OS (and QEMU - Part 2)"](https://redox-os.org/news/bochs-qemu-2/)**.
+1. ~~One of the major missing features in QEMU is KVM as it needs virtualization support on Redox.~~ **UPDATE2: Virtualization support is coming to Redox OS _soon_! The feature is called `revirt` and the project plan and implementation details are documented in the post - ["RSoC 2022: Revirt - Virtualization on Redox OS"](https://redox-os.org/news/rsoc-2022-revirt-1/)**
+2. ~~I am not sure about this, but `sdl` seems to be a supported library on `Redox OS`, so I'm guessing QEMU will work just fine with `GUI` as it supports `sdl` rendering~~ **UPDATE1: Redox DOES have static linking with `sdl`, with some known issues that are documented in the post - ["Bochs on Redox OS (and QEMU - Part 2)"](https://redox-os.org/news/bochs-qemu-2/)**.
 3. There seem to be many features in QEMU which optimize the execution of `guest` code, I have chosen to disable most of them either due to incompatibility (I guess `membarrier` is one of them) or in order to just make to build simple enough to run without any failures the first time.  
 
+<br/>
+<br/>
 
 # P.S. (Post Script)
 
@@ -93,17 +122,22 @@ During the course of this project (which still has so much exciting stuff to be
 
 Also, I shall make sure to keep the next blog post **more technical** as I wanted to make this one more introductory, but none the less, if you are still seeking technical information, ~~**refer the Gitlab WIKI link mentioned before**~~ (**UPDATE1**).  
 
+<br/>
+<br/>
+
 # Until later!
 
 If you've reached till this point, thank you for reading. I shall be coming back sometime ~~next month~~(UPDATE1) with more exciting updates!
 
-Until then, do keep safe, hale and hearty.  
-Bye! ❤❤❤
+Until then, keep well.  
+Bye! 💕💕💕
+
+<br/>
 
-You can find me on:
-- [Redox OS - GitLab](https://gitlab.redox-os.org/enygmator/) 
-- Redox OS - Mattermost chat - username: `enygmator`
-- [GitHub](http://github.com/enygmator/)
-- [LinkedIn](https://www.linkedin.com/in/ttarunaditya/)
-- [Twitter](https://twitter.com/Enygmator)
-- [Encrypted chat on Matrix (like _Element chat_)](https://matrix.to/#/@enygmator:matrix.org)
+You can find me on:  
+- [Redox OS - GitLab](https://gitlab.redox-os.org/enygmator/)  
+- Redox OS - Mattermost chat - username: `enygmator`  
+- [GitHub](http://github.com/enygmator/)  
+- [LinkedIn](https://www.linkedin.com/in/ttarunaditya/)  
+- [Twitter](https://twitter.com/Enygmator)  
+- [Encrypted chat on Matrix (like _Element chat_)](https://matrix.to/#/@enygmator:matrix.org)