Commits (28)
title: "Rsoc: Ion as a Library, week 5"
date: 2019-07-07T11:17:42-04:00
author: "AdminXVII"
# Wrapping up the Ion as a library project
## What is Ion
Ion is a fast, modern shell. Its powerful yet simple syntax almost eradicates the need for GNU's coreutils, by providing integrated, super-fast _methods_. With the methods providing native speed and modern features like non-forking expansion, Ion beats Dash in terms of performance in a wide variety of scenarios.
Features like exiting with error on command not found, invalid expansion and explicit typing in function signatures, as well as using Rust as its base language means a whole host of user and application errors simply can't happen, which POSIX shells lack.
Ion takes the Rust's promise for fast, simple, secure programming and introduces it to the world of shells.
## The work done as part of this project
It is now possible to embed Ion in any Rust application. Ion takes any Read instance and can execute it (so yes, it is possible to run Ion without ever collecting the script's binary stream). It takes care of expanding the input and managing the running applications in an efficient manner, with a comprehensive set of errors.
Ion is now the rust-based, pipe-oriented liblua alternative.
# This week's progression
## Summary of the work done
- Create a concrete example of the Ion API with an alternative to GNU's parallel. See section below
- Refactor the calc builtin's input methods
- Use liner for interactive input for better control support
- If stdin is not a tty, don't print the interactive help text & input
- Improve the shell options
- Use a more descriptive name for the option to control the tty
- Remove configuration related only to the interactive shell
- Give users more control
- Add callbacks for background pipeline events
- Add support for default shell redirection
- Add metadata to prepare for a release to crates.io
- Reduce blocking forking (see section below)
- Improve testing
- Add the cracauer shell test suite for proper error handling
- Use xargs to run all tests in parallel
- Only test for the cli parameters in the CI, not with the default test method
- Misc
- Improve the interactive completion
- Generate man pages for the builtins automatically, as well as the builtin page in the Ion Manual
## Performance improvement: non-forking sub-shell expansion
During shell expansion, it is quite common to deal with sub-shells (e.g. `echo $(echo a)`). Dash and Bash's way of doing it is forking the shell and redirecting its output to a pipe. The problem is that forking can be quite time-consuming, and the main process is effectively blocked, making the fork absolutely not useful.
For example, given the subshell expansion for `echo $(echo a)`, this is the two timelines:
main process: | part of expansion | forking | waiting | rest of expansion |
expansion fork: | forking | executing builtin |
main process: | part of expansion | executing builtin | rest of expansion |
Forking is quite cheap on Linux due to copy-on-write paging, yet it is still cheaper to not fork at all.
Another problem is that, in POSIX shells' case, the main process will be periodically woked-up to read partially the expansion's output. Since there is nothing to do with this read until the pipeline exits, this means the process keeps taking CPU time where it should not. Ion in contrast lets the pipe fill up and only read once, at the end of the pipeline execution, leaving your CPU in peace.
With a more down-to-earth approach, this means
for _ in {0..10000}
echo $(echo a) > /dev/null
now executes 25x faster than it did on a Galago Pro 3 with i7-8565U (turbo disabled). It is also 10x faster than the dash equivalent, and 30x faster than bash's.
## Ion as a library's first useful application: Redox's Parallel
The first application based upon Ion as a library has now been created! Redox's Parallel is an alternative to GNU's one, with a more ergonomic and faster shell at its core. It is already usable and released as a beta version. You can check out the code at https://gitlab.redox-os.org/redox-os/parallel (GPL).
## The next steps
- Once alpha testing phase is done, release to crates.io
- Improve expansion by reducing allocations
- Adapt more tests from the cracauer test set, and automate them
- Create non-forking function execution
- Explore the possibility for non-forking builtins
- Adapt the nix crate for redox use, or move away from it
title: "Rsoc: Improving Ion's UX, week 1"
date: 2019-07-015T17:58:24-04:00
author: "AdminXVII"
# Adding Nix support for Redox (not yet merged)
The libc crate was updated to use all the latest functionalities from Redox OS & Relibc ([PR](https://github.com/rust-lang/libc/pull/1438)). In the process, a patch was proposed to nix to support Redox OS ([PR](https://github.com/nix-rust/nix/pull/1098)). This means a whole host of applications that used nix will work out of the box on Redox OS! (A dependency upgrade will still be needed)
# Summary of the work done
- Add a return keyword for exiting functions with a code
- When disowning a process, redirect stdout & stderr to /dev/null
- Reduce lock contention on ctrl-c
- Fix a bug where the command expansion wouldn't reset the default pipes when an expansion error occured. Thanks to aleksator for filing the bug and helping me find the culprit.
- Bring back compilation on Redox by adding Nix (the crate, not the distro) support for redox
# The next steps
- Add plugins
- Add user-defined autocompletion
- Improve expansion by reducing allocations
- Adapt more tests from the cracauer test set, and automate them
- Create non-forking function execution
- Explore the possibility for non-forking builtins
title: "Rsoc: Improving Ion's UX, week 2"
date: 2019-07-21T12:12:30-04:00
author: "AdminXVII"
# Keyword of the week: plugins
## Summary of the work done this week
- Converting 36 plugins from the Oh-My-Zsh repo to Ion's syntax. Aliases and functions are provided as one-on-one conversion from the zsh alternative to provide a smoother inboarding experience.
- Creating a auto-documenting gitlab pages website to help users select plugins and making the experience as straightforward as possible.
## Converting the plugins
Userland plugins are great tools to make the user's workflow faster with a near-zero onboarding involvment. They allow users to have commands tailored to their workflow without much involvement from the shell and without impacting the maintainability of the rest of the system. As such, it is really a must have for a pleasing interactive shell.
Ion has now its own official repo for userland plugins! [Ion-plugins](https://gitlab.redox-os.org/redox-os/ion-plugins) aims to provide a simple, opt-in, centralized set of components to improve Ion's interactive UX. 36 plugins taken from the great [oh-my-zsh](https://github.com/robbyrussell/oh-my-zsh) repository are already converted to Ion's syntax and more are on the way (since Ion does not currently has command autocompletion, only aliases, functions and startup scripts are converted).
## Auto-documenting plugins
When choosing plugins, users want to know what is provided. They also like to have this info without having to open two webpage to access each plugin's README. On the other side, it's quite a hassle for contributors to have to update the README after each update.
Ion-plugins has the solution! After each commit, the aliases and functions are extracted from the source code of each plugin, along with the top-most comment for describing the plugin. With this info collected, a [global documentation webpage](http://redox-os.pages.redox-os.org/ion-plugins/) is generated, providing users with all the info needed and sparing contributors from the boring work of copying info from the source to the man page.
Lastly, users can select the plugins they want and the configuration needed is automatically generated. This does not mean the configuration is complex, quite the opposite. Everything is based upon a simple sourcing, making the core working of the plugins obvious. Simple, standard means for powerful deeds.
title: "Rsoc: Improving Ion's UX, week 3"
date: 2019-07-28T18:26:35-04:00
author: "AdminXVII"
# Plans for better autocompletion: Shellac
Ion's biggest missing feature at the moment is proper autocompletion support. It's quite annoying that even day-to-day commands like git can't be autocompleted. The problem is that autocompletion takes a lot of time from the community, so ideally autocompletion for commands should be shell-agnostic.
The solution is creating something akin to a language server. That way, you avoid (mostly) the N ⋅M problem, since you can share the definition files with all the other shells. By creating a server that replies over sockets also means you avoid the time necessary to start up the binary, and you allow caching mechanism. That means lower latency, so increased user experience.
It turns out that someone already has worked on the subject. The Oil shell got in touch with multiple shells including ZSH, Closh, Elvish to create what is to be called [Shellac](https://github.com/oilshell/oil/wiki/Shellac-Protocol-Proposal). This is still an early proposal, but this seems quite interesting for the Ion shell.
The improvements seeked are categorized below.
## Faster gathering of definitions
Adding definitions for every binary is a long process. The problem is that if you don't have good autocompletion, you don't get much userbase. This in turn means that your ecosystem is much smaller, so creating completion is much longer. The cycle is quite bad, and sharing completion with all shells should greatly reduce it.
It should also be easier to generate completion files from man pages, so that means small utilities should be able to provide completion to users a lot more quickly and without having to think about each and every shell out there.
## Fast completion
In Bash and ZSH's cases, completion is done via functions. That is in fact quite slow, since the shell need to interpret everything rather than run in bare metal. There is also the fact that shells tend to call binaries where it should be simple to run it directly in Rust/C. In 99% of the cases, static completion is sufficient and native speed should be expected. For the rest, it should be easy to avoid calling executable unless dynamic completion is needed (ex: git branches). Ideally latency should be minimal as to be unnoticeable on a mid-end computer, where now zsh can sometimes take seconds to complete. If shells can agree to run the shellac server as a daemon and communicate with unix domain sockets instead of standard pipes, the startup cost could even be avoided completely, as well as amortizing of buffer allocations. It should also be possible to implement caching facilities to make it real-time.
No actual implementation exists as of today, but this holds great promises for completion speed.
## Avoiding version skew
When new options for binaries are created, it takes a long time for it to reach the end user's completion system. Since completion is generally provided as third party repo, someone must take care of updating the definition for each shell in those completion collection. A shell agnostic completion facility with a formal grammar means distro could in fact pack the definition with the binary, just like it's man page. That means completion would always be up to date! If there exists tools to convert easily a man page to the grammar, that also means upstream maintainers will be more willing to update it than a manual update. No more version skew!
## Shell diversity
Right now it's hard for shells to get started, because of how much they must implement before gaining a day-to-day userbase. A shell-agnostic completion server allows a greater diversity of shells to start being implemented, which can never be bad. It also means Ion can benefit from the user base of ZSH, Fish, Oil, Elvish _et al._ for developing completion in case the proposition is accepted.
# Issues fixed and upgrades done this week in Ion
- Update the minimal rust version to build Ion to 1.35.0
- Follow rust's decisions and replace trim_{left,right} with trim_{start,end}
- Fix the set builtin
- Rename the calc builtin to math to avoid interfering with GNU's calc
- Fix a potential panic
- Make Ion work for Redox
- Add an integrated help page for the math (previously calc) builtin
title = "RSoC: Implementing ptrace for Redox OS - part 3"
author = "jD91mZM2"
date = "2019-07-07T08:46:13+02:00"
Table of Contents:
- [Merged!](#merged)
- [Accessing the memory of another process](#accessing-the-memory-of-another-process)
- [Accessing floating point registers](#accessing-floating-point-registers)
- [So... What's next?](#so-what-s-next)
[Pictures of ptrace :)](https://imgur.com/a/rdoOc7q)
# Merged!
Before I dive in to this week's actions, I am pleased to announce that
all the last weeks' work is merged! However, something went
hilariously wrong with [the
and the merge was [re-merged with a new
but however, with Jeremy as author. Trying to fix one mistake lead to
another, but at least the commit message now makes sense, so I'm not
complaining :)
This merge means you can now experiment with basic ptrace
functionality using only basic registers and
`PTRACE_SYSCALL`/`PTRACE_SINGLESTEP`. I have already opened the second
PR in the batch: [Ptrace memory reading and floating point registers
which will supply the "final bits" of the *initial* implementation,
before all the nitpicking of final concerns can start (not to
underestimate the importance and difficulty of these nitpicks - there
are some areas of ptrace that aren't even thought about yet and those
will need tending to)! I will comment on these changes in this blog
post, as there are some interesting things going on!
# Accessing the memory of another process
This is perhaps the most interesting change of all. I had to learn
about how processors handle memory: Each process/context has it's own
"page table" which translates parts (here "pages") of "virtual" memory
(in Linux and Redox, 4096 bytes long) to parts (here "frames") of
real, "physical" memory. This means that the pointer `0x12345` on one
process will point to different data than `0x12345` will for another
process. An exception is kernel memory, but only because all processes
map that memory each syscall to a pre-determined address (the virtual
address is still different from the physical one, so it's not much of
an "exception" really).
This paging memory is often portrayed as particularly complicatied -
and it probably is - but luckily I only had to touch the basics.
My first *successful* approach to this was by attaching to the target
process' page table and somehow save the needed memory, perhaps by
using the stack since kernel memory is hopefully mapped in both
processes. That worked, but because the array was on the stack it
could only transfer a limited set of bytes at a time.
![First successful attempt](https://i.imgur.com/nGZvXdn.png)
Then I talked to Jeremy, who told me the kernel would not support swap
memory for a while, so a better way would be to find out the physical
frame of the virtual page containing the requested address, and map it
to the current process as well, translating all addresses. This was
# Accessing floating point registers
This was really easy for one reason: Jeremy also said the kernel will
*never* touch the floating point registers of a context. So no new
code had to be written to preserve these registers, they are already
saved in the context switch. Since I also made the `syscall`
floating-point struct memory-compatible with the x86\_64 standard,
[the end
was basically just
unsafe { *(self.fx as *const FloatRegisters) }
with a bit of code to discard all the reserved bits to make sure the
user doesn't modify them.
An interesting bit here, though, is that the processor actually uses
80-bit floats. Therefore I got to have some fun to [interpret these in
the strace library](https://gitlab.redox-os.org/redox-os/strace-redox/blob/1579b98dcdacdec3477857bab1c0c9e64a5b117f/src/f80.rs)
# So... What's next?
First, there are a few concerns I have myself which I will need to
attend to in a later PR:
Help would be welcome over at the
[RFC](https://gitlab.redox-os.org/redox-os/rfcs/merge_requests/14) (be
warned: Redox GitLab does not seem to send notifications!), or any
other way to contact me: Hoping I won't get spammed to bits, I'll
leave my contact details also (I really need help with these
- Email: [me@krake.one](mailto:me@krake.one)
- Discord: jD91mZM2#1033
- IRC (Freenode/Mozilla): jD91mZM2
I've already gotten some help on the RFC which I am very, very happy
about. People with Linux ptrace experience give a lot of valueable
feedback about the shortcomings of the API and how the Redox API can
be designed better. Even though the basic implementation is done, it's
intentionally programmed so the biggest lowest-level bits are separate
from the main interface, which can therefore be replaced. It's not too
Secondly, I'm working on a compatibility layer for ptrace over at the
[ptrace branch of
title = "RSoC: Implementing ptrace for Redox OS - part 4"
author = "jD91mZM2"
date = "2019-07-13T11:07:22+02:00"
Table of Contents:
- [Merged (again)!](#merged-again)
- [redox-nix update](#redox-nix-update)
- [Relibc: ptrace compatibility](#relibc-ptrace-compatibility)
- [Issue #1](#issue-1-the-waitpid-interaction-with-ptrace-is-vastly-different-on-redox)
- [Issue #2](#issue-2-tracemes-are-not-implemented)
# Merged (again)!
Once again, [last weeks
was merged, which means the full ptrace feature was merged, and it's
time to start tackling the final issues which I have delayed for so
long. But, before that, I decided to try to get some basic [ptrace
compatibility in
so we could see just how far away software like gdb is from being
ported, and what concerns I haven't thought about yet.
# redox-nix update
That said, I took a little break from the madness, to instead lay my
focus on another interesting problem: Newer
[redoxer](https://gitlab.redox-os.org/redox-os/redoxer) couldn't be
compiled using [carnix](https://nest.pijul.com/pmeunier/carnix),
because of some dependency that used a cargo feature carnix didn't
support. Let me first explain what carnix is, and why this is a
The [Nix](https://nixos.org/nix/) package manager, in all it's glory,
is all about reproducibility. This means one should not have any
compilation cache that may make compilation work for one person but
not for another. This aspect is not dissimilar from the Redox package
manager, as it first downloads the source in `source/`, and then
copies it to `build/` before each build. (I'll not dive into details
in whether Redox will eventually get a more interesting package
manager, but the answer is yes, hopefully!)
Consider this: You are building a crate that requires the `openssl`
crate. You install the openssl C dependency and forget about it. Later
you decide to build a package for this crate. If you have compilation
cache, you might not have to rebuild the openssl crate, even if you
use sandboxing or otherwise uninstall the openssl C dependency just to
see whether it's required or not. Therefore you may forget to add that
as a C dependency, and break reproducibility.
Of course you don't *actually* want to rebuild everything on each tiny
change or update, though... That's why `carnix` throws cargo in the
trash can and rolls it's own solution: Create a package for each
dependency crate, and manually compile them with `rustc`. Throwing
cargo in the trash can, along with not being great for the environment
(one should really recycle things), also throws all the cool
cargo-specific there as well and leaves `carnix` to constantly
re-implement each feature, which of course breaks a lot of things.
This problem is why I started "yacna", Yet Another Cargo->Nix
Alternative. It aims to use `cargo` as a library to generate all the
needed commands using the completely standard way, and then modify the
commands only as little as possible to make things work. It's not
pretty, but locally I have managed to compile a simple crate with like
one dependency. And it supports feature flags and everything out of
the box, thanks to re-using Cargo. This is what I've spent much of the
week on, and thus I won't have too much to give that is Redox-related.
# Relibc: ptrace compatibility
The relibc `ptrace` implementation will always be a best-effort one,
as due to the nature of Redox's vastly different (read: superior)
ptrace API (which is more similar to the `/proc` filesystem almost
every experienced ptrace user keeps talking about), it'll always be a
hellish compatibility layer which keeps track of most things in
userspace. Note that Redox will, at least as long as I get anything to
say about it, never implement useless or otherwise ptrace "features"
for compatibility with Linux in the kernel. That's why it'll be very
difficult to get this to work in relibc.
The first things I did was to [implement a ptrace wrapper for
which was of course easy when the function and the syscall map 1:1.
Next, I had some fun and implemented the
synchronization primitive. It's interesting to implement something
which is so important you get right, as locks could be attempted
simultaneously and the whole goal is to handle that gracefully. I
spent some time to unify the Mutex and the Once code, as they are
surprisingly similar in a way.
Next up: Stop doing other things, although needed, and get to work!
## Issue #1: The waitpid interaction with ptrace is vastly different on Redox!
This had to be worked around by adding a specific ptrace handler, as
// First, allow ptrace to handle waitpid
let state = ptrace::init_state();
let sessions = state.sessions.lock();
if let Some(session) = sessions.get(&pid) {
if options & sys_wait::WNOHANG != sys_wait::WNOHANG {
let _ = (&mut &session.tracer).write(&[syscall::PTRACE_WAIT]);
res = Some(e(syscall::waitpid(pid as usize, &mut status, (options | sys_wait::WNOHANG) as usize)));
if res == Some(0) {
// WNOHANG, just pretend ptrace SIGSTOP:ped this
status = (syscall::SIGSTOP << 8) | 0x7f;
res = Some(pid as usize);
ptrace should also add `WUNTRACED` when in a session, but that is a
problem because of the below issue.
## Issue #2: Tracemes are not implemented
Tracemes are, in my opinion, useless. The parent could just attach to
the child using its process ID. It's just problematic:
`PTRACE_TRACEME` can be called from a child process so that the
parent, without help from the kernel or the process itself, can't
really know it's in a session. And the memory is not shared in
`fork`s. So somehow, here it needs to detect that, perhaps using
[IPC](https://gitlab.redox-os.org/redox-os/ipcd). Of course though, it
can't issue a blocking `read` and/or `write` in a nonblocking `ptrace`
call, and ugh. Another alternative is using shared memory, but that
means you need to tell one single `BTreeSet` to use a different
allocator, that also somehow dynamically allocates things in a fixed
view of memory.
For now, `PTRACE_TRACEME` is a no-op, and most ptrace operations
automatically attach to the child process it's trying to operate
on. Works great, except now `waitpid` can't add `WUNTRACED`
automagically before any ptrace operation is issued on the parent. So
having it a no-op is sadly not a solution. I am almost tempted to
implement the ptrace compatibility function as a specialized scheme
daemon which by taking care of all ptrace-related calls can share
memory and therefore implement this stuff easier. But while
`PTRACE_TRACEME` is the only issue, I think I'll try sticking to a
less overkill solution.
![Image of ptrace running on Redox](https://i.imgur.com/YWwEmxM.png)
title = "RSoC: Implementing ptrace for Redox OS - part 5"
author = "jD91mZM2"
date = "2019-07-22T20:28:47+02:00"
# Introduction
This week I've decided to skip trying to get GDB working *for now*
(there are so many issues it'll take forever to solve them), and
instead decided to finally give focus to the final concerns I had
about ptrace. Most changes this week was related to getting decent
behavior of child processes, although the design feels... suboptimal,
somehow (not sure why), so I feel I must be able to improve it better
Another change was security: Tracers running as a non-root user can
now in addition to only tracing processes running as the same user,
only trace processes that are directly or indirectly children of the
tracer. In the future this can easily be allowed with some kind of
capability, but currently in Redox there isn't a capability-like
system other than the simple (but really powerful) namespacing system
which sadly I don't think can be used for this.
## Catching child processes with ptrace
The first and most pressing issue was that there needed to be a way to
catch child processes (also threads, Redox makes no distinction) from
ptrace. It should be possible to both stop the child process, or let
it keep running uninterupted.
My design revolves around an event system: When a child process is
created it sends an event with the PID, as well as pauses the child
until we've determined if we want to trace it. Events interrupt any
wait, but returns `Ok(0)` to signify that the breakpoint may not have
been reached yet. Then one can `read` from the `proc:<pid>/trace` file
handle to obtain a few events at a time, which will also clear
them. Setting a breakpoint while an event is not yet read, will not
wait for that breakpoint to be reached, but rather return `Ok(0)`
With this design, you will be able to write code like this:
let mut written = tracer.write(&[PTRACE_SYSCALL])?;
while written == 0 {
let mut event = PtraceEvent::default();
// Assert we read exactly one event, as we know there must be at
// least one because of `written == 0`, and we only have storage
// for one event (room for optimization: you *can* read more at
// once)
assert_eq!(tracer.read(&mut event)?, mem::size_of::<PtraceEvent>());
// TODO: Handle event
// Retry the wait, the tracer is still running at this point and
// may or may not have reached the breakpoint we set.
written = tracer.write(&[PTRACE_WAIT])?
The implementation was actually quite simple, which was actually
somewhat of a goal for this design! (Redox is a microkernel,
Events are sent to a `VecDeque` that handles them. If the queue was
previously empty, send a `EVENT_READ` notification to the `event:`
scheme. Either way, notify the tracer file handle waiting for a
breakpoint to be reached.
I love the `WaitCondition` structure, it makes blocking and notifying
so easy :)
[Here you can see the
## Catching signals
Another change I made was related to signals. It's tricky to come up
with a design that just *feels* right, and I didn't really succeed
here. Let's start with the critera though. I'm not very used to ptrace
so I am far from an expert in even the simple task of giving criteria,
but I'll start with the most basic: The design has to be able to trace
system calls and instructions ran in signal handlers. The design also
has to be able to obtain the actual signal number, as well as run the
program normally up to a signal.
My design is requires 2 new constants:
- `PTRACE_EVENT_SIGNAL` is an event similar to the child-process event
above, but on each signal we only send the signal number.
- `PTRACE_SIGNAL` for running the program until any signal is reached,
which in case it stops *before* the signal handler is invoked. So if
the child sends a signal to itself, it will be somewhere inside the
`SYS_KILL` syscall.
Turns out these simple criteria are still pretty tricky to meet for
two reasons
Firstly, as a signal may happen inside a syscall, they will break the
unwritten rule that each syscall will raise 2 breakpoints in order. If
we take the example above about a program killing itself, we'll see
the following behavior
(pre-)syscall: SYS_KILL
(pre-)syscall: <inside the signal handler>
(post-)syscall: <inside the signal handler>
(pre-)syscall: SYS_SIGRETURN
(post-)syscall: SYS_KILL
This is annoying mostly because the program can't actually detect
whether it's the event is pre or post syscall. So it'll have to make
an intelligent guess using the signal event, which I guess could
work. But this is suboptimal and I have yet to solve this somehow.
Secondly, it is ~~possible~~, nay, *probable* that a signal is handled
using either `SIG_DFL` or `SIG_IGN`. Both of these are handled in the
kernel, and should probably not let you control. Plus, even I wanted
to allow this, the current way of setting breakpoints only works on
user-triggered syscalls (which is probably for the best). So no signal
breakpoint will be caused then, only a signal event. And you can't
abort the signal.
[Related code for signal handling
I think something to set as a goal with signals is to hit two birds
with one stone and use them for handling the `int3` instruction which
is used by debuggers to set breakpoints. That instruction causes a
`SIGTRAP` signal, something which I think could be used. Somehow there
has to be a way to inject code into a program (on Linux you search
memory maps for free space and inject machine code there, see [this
awesome article](https://www.linuxjournal.com/article/6210))
Once again, I'd be extremely thankful for any kind of feedback, even
negative. I'll go read the Linux man page on ptrace a few times now -
we're at that point.
title = "RSoC: Implementing ptrace for Redox OS - part 6"
author = "jD91mZM2"
date = "2019-07-29T07:59:35+02:00"
# Introduction
Hallå världen! We've got yet another week on our hands, and that must
surely mean another status report on ptrace? It does. If you recall
last week, I said my design just *felt* wrong, but I couldn't explain
it? This has been fixed, it now feels more right than ever, let's see
what's changed!
# Ptrace overhaul
After a tough session of brainstorming I rewrote the RFC, you can see
the process
[here](https://gitlab.redox-os.org/redox-os/rfcs/merge_requests/14/commits). This
design changes the dull `PTRACE_SINGLESTEP` to the new awesome
`PTRACE_STOP_SINGLESTEP`, with the new advantage of being a
non-exclusive operation! That's right, you can now set multiple
breakpoints with ptrace, and it will stop on the first one while
reporting which one it reached. How will it report that?
Everything is an event! Especially breakpoints, they now use events to
report which one was reached as well as any arguments that might be
useful. This lets you catch which signal caused `PTRACE_STOP_SIGNAL`
to be returned. The design was largely inspired by Linux' `PTRACE_O_*`
which is a non-exclusive way to stop at more places than just the one
you requested. And yes, this does mean I thoroughly read through
almost the entire `man ptrace` ;)
After a breakpoint is reached you can `read` from the trace file
descriptor to recieve one or more events. This lets you catch both
breakpoint and non-breakpoint events, where the non-breakpoints don't
stop the tracee. If you try to set a new breakpoint without `read`ing
all events, it will refuse to wait for the breakpoint to be set. In
non-blocking mode it won't wait anyway, although you can use
`PTRACE_FLAG_WAIT` to override this behavior.
![Image of a signal breakpoint](https://i.imgur.com/J8XkeW1.png)
Sysemu breakpoints are now more flexible. You don't have to decide
whether to emulate a syscall up-front, but rather once you recieve a
`PTRACE_STOP_PRE_SYSCALL` (the reason for separating pre- and
post-syscalls are described in the RFC) you can choose to add
`PTRACE_FLAG_SYSEMU` *to the next* ptrace operation.
## Bitflags!
Now that ptrace operations use a lot more bits, using a `u8` would be
too small. I increased this to a whooping `u64` which to me sounds
smaller than it is (I *currently* only use 16 bits anyway!). To
convert a 64-bit integer one can use `to_ne_bytes()`... I'm
kidding. You don't have to do that either!
I am experimenting with using the
[bitflags](https://crates.io/crates/bitflags) crate for creating type
wrappers around various flags in the redox_syscall crate, where ptrace
is one of them. This will not only have the benefits of using the type
system to ensure you don't mix and match different flags in an invalid
way, as well as giving a darn useful `Debug` implementation; it will
also have the benefit of letting me implement `Deref` on this struct
which will let you coerce `PtraceFlags` to `&[u8]`!
You can see if you also think the kernel gets cleaner with this change
[in this
commit](https://gitlab.redox-os.org/redox-os/kernel/merge_requests/107/diffs?commit_id=538ca49ee2de79c0f53b29782754778210671a9f). If
not, write to me and complain!
# Strace improvements
Strace has been separated into two possible compilation modes: Simple
and advanced. The simple code is what we had before, code that's easy
to read and understand where everything is synchronous. Advanced mode
is a new and exciting mode that uses the asynchronous interface (not
rust `async`, yet) to support more functions: Such as also tracing
child processes and threads.
[See the commit here](https://gitlab.redox-os.org/redox-os/strace-redox/commit/ab942219d92adcf303a933824b60c0b52924d96b)
# What's next?
Now that we're back to where we were (but with a much more scalable
system on our hands), I can get back to work implementing a way to
override signals and therefore handle `int3`. After that, a lot of the
final concerns and nitpicks I had are completed. Then there's also the
huge problem left of actually allowing the user to inject code...
See you the next week! Until then, make sure to stay hydrated in this
warmth 🍻
title = "RSoC: Implementing ptrace for Redox OS - part 7"
author = "jD91mZM2"
date = "2019-08-04T10:11:27+02:00"
# Introduction
Hallo Welt! Have you ever written a project and when it's done you
just think "Now what"? The project you've been working on for days,
weeks, maybe months I don't know, is just suddenly... without
I'm not trying to say my work here is done, but I do feel like we're
close... and that's frightening. In fact, this week has been all about
finishing things off (whenever I had time - life kept getting in the
way an annoying lot)
# Features of the week
The PR of both last week and this one was merged, so as I'm writing
this the features are actually available on Redox OS. All you have to
do is update your version!
## Ignoring signals
In [kernel commit
I made it possible to ignore signals and therefore stop signal
handlers from being invoked in the tracee, and handle it yourself
instead. A typical use for this could be a debugger that wants
`SIGINT` to just stop the tracee instead of quitting it.
I was about to add another flag, `PTRACE_FLAG_SIGIGNORE`, which would
when set just ignore any started signal. But when I thought about it
long and hard, I realized we already have something similar: sysemu!
Recent changes have already made sysemu work so you can activate it
after a pre-syscall event and it will just return with your own set
registers instead of handling the system call in the kernel. So it's
an awful lot like ignoring the system call... Aha. The flags were
merged, and [RFC commit
now has the flag `PTRACE_FLAG_IGNORE` which will handle both signals
and system calls, and is scalable.
## Handling int3
Turns out, however, that signal handlers can not actually handle
`int3`. The internal kernel function doesn't signal the process but
rather calls `exit` directly. Whether this behavior is desired or not
is something I should first ask you guys and then the project leader
Jeremy Soller. Would a process really benefit from being able to
register a signal handler to catch their own `int3`?
Until then, I made [kernel commit
use a new flag as specified by [RFC commit
let the tracer inspect the stopped tracee, and choose whether or not
to set `PTRACE_FLAG_IGNORE` for the next call and therefore not
perform the default behavior - exiting with `SIGTRAP`. Told you the
overhaul and `PTRACE_FLAG_IGNORE` designs were scalable!
# What now?
One big thing I thought I'd have to do is make memory maps readable,
like Linux' `/proc/<pid>/maps`. Turns out, unless I'm mistaken, that
most programs that wish to inject code into a tracee doesn't actually
use this but rather just writes directly to `rip` before running it,
reversing it, and resetting `rip` back again. This means I don't have
to let the user read memory maps, which is good because I don't know
if all the needed information is even available in Redox yet. And if a
user wants to inject a looot of code, they can just inject a syscall
that allocates a huge region and then write their code and jump to
I still have some things on my to-do list, but most are just testing
things or thinking through some implementation details. For example,
yesterday I resolved [an old
about catching the rsp of same-ring interrupts. This is unlikely to
ever be needed, but as I can't prove it will *never* happen I should
probably make sure it behaves correctly. A few other things I did was
[researching if modifying the CS register could be
as well as remove a TODO comment [about the safety of the memory
(which Jeremy had already gave [a thumbs up
Then I suppose I should go back to investigating if GDB can be
ported. I'm also debating whether it'd be worth it to "just" write a
debugger from scratch in rust that keeps both Linux and Redox's vastly
different systems in mind from the start. Pros would be using Rust and
that it'd be easier for me to make something basic than investigate
something as huge as GDB. Cons are that GDB is huge and I'd hardly be
able to re-implement the entirety of it. Input from anybody working on
GDB or a competitor would be absolutely great!
title = "RSoC: Implementing ptrace for Redox OS - part 8 - The finale"
author = "jD91mZM2"
date = "2019-08-18T16:16:27+02:00"
# Introduction
Bonjour le monde! Another week, another- hold on. It's been two
weeks. Time flies, doesn't it? Last week I was sadly on vacation and
couldn't devote much time to Redox, but this week I've been back!
Most of my work this week has been about GDB. I've asked around a
little about porting Redox OS in the #gdb IRC channel on freenode, and
had a poll about which way I should go forward: Port GDB or create my
own GDBserver from scratch in Rust. The votes have been received, and
the answer was:
## Building a GDB server in Rust
[The GDB Remote
is incredibly well-documented and a pleasure to work with. GDB is
really compatible with all kinds of targets, using new features only
on the targets that explicitly supports them.
I created the
library for parsing the main structure over GDB packets and performing
checksum verifications. The actual contents of the packets isn't
interpreted currently. Then I started work on a "gdbserver" program
(in rust, still) that aims to support both x86_64 Linux and Redox OS.
![GDB trying its best to understand the madness that my program
sends](https://i.imgur.com/k0Oj63q.png "GDB trying its best to
understand the madness that my program sends")
## Detect exits
[The RFC now specifies
and my fork of the kernel implements it. Robert O'Callahan, programmer
at Mozilla who works on the `rr` debugging utility suggested this
idea, as apparently it's very useful to be able to inspect the process
state *before* exit. Huge thanks to him for his input, it's incredibly
valuable. I will send out a merge request for the final changes soon,
and probably get the RFC merged.
## Final round up
My RSoC period will unfortunately end soon, together with the summer
vacation from school. This means I'll be unable to create patches as
rapidly and should spend the final few days to publish everything and
make sure everything is somewhat working. It's been a great summer and
I'm really happy to have gotten this opportunity again. Goodbye for
now, everyone!