Skip to content

Rework the graphics handling

Note: Some items are under both headings if they are both a new feature and require new ipc functionality.

Graphics features

  • Move text console support out of vesad to allow other graphics drivers to reuse it. (drivers!128 (merged))
  • Move from allowing fbcond and orbital to update on every change to updates limited by the gpu vsync.
  • Implement page flipping. (drivers!273 (merged))
  • Support for hardware backed cursor. This both decreases latency and improves energy efficiency by offloading rendering of the cursor to the display controller (on mobile devices this is an entirely separate part of the SoC by the way). Not every gpu and gpu interfaces support this though. VGA doesn't support it, but virtio-gpu does and so do all desktop gpu's. (drivers!260 (merged))
  • Support mode setting. That is changing display resolutions, display mirroring, selecting which part of a buffer to show, using multiple planes aside from the hardware backed cursor. (partial in drivers!273 (merged) and drivers!277 (merged))

Communication between parts of the graphics subsystem.

  • Better IPC serialization/deserialization. The current ad-hoc code is error prone and hard to keep in sync between components as you need to change everything in lockstep. Something like Cap'n Proto (Rust implementation) may be a better fit. It should allow keeping compatibility with older programs even when we add and remove commands. Edit: Cap'n Proto requires a C++ program for generating bindings, so another format may be better.
  • Implement handoff from vesad to other graphics drivers (drivers!213 (merged))
  • Let inputd pass the display fd directly to users.
  • Move the console multiplexing out of the individual graphics drivers. Instead have a protocol that allows swapping the framebuffer to be rendered by the graphics driver and then have an fbmuxd daemon which saves the current framebuffer for every virtual terminal and tells the graphics driver to swap it out whenever either switching between vt's or when the client requests the framebuffer to be swapped. The vga/vesa driver can then write it to the actual framebuffer while the virtio-gpu driver in the future could request the virtio-gpu device to change which framebuffer to use for scanout once it becomes possible for users to directly create buffers on the gpu. Fbmuxd would also handle multiplexing of input devices. Inputd can tell the graphics driver which fd should have control over the display scanout.
  • Allow delegating control over a virtual terminal to a different program. This would allow for example the login screen to delegate control to an instance of orbital running as the logged in user, reducing attack surface. Or for a VR game to get full control over page flipping and mode setting. It should also be possible for the delegating program to revoke access again for all the (in)directly delegated programs. This delegation should happen for both the display and input devices. Preferably with an option for the delegating program to keep accepting input such that it can for example listen for Super+L to lock the screen and show a password prompt for the current user.
  • Allow submitting other requests than flush requests to the graphics driver.
  • Allow running commands like shader invocations on the gpu using a 3d context. Initially just virgl (OpenGL) or venus (Vulkan) support for virtio-gpu would work. This final step would allow actual hardware acceleration.
Edited by bjorn3