planet on tony is coding

Linux 6.15, DRM scheduler, wedged events, sched_ext and more

Mon, 26 May 2025 00:00:00 +0000

The Linux 6.15 has just been released, bringing a lot of new features:

nova-core, the “base” driver for the new NVIDIA GPU driver, written in Rust. nova project will eventually replace Nouveau driver for all GSP-based GPUs.
RISC-V gained support for some extensions: BFloat16 floating-point, Zaamo, Zalrsc and ZBKB.
The fwctl subsystem has been merged. This new family of drivers acts as a transport layer between userspace and complex firmware. To understand more about its controversies and how it got merged, check out this LWN article.
Support for MacBook touch bars, both as a DRM driver and input source.
Support for Adreno 623 GPU.

As always, I suggest to have a look at the Kernel Newbies summary. Now, let’s have a look at Igalia’s contributions.

DRM wedged events

In 3D graphics APIs such Vulkan and OpenGL, there are some mechanisms that applications can rely to check if the GPU had reset (you can read more about this in the kernel documentation). However, there was no generic mechanism to inform userspace that a GPU reset has happened. This is useful because in some cases the reset affected not only the app involved in the reset, but the whole graphic stack and thus needs some action to recover, like doing a module rebind or even bus reset to recovery the hardware. For this release, we helped to add an userspace event for this, so a daemon or the compositor can listen to it and trigger some recovery measure after the GPU has reset. Read more in the kernel docs.

DRM scheduler work

In the DRM scheduler area, in preparation for the future scheduling improvements, we worked on cleaning up the code base, better separation of the internal and external interfaces, and adding formal interfaces at places where individual drivers had too much knowledge of the scheduler internals.

General GPU/DRM stack

In the wider GPU stack area we optimised the most frequent dma-fence single fence merge operation to avoid memory allocations and array sorting. This should slightly reduce the CPU utilisation with workloads which use the DRM sync objects heavily, such as the modern composited desktops using Vulkan explicit sync.

Some releases ago, we helped to enable async page flips in the atomic DRM uAPI. So far, this feature was only enabled for the primary plane. In this release, we added a mechanism for the driver to decide which plane can perform async flips. We used this to enable overlay planes to do async flips in AMDGPU driver.

We also fixed a bug in the DRM fdinfo common layer which could cause use after free after driver unbind.

Intel Xe driver improvements

On the Intel GPU specific front we worked on adding better Alderlake-P support to the new Intel Xe driver by identifying and adding missing hardware workarounds, fixed the workaround application in general and also made some other smaller improvements.

sched_ext

When developing and optimizing a sched_ext-based scheduler, it is important to understand the interactions between the BPF scheduler and the in-kernel sched_ext core. If there is a mismatch between what the BPF scheduler developer expects and how the sched_ext core actually works, such a mismatch could often be the source of bugs or performance issues.

To address such a problem, we added a mechanism to count and report the internal events of the sched_ext core. This significantly improves the visibility of subtle edge cases, which might easily slip off. So far, eight events have been added, and the events can be monitored through a BPF program, sysfs, and a tracepoint.

A few less bugs

As usual, as part of our work on diverse projects, we keep an eye on automated test results to look for potential security and stability issues in different kernel areas. We’re happy to have contributed to making this release a bit more robust by fixing bugs in memory management, network (SCTP), ext4, suspend/resume and other subsystems.

This is the complete list of Igalia’s contributions for this release:

Authored (75)

André Almeida

Angelos Oikonomopoulos

arm64: Don’t call NULL in do_compat_alignment_fixup()

Bhupesh

ext4: ignore xattrs past end

Changwoo Min

Gavin Guo

mm/huge_memory: fix dereferencing invalid pmd migration entry

Guilherme G. Piccoli

Luis Henriques

fuse: removed unused function fuse_uring_create() from header

Maíra Canal

Melissa Wen

Ricardo Cañuelo Navarro

Rodrigo Siqueira

Thadeu Lima de Souza Cascardo

Tvrtko Ursulin

Reviewed (30)

André Almeida

Christian Gmeiner

drm/sched: Use struct for drm_sched_init() params

Iago Toral Quiroga

Jose Maria Casanova Crespo

drm/v3d: Remove v3d->cpu_job

Luis Henriques

fuse: add default_request_timeout and max_request_timeout sysctls

Maíra Canal

Melissa Wen

Rodrigo Siqueira

Thadeu Lima de Souza Cascardo

scripts: add script to extract built-in firmware blobs

Tvrtko Ursulin

Tested (2)

Changwoo Min

sched_ext: Fix rq lock state in hotplug ops

Guilherme G. Piccoli

x86/split_lock: Fix the delayed detection logic

Acked (12)

Changwoo Min

Maíra Canal

drm/doc: Document KUnit expectations

Tvrtko Ursulin

Maintainer SoB (2)

Maíra Canal

drm/v3d: Add clock handling

Tvrtko Ursulin

dma-buf: add selftest for fence order after merge

Linux 6.14, an almost forgotten release

Fri, 28 Mar 2025 00:00:00 +0000

Linux 6.14 is the second release of 2025, and as usual Igalia took part on it. It’s a very normal release, except that it was release on Monday, instead of the usual Sunday release that has been going on for years now. The reason behind this? Well, quoting Linus himself:

I’d like to say that some important last-minute thing came up and delayed things.

But no. It’s just pure incompetence.

But we did not forget about it, so here’s our Linux 6.14 blog post!

A part of the development cycle for this release happened during late December, when a lot of maintainers and developers were taking their deserved breaks. As a result of this, this release contains less changes than usual as stated by LWN as the “lowest level of merge-window activity seen in years”. Nevertheless, some cool features made through this release:

NT synchronization primitives: Elizabeth Figura, from Codeweavers, is know from her work around improving Wine sync functions, like mutexes and semaphores. She was one the main collaborators behind the futex_waitv() work and now developed a virtual driver that is more compliant with the precise semantics that the NT kernel exposes. This allows Wine to behave closer to Windows without the need to create new syscalls, since this driver uses ioctl() as the front-end uAPI.
RWF_UNCACHED: Linux has two ways of dealing with storage I/O: buffered I/O (usually the preferred one) that stores data in a temporary buffer and regularly syncs the cache data with the device; and direct I/O that doesn’t use cache and always writes/reads synchronously with the storage device. Now a new mixed approach is available: uncached buffered I/O. This method is aimed to have a fast way to write or read data that will not be needed again in the short term. For reading, the device writes data in the buffer and as soon as the user finished reading the buffer, it’s cleared from the cache. For writing, as soon as userspace fills the cache, the device reads it and removes it from the cache. In this way we still have the advantage of using a fast cache but reducing the cache pressure.
amdgpu panic support: AMD developers added kernel panic support for amdgpu driver, “which displays a pretty user friendly message on the screen when a Linux kernel panic occurs” instead of just a black screen or a partial dmesg log.

As usual Kernel Newbies provides a very good summary, you should check it for more details: Linux 6.14 changelog. Now let’s jump to see what were the merged contributions by Igalia for this release!

DRM

For the DRM common infrastructure, we helped to land a standardization for DRM client memory usage reporting. Additionally, we contributed to improve and fix bugs found in drivers of AMD, Intel, Broadcom, and Vivante.

AMDGPU

For the AMD driver, we fixed bugs experienced by users of Cosmic Desktop Environment on several AMD hardware versions. One was uncovered with the introduction of overlay cursor mode, and a definition mismatch across the display driver caused a page-fault in the usage of multiple overlay planes. Another bug was related to division by zero on plane scaling. Also, we fixed regressions on VRR and MST generated by the series of changes to migrate AMD display driver from open-coded EDID handling to drm_edid struct.

Intel

For the Intel drivers, we fixed a bug in the xe GPU driver which prevented certain type of workarounds from being applied, helped with the maintainership of the i915 driver, handled external code contributions, maintained the development branch and sent several pull requests.

Raspberry Pi (V3D)

We fixed the GPU resets for the Raspberry Pi 4 as we found out to be broken as per a user bug report.

Also in the V3D driver, the active performance monitor is now properly stopped before being destroyed, addressing a potential use-after-free issue. Additionally, support for a global performance monitor has been added via a new DRM_IOCTL_V3D_PERFMON_SET_GLOBAL ioctl. This allows all jobs to share a single, globally configured perfmon, enabling more consistent performance tracking and paving the way for integration with user-space tools such as perfetto.

Your browser does not support the video tag.

A small video demo of perfetto integration with V3D

etnaviv

On the etnaviv side, fdinfo support has been implemented to expose memory usage statistics per file descriptor, enhancing observability and debugging capabilities for memory-related behavior.

sched_ext

Many BPF schedulers (e.g., scx_lavd) frequently call bpf_ktime_get_ns() for tracking tasks’ runtime properties. bpf_ktime_get_ns() eventually reads a hardware timestamp counter (TSC). However, reading a hardware TSC is not performant in some hardware platforms, degrading instructions per cycyle (IPC).

We addressed the performance problem of reading hardware TSC by leveraging the rq clock in the scheduler core, introducing a scx_bpf_now() function for BPF schedulers. Whenever the rq clock is fresh and valid, scx_bpf_now() provides the rq clock, which is already updated by the scheduler core, so it can reduce reading the hardware TSC. Using scx_bpf_now() reduces the number of reading hardware TSC by 50-80% (e.g., 76% for scx_lavd).

Assorted kernel fixes

Continuing our efforts on cleaning up kernel bugs, we provided a few fixes that address issues reported by syzbot with the goal of increasing stability and security, leveraging the fuzzing capabilities of syzkaller to bring to the surface certain bugs that are hard to notice otherwise. We’re addressing bug reports from different kernel areas, including drivers and core subsystems such as the memory manager. As part of this effort, several fixes were done for the probe path of the rtlwifi driver.

Check the complete list of Igalia’s contributions for the 6.14 release

Authored (38)

Changwoo Min

Christian Gmeiner

Luis Henriques

fuse: fix possible deadlock if rings are never initialized

Maíra Canal

Melissa Wen

Ricardo Cañuelo Navarro

Rodrigo Siqueira

Thadeu Lima de Souza Cascardo

Tvrtko Ursulin

Reviewed (36)

André Almeida

Christian Gmeiner

Iago Toral Quiroga

Jose Maria Casanova Crespo

drm/v3d: Assign job pointer to NULL before signaling the fence

Luis Henriques

Maíra Canal

Tvrtko Ursulin

Tested (2)

André Almeida

ASoC: cs35l41: Fallback to using HID for system_name if no SUB is available

Christian Gmeiner

hexagon: fix using plain integer as NULL pointer warning in cmpxchg

Acked (1)

Iago Toral Quiroga

drm/v3d: Fix miscellaneous documentation errors

Maintainer SoB (6)

Maíra Canal

Tvrtko Ursulin

Linux 6.13, I WANT A GUITAR PEDAL

Mon, 20 Jan 2025 00:00:00 +0000

Just as 2025 is starting, we got a new Linux release in mid January, tagged as 6.13. In the spirit of holidays, Linus Torvalds even announced during 6.13-rc6 that he would be building and raffling a guitar pedal for a random kernel developer!

As usual, this release comes with a pack of exciting news done by the kernel community:

This release has two important improvements for task scheduling: lazy preemption and proxy execution. The goal with lazy preemption is to find a better balance between throughput and response time. A secondary goal is being able to make it the preferred non-realtime scheduling policy for most cases. Tasks that really need a reschedule in a hurry will use the older TIF_NEED_RESCHED flag. A preliminary work for proxy execution was merged, which will let us avoid priority-inversion scenarios when using real time tasks with deadline scheduling, for use cases such as Android.
New important Rust abstractions arrived, such as VFS data structures and interfaces, and also abstractions for misc devices.
Lightweight guard pages: guard pages are used to raise a fatal signal when accessed. This feature had the drawback of having a heavy performance impact, but in this new release the flag MADV_GUARD_INSTALL was added for the madvise() syscall, offering a lightweight way to guard pages.

To know more about the community improvements, check out the summary made by Kernel Newbies.

Now let’s highlight the contributions made by Igalians for this release.

Case-insensitive support for tmpfs

Case sensitivity has been a traditional difference between Linux distros and MS Windows, with the most popular filesystems been in opposite sides: while ext4 is case sensitive, NTFS is case insensitive. This difference proved to be challenging when Windows apps, mainly games, started to be a common use case for Linux distros (thanks to Wine!). For instance, games running through Steam’s Proton would expect that the path assets/player.png and assets/PLAYER.PNG would point to be the same file, but this is not the case in ext4. To avoid doing workarounds in userspace, ext4 has support for casefolding since Linux 5.2.

Now, tmpfs joins the group of filesystems with case-insensitive support. This is particularly useful for running games inside containers, like the combination of Wine + Flatpak. In such scenarios, the container shares a subset of the host filesystem with the application, mounting it using tmpfs. To keep the filesystem consistent, with the same expectations of the host filesystem about the mounted one, if the host filesystem is case-insensitive we can do the same thing for the container filesystem too. You can read more about the use case in the patchset cover letter.

While the container frameworks implement proper support for this feature, you can play with it and try it yourself:

$ mount -t tmpfs -o casefold fs_name /mytmpfs
$ cd /mytmpfs # case-sensitive by default, we still need to enable it
$ mkdir a
$ touch a; touch A
$ ls
A  a
$ mkdir B; cd b
cd: The directory 'b' does not exist
$ # now let's create a case-insensitive dir
$ mkdir case_dir
$ chattr +F case_dir
$ cd case_dir
$ touch a; touch A
$ ls
a
$ mkdir B; cd b
$ pwd
$ /home/user/mytmpfs/case_dir/B

V3D Super Pages support

As part of Igalia’s effort for enhancing the graphics stack for Raspberry Pi, the V3D DRM driver now has support for Super Pages, improving performance and making memory usage more efficient for Raspberry Pi 4 and 5. Using Linux 6.13, the driver will enable the MMU to allocate not only the default 4KB pages, but also 64KB “Big Pages” and 1MB “Super Pages”.

To measure the difference that Super Pages made to the performance, a series of benchmarks where used, and the highlights are:

+8.36% of FPS boost for Warzone 2100 in RPi4
+3.62% of FPS boost for Quake 2 in RPi5
10% time reduction for the Mesa CI job v3dv-rpi5-vk-full:arm64
Aether SX2 emulator is more fluid to play

You can read a detailed post about this, with all benchmark results, in Maíra’s blog post, including a super cool PlayStation 2 emulation showcase!

New `transparent_hugepage_shmem=` command-line parameter

Igalia contributed new kernel command-line parameters to improve the configuration of multi-size Transparent Huge Pages (mTHP) for shmem. These parameters, transparent_hugepage_shmem= and thp_shmem=, enable more flexible and fine-grained control over the allocation of huge pages when using shmem.

The transparent_hugepage_shmem= parameter allows users to set a global default huge page allocation policy for the internal shmem mount. This is particularly valuable for DRM GPU drivers. Just as CPU architectures, GPUs can also take advantage of huge pages, but this is possible only if DRM GEM objects are backed by huge pages.

Since GEM uses shmem to allocate anonymous pageable memory, having control over the default huge page allocation policy allows for the exploration of huge pages use on GPUs that rely on GEM objects backed by shmem.

In addition, the thp_shmem= parameter provides fine-grained control over the default huge page allocation policy for specific huge page sizes.

By configuring page sizes and policies of huge-page allocations for the internal shmem mount, these changes complement the V3D Super Pages feature, as we can now tailor the size of the huge pages to the needs of our GPUs.

DRM and AMDGPU improvements

As usual in Linux releases, this one collects a list of improvements made by our team in DRM and AMDGPU driver from the last cycle.

Cosmic (the desktop environment behind Pop! OS) users discovered some bugs in the AMD display driver regarding the handling of overlay planes. These issues were pre-existing and came to light with the introduction of cursor overlay mode. They were causing page faults and divide errors. We debugged the issue together with reporters and proposed a set of solutions that were ultimately accepted by AMD developers in time for this release.

In addition, we worked with AMD developers to migrate the driver-specific handling of EDID data to the DRM common code, using drm_edid opaque objects to avoid handling raw EDID data. The first phase was incorporated and allowed the inclusion of new functionality to get EDID from ACPI. However, some dependencies between the AMD the Linux-dependent and OS-agnostic components were left to be resolved in next iterations. It means that next steps will focus on removing the legacy way of handling this data.

Also in the AMD driver, we fixed one out of bounds memory write, fixed one warning on a boot regression and exposed special GPU memory pools via the fdinfo common DRM framework.

In the DRM scheduler code, we added some missing locking, removed a couple of re-lock cycles for slightly reduced command submission overheads and clarified the internal documentation.

In the common dma-fence code, we fixed one memory leak on the failure path and one significant runtime memory leak caused by incorrect merging of fences. The latter was found by the community and was manifesting itself as a system out of memory condition after a few hours of gameplay.

sched_ext

The sched_ext landed in kernel 6.12 to enable the efficient development of BPF-based custom schedulers. During the 6.13 development cycle, the sched_ext community has made efforts to harden the code to make it more reliable and clean up the BPF APIs and documentation for clarity.

Igalia has contributed to hardening the sched_ext core code. We fixed the incorrect use of the scheduler run queue lock, especially during initializing and finalizing the BPF scheduler. Also, we fixed the missing RCU lock protections when the sched_core selects a proper CPU for a task. Without these fixes, the sched_ext core, in the worst case, could crash or raise a kernel oops message.

Other Contributions & Fixes

syzkaller, a kernel fuzzer, has been an important instrument to find kernel bugs. With the help of KASAN, a memory error detector, and syzbot, numerous such bugs have been reported and fixed.

Igalians have contributed to such fixes around a lot of subsystems (like media, network, etc), helping reduce the number of open bugs.

Check the complete list of Igalia’s contributions for the 6.13 release

Authored (70)

André Almeida

Changwoo Min

Christian Gmeiner

drm/v3d: Use v3d_perfmon_find()

Guilherme G. Piccoli

Maíra Canal

Melissa Wen

Thadeu Lima de Souza Cascardo

Tvrtko Ursulin

Reviewed (41)

André Almeida

Christian Gmeiner

Iago Toral Quiroga

Jose Maria Casanova Crespo

drm/v3d: Ensure job pointer is set to NULL after job completion

Juan A. Suarez

drm/vc4: Use vc4_perfmon_find()

Maíra Canal

Tvrtko Ursulin

Tested (1)

Christian Gmeiner

drm/etnaviv: Replace the ‘&pdev->dev’ with ‘dev’

Acked (5)

Changwoo Min

Maíra Canal

Maintainer SoB (6)

Maíra Canal

Installing kernel modules faster with multithread XZ

Fri, 03 Mar 2023 00:00:00 +0000

As a kernel developer, everyday I need to compile and install custom kernels, and any improvement in this workflow means to be more productive. While installing my fresh compiled modules, I noticed that it would be stuck in amdgpu compression for some time:

XZ      /usr/lib/modules/6.2.0-tonyk/kernel/drivers/gpu/drm/amd/amdgpu/amdgpu.ko.xz

XZ format

My target machine is the Steam Deck, that uses .xz for compressing the modules. Giving that we want gamers to be able to install as many games as possible, the OS shouldn’t waste much disk space. amdgpu, when compiled with debug symbols can use a good hunk of space. Here’s the comparison of disk size of the module uncompressed, and then with .zst and .xz compression:

360M amdgpu.ko
61M  amdgpu.ko.zst
38M  amdgpu.ko.xz

This more compact module comes with a cost: more CPU time for compression.

Multithread compression

When I opened htop, I saw that only a lonely thread was doing the hard work to compress amdgpu, even that compression is a task easily parallelizable. I then hacked scripts/Makefile.modinst so XZ would use as many threads as possible, with the option -T0. In my main build machine, modules_install was running 4 times faster!

# before the patch
$ time make modules_install -j16
Executed in  100.08 secs

# after the patch
$ time make  modules_install -j16
Executed in   28.60 secs

Then, I submitted a patch to make this default for everyone: [PATCH] kbuild: modinst: Enable multithread xz compression

However, as Masahiro Yamada noticed, we shouldn’t be spawning numerous threads in the build system without the user request. Until today we specify manually how many threads we should run with make -jX.

Hopefully, Nathan Chancellor suggested that the same results can be achieved using XZ_OPT=-T0, so we still can benefit from this without the patch. I experimented with different -TX and -jY values, but in my notebook the most efficient values were X = Y = nproc. You can check some results bellow:

$ make modules_install
174.83 secs

$ make modules_install -j8
100.55 secs

$ make modules_install XZ_OPT=-T0
81.51 secs

$ make modules_install -j8 XZ_OPT=-T0
53.22 sec

futex2 at Linux Plumbers Conference 2022

Fri, 23 Sep 2022 00:00:00 +0000

In-person conferences are finally back! After two years of remote conferences, the kernel development community got together in Dublin, Ireland, to discuss current problems that need collaboration to be solved.

As in the past editions, I took the opportunity to discuss about futex2, a project I’m deeply involved in. futex2 is a project to solve issues found in the current futex interface. This years’ session was about NUMA awaress of futex. Currently, futex allocates a single hash table in the kernel to store all data. If this is done in a NUMA machine, the table is allocated in a single node. This increases the cost of futex operations when done from any other node, given that will need to access a memory outside of it’s node.

To solve this, futex2 will allocate a table per NUMA node and have a NUMA-aware interface to tell the kernel which node to work on, using this struct:

struct futexX_numa {
   __uX value;
   __sX hint;
};

Where X can be a range of values [8, 16, 32, 64], for each available futex size in futex2. value is the futex value and hint is the node where you want to operate on. It can be -1 for the current node or [0, MAX_NUMNODES) to specify the node. So, for example:

struct futex32_numa f = {.value = 0, .hint = -1};

futex_wait(&f, 0, FUTEX_NUMA | FUTEX_32, NULL);

The session was short, but very productive, with Peter Zijlstra and Thomas Gleixner giving most feedback about the proposal. Here’s a list of ideas that were covered in the session and are to be consider when implementing this interface:

Userspace always initialize .hint = -1. The kernel then overwrites -1 with the current node being used by the first waiter. In that way, every following futex operation (wait or wake) will be guaranteed to be on the same node.
After the last waiter is waked, .hint can be set as -1 again.
To avoid racing, any change to the struct futex_numa needs to be done atomically for the whole structure.
The struct needs to have explicitly alignment and to be __packed.
Benchmark data will for sure attracts reviewers and users. perf bench is the right place to add that.
The local hash table doesn’t need to be as big as is the global futex table, probably num_possible_cpus() will be enough.
Support for a struct futex64_numa might be tricky, so we might just restrict for smaller sizes. hint may be always s16 giving that MAX_NUMNODES isn’t that huge.

Here are my slides from this year. If you need any help with futex, you can get in touch with me andrealmeid@igalia.com.

Apart from futex, there were other talks to highlight. As a trend in recent Plumber editions, Rust was once again a hot topic in the conference. From a single session at LPC 2020, this year we had Rust in the main track, in the Maintainers Summit and its own Microconference. The initial mistrust of adding a second language to the kernel is long gone, and now the conversations are around the final bits on how to get it merged (Will the patches go though the maintainers tree, or directly to Torvalds? Which Rust version to use? Stable or unstable?). This whole effort is a good showcase of the conference ability to join efforts from different companies, communities and projects along the stack (GCC, clang) to solve complex problems together. Oh, and if you are wondering, Rust will likely be available in Linux 6.1 or 6.2.

From the GPU world we had a session to discuss how to deal with accelerator drivers in the kernel, as there’s some disagreement on how to accept (and where to accept) these new drivers. Along with that, a BoF to talk about getting a userspace console to be used in case the graphics stack crashes and how to get an implementation of cgroups for GPUs.

The Kernel Testing & Dependability Microconference was a stage to get news from testing tools, like syzbot, kselftests, KernelCI and KUnit. There was some discussion on how to integrate all frameworks as well, given that there’s a lot of testing nowadays, but not much communication and deploying of all this. Last but not least, Tales Aparecida (my GSoC mentee) presented the challenges of using KUnit to test device drivers.

That’s it for 2022 edition. Thanks Igalia for sponsoring my travel to LPC and see ya next year!

Keeping a project bisectable

Thu, 04 Aug 2022 00:00:00 +0000

People write code. Test coverage is never enough. Some angry contributor will disable the CI. And we all write bugs.

But that’s OK, it part of the job. Programming is hard and sometimes we may miss a corner case, forget that numbers overflow and all other strange things that computers can do.

One easy thing that we can do to help the poor developer that needs to find what changed in the code that stopped their printer to work properly, is to keep the project bisectable.

A “bisectable” project is a project where one can reliably run git bisect, which is a very useful command to find a commit that introduces a bug. It works doing a binary search in the git history until finding the guilty commit. This process involves building each step of the bisect and running a test on each build to check if it’s good or bad (that you can magically automate with git bisect run). The problem is, if you can’t compile, you can’t tell if this commit is before or after the bug (it can even be the culpable commit itself!). Then you need to jump and try another commit and hope that it will compile, making the process more painful. A lot of build breakages along the commit history can easily discourage a brave bisecter.

Keep it buildable

To make sure we keep the bisectability of a project, we just need to make sure that every commit is buildable. Sometimes we split a feature in a bunch of cool commits but fail to see if they work as standalone changes. We can simply do a interactive rebase, building it at each step:

git rebase -i main --exec "make"

Replace main with the reference that you want to build from and make with your fancy build system and ta-da, let the magic happen. If any of the steps fails to build, the rebase stops (given that “any command that fails will interrupt the rebase, with exit code 1”) so you can investigate what happened.

Of course, there’s still some room for build breakages for different build configurations, but if you always follow this practice before upstreaming your patches you are building a better git log.

Discussion on Reddit.
Discussion on Hacker News

Create an ArchLinux image for kernel testing

Tue, 10 Mar 2020 00:00:00 +0000

In this tutorial, we’ll look on how to create a functional and simple ArchLinux virtual machine image, that can access the internet, display graphical windows and share a folder with the host to help you with your kernel development. The only requirement is that you need to have a working ArchLinux installation.

A virtual machine is useful in a lot of development scenarios, but it’s particularly essential in the Linux kernel development. It can be really time-consuming to install the kernel on your own system and then needing to reboot the machine just to see if your printk() is working. This is why this topic was already covered in Collabora’s blog, with Ezequiel explaining how to use virtme and Frédéric showing how to setup a minimal Debian to use with QEMU-KVM.

As already said by my colleagues, there’s no need to install a complete system from scratch, using a installation disc. If you’ve already used to Arch Linux, you probably know that the distro slogan is “Keep it Simple, Stupid”. So let’s try not to suffer in order to get a kernel development environment. It may seem to be complicated to achieve this, but once I explain all the concepts behind it, you will notice how this is very customizable and easily scriptable, so you can automate this part of your workflow.

Since we’re going to work with two systems at the same time, let’s explicitly state in which machine you should run the command:

(host)$ # for you real machine

(guest)$ # for the virtual machine

Creating a disk

We will need a disk to store our new OS in. Hopefully, we have no need for a physical device, a file will do the work. It’s up to you how much space you’ll allocate, but I recommend a minimum of 4 GB. We’re going to create a 5GB sparse file (a file that allocates space as needed) to be our disk, using the truncate command:

(host)$ truncate -s 5G arch_disk.raw

If you check the size with du -h arch_disk.raw, it’s going to say 0 (because we haven’t used it yet), but if you run du -h --apparent-size arch_disk.raw you can see the maximum size the file may expand.

Let’s add a file system at this file. This means that this file will be ready to contain files and folders, and will contain the new file system as its data. This will make our disk to look and behave as a single partition.

(host)$ mkfs.ext4 arch_disk.raw

Since we have a file that represents a partition, we can mount it. Create a directory to be the mounting point and mount it:

(host)$ mkdir mnt
(host)$ sudo mount arch_disk.raw mnt

Installing ArchLinux

Now that we have a disk, let’s place an initial system, just like when we are installing Arch. Install these packages:

(host)$ sudo pacman -S arch-install-scripts qemu

The first package has some scripts that are really helpful to install ArchLinux (e.g. pacstrap, arch-chroot) and the second one is the QEMU emulator.

Just remember to check if you are using a nice mirror close to you on top of your mirrorlist file (/etc/pacman.d/mirrorlist). This will speedup your download.

Now, let’s transform that formatted partition into a functional system. pacstrap is wrapper for pacman that will install in the first argument (mnt) the packages/groups (base, base-devel) and create the root filesystem, with the appropriated folders (lib, dev, tmp, home, …). You may also take this step to add more packages that you would like to have in the virtual environment, like vim for instance. If you want to enable internet access in the guest in the future, use this step to install the dhcpcd package. Just add then after base-devel in the following command:

(host)$ sudo pacstrap mnt base base-devel

Pro tip: if your host system is up-to-date, you can use the flag -c to just copy the packages from your host to the guest, instead of downloading them all again: pacstrap -c mnt ...

This will create the directory structure and install the basic packages. You can navigate through mnt/ and see an entire file system there, and even change the root to this new system:

(host)$ sudo arch-chroot mnt

If you use ls and pwd you will see that you are definitely in a guest system, and not in your host machine anymore. If you use uname -r you can see the kernel version was installed on your host system:

(guest)$ ls
bin   dev  home  lib64	     mnt  proc	run   srv  tmp	var
boot etc  lib	 lost+found  opt  root	sbin  sys  usr
(guest)$ ls home/
(guest)$ pwd
/
(guest)$ uname -r 4.20.7-arch1-1-ARCH

Once inside, it can be useful to set a password for the root user with passwd.

Use CTRL+D to exit from the guest and then sudo umount mnt to umount the disk. Let’s use a custom kernel now.

Using QEMU and your kernel

Now, for the next steps, you will need a compiled and functional Linux Kernel. You already have one, since you are running a GNU/Linux distribution, but if you still don’t have a custom kernel to make experiments, you can easily get one like this:

(host)$ git clone --depth=1 git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
(host)$ cd linux
(host)$ make x86_64_defconfig
(host)$ make kvm_guest.config
(host)$ make -j8

The goal of this article is not get in details on how to compile the kernel. For more information, check the Arch wiki article and have a look at ccache.

The git command will clone Linus’ tree source code of the kernel into your machine. With the argument --depth=1, you won’t clone all the commit history of the kernel and the download will be faster. If you want to have the history (which is advisable with you want to send patches to the project), remove this argument. The make ...config commands will create a basic kernel with virtualization powers for x86_64 machines.

make -j8 can take some time and some CPU usage. The -j8 argument will create 8 jobs to compile the kernel, change it according to the number of threads in your machine.

Let’s play with QEMU. We are going to use a lot of flags and arguments, so let’s check in details each one:

-hda disk.raw: specify that the arch_disk.raw file should be provided as the first hard disk in the emulated system.
-m 4G: amount of RAM we are going to loan to the virtual machine.
-nographic: QEMU will run on the terminal instead of a graphical window.
-kernel linux/arch/x86_64/boot/bzImage: define the file where your compressed kernel image is. You can also use the kernel installed on your machine, it should be somewhere in /boot/.
-smp 1: how many virtual CPUs QEMU will use.
-append "root=/dev/sda rw console=ttyS0 loglevel=5": kernel parameters.
- root defines which disk/partition has the root file system. In our case, it will be the first storage device¹, the hard disk arch_disk.raw.
- rw that we want to read and write to disk.
- console is to set the standard output of the kernel and of the PID 1.
- loglevel is to set how much log the kernel will output to the console and 7 is the highest and it will display all the kernel messages in the console prompt.
  - I don’t find the Audit logs useful. You can disable them with audit=0.
  - You can learn more about kernel parameters in the documentation;
--enable-kvm: this is to enable hardware acceleration to virtualization.

Feel free to change these parameters to suit your environment. Now run it all together:

(host)$ qemu-system-x86_64 -hda arch_disk.raw -m 4G -nographic \
        -kernel linux/.../bzImage \
        -append "root=/dev/sda rw console=ttyS0 loglevel=5" \
        --enable-kvm

When the prompt displays archlinux login:, just type root and enter. Use uname -r again to check which kernel you are running. When you are done, use CTRL+a then x to exit:

archlinux login: root
(guest)$ uname -r
5.0.0-rc1+

If you always use the root user, you may set to auto login using systemd. Type systemctl edit serial-getty@.service in the guest and a text editor will open. Add those lines and in the next login you should be good:

[Service]
ExecStart=
ExecStart=-/usr/bin/agetty --autologin root --noclear %I $TERM

Connecting to the internet

You may want to install some packages on your virtual machine or perform some network tests, but as it stands, you can’t reach the internet:

(guest)$ ping eff.org
ping: eff.org: Temporary failure in name resolution

Let’s use our host as a network bridge; login in our virtual machine with root user and then:

(guest)$ systemctl enable dhcpcd
(guest)$ systemctl start dhcpcd

Every time your system boots, it’ll start to run a DHCP service and connects to the internet using a “virtual wire” to the host.

(guest)$ ping eff.org
PING eff.org (173.239.79.196) 56(84) bytes of data.

Due to the configuration of the NAT, you won’t get ping answers like 64 bytes from vm1.eff.org (173.239.79.196): icmp_seq=1 ttl=46 time=197 ms. However, since we managed to solve the DNS (since ping showed to us the IP address of eff.org), we are sure that we have internet connection.

Displaying graphical windows

If you want to run graphical applications inside your virtual machine, you may follow one of the methods:

QEMU display: use QEMU display in graphical mode with a GUI compositor;
SSH with XForwarding: use your own terminal to SSH into the VM and then display the content using your host’s display.

This can be useful to test some subsystems like DRM or V4L2. Keep in mind that the performance won’t be the best and the functionality can be a bit buggy, so I recommend using lightweight apps like gstreamer and ffmpeg.

QEMU display

The first difference here is that we’re going to remove the -nographic argument from QEMU. Run it now and it should open a new window.

Now, make sure to have this module enabled at your kernel: DRM_BOCHS, a driver to help us displaying graphical content in QEMU. This means that if you open the .config file at your kernel source directory it should have this line: CONFIG_DRM_BOCHS=y. Remember to recompile the kernel each time you change it’s configuration.

We need software to manage the windows and display them on the screen, so we are going to install a lightweight Wayland compositor, weston:

(guest)$ pacman -S weston xorg-server-xwayland xorg-fonts-type1 xorg-xclock

Create a file ~/.config/weston.ini in guest and add this:

[core] xwayland=true

Run the weston command in guest, and the graphical interface should appear. You can use your mouse to open the weston-terminal at the top left corner. If your mouse is moving oddly, make sure your window zoom is set to the “Best fit” option in the QEMU window bar. Inside the terminal, run xclock and you should be able to check the hours inside your virtual machine. When you’re done, you can use Ctrl+Alt+Backspace to quit weston.

Screen capture showing xclock inside weston

This is just the basic setup using QEMU graphical mode. To improve performance and usability (like to have clipboard and multimonitor support) one can use a QEMU front end (like virt-manager) or SPICE to expand your options and features between guest and host. Check the ArchWiki to learn more about virtual graphic cards, graphics acceleration and SPICE.

XForwarding

This is a way to display graphical interfaces running in a remote host in the local machine. This means that the X server on your guest machine will forward the graphical input/output to the X client on your host machine. You can keep the argument -nographic in QEMU. Let’s get some packages that will help us (you need to install xorg-auth in both machines, the host and guest):

(host)$ sudo pacman -S xorg-xauth

(guest)$ pacman -S xorg-xauth xorg-xclock openssh xorg-fonts-type1

As in the last section, our goal here is to run the application xclock, a simple graphical clock as a proof of concept. If you try to run it now, this should happen:

(guest)$ xclock Error: Can't open display:

That’s why we are going to use SSH, to help us get the graphical output. The default TCP port of SSH is 22, but probably your localhost already has reserved this port. Besides this, the IP address QEMU gives to your VM usually isn’t routable. So, let’s map our guest 22 to another port, let’s say 1337, and expose this port using the hostfwd option. This can be done with these additional QEMU flags:

-net nic: creates a basic network card;
-net user,hostfwd=tcp::1337-:22: maps port host’s 1337 to guest’s 22.

If you try to access the machine now via ssh (with QEMU running with this new parameters), it won’t be possible yet:

(host)$ ssh root@localhost -p 1337 ssh_exchange_identification: read: Connection
reset by peer

In the guest, if you run systemctl status sshd, you can see that it isn’t running. Let’s configure sshd before running it. Let’s edit guest’s /etc/ssh/sshd_config to ensure that you have those lines uncommented and edited:

PermitRootLogin yes # allows root login with password via ssh
X11Forwarding yes # allows XForwarding

Let’s configure sshd to run at boot and to start now:

(guest)$ systemctl enable sshd
(guest)$ systemctl start sshd
(guest)$ systemctl status sshd
* sshd.service - OpenSSH Daemon
     Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: disabled)
     Active: active (running)
     ...

To have access to the guest without using a password every time, you can place your SSH public key inside the file ~/.ssh/authorized_keys. You can copy and paste your key in this file while the guest machine is running, or you can mount the guest rootfs and write in the file, like this:

(guest)$ # Stop guest machine using CTRL+A, X
(host)$ sudo mount arch_disk.raw mnt
(host)$ sudo su
(root-host)$ cat /home/user/.ssh/id_rsa.pub > mnt/root/.ssh/authorized_keys
(root-host)$ exit
(host)$ sudo umount mnt

Start the virtual machine again and, with a second terminal session, try to login:

(host)$ ssh root@localhost -p 1337
Last login: Mon Feb 18 17:57:15 2019 from 10.0.2.2
(guest)$

You can use exit command to close the session. If you don’t have a .Xauthority file on your home folder, it will prompt a warning, but don’t worry: after the warning, the file will be created. Access the virtual machine using ssh with the argument -X:

(host)$ ssh -X root@localhost -p 1337

And check the hours with xclock! To exit, you can also use CTRL+D.

Screen capture showing xclock with XForwarding

Shared folder between host and guest

Copy-pasting is definitely not the best way to send a file to your guest machine. Hopefully, we can easily solve this problem by sharing a folder between machines.

If you ran make x86_64_defconfig and make kvmconfig before the kernel compilation, you should already have the required modules enabled. If you get errors, please make sure your kernel has the following options enabled:

(host)$ grep 'VIRTIO_PCI=\|NET_9P=\|9P_FS=\|NET_9P_V\|IG_PCI=' .config
CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_PCI=y
CONFIG_VIRTIO_PCI=y
CONFIG_9P_FS=y

If something looks like # CONFIG_XXX is not set, you should enable it.

Now, elect a folder to be your shared holder, for example /home/user/shared. Then, we need to add more arguments to our QEMU command:

-fsdev local,id=fs1,path=/home/user/shared,security_model=none: this will add a new file system device to our emulation. Make sure to put the right directory at path. Don’t worry about security_model=none, this argument will let the permission of creating/modifying files inside the guest as if was created by the host user.
-device virtio-9p-pci,fsdev=fs1,mount_tag=shared_folder: this defines the name and type of the virtual device.

We need to edit our guest /etc/fstab. It should look like this:

# Static information about the filesystems.
# See fstab(5) for details.
# <file system> <dir> <type> <options> <dump> <pass>
shared_folder /root/host_folder 9p trans=virtio 0 0

This determines the mounting pointing of the shared folder. As long you are consistent, you can choose whatever name for shared_folder and host_folder. Reboot the guest machine and then:

(guest)$ ls /root/
host_folder

Adding a new user

For some testing or actions, it’s maybe useful to have a regular user, that is not the root one. For instance, makepkg refuses to run as root for safety reasons. The following commands should be used in the guest. Use useradd -m user -G wheel to create a new user with the name user, with a home directory and part of group wheel. You can set a password for it using passwd user. To use this account, just type su user. If you want to login to this user from SSH, remember to add the public key to /home/user/.ssh/authorized_keys. If you want to be able to run any sudo command without typing password, run as root user visudo and uncomment the line that says %wheel ALL=(ALL) NOPASSWD: ALL.

Conclusion

You may also want your kernel to have a custom name, it may be useful for you to organize your versions. You can do this changing the LOCALVERSION value at menuconfig or simply running make LOCALVERSION=, e.g.:

(host)$ make LOCALVERSION=-VM ...
(host)$ qemu-system-x86_64 ...
(guest)$ uname -r
5.0.0-rc1-VM

Now you can easily hack and test your kernel! I recommend you read this section of the ArchWiki, as you’ll find some cool tips to improve your VM performance. You may also want to create scripts and alias to not deal with all the flags, that will definitively make your life easier.

This content was originally posted at Collabora’s blog , as part of my Software Engineer job. It was slight updated and edited before being posted here.

The observant will notice that the argument used by QEMU to specify the first drive is hda, which results in the kernel enumerating a drive as sda. Historically IDE drives were labelled hdX (where X is an increasing drive letter), but for quite a while it has been typical for IDE drives to be accessed via an emulation layer in the SCSI subsystem, which labels drive sdX. Also in contrast to how most physical hard drives are used, we have not created a boot partition and partitions in the virtual drive, instead treating the whole device as a partition, and thus lacking the numerical suffix we would typically see when referring to specific partitions. ↩︎