That Time I Fixed a Blender Bug

(It’s been six months since I fixed the bug here; details will be slightly wrong if not outright hallucinated.)

On Linux, there’s this Compose key. When you press it, the next few keys aren’t actually typed, and instead append those characters into an IME (input method editor, but it’s mostly known by the acronym) key accumulator.

Guess what part Blender missed, if you were using Wayland? The part that I bolded in an obvious act of foreshadowing? (It’s my blog and only I get access to the precognition layer.) I ran into this problem, and it was ruining my preconception of Blender being this pristine monolith of perfection. I couldn’t even type characters like ¤ or ⋄ or — or ‽ anymore, at least not without going back (gasp!) and manually deleting keys that should have never made it into the text box.

Aside: This probably hadn’t come up before because the “Compose key on Wayland” is a niche within a niche. The same code path is also used for dead keys, which are something like: ` a → à; however, the ` key here is an entirely different keysym.

You can see the source code for compose-key handling here, and you may notice that composition is done by the application itself, and not by Wayland!

The logic here is something like this:

• When a key is typed, that’s a keysym. Blender receives a Wayland event for that keysym, and hands it over to the xkb compose statemachine (EDIT: note that contrary to the name, this function is actually a Blender function and not a libxkb function! foreshadowing!).

• The compose state machine, from the xkb (X keyboard, at a guess) library decides whether or not this keysym either:

  • Starts a new compose sequence,
  • continues this previous compose sequence,
  • accepts this compose sequence and returns a UTF-8 string,
  • or rejects this compose sequence.

• If the compose state machine doesn’t recognize a keysym, Blender then asks xkb what text should be typed for the specific keysym.

The bug here was that Blender’s wrapper around the xkb compose state machine only reported that the event was handled if xkb’s compose state machine accepted the compose sequence.

One two-line fix later, the <Compose> key worked again.

You may notice that I didn’t fix handling for the first character, or the character that causes a rejection, of the compose sequence. The first is easy: it doesn’t affect my case, where the <Compose> key press doesn’t have any UTF-8 text associated with it (and also I didn’t think of it at the time).

The second is a little tougher to justify. I pored through literature (typed <Compose> a ` into various text boxes that I found lying around) and couldn’t find consensus (the KDE application launcher didn’t consume the key; my browser’s searchbox did) so I decided to err towards adding less code. (In hindsight: noting this in the pull request would’ve been a great idea!) Campbell Barton chose to handle this case also, which (in hindsight again) was the better move.

Some of you may know that xkb_compose_state_get_utf8 doesn’t return the length of the truncated UTF-8 result, and instead returns how many bytes the replacement sequence takes up.

This means that any replacement sequence over five bytes long is an out-of-bounds write and (in my particular situation) a segmentation fault. This is admittedly an edge case; if you bind <Compose> a s d f to the first ninety characters of the bee movie script, the resulting potential security hole is kinda deserved.