Rethinking UI APIs

2017-09-14

A few weeks ago I wrote a rant, "React and Redux are a joke right?". While many of my friends got the point most of the comments clearly responded mostly to the title without really getting the point. That's my fault for burying the point instead of making it clear, and for picking a clickbait title although that did get people to look.

I wrote a response back then but wordpress ate it. Since then I spent 2 weeks redoing my blog to be a static blog and now I'm finally getting around to writing what I hope will clear up my point.

Maybe my point can best be summed up as "Let's rethink the entire UI paradigm. Let's consider alternatives to the standard retained mode GUI. Those APIs are making us write way too much code".

If you're not familiar with what a "retained mode GUI" is it's any graphic user interface that requires a tree of objects that represent the UI. You build up the tree of objects. It sticks around "is retained" even if your code is not running. The UI system looks at the tree of objects to show the UI. This is the single most common paradigm for UIs. Examples include the DOM and all the DOM elements. On iOS all the UI objects like UIButton, UILabel, UITextField. Android, MacOS, Windows etc all work this way.

There maybe many reasons for doing it that way and I'm not arguing to stop. Rather I'm just pointing out there's another style, called "Immediate Mode GUI", and if you go try and use it for a complex UI you'll find you probably need to write 1/5th as much code to get things done as you do with a retained mode GUI.

The issue with a retained mode GUI is having to manage all of those objects. You need some way to associate your data with their representation in the UI. You also need to marshal data between your copy of the data and the UI's copy of the data. On top of that most retained mode GUIs are not that fast so you have to minimize churn. You need to try to manipulate, create, and destroy as few of these GUI objects as possible otherwise your app will be unresponsive.

Contrast that with an Immediate Mode GUI. In an Immediate Mode GUI there are no objects, there is no (or nearly no) UI state at all. The entire UI is recreated every time it's rendered. This is no different than many game engines at a low level. You have a screen of pixels. To make things appear to move over time, to animate them, you erase the entire screen and re-draw every single pixel on the entire screen. An Immediate Mode GUI is similar, there's nothing to delete because nothing was created. Instead the UI is drawn by you by calling the functions in the Immediate Mode GUI API. This draws the entire UI and with the state of the mouse and keyboard also handles the one single UI object that's actually being interacted with.

What happens when you do this, all the code related to creating, deleting, and managing GUI objects disappears because there are no GUI objects. All the code related to marshaling data into and out of the GUI system disappears because the GUI system doesn't retain any data. All the code related to try to touch as few GUI objects as possible disappears since there are no objects to touch.

And so, that's what struck me when I was trying to write some performant code in React and some site suggested Redux. React as a pattern is fine. It or something close to it would work well in an Immediate Mode GUI. But in practice part of React's purpose is to minimize creation, deletion, and manipulation of GUI objects (DOM Elements for ReactDOM, native elements for React Native). To do that it tracks lots of stuff about those objects. It also has integrated functions to try to compare your data to the GUI's data. It has its this.setState system that gets more and more convoluted with requiring you not to set state directly and not even to inspect state directly as a change might be pending. All of that disappears in an Immediate Mode GUI. Redux is one of many suggested ways to take all that tracking a step further, to make it work across branches of your GUI object hierarchy. All of that disappears in an Immediate mode GUI.

Now I'm not saying there aren't other reason you might want command patterns. Nor am I saying you don't want more structure to the ways you manipulate your data. But, those ways should be 100% unrelated to your UI system. Your UI system should not be influencing how you use and store your data.

When I wrote the title of my rant "React and Redux are a joke right?" my mindset was realizing that the entire construct of React and Redux are their to manage a retained mode GUI. If we were using an Immediate Mode GUI we wouldn't need them. The joke is on us thinking we're making our lives easier when in reality we're just making it more complex by piling on solution on top of solution on top of solution when the real problem is below all of that.

Now, I'm not suggesting we all switch to Immediate Mode GUIs. Rather, I'm suggesting that experience writing a complex UI with an immediate mode GUI might influence ways to make things better. Maybe there's some new design between retained mode GUIs and Immediate Mode GUIs that would be best. Maybe a different API or paradigm all together. I don't know if it would lead anywhere. But, I think there's room for research. As a more concrete example the entire Unity3D UI is basically an immediate mode GUI. All of their 1000s of plugins use it. Their API could probably use a refactoring but Unity3D is clearly a complex UI and it's running using an Immediate Mode style system so it's at least some evidence that such a system can work.

There are lots of objections to Immediate Mode GUIs. The biggest is probably it's assumed they are CPU hogs. I have not written any benchmarks and it would be hard as the 2 paradigms are so different it would be like those benchmarks where a C++ person translates their benchmark to Haskell not really getting Haskell end up making a very inefficient benchmark.

That said, I think the CPU objection might be overblown and might possibly be the complete opposite with Immediate Mode GUIs using less CPU than their retained mode counterparts. The reason I think this is true is that Immediate Mode GUIs almost always run at 60fps (or smooth) and retained mode GUIs almost never run smooth. To make a retained mode GUI run smooth for a complex UI requires tons and tons of code. Code like React's checks for changes, like the state tracking stuff, all the componentDidMount, componentWillUnmount, shouldComponentMount stuff, various kinds of caches etc. All that code, possibly 5x to 10x the code of an Immediate Mode GUI is code that is using the CPU. And when I say 5x to 10x the code I don't necessarily mean just written code. I mean executed code. A loop comparing if properties changed is code that's running even if the code implementing the loop is small. Some languages and or system generate tons of code for you to help with these issues. You didn't have to personally write the code but it is there bloating your app, using CPU. When you see a retained mode GUI like a webpage not be entirely responsive or take more than 16ms to update that's because the CPU was running 100% and could not keep up. That situation rarely happens with an Immediate Mode GUI. That suggests to me it's possible the Immediate Mode GUI is actually using less CPU.

Other than Unity3D which is probably the most used Immediate Mode GUI in use but is tied to their editor the most popular Immediate Mode GUI is Dear ImGUI. Yes, it's only C++. Yes, it's not designed for the Web (or phones). Again, for like the 1000th time, that is not the point. The point is not Dear ImGUI as a solution. The point is Dear ImGUI as inspiration. In realizing how much extra code we're being made to write and or execute to deal with retained mode APIs. The point is to take a step back and consider that just maybe this whole ubiquitous retained mode GUI API should be revisited.

Comments