Could ImGUI be the future of GUIs?


A random un-thought out idea that came up was could something like Dear ImGUI ever be the future for a mainstream UI library?

For those that don't know what an Immediate Mode GUI or ImGUI is there's a semi famous video by Casey Muratori about it from 2005ish.

Most programmers that use an ImGUI style find it infinitely easier to make UIs with them than traditional retained mode GUIs. They also find them significantly more performant.

The typical retained mode object oriented GUI framework is a system where you basically create a scenegraph of GUI framework widgets. (windows, grids, slider, buttons, checkboxes, etc). You copy your data into those widgets. You then wait for events or callbacks to get told when a widget was edited. You then query the widget's values and copy them back into your data.

This pattern is used in practically every GUI system out there. Windows, WFP, HTML DOM, Apple UIKit, Qt, you name it 99% of GUI frameworks are Retained Mode Object Oriented Scenegraph GUIs.

A few problems with this GUI style are

In contrast in an ImGUI there are no objects and there is almost no state. The simple explanation of most ImGUIs is that you call functions like

// draw a button
if (ImGUI::Button("Click Me")) {
// draw slider
ImGUI::SliderFloat("Speed:" &someInstance.speed, 0.0f, 100.0f);

Button and Slider do two things.

  1. They append into a vector (array) the positions and texture coordinates needed to draw the widget (or not insert them if they'd be clipped off screen or outside the current window / clip rectangle)

  2. They check the position of the mouse pointer, state of keyboard, etc to manipulate that widget. If the data changed they return it immediately

So, pluses:

Possible minus

Perceived but probably not minuses

I guess I'm really curious. I know most GUI framework authors are skeptical that ImGUIs are a good pattern. AFAICT though, no one has really tried. As mentioned above most ImGUIs are used for game development. It would take a concerted effort to try to find the right patterns to completely replicate something as fancy as say Apple's UIKit. Could it be done and stay performant? Would it lose the performance by adding back in all the features? Does the basic design of an ImGUI mean it would end up keeping the perf and the easy of use? Would we find certain features are just impossible to really implement without a scenegraph?

Let me also add that to some degree React is similar to an ImGUI in usage. React has JSX but it's just a shorthand for function calls. The biggest differences would be

If we were to translate the code above into some imaginary ImReact it might be something like

const Button = (props) => {
  return ImGUI:Button(props.caption);

const SliderFloat = (props) => {
  return ImGUI:SliderFloat(props.caption, props.value, props.min, props.max);

const Form = (props) => {
  if (<Button caption="Click Me">) {
  <SliderFloat min="0" max="100" value="&props.speed" caption="Speed:" />

Just looking at that React code you can see the translation back into real code is really straight forward.

Not exactly sure how the update to speed would work but I guess I'm mixing C++ (ImGUI) with JavaScript (React). Typical ImGUIs either have the pattern of being able to pass in a pointer to a primitive, something JavaScript doesn't have. Or, they return the new value as in

newValue = ImGUI::SliderFloat(caption, currentValue, min, max);

which if you want to use the same as the Dear ImGUI C++ example you'd write

someInstance.speed = ImGUI::SliderFloat("Speed:", someInstance.speed, 0.0f, 100.0f);

So if we assumed that style of API then

const Button = (props) => {
  return ImGUI:Button(props.caption);

const SliderFloat = (props) => {
  return ImGUI:SliderFloat(props.caption, props.value, props.min, props.max);

const Form = (props) => {
  if (<Button caption="Click Me">) {
  props.speed = (<SliderFloat min="0" max="100" value="{props.speed}" caption="Speed:" />);

Notice the components are not returning virtual dom nodes since there's no need. The only thing we're really taking is JSX just to show that you could use a React style pattern if you wanted to.

Note: Don't get caught up in the direct state manipulation in the example. How you update state should not be dicated by your UI library. You're free the manage state anyway you please regardless of which UI system you use. Still the example shows how simple ImGUI style is.

state.value = ImGUI:SliderFloat(caption, value, min, max);

is certainly simpler than

// at init time
const slider = new SliderWidget(caption, state.value, min, max);
slider.onChange = function(newValue) {
  state.value = newValue;

// if state.value changed slider needs to show the new value
function updateSlider(newValue) {
  state.value = newValue;

Even worse now you need to some how call updateSlider either everywhere state.value is updated or you need to write some elaborate system so that all places that want to update state.value call into a system that tracks all the widgets and what state they reflect.

ImGUI libraries needs no such complication. There is no widget. Every frame whatever value is in the state is what's in the widget. This is the same promise of React but React ends up being hobbled by the fact that it's on top of slow retained mode GUI libraries.

As an example of complexity possible the most prolific ImGUI is Unity's Editor UI.

So at least there is some precedence of using an ImGUI in user facing app instead of just a game even if Unity itself is for making games.

There are also lots of screenshots of various ImGUI made UIs in the readme.

Here is also a live version of the included example in the Dear ImGUI library

If you decide to interact with it be aware that it's not actually been designed for the browser and so has issues that need fixing. Those issues can easily be fixed so don't get bogged down in nitpicking tiny issues. Rather, notice how complex the UI is and yet it's running at 60fps. Use the "examples" menu in the main window and open more windows. Expand the examples in the main window and see all kinds of live and complex widgets. Now imagine you tried to make just as complex UI using HTML/DOM/React. Not only would the HTML/DOM version have lots of pauses and likely not run 60fps but the code to actually implement it would probably be 5x to 10x as much code along multiple dimensions. One dimension is how much code you have to write to implement the UI using HTML/DOM and/or React vs ImGUI. The other dimension is how much code executes to get the UI on the screen. I suspect the amount of CPU instructions executed in the HTML/DOM version is up to 100x more than the ImGUI version.

Consider the ImGUI::Button function vs making <button> element.

For the <button> element

  1. HTMLButtonElement object as to be created.

    It has all of these properties that need to be set to something

     autofocus: boolean 
     disabled: boolean 
     form: object 
     formAction: string 
     formEnctype: string 
     formMethod: string 
     formNoValidate: boolean 
     formTarget: string 
     name: string 
     type: string 
     value: string 
     willValidate: boolean 
     validity: object ValidityState
     validationMessage: string 
     labels: object NodeList
     title: string 
     lang: string 
     translate: boolean 
     dir: string 
     dataset: object DOMStringMap
     hidden: boolean 
     tabIndex: number 
     accessKey: string 
     draggable: boolean 
     spellcheck: boolean 
     autocapitalize: string 
     contentEditable: string 
     isContentEditable: boolean 
     inputMode: string 
     offsetParent: object 
     offsetTop: number 
     offsetLeft: number 
     offsetWidth: number 
     offsetHeight: number 
     style: object CSSStyleDeclaration
     namespaceURI: string 
     localName: string 
     tagName: string 
     id: string 
     classList: object DOMTokenList
     attributes: object NamedNodeMap
     scrollTop: number 
     scrollLeft: number 
     scrollWidth: number 
     scrollHeight: number 
     clientTop: number 
     clientLeft: number 
     clientWidth: number 
     clientHeight: number 
     attributeStyleMap: object StylePropertyMap
     previousElementSibling: object 
     nextElementSibling: object 
     children: object HTMLCollection
     firstElementChild: object 
     lastElementChild: object 
     childElementCount: number 
     nodeType: number 
     nodeName: string 
     baseURI: string 
     isConnected: boolean 
     ownerDocument: object HTMLDocument
     parentNode: object 
     parentElement: object 
     childNodes: object NodeList
     firstChild: object 
     lastChild: object 
     previousSibling: object 
     nextSibling: object 
     nodeValue: object 
     textContent: string 
  2. More objects need to be created.

    Looking above we can see we need to create

    NodeList            // an empty list of children of this button
    HTMLCollection      // another empty list of children of this button
    StylePropertyMap    //
    NameNodeMap         // the attributes
    DOMTokenList        // the CSS classes as a list
    CSSStyleDeclaration // an object used to deal with CSS
    DOMStringMap        // empty but used for dataset attributes
    ValidityState       // ?? no idea

This is just creation time so far. Tons of properties need to be set to defaults, filled out with empty strings and or other objects need to be created and those objects also need all their properties filled out and as well may need deeper objects created.

Now that an HTMLButtonElememt exists it get inserted into the DOM

At render time the browser will walk the DOM, I'm sure there is some amount of caching but it needs to figure out where the button is. It will likely build some separate internal scene graph separate from the DOM itself which is rendering specific so 1000s more lines of code get executed.

Eventually it will get the to point to render the button. Here again it has to check the 100s of CSS attributes. Text color? Font size? Font Family? Text Shadow? Transform? Animation? Border? Multiple Borders? Background color? Background Image? Background gradient? Is it transparent? Is it on its own stacking context? Literally 100s of options.

Let's assume it's using nothing special, eventually it will generate some quad vertices to render font glyphs. It will likely render these glyphs into a texture or grid of textures for the stacking context. It does this as an optimization so ideally if a different stacking context has its content change but nothing in this stack context changes it can skip re-rendering the texture(s) for this context and just use the one it created last time.

I'm sure there's a 100 other steps I missing related to caching positions, marking things as computed so they don't get recomputed, and on and on.

Compare to ImGUI:Button which is just a function, not an object. All it effectively does is

  1. Clip the button rectangle to the current clip space and exit if it's completely clipped
  2. Insert the vertices for the rectangle of the button into the pre-allocated vertex array
  3. Insert the vertices for each glyph stopping when the first glyph is clipped by the button area.
  4. Return true if the mouse button was pressed and if its position is inside button rectangle, else false.

That's it

Note that those 4 steps also exist in the browser in HTML/DOM land except they are 4 steps of 100s.

So, in summary, ImGUI style is potentially much faster and easier to use. It's both easier to use in the simple case and easier to use in the complex case. The API is easier to use. It's easier to reason about. There is no state. There are no objects. There is no data marshalling. There are no events or callbacks. Because it's so fast when the UI gets complex no giant frameworks like React's virtual dom need to be created. Because of the speed little to no effort is required to workaround slowness like with the DOM. More research into ImGUI style UIs could lead to huge gains in productivity.


Thoughts on Magic Ink


If you've never read a Bret Victor paper or watched a Bret Victor presentation you're missing out. For paper's I'd recommend starting with Learnable Programming.

Recently via some random chain of events I stumbled on the Future of Coding podcast. The theme I guess could be summed up as "we're doing it wrong!". "It" being programming computers. I don't know if it's true that we're doing it wrong but it's definitely fun to think about.

In a couple of the podcasts the Bret Victor paper, Magic Ink, was brought up. I hadn't read it yet so I checked it out.

It's inspiring as usual for Bret Victor. I'm not sure if I can do a good job of summarizing it but my take away was that with only a little extra thought it becomes clear that lots of software could have far better user experiences.

Bret claims that interface designers should consider themselves graphic designers first and foremost and that a graphic designer's skill is supposed to be to make images and make it easy to understand and compare information as easily and quickly as possible. Many software applications have given little to no thought about what the user actually wants to accomplish, what data they need to see and how to present it for the app to be useful to them and how most apps make it tedious to get what the user really wants out of them and that maybe with a little more thought much of that could be fixed. I really liked that idea and many of the examples were very compelling.

The paper was written in or around 2005 so before nearly everyone had a PDA in their pocket like they do now. Of course lots of people did have PDAs on them in 2005 but internet access over cellular was still rare and expensive.

In any case I'm curious about some of the things that seem to have changed in the 13 years since that paper was posted.

I think probably the biggest example of something that sounded like a good idea at the time and maybe still is but it's probably not as clear.... Much software could be better if it took context into account. That much is somewhat obvious. One example given might be what a map like Google Maps shows you when you open it. If it can know your location it should probably start with a map of where you currently are. If you click the search bar maybe it should try to guess where you're most likely to want to go as possible suggestions like if it's 8am on a weekday maybe it would suggest your work or if it's 6pm on a weekday maybe it should suggest your home. There was a time when maps didn't have GPS data so they couldn't start by showing you where you are. I'm not sure even Google Maps looks at your location and the time of day and what day of the week to decide what options to show you when you click search.

Bret went on to suggest that if all apps communicated with each other then let's say you just read an email about a business trip you need to take. If you opened your maps app now and the email app and the maps app could talk to each other then there should be a way for the map app to guess that maybe what you want to look at right now is the location of the business trip.

Not sure that's a good suggestion BUT, in 2018 the idea that multiple apps would all share their data sounds super scary because we know now that every app would record everything, send it to their servers, share it with their various partners, most of them advertizing partners and we'd all have profiles built on us. Of course we have that today but not to the level that the paper was hoping to see in the future of 2005.

I'm really curious if Mr. Victor still sees that world as a possible future or if he's settled that it's impossible to do without too many privacy issues or if he's got ideas for solutions.

Another example of something that seems to have changed, the paper suggests using context to help searches. I wish Google did this (or did it better). One example in the paper is if you search for "Blender", "3DSMax", "Modo", and then search for "Maya" it should be abundantly clear that you mean "Maya" the 3D Software Application by Autodesk and not "Maya" the civilization from Central America.

I find these context mistakes infuriating, especially when I know the application has the data it needs. Sometime in 2012, while in Tokyo, I searched for "pizza" on Google Maps and it ended up showing me some place in Texas. Given it knew my GPS and even if it didn't know it at the moment it new my previous searches it seems really stupid to somehow think I was searching for anything in Texas without explicitly typing "Texas" in my search. Even typing "Texas" in my search doesn't seem like it should show me Texas USA if I'm in Tokyo as I could be looking for a store called "Texas" or a restaurant serving "Texas BBQ" or "Texas style Mexican food". It seems like it should require some pretty specific extra context for it to ever give me any results from the state of Texas if it knows I'm currently in Tokyo.

In any case though, of the problems I have with context based interfaces is inconsistency. In Google search maybe it's not so bad to get different results each time. Things change, there's new info that matches the search. But, there are time when I feel like consistency trumps context. I'm trying to think of a good example but until then I guess the way I'd explain it is I have muscle memory. Click this option, press down 3 times, press enter to select. Or click here to make a new fob, then click 2 inches to the right to set the fob's options. If a context aware app made it so my muscle memory failed often I think it would drive me nuts.

Not quite the same but an example in the paper is typing a zip code and having the app start zooming as digits are entered. Type a "9" and we know it's the west coast, followed by a "4" brings us to the SF Bay Area, followed by a "5" brings us to the Easy Bay.

That sounds great except ... is it? Maybe it's a different case but Google tried live search results. As you'd type each letter Google would change the results live on the page. Google since got rid of that feature. I'd be curious to know why. The claim is that it wasn't good for mobile so why not make it consistent. I'm not sure I buy that claim as all kinds of things are different on mobile. For one I don't have a mouse with 1 or 2 buttons that can hover over things and I have a tiny display.

In any case though I absolutely loathed the old "show them as you type instant search results". Let's make it clear, I'm not talking about the search suggestions that appear just below the search input area. I'm talking that Google used to update the entire page as I typed. The problem for me is I'm often referencing things on the page as I type. With Google making them disappear I couldn't reference them and it actually made this harder for me.

Given that I wondered if similar issues would happen with things suggested in the paper related to instantly showing new info as the user starts entering their data. I think I'm pretty glad Google Maps doesn't jump around as I type but waits for me to select something. I'm curious if others have noticed that too that trying to be too responsive can actually be annoying and or counter productive.

The last potion of the paper was about his award winning Bart app that ran as an accessory on MacOS. He went over in detail all the design decisions and it was certainly a beautiful app with lots of thought and care put into the design.

My personal reaction though was not quite as awe struck and it made me wonder if "Bart" wasn't also the product of a personal bubble.

The first thing that stuck out was it had this semi-fancy interface for choosing a destination in which it shows a map of the entire Bart system. The Bart system is not really that big. There are basically 5 lines, they all come out of Oakland so it's an extremely simple system. You could hover your mouse down the tracks to choose any station.

Compare to Tokyo where I live there are something like 40 lines and 2000 stations. Most of the line cross other lines, sometimes multiple times zig zagging here and there. Such a UI would arguably never work here.

But, thinking about it more at almost seemed like Mr. Victor missed is own advice. The paper points out that often software isn't helping the user do what they actually want to do and the Bart app is a perfect example of exactly that. No one is trying to get from one Bart station to another. They are actually trying to get from one place to another neither of which is a Bart station!

If I'm at the north side of Lake Merritt in Oakland and I want to go to the Metreon in San Francisco my goal is not to "take the Bart". It's to get from where I am to where I want to be. That might be bus, Uber, Lyft, Ferry, Bart, maybe even some carpool service.

That idea came up because the paper mentioned showing only one route and that clearly wouldn't work in Tokyo where there are often multiple routes. There's the fastest route which could depend on the time of day. There's the route that gets you to your destination soonest which might be different. In other words, if you leave right now one route might get you there 50 minutes from now. Another route might get you there 60 minutes from now but you leave 15 minutes later so only 45 minutes travel time. Yet another route might require less transfers, less walking either to the bus stop or train station or at transfer area. One route might take the bus, another a train. One route might be cheaper via more trips on the same company's lines instead of switching companies. There are at least 10 different train/subway/lightrail companies in Tokyo.

From my own apartment downtown I'm only about a 2-3 minute walk to a bus stop but when I ask how to get somewhere, depending on where it is it might be best for me to walk to one of the 5 stations within a 25 minute walk. If it's 5am and I'm planning to go to the beach there is no bus running so I need to walk 15 minutes to a major station where as at 7am its much faster to catch the bus to that station.

As another example from Shibuya to Azabujuban there are at least 4 routes.

Why would I pick one over the other? Well, Ginza line and Hanzomon Line run parallel but their platforms in Shibuya are 5-6 minute walk apart. If I'm nearer one or the other I'd pick the one I'm nearer. Hanzomon Line also skips one station so it might be faster. I have a similar dilemma at my destination as the Oedo platform and Namboku platforms are also 5-6 minutes apart so I might want to take into account my final destination. Another concern would be if I have a commuter pass then one of those routes might be free or 1/2 free. Yet another price consideration is even if I don't have a pass the bus is the cheapest option as it's one bus where as the 3 other routes each use two diferent train companies. The bus takes 25 minutes where as the train routes only take 12-15 minutes but, the bus starts at Shibuya so I'm almost guaranteed to have a seat. If I have the time I might perfer a comfortable seated ride in the bus vs standing on the train and having the 2-3 minute transfer walks. Yet another consideration would be if I'm carrying something heavy like if I just bought something maybe I'd prefer a cab/uber/lyft.

This all gets even worse if my destination is between stations. For example from my house to Enoshima, an island a little over an hour a away, I can go

Considerations? The last one is cheaper by $3. ($9 vs $12). The monorail might be more scenic. On the Shonan-Shinjuku line I can pay an extra $9 and get a fancy comfortable airplane like seat for 40 minutes of my trip.

The Bart app's one route design struck a cord knowning it wouldn't work in Tokyo which after a little thought made it clear it wasn't following its own suggestion and solving the actual user's problem of getting from A to B where A and B are not "Bart Stations".

In any case the paper is still amazing and thought provoking and you should totally read it and take away the bigger message. I'd love to hear your thoughts.


OffscreenCanvas and Commit


Chrome is planning to ship OffscreenCanvas.

I know lots of devs that have been wanting that feature for ages so it's exciting to see it finally here. What is OffscreenCanvas? It's basically the ability to draw to a canvas from a web worker.

Drawing a complex scene often takes lots of CPU power. By being able to move all those calculations to a web worker we can make sure the main thread, the one reading the keyboard, responding to the mouse, etc... has all the power it needs to stay responsive.

There was debate for a long time about how it should be done. Ian Hickson wrote one idea orginally and with zero review stuck it in the spec. MDN even documented it though it was never implemented by anyone. I wrote another proposal in around 2012 that pointed out the issues with the one in the spec and suggested another solution. That was never implemented either though it was referenced from time to time as a reminder of some of the issues invovled.

In any case the current solution that chrome appears about to ship is that WebGL and Canvas2D mostly work in workers exactly the same as they do outside if workers. There's a small amount of code you need to write to transfer control of a canvas to some object that will exist inside the worker. The worker then creates a WebGL context or a 2D context and renders just like it would if it was in the main page. Results show up automatically just like they do on the main page. In other words, for those familar with graphics programming, there is no explict present or swapBuffers call. The moment you call one of the rendering functions in the respective APIs the browser "queues a task to do the present/swap" when your event exits.

This is great as it's the path of least surprise. No crazy new changes are needed to your code.

Even better they added requestAnimationFrame to workers so a worker can effectively just do a standard render loop

function render(time) {

So far so good.

But, ... they are also considering adding something else which is an explicit present/swapbuffers function called commit. Normal JavaScript apps would likely not use this API. Rather it's an API for WebAssembly ports of native games.

The issue they are trying to solve is that most native games run in what's called a "spinloop". They have code like this

   while(!userWantsToExit) {

They never pause and never stop rendering they just run as fast as they can in a loop. By adding commit they feel they can better support native ports.

I see several problems with this approach and I hope I've convinced them to put the brakes and do a little more testing before releasing this API.

You can NOT use any other Web APIs with this model!

For those that don't know how JavaScript works it works on an event model. You provide functions to be called when certain events happen. Events include things like key pressed, mouse clicked, button clicked, slider moved, image downloaded, websocket message received, etc..

When one of these events arrives the browser calls the JavaScript function you assigned to that event. Your JavaScrpt runs AND THE BROWSER IS FROZEN until your JavaScript exits. Once your JavaScript exits the browser will run any other events on the list of events waiting to be run.

A spinloop like the one enabled by commit means your JavaScript never exits so you'll never process any other events. In other words, the worker rendering with a commit spinloop CAN NOT USE ANY OTHER WEB APIs. It can not receive messages from the main page. It can not download images. It can not read files or request data from a server. It can not use a websocket.

WOW! An api that removes the ability to use all other APIs!?!?!

When I asked about it I was told the solution is to use SharedArrayBuffers. SharedArrayBuffers are a way for workers and the main thread to share a chunk of memory with each other. They can all read and write from it and so they can use shared array buffers to commicate with each other.

Ok, I guess that works. It sounds like a ton of work. For example there is no way to get the raw data from an image in the current Web APIs. You can download images and use them in Canvas 2D and WebGL but as we've just pointed out you can't use those APIs in a worker using commit. Because you can't get the raw data you also can't download those imaegs in another worker or the main thread and pass them via sharedmemorybuffers into the render/commit worker. Soooo, you're left to write your own image decoders throwing away a bunch of the web API again. This is one reason why webassembly apps are so bloated is they include their own versions of image loading libraries even though libraries already exist in the browser.

I suppose that's a minor thing but that's not the end of it.

The next issue is what happens when your page that has worker that's using commit is not the front tab. This is a problem too.

With a normal requestAnimationFrame loop the only thing that happens is the browser stops sending animation frame events. It's still fully able to deliever other events. Events for fetching json, events for loading images, events loading other data. Your program can keep responding.

With commit it was suggested they'd just block commit forever until the tab is put in the front again. The problem then is that you've got the main page and or workers still receiving mesages but when they try to communicate those messages to the rendering worker that worker never responds. It's frozen. This will be a HUGE source of race bugs. Developers think their code works only to find it fails is subtle and hard to reproduce ways depending on when the user switches tabs. Not good.

Okay, so they suggested maybe they can throttle commit. They'll call it just once a second for example. Unfortunately we can show that's not a solution. Many GPU pages (and many even non-GPU pages) can be really slow. Here's a page that's really slow at least on my machine. When it's the front tab I can barely type. I'm glad that page exists as it's super educational so I think pages like that should exist. If I make some other tab the front tab my machine is back to normal and responsive. Now imagine if that page used commit and commit was only throttled. Imagine it was called once per second. The experience would be horrible and my machine would still seem unusable as once a second my machine would hiccup as it processes that graphics page offscreen. So no, throttling is not a viable solution. whatever solution happens must stop the rendering period.

So what solutions are their?

Well why do we need commit at all?

The reason some people think we need commit is because they want to support native ports to webassembly. A typical spinloop based C/C++ program might have some code like this

void main() {
  KeyboardSystem* keySys = new KeyboardSystem();
  GraphicsSyatem* gfxSys = new GraphicsSystem();
  DataSystem* dataSys = new DataSystem();

  GameData* gameData = dataSys.loadData();

  bool done = false;
  while(!done) {
    done = keySys.checkKeyboard();


The problem they see is that this code can't work in the browser's current system. Like I mentioned above the browser only calls your code via events. Your code needs to exit so that the browser can then process the next event. The code above never exits. If it does exit then keySys, gfxSys, dataSys and gameData would all be cleaned up which is not what we want.

Of course programmers can refactor their code so this isn't a problem but the people pushing for the commit are trying to to make it so those developers don't have to change their code and things will just work.

Here comes a place where we disagree. First, the amount of work to refactor that code is small. Of course the example above is small but I suspect even large native code bases would not take that much work to refactor to work with events. You'd need a few globals or singletons but otherwise you just split up your code

static KeyboardSystem* keySys;
static GraphicsSyatem* gfxSys;
static DataSystem* dataSys;

static GameData* gameData;

void init() {
  KeyboardSystem* keySys = new KeyboardSystem();
  GraphicsSyatem* gfxSys = new GraphicsSystem();
  DataSystem* dataSys = new DataSystem();

  GameData* gameData = dataSys.loadData();

void render()

void cleanup() {

now call init then call render on a requestAnimationFrame loop just like JavaScript. What was so hard?

Second is that even if native developers don't have to refactor that code there are tons of other places they have to refactor. There is no path from native to browser that does not require a bunch of work if you want users to have a good experence. As a simple example I tried porting some native code. The first thing I had to do was refactor to be event based. The app came up. But, then I needed to deal with the fact that the native app was hardcoded to, at compile time, decide what keys to use for Windows, Mac, Linux. That doens't work in the browser where depending on what machine the page is viewed the keys need to change at runtime. Ctrl-C vs Cmd-C for copy etc. For that particular app it would have been far more work to make it do the correct thing at runtime instead of compile time than it was to refactor to make it event based.

That wasn't the end of it though. Next up was the clipboard support. The native code was designed to expect it could read the clipboard on demand but that's not how the clipboard works in the browser. In the browser the user presses Paste (Ctrl-V or Cmd-V etc) and only then is the clipboard made available to the app via a clipboard event. In this way the page can't read the clipboard as data is being passed to other apps. It can only read it when the user has pasted into this app.

And those were just the start. The apps that use a spinloop are 99% games. Non-games are more often than not event based. Games have lots of issues needing far more data that most other native apps. No user wants to wait 5mintes to an hour for all that data to download so if the ported native apps hope to have any kind of audience they need to refactor to stream the data and ideally start up with a minimal amount of data while they continue to download the rest in the background.

They also need to be able to save state, read mods, and lots of other things which change drastically and all of which require lots of work to be a good user experience in a browser.

My point being that just adding commit will not be enough. There's a ton of work involved in bringing a native app to the browser and it not having a very bad user experience. By adding commit it just makes it slightly easier to barf bad content on the web. That shouldn't be encoraged. If devs are going to bring their native app to the browser they need to actually do the work to make it a good experience. Refactoring to be event based is the least of their problems.

I hope that at least gives some creedence to the idea that we shouldn't use the fact that many native games use a spinloop as an arguement to support spinloops. Let them refactor their apps.

The bigger issue is I don't believe there is actually a solution to the issues above about blocking commit in a spinloop. If you block I guarantee there will be race issues. The next most obvious solution is to provide some kind of API that lets developers stop rendering. They can use the focus and blur events to do their own throttling or commit can return some value saying effectively "the next time you call me I'm going to freeze so you'd better get ready". Another idea is the browser runs the spinloop a few more iterations but some other API lets the spinloop worker check if it's going to be frozen.

It really doesn't matter which of those solutions happen. The problem is they are solutions that require perfection. Developers are told do, A, then B, then C and it will work. Yet we know with 100% certainty that will not happen. Developers, especially web developers, never do things perfectly. An API that requires perfection to work correctly will basically never work correctly on the web. To put it another way, if developers have to deal with all the race conditions that come up from using sharedarraybuffers with a commit function that can block at anytime then likely the majority of pages will have race conditions that trigger randomly.

IMO that's not a solution we should chose. rAF just works. If you're not the front page rAF does not get called but other events still get processed. You can still communiate with the worker. Blocking commit doesn't work. The moment it's blocked ZERO commication with that worker can happen. You can't rescue it or nudge it out of it's blocked state. It's blocked. And, as pointed out above, thottling is not a solution.

So, in summary, I would argue commit should NOT be added to the set of web APIs period. Require devs to refactor to use events is the only reasonable solution IMO.


Why you should hang in there and learn git


A friend of mine was (is) struggling with learning git. I know what it's like. I was there. My progression was Source Safe -> CVS -> Subversion -> Perforce -> Mercurial -> Git. I found it frustrating at first and I didn't get it. Now that I do (mostly?) get it can't imagine switching back. So if you're frustrated learning git. You can't understand why it has to be so hard compared to what you're used to and you don't get the point. You feel like git adds nothing to what you're used to and it's just stupid then I hope this will help if only a little.

First off, an analogy. Imagine some one was working with a flat file system, no folders. They somehow have been able to get work done for years. You come along and say “You should switch to this new hierarchical file system. It has folders and allows you to organize better”. And, they’re like “WTF would I need folders for? I’ve been working just fine for years with a flat file system. I just want to get shit done. I don’t want to have to learn these crazy commands like cd and mkdir and rmdir. I don’t want to have to remember what folder I’m in and make sure I run commands in the correct folder. As it is things are simple. I type “rm filename” it gets deleted. Now I type “rm foldername” and I get an error. I then have to go read a manual on how to delete folders. I find out I can type “rmdir foldername” but I still get an error the folder is not empty. It’s effing making me insane. Why I can’t just do it like I’ve always done!”. And so it is with git.

One analogy with git is that a flat filesystem is 1 dimensional. A hierarchical file system is 2 dimensional. A filesystem with git is 3 dimensional. You switch in the 3rd dimension by changing branches with git checkout nameofbranch. If the branch does not exist yet (you want to create a new branch) then git checkout -b nameofnewbranch.

Git’s branches are effectively that 3rd dimension. They set your folder (and all folders below) to the state of the stuff committed to that branch.

What this enables is working on 5, 10, 20 things at once. Something I rarely did with cvs, svn, p4, or hg. Sure once in awhile I’d find some convoluted workflow to allow me to work on 2 things at once. Maybe they happened to be in totally unrelated parts of the code in which case it might not be too hard if I remembered to move the changed files for the other work before check in. Maybe I’d checkout the entire project in another folder so I'd have 2 or more copies of the project in separate folders on my hard drive. Or I’d backup all the files to another folder, checkout the latest, work on feature 2, check it back in, then copy my backedup folder back to my main work folder, and sync in the new changes or some other convoluted solution.

In git all that goes away. Because I have git style lightweight branches it becomes trivial to work on lots of different things and switch between them instantly. It’s that feature that I’d argue is the big difference. Look at most people’s local git repos and you’ll find they have 5, 10, 20 branches. One branch to work on bug ABC, another to work on bug DEF, another to update to docs, another to implement feature XYZ, another working on a longer term feature GHI, another to refactor the renderer, another to test out an experimental idea, etc. All of these branches are local to them only and have no effect on remote repos like github (unless they want them to).

If you’re used to not using git style lightweight branches and working on lots of things at once let me suggest it’s because all other VCSes suck in this area. You’ve been doing it so long that way you can’t even imagine it could be different. The same way in the hypothetical example above the guy with the flat filesystem can’t imagine why he’d ever need folders and is frustrated at having to remember what the current folder is, how to delete/rename a folder or how to move stuff between folders etc. All things he didn’t have to do with a flat system.

A big problem here is the word branch. Coming from cvs, svn, p4, and even hg the word "branch" means something heavy, something used to mark a release or a version. You probably rarely used them. I know I did not. That's not what branches are in git. Branches in git are a fundamental part of the git workflow. If you're not using branches often you're probably missing out on what makes git different.

In other words, I expect you won’t get the point of git style branches. You’ve been living happily without them not knowing what you’re missing, content that you pretty much only ever work on one thing at a time or find convoluted workarounds in those rare cases you really have to. git removes all of that by making branching the normal thing to do and just like the person that’s used to a hierarchical file system could never go back to a flat file system, the person that’s used to git style branches and working on multiple things with ease would never go back to a VCS that’s only designed to work on one thing at a time which is pretty much all other systems. But, until you really get how freeing it is to be able to make lots of branches and work on multiple things you’ll keep doing it the old way and not realize what you’re missing. Which is basically way all anyone can really say is “stick it out and when you get it you’ll get it”.

Note: I get that p4 has some features for working on multiple things. I also get that hg added some extensions to work more like git. For hg in particular though, while they added after the fact optional features to make it more like git go through pretty much any hg tutorial and it won't teach you that workflow. It's not the norm AFAICT where as in git it is the norm. That difference in base is what really set the two apart.

Let me also add that git is 4 dimensional. If branches are the 3rd dimension then versioning is the 4th. Every branch has a history of changes and you can go back to any version of any branch at anytime. Of course all VCSes have history and branches but again it's git's workflow that makes the differnce.

Let me also add that branches don't need to have anything in common. One branch might have your source. Another branch might have you docs. Whether that's common or not I don't no but it points out git doesn't care.

The most common example of this is probably github's github pages where github will, by default, serve as public a branch named "gh-pages". In several projects that branch has zero in common with the main branch. Instead some build script possibly running on a CI service builds the project's website and then checks it into the gh-pages branch. Whether using unrelated branches is a good practice or not AFAIK it's pretty much not done in other VCSes which I think hihglights a difference.


Software Development is never simple


Recently I wrote a little interval-timer.

I have a no-equipment interval workout, 30 rounds, 50 seconds each with a 10 second break between each one I've been doing for a while. I was using some online timer but it was buggy. It often displayed incorrectly and you had to resize your window once to get it work. It also didn't adjust to the window size well so if your window was the wrong aspect it wouldn't fit. Minor things but still annoying.

I checked out 5 or 6 others but they all required registration in order to try to sell you stuff or where covered in ads or didn't save your settings so you had to set them up every time or etc... etc...

I'd had it on my list of "This should be a simple few hour project, I should make my own" for at least a couple of years and finally recently I decided to do it.

Fighting CSS was, as always, no fun but eventually I got it to work, at least in modern current as of 2018/1 Firefox, Chrome, and Safari on desktop.

But! ... and I know this is normal but it's ridiculous how many small issues there have been.

First I thought "what the heck, it's a web page, let's make it work well on mobile (iOS) so I set the appropriate meta tags and futsed with the CSS a little and it comes up as expected. Except of course mobile has issues with how it computes full height 100vh and so my buttons at the bottom were off the screen. That is unless you saved the page to your home screen in which case iOS Safari goes fullscreen. Seemed good enough for me. SHIP IT!

So I post it to my facebook (friends only). First feedback I get is the controls weren't clear. Probably still aren't. One friend thought the circle arrow ↻ meant "go" instead of "rewind/reset" and I guess didn't recogonize the right pointing triangle ▶ as a "play" button. I made the circle arrow counter clockwise ↺ (not sure that helps) and added tooltips that say "reset", "start", and "stop" although that's only useful on desktop since you can't hover your finger on the phone.

Next friends complained it didn't run in iOS 10. I really didn't care when I wrote it, I'm on iOS 11, but then friends wanted to use it so I go look into it. Fortunately it was just adding prefixed CSS properties to fix it.

Then I was using URLs to store the settings like https://blabla?duration=50&rounds=30 etc.. but that meant if you added a bookmark and tried to change the settings you'd come back and your settings would be gone. Originally I thought putting them in the URL would let you have multiple timers but I doubt anyone would get that so I changed it to save the settings in local storage. You only get one timer. No, I'm not going to create a UI for multiple timers! I suspect most people just have one anyway and it's not too hard to change the 3 settings so it's probably good.

Then I realized I didn't handle hours for those few people that work out more than 1 hour so I added that. Speaking of which I also realize that entering seconds only is probably not a good UX. If you want 3 minute rounds you need to calculate in your head 3 * 60 = 180 seconds rather than put in 3 minutes 0 seconds but I'm seriously too lazy to deal with that and don't care. 😜

Ok, then I realized I was using requestAnimationFrame which doesn't run if the window is not visible. So for example you switch tabs to run music on youtube or something and the timer stops. Okay so I switched to using setTimeout and also made it set the title so you can still see the timer running even if it's not the current tab.

Then I noticed the tooltips I'd added above broke mobile. Buttons with tooltips required 2 clicks to work so I removed the tooltips.

Then I realized people were using it on the phone (I wasn't) and that the phone will go to sleep making it useless on the phone unless you manually prevent your phone from sleeping. I found a solution (which is friggen ridiculous) so now it doesn't let your phone sleep if the timer is running.

Then I realized the solution that prevents the phone from sleeping (which is to play a silent hidden video) stops your background music which is not very good for workouts. I found a solution which is to mute the video. I hope these 2 solutions continue to work.

Then I noticed that at least on iOS, if you add the page to your home screen, then, anytime you switch away from timer to another app and comeback it reloads the page meaning you lose your place. So today I made it save it's state constantly so if the page reloads it will continue. At the moment it continues as though it was still running. In other words if you're out of the timer for 3 minutes when you come back 3 minutes will have elapsed. I wasn't sure if that's what it should do or if I should pause the timer. As it is there so no way to rewind the timer a little which is probably the next thing I should consider adding.

So then I tried it on iOS but because of a bug it was actually pausing while switched away. That's when I noticed that when you come back, although the timer continues where it left off, because of limitations of mobile browsers the page is not allowed to make any sound unless the sound starts from a user interaction. Which means I'm basically forced to pause it and present a "continue where you left off" button. Or, just come back already paused and let the user press play.

And so this simple interval timer which I thought would take at most a few hours has now gobbled up a few days. Software dev is never simple. I wonder what the next issue will be 🤣


Why I Hate Software Dev


Ugh!!! This is why I hate computer dev! 🤣

So I decide I want to fix the CSS on for Windows. In particular I wanted to try to fix the scrollbars so they look like the MacOS version. I'm thinking it will only take a few mins.

So I decide to start in dev mode on Windows. (I normally do that dev on Mac).


Okay, F!!!, I don't want my machine effed up when different versions of meteor needed for one project vs another. Rant mode on, I wish all dev worked without installing anything globally or needing any kind of admin. As we learn of more and more exploits you should NEVER EVER BE ASKED FOR ADMIN. EVER!!!. I know it will be years or decades until this BS stops. Millions of machines will get powned by running un standboxed software that has exploits and installs via admin but until then I guess VMs it is 😡.

Eventually I got meteor running only to find out that tar on macOS and tar on Linux have different options and the ones I was using to re-write paths as I untar a backup won't work on Linux. I try restoring the DB manually but it doesn't work (no errors, but nothing showing up on the site).

I guess this is just par for the course. Doing new stuff is always a pain in the ass. It always takes time to setup something new and get it working and it's only after you've done that that you then go back to ignoring it because it's only important once every few months or years. Then next time you need to do it it's been so long that it's all changed and you have to spend the hours or even sometimes days getting your dev environmnet setup for your new project. Unfortunately new projects are getting more common or rather switching between many projects all with different needs for globally installed software is becoming far more common.

Still, it's super frustrating when you think something is going to only take a few mins and just getting to the point where you can actually do those few mins takes hours.

Finally I did what I probably should have done in the first place. I just run the mac version and go to http://<ipOfMac>:3000 on Windows, edit on Mac, check on Windows. I'm done in a few. Now if only all the browsers had a standard for scrollbar styling as it only works in Chrome.


Rethinking UI APIs


A few weeks ago I wrote a rant, "React and Redux are a joke right?". While many of my friends got the point most of the comments clearly responded mostly to the title without really getting the point. That's my fault for burying the point instead of making it clear, and for picking a clickbait title although that did get people to look.

I wrote a response back then but wordpress ate it. Since then I spent 2 weeks redoing my blog to be a static blog and now I'm finally getting around to writing what I hope will clear up my point.

Maybe my point can best be summed up as "Let's rethink the entire UI paradigm. Let's consider alternatives to the standard retained mode GUI. Those APIs are making us write way too much code".

If you're not familiar with what a "retained mode GUI" is it's any graphic user interface that requires a tree of objects that represent the UI. You build up the tree of objects. It sticks around "is retained" even if your code is not running. The UI system looks at the tree of objects to show the UI. This is the single most common paradigm for UIs. Examples include the DOM and all the DOM elements. On iOS all the UI objects like UIButton, UILabel, UITextField. Android, MacOS, Windows etc all work this way.

There maybe many reasons for doing it that way and I'm not arguing to stop. Rather I'm just pointing out there's another style, called "Immediate Mode GUI", and if you go try and use it for a complex UI you'll find you probably need to write 1/5th as much code to get things done as you do with a retained mode GUI.

The issue with a retained mode GUI is having to manage all of those objects. You need some way to associate your data with their representation in the UI. You also need to marshal data between your copy of the data and the UI's copy of the data. On top of that most retained mode GUIs are not that fast so you have to minimize churn. You need to try to manipulate, create, and destroy as few of these GUI objects as possible otherwise your app will be unresponsive.

Contrast that with an Immediate Mode GUI. In an Immediate Mode GUI there are no objects, there is no (or nearly no) UI state at all. The entire UI is recreated every time it's rendered. This is no different than many game engines at a low level. You have a screen of pixels. To make things appear to move over time, to animate them, you erase the entire screen and re-draw every single pixel on the entire screen. An Immediate Mode GUI is similar, there's nothing to delete because nothing was created. Instead the UI is drawn by you by calling the functions in the Immediate Mode GUI API. This draws the entire UI and with the state of the mouse and keyboard also handles the one single UI object that's actually being interacted with.

What happens when you do this, all the code related to creating, deleting, and managing GUI objects disappears because there are no GUI objects. All the code related to marshaling data into and out of the GUI system disappears because the GUI system doesn't retain any data. All the code related to try to touch as few GUI objects as possible disappears since there are no objects to touch.

And so, that's what struck me when I was trying to write some performant code in React and some site suggested Redux. React as a pattern is fine. It or something close to it would work well in an Immediate Mode GUI. But in practice part of React's purpose is to minimize creation, deletion, and manipulation of GUI objects (DOM Elements for ReactDOM, native elements for React Native). To do that it tracks lots of stuff about those objects. It also has integrated functions to try to compare your data to the GUI's data. It has its this.setState system that gets more and more convoluted with requiring you not to set state directly and not even to inspect state directly as a change might be pending. All of that disappears in an Immediate Mode GUI. Redux is one of many suggested ways to take all that tracking a step further, to make it work across branches of your GUI object hierarchy. All of that disappears in an Immediate mode GUI.

Now I'm not saying there aren't other reason you might want command patterns. Nor am I saying you don't want more structure to the ways you manipulate your data. But, those ways should be 100% unrelated to your UI system. Your UI system should not be influencing how you use and store your data.

When I wrote the title of my rant "React and Redux are a joke right?" my mindset was realizing that the entire construct of React and Redux are their to manage a retained mode GUI. If we were using an Immediate Mode GUI we wouldn't need them. The joke is on us thinking we're making our lives easier when in reality we're just making it more complex by piling on solution on top of solution on top of solution when the real problem is below all of that.

Now, I'm not suggesting we all switch to Immediate Mode GUIs. Rather, I'm suggesting that experience writing a complex UI with an immediate mode GUI might influence ways to make things better. Maybe there's some new design between retained mode GUIs and Immediate Mode GUIs that would be best. Maybe a different API or paradigm all together. I don't know if it would lead anywhere. But, I think there's room for research. As a more concrete example the entire Unity3D UI is basically an immediate mode GUI. All of their 1000s of plugins use it. Their API could probably use a refactoring but Unity3D is clearly a complex UI and it's running using an Immediate Mode style system so it's at least some evidence that such a system can work.

There are lots of objections to Immediate Mode GUIs. The biggest is probably it's assumed they are CPU hogs. I have not written any benchmarks and it would be hard as the 2 paradigms are so different it would be like those benchmarks where a C++ person translates their benchmark to Haskell not really getting Haskell end up making a very inefficient benchmark.

That said, I think the CPU objection might be overblown and might possibly be the complete opposite with Immediate Mode GUIs using less CPU than their retained mode counterparts. The reason I think this is true is that Immediate Mode GUIs almost always run at 60fps (or smooth) and retained mode GUIs almost never run smooth. To make a retained mode GUI run smooth for a complex UI requires tons and tons of code. Code like React's checks for changes, like the state tracking stuff, all the componentDidMount, componentWillUnmount, shouldComponentMount stuff, various kinds of caches etc. All that code, possibly 5x to 10x the code of an Immediate Mode GUI is code that is using the CPU. And when I say 5x to 10x the code I don't necessarily mean just written code. I mean executed code. A loop comparing if properties changed is code that's running even if the code implementing the loop is small. Some languages and or system generate tons of code for you to help with these issues. You didn't have to personally write the code but it is there bloating your app, using CPU. When you see a retained mode GUI like a webpage not be entirely responsive or take more than 16ms to update that's because the CPU was running 100% and could not keep up. That situation rarely happens with an Immediate Mode GUI. That suggests to me it's possible the Immediate Mode GUI is actually using less CPU.

Other than Unity3D which is probably the most used Immediate Mode GUI in use but is tied to their editor the most popular Immediate Mode GUI is Dear ImGUI. Yes, it's only C++. Yes, it's not designed for the Web (or phones). Again, for like the 1000th time, that is not the point. The point is not Dear ImGUI as a solution. The point is Dear ImGUI as inspiration. In realizing how much extra code we're being made to write and or execute to deal with retained mode APIs. The point is to take a step back and consider that just maybe this whole ubiquitous retained mode GUI API should be revisited.


CSS Grid - Fail?


I'm probably just missing the solution but I'm getting the impression CSS Grid really isn't the right solution and we'll be asking for another soon.

If you haven't used CSS Grid it's way of using CSS to declare that one element is a grid of NxM cells. Almost like a table except CSS Grid happens in CSS. The cool parts are that you can name groups of cells and then you can tell direct children of the grid what group of cells they cover just by name. Because it's CSS it's also easy to use media queries to change the layout based on the size of the user's browser.

That all sounds great, except AFAICT a grid doesn't actually solve this issue. Here's an example. Here's a video about CSS Grid

Note the screenshot for the video itself. In case it gets changed at the time I wrote this post it looked like this

Seeing that thumbnail you'd expect that layout is what you'd use CSS Grid for but if you watch the video they never make a grid like that.

When I look at the thumbnail I see 2 columns.

In the left column I see an upper half and a lower half.

The lower half is split into 2 columns itself

The left of those 2 columns is also split into an upper and lower part.

On the right I see a 3 row layout which each row being 1/3rd of the height of the entire thing.

So, can we really make that layout with CSS Grid and make it responsive? I suppose the answer is that we can but I doubt the way we have to do it is really what CSS grid people want. Remember CSS Grid is just that, a grid, so in order to make the layout above we'd need a grid like this

That hardly seems reasonable. The left column split into 2 50% height parts really shouldn't care about the right split into three 33% parts. The left top half really shouldn't have to care the the left bottom half needs to split into 2. And it all gets more messed up by the left most bottom area being split horizontally. Everything gets conflated this way. If you want to add another split somewhere you might have to redo a ton of stuff. That's not a reasonable way to go about this.

The more obvious way to do this is to nest grids just like we used to nest tables. In other words what we want is this

That's 5 grids. The outer one with 2 sides. The inner right one with three 33% rows. The left with two 50% rows and so on.

The problem is now we lose the ability to place things arbitrarily because grids only affect their direct children.

It seems like we need yet another new css type. Let's call it css layout. I have no idea how that would work. The problem comes back to separating content from style which means in order to do this we'd need some way to specify a hierarchy in CSS since that hierarchy shouldn't be expressed in HTML.

I have no idea what that expression would be, some large json looking like structure in CSS like this?

.mylayout {
  display: layout;
  layout-spec: "{
    rows: [
      { height: 100%;
        columns: [
          { width: 50%;
            name: left-column;
            rows: [
              { height: 50%;
                name: top-left;
              { height: 50%;
                name: bottom-left;
                columns: [
                  { width: 50%;
                    name: bottom-left-left;
                    rows: [
                       { height: auto;
                         name: bottom-left-left-top;
                       { height: auto;
                         name: bottom-left-left-bottom;
                  { width: 50%;
                    name: bottom-left-right;
          { width: 50%;
            name: right-column;
            rows: [
              { height: 33%; name: right-top; }
              { height: 33%; name: right-middle; }
              { height: 33%; name: right-bottom; }

It seems arguably what's needed though. Unless I'm missing something CSS Grid really doesn't solve all the layout issues what we've been trying to solve since nested table days.


Trying to help noobs is SOOOO FRUSTRATING!


I often wonder if I'd like to teach. It's certainly fun to teach when the students are easy to teach ? But where's the challenge in that?

I wrote (and and I answer tons of WebGL questions on stackoverflow but sometimes it's sooooooooooooooo frustrating.

I'm trying to take those frustrations as an opportunity to learn better how to teach, how to present things, how to be patient, etc but still...


Simplifying HappyFunTimes


I’m feeling rather stupid.

So a little over two years ago I started taking the code from PowPow and turning it into a library. That became HappyFunTimes which I’ve spent the better part of 2 years working on.