OffscreenCanvas and Commit

2018-07-04

Chrome is planning to ship OffscreenCanvas.

I know lots of devs that have been wanting that feature for ages so it's exciting to see it finally here. What is OffscreenCanvas? It's basically the ability to draw to a canvas from a web worker.

Drawing a complex scene often takes lots of CPU power. By being able to move all those calculations to a web worker we can make sure the main thread, the one reading the keyboard, responding to the mouse, etc... has all the power it needs to stay responsive.

There was debate for a long time about how it should be done. Ian Hickson wrote one idea orginally and with zero review stuck it in the spec. MDN even documented it though it was never implemented by anyone. I wrote another proposal in around 2012 that pointed out the issues with the one in the spec and suggested another solution. That was never implemented either though it was referenced from time to time as a reminder of some of the issues invovled.

In any case the current solution that chrome appears about to ship is that WebGL and Canvas2D mostly work in workers exactly the same as they do outside if workers. There's a small amount of code you need to write to transfer control of a canvas to some object that will exist inside the worker. The worker then creates a WebGL context or a 2D context and renders just like it would if it was in the main page. Results show up automatically just like they do on the main page. In other words, for those familar with graphics programming, there is no explict present or swapBuffers call. The moment you call one of the rendering functions in the respective APIs the browser "queues a task to do the present/swap" when your event exits.

This is great as it's the path of least surprise. No crazy new changes are needed to your code.

Even better they added requestAnimationFrame to workers so a worker can effectively just do a standard render loop

function render(time) {
  renderScene();
}
requestAnimationFrame(render);

So far so good.

But, ... they are also considering adding something else which is an explicit present/swapbuffers function called commit. Normal JavaScript apps would likely not use this API. Rather it's an API for WebAssembly ports of native games.

The issue they are trying to solve is that most native games run in what's called a "spinloop". They have code like this

   initialize();
   while(!userWantsToExit) {
      readUserInput();
      renderScene();
      glSwapBuffers();
   }

They never pause and never stop rendering they just run as fast as they can in a loop. By adding commit they feel they can better support native ports.

I see several problems with this approach and I hope I've convinced them to put the brakes and do a little more testing before releasing this API.

You can NOT use any other Web APIs with this model!

For those that don't know how JavaScript works it works on an event model. You provide functions to be called when certain events happen. Events include things like key pressed, mouse clicked, button clicked, slider moved, image downloaded, websocket message received, etc..

When one of these events arrives the browser calls the JavaScript function you assigned to that event. Your JavaScrpt runs AND THE BROWSER IS FROZEN until your JavaScript exits. Once your JavaScript exits the browser will run any other events on the list of events waiting to be run.

A spinloop like the one enabled by commit means your JavaScript never exits so you'll never process any other events. In other words, the worker rendering with a commit spinloop CAN NOT USE ANY OTHER WEB APIs. It can not receive messages from the main page. It can not download images. It can not read files or request data from a server. It can not use a websocket.

WOW! An api that removes the ability to use all other APIs!?!?!

When I asked about it I was told the solution is to use SharedArrayBuffers. SharedArrayBuffers are a way for workers and the main thread to share a chunk of memory with each other. They can all read and write from it and so they can use shared array buffers to commicate with each other.

Ok, I guess that works. It sounds like a ton of work. For example there is no way to get the raw data from an image in the current Web APIs. You can download images and use them in Canvas 2D and WebGL but as we've just pointed out you can't use those APIs in a worker using commit. Because you can't get the raw data you also can't download those imaegs in another worker or the main thread and pass them via sharedmemorybuffers into the render/commit worker. Soooo, you're left to write your own image decoders throwing away a bunch of the web API again. This is one reason why webassembly apps are so bloated is they include their own versions of image loading libraries even though libraries already exist in the browser.

I suppose that's a minor thing but that's not the end of it.

The next issue is what happens when your page that has worker that's using commit is not the front tab. This is a problem too.

With a normal requestAnimationFrame loop the only thing that happens is the browser stops sending animation frame events. It's still fully able to deliever other events. Events for fetching json, events for loading images, events loading other data. Your program can keep responding.

With commit it was suggested they'd just block commit forever until the tab is put in the front again. The problem then is that you've got the main page and or workers still receiving mesages but when they try to communicate those messages to the rendering worker that worker never responds. It's frozen. This will be a HUGE source of race bugs. Developers think their code works only to find it fails is subtle and hard to reproduce ways depending on when the user switches tabs. Not good.

Okay, so they suggested maybe they can throttle commit. They'll call it just once a second for example. Unfortunately we can show that's not a solution. Many GPU pages (and many even non-GPU pages) can be really slow. Here's a page that's really slow at least on my machine. When it's the front tab I can barely type. I'm glad that page exists as it's super educational so I think pages like that should exist. If I make some other tab the front tab my machine is back to normal and responsive. Now imagine if that page used commit and commit was only throttled. Imagine it was called once per second. The experience would be horrible and my machine would still seem unusable as once a second my machine would hiccup as it processes that graphics page offscreen. So no, throttling is not a viable solution. whatever solution happens must stop the rendering period.

So what solutions are their?

Well why do we need commit at all?

The reason some people think we need commit is because they want to support native ports to webassembly. A typical spinloop based C/C++ program might have some code like this

void main() {
  KeyboardSystem* keySys = new KeyboardSystem();
  GraphicsSyatem* gfxSys = new GraphicsSystem();
  DataSystem* dataSys = new DataSystem();

  GameData* gameData = dataSys.loadData();

  bool done = false;
  while(!done) {
    done = keySys.checkKeyboard();
    gfxSys.renderScene(gameData);
    glSwapBuffers();
  }

  gameData.cleanup();
  dataSys.cleanup();
  gfxSys.cleanup();
  keySys.cleanup();
}

The problem they see is that this code can't work in the browser's current system. Like I mentioned above the browser only calls your code via events. Your code needs to exit so that the browser can then process the next event. The code above never exits. If it does exit then keySys, gfxSys, dataSys and gameData would all be cleaned up which is not what we want.

Of course programmers can refactor their code so this isn't a problem but the people pushing for the commit are trying to to make it so those developers don't have to change their code and things will just work.

Here comes a place where we disagree. First, the amount of work to refactor that code is small. Of course the example above is small but I suspect even large native code bases would not take that much work to refactor to work with events. You'd need a few globals or singletons but otherwise you just split up your code

static KeyboardSystem* keySys;
static GraphicsSyatem* gfxSys;
static DataSystem* dataSys;

static GameData* gameData;

void init() {
  KeyboardSystem* keySys = new KeyboardSystem();
  GraphicsSyatem* gfxSys = new GraphicsSystem();
  DataSystem* dataSys = new DataSystem();

  GameData* gameData = dataSys.loadData();
}

void render()
   keySys.checkKeyboard();
   gfxSys.renderScene(gameData);
}

void cleanup() {
  gameData.cleanup();
  dataSys.cleanup();
  gfxSys.cleanup();
  keySys.cleanup();
}

now call init then call render on a requestAnimationFrame loop just like JavaScript. What was so hard?

Second is that even if native developers don't have to refactor that code there are tons of other places they have to refactor. There is no path from native to browser that does not require a bunch of work if you want users to have a good experence. As a simple example I tried porting some native code. The first thing I had to do was refactor to be event based. The app came up. But, then I needed to deal with the fact that the native app was hardcoded to, at compile time, decide what keys to use for Windows, Mac, Linux. That doens't work in the browser where depending on what machine the page is viewed the keys need to change at runtime. Ctrl-C vs Cmd-C for copy etc. For that particular app it would have been far more work to make it do the correct thing at runtime instead of compile time than it was to refactor to make it event based.

That wasn't the end of it though. Next up was the clipboard support. The native code was designed to expect it could read the clipboard on demand but that's not how the clipboard works in the browser. In the browser the user presses Paste (Ctrl-V or Cmd-V etc) and only then is the clipboard made available to the app via a clipboard event. In this way the page can't read the clipboard as data is being passed to other apps. It can only read it when the user has pasted into this app.

And those were just the start. The apps that use a spinloop are 99% games. Non-games are more often than not event based. Games have lots of issues needing far more data that most other native apps. No user wants to wait 5mintes to an hour for all that data to download so if the ported native apps hope to have any kind of audience they need to refactor to stream the data and ideally start up with a minimal amount of data while they continue to download the rest in the background.

They also need to be able to save state, read mods, and lots of other things which change drastically and all of which require lots of work to be a good user experience in a browser.

My point being that just adding commit will not be enough. There's a ton of work involved in bringing a native app to the browser and it not having a very bad user experience. By adding commit it just makes it slightly easier to barf bad content on the web. That shouldn't be encoraged. If devs are going to bring their native app to the browser they need to actually do the work to make it a good experience. Refactoring to be event based is the least of their problems.

I hope that at least gives some creedence to the idea that we shouldn't use the fact that many native games use a spinloop as an arguement to support spinloops. Let them refactor their apps.

The bigger issue is I don't believe there is actually a solution to the issues above about blocking commit in a spinloop. If you block I guarantee there will be race issues. The next most obvious solution is to provide some kind of API that lets developers stop rendering. They can use the focus and blur events to do their own throttling or commit can return some value saying effectively "the next time you call me I'm going to freeze so you'd better get ready". Another idea is the browser runs the spinloop a few more iterations but some other API lets the spinloop worker check if it's going to be frozen.

It really doesn't matter which of those solutions happen. The problem is they are solutions that require perfection. Developers are told do, A, then B, then C and it will work. Yet we know with 100% certainty that will not happen. Developers, especially web developers, never do things perfectly. An API that requires perfection to work correctly will basically never work correctly on the web. To put it another way, if developers have to deal with all the race conditions that come up from using sharedarraybuffers with a commit function that can block at anytime then likely the majority of pages will have race conditions that trigger randomly.

IMO that's not a solution we should chose. rAF just works. If you're not the front page rAF does not get called but other events still get processed. You can still communiate with the worker. Blocking commit doesn't work. The moment it's blocked ZERO commication with that worker can happen. You can't rescue it or nudge it out of it's blocked state. It's blocked. And, as pointed out above, thottling is not a solution.

So, in summary, I would argue commit should NOT be added to the set of web APIs period. Require devs to refactor to use events is the only reasonable solution IMO.

Comments

Game Credits Out Of Control

2018-05-15

Dear Games Credits, we need to have a talk. You are out of control and you seem to have lost the point of "credits".

As a perfect example, the PS4 God of War lists way too many people. They literally roll for 30 MINUTES!!

Imagine if the credits for Lord of the Rings listed not only Tolkien but also the person(s) that typeset the book. The person(s) that printed the book. The person(s) that managed the people that typeset the book. The person(s) that managed the people that printed the book. The person(s) who provided HR for the persons that printed and typeset the book. The president of the company that typeset and printed the book. The people that made the ink the book was printed with. The people that managed the people that make the ink the book was printed with. The assistants to the people that managed the people that made the ink the book was printed with. The people that delivered the ingredients to the ink factory to make the ink and on and on.

Sure, at some level you should be thankful for all of those people but that doens't mean they need to or should be listed in the credits.

Yes, who gets in the credits and who doesn't can be a contentious issue in the middle but it shouldn't be at the edges. The director of the game should get credit. The assistant to the salesperson in the foreign office should not. The caterer to the company should not get credit anymore than the cook at the burger place across the street should get credit no matter how much you love each of them. The art director should get credit. The team that makes the console's online store should not get credit. By that account every game on steam should be listing all members of Valve in their credits.

I don't know how to concretely define who should get credit. Off the top of my head credit should be given to those who materially contributed to the creation of the product. The keyword there is "creation". If you created parts of the content you could get credit. If you only indirectly supported people who created parts of the content you probably should not get credit with few exceptions.

Listing all the artists at an outsourcing company might make sense if all of those artists actually worked on the product. Listing the teams and managers that maintain the Playstation Network used by all games does not make sense anymore than listing the people that made the capaitors that sit on the electronics inside the machine or the guys that mixed the plastic for the power cables, or the captians that piloted the cargo ships that delivered the PS4s.

Serilousy, it's really a punch in the face to the actual God of War team that spent blood and tears working for 5-6 years on this game to then have their names diluted to the point of homeopathy with an extra 25 minutes of names. Just STOP IT. PLEASE!

Comments

Atarisoft Mario Bros for Commodore 64

2018-02-15

A friend recently found this video of someone playing the Mario Bros for Atarisoft that my friend John Alvarado and I created right out of high school when we were 19 years old.

I'm embarrassed to say at 19 I couldn't figure out how to make the platforms warp when you bonk your head into them given the sprite limits on the C64. Similarly I could not figure out how to make the platforms freeze for some reason. (Of course I could do it now ..... probably )

The problem was the C64, in that mode, is tilemap based so the best I could do is change the 4-6 tiles above your head which wouldn't match where you actually hit your head, at least if I only had one set of tiles. Maybe if I had a set for every other pixel or something but you only get 256 total but assumign we just needed every 4 pixels and 4 frames of animation that would 6 x 4 x 4 or 144 tiles. Some of the tiles are needed for numbers etc. Maybe I should have done that! No idea if I thought of that and was just too lazy or didn't think of it. Of course it would need a few more for the edges and then a different set for the bricks and another set for the ice so maybe it isn't possible. Less positions and less frames maybe. 10 tiles (edges) by 2 frames by 2 positions by 3 tile types is 120 tiles ... I wonder if it would have fit.

Looking at the Ocean version whichd did ship I see their solutions. They don't have any end pieces and the just push up whole tiles. Either 2 or 3.

What else I can add .... I remember being super late on it and getting flown out to Atarisoft to get asked when we'd be done. For some reason I seem to remember we went to some restaurant where I was introduced to fried bananas.

It was a cartridge so IIRC it had to fit in 8k. (Could be confused there). I know Centipede was 8k. Centipede was a port from Atari to C64. The Atari version only used around 6k of the 8k cart so we had room to add animation and a few other things.

Got paid $10k. Still feel guilty about that since it didn't ship. I don't think any of that was shared with John. Sorry John!

Comments below the video for our version seem be pretty positive. They say it plays better than the Ocean version. Of course it helps that both John and I were totally addicted to the arcade version. We'd play together all the time. Such a great co-op game!

Comments

Why you should hang in there and learn git

2018-01-24

A friend of mine was (is) struggling with learning git. I know what it's like. I was there. My progression was Source Safe -> CVS -> Subversion -> Perforce -> Mercurial -> Git. I found it frustrating at first and I didn't get it. Now that I do (mostly?) get it can't imagine switching back. So if you're frustrated learning git. You can't understand why it has to be so hard compared to what you're used to and you don't get the point. You feel like git adds nothing to what you're used to and it's just stupid then I hope this will help if only a little.

First off, an analogy. Imagine some one was working with a flat file system, no folders. They somehow have been able to get work done for years. You come along and say “You should switch to this new hierarchical file system. It has folders and allows you to organize better”. And, they’re like “WTF would I need folders for? I’ve been working just fine for years with a flat file system. I just want to get shit done. I don’t want to have to learn these crazy commands like cd and mkdir and rmdir. I don’t want to have to remember what folder I’m in and make sure I run commands in the correct folder. As it is things are simple. I type “rm filename” it gets deleted. Now I type “rm foldername” and I get an error. I then have to go read a manual on how to delete folders. I find out I can type “rmdir foldername” but I still get an error the folder is not empty. It’s effing making me insane. Why I can’t just do it like I’ve always done!”. And so it is with git.

One analogy with git is that a flat filesystem is 1 dimensional. A hierarchical file system is 2 dimensional. A filesystem with git is 3 dimensional. You switch in the 3rd dimension by changing branches with git checkout nameofbranch. If the branch does not exist yet (you want to create a new branch) then git checkout -b nameofnewbranch.

Git’s branches are effectively that 3rd dimension. They set your folder (and all folders below) to the state of the stuff committed to that branch.

What this enables is working on 5, 10, 20 things at once. Something I rarely did with cvs, svn, p4, or hg. Sure once in awhile I’d find some convoluted workflow to allow me to work on 2 things at once. Maybe they happened to be in totally unrelated parts of the code in which case it might not be too hard if I remembered to move the changed files for the other work before check in. Maybe I’d checkout the entire project in another folder so I'd have 2 or more copies of the project in separate folders on my hard drive. Or I’d backup all the files to another folder, checkout the latest, work on feature 2, check it back in, then copy my backedup folder back to my main work folder, and sync in the new changes or some other convoluted solution.

In git all that goes away. Because I have git style lightweight branches it becomes trivial to work on lots of different things and switch between them instantly. It’s that feature that I’d argue is the big difference. Look at most people’s local git repos and you’ll find they have 5, 10, 20 branches. One branch to work on bug ABC, another to work on bug DEF, another to update to docs, another to implement feature XYZ, another working on a longer term feature GHI, another to refactor the renderer, another to test out an experimental idea, etc. All of these branches are local to them only and have no effect on remote repos like github (unless they want them to).

If you’re used to not using git style lightweight branches and working on lots of things at once let me suggest it’s because all other VCSes suck in this area. You’ve been doing it so long that way you can’t even imagine it could be different. The same way in the hypothetical example above the guy with the flat filesystem can’t imagine why he’d ever need folders and is frustrated at having to remember what the current folder is, how to delete/rename a folder or how to move stuff between folders etc. All things he didn’t have to do with a flat system.

A big problem here is the word branch. Coming from cvs, svn, p4, and even hg the word "branch" means something heavy, something used to mark a release or a version. You probably rarely used them. I know I did not. That's not what branches are in git. Branches in git are a fundamental part of the git workflow. If you're not using branches often you're probably missing out on what makes git different.

In other words, I expect you won’t get the point of git style branches. You’ve been living happily without them not knowing what you’re missing, content that you pretty much only ever work on one thing at a time or find convoluted workarounds in those rare cases you really have to. git removes all of that by making branching the normal thing to do and just like the person that’s used to a hierarchical file system could never go back to a flat file system, the person that’s used to git style branches and working on multiple things with ease would never go back to a VCS that’s only designed to work on one thing at a time which is pretty much all other systems. But, until you really get how freeing it is to be able to make lots of branches and work on multiple things you’ll keep doing it the old way and not realize what you’re missing. Which is basically way all anyone can really say is “stick it out and when you get it you’ll get it”.


Note: I get that p4 has some features for working on multiple things. I also get that hg added some extensions to work more like git. For hg in particular though, while they added after the fact optional features to make it more like git go through pretty much any hg tutorial and it won't teach you that workflow. It's not the norm AFAICT where as in git it is the norm. That difference in base is what really set the two apart.

Let me also add that git is 4 dimensional. If branches are the 3rd dimension then versioning is the 4th. Every branch has a history of changes and you can go back to any version of any branch at anytime. Of course all VCSes have history and branches but again it's git's workflow that makes the differnce.

Let me also add that branches don't need to have anything in common. One branch might have your source. Another branch might have you docs. Whether that's common or not I don't no but it points out git doesn't care.

The most common example of this is probably github's github pages where github will, by default, serve as public a branch named "gh-pages". In several projects that branch has zero in common with the main branch. Instead some build script possibly running on a CI service builds the project's website and then checks it into the gh-pages branch. Whether using unrelated branches is a good practice or not AFAIK it's pretty much not done in other VCSes which I think hihglights a difference.

Comments

Software Development is never simple

2018-01-11

Recently I wrote a little interval-timer.

I have a no-equipment interval workout, 30 rounds, 50 seconds each with a 10 second break between each one I've been doing for a while. I was using some online timer but it was buggy. It often displayed incorrectly and you had to resize your window once to get it work. It also didn't adjust to the window size well so if your window was the wrong aspect it wouldn't fit. Minor things but still annoying.

I checked out 5 or 6 others but they all required registration in order to try to sell you stuff or where covered in ads or didn't save your settings so you had to set them up every time or etc... etc...

I'd had it on my list of "This should be a simple few hour project, I should make my own" for at least a couple of years and finally recently I decided to do it.

Fighting CSS was, as always, no fun but eventually I got it to work, at least in modern current as of 2018/1 Firefox, Chrome, and Safari on desktop.

But! ... and I know this is normal but it's ridiculous how many small issues there have been.

First I thought "what the heck, it's a web page, let's make it work well on mobile (iOS) so I set the appropriate meta tags and futsed with the CSS a little and it comes up as expected. Except of course mobile has issues with how it computes full height 100vh and so my buttons at the bottom were off the screen. That is unless you saved the page to your home screen in which case iOS Safari goes fullscreen. Seemed good enough for me. SHIP IT!

So I post it to my facebook (friends only). First feedback I get is the controls weren't clear. Probably still aren't. One friend thought the circle arrow ↻ meant "go" instead of "rewind/reset" and I guess didn't recogonize the right pointing triangle ▶ as a "play" button. I made the circle arrow counter clockwise ↺ (not sure that helps) and added tooltips that say "reset", "start", and "stop" although that's only useful on desktop since you can't hover your finger on the phone.

Next friends complained it didn't run in iOS 10. I really didn't care when I wrote it, I'm on iOS 11, but then friends wanted to use it so I go look into it. Fortunately it was just adding prefixed CSS properties to fix it.

Then I was using URLs to store the settings like https://blabla?duration=50&rounds=30 etc.. but that meant if you added a bookmark and tried to change the settings you'd come back and your settings would be gone. Originally I thought putting them in the URL would let you have multiple timers but I doubt anyone would get that so I changed it to save the settings in local storage. You only get one timer. No, I'm not going to create a UI for multiple timers! I suspect most people just have one anyway and it's not too hard to change the 3 settings so it's probably good.

Then I realized I didn't handle hours for those few people that work out more than 1 hour so I added that. Speaking of which I also realize that entering seconds only is probably not a good UX. If you want 3 minute rounds you need to calculate in your head 3 * 60 = 180 seconds rather than put in 3 minutes 0 seconds but I'm seriously too lazy to deal with that and don't care. 😜

Ok, then I realized I was using requestAnimationFrame which doesn't run if the window is not visible. So for example you switch tabs to run music on youtube or something and the timer stops. Okay so I switched to using setTimeout and also made it set the title so you can still see the timer running even if it's not the current tab.

Then I noticed the tooltips I'd added above broke mobile. Buttons with tooltips required 2 clicks to work so I removed the tooltips.

Then I realized people were using it on the phone (I wasn't) and that the phone will go to sleep making it useless on the phone unless you manually prevent your phone from sleeping. I found a solution (which is friggen ridiculous) so now it doesn't let your phone sleep if the timer is running.

Then I realized the solution that prevents the phone from sleeping (which is to play a silent hidden video) stops your background music which is not very good for workouts. I found a solution which is to mute the video. I hope these 2 solutions continue to work.

Then I noticed that at least on iOS, if you add the page to your home screen, then, anytime you switch away from timer to another app and comeback it reloads the page meaning you lose your place. So today I made it save it's state constantly so if the page reloads it will continue. At the moment it continues as though it was still running. In other words if you're out of the timer for 3 minutes when you come back 3 minutes will have elapsed. I wasn't sure if that's what it should do or if I should pause the timer. As it is there so no way to rewind the timer a little which is probably the next thing I should consider adding.

So then I tried it on iOS but because of a bug it was actually pausing while switched away. That's when I noticed that when you come back, although the timer continues where it left off, because of limitations of mobile browsers the page is not allowed to make any sound unless the sound starts from a user interaction. Which means I'm basically forced to pause it and present a "continue where you left off" button. Or, just come back already paused and let the user press play.

And so this simple interval timer which I thought would take at most a few hours has now gobbled up a few days. Software dev is never simple. I wonder what the next issue will be 🤣

Comments

Why I Hate Software Dev

2018-01-03

Ugh!!! This is why I hate computer dev! 🤣

So I decide I want to fix the CSS on vsa.com for Windows. In particular I wanted to try to fix the scrollbars so they look like the MacOS version. I'm thinking it will only take a few mins.

So I decide to start vsa.com in dev mode on Windows. (I normally do that dev on Mac).

Ok

Okay, F!!!, I don't want my machine effed up when different versions of meteor needed for one project vs another. Rant mode on, I wish all dev worked without installing anything globally or needing any kind of admin. As we learn of more and more exploits you should NEVER EVER BE ASKED FOR ADMIN. EVER!!!. I know it will be years or decades until this BS stops. Millions of machines will get powned by running un standboxed software that has exploits and installs via admin but until then I guess VMs it is 😡.

Eventually I got meteor running only to find out that tar on macOS and tar on Linux have different options and the ones I was using to re-write paths as I untar a backup won't work on Linux. I try restoring the DB manually but it doesn't work (no errors, but nothing showing up on the site).

I guess this is just par for the course. Doing new stuff is always a pain in the ass. It always takes time to setup something new and get it working and it's only after you've done that that you then go back to ignoring it because it's only important once every few months or years. Then next time you need to do it it's been so long that it's all changed and you have to spend the hours or even sometimes days getting your dev environmnet setup for your new project. Unfortunately new projects are getting more common or rather switching between many projects all with different needs for globally installed software is becoming far more common.

Still, it's super frustrating when you think something is going to only take a few mins and just getting to the point where you can actually do those few mins takes hours.

Finally I did what I probably should have done in the first place. I just run the mac version and go to http://<ipOfMac>:3000 on Windows, edit on Mac, check on Windows. I'm done in a few. Now if only all the browsers had a standard for scrollbar styling as it only works in Chrome.

Comments

Panopticlick Hyperbole

2017-12-27

Pantopiclick is an Electron Frontier Foundation website intended to show you how easy it is to track you on the internet. As part of this it creates a browser fingerprint. In other words it looks at all the various pieces of info a browser makes available that might be different with your computer vs other computers and turns them into a signature.

Examples include your User Agent string (how the browser shows which version of which browser you're using), a canvas finger print in which the fact that different computers/GPUs/browser might render the canvas slightly differently and those differences can be used to tell one from another. There's screen size. Your machine might be 1280x720 and someone else's might be 1920x1080 or whatever. Yet another is your time zone. The time zone itself is not available but your computer's local time is which means they can derive your time zone. There's what headers your browser sends. There's also a WebGL finger print which is similar to the canvas finger print. WebGL has many queriable features like how many textures it can use, what extensions are available, etc that are different for different drivers/GPUs/OSes. You also have a language preference which the site can use to give you a webpage in your desired language. Also what fonts are available can be checked if you have flash enabled and to some degree even if you don't. It can also check which plugins you have installed.

The site takes all of that info, makes a siganture, and then tells you how unique your browser is.

In my case it claims my Mid 2014 MacBook Pro is nearly 1 in a million. To put that in perspective it's claiming for basically all of Japan there are at most 14 other people that have the same machine using the same setup (Chrome, Language = English, Timezone = Japan, Screen = 1680x1050, certain fonts, etc.)

That may be true. I have no idea but it seemed a little fishy.

So I thought let's try my iPhone. I did this originally with my iPhone6S+ but it really doesn't matter. Checking with my iPhone X using Safari in Los Angeles with the language set to English it tells me that my browser is one of 493332.5 browsers.

Well let's analyze that number.

First it knows my time zone is Pacific Standard Time (PST). Other than that 100% of all the other info should match every iPhone X in the Pacific Standard Timezone. You can't install plugins on iPhones so the plugins list will be the same for every phone. The screen size is the same for every iPhoneX. The canvas and WebGL signatures will be the same for every iPhoneX. You can't install fonts either so the font list will be the same for every iPhoneX. Similarly the User Agent will be the same or at least the same for everyone on the same version of the OS.

Checking various info we can make a conservative estimate there are at least 53 million people in the PST as it includes cities like Los Angeles, San Diego, Seattle, San Fransico, Portland and others. According other stats at least 50% of those people own a smartphone. According to more stats iPhone has at least a 20% market share in the PST. People upgrade iPhones pretty quickly but let's assume conservatively that only 10% of those people that have an iphone have an iPhoneX. That means there are at least 53,000,000 * 50% * 20% * 10% or 530,000 iPhoneXs in the PST. That doesn't seem like an unreasoable number.

And yet based on Pantopticlick it's telling me I'm one out of 493332.5. In other words it's telling me that in all of the PST there are only at most 10 (ten) iPhoneXs. That's it. 10 phones! Clearly that's pure BULLSHIT!

In their defense they're claiming it's one of 493332.5 in their database. Their database is rotated every 3 months so if no other iPhoneXs have visited their site from the PST then yes the number might be accurate. But that's an irrelevant point. If a company wants to track people across sites it wants to track lots of people across lots of sites. Each site will have 1000s or millions of visitors. All 500k plus iPhoneX users in the PST are indistguishable by fingerprint across all of those sites so using this figureprint will not help them track those users in the least.

I'm all for the EFF warning people about the issues of tech. I'm a sponser of the EFF giving a not so small donation each year. But COME ON! Exaggerating these issues does nothing to help the EFFs case. Instead it makes the EFF seem unstrustworthy. If I can poke huge holes in this how do I not know that all the EFFs other press releases and issues are not similarly exaggerated.

EFF, please stop this hyperbole. You're only hurting your own cause.

Comments

Apple Privacy Snafu

2017-09-26

Apple recently posted a kind of privacy manifesto trying to distinguish themselves from other companies that might take privacy less serious.

So, imagine my surprise when I took my Macbook Pro in for service and was asked to give them my login username and password. NO YOU CAN NOT HAVE MY LOGIN NAME AND PASSWORD!

The genius at the Genius Bar was going to type it into some form so the repair staff can log into my computer. This to me seems akin to asking for my bank PIN number. You're told again and again NEVER TELL YOUR PIN NUMBER EVEN TO A BANK REPRESENTATIVE. Sure the bank itself may or may not be trustworthy but you have to trust every person in between not to read that password.

There's all kinds of stuff on my computer. My privacy keys for various servers. My bank, investment, and insurance info. Even my gmail is already logged in as in open the browser, go to gmail.com and my email will come up letting any user that has access start going through all my accounts and asking for password resets.

No company that claims to be taking privacy seriously should have their employees asking for usernames and passwords.

I told the Genius person, No, he could not have my account info. He then suggested I make a test account and give them that password. Again, NO EFFING WAY!.

First off, my entire hard disc is encrypted. It used to be that when you'd boot the machine it would ask for a password to unlock the drive. Only then would it really boot the computer and ask you to login to your account.

Apple changed that sometime in the last 1 or 2 OS upgrades to somthing much much less secure. Now, if I understand correctly, the key to unlock the drive is stored on the computer itself encrypted and all users of the computer have their own encrypted copy of the key. This means any user can unlock the drive. In other words, if you have a test account enabled your machine will allow the test user to unlock the drive. Once the drive is unlocked they can scan the entire machine sector by sector to read the data out. Before, they couldn't use the machine at all until the drive was unlocked.

Maybe I'm missing something but I don't see how giving Apple service my password is not insecure in the extreme.

To top that off it's not clear why they need the password in the first place. Macs easily boot from USB drives so they can plug in a USB drive if they want to see the computer work. If they want boot off an internal drive they can swap the internal drive with their own service drive. No reason for me to unlock my data for them.

In the end I told the guy I'd reformat the disk before bringing it in for service. It's hard to believe a company that claims to take privacy seriously didn't offer that as the first choice. In fact they didn't even suggest it. I had to suggest it.

Seriously, Apple. Having employees ask for a user's login credentials AND typing those credentials into some database seems incompatible with your privacy stance. Here's hoping you'll fix this.

Comments

Rethinking UI APIs

2017-09-14

A few weeks ago I wrote a rant, "React and Redux are a joke right?". While many of my friends got the point most of the comments clearly responded mostly to the title without really getting the point. That's my fault for burying the point instead of making it clear, and for picking a clickbait title although that did get people to look.

I wrote a response back then but wordpress ate it. Since then I spent 2 weeks redoing my blog to be a static blog and now I'm finally getting around to writing what I hope will clear up my point.

Maybe my point can best be summed up as "Let's rethink the entire UI paradigm. Let's consider alternatives to the standard retained mode GUI. Those APIs are making us write way too much code".

If you're not familiar with what a "retained mode GUI" is it's any graphic user interface that requires a tree of objects that represent the UI. You build up the tree of objects. It sticks around "is retained" even if your code is not running. The UI system looks at the tree of objects to show the UI. This is the single most common paradigm for UIs. Examples include the DOM and all the DOM elements. On iOS all the UI objects like UIButton, UILabel, UITextField. Android, MacOS, Windows etc all work this way.

There maybe many reasons for doing it that way and I'm not arguing to stop. Rather I'm just pointing out there's another style, called "Immediate Mode GUI", and if you go try and use it for a complex UI you'll find you probably need to write 1/5th as much code to get things done as you do with a retained mode GUI.

The issue with a retained mode GUI is having to manage all of those objects. You need some way to associate your data with their representation in the UI. You also need to marshal data between your copy of the data and the UI's copy of the data. On top of that most retained mode GUIs are not that fast so you have to minimize churn. You need to try to manipulate, create, and destroy as few of these GUI objects as possible otherwise your app will be unresponsive.

Contrast that with an Immediate Mode GUI. In an Immediate Mode GUI there are no objects, there is no (or nearly no) UI state at all. The entire UI is recreated every time it's rendered. This is no different than many game engines at a low level. You have a screen of pixels. To make things appear to move over time, to animate them, you erase the entire screen and re-draw every single pixel on the entire screen. An Immediate Mode GUI is similar, there's nothing to delete because nothing was created. Instead the UI is drawn by you by calling the functions in the Immediate Mode GUI API. This draws the entire UI and with the state of the mouse and keyboard also handles the one single UI object that's actually being interacted with.

What happens when you do this, all the code related to creating, deleting, and managing GUI objects disappears because there are no GUI objects. All the code related to marshaling data into and out of the GUI system disappears because the GUI system doesn't retain any data. All the code related to try to touch as few GUI objects as possible disappears since there are no objects to touch.

And so, that's what struck me when I was trying to write some performant code in React and some site suggested Redux. React as a pattern is fine. It or something close to it would work well in an Immediate Mode GUI. But in practice part of React's purpose is to minimize creation, deletion, and manipulation of GUI objects (DOM Elements for ReactDOM, native elements for React Native). To do that it tracks lots of stuff about those objects. It also has integrated functions to try to compare your data to the GUI's data. It has its this.setState system that gets more and more convoluted with requiring you not to set state directly and not even to inspect state directly as a change might be pending. All of that disappears in an Immediate Mode GUI. Redux is one of many suggested ways to take all that tracking a step further, to make it work across branches of your GUI object hierarchy. All of that disappears in an Immediate mode GUI.

Now I'm not saying there aren't other reason you might want command patterns. Nor am I saying you don't want more structure to the ways you manipulate your data. But, those ways should be 100% unrelated to your UI system. Your UI system should not be influencing how you use and store your data.

When I wrote the title of my rant "React and Redux are a joke right?" my mindset was realizing that the entire construct of React and Redux are their to manage a retained mode GUI. If we were using an Immediate Mode GUI we wouldn't need them. The joke is on us thinking we're making our lives easier when in reality we're just making it more complex by piling on solution on top of solution on top of solution when the real problem is below all of that.

Now, I'm not suggesting we all switch to Immediate Mode GUIs. Rather, I'm suggesting that experience writing a complex UI with an immediate mode GUI might influence ways to make things better. Maybe there's some new design between retained mode GUIs and Immediate Mode GUIs that would be best. Maybe a different API or paradigm all together. I don't know if it would lead anywhere. But, I think there's room for research. As a more concrete example the entire Unity3D UI is basically an immediate mode GUI. All of their 1000s of plugins use it. Their API could probably use a refactoring but Unity3D is clearly a complex UI and it's running using an Immediate Mode style system so it's at least some evidence that such a system can work.

There are lots of objections to Immediate Mode GUIs. The biggest is probably it's assumed they are CPU hogs. I have not written any benchmarks and it would be hard as the 2 paradigms are so different it would be like those benchmarks where a C++ person translates their benchmark to Haskell not really getting Haskell end up making a very inefficient benchmark.

That said, I think the CPU objection might be overblown and might possibly be the complete opposite with Immediate Mode GUIs using less CPU than their retained mode counterparts. The reason I think this is true is that Immediate Mode GUIs almost always run at 60fps (or smooth) and retained mode GUIs almost never run smooth. To make a retained mode GUI run smooth for a complex UI requires tons and tons of code. Code like React's checks for changes, like the state tracking stuff, all the componentDidMount, componentWillUnmount, shouldComponentMount stuff, various kinds of caches etc. All that code, possibly 5x to 10x the code of an Immediate Mode GUI is code that is using the CPU. And when I say 5x to 10x the code I don't necessarily mean just written code. I mean executed code. A loop comparing if properties changed is code that's running even if the code implementing the loop is small. Some languages and or system generate tons of code for you to help with these issues. You didn't have to personally write the code but it is there bloating your app, using CPU. When you see a retained mode GUI like a webpage not be entirely responsive or take more than 16ms to update that's because the CPU was running 100% and could not keep up. That situation rarely happens with an Immediate Mode GUI. That suggests to me it's possible the Immediate Mode GUI is actually using less CPU.

Other than Unity3D which is probably the most used Immediate Mode GUI in use but is tied to their editor the most popular Immediate Mode GUI is Dear ImGUI. Yes, it's only C++. Yes, it's not designed for the Web (or phones). Again, for like the 1000th time, that is not the point. The point is not Dear ImGUI as a solution. The point is Dear ImGUI as inspiration. In realizing how much extra code we're being made to write and or execute to deal with retained mode APIs. The point is to take a step back and consider that just maybe this whole ubiquitous retained mode GUI API should be revisited.

Comments

React and Redux are a joke right?

2017-08-19

Note: follow up here

This is probably an ill thought through rant but no one reads my blog anyway. I'd tweet where at least a few people read but 140 characters is not enough.

I've been using react for some personal projects and it doesn't completely suck. I like its component nature.

That said, eventually you want to start displaying data and then you run into this issue which is state in react.

Each component in react can have props (data passed down to the component from a parent component) and state (data that is managed by the component itself). React keeps its own retained mode "Virtual DOM" (an internal tree representation of all the HTML elements it can/will generate). When you update state you have to do it in this special way because React wants to know what you changed. This way it can compare the old state to the new state, if nothing changed then it knows it doesn't have to re−render (generate or modify the actual browser elements). Sounds like a win right?

In the context of the browser that's probably a win but let's take a step back from the browser.

Imagine you have an ordered array of people

const orderedListOfPeople = [
  { name: "Andrea",  location: "Brussels", },
  { name: "Fiona",   location: "Antwerp", },
  { name: "Lorenzo", location: "Berlin", },
  { name: "Gregg",   location: "Tokyo", },
  { name: "Rumi",    location: "London", },
  { name: "Tami",    location: "Los Angeles", },
];

Your component either has that as state or it's passed down from the state of some parent as props.

Let's say you're receiving your list of people async (over the net) so you have code to insert a person

function insertPerson(person) {
   const ndxToInsertAt = getIndexToInsertAt(orderedListOfPeople);
   orderedListOfPeople.splice(ndxToInsertAt, 0, person);
}

In the react world though you can't *mutate* your data. You have to make copies

function insertPerson(component, person) {
   const state = component.state;
   const orderedListOfPeople = state.orderedListOfPeople;
   const ndxToInsertAt = getIndexToInsertAt(orderedListOfPeople);

   // must make a new array, can't mutate
   const newOrderedListOfPeople = [
      orderedListOfPeople.slice(0, ndxToInsertAt),  // people before
      person,
      orderedListOfPeople.slice(ndxToInsert),       // people after
   ];

   component.setState({
     orderedListOfPeople: newOrderedListOfPeople,
   });
}

Look at all that extra code just so react and check what changed and what didn't

Let say your request receives an array of people.

getMorePeopleAsync(people => people.forEach(insertPerson));

If you get 3 people react will end up re−rendering 3 times because of the code above. Each call to `setState` triggers a re−render. You have to ADD MORE CODE to work around that issue.

And the hole keeps getting deeper.

Let's say you just want to change the location of a user. Normal code

orderedListOfPeople[indexOfPerson].location = newLocation;

But again react needs to know what you changed, it wants a you to make a new array so

// make a copy of the people array
const newOrderedListOfPeople = state.orderedListOfPeople.slice();

// must copy the person, can't mutate
const newPerson = Object.assign({}, newOrderedListOfPeople[indexOfPerson]); 
newPerson.location = newLocation;
newOrderedListOfPeople[indexOfPerson] = newPerson; 

component.setState({
  orderedListOfPeople: newOrderedListOfPeople,
});

Ok, that's a pain so they came up with immutability-helpers so you can do this

const mutation = {};
mutation[indexOfPerson] = { location: {$set: newLocation} };
component.setState({
  orderedListOfPeople: update(state.orderedListOfPeople, mutation),
}); 

MORE CODE!!

Now imagine you have a tree structure. You end up having to write a mutation description generation function. Given a certain child node in your tree you need to be able to generate a description of how to mutate it. For example if you had this tree

{
  name: "root",
  children: [
    { ... },
    { 
       name: "middle",
       children: [
         { ... },
         {
           name: "leaf",
           children: [],
         },
       ],
    },
  ],
},

In normal code if I have a reference to "leaf" and I want to change its name it's just

leaf.name = newName

In react land using immutable−helpers, first I'd have to add parent references to all the nodes

// no way to reference the parents with static declaration so
function makeNode(name, parent) {
  const node = {
    name: name,
    parent: parent,
    children: [],
  };
  parent.children.push(node);
  return node;
}

const root = makeNode("root", null);
const middle = makeNode("middle", root);
const leaf = makeNode("leaf", middle);

Now, given a reference to leaf I'd have to do something like

function generateMutation(node, innerMutation) {
   if (node.parent) {
     const ndx = node.parent.indexOf(node);
     const mutation = {};
     mutation[ndx] = innerMutation;
     return generateMutation(node.parent, mutation);
   } 
   return innerMutation;
}

const mutation = generateMutation(leaf, {name: newName});
component.setState({
  orderedListOfPeople: update(state.orderedListOfPeople, mutation);
});

SO MUCH CODE!!! All just to set one field of a node in a tree.

Remember what I wrote above that that if you want to modify 3 things, 3 calls to setState will end up re−rendering 3 times. Well imagine the contortions you need to go through to merge the mutation above so it handles 3 arbitrary updates at once.

So then you go looking for other solutions. A popular one is called redux. Effectively instead of directly manipulating data you WRITE LOTS OF CODE to indirectly manipulate data. The set location example above you'd first write a function that made a copy of the person and set the new location. You'd call that an action. You can think of actions like the action in transACTION. You're basically building an transaction system for indirectly manipulating your data.

Let's go back to just setting the location. Redux would want something like this. First they want you to make function to generate *actions*.

function setLocationAction(indexOfPerson, newLocation) {
  return {
    type: SET_LOCATION,
    indexOfPerson: indexOfPerson,
    newLocation: newLocation, 
  };
}

All the function above does is make on object used to describe the thing you want to happen. The type field will be used later to execute different user supplied code based on the type. So we write that code

function setLocationReducer(people, action) {
  // copy the person since we can't mutate the existing person
  const newPerson = Object.assign({}, people[action.indexOfPerson]);  
  newPerson.location = action.newLocation;
  // copy the persons list as no mutation allowed
  const newPeople = [
     people.slice(0, action.indexOfPerson),  // everything before person
     newPerson,                              // modified person
     people.slice(action.indexOfPerson + 1); // everything after person
  ];
  return newPeople;
}

Now you register setLocationReducer with redux and then when you want to set the location you'd do something like

dispatch(setLocationAction(indexOfPerson, newLocation));

That generates an *action* object, then using the *type* field ends up calling setLocationReducer all to indirectly set the location of a person

NOTE: I'm new to redux so the example above might not be perfect but that's irrelevant to my point. Please keep reading.

So what about trees of data with redux? The docs tell you try not have nested data. Instead put things in flat arrays and use indices to reference things in other flat arrays.

Read around the net about how everyone seems to think redux is the bees knees (as in they love it).

But take a step back. All of this extra code is because of React. React wants to know which data changed so it doesn't re−render too much. People say React makes the UI simpler but it makes dealing with our data MUCH HARDER! Why is the UI library dictating how we store and manipulate our data!!!

Now, there may be good reasons. I get we're working in the browser, the browser uses the DOM. The DOM is complicated. Each node is very heavy, hundreds of properties. Each node has hundreds of CSS styles directly or indirectly. The browser handles all of this and multiple non−ASCII languages and fancy styles and CSS animation and all kinds of other stuff.

BUT, ... Go try using something like Dear ImGUI. You store your data however you want. You manipulate your data however you want. You then have your UI use that data in place. No extra code just manipulate your data just because the UI framework needs it. And it's running at 60fps with very complicated UIs. No jank! Want to see it running live here's the sample code running in the browser.

Now I get that ImGUI does much less than the browser. See 2 paragraphs up. But that's not the point. The point is to look at how much of a pain it is to use React (and probably the DOM in general). To notice all the hoops it's asking you to jump through. To notice that maybe your UI code got simpler than direct DOM manipulation but your data management code got 10x harder. It feels like we need to take a step back and re−evaluate how we got in this mess. There has got to be a better way!. We shouldn't need immutability helpers and/or redux and actions and reducers. Those are all solution for a problem we shouldn't have in the first place. We should just be able to have our data and manipulate it directly and we should be able to have a practically stateless UI system that can run at 60fps no jank.

Maybe someone needs to start the browser over. window.ImGUI. The no DOM use path. I'm sure that idea has issues. That's not the point again. The point is to see how much extra work we're being told to do, how many hoops we're being made to jump through, and then consider if there are better solutions. I don't know what those solutions are and I'm really not trying to dis React and Redux and the whole environment. Rather I just feel like setState, immutability helpers, redux, and all the other solutions are going down the wrong path solving a problem that shouldn't be there in the first place. If we solved the original issue (maybe that the DOM is slow and possibly the wrong solution) then the need to do all this extra work would disappear.

Comments
older