Rethinking UI APIs

2017-09-14

A few weeks ago I wrote a rant, "React and Redux are a joke right?". While many of my friends got the point most of the comments clearly responded mostly to the title without really getting the point. That's my fault for burying the point instead of making it clear, and for picking a clickbait title although that did get people to look.

I wrote a response back then but wordpress ate it. Since then I spent 2 weeks redoing my blog to be a static blog and now I'm finally getting around to writing what I hope will clear up my point.

Maybe my point can best be summed up as "Let's rethink the entire UI paradigm. Let's consider alternatives to the standard retained mode GUI. Those APIs are making us write way too much code".

If you're not familiar with what a "retained mode GUI" is it's any graphic user interface that requires a tree of objects that represent the UI. You build up the tree of objects. It sticks around "is retained" even if your code is not running. The UI system looks at the tree of objects to show the UI. This is the single most common paradigm for UIs. Examples include the DOM and all the DOM elements. On iOS all the UI objects like UIButton, UILabel, UITextField. Android, MacOS, Windows etc all work this way.

There maybe many reasons for doing it that way and I'm not arguing to stop. Rather I'm just pointing out there's another style, called "Immediate Mode GUI", and if you go try and use it for a complex UI you'll find you probably need to write 1/5th as much code to get things done as you do with a retained mode GUI.

The issue with a retained mode GUI is having to manage all of those objects. You need some way to associate your data with their representation in the UI. You also need to marshal data between your copy of the data and the UI's copy of the data. On top of that most retained mode GUIs are not that fast so you have to minimize churn. You need to try to manipulate, create, and destroy as few of these GUI objects as possible otherwise your app will be unresponsive.

Contrast that with an Immediate Mode GUI. In an Immediate Mode GUI there are no objects, there is no (or nearly no) UI state at all. The entire UI is recreated every time it's rendered. This is no different than many game engines at a low level. You have a screen of pixels. To make things appear to move over time, to animate them, you erase the entire screen and re-draw every single pixel on the entire screen. An Immediate Mode GUI is similar, there's nothing to delete because nothing was created. Instead the UI is drawn by you by calling the functions in the Immediate Mode GUI API. This draws the entire UI and with the state of the mouse and keyboard also handles the one single UI object that's actually being interacted with.

What happens when you do this, all the code related to creating, deleting, and managing GUI objects disappears because there are no GUI objects. All the code related to marshaling data into and out of the GUI system disappears because the GUI system doesn't retain any data. All the code related to try to touch as few GUI objects as possible disappears since there are no objects to touch.

And so, that's what struck me when I was trying to write some performant code in React and some site suggested Redux. React as a pattern is fine. It or something close to it would work well in an Immediate Mode GUI. But in practice part of React's purpose is to minimize creation, deletion, and manipulation of GUI objects (DOM Elements for ReactDOM, native elements for React Native). To do that it tracks lots of stuff about those objects. It also has integrated functions to try to compare your data to the GUI's data. It has its this.setState system that gets more and more convoluted with requiring you not to set state directly and not even to inspect state directly as a change might be pending. All of that disappears in an Immediate Mode GUI. Redux is one of many suggested ways to take all that tracking a step further, to make it work across branches of your GUI object hierarchy. All of that disappears in an Immediate mode GUI.

Now I'm not saying there aren't other reason you might want command patterns. Nor am I saying you don't want more structure to the ways you manipulate your data. But, those ways should be 100% unrelated to your UI system. Your UI system should not be influencing how you use and store your data.

When I wrote the title of my rant "React and Redux are a joke right?" my mindset was realizing that the entire construct of React and Redux are their to manage a retained mode GUI. If we were using an Immediate Mode GUI we wouldn't need them. The joke is on us thinking we're making our lives easier when in reality we're just making it more complex by piling on solution on top of solution on top of solution when the real problem is below all of that.

Now, I'm not suggesting we all switch to Immediate Mode GUIs. Rather, I'm suggesting that experience writing a complex UI with an immediate mode GUI might influence ways to make things better. Maybe there's some new design between retained mode GUIs and Immediate Mode GUIs that would be best. Maybe a different API or paradigm all together. I don't know if it would lead anywhere. But, I think there's room for research. As a more concrete example the entire Unity3D UI is basically an immediate mode GUI. All of their 1000s of plugins use it. Their API could probably use a refactoring but Unity3D is clearly a complex UI and it's running using an Immediate Mode style system so it's at least some evidence that such a system can work.

There are lots of objections to Immediate Mode GUIs. The biggest is probably it's assumed they are CPU hogs. I have not written any benchmarks and it would be hard as the 2 paradigms are so different it would be like those benchmarks where a C++ person translates their benchmark to Haskell not really getting Haskell end up making a very inefficient benchmark.

That said, I think the CPU objection might be overblown and might possibly be the complete opposite with Immediate Mode GUIs using less CPU than their retained mode counterparts. The reason I think this is true is that Immediate Mode GUIs almost always run at 60fps (or smooth) and retained mode GUIs almost never run smooth. To make a retained mode GUI run smooth for a complex UI requires tons and tons of code. Code like React's checks for changes, like the state tracking stuff, all the componentDidMount, componentWillUnmount, shouldComponentMount stuff, various kinds of caches etc. All that code, possibly 5x to 10x the code of an Immediate Mode GUI is code that is using the CPU. And when I say 5x to 10x the code I don't necessarily mean just written code. I mean executed code. A loop comparing if properties changed is code that's running even if the code implementing the loop is small. Some languages and or system generate tons of code for you to help with these issues. You didn't have to personally write the code but it is there bloating your app, using CPU. When you see a retained mode GUI like a webpage not be entirely responsive or take more than 16ms to update that's because the CPU was running 100% and could not keep up. That situation rarely happens with an Immediate Mode GUI. That suggests to me it's possible the Immediate Mode GUI is actually using less CPU.

Other than Unity3D which is probably the most used Immediate Mode GUI in use but is tied to their editor the most popular Immediate Mode GUI is Dear ImGUI. Yes, it's only C++. Yes, it's not designed for the Web (or phones). Again, for like the 1000th time, that is not the point. The point is not Dear ImGUI as a solution. The point is Dear ImGUI as inspiration. In realizing how much extra code we're being made to write and or execute to deal with retained mode APIs. The point is to take a step back and consider that just maybe this whole ubiquitous retained mode GUI API should be revisited.

Comments

React and Redux are a joke right?

2017-08-19

Note: follow up here

This is probably an ill thought through rant but no one reads my blog anyway. I'd tweet where at least a few people read but 140 characters is not enough.

I've been using react for some personal projects and it doesn't completely suck. I like its component nature.

That said, eventually you want to start displaying data and then you run into this issue which is state in react.

Each component in react can have props (data passed down to the component from a parent component) and state (data that is managed by the component itself). React keeps its own retained mode "Virtual DOM" (an internal tree representation of all the HTML elements it can/will generate). When you update state you have to do it in this special way because React wants to know what you changed. This way it can compare the old state to the new state, if nothing changed then it knows it doesn't have to re−render (generate or modify the actual browser elements). Sounds like a win right?

In the context of the browser that's probably a win but let's take a step back from the browser.

Imagine you have an ordered array of people

const orderedListOfPeople = [
  { name: "Andrea",  location: "Brussels", },
  { name: "Fiona",   location: "Antwerp", },
  { name: "Lorenzo", location: "Berlin", },
  { name: "Gregg",   location: "Tokyo", },
  { name: "Rumi",    location: "London", },
  { name: "Tami",    location: "Los Angeles", },
];

Your component either has that as state or it's passed down from the state of some parent as props.

Let's say you're receiving your list of people async (over the net) so you have code to insert a person

function insertPerson(person) {
   const ndxToInsertAt = getIndexToInsertAt(orderedListOfPeople);
   orderedListOfPeople.splice(ndxToInsertAt, 0, person);
}

In the react world though you can't *mutate* your data. You have to make copies

function insertPerson(component, person) {
   const state = component.state;
   const orderedListOfPeople = state.orderedListOfPeople;
   const ndxToInsertAt = getIndexToInsertAt(orderedListOfPeople);

   // must make a new array, can't mutate
   const newOrderedListOfPeople = [
      orderedListOfPeople.slice(0, ndxToInsertAt),  // people before
      person,
      orderedListOfPeople.slice(ndxToInsert),       // people after
   ];

   component.setState({
     orderedListOfPeople: newOrderedListOfPeople,
   });
}

Look at all that extra code just so react and check what changed and what didn't

Let say your request receives an array of people.

getMorePeopleAsync(people => people.forEach(insertPerson));

If you get 3 people react will end up re−rendering 3 times because of the code above. Each call to `setState` triggers a re−render. You have to ADD MORE CODE to work around that issue.

And the hole keeps getting deeper.

Let's say you just want to change the location of a user. Normal code

orderedListOfPeople[indexOfPerson].location = newLocation;

But again react needs to know what you changed, it wants a you to make a new array so

// make a copy of the people array
const newOrderedListOfPeople = state.orderedListOfPeople.slice();

// must copy the person, can't mutate
const newPerson = Object.assign({}, newOrderedListOfPeople[indexOfPerson]); 
newPerson.location = newLocation;
newOrderedListOfPeople[indexOfPerson] = newPerson; 

component.setState({
  orderedListOfPeople: newOrderedListOfPeople,
});

Ok, that's a pain so they came up with immutability-helpers so you can do this

const mutation = {};
mutation[indexOfPerson] = { location: {$set: newLocation} };
component.setState({
  orderedListOfPeople: update(state.orderedListOfPeople, mutation),
}); 

MORE CODE!!

Now imagine you have a tree structure. You end up having to write a mutation description generation function. Given a certain child node in your tree you need to be able to generate a description of how to mutate it. For example if you had this tree

{
  name: "root",
  children: [
    { ... },
    { 
       name: "middle",
       children: [
         { ... },
         {
           name: "leaf",
           children: [],
         },
       ],
    },
  ],
},

In normal code if I have a reference to "leaf" and I want to change its name it's just

leaf.name = newName

In react land using immutable−helpers, first I'd have to add parent references to all the nodes

// no way to reference the parents with static declaration so
function makeNode(name, parent) {
  const node = {
    name: name,
    parent: parent,
    children: [],
  };
  parent.children.push(node);
  return node;
}

const root = makeNode("root", null);
const middle = makeNode("middle", root);
const leaf = makeNode("leaf", middle);

Now, given a reference to leaf I'd have to do something like

function generateMutation(node, innerMutation) {
   if (node.parent) {
     const ndx = node.parent.indexOf(node);
     const mutation = {};
     mutation[ndx] = innerMutation;
     return generateMutation(node.parent, mutation);
   } 
   return innerMutation;
}

const mutation = generateMutation(leaf, {name: newName});
component.setState({
  orderedListOfPeople: update(state.orderedListOfPeople, mutation);
});

SO MUCH CODE!!! All just to set one field of a node in a tree.

Remember what I wrote above that that if you want to modify 3 things, 3 calls to setState will end up re−rendering 3 times. Well imagine the contortions you need to go through to merge the mutation above so it handles 3 arbitrary updates at once.

So then you go looking for other solutions. A popular one is called redux. Effectively instead of directly manipulating data you WRITE LOTS OF CODE to indirectly manipulate data. The set location example above you'd first write a function that made a copy of the person and set the new location. You'd call that an action. You can think of actions like the action in transACTION. You're basically building an transaction system for indirectly manipulating your data.

Let's go back to just setting the location. Redux would want something like this. First they want you to make function to generate *actions*.

function setLocationAction(indexOfPerson, newLocation) {
  return {
    type: SET_LOCATION,
    indexOfPerson: indexOfPerson,
    newLocation: newLocation, 
  };
}

All the function above does is make on object used to describe the thing you want to happen. The type field will be used later to execute different user supplied code based on the type. So we write that code

function setLocationReducer(people, action) {
  // copy the person since we can't mutate the existing person
  const newPerson = Object.assign({}, people[action.indexOfPerson]);  
  newPerson.location = action.newLocation;
  // copy the persons list as no mutation allowed
  const newPeople = [
     people.slice(0, action.indexOfPerson),  // everything before person
     newPerson,                              // modified person
     people.slice(action.indexOfPerson + 1); // everything after person
  ];
  return newPeople;
}

Now you register setLocationReducer with redux and then when you want to set the location you'd do something like

dispatch(setLocationAction(indexOfPerson, newLocation));

That generates an *action* object, then using the *type* field ends up calling setLocationReducer all to indirectly set the location of a person

NOTE: I'm new to redux so the example above might not be perfect but that's irrelevant to my point. Please keep reading.

So what about trees of data with redux? The docs tell you try not have nested data. Instead put things in flat arrays and use indices to reference things in other flat arrays.

Read around the net about how everyone seems to think redux is the bees knees (as in they love it).

But take a step back. All of this extra code is because of React. React wants to know which data changed so it doesn't re−render too much. People say React makes the UI simpler but it makes dealing with our data MUCH HARDER! Why is the UI library dictating how we store and manipulate our data!!!

Now, there may be good reasons. I get we're working in the browser, the browser uses the DOM. The DOM is complicated. Each node is very heavy, hundreds of properties. Each node has hundreds of CSS styles directly or indirectly. The browser handles all of this and multiple non−ASCII languages and fancy styles and CSS animation and all kinds of other stuff.

BUT, ... Go try using something like Dear ImGUI. You store your data however you want. You manipulate your data however you want. You then have your UI use that data in place. No extra code just manipulate your data just because the UI framework needs it. And it's running at 60fps with very complicated UIs. No jank! Want to see it running live here's the sample code running in the browser.

Now I get that ImGUI does much less than the browser. See 2 paragraphs up. But that's not the point. The point is to look at how much of a pain it is to use React (and probably the DOM in general). To notice all the hoops it's asking you to jump through. To notice that maybe your UI code got simpler than direct DOM manipulation but your data management code got 10x harder. It feels like we need to take a step back and re−evaluate how we got in this mess. There has got to be a better way!. We shouldn't need immutability helpers and/or redux and actions and reducers. Those are all solution for a problem we shouldn't have in the first place. We should just be able to have our data and manipulate it directly and we should be able to have a practically stateless UI system that can run at 60fps no jank.

Maybe someone needs to start the browser over. window.ImGUI. The no DOM use path. I'm sure that idea has issues. That's not the point again. The point is to see how much extra work we're being told to do, how many hoops we're being made to jump through, and then consider if there are better solutions. I don't know what those solutions are and I'm really not trying to dis React and Redux and the whole environment. Rather I just feel like setState, immutability helpers, redux, and all the other solutions are going down the wrong path solving a problem that shouldn't be there in the first place. If we solved the original issue (maybe that the DOM is slow and possibly the wrong solution) then the need to do all this extra work would disappear.

Comments

CSS Grid - Fail?

2017-08-07

I'm probably just missing the solution but I'm getting the impression CSS Grid really isn't the right solution and we'll be asking for another soon.

If you haven't used CSS Grid it's way of using CSS to declare that one element is a grid of NxM cells. Almost like a table except CSS Grid happens in CSS. The cool parts are that you can name groups of cells and then you can tell direct children of the grid what group of cells they cover just by name. Because it's CSS it's also easy to use media queries to change the layout based on the size of the user's browser.

That all sounds great, except AFAICT a grid doesn't actually solve this issue. Here's an example. Here's a video about CSS Grid

Note the screenshot for the video itself. In case it gets changed at the time I wrote this post it looked like this

Seeing that thumbnail you'd expect that layout is what you'd use CSS Grid for but if you watch the video they never make a grid like that.

When I look at the thumbnail I see 2 columns.

In the left column I see an upper half and a lower half.

The lower half is split into 2 columns itself

The left of those 2 columns is also split into an upper and lower part.

On the right I see a 3 row layout which each row being 1/3rd of the height of the entire thing.

So, can we really make that layout with CSS Grid and make it responsive? I suppose the answer is that we can but I doubt the way we have to do it is really what CSS grid people want. Remember CSS Grid is just that, a grid, so in order to make the layout above we'd need a grid like this

That hardly seems reasonable. The left column split into 2 50% height parts really shouldn't care about the right split into three 33% parts. The left top half really shouldn't have to care the the left bottom half needs to split into 2. And it all gets more messed up by the left most bottom area being split horizontally. Everything gets conflated this way. If you want to add another split somewhere you might have to redo a ton of stuff. That's not a reasonable way to go about this.

The more obvious way to do this is to nest grids just like we used to nest tables. In other words what we want is this

That's 5 grids. The outer one with 2 sides. The inner right one with three 33% rows. The left with two 50% rows and so on.

The problem is now we lose the ability to place things arbitrarily because grids only affect their direct children.

It seems like we need yet another new css type. Let's call it css layout. I have no idea how that would work. The problem comes back to separating content from style which means in order to do this we'd need some way to specify a hierarchy in CSS since that hierarchy shouldn't be expressed in HTML.

I have no idea what that expression would be, some large json looking like structure in CSS like this?

.mylayout {
  display: layout;
  layout-spec: "{
    rows: [
      { height: 100%;
        columns: [
          { width: 50%;
            name: left-column;
            rows: [
              { height: 50%;
                name: top-left;
              }
              { height: 50%;
                name: bottom-left;
                columns: [
                  { width: 50%;
                    name: bottom-left-left;
                    rows: [
                       { height: auto;
                         name: bottom-left-left-top;
                       }
                       { height: auto;
                         name: bottom-left-left-bottom;
                       }
                    ];
                  }
                  { width: 50%;
                    name: bottom-left-right;
                  }
                ];
              }
            ];
          }
          { width: 50%;
            name: right-column;
            rows: [
              { height: 33%; name: right-top; }
              { height: 33%; name: right-middle; }
              { height: 33%; name: right-bottom; }
            ];
          }
        ]
      }       
    ]
  }";
}

It seems arguably what's needed though. Unless I'm missing something CSS Grid really doesn't solve all the layout issues what we've been trying to solve since nested table days.

Comments

Visual Studio Code Wishlist

2017-08-01

I'm in the process of switching to or at least trying out Visual Studio Code.

I'm not familiar with every editor but I certainly know that an expandable editor is not a new thing. Emacs is as old as dirt and has been customizable since it was first created. vi has been customizable just as long I'm guessing. Brief, a DOS based editor from the late 80s was probably the first editor I customized a bunch with custom scripts.

For whatever reason though Visual Studio Code feels like it's done something new. I think the biggest thing it did (and I get that I might have gotten this from Atom) is not just to have an extension system but to actually integrate an extension browser/market directly in the product. Most other extensible editors require digging deeply into the manual to find out how to add an extension and then you're left on your own to find them. VSCode on the other hand asks you when you start or you can just click one of the main icons.

If that was all it brought that might be enough but it seems to have brought a few other innovations (expect to get corrected in the comments). One is that Microsoft designed an asynchronous language server protocol for heavy processes to run outside the editor. So for example if you want to have a C++ linter that runs LLVM to find places to warn you you don't have to put it in the editor and have it slow down your editing. Rather you run it as an external process, it can take as long as it needs and provide the result asynchronously.

Some editors did stuff like that but they didn't define a standard for it, instead it was just built in to their few custom linters. VSCode on the other hand, because it defined a standard opened that up to add support for any language. Write the linter, interface using the protocol, install, BOOM! Your editor now supports a new language. They even suggested other editors consider supporting the same protocol so they can take advantage of the same linters.

Maybe other editors support a similar feature but I was kind of blown away at some of the lint warnings I saw. For example I recently started playing with React. I followed the docs but of course things change over time. I followed some instructions that said to install the React linter into VSCode and suddently I was getting super React specific linter warnings directly in my editor. Is that a common thing anywhere else? I've certainly seen lint warnings for the language itself but I've never seen them for specific libraries.

It also helps, at least for JavaScript, that there's is almost a defacto standard for declaring your dependencies. This means the editor can easily look them up and know which linters to run. Of course you can also manually specify them but dang, that's pretty awesome.

Another feature I hadn't seen so much before is AST based linters that can expose correctors. I've seen a linter tell me "2 spaces required here" or "must use double quotes" but I hadn't seen "Fix all occurrences of this issue" before. Of course I've seen re−formatters before but they were generally all or nothing. You run your entire file through, it reformats everything, you get stuff out the other end. Some of you might like that kind of stuff but I'm not a fan as I find I generally want more flexibility than most coding standards provide.

This is not VSCode related but that brings up yet another thing which is the most common linter for JavaScript is eslint and not only is it massively configurable but it also takes plugins. That means unlike most linters I'm familiar with eslint you can relatively easily customize it for your personal style instead of just having to pick one of a few presets. Similarly the way these are specified are someone standardized so VSCode can adapt to difference lint settings per project.

Unfortunately VSCode is still missing many things I'm used to. Some might be easy to add as plugins. Others will require big changes I can only pray the team will make

Big needs include recordable keyboard macros. That's a feature I've used often for over 30 years and it's a little hard to believe that it wasn't part of the MVP for VSCode. It even exists in Visual Studio.aspx). It requires a certain level of architecture at a deep level. Hopefully that architecture is already in place.

Another personal big need is support for unlimited window splitting. As it is VSCode is hardcoded to 3 vertical splits period. No horizontal splits. No 2x2. Emacs since the 80s has had unlimited spitting. In this age of 30inch monitors a limit of 3 splits, and only vertical splits, seems very limiting.

Yet another is support for column editing. VSCode has support for multiple cursors and that's great but it's not a substitute for column editing.

column editing

multi-cursor editing

Similarly in my old editor there are at least 3 modes of selecting. (1) Normal character to character select. (2) line select (3) column select. You press Alt−M (or just hold shift) for the normal character to character select, Alt−L switches to line select, Alt−C to column select. Moving the cursor keys once in selection mode adjusts the selection. Pressing almost anything else stops selecting which basically means you need the editor to tell you some other key was pressed. Just assigning a function to get called on Alt−L, Alt−M, Alt−C is not enough.

Mapping the keys might be easy but handling virtual space not so much.

Those things probably need core changes to the editor's internals. The following can probably just be done with extensions some of which might already exist.

Less mouse dependence. I could be I'm just not used to it yet but in my previous editor there is an optional custom open dialog that doesn't need a mouse. The default is to use the OS's open dialog but when editing I find that a PITA. Another is a buffer selector. Some people might see this as the same as file selector but for some reason I find it useful to be able to switch among the short list of files I have open rather than the entire list of files in my project. Plugins for those probably would not be that hard.

Another is to be able to search and replace based on coloring or the AST. Example: search and replace "for" with "four" but only in strings. Without that feature every for loop would get replaced. Or search and replace "which" with "witch" but only in identifiers.

More auto formatting would be nice. It's possible it already exists and I just have to configure or turn it on. As a higher level example my old editor could auto wrap comments

And it's also aware when I'm in a comment block and knows to indent the next line with a comment. It's probably just a minor tweak to the current auto indent which looks at the previous line.

This one might require callback/event support but my previous editor keeps backups and I can visit and diff them

You probably thinking I should use git for that but that's not really the point. The point is this just happens in the background and saves me from when I haven't committed to git yet or when I'm working on a file not related to software dev.

One more is safe multi−file search and replace. Basically my current editor can search and replace across files and then undo the entire thing. Of course without that feature I could manually backup first, then do my search and replace but it's so much nicer just to be able to do it without thinking about it.

I'm not entirely sure I want to switch but I do find some of the new features compelling so I'm hoping I can get VSCode customized to surpass my previous editor.

Comments

Does Chrome need to change it's mic/camera permission UX?

2017-07-18

I'm a little worried about webcam and mic access in browsers at least as currently implemented. (and native apps too I suppose but less so)

As it is, in Chrome, if you let a webpage access your camera (or mic) that page's domain gets permanent permission to access the camera or mic whenever it wants forever, no questions asked again.

I recently visited one of the early HTML5 Webcam demos that I hadn't visited in years. I expected to get asked for permission to access the camera. Instead the camera just came on. That does not seem like a good model for web sites that are covered in ads and scripts from all over the net.

I'm sure the Chromium team was thinking of supporting hangouts when designing webcam support and I might be convinced that if hangouts always had access to the camera that might be no worse than a native app. But, it's the browser it's not a native app, it's untrusted code.

Even for communications sites though if I run them in an iframe they get camera permission. In other words, say yes just once and now that domain can follow you all over the net and use your camera and mic, at least as of Chrome 59. Did you ever use your mic on facebook.com to make a call? Well now any page that has facebook social embeds can now use your mic without asking.

I don't know what the best permission UX is. Always asking might might be tedious for actual communication websites (messenger, hangouts, slack?, ...) but not asking sucks for the open web. It even sucks on a communications website if the camera or mic is not something I use often. I don't want any app to have permission to spy on me. Imagine slack.com, imagine you use it to do a video call just once, then not again for 6 months even though you're text chatting constantly. During that entire time, at any time slack could have been accessing your mic or you camera. I personally want to opt−in to always ask. I think I'd prefer this even in native apps but especially for the web. The UX doesn't have to suck. Clicking "call" and having the browser say "this app wants to access your mic, Y/N" doesn't seem like a burden.

Here's a demo of the issue.

At a minimum it seems like iframes should not get automatic permission even if that domain had permission before. Otherwise some ad company will make a compelling camera demo just to get you to say yes once to using the camera on their domain. Once they do all their ads all over the net can start spying on you or using the mic to track you.

Even then though there are plenty of sites that allow users to post JavaScript, give access to one user's page and all users' pages get access. So nice person makes camera demo, bad person takes advantage that that domain now has access to your mic or camera.

I filed a bug on this about 5 months ago but no word yet. If you think this is an important issue consider starring the bug. Until then, if this concerns you go to chrome://settings/content/camera in Chrome and remove all the sites you've enabled the camera for. Do the same for the microphone by going to chrome://settings/content/microphone.

If you use a site that needs access to the camera or the mic, once you've given it permission to use the camera or the mic a small icon will appear in the URL bar. When you're done with the site you can click that icon to remove permission. That's tedious and you're likely to forget but it's better than nothing for now.

Comments

Dear Apple, Please make your app stores more internationally friendly

2017-07-05

I know we are in the minority but as a person living abroad (Japan) it's extremely frustrating to use the iOS (and Mac) App stores.

Because my native language is English and I'm from the USA I want to use the USA store. But, sometimes I have to use the Japanese store, for example to download a banking for local banks who's apps are only available on the Japanese store. Doing this is extremely tedious. The steps are:

1. Go to your home page, swipe to whatever page your settings are on and click the settings app.

2. Scroll down, find, and pick "iTunes and App Store"

3. Click your Apple ID (it's not clear at all this is clickable)

4. Pick "View Apple ID" (It's not clear at all this is where you'd change the store)

5. Enter you password. I have a long password with both numbers and symbols. It's super annoying to type on the phone. No idea why this is Japanese right now.

6. Pick "Country/Region".

7. Pick "Change Country or Region" (think of this as "yes, I really want to do this").

Imagine the frustration of students etc who don't have a local credit card with which to sign up. I'm guessing they beg their friends.

8. Select the region you want.

9. Enter your local address and postal code.

10. Enter your local phone number

11. Enter a local credit card, its expiration date and its code.

All of this takes several minutes.

Then, when I need to switch back I have to go through the same tedious process.

To give an example, Saturday I wanted to checkout Japanese podcasts. The easiest way to access the most popular ones seemed to be to use the builtin Apple iOS Podcast app. So I click it but of course since I'm on the USA store it shows the USA list. I sigh and decide I'll do this later. A couple of days later I finally get around to switching stores. I check the podcasts just to see it's showing me the Japanese lists. It is but don't download any yet. Today I need some network info app for iOS. I find one called "Network Utility" on iOS and download it. It's not showing the info I need. I check their website to see if there's an example of it showing the info I need. It's possible the info doesn't exist on my phone. I see the same app exists on the Mac App Store. I check it out, I can't tell but I know the info I want to display on iOS *is* available on my Mac. In other words if the Mac App displays the info I want then maybe the iOS one does too. There's no free version but it's only 99¢ so I click "buy" and am then told I can't use the account to buy it. In other words because I switched my region to Japan and this mac app is on the USA store I can't buy it without going through all the shit above to switch back to the USA region.

Now I get I'm in the minority, most people stay in whatever region they were born in so this issue probably affects few people (maybe tourists?). But at the same time why does it have to be this way? Why can't I just register addresses and payment info for both stores and switch easily without having to re−enter it all every time? Or better why I can't have it automatically use the correct info. Or even better than that why can't there just be one store? Let me select a region for the charts and recommendations but otherwise put it all on the same store! Come on Apple! You're the leaders in UI/UX. You can do this!

Comments

Don't disable web security!!!

2017-06-27

This basic question is all over stack overflow.

People ask how can they access files when developing HTML locally. They make a .HTML file, then open it in Chrome. They add a script that needs to access an image for canvas or WebGL or whatever and find they can't. So they ask on Stack Overflow and the most common answer is some form of "Start Chrome with the option −−disasble−web−security" (or one of 5 or 6 other similar flags)

I keep screaming DON'T DO THAT! but almost no one listens. In fact not only that the downvote my answers.

Well here's two proof of concepts of why it's ill−advised to disable web security.

The first one is an example that will get your stack overflow or github username if you are logged in and you started chrome with --disable-web-security. Of course you probably don't care that someone is looking up your username on various sites but that's not really the point. The point is some webpage not related to those sites was able to access data from those sites. The same webpage could access any other site. You bank, your google account, all because you disabled security.

You might say "I'd never run a script like that" but you likely run lots of 3rdparty scripts.

The second example will show files from your hard drive. It could upload them to a remote server. Which files some baddie would want I have no idea. The point is not to show uploading dangerous files. The point is only to show if you disable web security it's possible for a script, your own or a 3rd party one to access your local files.

Many of you will be thinking "I'd never do either of those" but I think that's being short sighted. I know I often forget which browser I'm in, the dev one or the non−dev one. If I mistakenly used the dev one with web security disabled then oops.

Of course you might also be thinking you'd never do any of the things above. You're running your own hand coded webpages with scripts and not using any 3rd party libraries and you never use the wrong browser. But again, that's not the point. The point is you turned off security. The point is not to enumerate all the ways you might get hacked or have data stolen or accounts manipulated. The point is if you disable web security you've made yourself more vulnerable period.

This is especially frustrating because the better solution is so simple. Just run a simple local server! It will take you all of 2 minutes at most. Here's one I wrote for those people not comfortable with the command line. Here's also 6 or 7 others.

Comments

Sony Playlink

2017-06-20

So Sony announced Playlink at E3 this year 2017.

It's a little frustrating. It's basically HappyFunTimes.

The part that's frustrating is in that video Shuhei Yoshida is happily playing a Playlink game and yet about a year ago I saw Shuhei Yoshida at a Bitsummit 2016 party and I showed him a video of happyfuntimes. He told me using phones as a controller was a stupid idea. Is objection was that phone controls are too mushy and laggy. He didn't have the foresight to see that not all games need precise controls to be fun. And yet here he is a year later playing Playlink which is the same thing as happyfuntimes.

In 2014 I also showed Konou Tsutomu happyfuntimes at a party and even suggested a "Playstation Party" channel with games for more than 4 players using the system. My hope was maybe he'd get excited about it and bring it up at Sony but he seemed uninterested.

Some people don't see the similarity but I'd like to point out that there are

And many others.

And of course there are also games you could not easily play on PS4 like this giant game where players control bunnies or this game using 6 screens.

And to Shuhei's objection that the controls are not precise enough

You just have to design the right games. Happyfuntimes would not be any good for Street Fighter but that doesn't mean there aren't still an infinite variety of games it would be good for.

In any case I'm not upset about it. I doubt that Shuhei had much to do with Playlink directly I doubt Konou even brought it up at Sony. I think the basic idea is actually a pretty obvious idea. Mostly it just reinforces something I already knew. That pitching and negotiation skills are incredibly important. If I had really wanted happyfuntimes to become PS4 Playlink I should have pitched it much harder and more officially than showing it casually at a party to gauge interest. It's only slightly annoying to have been shot down by the same guy that announced their own version of the same thing. ?

In any case, if you want to experiment with games that support lots of players happyfuntimes is still a free and open source project available for Unity and/or HTML5/Electron. I'd love to see and play your games!

Comments

Wishing for more Sandboxes

2017-06-19

I'm starting to wish that nearly all desktop apps ran in a very tight sandbox the same way they do on iOS.

Windows is trying to do this with the Windows store and Apple is trying to do it with the Mac App Store. The problem is two folder. One is they started with unsandboxed systems and so have decades of legacy software that expects to be unsandboxed. The other is they've conflated sandboxes and their app stores. Those 2 things should be separated.

Apps like Photoshop, Lightroom, Microsoft Word, gIMP, Blender, Maya, etc should not need system wide access.

To be clear I am **NOT** suggesting that there should be an app store or there should be an approval process for apps. Rather I'm suggesting that the OS should default to running each app in a sandbox with that app unable to get outside its sandbox without user permission. The permission system should be designed well (like I think it mostly is on iOS) so a native app should not be able to access your entire hard drive by default. It should not be able to read files from other apps by default. It should not be able to use your camera or mic or get GPS info by default. It should not be able to supply notifications by default or read your contacts. All of those things should be requested of the user at use time like iOS does (and I think Android is in the process of doing).

This might seem unrelated but it came up recently when a user on Stack Overflow asked how to make an Electron app from their HTML5 WebGL game. There are a few steps but over all it's pretty easy. If you're not familiar with Electron it's basically a version of Chrome that you can bundle as an app with your own HTML/CSS/JavaScript but unlike a normal webpage your JavaScript can access native features like files, OS level menus, OS level networking, etc.

And there in is the issue. The issue is it's common to use 3rd party scripts in your HTML5 apps. Maybe you're including JQuery or Three.js from a CDN. Maybe like many mobile apps you're downloading your HTML/CSS/JavaScript from your own servers like myautoupdatingapp.com. By doing that you've just made it possible for the people controlling the CDN or that hacks your server or the people that buy your domain to own every machine that's running your app. This is something that's not true with a browser doing the same thing because the browser does not allow JavaScript to access all those native things. It's only Electron that does this.

This means I have to trust every developer using Electron to not do either of those things.

On the other hand, this is exactly what iOS was designed to handle. You don't have to trust the app to the same level because the OS doesn't let the app read and write files to the entire machine. The OS doesn't let the app access the camera or the mic without first asking the user for permission.

This isn't the first time this kind of thing has happened. I'm sure there's plenty of other cases. One for me is XBMC/Kodi where there are plugins but no sandbox which means every plugin could be hacking your system. Many of those plugins are for websites that are arguably doing questionable things so why should I trust them not to do questionable things to my machine?

I'd even take it so far as I wish it it was easier to do this in the terminal/shell. If I'm trying out a new project there is often a build step or setup step or even the project itself. Those steps often allow code to run, code I don't want to have to trust. Of course in those cases I could run them in a VM and maybe I should start doing that more. I'm just wishing that that was easier than it is today. Like it kind if wish it was an OS level thing. I'd type something like

mkdir test & cd test & start VM

or

mkdir test & cd test & start standbox

Then I could

git clone someproject .
./configure
make

or

git clone somejsproj .
npm install

And not have to trust the 1000+ contributors above that they weren't doing something bad intentionally or unintentionally.

Unfortunately without a push by Apple and/or Microsoft it's unlikely the big software companies like Adobe are going to switch to their apps to the sandboxed systems.

IMO both companies need to separate their sandboxes (good) from their stores (bad). They then need to make it harder to run un−sandboxed apps. Not impossible, some apps probably need system level access if they provide system level services. But, they need to start making it the normal that the apps themselves are sandboxed.

Comments

NES/Famicom, A Visual Compendium - Corrections

2017-04-07

I very nice book came out with images from tons of NES games called "NES/Famicom, A Visual Compendium". I saw several of my friends had backed the kickstarter and when they go their copies they mentioned one of my games was in it so I had to buy a copy.

It's a gorgeous book. I'm not 100% sure I'm into the sharp emulator captures graphics as they look absolutely nothing at all like the original games. As art they're very cool but as representations of what those games looked like they are far off the mark. You can see a comparison on this article and see how when blown up in an emulator they look all blocky but back when the came out they looked smooth. Still as graphic art it's cool to see them in the book.

That said I looked up M.C. Kids and was a little disappointed to see things reported incorrectly. I not blaming anyone in particular. I assume it's an issue of like the game "telephone" where as the message got passed from person to person it got re−interpreted and ended up in it's present form

For M.C Kids it implies Rene did all the enemies but that was not the case. I'm not sure the percent of enemies created by Rene but IIRC Darren Bartlett and Ron Miller both did enemies as well.

I then notice there was an unreleased section and sure enough there was Robocop vs Terminator listed.

It's also wrong. It's says "Graeme Devine moved me from Caesars Palace Gameboy to Robocop Vs Terminator". What actually happened though is Graeme Devine moved me from Caesars Palace to Terminator NES, not Robocop vs Terminator. I worked on an animation tool to be shared between Terminator NES and M.C. Kids NES. Later I was asked to work on M.C Kids and Terminator was given to David Parry to make Terminator for Sega Genesis. When I finished M.C. Kids and I was no longer at Virgin Games I got a contract from Interplay to code Robocop Vs Terminator for NES.

Comments
older
newer