What if good code colorization came before naming standards?

2022-06-04

Related to this post on time wasted because of naming standards, I just ran into this 2018 talk about tree-sitter. A fast language parser for code colorization written by Max Brunsfeld at github.

It seems pretty clear something like this had to be developed later than naming standards but its at least interesting to imagine what if it came first? Would we even need naming standards?

In the talk they point out that most editors use a complex set of regular expressions to guess how to color things.

Here's a pretty typical example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
class Foo {
 public:
  static Foo* create(int bar);
  int getBar();

 private:
  explicit Foo(int bar);
  int bar;
};

Foo::Foo(int bar) : bar(bar) {};

Foo* Foo::create(int bar) {
  return new Foo(bar);
}

int Foo::getBar() {
  return bar;
}

What to notice:

This is because most of the colorizers have no actual knowledge of the language. They just have a list of known language keywords and some regular expressions to guess at what is a type, a function, a string, a comment.

What if the colorizer actually understood the language?

For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
class Foo {
 public:
  static Foo* create(int bar);
  int getBar();

 private:
  explicit Foo(int bar);
  int bar;
};

Foo::Foo(int bar) : bar(bar) {};

Foo* Foo::create(int bar) {
  return new Foo(bar);
}

int Foo::getBar() {
  return bar;
}

What to notice: Every type is green, every function is yellow, every bar is red when it's a member of a class. This means we don't need to name it _bar or mBar or bar_ as many style guides would suggest because the editor knows what it is and shows us by color.

We could also distinguish between member functions and global functions

void Foo::someMethod() {
  doThis();  // is this a member function or a global function?
  doThat();  // is this a member function or a global function?
}

Some of these issues go away by language design. In Python and JavaScript a member function and a property both have to be accessed by self / this so yes, there are other solutions than just coloring and naming conventions to help make code more understandable at a glance.

I haven't used tree-sitter directly (apparently it's used on Github for colorization though). I just found the idea that a language parsing colorizer could help make code more readable and help distinguish between things that naming conventions are often used for. I get that color isn't everywhere so it's maybe not a solution but it's still fun to think about what other ways we could make it easier to grok the code.

PS: The coloring above is hand-written and not via tree-sitter.

Comments

How many man years are wasted with western naming convensions?

2022-06-02

In most (all?) western languages there's the concept of UPPER case and lower case. I'm only guessing this is one reason why we have different styles of naming convensions. Commonly we have things like

ALL_UPPER_CASE_SNAKE_CASE_IS_A_CONSTANT
CaptializedCamelCaseAreClassNames
lower_case_snake_case_is_a_variable_or_member
lowerCaseCamelCaseIsAVariableOrMember

We often bake these into coding standards. Google's, Mozilla's, Apple's, Microsoft's

Being a person that grew up with a native language that has the concept of UPPER/lower case some of this seems normal but several languages, off the top of my head (Japanese, Chinese, Korean, Arabic, Hindi) have no UPPER/lower case. If programming had been invented or made popular by people whose native language was one of those would we even have the concept of camelCase?

In any case, the fact that we have these styles often leads to extra work.

For Example, I'm implementing OpenGL which has constants like GL_MAX_TEXTURE_SIZE. In the code, our style guide uses mCamelCase for class members so we have something like this

struct Limits {
  unsigned mMaxTextureSize;
  ...

There's then code that is effectively

void glGetIntegerv(GLenum pname, GLint* data) {
  const Limits& limits = getCurrentContext()->getLimits();
  switch (pname) {
    case GL_MAX_TEXTURE_SIZE:
      *data = limits.mMaxTextureSize;
      return;
    ...

Notice all this busy work. Someone had to translate GL_MAX_TEXTURE_SIZE into mMaxTextureSize. We could have instead done this

struct Limits {
  unsigned GL_MAX_TEXTURE_SIZE;
  ...
};

void glGetIntegerv(GLenum pname, GLint* data) {
  const Limits& limits = getCurrentContext()->getLimits();
  switch (pname) {
    case GL_MAX_TEXTURE_SIZE:
      *data = limits.GL_MAX_TEXTURE_SIZE;
      return;
    ...

In this second case, seaching for GL_MAX_TEXTURE_SIZE will find all references to the concept/limit we're interested in. In the previous case things are separated and we either have to search for each individually or we have write some more complex query. Further, we need to be aware of the coding style. Maybe in a different code base it's like this

struct Limits {
  unsigned max_texture_size;
  ...
};

void glGetIntegerv(GLenum pname, GLint* data) {
  const Limits& limits = getCurrentContext()->getLimits();
  switch (pname) {
    case GL_MAX_TEXTURE_SIZE:
      *data = limits.max_texture_size;
      return;
    ...

In fact I've worked in projects that are made from multiple parts where different parts have their own coding standards and yet more work is require to translate between the different choices.

The point is, time is spent translating to <-> from one form GL_MAX_TEXTURE_SIZE to another maxTextureSize.

You can automatic this process. Maybe you auto generate struct Limits above but you still ended up having to write code to do the translation from one form to another, you still have the search problem, and you still need to know which to reference.

It just had me wondering, how many man years of work would be saved if we didn't have this translation step which arguably only exists because of man made style guides, arguably influenced by the fact that western languages have the concept of letter case? I suspect, over all of the millions of programmers in the world, it's 100s or 1000s of man years of work per year possibly wasted just because of the effort of converting these forms.

Comments

ImHUI - first thoughts

2021-02-27

I'm a fan (and a sponsor) of Dear ImGUI. I've written a couple of previous articles on it including this one and this one

Lately I thought, I wonder what it would be like to try to make an HTML library that followed a similar style of API.

NOTE: This is not Dear ImGUI running in JavaScript. For that see this repo. The difference is most ImGUI libraries render their own text and graphics. More specifically they generate arrays of vertex positions, texture coordinates, and vertex colors for the glyphs and other lines and rectangles for your UI. You draw each array of vertices using whatever method you feel like. (WebGL, OpenGL, Vulkan, DirectX, Unreal, Unity, etc...)

This experiment is instead actually using HTML elements like <div> <input type="text">, <input type="range">, <button> etc...

This has pluses and minus.

The minus is it's likely not as fast as Dear ImGUI (or other ImGUI) libraries, especially if you've got a complex UI that updates at 60fps?

On the other hand it might actually be faster for many use cases. See below

The pluses are

Thoughts so far

So, what have I noticed so far...

Simpler to get your data in/out

What's nice about the ImGUI style of stateless UI is that you don't have to setup event handlers nor really marshall data in and out of UI widgets.

Consider standard JavaScript. If you have an <input type="text"> you probably have code something like this

const elem = document.createElement('input');
elem.type = 'text';
elem.addEventListener('input', (e) => {
  someObject.someProperty = elem.value;
});

You'd also need someway to update the element if the value changes

// when someObject.someProperty changes
elem.value = someObject.someProperty;

You now need some system for tracking when you update someObject.someProperty.

React makes this sightly easier. It handles the updating. It doesn't handle the getting.

function MyTextInput() {
  return {
    <input
      value={someObject.someProperty}
      onChange={function(e) { someObject.someProperty = this.value; }>
  }
}

Of course that faux react code above won't work. You need to use state or some other solution so that react knows to re-render when you change someObject.someProperty.

function MyTextInput() {
  const [value, setValue] = useState(someObject.someProperty);
  return {
    <input
      value={value}
      onChange={function(e) { setValue(this.value); }>
  }
}

So now React will, um, react to the state changing but it won't react to someObject.someProperty changing, like say if you selected a different object. So you have to add more code. The code above also provides no way to get the data back into someObject.someProperty so you have to add more code.

In C++ ImGUI style you'd do one of these

  // pass the value in, get the new value out
  someObject.someProperty = ImGUI::textInput(someObject.someProperty);

or

  // pass by reference. updates automatically
  ImGUI::textInput(someObject.someProperty);  

JavaScript doesn't support passing by reference so we can't do the 2nd style, OR, we could pass in some getter/setter pair to let the code change values.

  // pass the value in, get the new value out (still works in JS)
  someObject.someProperty = textInput(someObject.someProperty);

  // use a getter/setter generator
  textInput(gs(someObject, 'someProperty'));

Where gs is defined something like

function gs(obj, propertyName) {
  return {
    get() { return obj[propertyName]; },
    set(v) { obj[propertyName] = v; },
  };
}

In any case, it's decidedly simpler than either vanilla JS or React. There is no other code to get the new value from the UI back to your data storage. It just happens.

Less Flexible?

I'm not sure how to describe this. Basically I notice all the HTML/CSS features I'm throwing away using ImGUI because I know what HTML elements I'm creating underneath.

Consider the text function. It just takes a string and adds a row to the current UI

ImGUI::text('this text appears in a row by itself');

There's no way to set a className. There's no way to choose span instead of div or sub or sup or h1 etc..

Looking through the ImGUI code I see lots of stateful functions to help this along so (making up an example) one solution is some function which sets which type will be created

ImGUI::TextElement('div')
ImGUI::text('this text is put in a div');
ImGUI::text('this text is put in a div too');
ImGUI::TextElement('span')
ImGUI::text('this text is put in a span');

The same is true for which class name to use or adding on styles etc. I see those littered throughout the ImGUI examples. As another example

ImGUI::text('this text is on its own line');
ImGUI::text('this text can is not'); ImGui::SameLine();

Is that a plus? A Minus? Should I add more optional parameters to functions

ImHUI.text(msg: string, className?: string, type?: string)

or

ImGUI.text(msg: string, attr: Record<string, any>)

where you could do something like

ImGUI.text("hello world", {className: 'glow', style: {color: 'red'}});

I'm not yet sure what's the best direction here.

Higher level = Easier to Use

One thing I've noticed is that, at least with Dear ImGUI, more things are decided for you. Or maybe that's another way of saying Dear ImGUI is a higher level library than React or Vanilla JS/HTML.

As a simple example

ImGUI::sliderFloat("Degrees", myFloatVariable, -360, +360);

Effectively represents 4 separate concepts

  1. A label ("degrees") probably made with a <div>
  2. A slider. In HTML made with <input type="range">
  3. A number display. In this case probably a separate <div>
  4. A container for all 3 of those pieces.

So, is ImGUI actually simpler than HTML or is it just the fact that it has higher level components?

In other words, to do that with raw HTML requires creating 4 elements, childing the first 3 into one of them, responding to input events, updating the number display when an input event arrives. Updating both the number display and the <input> element's value if the value changes externally to the UI widgets.

But, if I had existing higher level UI components that already handled is that enough to make things easier? Meaning how much of Dear ImGUI's ease of use comes from its paradigm and how much from a large library of higher level widgets?

This is kind of like comparing programming languages. For given language, how much of the perceived benefit comes from the language itself and how much from the standard libraries or common environment it runs in.

Notes in implementation

getter setters vs direct assignment

ImGUI uses C++ ability to pass by reference. JavaScript has no ability to pass by reference. In other words in C++ I can do this

void multBy2(int& v) {
  v *= 2;
}

int foo = 123;
multBy2(foo);
cout << foo;    // prints 246

There is no way to do this in JavaScript.

Following the Dear ImGUI API I first tried to work around this by requiring you pass in an getter-setter like this

var foo = 123;
var fooGetterSetter =  {
  get() { return foo; }
  set(v) { foo = v; }
};

which you could then use like this

// slider that goes from 0 to 200
ImHUI.sliderFloat("Some Value", fooGetterSetter, 0, 200);

Of course if the point of using one of these libraries is ease of use then it sucks to have to make getter-setters.

I thought maybe I could make getter setter generators like the one gs shown above. It means for the easiest usage you're required to use objects so instead of bare foo you'd do something like

const data = {
  foo: 123,
};

...

// slider that goes from 0 to 200
ImHUI.sliderFloat("Some Value", gs(data, 'foo'), 0, 200);

That has 2 problems though. One is that it can't be type checked because you have to pass in a string to gs(object: Object, propertyName: string).

The other is it's effectively generating a new getter-setter on every invocation. To put it another way, while the easy to type code looks like the line just above, the performant code would require creating a getter-setter at init time like this

const data = {
  foo: 123,
};
const fooGetterSetter = gs(data, 'foo');

...
// slider that goes from 0 to 200
ImHUI.sliderFloat("Some Value", fooGetterSetter, 0, 200);

I could probably make some function that generates getters/setters for all properties but that also sounds yuck as it removes you from your data.

const data = {
  foo: 123,
};
const dataGetterSetters = generateGetterSetters(data)

...

// slider that goes from 0 to 200
ImHUI.sliderFloat("Some Value", dataGetterSetter.foo, 0, 200);

Another solution would be to require using an object and then make all the ImHUI functions take an object and a property name as in

// slider that goes from 0 to 200
ImHUI.sliderFloat("Some Value", data, 'foo', 0, 200);

That has the same issue though that because you're passing in a property name by string it's error prone and types can't be checked.

So, at least at the moment, I've ended up changing it so you pass in the value and it passes back a new one

// slider that goes from 0 to 200
foo = ImHUI.sliderFloat("Some Value", foo, 0, 200);

// or

// slider that goes from 0 to 200
data.foo = ImHUI.sliderFloat("Some Value", data.foo, 0, 200);

It's far more performant than using getter-setters, on top of being more performant than generating getter-setters. Further it's type safe. Eslint or TypeScript can both warn you about non-existing properties and possibly type mis-matches.

Figuring out the smallest building blocks

The 3rd widget I created was the sliderFloat which as I pointed out above consists of 4 elements, a div for the label, a div for the displayed value, an input[type=range] for the slider, and a container to arrange them. When I first implemented it I made a class that manages all 4 elements. But later I realized each of those 4 elements is useful on its own so the current implementation is just nested ImHUI calls. A sliderFloat is

function slideFloat(label: string, value: number, min: number = 0, max: number = 1) {
  beginWrapper('slider-float');
    value = sliderFloatNode(value, min, max);
    text(value.toFixed(2));
    text(prompt);
  endWrapper();
  return value;
}

The question for me is, what are the smallest building blocks?

For example a draggable window is currently hand coded as a combination of parts. There's the outer div, it's scalable. There's the title bar for the window, it has the text for the title and it's draggable to move the window around. Can I separate those so a window is built from these lower-level parts? That's something to explore.

Diagrams, Images, Graphs

You can see in the current live example I put in a version of ImGUI::plotLines which takes a list of values and plots them as a 2D line. The current implementation creates a 2D canvas using a canvasNode which returns a Canvas2DRenderingContext. In other words, if you want to draw something live you can build your own widget like this

function circleGraph(zeroToOne: number) {
  const ctx = canvasNode();
  const {width, height} = ctx.canvas;
  const radius = Math.min(width, height);
  ctx.beginPath();
  ctx.arc(width /2, height / 2, radius, 0, Math.PI * 2 * zeroToOne);
  ctx.fill();
}

The canvas will be auto-sized to fit its container so you just draw stuff on it.

The thing is, the canvas 2D api is not that fast. At what point should I try to use WebGL or let you use WebGL. If I use WebGL there's the context limit issues. Just something to think about. Given the way ImGUIs work if you have 1000 lines to draw then every time the UI updates you have to draw all 1000 lines. In C++ ImGUI that's just inserting some data into the vertex buffers being generated, but in JavaScript, with Canvas 2D, it's doing a lot more work to call into the Canvas2D API.

It's something to explore.

So far it's just an Experiment

I have no idea where this is going. I don't have any projects that need a GUI like this at the moment but maybe if I can get it into something I think is kind of stable I'd consider using it over something like dat.gui which is probably far and way the most common UI library for WebGL visualizations.

Comments

Done Answering Questions Stack Overflow

2021-02-16

Over the last ~9 years I've spent way too much time answering questions on stack overflow. I don't know why. I want to say it's because I like helping people. It's certainly not for "internet points". In fact I despise the gamification on Stack Overflow so much I tried to hide it from myself. Here is what my view of Stack Overflow looks like

Notice all the points are missing. I feel an unhealthy influence of points on all sites that have them so I turn them off. I'm convinced someday some scientific research will show they are detrimental to well being and will push to ban them or at least shame them out of existence.

In any case, yea, I spent way to much time answering questions on Stack Overflow. At the time I wrote this I had answered 27% of all the WebGL tagged questions on the site. Including other topics in total over 1900 answers. I also edited the tags of hundreds of wrongly tagged questions.

Many of my answers took hours to write. It could be figuring out a working solution or it could be debugging someone's code. I generally tried to post working code in as many answers as appropriate since in my opinion, working code is almost always better than just an explanation.

As a recent example, someone was trying to glue together two libraries and was running into issues. I got their minimal repo runnable, tracked down the issue, posted a working solution, and filed a bug report on one of the libraries. The entire process took about 2.5 hours.

I've also pointed out before that I wrote webglfundamentals.org and webgl2fundamentals.org in response to questions on stack overflow. WebGL is a verbose API. People ask questions and there is no simple answer. You could just give them some code that happens to work but they likely need 16 chapters of tutorials to understand that code. That's way too much for stack overflow.

So, 9 years ago I started writing articles to explain WebGL. I tried to go out of my way not have them be self promoting. The don't say "WebGL articles by GREGG TAVARES" In fact, except for the copyright license hidden in the code comments, IIRC my name is no where on the website. I'd even be happy to remove my name from the license though I'm not quite sure what legal implications there are. Can I just make something up like "copyright webglfundamentals.org"? I have no idea.

I even moved them from my github account to an organization. The hope was I could find more people to contribute if there was an org so you can participate in the org and not in my personal site. The sites are under "gfxfundametnals" not "greggman". Unfortunately no one has stepped up to write anything, though several volunteers have translated the articles into Chinese, Japanese, Russian, Korean, and other languages.

In any case, once I'd written the articles I would point people to them on Stack Overflow when it seemed appropriate. If, based on the issues they are having, someone is clearly new to WebGL, I might leave an answer that answers their specific question and then also leave a link to the effect of

You might find these articles useful.

If someone else had already written a good answer I might just add the same as a comment under the question.

Similarly if one of the articles addressed their particular issue I might link directly to it. Of course if I was answering I'd always leave a full answer, not just a link. I've been doing this for the least 9 years. It's clearly and unambiguously helpful to the user that asked the question as well as users reading later.

An example of this came up recently. Someone asked a question about how to use mat4 attributes. Someone else left an okay answer that answered the question, though it didn't give a good example. But, given the answer was good enough, I added a comment. "You might find this article useful..." because the article has a better example.

There were 2 other parts to the comment.

  1. The answer stated something incorrect. They claimed drawing different shapes with instancing is impossible. My comment pointed out it was not impossible and specified how to do it.

  2. That brought up another point which is if you want to draw multiple different models in a single draw call, I'd written an example to do that in a stack overflow answer and so I linked to it.

The next day I went to check if there was a new comment, in particular to see if the answerer had addressed their incorrect "it's impossible" blurb. They had, they'd removed that part of the answer. But, further my comment had been deleted!?!?!

WTF!!!!

The comment was triple useful. It was useful because it explained how something was possible. It was useful because it linked to a better working example the questioner needed. And, it was useful because it linked to a more flexible solution.

I didn't know this at the time but there is no record of deleted comments. I'd thought maybe I was dreaming. That 2.5 hours I spent on some other answer happened between 4am and 6am. I meant to go to sleep but got sucked into debugging. When I was finished, I checked for more questions, saw this one, and added the comment, but maybe I was too tired and forgot to press "submit"?

So I left the comment again, this time under the question itself since the answer had removed the part about something being impossible. This time I took a screenshot just so I'd know my memory wasn't bad.

I checked back later in the day to find the comment deleted. This prompted me to ask on meta, the stack overflow about stack overflow, what to do about on topic comments being over zealously deleted.

This is when I found out a bunch of things I didn't know

  1. Comments can be deleted by any moderator with for any reason. They don't like you? They can delete all your comments. They hate LGBT people and believe you're LGBT? They can delete your comments. This is one reason why there is no visible comment history.

  2. Comments are apparently meant to be ephemeral.

    Several people claimed comments have absolutely zero value. Therefore their deletion is irrelevant.

I found both of these claims rather ludicrous. Comments have a voting system. Some comments get hundreds of vote. Why would anyone design a voting system for something that has zero value?

Links to other stack overflow questions and answers in comments are scanned and used to show related links on the right side bar. If comments have zero value why would anyone make a system to scan them and display their info?

People can even link directly to other comments. What would be the point of implementing the ability to link to something that has zero value?

But further, I found that, according to various members, the links I'd been leaving are considered spam!!!!

According to these people, the links are nothing but self serving self promotion. More than worthless they considered them actively bad and I was a bad person for spamming the site with them. Here I was spending a few hundred hours writing these articles for users of stack overflow to reference when they needed more than would fit in an answer but apparently trying to tell them about these articles was against the rules.

Some claimed, though it was frowned on, it was slightly less shitty spam if I spelled out I wrote the articles when linking to them. There was no guarantee they wouldn't still be deleted, only that it was marginally less shitty if I declared my supposed conflict of interest.

To put it another way, if someone else posted the links it would be more okay because there is no conflict of interest. I don't buy that though. They're basically saying the exact same comment by person A is ok but by person B is not. That's effing stupid. Either the comment is useful to people reading it or it's not. Who posted it is irrelevant.

Well, this is straw that broke the camel's back.

I'm Done Answering Questions on Stack Overflow

Spending all the time answering people's questions and writing these article to help them was nothing but a burden anyway so I guess I should be thankful Stack Overflow corrected my delusion that I was being helpful and made it clear I was just a self serving spammer.

It's probably for the best anyway. I'll find some more productive way to use my time. To be clear, a bit has flipped in my head. My joy or compulsion or whatever it was that made me want to participate on Stack Overflow is gone or curred. Time to move on.

Comments

The Day Unity Broke The Internet

2021-02-03

Okay, hyperbole aside, Unity mistakenly hardcoded checking the browser's userAgent for MacOS 10 in their "WebGL" game support. MacOS 11 shipped a few months ago and Chrome/Edge started reporting MacOS 11 when Chrome/Edge is run on MacOS 11.

Result: Nearly all existing Unity games on itch.io, simmer.io, github.io, as well as Unity based visualizations on science sites, corporate training sites etc, stopped working for anyone running MacOS 11 using Chrome/Edge (and probably Brave, Vivaldi, etc...?)

It's an interesting issue

Unity shouldn't have hard coded checking for MacOS 10 and failing if it didn't exist

That's just a bug and they have since fixed it. Though ... DOH! Did it really take much thought not to write code that failed if not "10"?

Unfortunately there are many years of games and visualizations out there. It's unlikely most of those games will get updated. Further, even though Unity's main fix is to fix the bug in Unity itself, to apply it you'd have to re-compile your game in a newer version of Unity. Not only is that time consuming but it's no improbable your old project is not compatible with current versions of Unity and will require a bunch of work refactoring the code.

Fortunately you can just replace the file with the issue with a patched version of the file. Luckily that solves the issue and doesn't require you to recompile your game. Still, there will be 1000s of games that don't get this update. If we're lucky some of the sites will just do this automatically for their users.

But BTW, users are still uploading new games even today (Feb 2021) that have this bug as they are using older versions of Unity. Maybe some sites could check and warn the user?

It's been best practice for over a decade to NOT look at userAgent

This MDN article spells out why you shouldn't be looking at userAgent. You should instead be doing feature detection.

Unfortunately reality doesn't always meet expectations. Many web APIs have quirks that can not be easily detected. I didn't dig through the Unity code to see if what they were checking for was a "it can't be helped" kind of issue or a "we didn't know we could feature detect this" issue, but do know I personally have run into these kinds of issues and I also know, sometimes I could try to feature detect but it would be a PITA, meaning checking "If Safari, fall back to X" might take 2 lines of code where as checking that whatever browser I'm using actually follows the spec might take 50 lines of code and I'm lazy ๐Ÿ˜…

The userAgent is going away

Or at least in theory all the browser vendors have suggested they plan to get rid of the userAgent string or freeze it. Here's Chrome plans.

It's not clear what they are replacing it with is all that much better. It's better in that it needs less parsing? It sounds like it still provides all the same info to hang yourself with and to be tracked.

But, in some ways, it does possibly let Unity off the hook. AFAIK Chrome may decide to change their version string claiming MacOS 10 even on MacOS 11. Safari and Firefox already do this, I'm guessing for similar reasons, too many poorly coded sites broken. You might think Safari and Firefox don't report MacOS 11 because of tracking but if preventing tracking was their goal they wouldn't report the version of the browser in the userAgent, which they do.

Sometimes I want the userAgent

I recently wanted to write some software to check how many users can use feature X and I wanted to do it by checking which OS and which browser they are on so I can see for example, 70% of users on Safari, Mac can use feature X and 60% of users on Chrome, Android can use the same feature.

That seems like a reasonable thing to want to know so as much as I don't like being tracked I'm also not sure getting rid of the data available via userAgent is the best thing. It doesn't appear that data is going away though, just changing.

So?

I wrote "Unity broke the internet" mostly because Unity's many years old bug, spread over thousands of sites, potentially forced the browsers to work around those sites rather than progress forward. Unfortunately it's not the first time that's happened


Apparently the same thing happened to Firefox and they ended up adjusting their userAgent string. That rabbit hole lead me to this horror fest! ๐Ÿ˜ฑ

Comments

Zip - How not to design a file format.

2021-01-16

The Zip file format is now 32 years old. You'd think being 32 years old the format would be well documented. Unfortunately it's not.

I have a feeling this is like many file formats. They aren't designed, rather the developer just makes it up as they go. If it gets popular other people want to read and/or write them. They either try to reverse engineer the format OR they ask for specs. Even if the developer writes specs they often forget all the assumptions their original program makes. Those are not written down and hence the spec is incomplete. Zip is such a format.

Zip claims its format is documented in a file called APPNOTE.TXT which can be found here.

The short version is, a zip file consists of records, each record starts with some 4 byte marker that generally takes the form

0x50, 0x4B, ??, ??

Where the 0x50, 0x4B are the letters PK standing for "Phil Katz", the person who made the zip format. The two ?? are bytes that identify the type of the record. Examples

0x50 0x4b 0x03 0x04   // a local file record
0x50 0x4b 0x01 0x02   // a central directory file record
0x50 0x4b 0x06 0x06   // an end of central directory record

Records do NOT follow any standard pattern. To read or even skip a record you must know its format. What I mean is there are several other formats that follow some convention like each record id is followed by the length of the record. So, if you see an id, and you don't understand it, you just read the length, skip that many bytes (*), and you'll be at the next id. Examples of this type include most video container formats, jpgs, tiff, photoshop files, wav files, and many others.
(*) some formats require rounding the length up to the nearest multiple of 4 or 16.

Zip does NOT do this. If you see an id and you don't know how that type of record's content is structured there is no way to know how many bytes to skip.

APPNOTE.TXT says the following things

4.1.9 ZIP files MAY be streamed, split into segments (on fixed or on removable media) or "self-extracting". Self-extracting ZIP files MUST include extraction code for a target platform within the ZIP file.

4.3.1 A ZIP file MUST contain an "end of central directory record". A ZIP file containing only an "end of central directory record" is considered an empty ZIP file. Files MAY be added or replaced within a ZIP file, or deleted. A ZIP file MUST have only one "end of central directory record". Other records defined in this specification MAY be used as needed to support storage requirements for individual ZIP files.

4.3.2 Each file placed into a ZIP file MUST be preceded by a "local file header" record for that file. Each "local file header" MUST be accompanied by a corresponding "central directory header" record within the central directory section of the ZIP file.

4.3.3 Files MAY be stored in arbitrary order within a ZIP file. A ZIP file MAY span multiple volumes or it MAY be split into user-defined segment sizes. All values MUST be stored in little-endian byte order unless otherwise specified in this document for a specific data element.

4.3.6 Overall .ZIP file format:

      [local file header 1]
      [encryption header 1]
      [file data 1]
      [data descriptor 1]
      . 
      .
      .
      [local file header n]
      [encryption header n]
      [file data n]
      [data descriptor n]
      [archive decryption header] 
      [archive extra data record] 
      [central directory header 1]
      .
      .
      .
      [central directory header n]
      [zip64 end of central directory record]
      [zip64 end of central directory locator] 
      [end of central directory record]
   

4.3.7 Local file header:

      local file header signature     4 bytes  (0x04034b50)
      version needed to extract       2 bytes
      general purpose bit flag        2 bytes
      compression method              2 bytes
      last mod file time              2 bytes
      last mod file date              2 bytes
      crc-32                          4 bytes
      compressed size                 4 bytes
      uncompressed size               4 bytes
      file name length                2 bytes
      extra field length              2 bytes

      file name (variable size)
      extra field (variable size)
   

4.3.8 File data

Immediately following the local header for a file SHOULD be placed the compressed or stored data for the file. If the file is encrypted, the encryption header for the file SHOULD be placed after the local header and before the file data. The series of [local file header][encryption header] [file data][data descriptor] repeats for each file in the .ZIP archive.

Zero-byte files, directories, and other file types that contain no content MUST NOT include file data.

4.3.12 Central directory structure:

      [central directory header 1]
      .
      .
      . 
      [central directory header n]
      [digital signature] 
   

File header:

        central file header signature   4 bytes  (0x02014b50)
        version made by                 2 bytes
        version needed to extract       2 bytes
        general purpose bit flag        2 bytes
        compression method              2 bytes
        last mod file time              2 bytes
        last mod file date              2 bytes
        crc-32                          4 bytes
        compressed size                 4 bytes
        uncompressed size               4 bytes
        file name length                2 bytes
        extra field length              2 bytes
        file comment length             2 bytes
        disk number start               2 bytes
        internal file attributes        2 bytes
        external file attributes        4 bytes
        relative offset of local header 4 bytes

        file name (variable size)
        extra field (variable size)
        file comment (variable size)
   

4.3.16 End of central directory record:

      end of central dir signature    4 bytes  (0x06054b50)
      number of this disk             2 bytes
      number of the disk with the
      start of the central directory  2 bytes
      total number of entries in the
      central directory on this disk  2 bytes
      total number of entries in
      the central directory           2 bytes
      size of the central directory   4 bytes
      offset of start of central
      directory with respect to
      the starting disk number        4 bytes
      .ZIP file comment length        2 bytes
      .ZIP file comment       (variable size)
   

There are other details involving encryption, larger files, optional data, but for the purposes of this post this is all we need. We need one more piece of info, how to make a self extracting archive.

To do so we could look back to ZIP2EXE.exe which shipped with pkzip in 1989 and see what it does but it's easier look at Info-Zip to see what happens.

How do I make a DOS (or other non-native) self-extracting archive under Unix?

The procedure is basically described in the UnZipSFX man page. First grab the appropriate UnZip binary distribution for your target platform (DOS, Windows, OS/2, etc.), as described above; we'll assume DOS in the following example. Then extract the UnZipSFX stub from the distribution and prepend as if it were a native Unix stub:

> unzip unz552x3.exe unzipsfx.exe                // extract the DOS SFX stub
> cat unzipsfx.exe yourzip.zip > yourDOSzip.exe  // create the SFX archive
> zip -A yourDOSzip.exe                          // fix up internal offsets
> 

That's it. You can still test, update and delete entries from the archive; it's a fully functional zipfile.

So given all of that let's go over some problems.

How do you read a zip file?

This is undefined by the spec.

There are 2 obvious ways.

  1. Scan from the front, when you see an id for a record do the appropriate thing.

  2. Scan from the back, find the end-of-central-directory-record and then use it to read through the central directory, only looking at things the central directory references.

Scanning from the back is how the original pkunzip works. For one it means if you ask for some subset of files it can jump directly to the data you need instead of having to scan the entire zip file. This was especially important if the zip file spanned multiple floppy disks.

But, 4.1.9 says you can stream zip files. How is that possible? What if there is some local file record that is not referenced by the central directory? Is that valid? This is undefined.

4.3.1 states

Files MAY be added or replaced within a ZIP file, or deleted.

Okay? That suggests the central directory might not reference all the files in the zip file because otherwise this statement about files being added, replaced, or delete has no point to be in the spec.

If I have file1.zip that contains files, A, B, C and I generate file2.zip that only contains files A, B. Those are just 2 independent zip files. It makes zero sense to put in the spec that you can add, replace, and delete files unless that knowledge some how affects the format of a zip file.

In other words. If you have

  [local file A]
  [local file B]
  [local file C]
  [central directory file A]
  [central directory file C]
  [end of central directory]

Then clearly B is deleted as the central directory doesn't reference it. On the other hand, if there's no [local file B] then you just have an independent zip file, independent of some other zip file that has B in it. No need for the spec to even mention that situation.

Similarly if you had

  [local file A (old)]
  [local file B]
  [local file C]
  [local file A (new)]
  [central directory file A(new)]
  [central directory file B]
  [central directory file C]
  [end of central directory]

Then A (old) has been replaced by A (new) according to the central directory. If on the other hand there is no [local file A (old)] you just have an independent zip file.

You might think this is nonsense but you have to remember, pkzip comes from the era of floppy disks. Reading an entire zip file's contents and writing out a brand new zip file could be an extremely slow process. In both cases, the ability to delete a file just by updating the central directory, or to add a file by reading the existing central directory, appending the new data, then writing a new central directory, is a desirable feature. This would be especially true if you had a zip file that spanned multiple floppy disks; something that was common in 1989. You'd like to be able to update a README.TXT in your zip file without having to re-write multiple floppies.

In discussion with PKWare, they state the following

The format was originally intended to be written front-to-back so the central directory and end of central directory record could be written out last after all files included in the ZIP are known and written. If adding files, changes can applied without rewriting the entire file. This was how the original PKZIP application was designed to write .ZIP files. When reading, it will read the ZIP file end of central directory first to locate the central directory and then seek to any files it needs to access

Of course "add" is different than "delete" and "replace".

Whether or not having local files not referenced by the central directory is undefined by the spec. It is only implied by the mention of:

Files MAY be added or replaced within a ZIP file, or deleted.

If it is valid for the central directory to not reference all the local files then reading a zip file by scanning from the front may fail. Without special care you'd get files that aren't supposed to exist or errors from trying to overwrite existing files.

But, that contradicts 4.1.9 that says zip files maybe be streamed. If zip files can be streamed then both of the example above would fail because in the first case we'd see file B and in the second we'd see file A (old) before we saw that the central directory doesn't reference them. If you have to wait for the central directory before you can correctly use any of the entries then functionally you can not stream zip files.

Can the self extracting portion have any zip IDs in it?

Seeing the instructions for how to create a self extracting zip file above, we just prepend some executable code to the front of the file and then fix the offsets in the central directory.

So let's say your self extractor has code like this

switch (id) {
  case 0x06054b50:
    read_end_of_central_directory();
    break;
  case 0x04034b50:
    read_local_file_record();
    break;
  case 0x02014b50:
    read_center_file_record();
    break;
  ...
}

Given the code above, it's likely those values 0x06054b50, 0x04034b50, 0x02014b50 will appear in binary in the self extracting portion of the zip file at the front of the file. If you read a zip file by scanning from the front your scanner my see those ids and mis-interpret them as a zip records.

In fact you can imagine a self extractor with a zip file in it like this

// data for a zip file that contains
//   LICENSE.txt
//   README.txt
//   player.exe
const unsigned char[] runtimeAndLicenseData = {
  0x50, 0x4b, 0x03, 0x04, ??, ??, ...
};

int main() {
   extractZipFromFile(getPathToSelf());
   extractZipFromMemory(runtimeAndLicenseData, sizeof(runtimeAndLicenseData));
}

Now there's a zip file in the self extractor. Any reader that reads from the front would see this inner zip file and fail. Is that a valid zip file? This is undefined by the spec.

I tested this. The original PKUnzip.exe in DOS, the Windows Explorer, MacOS Finder, Info-Zip (the unzip included in MacOS and Linux), all clearly read from the back and see the files after the self extractor. 7z, Keka, see the embedded zip inside the self extractor.

Is that failure or is that a bad zip file? The APPNOTE.TXT does not say. I think it should be explicit here and I think it's one of those unstated assumptions. PKunzip scans from the back and so this just happens to work but the fact of how it happens to work is never documented. The issue that the data in the self-extractor might happen to resemble a zip file is just glossed over. Similarly streaming will likely fail if it hasn't already from the previous issues.

You might think this is a non issue but their are 100s of thousands of self extracting zip files out there from the 1990s in the archives. A forward scanner might fail to read these.

Can the zip comment contain zip IDs in it?

If you go look at 4.3.16 above you'll see the end of a zip file is a variable length comment. So, if you're doing backward scanning you basically read from the back of the file looking for 0x50 0x4B 0x05 0x06 but what if that sequence of bytes is in the comment?

I'm sure Phil Katz never gave it a second thought. He just assumed people would put the equivalent of a README.txt in there. As such it would only have values from 0x20 to 0x7F with maybe a 0x0D (carriage return), 0x0A (linefeed), 0x09 (tab) and maybe 0x06 (bell).

Unfortunately all of those values in the ids are valid ASCII, even utf-8. We already went over 0x50 = P and 0x4B = K. 0x06 is "Bell" in ASCII (makes a noise or flashes the screen). 0x05 is "Enquiry".

The APPNOTE.TXT should arguably explicitly specify if this is invalid. Indirectly 4.3.1 says

A ZIP file MUST have only one "end of central directory record"

But what does that mean? Does that mean the bytes 0x50 0x4B 0x05 0x06 can't appear in the comment nor the self extracting code? Does it mean the first time you see that scanning from the back you don't try to find a second match?

If you scan from the front and run into none of the issues mentioned before, then a forward scanner would successfully read this. On the other hand, pkunzip itself would fail.

What if the offset to the central directory is 1,347,093,766?

That offset is 0x504b0506 so it will appear to be end central directory header. I think 1.3gig zip file wasn't even on the radar when zip was created and in fact extensions were required to handle files larger then 4gig. But, it does show one more way the format is poorly designed.

What's a good design?

There's certainly debate to be had about what a good design would be but somethings are arguably easy to decide if we could start over.

  1. It would have been better if records had a fixed format like id followed by size so that you can skip a record you don't understand.

  2. It would have been better if the last record at the end of the file was just an offset-to-end-of-central-directory record as in

       0x504b0609 (id: some id is not in use)
       0x04000000 (size of data of record)
       0x???????? (relative offset to end-of-central-directory)
       

    Then there would be no ambiguity for reading from the back.

    1. Read the last 12 bytes
    2. Check the first 8 are 0x50 0x4b 0x06 0x09 0x04 0x00 0x00 0x00. If not, fail.
    3. Read the offset and go to the end-of-central-directory

    Or, conversely, put the comment in its own record and write it before the central directory and put an offset to it in the end-of-central-directory-record. Then at least this issue of scanning over the comment would disappear.

  3. Be clear about what data can appear in a self extracting stub.

    If you want to support reading from the front it seems required to state that the self extracting portion can't appear to have any records.

    This is hard to enforce unless you specifically wrote some validator. If you just check based on whether your own app can read the zip file then, as it stands now, Pkzip, pkunzip, info-zip (the zip in MacOS, Linux), Windows Explorer, and MacOS all don't care what's in the self extracting portion so they aren't useful for validation. You must explicitly state that you must scan from the back in the spec or write a validator that rejects zip that are not forward scanable and state in the spec why.

  4. Be clear if the central directory can disagree with local file records

  5. Be clear if random data can appear between records

    A backward scanner does not care what's between records. It only cares it can find the central directory and it only reads what that central directory points to. That means there can be any random data between records (or some at least some records).

    Be explicit if this is okay or not okay. Don't rely on implicit diagrams.

What to do, how to fix?

If I was to to guess all of these issues are implementation details that didn't make it into the APPNOTE.TXT. What I believe the APPNOTE.TXT really wants to say is "a valid zip file is one that pkzip can manipulate and pkunzip can correctly unzip. Instead it defines things in such a way that various implementations can make files that other implementations can't read.

Of course with 32 years of zip files out their we can't fix the format. What PKWare could do is get specific about these edge cases. If it was me I'd add these sections to the APPNOTE.TXT

4.3.1 A ZIP file MUST contain an "end of central directory record". A ZIP file containing only an "end of central directory record" is considered an empty ZIP file. Files MAY be added or replaced within a ZIP file, or deleted. A ZIP file MUST have only one "end of central directory record". Other records defined in this specification MAY be used as needed to support storage requirements for individual ZIP files.

The "end of central directory record" must be at the end of the file and the sequence of bytes, 0x50 0x4B 0x05 0x06, must not appear in the comment.

The "central directory" is the authority on the contents of the zip file. Only the data it references are valid to read from the file. This is because (1) the contents of the self extracting portion of the file is undefined and might be appear to contain zip records when in fact they are not related to the zip file and (2) the ability to add, update, and delete files in a zip file stems from the fact that it is only the central directory that knows which local files are valid.

That would be one way. I believe this will read the 100s of millions of existing zip files out there.

On the other hand, if PKWare claims such files that have these issues don't exist then this would work as well

4.3.1 A ZIP file MUST contain an "end of central directory record". A ZIP file containing only an "end of central directory record" is considered an empty ZIP file. Files MAY be added or replaced within a ZIP file, or deleted. A ZIP file MUST have only one "end of central directory record". Other records defined in this specification MAY be used as needed to support storage requirements for individual ZIP files.

The "end of central directory record" must be at the end of the file and the sequence of bytes, 0x50 0x4B 0x05 0x06, must not appear in the comment.

There can be no [local file records] that do not appear in the central directory. This guarantee is required so reading a file front to back provides the same results as reading it back to front. Any file that does not follow this rule is an invalid zip file.

A self extracting zip file must not contain any of the sequences of record ids listed in this document as they maybe mis-interpreted by forward scanning zip readers. Any file that does not follow this rule is an invalid zip file.

I hope they will update the APPNOTE.TXT so that the various zip readers and zip creators can agree on what's valid.

Unfortunately I feel like pkware doesn't want to be clear here. Their POV seems to be that zip is an ambiguous format. If you want to read by scanning from the front then just don't try to read files you can't read that way. They're still valid zip files and but the fact that you can't read them is irrelevant. It's just your choice to fail to support those.

I suppose that's a valid POV. Few if any zip libraries handle every feature of zip. Still, it would be nice to know if you're intentionally not handling something or if you're just reading the file wrong and getting lucky that it works sometimes.

The reason all this came up is I wrote a javascript unzip library. There are tons out here but I had special needs the other libraries I found didn't handle. In particular I needed a library that let me read a single file from a large zip as fast as possible. That means backward scanning, finding the offset to the desired file, and just decompressing that single file. Hopefully others find it useful.


You might find this history of Zip fascinating

Comments

JSBenchIt.org and JSGist.org

2020-11-09

I recently made 2 new sites. They came about like this.

Once in a while I want to benchmark solutions in JavaScript just to see how much slower one solution is vs another. I used to use jsperf.com but sometime in early 2020 or 2019 it disappeared.

Searching around I found 2 others, jsbench.me. Trying them they both have their issues. Jsbench.me is ugly. Probably not fair but whatever. Using it bugged me. Jsben.ch, at least as of this writing had 2 issues when I tried to use it. One is that if my code had a bug the site would crash as in it would put up a full window UI that blocks everything with a "running..." message and then never recovers. The result was all the code I typed in was lost and I'd have to start over. The other is it has a 4k limit. 4k might sound like a lot but I ran into that limit trying to test a fairly simple thing. I managed to squeeze my test in with some work but worse, there's no contact info anywhere except a donate button that leads directly to the donation site, not a contact sight so there's no way to even file a bug let alone make a suggestion.

In any case I put up with it for 6 months or so but then one day about a month ago, I don't remember what triggered it but I figured I could make my own site fairly quickly where I'm sure in my head quickly meant 1-3 days max. ๐Ÿ˜‚

benchmark.js

So, this is what happened. First I decide I should use benchmark.js mostly because I suck at math and it claims "statistically significant results". I have no idea what that means ๐Ÿ˜… but, a glance at the code shows some math happening that's more than I'd do if I just wrote my own timing functions.

Unfortunately I'd argue benchmark.js is actually not a very good library. They made up some username or org name called "bestiejs" to make it sound like it's good and they claim tests and docs but the docs are horrible auto-generated docs. They don't actually cover how to use the library they just list a bunch of classes and methods and it's left to you to figure out which functions to call and when and why. There's also some very questionable design choices like the way you add setup code is by manually patching the prototype of one of their classes. WAT?!?

I thought about writing my own anyway and trying to extract the math parts but eventually I got things working enough and just wanted to move on.

Personal Access Tokens

I also didn't want to run a full server with database and everything else so I decided I'd see if it was possible to store the data in a github gist. It turns out yes, it's possible but I also learned there is no way to make a static website that supports Oauth to let the user login to github.

A workaround is a user can make a Personal Access Token which is a fancy way of basically making a special password that is given certain permissions. So, in order to save the user would have to go to github, manually make a personal access token, paste it into jsbenchit.org and then they could save. It worked! ๐ŸŽ‰

As I got it working I released I could also make a site similar to jsfiddle or codepen with only minor tweaks to the UI so I started on that too.

ยฏ\(ใƒ„)/ยฏ

Running User Code

Both sites run arbitrary user code and so if I didn't do something people could write code that steals the personal access token. That's no good. Stealing their own token is not an issue but passing a benchmark or jsgist to another user would let them steal that user's token.

The solution is to run the user's code in an iframe on another domain. That domain can't read any of the data from the main site so problem is solved.

Unfortunately I ran into a new problem. Well, maybe it's not so new. The problem is since the servers are static I can't serve the user's code like a normal site would. If you look at jsfiddle, codepen, and stack overflow snippets you'll see they run the code from a server served page generated using the user's code. With a static site I don't have that option.

To work around it I generated a blob, make a URL to the blob and have the browser load that in an iframe. I use this solution on webgfundmentals.org, webgl2fundamentals.org, and threejsfundamentals.org It works but it has other problems. One is since I can't serve any files whatsoever I have to re-write URLs if you use more than 1 file.

Take for example something that uses workers. You usually need a minimum of 2 files. An HTML file with a <script> section that launches a worker and the worker's script is in another file. So you start with main.html that loads worker.js but you end up with blob:1234-1314523-1232 for main.html but it's still referencing worker.js and you have to some how find that and change it to the blob url that was generated for worker.js. I actually implemented this solution on those sites I mentioned above but it only works because I wrote all the examples that are running live and the solutions only handle the small number of cases I needed to work.

The second problem with the blob solution they are no good for debugging. Every time the user clicks "run" new blobs are created so any breakpoints you set last time you ran it don't apply to the new blob since they're associated with a URL and that URL has just changed.

Looking into it I found out I could solve both problems with a service worker. The main page starts the service worker then injects the filename/content of each file into the service worker. It then references those files as normal URLs so the don't change. Both problems are solved. ๐Ÿ˜Š

I went on to continue making the sites even though I was way past the amount of time I thought I'd be spending on them.

Github Login

In using the sites I ran into a new problems. Using a personal access token sucked! I have at least 4 computers I want to run these on. A Windows PC, a Mac, an iPhone, and an Android phone. When I'd try to use a different PC I needed to either go through the process of making a new personal access token, or I needed to find someway to pass that token between machines, like email it to myself. ๐Ÿคฎ

I wondered if I could get the browser to save it as a password. It turns out, yes, if you use a <form> and an <input type="password"> and you apply the correct incantation when the user clicks a "submit" button the browser will offer to save the personal access token as a password.

Problem solved? No ๐Ÿ˜ญ

A minor issue is there's no username but the browsers assume it's always username + password. That's not that big a deal, I can supply a fake username though it will probably confuse users.

A major issue though is that passing between machines via a browser's password manager doesn't help pass between different browsers. If I want to test Firefox vs Chrome vs Safari then I was back to the same problem of keeping track of a personal access token somewhere.

Now I was entering sunk cost issues. I'd spent a bunch of time getting this far but the personal access token issues seemed like they'd likely make no one use either site. If no one is going to use it then I've wasted all the time I put in already.

Looking into it more it turns out the amount of "server" need to support oauth so that users could log in with github directly is actually really tiny. No storage is needed, almost nothing.

Basically they way Oauth works is

  1. user clicks "login with github"
  2. static page opens a popup to https://github.com/login/oauth/authorize and passes it an app id (called client_id), the permissions the app wants, and something called "state" which you make up.
  3. The popup shows github's login and asks for permission for the app to use whatever features it requested.
  4. If the user picks "permission granted" then the github page redirects to some page you pre-registered when you registered your app with github. For our case this would be https://jsbenchit.org/auth.html. To this page the redirect includes a "code" and the "state" passed at step 2.
  5. The auth.html page either directly or by communicating with the page that opened the popup, first verifies that the "state" matches what was sent at step 2. If not something is fishy, abort. Otherwise it needs to contact github at a special URL and pass the "code", the "client_id" and a "client_secret".

    Here's the part that needs a server. The page can't send the secret because then anyone could read the secret. So, the secret needs to be on a server. So,

  6. The page needs to contact a server you control that contains the secret and passes it the "code". That server then contacts github, passes the "code", "client_id", and "client_secret" to github. In response github will return an access token which is exactly the same as a "personal access token" except the process for getting one is more automated.
  7. The page gets the access token from the server and starts using it to access github's API

If you were able to follow that the short part is you need a server, and all it has to do is given a "code", contact github, pass the "code", "client_id" and "client_secret" on to github, and pass back the resulting token.

Pretty simple. Once that's done the server is no longer needed. The client will function without contacting that server until and unless the token expires. This means that server can be stateless and basically only takes a few lines of code to run.

A found a couple of solutions. One is called Pizzly. It's overkill for my needs. It's a server that provides the oauth server in step 6 above but it also tracks the tokens for you and proxies all other github requests, or requests to whatever service you're using. So your client side code just gets a pizzly user id which gets translated for you to the correct token.

I'm sure that's a great solution but it would mean paying for a much larger server, having to back up user accounts, keep applying patches as security issues are found. It also means paying for all bandwidth between the browser can github because pizzly is in the middle.

Another repo though made it clear how simple the issue can be solved. It's this github-secret-keeper repo. It runs a few line node server. I ran the free example on heroku and it works! But, I didn't want to make an heroku account. It seemed too expensive for what I needed it for. I also didn't feel like setting up a new dynamo at Digital Ocean and paying $5 a month just to run this simple server that I'd have to maintain.

AWS Lambda

I ended up making an AWS Lambda function to do this which added another 3 days or so to try to learn enough AWS to get it done.

I want to say the experience was rather poor IMO. Here's the problem. All the examples I found showed lambda doing node.js like stuff, accepting a request, reading the parameters, and returning a response. Some showed the parameters already parsed and the response being structured. Trying that didn't work and it turns out the reason is AWS for this feature is split into 2 parts.

Part 1 is AWS Lambda which just runs functions in node.js or python or Java etc...

Part 2 is AWS API Gateway which provides public facing endpoints (URLS) and routes them to different services on AWS, AWS Lambda being one of those targets.

It turns out the default in AWS API Gateway doesn't match any of the examples I saw. In particular the default in AWS API Gateway is that you setup a ton of rules to parse and validate requests and parameters and only if they parse correctly and pass all the validation do they get forwarded to the next service. But that's not really what the example shown wanted. Instead they wanted AWS API Gateway to effectively just pass through the request. That's not the default and I'd argue it not being the default is a mistake.

My guess is that service was originally written in Java. Because Java is strongly typed it was natural to think in terms of making the request fit strong types. Node.js on the other hand, is loosely typed. It's trivial to take random JSON, look at the data you care about, ignore the rest, and move on with your life.

In any case I finally figured out how to get AWS API Gateway to do what all the AWS Lambda examples I was seeing needed and it started working.

My solution is here if you want to use it for github or any Oauth service.

CSS and Splitting

Next up was spitting and CSS. I still can't claim to be a CSS guru in any way shape or form and several times I year I run into infuriating CSS issues where I thought I'd get something done in 15 minutes but turns into 15 minutes of the work I thought I was going to do and 1 to 4 hours of trying to figure out why my CSS is not working.

I think there are 2 big issues.

  1. is that Safari doesn't match Chrome and Firefox so you get something working only to find it doesn't work on Safari

  2. Nowhere does it seem to be documented how to make children always fill their parents. This is especially important if you're trying to make a page that acts more like an app where the data available should always fit on the screen vs a webpage that be been as tall as all the content.

    To be more clear you try (or I try) to make some layout like

       +--------------------+
       |                    |
       +---+------------+---+
       |   |            |   |
       |   |            |   |
       |   |            |   |
       +---+-----+------+---+
       |         |          |
       +---------+----------+
       

    and I want the entire thing to fill the screen and the contents of each area expand to use all of it. For whatever reason it never "just works". I'd think this would be trivial but something about it is not or at least not for me. It's always a bunch of sitting in the dev tools and adding random height: 100% or min-height: 0 or flex: 1 1 auto; or position: relative to various places in the hope things get fixed and they don't break something else or one of the other browsers. I'd think this would be common enough that the solution would be well documented on MDN or CSS Tricks or some place but it's not or at least I've never found it. Instead there's just all us clueless users reading the guesses of other clueless users sharing their magic solutions on Stack Overflow.

    I often wonder if any of the browser makers or spec writers ever actually use the stuff they make and why they don't work harder to spread the solutions.

    Any any case my CSS seems to be doing what I want at the moment

That said, I also ran into the issue that I needed a splitter control that let you drag the divider between two areas to adjust their sizes. There's 3 I found but they all had issues. One was out of date, and unmaintained and got errors with current React. Yea, I used react. Maybe that was a bad decision. Still not sure.

After fighting with the other solutions I ended up writing my own so that was a couple of days of working through issues.

Disqus

Next up was comments. I don't know why I felt compelled to add comments but I did. I felt like people being able to comment would be net positive. Codepen allows comments. The easiest thing to do is just tack on disqus. Similar to the user code issue though I can't use disqus directly on the main site otherwise they could steal the access token.

So, setup another domain, put disqus in an iframe. The truth is disqus already puts itself in an iframe but at the top level it does this with a script on your main page which means they can steal secrets if they want. So, yea, 3rd domain (2nd was for user code).

The next problem is there is no way in the browser to size an iframe to fit its content. It seems ridiculous to have that limitation in 2020 but it's still there. The solution is the iframe sends messages to the parent saying what size its content is and then the parent can adjust the size of the iframe to match. It turns out this is how disqus itself works. The script it uses to insert an iframe listens for messages to resize the iframe.

Since I was doing iframe in iframe I needed to re-implement that solution.

It worked, problem solved..... or is it? ๐Ÿ˜†

Github Comments

It's a common topic on tech sites but there is a vocal minority that really dislike disqus. I assume it's because they are being tracked across the net. One half solution is you put a click through so that by default disqus doesn't load but the user can click "load comments" which is effectively an opt in to being tracked.

The thing is, gists already support comments. If only there was a way to use them easily on a 3rd party site like disqus. There isn't so, ....

There's an API where you can get the comments for a gist. You then have to format them from markdown into HTML. You need to sanitize them because it's user data and you don't want people to be able to insert JavaScript. I was already running comments on a 3rd domain so at least that part is already covered.

In any case it wasn't too much work to get existing comments displayed. New comments was more work though.

Github gists display as follows

+----------+
| header   |
+----------+
| files    |
|          |
|          |
+----------+
| comments |
|          |
|          |
+----------+
| new      |
| comment  |
| form     |
+----------+

that comment form is way down the page. If there was an id to jump to I could have possibly put that page in an iframe and just use a link like https://gist.github.com/<id>/#new-comment-form. to get the form to appear in a useful way. That would give the full github comment UI which includes drag and drop image attachments amount other things. Even if putting it in an iframe sucked I could have just had a link in the form of

<a href="https://gist.github.com/<id>/#new-comment-form">click here to leave a comment</a>

But, no such ID exits, nor does any standalone new comment form page.

So, I ended up adding a form. But for a preview we're back to the problem of user data on a page that has access to a github token.

The solution to put the preview on a separate page served from the comments domain and send a message with new content when the user asks for a preview. That way, even if we fail to fully sanitize the user content can't steal the tokens.

Embedding

Both sites support embedding

jsgist.org just uses iframes.

JsBenchIt supports 2 embed modes. One, uses an iframe.

+ there's no security issues (like I can't see any data on whatever site you embedded it)

- It's up to you to make your iframe fit the results

The other mode uses a script

+ it can auto size the frame

- if I was bad I could change the script and steal your login credentials for whatever site you embed it on.

Of course I'd never do that but just to be aware. Maybe someone hacks my account or steals my domain etc... This same problem exists for any scripts you use from a site you don't control like query from a CDN for example so it's not uncommon to use a script. Just pointing out the trade off.

I'm not sure what the point of embedding the benchmarks is but I guess you could show off your special solution or, show how some other solution is slow, or maybe post one and encourage others to try to beat it.

Closing Thoughts

I spent about a month, 6 to 12hrs a day on these 2 sites so far. There's a long list of things I could add, especially to jsgist.org. No idea if I will add those things. jsbenchit.org has a point for me because I didn't like the existing solution. jsgist.org has much less of a point because are are 10 or sites that already do something like this in various ways. jsfiddle, codepen, jsbin, codesandbox, glitch, github codespaces, plunkr, and I know there are others so I'm not sure what the point was. It started as just a kind of "oh, yea, I could do that too" while making jsbenchit and honestly I spent probably spent 2/3rds of the time there vs the benchmark site.

I honestly wish I'd find a way to spend this kind of time on something that has some hope of generating income, not just income but also something I'm truly passionate about. Much of this feels more like the procrastination project that one does to avoid doing the thing they should really do.

That said, the sites are live, they seem to kind of work though I'm sure there are still lurking bugs. Being stored in gists the data is yours. There is no tracking on the site. The sites are also open source so pull requests welcome.

Comments

GitHub has a Permission Problem.

2020-09-27

TL;DR: Thousands of developers are giving 3rd parties write access to their github repos. This is even more irresponsible than giving out your email password or your computer's password since your github repos are often used by more than just you. The tokens given to 3rd parties are just like passwords. A hacker that breaches a company that has that info will suddenly have write access to every github repo the breached company had tokens for.

github should work to stop this irresponsible practice.

I really want to scream about security on a great many fronts but today let's talk about github.

What the actual F!!!

How is this not a 2000 comment topic on HN and Github not shamed into fixing this?

Github's permission systems are irresponsible in the extreme!!

Lots of sites let you sign up via github. Gatsby is one. Here's the screen you get when you try to sign up via your github account.

Like seriously, WTF does "Act on your behalf" mean? Does it mean Gatsby can have someone assassinated on my behalf? Can they take out a mortgage on my behalf? Can they volunteer me for the Peace Corps on my behalf? More seriously can they scan all my private repos on my behalf? Insert trojans in my code on my behalf? Open pull requests on other people's projects on my behalf? Log in to every other service I've connected to my github account on my behalf? Delete all my repos on my behalf? Add users to my projects on my behalf? Change my password on my behalf?

This seems like the most ridiculous permission ever!

I bought this up with github and they basically threw up their hands and said "Well, at least we're telling you something". No you're not. You're effectively telling me absolutely nothing except that you're claiming if I click through you're giving that company permission to do absolute anything. How is that useful info?

But, just telling me isn't really the point. The point is each service should be required to use as small of permissions as is absolutely necessary. If I sign up for a service, the default should be no permissions except getting my email address. If a service is supposed to work with a repo (like gatsby is) then github should provide an interface such that gatsby tells github "Give me a list of repos the user wants me to use" and github present the UI to select an existing one or create new ones and when finished, only those repos are accessible and only with the minimal permissions need.

This isn't entirely github's fault though, the majority of the development community seems asleep as well.

Let's imagine your bank let you sign in to 3rd party services in a similar manner. How many people would click through on "Let ACME corp act on your behalf on your Citibank Account". I think most people would be super scared of permissions like that. Instead they'd want very specific permission like, only permission to deposit money, or only permission to read the balance, or only permission to read transactions, etc...

Github providing blanket permissions to so many companies is a huge recipe for disaster just waiting to happen. If any one of those companies gets hacked, or has insider help, or has a disgruntled employee, suddenly your trade secrets are stolen, your unreleased app is leaked, your software is hacked with a trojan and all your customers sue you for the loss of work your app caused. It could be worse, you could run an open source library so by hacking ACME corp the bad guys can hack your library and via that hack everyone using your library.

I get why github does it and/or why the apps do it. For example check out Forestry. They could ask for minimal permissions and good on them for providing a path to go that route. They ask for greater permissions so that they can do all the steps for you. I get that. But if you allow them blanket access to your github (or gitlab), YOU SHOULD ARGUABLY BE DISQUALIFIED FROM BEING A SOFTWARE DEVELOPER!!!

The fact that you trusted some random 3rd party with blanket permissions to edit all of your repos and change all of your permissions is proof you don't know WTF you're doing and you can't be trusted. It's like if someone asked you for the password to your computer. If you give it out you're not computer literate!

boy: "Is it ok I set my password to your birthday?

girl: "Then your password is meaningless!"

Here's the default permissions Forestry asks for if you follow their recommended path.

First let's explain what Forestry is. It's a UI for editing blog posts through git so you can have a nice friendly interface for your git based static site generator. That's great! But, at most it only needs access to a single repo. Not all your public repos! If you click through and picked "Authorize" that's no different then giving them the password to your computer. Maybe worse because at least your hacked computer will probably only affect you.

Further, the fact that companies like Forestry even ask for this should be shameful! Remember when various companies like Facebook, Yelp, when you signed up they'd ask for your the username and password for your email account? Remember how pretty much every tech person on the planet knew that was seriously irresponsible to even ask? Well this is no different. It's entirely irresponsible for Forestry to ask for these kind of blanket permissions! It's entirely irresponsible for any users to give them these permissions! How are all the tech leaders seemingly asleep at calling this out?

Like I mentioned above, part of this arguably lies at Github's feet. Forestry does this because github provides no good flow to do it well so Forestry is left with 2 options (1) be entirely irresponsible but make it easy for the user to use their service, (2) be responsible but lose sales because people can't get setup easily.

Instead it should be a sign they're an irresponsible and untrustworthy company that they ask for these kinds of permissions at all. And further, github should be should also be ashamed their system encourages these kinds of blanket permissions.

Think of it this way. There are literally millions of open source libraries. npm has over a million libraries and that's just JavaScript. Add in python, C++, Java, C#, ruby, and all the other projects on github. Hundreds of thousands of developers wrote those libraries. How many of those developers have given out the keys their repos so random 3rd parties can hack their libraries? Maybe they gave too broad permissions to some code linting site. Maybe they gave too broad permissions to some project monitoring site. Maybe they gave too broad permissions just to join a forum using their github account. Isn't that super irresponsible? They've opened a door by using the library and they're letting more people in the door. That can't be good.

I don't blame the devs so much as github for effectively making this common. Github needs to take security seriously and that means working to make issues like this the exception, not the rule. It should be the easiest thing to do to allow a 3rd party minimal access to your repo. It should be much harder to give them too much access. There should be giant warnings that you're about to do something super irresponsible and that you should probably not be trusting the company asking for these permissions.

Call it out!

I don't mean to pick on Forestry. 100s of other github (and gitlab?) integrations have the same issues. Forestry was just the latest one I looked at. I've seen various companies have this issue for years now and I've been seriously surprised this hasn't been a bigger topic.


Don't clutter the UX with meaningless info

Look above at the Github permissions. Reading public info should not even be listed! It's already obvious that all your public info can be read by the app. That's the definition of public! There's no reason to tell me it might read it. It doesn't need permission to do so.

Comments

Comparing Code Styles

2020-07-03

In a few projects in the past I made these functions

function createElem(tag, attrs = {}) { 
  const elem = document.createElement(tag);
  for (const [key, value] of Object.entries(attrs)) {
    if (typeof value === 'object') {
      for (const [k, v] of Object.entries(value)) {
        elem[key][k] = v;
      }
    } else if (elem[key] === undefined) {
      elem.setAttribute(key, value);
    } else {
      elem[key] = value;
    }
  }
  return elem;
}

function addElem(tag, parent, attrs = {}) {
  const elem = createElem(tag, attrs);
  parent.appendChild(elem);
  return elem;
}

It let's you create an element and fill the various parts of it relatively tersely.

For example

const form = addElem('form', document.body);

const checkbox = addElem('input', form, {
  type: 'checkbox',
  id: 'debug',
  className: 'bullseye',
});

const label = addElem('label', form, {
  for: 'debug',
  textContent: 'debugging on',
  style: {
    background: 'red';
  },
});

With the built in browser API this would be

const form = document.createElement('form');
document.body.appendChild(form);

const checkbox = document.createElement('input');
form.appendChild(checkbox);
checkbox.type = 'checkbox';
checkbox.id = 'debug';
checkbox.className = 'bullseye';

const label = document.createElement('label');
form.appendChild(label);
form.for = 'debug';
form.textContent = 'debugging on';
form.style.background = 'red';

Recently I saw someone post they use a function more like this

function addElem(tag, attrs = {}, children  = []) {
  const elem = createElem(tag, attrs);
  for (const child of children) {
    elem.appendChild(child);
  }
  return elem;
}

The difference to mine was you pass in the children, not the parent. This suggests a nested style like this

document.body.appendChild(addElem('form', {}, [
  addElem('input', {
    type: 'checkbox',
    id: 'debug',
    className: 'bullseye',
  }),
  addElem('label', {
    for: 'debug',
    textContent: 'debugging on',
    style: {
      background: 'red';
    },
  }),
]);

I tried it out recently when refactoring someone else's code. No idea why I decided to refactor but anyway. Here's the original code

function createTableData(thead, tbody) {
  const row = document.createElement('tr');
  {
    const header = document.createElement('th');
    header.className = "text sortcol";
    header.textContent = "Library";
    row.appendChild(header);
    for(const benchmark of Object.keys(testData)) {
      const header = document.createElement('th');
      header.className = "number sortcol";
      header.textContent = benchmark;
      row.appendChild(header);
    };
  }
  {
    const header = document.createElement('th');
    header.className = "number sortcol sortfirstdesc";
    header.textContent = "Average";
    row.appendChild(header);
    thead.appendChild(row);
    for (let i = 0; i < libraries.length; i++) {
      const row = document.createElement('tr');
      row.id = libraries[i] + '_row';
      {
        const data = document.createElement('td');
        data.style.backgroundColor = colors[i];
        data.style.color = '#ffffff';
        data.style.fontWeight = 'normal';
        data.style.fontFamily = 'Arial Black';
        data.textContent = libraries[i];
        row.appendChild(data);
        for(const benchmark of Object.keys(testData)) {
          const data = document.createElement('td');
          data.id = `${benchmark}_${library_to_id(libraries[i])}_data`;
          data.textContent = "";
          row.appendChild(data);
        };
      }
      {
        const data = document.createElement('td');
        data.id = library_to_id(libraries[i]) + '_ave__data'
        data.textContent = "";
        row.appendChild(data);
        tbody.appendChild(row);
      }
    };
  }
}

While that code is verbose it's relatively easy to follow.

Here's the refactor

function createTableData(thead, tbody) {
  thead.appendChild(addElem('tr', {}, [
    addElem('th', {
      className: "text sortcol",
      textContent: "Library",
    }),
    ...Object.keys(testData).map(benchmark => addElem('th', {
      className: "number sortcol",
      textContent: benchmark,
    })),
    addElem('th', {
      className: "number sortcol sortfirstdesc",
      textContent: "Average",
    }),
  ]));
  for (let i = 0; i < libraries.length; i++) {
    tbody.appendChild(addElem('tr', {
      id: `${libraries[i]}_row`,
    }, [
      addElem('td', {
        style: {
          backgroundColor: colors[i],
          color: '#ffffff',
          fontWeight: 'normal',
          fontFamily: 'Arial Black',
        },
        textContent: libraries[i],
      }),
      ...Object.keys(testData).map(benchmark => addElem('td', {
        id: `${benchmark}_${library_to_id(libraries[i])}_data`,
      })),
      addElem('td', {
        id: `${library_to_id(libraries[i])}_ave__data`,
      }),
    ]));
  };
}

I'm not entirely sure I like it better. What I noticed when I was writing it is I found myself having a hard time keeping track of the opening and closing braces, parenthesis, and square brackets. Effectively it's one giant expression instead of multiple individual statements.

Maybe if it was JSX it might hold the same structure but be more readable? Let's assume we could use JSX here then it would be

function createTableData(thead, tbody) {
  thead.appendChild((
    <tr>
      <th className="text sortcol">Library</th>
      (
        Object.keys(testData).map(benchmark => (
          <th className="number sortcol">{benchmark}</th>
        ))
      )
      <th className="number sortcol sortfirstdesc">Average</th>
   </tr>
  ));
  for (let i = 0; i < libraries.length; i++) {
    tbody.appendChild((
      <tr id={`libraries[i]}_row`}>
        <td style={{
          backgroundColor: colors[i],
          color: '#ffffff',
          fontWeight: 'normal',
          fontFamily: 'Arial Black',
        }}>{libraries[i]}</td>
        Object.keys(testData).map(benchmark => (
           <td id={`${benchmark}_${library_to_id(libraries[i])}_data`} />
        ))
        <td id={`${library_to_id(libraries[i])}_ave__data`} />
      </tr>
    ));
  };
}

I really don't know which I like best. I'm sure I don't like the most verbose raw browser API version. The more terse it gets though the harder it seem to be to read it.

Maybe I just need to come up with a better way to format?

I mostly wrote this post only because after the refactor I wasn't sure I was diggin it but writing all this out I have no ideas on how to fix my reservations. I did feel a little like was solving a puzzle unrelated to the task and hand to generate one giant expression.

Maybe my spidey senses are telling me it will be hard to read or edit later? I mean I do try to break down expressions into smaller parts now more than I did in the past. In the past I might have written something like

const dist = Math.sqrt((x2 - x1) ** 2 + (y2 - y1) ** 2;

but now-a-days I'd be much more likely to write something like

const dx = x2 - x1;
const dy = y2 - y1;
const distSq = dx * dx + dy * dy;
const dist = Math.sqrt(distSq);

Maybe with such a simple equation it's hard to see why I prefer spell it out. Maybe I prefer to spell it out because often I'm writing tutorials. Certainly my younger self thought terseness was "cool" but my older self finds terseness for the sake of terseness to be mis-placed. Readability, understandability, editability, comparability I value over terseness.

hmmm....๐Ÿค”

Comments

OpenGL Trivia

2020-06-10

I am not an OpenGL guru and I'm sure someone who is a guru will protest loudly and rudely in the comments below about a something that's wrong here at some point but ... I effectively wrote an OpenGL ES 2.0 driver for Chrome. During that time I learned a bunch of trivia about OpenGL that I think is probably not common knowledge.

Until OpenGL 3.1 you didn't need to call glGenBuffers, glGenTextures, glGenRenderbuffer, glGenFramebuffers

You still don't need to call them if you're using the compatibility profile.

The spec effectively said that all the glGenXXX functions do is manage numbers for you but it was perfectly fine to make up your own numbers

const id = 123;
glBindBuffer(GL_ARRAY_BUFFER, id);
glBufferData(GL_ARRAY_BUFFER, sizeof(data), data, GL_STATIC_DRAW);

I found this out when running the OpenGL ES 2.0 conformance tests against the implementation in Chrome as they test for it.

Note: I am not suggesting you should not call glGenXXX!. I'm just pointing out the triva that they don't/didn't need to be called.

Texture 0 is the default texture.

You can set it the same as any other texture

glBindTexture(GL_TEXTURE_2D, 0);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA8, 1, 1, 0, GL_RGBA, GL_UNSIGNED_BYTE, oneRedPixel);

Now if you happen to use the default texture it will be red.

I found this out as well when running the OpenGL ES 2.0 conformance tests against the implementation in Chrome as they test for it. It was also a little bit of a disappointment to me that WebGL didn't ship with this feature. I brought it up with the committee when I discovered it but I think people just wanted to ship rather than go back and revisit the spec to make it compatible with OpenGL and OpenGL ES. Especially since this trivia seems not well known and therefore rarely used.

Compiling a shader is not required to fail even if there are errors in your shader.

The spec, at least the ES spec, says that glCompileShader can always return success. The spec only requires that glLinkProgram fail if the shaders are bad.

I found this out as well when running the OpenGL ES 2.0 conformance tests against the implementation in Chrome as they test for it.

This trivia is unlikely to ever matter to you unless you're on some low memory embedded device.

There were no OpenGL conformance tests until 2012-ish

I don't know the actual date but when I was using the OpenGL ES 2.0 conformance tests they were being back ported to OpenGL because there had never been an official set of tests. This is one reason there are so many issues with various OpenGL implementations or at least were in the past. Tests now exist but of course any edge case they miss is almost guaranteed to show inconsistencies across implementations.

This is also a lesson I learned. If you don't have comprehensive conformance tests for your standards, implementations will diverge. Making them comprehensive is hard but if you don't want your standard to devolve into lots of non-standard edge cases then you need to invest the time to make comprehensive conformance tests and do you best to make them easily usable with implementations other than your own. Not just for APIs, file formats are another place comprehensive conformance tests would likely help to keep the non-standard variations at a minimum.

Here are the WebGL2 tests as examples and here are the OpenGL tests. The OpenGL ones were not made public until 2017, 25yrs after OpenGL shipped.

Whether or not fragments get clipped by the viewport is implementation specific

This may or may not be fixed in the spec but it is not fixed in actual implementations. Originally the viewport setting set by glViewport only clipped vertices (and or the triangles they create). but for example, draw a 32x32 size POINTS point say 2 pixels off the edge of the viewport, should the 14 pixels still in the viewport be drawn? NVidia says yes, AMD says no. The OpenGL ES spec says yes, the OpenGL spec says no.

Arguably the answer should be yes otherwise POINTS are entirely useless for any size other than 1.0

POINTS have a max size. That size can be 1.0.

I don't think it's trivia really but it might be. Plenty of projects might use POINTS for particles and they expand the size based on the distance from the camera but it turns out they may never expand or they might be limited to some size like 64x64.

I find this very strange that there is a limit. I can imagine there is/was dedicated hardware to draw points in the past. It's relatively trivial to implemented them yourself using instanced drawing and some trivial math in the vertex shader that has no size limit so I'm surprised that GPUs just don't use that method and not have a size limit.

But whatever, it's how it is. Basically you should not use POINTS if you want consistent behavior.

LINES have a max thickness of 1.0 in core OpenGL

Older OpenGL and therefore the compatibility profile of OpenGL supports lines of various thicknesses although like points above the max thickness was driver/GPU dependant and allowed to be just 1.0. But, in the core spec as of OpenGL 3.0 only 1.0 is allowed period.

The funny thing is the spec still explains how glLineWidth works. It's only buried in the appendix that it doesn't actually work.

E.2.1 Deprecated But Still Supported Features

The following features are deprecated, but still present in the core profile. They may be removed from a future version of OpenGL, and are removed in a forward compatible context implementing the core profile.

  • Wide lines - LineWidth values greater than 1.0 will generate an INVALID_VALUE error.

The point is, except for maybe debugging you probably don't want to use LINES and instead you need to rasterize lines yourself using triangles.

You don't need to setup any attributes or buffers to render.

This comes up from needing to make the smallest repos either to post on stack overflow or to file a bug. Let's assume you're using core OpenGL or OpenGL ES 2.0+ so that you're required to write shaders. Here's the simplest code to test a texture

const GLchar* vsrc = R"(#version 300
void main() {
  gl_Position = vec4(0, 0, 0, 1);
  gl_PointSize = 100.0;
})";

const GLchar* fsrc = R"(#version 300
precision highp float;
uniform sampler2D tex;
out vec4 color;
void main() {
  color = texture(tex, gl_PointCoord);
})";

GLuint prg = someUtilToCompileShadersAndLinkToProgram(vsrc, fsrc);
glUseProgram(prg);

// this block only needed in GL, not GL ES
{
    glEnable(GL_PROGRAM_POINT_SIZE);
    GLuint vertex_array;
    glGenVertexArrays(1, &vertex_array);
    glBindVertexArray(vertex_array);
}

const GLubyte oneRedPixel[] = { 0xFF, 0x00, 0x00, 0xFF };
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA8, 1, 1, 0, GL_RGBA, GL_UNSIGNED_BYTE, oneRedPixel);

glDrawArrays(GL_POINTS, 0, 1);

Note: no attributes, no buffers, and I can test things about textures. If I wanted to try multiple things I can just change the vertex shader to

const GLchar* vsrc = R"(#version 300
layout(location = 0) in vec4 position;
void main() {
  gl_Position = position;
  gl_PointSize = 100.0;
})";

And then use glVertexAttrib to change the position. Example

glVertexAttrib2f(0, -0.5, 0);  // draw on left
glDrawArrays(GL_POINTS, 0, 1);
...
glVertexAttrib2f(0,  0.5, 0);  // draw on right
glDrawArrays(GL_POINTS, 0, 1);

Note that even if we used this second shader and didn't call glVertexAttrib we'd get a point in the center of the viewport. See next item.

PS: This may only work in the core profile.

The default attribute value is 0, 0, 0, 1

I see this all the time. Someone declares a position attribute as vec3 and then manually sets w to 1.

in vec3 position;
uniform mat4 matrix;
void main() {
  gl_Position = matrix * vec4(position, 1);
}

The thing is for attributes w defaults to 1.0 so this will work just as well

in vec4 position;
uniform mat4 matrix;
void main() {
  gl_Position = matrix * position;
}

It doesn't matter that you're only supplying x, y, and z from your attributes. w defaults to 1.

Framebuffers are cheap and you should create more of them rather than modify them.

I'm not sure if this is well known or not. It partly falls out from understanding the API.

A framebuffer is a tiny thing that just consists of a collection of references to textures and renderbuffers. Therefore don't be afraid to make more.

Let's say your doing some multipass post processing where you swap inputs and outputs.

texture A as uniform input => pass 1 shader => texture B attached to framebuffer texture B as uniform input => pass 2 shader => texture A attached to framebuffer texture A as uniform input => pass 3 shader => texture B attached to framebuffer texture B as uniform input => pass 4 shader => texture A attached to framebuffer ...

You can implement this in 2 ways

  1. Make one framebuffer, call gl.framebufferTexture2D to set which texture to render to between passes.

  2. Make 2 framebuffers, attach texture A to one and texture B to the other. Bind the other framebuffer between passes.

Method 2 is better. Every time you change the settings inside a framebuffer the driver potentially has to check a bunch of stuff at render time. Don't change anything and nothing has to be checked again.

This arguably includes glDrawBuffers which is also framebuffer state. If you need multiple settings for glDrawBuffers make a different framebuffer with the same attachments but different glDrawBuffers settings.

Arguably this is likely a trivial optimization. The more important point is framebuffers themselves are cheap.

TexImage2D the API leads to interesting complications

Not too many people seem to be aware of the implications of TexImage2D. Consider that in order to function on the GPU your texture must be setup with the correct number of mip levels. You can set how many. It could be 1 mip. It could be a a bunch but they each have to be the correct size and format. Let's say you have a 8x8 texture and you want to do the standard thing (not setting any other texture or sampler parameters). You'll also need a 4x4 mip, a 2x2 mip, an 1x1 mip. You can get those automatically by uploading the level 0 8x8 mip and calling glGenerateMipmap.

Those 4 mip levels need to copied to the GPU, ideally without wasting a lot of memory. But look at the API. There's nothing in that says I can't do this

glTexImage2D(GL_TEXTURE_2D, 0, 8, 8, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixelData8x8);
glTexImage2D(GL_TEXTURE_2D, 1, 20, 40, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixelData40x20);
glTexImage2D(GL_TEXTURE_2D, 2, 10, 20, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixelData20x10);
glTexImage2D(GL_TEXTURE_2D, 3, 5, 10, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixelData10x5);
glTexImage2D(GL_TEXTURE_2D, 4, 2, 5, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixelData5x2);
glTexImage2D(GL_TEXTURE_2D, 5, 1, 2, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixelData2x1);
glTexImage2D(GL_TEXTURE_2D, 6, 1, 1, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixelData1x1);

If it's not clear what that code does a normal mipmap looks like this

but the mip chain above looks like this

Now, the texture above will not render but the code is valid, no errors, and, I can fix it by adding this line at the bottom

glTexImage2D(GL_TEXTURE_2D, 0, 40, 80, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixelData80x40);

I can even do this

glTexImage2D(GL_TEXTURE_2D, 6, 1000, 1000, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixelData1000x1000);
glTexImage2D(GL_TEXTURE_2D, 6, 1, 1, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixelData1x1);

or this

glTexImage2D(GL_TEXTURE_2D, 6, 1000, 1000, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixelData1000x1000);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAX_LEVEL, 5);

Do you see the issue? The API can't actually know anything about what you're trying to do until you actually draw. All the data you send to each mip just has to sit around until you call draw because there's no way for the API to know beforehand what state all the mips will be until you finally decide to draw with that texture. Maybe you supply the last mip first. Maybe you supply different internal formats to every mip and then fix them all later.

Ideally you'd specify the level 0 mip and then it would be an error to specify any other mip that does not match. Same internal format, correct size for the current level 0. That still might not be perfect because on changing level 0 all the mips might be the wrong size or format but it could be that changing level 0 to a different size invalidates all the other mip levels.

This is specifically why TexStorage2D was added but TexImage2D is pretty much ingrained at this point

Comments
older