So, of course I was super excited when the next Zelda game came out. Zelda: Tears of the Kingdom (TotK).
Well… Sadly, so far, it's been a huge disappointment. I'm 60 hours into the game. Was going to wait until I finished because maybe by the end I'll have changed my mind. But then I though, no, I should write what I'm feeling now. Regardless of how well it ends I've got lots of time in the game already and want to record how I felt for 50 or so of the last 60 hours.
Probably the single biggest disappointment is TotK takes place in the same world as BotW. I know lots of fans like that, but you can read how disappointed I was with that in A Link Between Worlds. Lots of people love that about both games. Me, it robbed me of the #1 joy I got out of BotW, that is, discovering new places. Almost everywhere I go in TotK I've already been there. The joy of discovery is removed. I remember playing BotW and climbing a mountain and feeling wonder at seeing the spiral Rist Peninsula. I remember seeing Eventide Island the first time and thing OMG! I can go all the way over to that island!? I remember the joy I felt the first time I crossed The Tabantha Great Bridge and saw how deep the canyon was. I remember the first time I discovered the massive Forgotten Temple. And 30-50 other just as "wow" and wondrous moments. The first time I saw a dragon. The first time I saw a dragon up close. The first time it wasn't raining near Floria Bridge. The first time I saw Skull Lake. The first time I saw Lomei Labyrinth Island. The first time I saw Leviathan Bones. And on and on.
All of that joy is missing from TotK because I've already been to all of these places. There's a few new places in the sky, but so far, none of them have been impressive.
In TotK you can build things. They took the Magnesis power from BotW and added that when you move one item next to another you can pick "Attach" and they'll get glued together. The difference then is, in BotW you'd be near water that you want to cross, you'd have to go find a raft. In TotK, you instead have to go find parts. For example, find 3 logs or cut down 3 trees for 3 logs, then you can glue the logs together, now you have a raft. It takes a couple of minutes to build the raft. This makes TotK more tedious than BotW. I didn't really want to build the raft, I just wanted to cross the river. Being able to build things is a great idea but it's also unfortunately a chore. Maybe I'm not being creative enough but mostly it's pretty obvious what to build and how to build it.
In BotW, after Link gets off the plateau, it's suggested he should go east. The enemies and things encountered in that direction are designed for the beginning player. Of course the player is free to go anywhere, but if they go in the order suggested they'll likely get a better experience as enemies will be weaker, shrines will have stuff that trains them. Etc...
In TotK, unless I missed it, no such direction happened. I ended up going to Rito Village first because that was what some character in the game suggested. 30-40hrs in, I was going east from the center of Hyrule, and it's clear the designers wished I'd gone that way first as the training shrines are all there. Training you to use arrows, training you to parry, training you to throw weapons, etc… It's possible I missed the hint but it feels like there was no guidance suggesting I go that direction first.
I cleared 3 bosses (Rito, Goron, Zora) with no pants. Why? Because I never found any source of money in the first 20-30 hours of play so all I could afford was cold armor (500) and cold headgear (750). Pants cost (1000). Later I needed flame guard armor and had to use all my money to buy just the top. I didn't have enough money to buy pants, nor did I run into any source of money that far into the game.
Here's my character, 60 hours into the game!
Another wall came up when I went to my 4th boss (Gerudo). The 2nd phase of the boss way too hard. I quit the boss, went and made 25 meals for hearts, and even then I could tell there was no way I was going to beat it when 12 or so fast moving Gibdos each take all of my hit points with 1 hit.
After dying too many times I finally gave in and checked online, the first time I'd done so. According to what I read, my armor isn't up to the fight. Now I've spent 30+ hours trying to upgrade my armor but it's a super slog. I need to unlock all of the fairies. Each one requires it's own side quest or 2. Once I've unlocked them I have to go item hunting which will be another 10+ hours. I actually have money now (~3000) and lots of gems but I know no where to buy good armor. I found the fashion armor. I got some sticky armor. But I have yet to get any of my armor upgraded more than 2 points past what it was 30 hours ago.
This is my collection of weapons 60hrs in!
This is my collection of shields.
Where are the weapons!?!?!?!? The shield collections looks ok for 60hrs but the weapons do not. Where are they?
I complained about the fighting in BotW. I found it not as fun as previous Zelda games. Fighting in TotK hasn't changed so that's the same. I get that I suck at it because I can watch videos of people who don't. But, for whatever reason, unlike every other Zelda, I've never gotten the hang of fighting in either BotW nor TotK. As such, I avoid fights as much as possible because basically the odds of me dying are around 1 out of 3. Especially if the enemy is a Lizalfos. They run fast, they take my weapon and/or shield leaving me defenseless.
Taking on a single enemy is something I can often handle but taking on 3 or more I'm more often than not going to die.
I complained about this in BotW as well. I wish there was a combat trainer in some village near the beginning of the game. He'd ask if you want to be trained and you could pick yes or no. That way, people who hated the mandatory training from previous Zelda games could skip it, but people like me, who want to train in a place where you don't lose any hit points and never die, would have a place to learn how to actually fight.
In BotW I basically avoided as many fights as I could and skipped all the shrines with medium or hard tests of combat until after I'd finished the game. In TotK it's been similar. I'm avoiding fights for the most part.
Surprisingly, in both games, the bosses (well most of them), were easy or about the same level as previous Zelda games so it's super surprising that combat from random monsters in the world is so friggen difficult.
The world of TotK is not as interesting as BotW. Yes, it's the same map but things have changed.
In BotW there were signs all over the world of ancient times, ruins, fields of dead guardians, it felt epic. In TotK the world is covered with rubble from some sky people's world falling down. For whatever reason, I'm not finding the TotK world compelling.
In BotW I'd come across a field of broken guardians next to a large thick stone wall. It was clearly the site of an epic battle. Stuff all over BotW's world suggests the place has history. Nothing in the world of TotK has made me wonder anything at all. The idea of a Luputa like civilization in the sky is interesting but nothing about the world presented in the sky in TotK suggests anything interesting actually happened there. Instead it's all just stuff designed around gameplay, not around what a civilization in the sky might be like.
It was a mistake to use the same world as BotW in that there's no consistency. Of course Zelda games have never been consistent but also, except for "A Link Between Worlds" (which I was also disappointed with), no Zelda game has had anything to do with any other Zelda game.
TotK though, because it's in the same world and because that world is so detailed, it arguably needs more consistency. All of this talk of a world in the sky that's always been there and is the source of the clean water in Zora's Domain, etc does match BotW. The fact that all the old shrines are gone but have magically been replaced by knew ones yet Kakariko Village and Hateno Village are basically unchanged makes no sense. Of course, going from the first principle (no Zelda's share anything) it doesn't matter. But, the fact that this Zelda is the same world, Zelda even references Link saving Hyrule previously, means that all those inconsistencies are highlighted. If they'd just made a new world that would disappear.
First off, what do these 6 pictures have in common?
Now look at this
During my first 60hrs, I saw the red gloom covered pits, always from a distance, always from ground level. I thought I was supposed to avoid them! Especially because I thought they were the home of these
Those gloom hands are super scary. The screen changes color, the music gets super tense. As soon as I ran into one I beamed out! So, I avoided these gloom covered holes for fear gloom hands would come out.
Some characters seemed to suggest I should check out some "chasms" and so I kept wondering when I'd run into a chasm knowing that a chasm looks like those 6 examples above, not a pit/crater/hole. In fact there are at last 4 chasms in BotW. Tanagar Canyon, Gerudo Canyon, Karusa Valley, Tempest Gulch. All of those are chasms.
At the 60hr mark, I finally decided to check online, where could I find weapons. The first post I found said, inside the "Hyrule Field Chasm" and marked it on the map. I'm like WTF? There's a chasm there? I go look and find it just one of these pits, not a chasm. So yea, because of poor localization or because the translator didn't bother to look up what a chasm is, the "chasms" are mis-named. 🤬
I was kind pissed off I'd missed this for 60hrs (though I had been in the one from the Goron boss, which to be honest was the only "wow" moment for me in the game so far). I was wondering when I'd find other entrances, especially since someone gave me a map marking some spot in the dark far west from Death Mountain. Now I knew.
On the other hand, I was excited, hoping this was where I'd find the things I'd been missing. Namely, discovering interesting places that filled me with wonder.
Well ... after 10hrs of exploring, no, the dark world doesn't provide what I was missing. In fact, it's super boring!
I literally spent 6-7 hours just trying to find anything interesting, going from lightroot to lightroot. This is what I opened
That entire area had nothing. 5 or so hours in I saw on the map there appeared to be something of interest at the far north but I couldn't find a way to access it. I tried diving into the pit under Hyrule Castle but I didn't find a way to the stuff on the map, even though it marked me as just north of it. I eventually gave up on that. I eventually found some stairs with flames and was hoping it was a temple or dungeon. No, it was just a place to use "Ascend" and deposited me on a tower at the Bridge of Hylia.
At the 6-7 hour point I finally found "Autobuild" and thought maybe that would open something new. Nope. The characters that gave it to me pointed some direction that led to some mine carts. I explored them but found nothing. I spent another couple of hours opening more lightroots and still nothing.
This includes an hour or so of "grinding" since I ran out of arrows and all of my bows broke from shooting giant brightbloom seeds. I know Zelda has always had some amount of grind but it feels worse in TotK, probably because I'm not enjoying the game. First I needed to go get money, then I needed to buy arrows, then I need to find bows. So yea, about an hour.
The dark hasn't saved TotK for me, in fact it's had the opposite effect. I like it even less given how boring the dark has been. It's like some bad filler content.
Zelda games have never had a ton of story. They're all about the game play. But, TotK has the worst so far. Let me put that another way, TotK has an interesting story premise. It's just that individual parts make no sense.
In one scene, Ganondorf appears and magically stabs someone in the back. The fact that he could do that invalidates all his other actions and the rest of the story. If he can just magically kill anyone then he should have killed Zelda and the King and everyone who stands in his way.
The scene where the Queen says Zelda is hiding that she wants to help is some of the most silly childish writing ever.
The scene where Ganondorf appears before King Raura, Queen Sonia and Zelda pledging allegiance, doesn't seem like it makes any sense, Zelda is from the future and knows who Ganondorf is, so her reaction to seeing him (not sure she trusts him), makes no sense. She knows exactly who he is.
Things I like about TotK.
Recipes
I still find cooking tedious. I don't hate it. But it is annoying to have to take 5-10 minutes cooking for a big battle. In BotW you had to just memoize the recipes. In TotK it memorizes them for you.
Unfortunately I found them mostly useless. For whatever reason, I remember rarely not having the ingredients to make good recipes in BotW. In TotK it feels like I rarely have the right ingredients. I click on some ingredient and look at what recipes it can be used in and I never have everything needed. Maybe the fact that I have recipes memorized for me just ends up pointing out what I don't have more than BotW. I don't know what changed, all I know is I have very few good meals.
Ascend
One of my favorite features of BotW is how open the world is because you can climb almost anything. It was a big departure from most other games where you're stuck behind various barriers.
That said, it was often a little slow to climb a large mountain.
In TotK you get the "Ascend" power, which lets you swim vertically through things. With this ability, and the fact that the world is littered with parts that fell from the sky, many mountains that were slow to climb now have areas where you get can under one of these fallen sky parts and beam yourself through them. This makes climbing things slightly less slow and a little more fun.
Of course it also opens up a bunch of interesting puzzles.
Fuze
I'm a little mixed on fuze. I think it's interesting that you can upgrade your weapons, your shields, and your arrows, in like 50+ ways each. Put a flame emitter on your shield and it emits flames anytime you put up your shield. Put a lightning emitter on the end of a staff and you can electrify anything it touches. Put a rock on the end of a log and you have something that can crush rocks.
On the other hand, with arrows, you no longer have a supply of types of arrows. In previous Zelda games, you'd have, separately, normal arrows, fire arrows, freeze arrows, bomb arrows, electric arrows. So you could collect say, 50 fire arrows and shoot them fast.
In TotK you only have regular arrows and on each shot you can pause the game and for this one single shot, fuze something to the arrow. Fuze a flame fruit and you get a fire arrow.
The variety is good. I feel like I'm almost never out of fire arrows. Plus you can fuze other things like Keese Eyes that give you homing arrows. On the other hand, shooting arrows is now tedious and interrupts the combat because every single shot you pause the game and pick from a list of 200+ items.
Vehicles
IIRC, BotW didn't have many vehicles. There was a bonus one from the DLC but otherwise I don't remember any. I guess the Goron mine carts?
In TotK there are a few. Floating platforms, gliders, wheeled platforms, and more. They've been both interesting and so far kind of feel like an after thought.
I like that they exist but, 60 hours in, there's only been a few places where they were actually needed.
I've thought about quitting and not finishing TotK. That's a first for me in a Zelda game. Again, it's my favorite video game series. The only amiibo I own is a BotW guardian.
I have Zelda fan art posters on my walls
I even have Zelda key chains
and Zelda coasters
In other words, I'm a huge Zelda fan, not a hater. It's really disappointing to find I'm not enjoying TotK as much as I had hoped.
At 70hrs, which is probably the 3rd most I've played any game ever (BotW being #1), I think I'm done. I want to see the end but I'm sick of just grinding, trying to find armor so I can survive a boss fight. I can go dive in some other pit but if it's just more grinding from lightroot to lightroot what's the point?
Thoughts after finishing 15 days after I wrote the stuff above.
According to my profile I "Played for 105 hours or more" so that's 35 hours more than when I wrote my thoughts above. Those 35 hours felt like another 70 and I'm actually surprised it claims only 105 hours given it's been two weeks but whatever 🤷♂️
So, apparently I didn't need to check the "chasms". Some time after I got the Master Sword, a character told me to follow him down one of the "chasms" and that led to the things you're supposed to do down there. In other words, the 10 hours I spent trying to find anything down there were mostly pointless and my experience would have been better if I'd not looked online and not followed the advice to go into into a "chasm".
Still, I did feel like the dark world is mostly filler. Unlike the world above which has snow areas, mountain areas, forests, jungles, beaches, clifts, deserts, etc... The dark world is pretty much the same all over. Once the characters told me what to do down there it wasn't nearly as tedious although I'd already lit up many of the places they directed me to go.
Mineru's addition seemed wasted, or else I didn't figure out how to use it. For something so late in the game with so much flexibility, it seemed like it might add lots of new and interesting gameplay, but in the end I mostly ignored it. I'll have to go online to see what I missed I guess.
While I had lots of issues with the details in the story and how much of it didn't make any sense, including Ganondorf's last act, I did end up enjoying Zelda's arc. That part was good.
I still found it too hard. I spent I think literally a week or more trying to beat Ganondorf in the last boss fight.
First, after a few tries, it was clear to me I didn't have enough of the right meals to survive so I beamed out. That means you have to start the entire sequence over, fight your way into the boss area, go through 5 waves of Ganondorf summoning swarms of enemies, before you can get back to the main fight. The whole thing felt so tedious to me, spending several hours getting the right ingredients to make the types of meals needed to survive and getting the gloom resistant armor and upgrading it. I only managed to upgrade it once per piece as looking at the requirements for twice would have easily required another 3-5 hours of nothing but battles with giant gloom monsters in the dark 🙄
One you're actually fighting Ganondorf you're required to Flurry Rush him which you can only do after you execute a Perfect Parry or Perfect Dodge. Again I'm going to complain that I wish there was a place you could choose to train that was like older Zelda games where some teacher would tell you exactly when to do the move and not let you out until you'd done it several times but at the same time actually let you practice quickly.
As it was, I had to learn by fighting Ganondorf 60+ times and it felt like ass to wait for the death screen, wait for the reload, etc. After a few times I'd get frustrated, feel like throwing my controller through my TV, and so quit the game and wait a few hours or the next day to try again. Worse, in Ganondorf's 3rd phase, you have to Perfect Parry/Dodge twice in a row and I could rarely do it.
In the final battle where I beat him, I made it through the first two phases without taking a single hit. In other words, I'd learned to correctly Perfect Dodge. But, on the 3rd phase it was still super frustrating I couldn't do it in this phase and he'd hit me 4 out of 5 times and only 1 out of 5 would be able to do the double Perfect Dodge. Even a single Perfect Dodge was hard. The point being I needed a place to train so that this battle felt good. I never felt like I was doing it wrong since I was doing it exactly the same as the previous two phases. Rather, I felt like the game wasn't making it clear what I was suppose to be doing. When I managed to pull off a Perfect Dodge it just felt like luck as to me it felt like I was pushing the buttons at the same time every time.
Once I'd beaten the game I went back in to check a few things I still had marked on the map. I checked out a couple of sky places I'd never been to and for one, the only way I could see to get to the top was to build a flying machine.
Watching some videos it's clear I missed quite a few interesting things I could maybe have built? On the other hand, many of them are things that don't interest me. I had this same issue in BotW. There wasn't building but there was physics in BotW and watching videos of creative ways I could attack groups of outdoor enemies using these techniques was interesting. The thing is, I didn't want to fight the enemies, I wanted to "continue the adventure" so taking the time to setup some special way of attacking enemies just felt like a waste of time. I'm not saying others shouldn't enjoy that activity. Only that I didn't enjoy it. My goal wasn't to fight as many enemies as possible, it was to go to the next goal, discover the next interesting place, advance the story. Except for bosses and enemies in dungeons, the outdoor enemies are just things in the way of what I actually want to do.
There's some crazy contraptions people built in that video above. It's just that building those contraptions doesn't advance me toward completing the game.
It's hard for me to say what I'd feel if I'd never played BotW and only played TotK. I still feel like BotW is a better game even though in way TotK is all of BotW plus more.
I think the issue for me is, BotW was all about discovering the various areas of Hyrule. For me, discovering each area was 60-70% of the joy I got. If I'd never played BotW, maybe I would have enjoyed TotK more, but, the game feels designed for people that played BotW. I feel like BotW was designed to get you to explore the world, by which I mean, based on what the characters you meet tell you, you end up wanting to go to each place. In TotK I feel like that's less true. It's hard to say if that feeling is real or it's only because I've been to all these places already in BotW.
Partly it's that TotK is 1.8x larger than BotW so if they'd directed you to explore all of the BotW parts the game would be way too long. Instead they mostly just direct you to visit some parts plus much of the new stuff and leave the rest as random playground.
In any case, BotW is still my favorite Zelda and TotK, while it had a few great highlights, is much further down the list if was to rank every Zelda.
Here's hoping the next one is an entirely new world.
]]>In the movie Bunny Watson (Katherine Hepburn) runs the research department of some corporation. Other divisions call the research department anytime they need info like "Give me 5 interesting facts about Kenya?". Or "Who is the head of India and what is their title?". The research department gives you the answers or if they don't know, they will go research them and get back to you.
Richard Sumner (Spencer Tracy) is a computer engineer who's been hired to install a computer that will give these types of answers. Once they turn it on it works exactly like ChatGPT. You ask it a question in English and it gives you a few sentences worth of answer.
That pretty much the exact experience of using ChatGPT was in this 66+ year old move and that I happened to watch it in December 2022 around the time ChatGPT came out was a really interesting coincidence.
I'm a fan of several Katherine Hepburn, Spencer Tracy movies but I can't fully recommend this one. Still, others loved it more than me so if you want to watch it it's available on Amazon.
Also, There's a few famous, good, relatively recent AI movies, "Her", "Ex Machina". Some people consider The Matrix an AI movie though it's arguably more fantasy. The base of the Terminator movies is about AI trying to kill humans.
But, if you've never seen it, my favorite AI movie of all time is still
I was going to post a link to the trailer but IMO all of the trailers have too many spoilers. Not that you can't imagine what will happen based solely on the premise. The United States government designs the most powerful computer ever, to run the nation's defenses. What could possibly go wrong? Still, there are a few twists the trailer spoils so avoid it and JUST WATCH THE MOVIE!. It's only 1hr and 40 minutes and it's pretty taut all the way though.
Sadly I don't know where to recommend watching it. It's not on Amazon Prime.
ATM it is here on Vimeo and here on the Archive. I have to believe both are illegal uploads since it's a movie owned by Universal but apparently ATM there is no other way to watch it except to buy a DVD/Blu-Ray and a player. (I don't even own one anymore)
Some of my favorite images from any movie ever are of the size of the computer in the opening scenes.
Note: If you're the type of person that will laugh at the 1970s computers and because of that ridicule the movie then you're missing out.. For me, I grew up in the 70s and learned on computer with slow 30 characters per second (300 baud) text only displays and teletype terminals. The fact that the computers don't look like modern computers doesn't detract from the movie in any way IMO but if you lack the imagination to go there then it's not for you.
]]>At some point in traveling I'd rent a car from Hertz. I'd get to the airport, ride the Hertz shuttle to the Hertz location. There, they'd announce something to the effect of
If you're a gold member look up your car on the dash board and then go straight to your car.
The rest of us had to go into the Hertz office and stand in line at the counter for 10-30 minutes depending on how crowded the counter is.
I ended up joining the gold club which is free.
Since then I've even been in locations where they say
Just pick any car you want in this section
This all got me wondering though, what is the benefit to Hertz of making people go to the counter? As far as I can tell they are just bleeding money. They could have nearly everyone just walk to their car, saving them on having to have 3 to 20 people at the counters and all the counter equipment.
Signing up for a gold membership, if I recall, is just about setting your preferences and verifying your data. There's every incentive for them to do this for all customers before they get to the car rental area. Even if those customer don't sign up for the "gold club" it would still save Hertz a bunch of work.
Note that even after you pick up the car, you drive to the exit and there are gates with employees who check your license and hand you your rental agreement. It usually takes 30 to 60 seconds. Which begs the question, why does it take so damn long at the counter? There, it feels like it's around 10 minutes per customer and the agent at the counter is typing constantly. What are they typing? This is true, even if you have a reservation which means you've already registered all the data they need!
It's seriously ridiculous. Even the airlines at least have automated agents where you type in your name, scan your id, it prints your boarding pass. You drop off your luggage. Done!
Hertz has to be throwing away 100s of millions of dollars a year by not doing both things above. (1) just letting you walk straight to your car (2) automating as much as possible.
But who knows. Maybe a Hertz employee can explain why it's possible to do it, as evidenced by the Gold Club, and why they don't just do it for everyone and save so much time and money.
]]>It seems pretty clear something like this had to be developed later than naming standards but its at least interesting to imagine what if it came first? Would we even need naming standards?
In the talk they point out that most editors use a complex set of regular expressions to guess how to color things.
Here's a pretty typical example:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | class Foo { public: static Foo* create(int bar); int getBar(); private: explicit Foo(int bar); int bar; }; Foo::Foo(int bar) : bar(bar) {}; Foo* Foo::create(int bar) { return new Foo(bar); } int Foo::getBar() { return bar; } |
What to notice:
Foo
is only green in 2 places, 7 others are not colorized.bar
. bar
as an argument to create
and Foo::Foo
and
bar
as a member of an instance of Foo
. This is because most of the colorizers have no actual knowledge of the language. They just have a list of known language keywords and some regular expressions to guess at what is a type, a function, a string, a comment.
What if the colorizer actually understood the language?
For example:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | class Foo { public: static Foo* create(int bar); int getBar(); private: explicit Foo(int bar); int bar; }; Foo::Foo(int bar) : bar(bar) {}; Foo* Foo::create(int bar) { return new Foo(bar); } int Foo::getBar() { return bar; } |
What to notice:
Every type is green, every function is yellow, every bar
is
red when it's a member of a class. This means we don't need to name it _bar
or
mBar
or bar_
as many style guides would suggest because the editor knows
what it is and shows us by color.
We could also distinguish between member functions and global functions
void Foo::someMethod() { doThis(); // is this a member function or a global function? doThat(); // is this a member function or a global function? }
Some of these issues go away by language design. In Python and JavaScript a
member function and a property both have to be accessed by self
/ this
so
yes, there are other solutions than just coloring and naming conventions to help
make code more understandable at a glance.
I haven't used tree-sitter directly (apparently it's used on Github for colorization though). I just found the idea that a language parsing colorizer could help make code more readable and help distinguish between things that naming conventions are often used for. I get that color isn't everywhere so it's maybe not a solution but it's still fun to think about what other ways we could make it easier to grok the code.
PS: The coloring above is hand-written and not via tree-sitter.
]]>ALL_UPPER_CASE_SNAKE_CASE_IS_A_CONSTANT CaptializedCamelCaseAreClassNames lower_case_snake_case_is_a_variable_or_member lowerCaseCamelCaseIsAVariableOrMember
We often bake these into coding standards. Google's, Mozilla's, Apple's, Microsoft's
Being a person that grew up with a native language that has the concept of UPPER/lower case some of this seems normal but several languages, off the top of my head (Japanese, Chinese, Korean, Arabic, Hindi) have no UPPER/lower case. If programming had been invented or made popular by people whose native language was one of those would we even have the concept of camelCase?
In any case, the fact that we have these styles often leads to extra work.
For Example, I'm implementing OpenGL which has constants like GL_MAX_TEXTURE_SIZE
.
In the code, our style guide uses mCamelCase
for class members so we have
something like this
struct Limits { unsigned mMaxTextureSize; ...
There's then code that is effectively
void glGetIntegerv(GLenum pname, GLint* data) { const Limits& limits = getCurrentContext()->getLimits(); switch (pname) { case GL_MAX_TEXTURE_SIZE: *data = limits.mMaxTextureSize; return; ...
Notice all this busy work. Someone had to translate GL_MAX_TEXTURE_SIZE
into mMaxTextureSize
.
We could have instead done this
struct Limits { unsigned GL_MAX_TEXTURE_SIZE; ... }; void glGetIntegerv(GLenum pname, GLint* data) { const Limits& limits = getCurrentContext()->getLimits(); switch (pname) { case GL_MAX_TEXTURE_SIZE: *data = limits.GL_MAX_TEXTURE_SIZE; return; ...
In this second case, seaching for GL_MAX_TEXTURE_SIZE
will find all references to the concept/limit
we're interested in. In the previous case things are separated and we either have to search for
each individually or we have write some more complex query. Further, we need to be aware
of the coding style. Maybe in a different code base it's like this
struct Limits { unsigned max_texture_size; ... }; void glGetIntegerv(GLenum pname, GLint* data) { const Limits& limits = getCurrentContext()->getLimits(); switch (pname) { case GL_MAX_TEXTURE_SIZE: *data = limits.max_texture_size; return; ...
In fact I've worked in projects that are made from multiple parts where different parts have their own coding standards and yet more work is require to translate between the different choices.
The point is, time is spent translating to <-> from one form GL_MAX_TEXTURE_SIZE
to another maxTextureSize
.
You can automatic this process. Maybe you auto generate struct Limits
above but you still ended up having to write code
to do the translation from one form to another, you still have the search problem, and you still need to know which to reference.
It just had me wondering, how many man years of work would be saved if we didn't have this translation step which arguably only exists because of man made style guides, arguably influenced by the fact that western languages have the concept of letter case? I suspect, over all of the millions of programmers in the world, it's 100s or 1000s of man years of work per year possibly wasted just because of the effort of converting these forms.
]]>Lately I thought, I wonder what it would be like to try to make an HTML library that followed a similar style of API.
NOTE: This is not Dear ImGUI running in JavaScript. For that see this repo. The difference is most ImGUI libraries render their own text and graphics. More specifically they generate arrays of vertex positions, texture coordinates, and vertex colors for the glyphs and other lines and rectangles for your UI. You draw each array of vertices using whatever method you feel like. (WebGL, OpenGL, Vulkan, DirectX, Unreal, Unity, etc...)
This experiment is instead actually using HTML elements like <div>
<input type="text">
, <input type="range">
, <button>
etc...
This has pluses and minus.
The minus is it's likely not as fast as Dear ImGUI (or other ImGUI) libraries, especially if you've got a complex UI that updates at 60fps?
On the other hand it might actually be faster for many use cases. See below
The pluses are
If you don't update anything in the UI it doesn't use any CPU
In Dear ImGUI you must re-render the UI every frame if you non UI stuff (game/app) changes.
Styling is free (use CSS)
Most ImGUI libraries have very minimal styling features but here we get all of CSS to style.
Layout is free (word wrap, sizing, grids, spacing
Most layout happen outside the library based on css. In Dear ImGUI all layout happens every frame by the library. Here, if we want 4 elements side by side we just surround them with flex or grid and the browser handles layout.
Supports all of Unicode
Most ImGUI libraries only handle a small number of glyphs. They may or may not handle colored emoji 🍎🍐🍇🐯🐻🦁🦁😉🤣 or Japanese(日本語), Korean(한국어), Chinese(汉语). I don't think any handle right to left languages like Arabic(عربي).
Supports more fonts
This might not be fair, I'm sure Dear ImGUI can use more than one font. The thing is it's likely cumbersome, especially multiple sizes in multiple styles where as the browser excels at this.
Supports Language Input
Many ImGUI libraries have issues with language input. Because they are rendering their own text they have 2nd class support for complex input and input method editors.
Supports Accessability
Most ImGUI libraries are not very accessability friendly. Because they just ultimately render pixels there is not much for an accessability feature to look at. With HTML, it's far easier for the browser and or OS to look into the content of the elements.
Automatic zoom support
The browser zooms. In Dear ImGUI you'd have to manually code zooming support?
Automatic HD-DPI support
The browser will render text and most other widgets taking into account the user's device's HD-DPI features.
In general, less code than Dear ImGUI should be executing if you are not updating 1000s of values per frame.
Consider, Dear ImGUI is mostly stateless AFAIK. That means things like word wrapping a string or computing the size of column might need to be done on every render. In ImHUI's case, that's handled by the browser and if the contents of an element has not changed much of that is cached.
So, what have I noticed so far...
What's nice about the ImGUI style of stateless UI is that you don't have to setup event handlers nor really marshall data in and out of UI widgets.
Consider standard JavaScript. If you have an <input type="text">
you probably have
code something like this
const elem = document.createElement('input'); elem.type = 'text'; elem.addEventListener('input', (e) => { someObject.someProperty = elem.value; });
You'd also need someway to update the element if the value changes
// when someObject.someProperty changes elem.value = someObject.someProperty;
You now need some system for tracking when you update someObject.someProperty
.
React makes this sightly easier. It handles the updating. It doesn't handle the getting.
function MyTextInput() { return { <input value={someObject.someProperty} onChange={function(e) { someObject.someProperty = this.value; }> } }
Of course that faux react code above won't work. You need to use state or some other solution
so that react knows to re-render when you change someObject.someProperty
.
function MyTextInput() { const [value, setValue] = useState(someObject.someProperty); return { <input value={value} onChange={function(e) { setValue(this.value); }> } }
So now React will, um, react to the state changing but it won't react
to someObject.someProperty
changing, like say if you selected a different
object. So you have to add more code. The code above also provides no way
to get the data back into someObject.someProperty
so you have to add more
code.
In C++ ImGUI style you'd do one of these
// pass the value in, get the new value out someObject.someProperty = ImGUI::textInput(someObject.someProperty);
or
// pass by reference. updates automatically ImGUI::textInput(someObject.someProperty);
JavaScript doesn't support passing by reference so we can't do the 2nd style, OR, we could pass in some getter/setter pair to let the code change values.
// pass the value in, get the new value out (still works in JS) someObject.someProperty = textInput(someObject.someProperty); // use a getter/setter generator textInput(gs(someObject, 'someProperty'));
Where gs
is defined something like
function gs(obj, propertyName) { return { get() { return obj[propertyName]; }, set(v) { obj[propertyName] = v; }, }; }
In any case, it's decidedly simpler than either vanilla JS or React. There is no other code to get the new value from the UI back to your data storage. It just happens.
I'm not sure how to describe this. Basically I notice all the HTML/CSS features I'm throwing away using ImGUI because I know what HTML elements I'm creating underneath.
Consider the text
function. It just takes a string and adds a row to the current UI
ImGUI::text('this text appears in a row by itself');
There's no way to set a className. There's no way to choose span
instead of div
or
sub
or sup
or h1
etc..
Looking through the ImGUI code I see lots of stateful functions to help this along so (making up an example) one solution is some function which sets which type will be created
ImGUI::TextElement('div') ImGUI::text('this text is put in a div'); ImGUI::text('this text is put in a div too'); ImGUI::TextElement('span') ImGUI::text('this text is put in a span');
The same is true for which class name to use or adding on styles etc. I see those littered throughout the ImGUI examples. As another example
ImGUI::text('this text is on its own line'); ImGUI::text('this text can is not'); ImGui::SameLine();
Is that a plus? A Minus? Should I add more optional parameters to functions
ImHUI.text(msg: string, className?: string, type?: string)
or
ImGUI.text(msg: string, attr: Record<string, any>)
where you could do something like
ImGUI.text("hello world", {className: 'glow', style: {color: 'red'}});
I'm not yet sure what's the best direction here.
One thing I've noticed is that, at least with Dear ImGUI, more things are decided for you. Or maybe that's another way of saying Dear ImGUI is a higher level library than React or Vanilla JS/HTML.
As a simple example
ImGUI::sliderFloat("Degrees", myFloatVariable, -360, +360);
Effectively represents 4 separate concepts
<div>
<input type="range">
<div>
So, is ImGUI actually simpler than HTML or is it just the fact that it has higher level components?
In other words, to do that with raw HTML requires creating 4 elements,
childing the first 3 into one of them, responding to input
events,
updating the number display when an input
event arrives. Updating
both the number display and the <input>
element's value if the value
changes externally to the UI widgets.
But, if I had existing higher level UI components that already handled is that enough to make things easier? Meaning how much of Dear ImGUI's ease of use comes from its paradigm and how much from a large library of higher level widgets?
This is kind of like comparing programming languages. For given language, how much of the perceived benefit comes from the language itself and how much from the standard libraries or common environment it runs in.
ImGUI uses C++ ability to pass by reference. JavaScript has no ability to pass by reference. In other words in C++ I can do this
void multBy2(int& v) { v *= 2; } int foo = 123; multBy2(foo); cout << foo; // prints 246
There is no way to do this in JavaScript.
Following the Dear ImGUI API I first tried to work around this by requiring you pass in an getter-setter like this
var foo = 123; var fooGetterSetter = { get() { return foo; } set(v) { foo = v; } };
which you could then use like this
// slider that goes from 0 to 200 ImHUI.sliderFloat("Some Value", fooGetterSetter, 0, 200);
Of course if the point of using one of these libraries is ease of use then it sucks to have to make getter-setters.
I thought maybe I could make getter setter generators like the one gs
shown above.
It means for the easiest usage you're required to use objects so instead of bare foo
you'd do something like
const data = { foo: 123, }; ... // slider that goes from 0 to 200 ImHUI.sliderFloat("Some Value", gs(data, 'foo'), 0, 200);
That has 2 problems though. One is that it can't be type checked because you have
to pass in a string to gs(object: Object, propertyName: string)
.
The other is it's effectively generating a new getter-setter on every invocation. To put it another way, while the easy to type code looks like the line just above, the performant code would require creating a getter-setter at init time like this
const data = { foo: 123, }; const fooGetterSetter = gs(data, 'foo'); ... // slider that goes from 0 to 200 ImHUI.sliderFloat("Some Value", fooGetterSetter, 0, 200);
I could probably make some function that generates getters/setters for all properties but that also sounds yuck as it removes you from your data.
const data = { foo: 123, }; const dataGetterSetters = generateGetterSetters(data) ... // slider that goes from 0 to 200 ImHUI.sliderFloat("Some Value", dataGetterSetter.foo, 0, 200);
Another solution would be to require using an object and then make all the ImHUI functions take an object and a property name as in
// slider that goes from 0 to 200 ImHUI.sliderFloat("Some Value", data, 'foo', 0, 200);
That has the same issue though that because you're passing in a property name by string it's error prone and types can't be checked.
So, at least at the moment, I've ended up changing it so you pass in the value and it passes back a new one
// slider that goes from 0 to 200 foo = ImHUI.sliderFloat("Some Value", foo, 0, 200); // or // slider that goes from 0 to 200 data.foo = ImHUI.sliderFloat("Some Value", data.foo, 0, 200);
It's far more performant than using getter-setters, on top of being more performant than generating getter-setters. Further it's type safe. Eslint or TypeScript can both warn you about non-existing properties and possibly type mis-matches.
The 3rd widget I created was the sliderFloat
which as I pointed out above
consists of 4 elements, a div for the label, a div for the displayed value,
an input[type=range] for the slider, and a container to arrange them.
When I first implemented it I made a class that manages all 4 elements.
But later I realized each of those 4 elements is useful on its own so the
current implementation is just nested ImHUI calls. A sliderFloat
is
function slideFloat(label: string, value: number, min: number = 0, max: number = 1) { beginWrapper('slider-float'); value = sliderFloatNode(value, min, max); text(value.toFixed(2)); text(prompt); endWrapper(); return value; }
The question for me is, what are the smallest building blocks?
For example a draggable window is currently hand coded as a combination of parts. There's the outer div, it's scalable. There's the title bar for the window, it has the text for the title and it's draggable to move the window around. Can I separate those so a window is built from these lower-level parts? That's something to explore.
You can see in the current live example I put in a version of ImGUI::plotLines
which takes a list of values and plots them as a 2D line. The current implementation
creates a 2D canvas using a canvasNode
which returns a Canvas2DRenderingContext
.
In other words, if you want to draw something live you can build your own widget
like this
function circleGraph(zeroToOne: number) { const ctx = canvasNode(); const {width, height} = ctx.canvas; const radius = Math.min(width, height); ctx.beginPath(); ctx.arc(width /2, height / 2, radius, 0, Math.PI * 2 * zeroToOne); ctx.fill(); }
The canvas will be auto-sized to fit its container so you just draw stuff on it.
The thing is, the canvas 2D api is not that fast. At what point should I try to use WebGL or let you use WebGL. If I use WebGL there's the context limit issues. Just something to think about. Given the way ImGUIs work if you have 1000 lines to draw then every time the UI updates you have to draw all 1000 lines. In C++ ImGUI that's just inserting some data into the vertex buffers being generated, but in JavaScript, with Canvas 2D, it's doing a lot more work to call into the Canvas2D API.
It's something to explore.
I have no idea where this is going. I don't have any projects that need a GUI like this at the moment but maybe if I can get it into something I think is kind of stable I'd consider using it over something like dat.gui which is probably far and way the most common UI library for WebGL visualizations.
]]>Notice all the points are missing. I feel an unhealthy influence of points on all sites that have them so I turn them off. I'm convinced someday some scientific research will show they are detrimental to well being and will push to ban them or at least shame them out of existence.
In any case, yea, I spent way to much time answering questions on Stack Overflow. At the time I wrote this I had answered 27% of all the WebGL tagged questions on the site. Including other topics in total over 1900 answers. I also edited the tags of hundreds of wrongly tagged questions.
Many of my answers took hours to write. It could be figuring out a working solution or it could be debugging someone's code. I generally tried to post working code in as many answers as appropriate since in my opinion, working code is almost always better than just an explanation.
As a recent example, someone was trying to glue together two libraries and was running into issues. I got their minimal repo runnable, tracked down the issue, posted a working solution, and filed a bug report on one of the libraries. The entire process took about 2.5 hours.
I've also pointed out before that I wrote webglfundamentals.org and webgl2fundamentals.org in response to questions on stack overflow. WebGL is a verbose API. People ask questions and there is no simple answer. You could just give them some code that happens to work but they likely need 16 chapters of tutorials to understand that code. That's way too much for stack overflow.
So, 9 years ago I started writing articles to explain WebGL. I tried to go out of my way not have them be self promoting. The don't say "WebGL articles by GREGG TAVARES" In fact, except for the copyright license hidden in the code comments, IIRC my name is no where on the website. I'd even be happy to remove my name from the license though I'm not quite sure what legal implications there are. Can I just make something up like "copyright webglfundamentals.org"? I have no idea.
I even moved them from my github account to an organization. The hope was I could find more people to contribute if there was an org so you can participate in the org and not in my personal site. The sites are under "gfxfundametnals" not "greggman". Unfortunately no one has stepped up to write anything, though several volunteers have translated the articles into Chinese, Japanese, Russian, Korean, and other languages.
In any case, once I'd written the articles I would point people to them on Stack Overflow when it seemed appropriate. If, based on the issues they are having, someone is clearly new to WebGL, I might leave an answer that answers their specific question and then also leave a link to the effect of
You might find these articles useful.
If someone else had already written a good answer I might just add the same as a comment under the question.
Similarly if one of the articles addressed their particular issue I might link directly to it. Of course if I was answering I'd always leave a full answer, not just a link. I've been doing this for the least 9 years. It's clearly and unambiguously helpful to the user that asked the question as well as users reading later.
An example of this came up recently. Someone asked a question about how to use mat4
attributes.
Someone else left an okay answer that answered the question, though it didn't give a good
example. But, given the answer was good enough, I added a comment. "You might find
this article useful..."
because the article has a better example.
There were 2 other parts to the comment.
The answer stated something incorrect. They claimed drawing different shapes with instancing is impossible. My comment pointed out it was not impossible and specified how to do it.
That brought up another point which is if you want to draw multiple different models in a single draw call, I'd written an example to do that in a stack overflow answer and so I linked to it.
The next day I went to check if there was a new comment, in particular to see if the answerer had addressed their incorrect "it's impossible" blurb. They had, they'd removed that part of the answer. But, further my comment had been deleted!?!?!
The comment was triple useful. It was useful because it explained how something was possible. It was useful because it linked to a better working example the questioner needed. And, it was useful because it linked to a more flexible solution.
I didn't know this at the time but there is no record of deleted comments. I'd thought maybe I was dreaming. That 2.5 hours I spent on some other answer happened between 4am and 6am. I meant to go to sleep but got sucked into debugging. When I was finished, I checked for more questions, saw this one, and added the comment, but maybe I was too tired and forgot to press "submit"?
So I left the comment again, this time under the question itself since the answer had removed the part about something being impossible. This time I took a screenshot just so I'd know my memory wasn't bad.
I checked back later in the day to find the comment deleted. This prompted me to ask on meta, the stack overflow about stack overflow, what to do about on topic comments being over zealously deleted.
This is when I found out a bunch of things I didn't know
Comments can be deleted by any moderator with for any reason. They don't like you? They can delete all your comments. They hate LGBT people and believe you're LGBT? They can delete your comments. This is one reason why there is no visible comment history.
Comments are apparently meant to be ephemeral.
Several people claimed comments have absolutely zero value. Therefore their deletion is irrelevant.
I found both of these claims rather ludicrous. Comments have a voting system. Some comments get hundreds of vote. Why would anyone design a voting system for something that has zero value?
Links to other stack overflow questions and answers in comments are scanned and used to show related links on the right side bar. If comments have zero value why would anyone make a system to scan them and display their info?
People can even link directly to other comments. What would be the point of implementing the ability to link to something that has zero value?
But further, I found that, according to various members, the links I'd been leaving are considered spam!!!!
According to these people, the links are nothing but self serving self promotion. More than worthless they considered them actively bad and I was a bad person for spamming the site with them. Here I was spending a few hundred hours writing these articles for users of stack overflow to reference when they needed more than would fit in an answer but apparently trying to tell them about these articles was against the rules.
Some claimed, though it was frowned on, it was slightly less shitty spam if I spelled out I wrote the articles when linking to them. There was no guarantee they wouldn't still be deleted, only that it was marginally less shitty if I declared my supposed conflict of interest.
To put it another way, if someone else posted the links it would be more okay because there is no conflict of interest. I don't buy that though. They're basically saying the exact same comment by person A is ok but by person B is not. That's effing stupid. Either the comment is useful to people reading it or it's not. Who posted it is irrelevant.
Well, this is straw that broke the camel's back.
Spending all the time answering people's questions and writing these article to help them was nothing but a burden anyway so I guess I should be thankful Stack Overflow corrected my delusion that I was being helpful and made it clear I was just a self serving spammer.
It's probably for the best anyway. I'll find some more productive way to use my time. To be clear, a bit has flipped in my head. My joy or compulsion or whatever it was that made me want to participate on Stack Overflow is gone or curred. Time to move on.
]]>Result: Nearly all existing Unity games on itch.io, simmer.io, github.io, as well as Unity based visualizations on science sites, corporate training sites etc, stopped working for anyone running MacOS 11 using Chrome/Edge (and probably Brave, Vivaldi, etc...?)
It's an interesting issue
That's just a bug and they have since fixed it. Though ... DOH! Did it really take much thought not to write code that failed if not "10"?
Unfortunately there are many years of games and visualizations out there. It's unlikely most of those games will get updated. Further, even though Unity's main fix is to fix the bug in Unity itself, to apply it you'd have to re-compile your game in a newer version of Unity. Not only is that time consuming but it's no improbable your old project is not compatible with current versions of Unity and will require a bunch of work refactoring the code.
Fortunately you can just replace the file with the issue with a patched version of the file. Luckily that solves the issue and doesn't require you to recompile your game. Still, there will be 1000s of games that don't get this update. If we're lucky some of the sites will just do this automatically for their users.
But BTW, users are still uploading new games even today (Feb 2021) that have this bug as they are using older versions of Unity. Maybe some sites could check and warn the user?
userAgent
This MDN article
spells out why you shouldn't be looking at userAgent
. You should instead be doing feature detection.
Unfortunately reality doesn't always meet expectations. Many web APIs have quirks that can not be easily detected. I didn't dig through the Unity code to see if what they were checking for was a "it can't be helped" kind of issue or a "we didn't know we could feature detect this" issue, but do know I personally have run into these kinds of issues and I also know, sometimes I could try to feature detect but it would be a PITA, meaning checking "If Safari, fall back to X" might take 2 lines of code where as checking that whatever browser I'm using actually follows the spec might take 50 lines of code and I'm lazy 😅
userAgent
is going awayOr at least in theory all the browser vendors have suggested they plan to get rid of the userAgent
string or freeze it. Here's Chrome plans.
It's not clear what they are replacing it with is all that much better. It's better in that it needs less parsing? It sounds like it still provides all the same info to hang yourself with and to be tracked.
But, in some ways, it does possibly let Unity off the hook. AFAIK Chrome may decide to change their version string claiming MacOS 10 even on MacOS 11. Safari and Firefox already do this, I'm guessing for similar reasons, too many poorly coded sites broken. You might think Safari and Firefox don't report MacOS 11 because of tracking but if preventing tracking was their goal they wouldn't report the version of the browser in the userAgent, which they do.
userAgent
I recently wanted to write some software to check how many users can use feature X and I wanted to do it by checking which OS and which browser they are on so I can see for example, 70% of users on Safari, Mac can use feature X and 60% of users on Chrome, Android can use the same feature.
That seems like a reasonable thing to want to know so as much as I don't like being
tracked I'm also not sure getting rid of the data available via userAgent
is
the best thing. It doesn't appear that data is going away though, just changing.
I wrote "Unity broke the internet" mostly because Unity's many years old bug, spread over thousands of sites, potentially forced the browsers to work around those sites rather than progress forward. Unfortunately it's not the first time that's happened
Apparently the same thing happened to Firefox and they ended up adjusting their userAgent string. That rabbit hole lead me to this horror fest! 😱
]]>I have a feeling this is like many file formats. They aren't designed, rather the developer just makes it up as they go. If it gets popular other people want to read and/or write them. They either try to reverse engineer the format OR they ask for specs. Even if the developer writes specs they often forget all the assumptions their original program makes. Those are not written down and hence the spec is incomplete. Zip is such a format.
Zip claims its format is documented in a file called APPNOTE.TXT which can be found here.
The short version is, a zip file consists of records, each record starts with some 4 byte marker that generally takes the form
0x50, 0x4B, ??, ??
Where the 0x50, 0x4B are the letters PK
standing for "Phil Katz", the person who made the zip format.
The two ?? are bytes that identify the type of the record. Examples
0x50 0x4b 0x03 0x04 // a local file record 0x50 0x4b 0x01 0x02 // a central directory file record 0x50 0x4b 0x06 0x06 // an end of central directory record
Records do NOT follow any standard pattern. To read or even skip a record you must know its format.
What I mean is there are several other formats that follow some convention like each record id
is followed by the length of the record. So, if you see an id, and you don't understand it, you just
read the length, skip that many bytes (*), and you'll be at the next id. Examples of this type include
most video container formats, jpgs, tiff, photoshop files, wav files, and many others.
(*) some formats require rounding the length up to the nearest multiple of 4 or 16.
Zip does NOT do this. If you see an id and you don't know how that type of record's content is structured there is no way to know how many bytes to skip.
APPNOTE.TXT says the following things
4.1.9 ZIP files MAY be streamed, split into segments (on fixed or on removable media) or "self-extracting". Self-extracting ZIP files MUST include extraction code for a target platform within the ZIP file.
4.3.1 A ZIP file MUST contain an "end of central directory record". A ZIP file containing only an "end of central directory record" is considered an empty ZIP file. Files MAY be added or replaced within a ZIP file, or deleted. A ZIP file MUST have only one "end of central directory record". Other records defined in this specification MAY be used as needed to support storage requirements for individual ZIP files.
4.3.2 Each file placed into a ZIP file MUST be preceded by a "local file header" record for that file. Each "local file header" MUST be accompanied by a corresponding "central directory header" record within the central directory section of the ZIP file.
4.3.3 Files MAY be stored in arbitrary order within a ZIP file. A ZIP file MAY span multiple volumes or it MAY be split into user-defined segment sizes. All values MUST be stored in little-endian byte order unless otherwise specified in this document for a specific data element.
4.3.6 Overall .ZIP file format:
[local file header 1] [encryption header 1] [file data 1] [data descriptor 1] . . . [local file header n] [encryption header n] [file data n] [data descriptor n] [archive decryption header] [archive extra data record] [central directory header 1] . . . [central directory header n] [zip64 end of central directory record] [zip64 end of central directory locator] [end of central directory record]
4.3.7 Local file header:
local file header signature 4 bytes (0x04034b50) version needed to extract 2 bytes general purpose bit flag 2 bytes compression method 2 bytes last mod file time 2 bytes last mod file date 2 bytes crc-32 4 bytes compressed size 4 bytes uncompressed size 4 bytes file name length 2 bytes extra field length 2 bytes file name (variable size) extra field (variable size)
4.3.8 File data
Immediately following the local header for a file SHOULD be placed the compressed or stored data for the file. If the file is encrypted, the encryption header for the file SHOULD be placed after the local header and before the file data. The series of [local file header][encryption header] [file data][data descriptor] repeats for each file in the .ZIP archive.
Zero-byte files, directories, and other file types that contain no content MUST NOT include file data.
4.3.12 Central directory structure:
[central directory header 1] . . . [central directory header n] [digital signature]
File header:
central file header signature 4 bytes (0x02014b50) version made by 2 bytes version needed to extract 2 bytes general purpose bit flag 2 bytes compression method 2 bytes last mod file time 2 bytes last mod file date 2 bytes crc-32 4 bytes compressed size 4 bytes uncompressed size 4 bytes file name length 2 bytes extra field length 2 bytes file comment length 2 bytes disk number start 2 bytes internal file attributes 2 bytes external file attributes 4 bytes relative offset of local header 4 bytes file name (variable size) extra field (variable size) file comment (variable size)
4.3.16 End of central directory record:
end of central dir signature 4 bytes (0x06054b50) number of this disk 2 bytes number of the disk with the start of the central directory 2 bytes total number of entries in the central directory on this disk 2 bytes total number of entries in the central directory 2 bytes size of the central directory 4 bytes offset of start of central directory with respect to the starting disk number 4 bytes .ZIP file comment length 2 bytes .ZIP file comment (variable size)
There are other details involving encryption, larger files, optional data, but for the purposes of this post this is all we need. We need one more piece of info, how to make a self extracting archive.
To do so we could look back to ZIP2EXE.exe
which shipped with pkzip in 1989
and see what it does but it's easier look at Info-Zip to see what happens.
How do I make a DOS (or other non-native) self-extracting archive under Unix?
The procedure is basically described in the UnZipSFX man page. First grab the appropriate UnZip binary distribution for your target platform (DOS, Windows, OS/2, etc.), as described above; we'll assume DOS in the following example. Then extract the UnZipSFX stub from the distribution and prepend as if it were a native Unix stub:
> unzip unz552x3.exe unzipsfx.exe // extract the DOS SFX stub > cat unzipsfx.exe yourzip.zip > yourDOSzip.exe // create the SFX archive > zip -A yourDOSzip.exe // fix up internal offsets >That's it. You can still test, update and delete entries from the archive; it's a fully functional zipfile.
So given all of that let's go over some problems.
This is undefined by the spec.
There are 2 obvious ways.
Scan from the front, when you see an id for a record do the appropriate thing.
Scan from the back, find the end-of-central-directory-record and then use it to read through the central directory, only looking at things the central directory references.
Scanning from the back is how the original pkunzip works. For one it means if you ask for some subset of files it can jump directly to the data you need instead of having to scan the entire zip file. This was especially important if the zip file spanned multiple floppy disks.
But, 4.1.9 says you can stream zip files. How is that possible? What if there is some local file record that is not referenced by the central directory? Is that valid? This is undefined.
4.3.1 states
Files MAY be added or replaced within a ZIP file, or deleted.
Okay? That suggests the central directory might not reference all the files in the zip file because otherwise this statement about files being added, replaced, or delete has no point to be in the spec.
If I have file1.zip
that contains files, A
, B
, C
and I generate file2.zip
that only contains files A
, B
. Those are just 2 independent zip files.
It makes zero sense to put in the spec that you can add, replace, and
delete files unless that knowledge some how affects the format
of a zip file.
In other words. If you have
[local file A] [local file B] [local file C] [central directory file A] [central directory file C] [end of central directory]
Then clearly B
is deleted as the central directory doesn't reference
it. On the other hand, if there's no [local file B]
then you just
have an independent zip file, independent of some other zip file that
has B
in it. No need for the spec to even mention that situation.
Similarly if you had
[local file A (old)] [local file B] [local file C] [local file A (new)] [central directory file A(new)] [central directory file B] [central directory file C] [end of central directory]
Then A (old)
has been replaced by A (new)
according
to the central directory. If on the other hand there is
no [local file A (old)]
you just have an independent zip file.
You might think this is nonsense but you have to remember, pkzip comes from the era of floppy disks. Reading an entire zip file's contents and writing out a brand new zip file could be an extremely slow process. In both cases, the ability to delete a file just by updating the central directory, or to add a file by reading the existing central directory, appending the new data, then writing a new central directory, is a desirable feature. This would be especially true if you had a zip file that spanned multiple floppy disks; something that was common in 1989. You'd like to be able to update a README.TXT in your zip file without having to re-write multiple floppies.
In discussion with PKWare, they state the following
The format was originally intended to be written front-to-back so the central directory and end of central directory record could be written out last after all files included in the ZIP are known and written. If adding files, changes can applied without rewriting the entire file. This was how the original PKZIP application was designed to write .ZIP files. When reading, it will read the ZIP file end of central directory first to locate the central directory and then seek to any files it needs to access
Of course "add" is different than "delete" and "replace".
Whether or not having local files not referenced by the central directory is undefined by the spec. It is only implied by the mention of:
Files MAY be added or replaced within a ZIP file, or deleted.
If it is valid for the central directory to not reference all the local files then reading a zip file by scanning from the front may fail. Without special care you'd get files that aren't supposed to exist or errors from trying to overwrite existing files.
But, that contradicts 4.1.9 that says zip files maybe be
streamed. If zip files can be streamed then both of the example
above would fail because in the first case we'd see file B
and in the second we'd see file A (old)
before we saw that
the central directory doesn't reference them. If you have
to wait for the central directory before you can correctly
use any of the entries then functionally
you can not stream zip files.
Seeing the instructions for how to create a self extracting zip file above, we just prepend some executable code to the front of the file and then fix the offsets in the central directory.
So let's say your self extractor has code like this
switch (id) { case 0x06054b50: read_end_of_central_directory(); break; case 0x04034b50: read_local_file_record(); break; case 0x02014b50: read_center_file_record(); break; ... }
Given the code above, it's likely those values 0x06054b50
, 0x04034b50
, 0x02014b50
will appear in binary
in the self extracting portion of the zip file at the front of the file. If you read a zip file by scanning
from the front your scanner my see those ids and mis-interpret them as a zip records.
In fact you can imagine a self extractor with a zip file in it like this
// data for a zip file that contains // LICENSE.txt // README.txt // player.exe const unsigned char[] runtimeAndLicenseData = { 0x50, 0x4b, 0x03, 0x04, ??, ??, ... }; int main() { extractZipFromFile(getPathToSelf()); extractZipFromMemory(runtimeAndLicenseData, sizeof(runtimeAndLicenseData)); }
Now there's a zip file in the self extractor. Any reader that reads from the front would see this inner zip file and fail. Is that a valid zip file? This is undefined by the spec.
I tested this. The original PKUnzip.exe in DOS, the Windows Explorer, MacOS Finder, Info-Zip (the unzip included in MacOS and Linux), all clearly read from the back and see the files after the self extractor. 7z, Keka, see the embedded zip inside the self extractor.
Is that failure or is that a bad zip file? The APPNOTE.TXT does not say. I think it should be explicit here and I think it's one of those unstated assumptions. PKunzip scans from the back and so this just happens to work but the fact of how it happens to work is never documented. The issue that the data in the self-extractor might happen to resemble a zip file is just glossed over. Similarly streaming will likely fail if it hasn't already from the previous issues.
You might think this is a non issue but their are 100s of thousands of self extracting zip files out there from the 1990s in the archives. A forward scanner might fail to read these.
If you go look at 4.3.16 above you'll see the end of a zip file
is a variable length comment. So, if you're doing backward scanning you
basically read from the back of the file looking for
0x50 0x4B 0x05 0x06
but what if that sequence of bytes is in the
comment?
I'm sure Phil Katz never gave it a second thought. He just assumed people would put the equivalent of a README.txt in there. As such it would only have values from 0x20 to 0x7F with maybe a 0x0D (carriage return), 0x0A (linefeed), 0x09 (tab) and maybe 0x06 (bell).
Unfortunately all of those values in the ids are valid ASCII, even utf-8. We already
went over 0x50 = P
and 0x4B = K
. 0x06 is "Bell" in ASCII (makes a noise or flashes
the screen). 0x05 is "Enquiry".
The APPNOTE.TXT should arguably explicitly specify if this is invalid. Indirectly 4.3.1 says
A ZIP file MUST have only one "end of central directory record"
But what does that mean? Does that mean the bytes
0x50 0x4B 0x05 0x06
can't appear in the comment nor the self extracting
code? Does it mean the first time you see that scanning from the back
you don't try to find a second match?
If you scan from the front and run into none of the issues mentioned before, then a forward scanner would successfully read this. On the other hand, pkunzip itself would fail.
That offset is 0x504b0506
so it will appear to be end central directory header.
I think 1.3gig zip file wasn't even on the radar when zip was created and in
fact extensions were required to handle files larger then 4gig. But, it does
show one more way the format is poorly designed.
There's certainly debate to be had about what a good design would be but somethings are arguably easy to decide if we could start over.
It would have been better if records had a fixed format like id followed by size so that you can skip a record you don't understand.
It would have been better if the last record at the end of the file was just an
offset-to-end-of-central-directory
record as in
0x504b0609 (id: some id is not in use) 0x04000000 (size of data of record) 0x???????? (relative offset to end-of-central-directory)
Then there would be no ambiguity for reading from the back.
0x50 0x4b 0x06 0x09 0x04 0x00 0x00 0x00
. If not, fail.Or, conversely, put the comment in its own record and write it
before the central directory and put an offset to it in
the end-of-central-directory-record
. Then at least this issue
of scanning over the comment would disappear.
Be clear about what data can appear in a self extracting stub.
If you want to support reading from the front it seems required to state that the self extracting portion can't appear to have any records.
This is hard to enforce unless you specifically wrote some validator. If you just check based on whether your own app can read the zip file then, as it stands now, Pkzip, pkunzip, info-zip (the zip in MacOS, Linux), Windows Explorer, and MacOS all don't care what's in the self extracting portion so they aren't useful for validation. You must explicitly state that you must scan from the back in the spec or write a validator that rejects zip that are not forward scanable and state in the spec why.
Be clear if the central directory can disagree with local file records
Be clear if random data can appear between records
A backward scanner does not care what's between records. It only cares it can find the central directory and it only reads what that central directory points to. That means there can be any random data between records (or some at least some records).
Be explicit if this is okay or not okay. Don't rely on implicit diagrams.
If I was to to guess all of these issues are implementation details that didn't make it into the APPNOTE.TXT. What I believe the APPNOTE.TXT really wants to say is "a valid zip file is one that pkzip can manipulate and pkunzip can correctly unzip. Instead it defines things in such a way that various implementations can make files that other implementations can't read.
Of course with 32 years of zip files out their we can't fix the format. What PKWare could do is get specific about these edge cases. If it was me I'd add these sections to the APPNOTE.TXT
4.3.1 A ZIP file MUST contain an "end of central directory record". A ZIP file containing only an "end of central directory record" is considered an empty ZIP file. Files MAY be added or replaced within a ZIP file, or deleted. A ZIP file MUST have only one "end of central directory record". Other records defined in this specification MAY be used as needed to support storage requirements for individual ZIP files.
The "end of central directory record" must be at the end of the file and the sequence of bytes,
0x50 0x4B 0x05 0x06
, must not appear in the comment.The "central directory" is the authority on the contents of the zip file. Only the data it references are valid to read from the file. This is because (1) the contents of the self extracting portion of the file is undefined and might be appear to contain zip records when in fact they are not related to the zip file and (2) the ability to add, update, and delete files in a zip file stems from the fact that it is only the central directory that knows which local files are valid.
That would be one way. I believe this will read the 100s of millions of existing zip files out there.
On the other hand, if PKWare claims such files that have these issues don't exist then this would work as well
4.3.1 A ZIP file MUST contain an "end of central directory record". A ZIP file containing only an "end of central directory record" is considered an empty ZIP file. Files MAY be added or replaced within a ZIP file, or deleted. A ZIP file MUST have only one "end of central directory record". Other records defined in this specification MAY be used as needed to support storage requirements for individual ZIP files.
The "end of central directory record" must be at the end of the file and the sequence of bytes,
0x50 0x4B 0x05 0x06
, must not appear in the comment.There can be no [local file records] that do not appear in the central directory. This guarantee is required so reading a file front to back provides the same results as reading it back to front. Any file that does not follow this rule is an invalid zip file.
A self extracting zip file must not contain any of the sequences of record ids listed in this document as they maybe mis-interpreted by forward scanning zip readers. Any file that does not follow this rule is an invalid zip file.
I hope they will update the APPNOTE.TXT so that the various zip readers and zip creators can agree on what's valid.
Unfortunately I feel like pkware doesn't want to be clear here. Their POV seems to be that zip is an ambiguous format. If you want to read by scanning from the front then just don't try to read files you can't read that way. They're still valid zip files and but the fact that you can't read them is irrelevant. It's just your choice to fail to support those.
I suppose that's a valid POV. Few if any zip libraries handle every feature of zip. Still, it would be nice to know if you're intentionally not handling something or if you're just reading the file wrong and getting lucky that it works sometimes.
The reason all this came up is I wrote a javascript unzip library. There are tons out here but I had special needs the other libraries I found didn't handle. In particular I needed a library that let me read a single file from a large zip as fast as possible. That means backward scanning, finding the offset to the desired file, and just decompressing that single file. Hopefully others find it useful.
You might find this history of Zip fascinating
]]>I've been listening to music via my iPhone for many years playing my collection of mp3s. I guess that dates me as I'm not using Spotify or Apple Music or Youtube Music but, I haven't had any luck using any of those services.
As some examples, Spotify, I picked "Caro Emerald" Radio. I'd classify her as modern Swing
and Spotify played "How Would You Feel" by Kzezip who I'd classify as pop.
Another example I put in Prince Radio
And Spotify played rap. Prince had nothing to do with rap. The list from when I pasted it above is basically "Hits from the 80s" but that's not what I want if pick "Prince Radio". I want music that sounds similar to Prince. Maybe Windy and Lisa, or maybe Rick James? or maybe some bands I never heard of. Checking the list though Spotify will give me Huey Lewis & The News. I have nothing against them, I like their songs, but they aren't similar to Prince.
An example from Youtube Music, I put in "Fuck you till your Groovy" by Jill Jones and pick "Radio"
And it played "All Night Long" by Lionel Richie. WAT!???!
Note: I got the Jill Jones recommendation from doing the same thing on Google Play Music who's radio feature actually seemed to work, or rather actually played music similar to the artist and not just music popular by people who like that artist.
Another Youtube Music Example, I put in "Swingrowers" radio which is an electro swing band.
And youtube played "Bliss on Mushrooms" by Infected Mushroom, an Industrial Band
WTF!!
Anyway, all this means I sadly I keep going back to just my own collection of mp3s because trying any of the other services means hitting "no" or "don't like" for 9 of 10 songs.
All that was really beside the point though. What I wanted to write about was I'd been listening to music via my iPhone for years and recently went through my entire list of ~8500 songs trying to make a playlist and that's when it became clear to me,
In particular I noticed lots of songs I never heard my iPhone play and conversely there were some albums it would seem to play way too often. One example is I have this album called "Pure Sugar" by Pure Sugar. It's house music from 1998
As far I know it was some album I bought at a record store,probably in 1998, because on a short listen it sounded ok and back then buying CD was the only way to add to your music collection. I had over 1100 CDs at the time.
I have no particular affinity for this CD. I wouldn't add any song on the album to a playlist but if you like house music as like background music it's fine. I can listen to the entire album which is better than most.
In any way, out of ~8500 songs my iPhone seemed to pick songs from this album all the friggen time?!?! I never really gave it much thought because I just assumed it was bad luck or one of this weird artifacts of random selection but then, when I was going through the entire list of songs and seeing all the stuff not being played I started to be clear something was broken.
I didn't actually figure out what the problem was. I've never rated any albums or tracks. iTunes/Music apparently auto rates albums. No idea what it does to do that. If it's by the number of times played that would suck because it would re-enforce its bad choices. If it's by looking up on the net other people's opinions that would suck too as I don't want other people choosing music for me from my own collection. I also have no idea if the rating are used to pick random tracks.
If shuffling is related to rating, apparently the solution is to set all the ratings to 1. That way the app will assume you set it and won't auto rate.
I actually have no idea if that works. Instead I switched to using a different music app and suddenly I'm hearing much more of my collection than I was on the built in app.
We'll see how it goes. I have no idea how the new app chooses songs though. I can think of lots of algorithms. The simplest would just be to pick a random track from all tracks.
I'm pretty confident the app isn't doing that because it's also played too many tracks from the same albums. In other words lets say it played a song from "Unbreakable" by Janet Jackson. Within about 10 songs I'd hear another song from the same album, and 10 songs later yet another. I'm not sure what the odds of that are but I think they are low for 8500 tracks. I might guess that it picks a random album and then a random track. Would that be more likely to hear songs from the same albums? Or maybe it picks N albums and then picks random tracks from just those albums trying to make a theme? I have no idea.
I wrote some code to try just picking songs at random and to see how often it picks a song from some album of the last 20 albums.
It just keeps picking songs at random until it's played every
song at least once. Using JavaScript's Math.random()
function,
for ~8500 tracks it would have to play around 80k tracks
until it's played every track once. During that time at least
one track would have been played ~25 times. Also
about one out of 36 tracks will be from the same album
as one of the last 20 albums played. That wouldn't remotely
explain getting 3+ songs from the same album in say 60 songs.
Yes I know random = random but in my own tests that situation
rarely comes up.
Apparently the there's also a difference between "random" and "shuffle". "Shuffle" is supposed to by like a deck of card. You put take all the tracks and "shuffle" them, then play the tracks in the shuffled order. I suppose I should check that.
Well, according to that on average every ~40 tracks I'll get a track from the same album. Maybe that's what I'm experiencing.
It's ridiculous how much of my collection I'm being re-introduced to since I switched players
In any case, within the first 60 tracks on the new app it played 2 songs from "Pure Sugar"!!! 😭😅🤣🤯
]]>Once in a while I want to benchmark solutions in JavaScript just to see how much slower one solution is vs another. I used to use jsperf.com but sometime in early 2020 or 2019 it disappeared.
Searching around I found 2 others, jsbench.me. Trying them they both have their issues. Jsbench.me is ugly. Probably not fair but whatever. Using it bugged me. Jsben.ch, at least as of this writing had 2 issues when I tried to use it. One is that if my code had a bug the site would crash as in it would put up a full window UI that blocks everything with a "running..." message and then never recovers. The result was all the code I typed in was lost and I'd have to start over. The other is it has a 4k limit. 4k might sound like a lot but I ran into that limit trying to test a fairly simple thing. I managed to squeeze my test in with some work but worse, there's no contact info anywhere except a donate button that leads directly to the donation site, not a contact sight so there's no way to even file a bug let alone make a suggestion.
In any case I put up with it for 6 months or so but then one day about a month ago, I don't remember what triggered it but I figured I could make my own site fairly quickly where I'm sure in my head quickly meant 1-3 days max. 😂
So, this is what happened. First I decide I should use benchmark.js mostly because I suck at math and it claims "statistically significant results". I have no idea what that means 😅 but, a glance at the code shows some math happening that's more than I'd do if I just wrote my own timing functions.
Unfortunately I'd argue benchmark.js is actually not a very good library. They made up some username or org name called "bestiejs" to make it sound like it's good and they claim tests and docs but the docs are horrible auto-generated docs. They don't actually cover how to use the library they just list a bunch of classes and methods and it's left to you to figure out which functions to call and when and why. There's also some very questionable design choices like the way you add setup code is by manually patching the prototype of one of their classes. WAT?!?
I thought about writing my own anyway and trying to extract the math parts but eventually I got things working enough and just wanted to move on.
I also didn't want to run a full server with database and everything else so I decided I'd see if it was possible to store the data in a github gist. It turns out yes, it's possible but I also learned there is no way to make a static website that supports Oauth to let the user login to github.
A workaround is a user can make a Personal Access Token which is a fancy way of basically making a special password that is given certain permissions. So, in order to save the user would have to go to github, manually make a personal access token, paste it into jsbenchit.org and then they could save. It worked! 🎉
As I got it working I released I could also make a site similar to jsfiddle or codepen with only minor tweaks to the UI so I started on that too.
¯\(ツ)/¯
Both sites run arbitrary user code and so if I didn't do something people could write code that steals the personal access token. That's no good. Stealing their own token is not an issue but passing a benchmark or jsgist to another user would let them steal that user's token.
The solution is to run the user's code in an iframe on another domain. That domain can't read any of the data from the main site so problem is solved.
Unfortunately I ran into a new problem. Well, maybe it's not so new. The problem is since the servers are static I can't serve the user's code like a normal site would. If you look at jsfiddle, codepen, and stack overflow snippets you'll see they run the code from a server served page generated using the user's code. With a static site I don't have that option.
To work around it I generated a blob, make a URL to the blob and have the browser load that in an iframe. I use this solution on webgfundmentals.org, webgl2fundamentals.org, and threejsfundamentals.org It works but it has other problems. One is since I can't serve any files whatsoever I have to re-write URLs if you use more than 1 file.
Take for example something that uses workers. You usually need a minimum of 2 files. An HTML file
with a <script>
section that launches a worker and the worker's script is in another file.
So you start with main.html
that loads worker.js
but you end up with blob:1234-1314523-1232
for main.html
but it's still referencing worker.js
and you have to some how find that and
change it to the blob url that was generated for worker.js
. I actually implemented this solution on
those sites I mentioned above but it only works because I wrote all the examples that are running
live and the solutions only handle the small number of cases I needed to work.
The second problem with the blob solution they are no good for debugging. Every time the user clicks "run" new blobs are created so any breakpoints you set last time you ran it don't apply to the new blob since they're associated with a URL and that URL has just changed.
Looking into it I found out I could solve both problems with a service worker. The main page starts the service worker then injects the filename/content of each file into the service worker. It then references those files as normal URLs so the don't change. Both problems are solved. 😊
I went on to continue making the sites even though I was way past the amount of time I thought I'd be spending on them.
In using the sites I ran into a new problems. Using a personal access token sucked! I have at least 4 computers I want to run these on. A Windows PC, a Mac, an iPhone, and an Android phone. When I'd try to use a different PC I needed to either go through the process of making a new personal access token, or I needed to find someway to pass that token between machines, like email it to myself. 🤮
I wondered if I could get the browser to save it as a password. It turns out, yes,
if you use a <form>
and an <input type="password">
and you apply the correct
incantation when the user clicks a "submit" button the browser will offer to save
the personal access token as a password.
Problem solved? No 😭
A minor issue is there's no username but the browsers assume it's always username + password. That's not that big a deal, I can supply a fake username though it will probably confuse users.
A major issue though is that passing between machines via a browser's password manager doesn't help pass between different browsers. If I want to test Firefox vs Chrome vs Safari then I was back to the same problem of keeping track of a personal access token somewhere.
Now I was entering sunk cost issues. I'd spent a bunch of time getting this far but the personal access token issues seemed like they'd likely make no one use either site. If no one is going to use it then I've wasted all the time I put in already.
Looking into it more it turns out the amount of "server" need to support oauth so that users could log in with github directly is actually really tiny. No storage is needed, almost nothing.
Basically they way Oauth works is
https://github.com/login/oauth/authorize
and
passes it an app id (called client_id), the permissions the app wants, and
something called "state" which you make up.https://jsbenchit.org/auth.html
. To this page
the redirect includes a "code" and the "state" passed at step 2.The auth.html
page either directly or by communicating with the page
that opened the popup, first verifies that the "state" matches
what was sent at step 2. If not something is fishy, abort.
Otherwise it needs to contact github at a special URL and pass
the "code", the "client_id" and a "client_secret".
Here's the part that needs a server. The page can't send the secret because then anyone could read the secret. So, the secret needs to be on a server. So,
If you were able to follow that the short part is you need a server, and all it has to do is given a "code", contact github, pass the "code", "client_id" and "client_secret" on to github, and pass back the resulting token.
Pretty simple. Once that's done the server is no longer needed. The client will function without contacting that server until and unless the token expires. This means that server can be stateless and basically only takes a few lines of code to run.
A found a couple of solutions. One is called Pizzly. It's overkill for my needs. It's a server that provides the oauth server in step 6 above but it also tracks the tokens for you and proxies all other github requests, or requests to whatever service you're using. So your client side code just gets a pizzly user id which gets translated for you to the correct token.
I'm sure that's a great solution but it would mean paying for a much larger server, having to back up user accounts, keep applying patches as security issues are found. It also means paying for all bandwidth between the browser can github because pizzly is in the middle.
Another repo though made it clear how simple the issue can be solved. It's this github-secret-keeper repo. It runs a few line node server. I ran the free example on heroku and it works! But, I didn't want to make an heroku account. It seemed too expensive for what I needed it for. I also didn't feel like setting up a new dynamo at Digital Ocean and paying $5 a month just to run this simple server that I'd have to maintain.
I ended up making an AWS Lambda function to do this which added another 3 days or so to try to learn enough AWS to get it done.
I want to say the experience was rather poor IMO. Here's the problem. All the examples I found showed lambda doing node.js like stuff, accepting a request, reading the parameters, and returning a response. Some showed the parameters already parsed and the response being structured. Trying that didn't work and it turns out the reason is AWS for this feature is split into 2 parts.
Part 1 is AWS Lambda which just runs functions in node.js or python or Java etc...
Part 2 is AWS API Gateway which provides public facing endpoints (URLS) and routes them to different services on AWS, AWS Lambda being one of those targets.
It turns out the default in AWS API Gateway doesn't match any of the examples I saw. In particular the default in AWS API Gateway is that you setup a ton of rules to parse and validate requests and parameters and only if they parse correctly and pass all the validation do they get forwarded to the next service. But that's not really what the example shown wanted. Instead they wanted AWS API Gateway to effectively just pass through the request. That's not the default and I'd argue it not being the default is a mistake.
My guess is that service was originally written in Java. Because Java is strongly typed it was natural to think in terms of making the request fit strong types. Node.js on the other hand, is loosely typed. It's trivial to take random JSON, look at the data you care about, ignore the rest, and move on with your life.
In any case I finally figured out how to get AWS API Gateway to do what all the AWS Lambda examples I was seeing needed and it started working.
My solution is here if you want to use it for github or any Oauth service.
Next up was spitting and CSS. I still can't claim to be a CSS guru in any way shape or form and several times I year I run into infuriating CSS issues where I thought I'd get something done in 15 minutes but turns into 15 minutes of the work I thought I was going to do and 1 to 4 hours of trying to figure out why my CSS is not working.
I think there are 2 big issues.
is that Safari doesn't match Chrome and Firefox so you get something working only to find it doesn't work on Safari
Nowhere does it seem to be documented how to make children always fill their parents. This is especially important if you're trying to make a page that acts more like an app where the data available should always fit on the screen vs a webpage that be been as tall as all the content.
To be more clear you try (or I try) to make some layout like
+--------------------+ | | +---+------------+---+ | | | | | | | | | | | | +---+-----+------+---+ | | | +---------+----------+
and I want the entire thing to fill the screen and the contents of each area expand to use all of it.
For whatever reason it never "just works". I'd think this would be trivial but something about it
is not or at least not for me. It's always a bunch of sitting in the dev tools and adding random
height: 100%
or min-height: 0
or flex: 1 1 auto;
or position: relative
to various places
in the hope things get fixed and they don't break something else or one of the other browsers.
I'd think this would be common enough that the solution would be well documented on MDN or CSS Tricks
or some place but it's not or at least I've never found it. Instead there's just all us clueless
users reading the guesses of other clueless users sharing their magic solutions on Stack Overflow.
I often wonder if any of the browser makers or spec writers ever actually use the stuff they make and why they don't work harder to spread the solutions.
Any any case my CSS seems to be doing what I want at the moment
That said, I also ran into the issue that I needed a splitter control that let you drag the divider between two areas to adjust their sizes. There's 3 I found but they all had issues. One was out of date, and unmaintained and got errors with current React. Yea, I used react. Maybe that was a bad decision. Still not sure.
After fighting with the other solutions I ended up writing my own so that was a couple of days of working through issues.
Next up was comments. I don't know why I felt compelled to add comments but I did. I felt like people being able to comment would be net positive. Codepen allows comments. The easiest thing to do is just tack on disqus. Similar to the user code issue though I can't use disqus directly on the main site otherwise they could steal the access token.
So, setup another domain, put disqus in an iframe. The truth is disqus already puts itself in an iframe but at the top level it does this with a script on your main page which means they can steal secrets if they want. So, yea, 3rd domain (2nd was for user code).
The next problem is there is no way in the browser to size an iframe to fit its content. It seems ridiculous to have that limitation in 2020 but it's still there. The solution is the iframe sends messages to the parent saying what size its content is and then the parent can adjust the size of the iframe to match. It turns out this is how disqus itself works. The script it uses to insert an iframe listens for messages to resize the iframe.
Since I was doing iframe in iframe I needed to re-implement that solution.
It worked, problem solved..... or is it? 😆
It's a common topic on tech sites but there is a vocal minority that really dislike disqus. I assume it's because they are being tracked across the net. One half solution is you put a click through so that by default disqus doesn't load but the user can click "load comments" which is effectively an opt in to being tracked.
The thing is, gists already support comments. If only there was a way to use them easily on a 3rd party site like disqus. There isn't so, ....
There's an API where you can get the comments for a gist. You then have to format them from markdown into HTML. You need to sanitize them because it's user data and you don't want people to be able to insert JavaScript. I was already running comments on a 3rd domain so at least that part is already covered.
In any case it wasn't too much work to get existing comments displayed. New comments was more work though.
Github gists display as follows
+----------+ | header | +----------+ | files | | | | | +----------+ | comments | | | | | +----------+ | new | | comment | | form | +----------+
that comment form is way down the page. If there was an id to jump to I could have possibly
put that page in an iframe and just use a link like https://gist.github.com/<id>/#new-comment-form
.
to get the form to appear in a useful way. That would give the full github comment UI which
includes drag and drop image attachments amount other things. Even if putting it in an iframe
sucked I could have just had a link in the form of
<a https://gist.github.com/<id>/#new-comment-form>click here to leave a comment</a>
But, no such ID exits, nor does any standalone new comment form page.
So, I ended up adding a form. But for a preview we're back to the problem of user data on a page that has access to a github token.
The solution to put the preview on a separate page served from the comments domain and send a message with new content when the user asks for a preview. That way, even if we fail to fully sanitize the user content can't steal the tokens.
Both sites support embedding
jsgist.org just uses iframes.
JsBenchIt supports 2 embed modes. One, uses an iframe.
+ there's no security issues (like I can't see any data on whatever site you embedded it)
- It's up to you to make your iframe fit the results
The other mode uses a script
+ it can auto size the frame
- if I was bad I could change the script and steal your login credentials for whatever site you embed it on.
Of course I'd never do that but just to be aware. Maybe someone hacks my account or steals my domain etc... This same problem exists for any scripts you use from a site you don't control like query from a CDN for example so it's not uncommon to use a script. Just pointing out the trade off.
I'm not sure what the point of embedding the benchmarks is but I guess you could show off your special solution or, show how some other solution is slow, or maybe post one and encourage others to try to beat it.
I spent about a month, 6 to 12hrs a day on these 2 sites so far. There's a long list of things I could add, especially to jsgist.org. No idea if I will add those things. jsbenchit.org has a point for me because I didn't like the existing solution. jsgist.org has much less of a point because are are 10 or sites that already do something like this in various ways. jsfiddle, codepen, jsbin, codesandbox, glitch, github codespaces, plunkr, and I know there are others so I'm not sure what the point was. It started as just a kind of "oh, yea, I could do that too" while making jsbenchit and honestly I spent probably spent 2/3rds of the time there vs the benchmark site.
I honestly wish I'd find a way to spend this kind of time on something that has some hope of generating income, not just income but also something I'm truly passionate about. Much of this feels more like the procrastination project that one does to avoid doing the thing they should really do.
That said, the sites are live, they seem to kind of work though I'm sure there are still lurking bugs. Being stored in gists the data is yours. There is no tracking on the site. The sites are also open source so pull requests welcome.
]]>It's surprising the number of services out there that will tell you to embed their JavaScript into your webpage. If you do that then those scripts could be reading all the data on the page including login credientials, your credit card number, contact info, whatever else is on the page.
In other words, for example, to use the disqus comment service you effectively add a script like this
<script http://games.greggman.comyourblog.disqus.com/embed.js></script>
Disqus uses that to insert an iframe and then show all the comments and the UI for adding more. I kind of wanted to add comments to the site above via disqus but there's no easy way to do it securely. The best I can think of is I can make a 2nd domain so that on the main page I create an iframe that links to the 2nd domain and that 2nd domain then includes that disqus script.
I'm not dissing disqus, I'm just more surprised this type of issue is not called out more as the security issue it is.
I looked into how codepen allows embedding a pen recently. Here's the UI for embedding
Notice of the 4 methods they mark HTML as recommended. Well if you dig through the HTML you see it does this
<script async https://static.codepen.io/assets/embed/ei.js></script>
Yes, it powns your page. Fortunately they offer using an iframe but it's surprising to me they recommend the insecure, we own your site, embed our script directly on your page option over the others. In fact I'd argue it's irresponsible for them offer that option at all. I'm not trying to single out codepen, it's common across may companies. Heck, Google Analytics is probably the most common embedded script with Facebook's being second.
I guess what goes through most people's heads who make this stuff is "we're trustworthy so nothing to worry about". Except,
It sets a precedent to trust all such similar sites offering embedded scripts
I might be able to trust "you" but I can I trust all your employees and successors?
We're basically setting up a world of millions of effectively compromised sites and then praying that it doesn't become an issue sometime in the future.
Even if I trust you you could be compelled to use your backdoor.
I suppose this is unlikely but who knows. Maybe the FBI comes knocking requesting that for a specific site you help them steal credientials because they see your script is on the site they want to hack or get info from.
Anyway, I do have comments on this site by disqus using their script and I have google analytics on here too. This site though has no login, there are no credientials or anything else to steal. For the new site though I'll have to decide on whether or not I want to run comments at all and if so setup the second domain.
]]>TL;DR: Thousands of developers are giving 3rd parties write access to their github repos. This is even more irresponsible than giving out your email password or your computer's password since your github repos are often used by more than just you. The tokens given to 3rd parties are just like passwords. A hacker that breaches a company that has that info will suddenly have write access to every github repo the breached company had tokens for.
github should work to stop this irresponsible practice.
I really want to scream about security on a great many fronts but today let's talk about github.
What the actual F!!!
How is this not a 2000 comment topic on HN and Github not shamed into fixing this?
Github's permission systems are irresponsible in the extreme!!
Lots of sites let you sign up via github. Gatsby is one. Here's the screen you get when you try to sign up via your github account.
Like seriously, WTF does "Act on your behalf" mean? Does it mean Gatsby can have someone assassinated on my behalf? Can they take out a mortgage on my behalf? Can they volunteer me for the Peace Corps on my behalf? More seriously can they scan all my private repos on my behalf? Insert trojans in my code on my behalf? Open pull requests on other people's projects on my behalf? Log in to every other service I've connected to my github account on my behalf? Delete all my repos on my behalf? Add users to my projects on my behalf? Change my password on my behalf?
This seems like the most ridiculous permission ever!
I bought this up with github and they basically threw up their hands and said "Well, at least we're telling you something". No you're not. You're effectively telling me absolutely nothing except that you're claiming if I click through you're giving that company permission to do absolute anything. How is that useful info?
But, just telling me isn't really the point. The point is each service should be required to use as small of permissions as is absolutely necessary. If I sign up for a service, the default should be no permissions except getting my email address. If a service is supposed to work with a repo (like gatsby is) then github should provide an interface such that gatsby tells github "Give me a list of repos the user wants me to use" and github present the UI to select an existing one or create new ones and when finished, only those repos are accessible and only with the minimal permissions need.
This isn't entirely github's fault though, the majority of the development community seems asleep as well.
Let's imagine your bank let you sign in to 3rd party services in a similar manner. How many people would click through on "Let ACME corp act on your behalf on your Citibank Account". I think most people would be super scared of permissions like that. Instead they'd want very specific permission like, only permission to deposit money, or only permission to read the balance, or only permission to read transactions, etc...
Github providing blanket permissions to so many companies is a huge recipe for disaster just waiting to happen. If any one of those companies gets hacked, or has insider help, or has a disgruntled employee, suddenly your trade secrets are stolen, your unreleased app is leaked, your software is hacked with a trojan and all your customers sue you for the loss of work your app caused. It could be worse, you could run an open source library so by hacking ACME corp the bad guys can hack your library and via that hack everyone using your library.
I get why github does it and/or why the apps do it. For example check out Forestry. They could ask for minimal permissions and good on them for providing a path to go that route. They ask for greater permissions so that they can do all the steps for you. I get that. But if you allow them blanket access to your github (or gitlab), YOU SHOULD ARGUABLY BE DISQUALIFIED FROM BEING A SOFTWARE DEVELOPER!!!
The fact that you trusted some random 3rd party with blanket permissions to edit all of your repos and change all of your permissions is proof you don't know WTF you're doing and you can't be trusted. It's like if someone asked you for the password to your computer. If you give it out you're not computer literate!
boy: "Is it ok I set my password to your birthday?
girl: "Then your password is meaningless!"
Here's the default permissions Forestry asks for if you follow their recommended path.
First let's explain what Forestry is. It's a UI for editing blog posts through git so you can have a nice friendly interface for your git based static site generator. That's great! But, at most it only needs access to a single repo. Not all your public repos! If you click through and picked "Authorize" that's no different then giving them the password to your computer. Maybe worse because at least your hacked computer will probably only affect you.
Further, the fact that companies like Forestry even ask for this should be shameful! Remember when various companies like Facebook, Yelp, when you signed up they'd ask for your the username and password for your email account? Remember how pretty much every tech person on the planet knew that was seriously irresponsible to even ask? Well this is no different. It's entirely irresponsible for Forestry to ask for these kind of blanket permissions! It's entirely irresponsible for any users to give them these permissions! How are all the tech leaders seemingly asleep at calling this out?
Like I mentioned above, part of this arguably lies at Github's feet. Forestry does this because github provides no good flow to do it well so Forestry is left with 2 options (1) be entirely irresponsible but make it easy for the user to use their service, (2) be responsible but lose sales because people can't get setup easily.
Instead it should be a sign they're an irresponsible and untrustworthy company that they ask for these kinds of permissions at all. And further, github should be should also be ashamed their system encourages these kinds of blanket permissions.
Think of it this way. There are literally millions of open source libraries. npm has over a million libraries and that's just JavaScript. Add in python, C++, Java, C#, ruby, and all the other projects on github. Hundreds of thousands of developers wrote those libraries. How many of those developers have given out the keys their repos so random 3rd parties can hack their libraries? Maybe they gave too broad permissions to some code linting site. Maybe they gave too broad permissions to some project monitoring site. Maybe they gave too broad permissions just to join a forum using their github account. Isn't that super irresponsible? They've opened a door by using the library and they're letting more people in the door. That can't be good.
I don't blame the devs so much as github for effectively making this common. Github needs to take security seriously and that means working to make issues like this the exception, not the rule. It should be the easiest thing to do to allow a 3rd party minimal access to your repo. It should be much harder to give them too much access. There should be giant warnings that you're about to do something super irresponsible and that you should probably not be trusting the company asking for these permissions.
Call it out!
I don't mean to pick on Forestry. 100s of other github (and gitlab?) integrations have the same issues. Forestry was just the latest one I looked at. I've seen various companies have this issue for years now and I've been seriously surprised this hasn't been a bigger topic.
]]>Don't clutter the UX with meaningless info
Look above at the Github permissions. Reading public info should not even be listed! It's already obvious that all your public info can be read by the app. That's the definition of public! There's no reason to tell me it might read it. It doesn't need permission to do so.
I get the impression that for many topics, youtube is more popular than web pages. Note: I have zero proof but it doesn't really matter for the point of this article.
Let's imaging there is a website that teaches JavaScript, for example this one.
Note: I have no idea how many people go to that site but compare it to this youtube channel which has millions of views.
For example this one video has 2.8 million views and it's just one of 100s of videos.
I have no idea but I suspect the youtube channel is far more viewed than the website.
Why is that?
At first I thought it was obvious, it's because more people like videos more than they like text for these topics. It's certainly easy to believe. Especially the younger generation, pretty much anyone under 25 has grown up with YouTube as part of their life.
There are lots of arguments to be made for video for learning coding. Seeing someone walk through the steps can be better than reading about how to do it. For one, it's unlikely someone writing a tutorial is going to remember to detail everything where as someone making a video is at least likely showing the actual steps on the video. Small things they might have forgotten to write down appear in the video.
On the other hand, video sucks for reference and speed. I can't currently search the content of video. While I can cue a video and set it to different time points that's much worse than being able to skim or jump to the middle of an article.
Anyway, there are certainly valid reason why a video might be more popular than an article on the same topic.
What if one of the major reasons why videos are more popular than articles is because of YouTube itself. You go to youtube and based on what you watched before it recommends other things to watch. You watch one video on how to code in JavaScript and it's going to recommend watching more videos about programming in JavaScript and programming in general. It's also going to ask you to subscribe to those channels. You might even be setup to get emails when a youtuber posts a new video to their channel.
So, Imagine Google's home page worked the same way. Imagine instead of this
It looked more like this
Even before you searched you'd see recommendations based on things
you searched for or viewed before. You'd see things you subscribed to.
You'd see marks for stuff you'd read before. Your history would
be part of the website just like it is on youtube. Google could even keep the [+]
button in top right
which would lead to sites to create your content.
I can hear a lot of various responses.
Horror.
We already have enough supposed problems with algorithms running YouTube and FB, and Twitter, and even Google's search results. Do we need more?
Google Reader
Some might say it's been done. It was called Google Reader. Or other RSS feed readers. That's arguably not the same though. I know of no RSS feed readers that use the algorithm to recommend what to read next. Generally they just show you a list of all your feeds where feeds with new items show bolded. Further they don't recommend new content.
Portals
Portals used to be thing like the home page of Yahoo. In fact in other countries Portals are still bigger than Google. Yahoo Japan is the #1 page in Japan. Baidu I believe is the #1 page in China. Naver is #1 in Korea. All of those are portals. They have a search feature but the front page is covered in articles and links
The difference is those portals rarely recommend things based on your preferences or subscriptions. Instead they are almost exclusively designed like the front page of a newspaper. They feature what some editors decided should be featured or they feature whatever is most popular. They don't show what you want featured or rather not what some algorithm guessed what you would personally be interested in.
Google even used to have a portal and while it had some customization options it was mostly like all other portals shoveling "top news" in different catagories at you instead of showing you your interests.
Not In Google's Interests
You could argue because Youtube hosts all the content they have more of an incentive to keep you on youtube. But is that really true? Google makes most of their money via ads. AFAIK they make more off youtube than on youtube. Suggesting things to read will only lead people to more sites likely serving Google's ads.
Also, having the site be more useful by suggesting web pages only makes it more likely to visit the Google home page more often to see what's new to check out.
I think it would be an interesting experiment. If not Google's
current home page than some new one, youweb.com
or something.
Like youtube it would mark what you've already read. Like youtube
it would allow people to make channels. RSS is ready in place
to let people add their channels. Not sure how many systems still
support this but there was a standard for learning where the
page is for adding new content so clicking the [+]
button could
take you there, where ever it is and Google could suggest places
if you want to start from scratch including squarespace or wordpress.com or even blogger 😂
I think it might be super useful to have more sites recommended to me based on my interests. I watch youtube. I look at the recommendations. In fact I appreciate the recommendations. Why should websites be any different? Unlike Youtube the web is more decentralized so that's actually a win over Youtube. Why shouldn't Google (or someone) offer this service?
I'm honestly surprised it hasn't been done already. It probably has but I just forgot or didn't notice.
This might also make the tracking more visible. People claim Google knows all the sites you visit. Well, why not show it? If there's a Google analytics script on some site and Google recorded you went there, then you go to Google's home page and there in your history, just like Youtube's history, is a list of the sites you've visited. This would make it far more explicit so advocates for privacy could more easily point to it and say LOOK!. It might also get people to pursue more ways to have things not get tracked. But, I suspect lots of people would also find it super useful and having Google recommend stuff based on that would seem natural given the interface. As it is now all they use that data for is to serve ads they think you might be interested in. Using that data to recommend pages seems more directly useful to me. Something I want, an article on a topic I'm interested in, vs something they want, to show me ad. And it seems like no loss to them. They'll still get a chance to show me the ad.
Oh well, I expect the majority of people who will respond to this to be anti-Google and so anti this idea. I still think the idea is an interesting one. No site I know of recommends content for me in a way similar to Youtube. I'd like to try it out and see how it goes.
Someone pointed out Chrome for Android and iOS has the "suggested articles" feature but trying it out it completely fails.
First off I turned in on and for me it recommended nothing but Japanese articles. Google knows my browsing history. It knows that 99% of the articles I read are English. The fact that it recommended articles in Japanese shows it completely failed to be anything like the youtube experience I'm suggesting. In fact Google claims the suggestions are based on my Web & App Activity but checking my Web & App Activity there is almost zero Japanese.
Further, there is no method to "Subscribe" to a channel, for whatever definition of "channel". There is nothing showing me articles I've read, though given my rant on Youtube showing me articles I've read maybe that's a good thing? I mean I can go into my account and check my activity but what I want is to be able to go to a page for a specific channel and see the list of all that channel's content and see which articles I've read and which I haven't.
So while it's a step toward providing a youtube like experience it's completely failing to come close to matching it.
Note: I believe "channels" are important. When you watch a youtube video most creators say "Click Subscribe!". It's arguably an important part of the youtube experience and needs to be present if we're trying to see what it would be like to bring that same experience to the web. Most sites already have feeds so this is arguably something relatively easy for Google or whoever is providing this youtube like web experience to implement a "channels" feature.
]]>Caveat, maybe I'm full of it and there are reasons for the UI the way it is. I doubt it. 😁
Youtube's recommendations drive me crazy. I'm sure they have stats or something that says their recommendations are perfect on average but maybe it's possible different people respond better to different kinds of recommendation systems?
As an example some people might like to watch the same videos again and again. Others might only want new videos (me!). So, when my recommendations are 10-50% for videos I've already watched it's a complete waste of time and space.
Here are some recommendations
You can see 2 of them have a red bar underneath. This signifies that I've watched this video. Don't recommend it to me please!!!
But it gets worse. Here's some more recommendations. The highlighted video I've already watched so I don't want it recommended.
I click the 3 dots and tell it "Not interested"
I then have to worry that youtube thinks I hate that channel which is not the case so this appears
Clicking "Tell Us Why" I get this
So I finally choose "I already watched this video".
WHY DID THAT TAKE 4 STEPS!??!?!
It could be 3 steps
It could even be 2 steps
Why is that 4 steps? What UX guidelines or process decided this needed to be 4 steps? It reminds me of the Windows Off Menu fiasco.
It gets worse though. Youtube effectively calls me a liar!
After those steps above I go to the channel for that user and you'll notice the video I marked as "I already watched the video" is not marked as watched with the read bar.
Imagine if in gmail you marked a message as read but Google decided, nope, we're going to keep it marked as un-read because we know better than you! I get it I guess. The red bar is not a "I watched this already" it's a "how much of this have I watched". Well, if I mark it as watched then mark it as 100% watched!!!
I'm also someone who would prefer to separate music from videos. If I want music I'll go to some music site, maybe even youtube music 🤮 Youtube seems to often fill my recommendations with 10-50% music playlists. STOP IT! You're not getting me to watch more videos (or listen to more music). You're just wasting my time.
Here 5 of 12 recommendations are for music! I'm on YouTUBE to watch things, not listen to things.
Now, maybe some users looking for something to watch end up clicking on 1-2 hr music videos or playlist. Fine, let me turn off all music so I can opt out of it. Pretty please 🥺 I'm happy to go to youtube.com/music or something if I want music from youtube or I'll search for it directly but in general if I to go youtube and I'm looking for recommendations I'm there to watch something.
Please Youtube, let help me help you surface more videos I want to watch. Make it easier to me to tell you I've already watched the video and mark them as watched so when I'm glancing at videos in a channel It's easy to see what I have and haven't watched. Let me separate looking for music from looking for videos. Thank you 🙇♀️
]]>function createElem(tag, attrs = {}) { const elem = document.createElement(tag); for (const [key, value] of Object.entries(attrs)) { if (typeof value === 'object') { for (const [k, v] of Object.entries(value)) { elem[key][k] = v; } } else if (elem[key] === undefined) { elem.setAttribute(key, value); } else { elem[key] = value; } } return elem; } function addElem(tag, parent, attrs = {}) { const elem = createElem(tag, attrs); parent.appendChild(elem); return elem; }
It let's you create an element and fill the various parts of it relatively tersely.
For example
const form = addElem('form', document.body); const checkbox = addElem('input', form, { type: 'checkbox', id: 'debug', className: 'bullseye', }); const label = addElem('label', form, { for: 'debug', textContent: 'debugging on', style: { background: 'red'; }, });
With the built in browser API this would be
const form = document.createElement('form'); document.body.appendChild(form); const checkbox = document.createElement('input'); form.appendChild(checkbox); checkbox.type = 'checkbox'; checkbox.id = 'debug'; checkbox.className = 'bullseye'; const label = document.createElement('label'); form.appendChild(label); form.for = 'debug'; form.textContent = 'debugging on'; form.style.background = 'red';
Recently I saw someone post they use a function more like this
function addElem(tag, attrs = {}, children = []) { const elem = createElem(tag, attrs); for (const child of children) { elem.appendChild(child); } return elem; }
The difference to mine was you pass in the children, not the parent. This suggests a nested style like this
document.body.appendChild(addElem('form', {}, [ addElem('input', { type: 'checkbox', id: 'debug', className: 'bullseye', }), addElem('label', { for: 'debug', textContent: 'debugging on', style: { background: 'red'; }, }), ]);
I tried it out recently when refactoring someone else's code. No idea why I decided to refactor but anyway. Here's the original code
function createTableData(thead, tbody) { const row = document.createElement('tr'); { const header = document.createElement('th'); header.className = "text sortcol"; header.textContent = "Library"; row.appendChild(header); for(const benchmark of Object.keys(testData)) { const header = document.createElement('th'); header.className = "number sortcol"; header.textContent = benchmark; row.appendChild(header); }; } { const header = document.createElement('th'); header.className = "number sortcol sortfirstdesc"; header.textContent = "Average"; row.appendChild(header); thead.appendChild(row); for (let i = 0; i < libraries.length; i++) { const row = document.createElement('tr'); row.id = libraries[i] + '_row'; { const data = document.createElement('td'); data.style.backgroundColor = colors[i]; data.style.color = '#ffffff'; data.style.fontWeight = 'normal'; data.style.fontFamily = 'Arial Black'; data.textContent = libraries[i]; row.appendChild(data); for(const benchmark of Object.keys(testData)) { const data = document.createElement('td'); data.id = `${benchmark}_${library_to_id(libraries[i])}_data`; data.textContent = ""; row.appendChild(data); }; } { const data = document.createElement('td'); data.id = library_to_id(libraries[i]) + '_ave__data' data.textContent = ""; row.appendChild(data); tbody.appendChild(row); } }; } }
While that code is verbose it's relatively easy to follow.
Here's the refactor
function createTableData(thead, tbody) { thead.appendChild(addElem('tr', {}, [ addElem('th', { className: "text sortcol", textContent: "Library", }), ...Object.keys(testData).map(benchmark => addElem('th', { className: "number sortcol", textContent: benchmark, })), addElem('th', { className: "number sortcol sortfirstdesc", textContent: "Average", }), ])); for (let i = 0; i < libraries.length; i++) { tbody.appendChild(addElem('tr', { id: `${libraries[i]}_row`, }, [ addElem('td', { style: { backgroundColor: colors[i], color: '#ffffff', fontWeight: 'normal', fontFamily: 'Arial Black', }, textContent: libraries[i], }), ...Object.keys(testData).map(benchmark => addElem('td', { id: `${benchmark}_${library_to_id(libraries[i])}_data`, })), addElem('td', { id: `${library_to_id(libraries[i])}_ave__data`, }), ])); }; }
I'm not entirely sure I like it better. What I noticed when I was writing it is I found myself having a hard time keeping track of the opening and closing braces, parenthesis, and square brackets. Effectively it's one giant expression instead of multiple individual statements.
Maybe if it was JSX it might hold the same structure but be more readable? Let's assume we could use JSX here then it would be
function createTableData(thead, tbody) { thead.appendChild(( <tr> <th className="text sortcol">Library</th> ( Object.keys(testData).map(benchmark => ( <th className="number sortcol">{benchmark}</th> )) ) <th className="number sortcol sortfirstdesc">Average</th> </tr> )); for (let i = 0; i < libraries.length; i++) { tbody.appendChild(( <tr id={`libraries[i]}_row`}> <td style={{ backgroundColor: colors[i], color: '#ffffff', fontWeight: 'normal', fontFamily: 'Arial Black', }}>{libraries[i]}</td> Object.keys(testData).map(benchmark => ( <td id={`${benchmark}_${library_to_id(libraries[i])}_data`} /> )) <td id={`${library_to_id(libraries[i])}_ave__data`} /> </tr> )); }; }
I really don't know which I like best. I'm sure I don't like the most verbose raw browser API version. The more terse it gets though the harder it seem to be to read it.
Maybe I just need to come up with a better way to format?
I mostly wrote this post only because after the refactor I wasn't sure I was diggin it but writing all this out I have no ideas on how to fix my reservations. I did feel a little like was solving a puzzle unrelated to the task and hand to generate one giant expression.
Maybe my spidey senses are telling me it will be hard to read or edit later? I mean I do try to break down expressions into smaller parts now more than I did in the past. In the past I might have written something like
const dist = Math.sqrt((x2 - x1) ** 2 + (y2 - y1) ** 2;
but now-a-days I'd be much more likely to write something like
const dx = x2 - x1; const dy = y2 - y1; const distSq = dx * dx + dy * dy; const dist = Math.sqrt(distSq);
Maybe with such a simple equation it's hard to see why I prefer spell it out. Maybe I prefer to spell it out because often I'm writing tutorials. Certainly my younger self thought terseness was "cool" but my older self finds terseness for the sake of terseness to be mis-placed. Readability, understandability, editability, comparability I value over terseness.
hmmm....🤔
]]>glGenBuffers
, glGenTextures
, glGenRenderbuffer
, glGenFramebuffers
You still don't need to call them if you're using the compatibility profile.
The spec effectively said that all the glGenXXX
functions do is manage numbers for you but it was
perfectly fine to make up your own numbers
const id = 123; glBindBuffer(GL_ARRAY_BUFFER, id); glBufferData(GL_ARRAY_BUFFER, sizeof(data), data, GL_STATIC_DRAW);
I found this out when running the OpenGL ES 2.0 conformance tests against the implementation in Chrome as they test for it.
Note: I am not suggesting you should not call glGenXXX!. I'm just pointing out the triva that they don't/didn't need to be called.
You can set it the same as any other texture
glBindTexture(GL_TEXTURE_2D, 0); glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA8, 1, 1, 0, GL_RGBA, GL_UNSIGNED_BYTE, oneRedPixel);
Now if you happen to use the default texture it will be red.
I found this out as well when running the OpenGL ES 2.0 conformance tests against the implementation in Chrome as they test for it. It was also a little bit of a disappointment to me that WebGL didn't ship with this feature. I brought it up with the committee when I discovered it but I think people just wanted to ship rather than go back and revisit the spec to make it compatible with OpenGL and OpenGL ES. Especially since this trivia seems not well known and therefore rarely used.
The spec, at least the ES spec, says that glCompileShader
can always return success.
The spec only requires that glLinkProgram
fail if the shaders are bad.
I found this out as well when running the OpenGL ES 2.0 conformance tests against the implementation in Chrome as they test for it.
This trivia is unlikely to ever matter to you unless you're on some low memory embedded device.
I don't know the actual date but when I was using the OpenGL ES 2.0 conformance tests they were being back ported to OpenGL because there had never been an official set of tests. This is one reason there are so many issues with various OpenGL implementations or at least were in the past. Tests now exist but of course any edge case they miss is almost guaranteed to show inconsistencies across implementations.
This is also a lesson I learned. If you don't have comprehensive conformance tests for your standards, implementations will diverge. Making them comprehensive is hard but if you don't want your standard to devolve into lots of non-standard edge cases then you need to invest the time to make comprehensive conformance tests and do you best to make them easily usable with implementations other than your own. Not just for APIs, file formats are another place comprehensive conformance tests would likely help to keep the non-standard variations at a minimum.
Here are the WebGL2 tests as examples and here are the OpenGL tests. The OpenGL ones were not made public until 2017, 25yrs after OpenGL shipped.
This may or may not be fixed in the spec but it is not fixed in actual implementations.
Originally the viewport setting set by glViewport
only clipped vertices (and or the triangles they create).
but for example, draw a 32x32 size POINTS
point say 2 pixels off the edge of the viewport, should the
14 pixels still in the viewport be drawn? NVidia says yes, AMD says no. The OpenGL ES spec says yes, the
OpenGL spec says no.
Arguably the answer should be yes otherwise POINTS
are entirely useless for any size other than 1.0
POINTS
have a max size. That size can be 1.0.I don't think it's trivia really but it might be. Plenty of projects might use POINTS
for
particles and they expand the size based on the distance from the camera but it turns out they
may never expand or they might be limited to some size like 64x64.
I find this very strange that there is a limit. I can imagine there is/was dedicated hardware to draw points in the past. It's relatively trivial to implemented them yourself using instanced drawing and some trivial math in the vertex shader that has no size limit so I'm surprised that GPUs just don't use that method and not have a size limit.
But whatever, it's how it is. Basically you should not use POINTS
if you want consistent behavior.
LINES
have a max thickness of 1.0 in core OpenGLOlder OpenGL and therefore the compatibility profile of OpenGL supports lines of various thicknesses although like points above the max thickness was driver/GPU dependant and allowed to be just 1.0. But, in the core spec as of OpenGL 3.0 only 1.0 is allowed period.
The funny thing is the spec still explains how glLineWidth
works. It's only buried
in the appendix that it doesn't actually work.
E.2.1 Deprecated But Still Supported Features
The following features are deprecated, but still present in the core profile. They may be removed from a future version of OpenGL, and are removed in a forward compatible context implementing the core profile.
- Wide lines - LineWidth values greater than 1.0 will generate an
INVALID_VALUE
error.
The point is, except for maybe debugging you probably don't want to use LINES
and instead you need to rasterize lines yourself using triangles.
This comes up from needing to make the smallest repos either to post on stack overflow or to file a bug. Let's assume you're using core OpenGL or OpenGL ES 2.0+ so that you're required to write shaders. Here's the simplest code to test a texture
const GLchar* vsrc = R"(#version 300 void main() { gl_Position = vec4(0, 0, 0, 1); gl_PointSize = 100.0; })"; const GLchar* fsrc = R"(#version 300 precision highp float; uniform sampler2D tex; out vec4 color; void main() { color = texture(tex, gl_PointCoord); })"; GLuint prg = someUtilToCompileShadersAndLinkToProgram(vsrc, fsrc); glUseProgram(prg); // this block only needed in GL, not GL ES { glEnable(GL_PROGRAM_POINT_SIZE); GLuint vertex_array; glGenVertexArrays(1, &vertex_array); glBindVertexArray(vertex_array); } const GLubyte oneRedPixel[] = { 0xFF, 0x00, 0x00, 0xFF }; glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA8, 1, 1, 0, GL_RGBA, GL_UNSIGNED_BYTE, oneRedPixel); glDrawArrays(GL_POINTS, 0, 1);
Note: no attributes, no buffers, and I can test things about textures. If I wanted to try multiple things I can just change the vertex shader to
const GLchar* vsrc = R"(#version 300 layout(location = 0) in vec4 position; void main() { gl_Position = position; gl_PointSize = 100.0; })";
And then use glVertexAttrib
to change the position. Example
glVertexAttrib2f(0, -0.5, 0); // draw on left glDrawArrays(GL_POINTS, 0, 1); ... glVertexAttrib2f(0, 0.5, 0); // draw on right glDrawArrays(GL_POINTS, 0, 1);
Note that even if we used this second shader and didn't call glVertexAttrib
we'd get a point in the center of the viewport. See next item.
PS: This may only work in the core profile.
I see this all the time. Someone declares a position attribute as vec3
and then manually sets w
to 1.
in vec3 position; uniform mat4 matrix; void main() { gl_Position = matrix * vec4(position, 1); }
The thing is for attributes w
defaults to 1.0 so this will work just as well
in vec4 position; uniform mat4 matrix; void main() { gl_Position = matrix * position; }
It doesn't matter that you're only supplying x, y, and z from your attributes.
w
defaults to 1.
I'm not sure if this is well known or not. It partly falls out from understanding the API.
A framebuffer is a tiny thing that just consists of a collection of references to textures and renderbuffers. Therefore don't be afraid to make more.
Let's say your doing some multipass post processing where you swap inputs and outputs.
texture A as uniform input => pass 1 shader => texture B attached to framebuffer texture B as uniform input => pass 2 shader => texture A attached to framebuffer texture A as uniform input => pass 3 shader => texture B attached to framebuffer texture B as uniform input => pass 4 shader => texture A attached to framebuffer ...
You can implement this in 2 ways
Make one framebuffer, call gl.framebufferTexture2D
to set which texture to render to
between passes.
Make 2 framebuffers, attach texture A to one and texture B to the other. Bind the other framebuffer between passes.
Method 2 is better. Every time you change the settings inside a framebuffer the driver potentially has to check a bunch of stuff at render time. Don't change anything and nothing has to be checked again.
This arguably includes glDrawBuffers
which is also framebuffer state. If you need
multiple settings for glDrawBuffers
make a different framebuffer with the same attachments
but different glDrawBuffers
settings.
Arguably this is likely a trivial optimization. The more important point is framebuffers themselves are cheap.
Not too many people seem to be aware of the implications of TexImage2D. Consider that
in order to function on the GPU your texture must be setup with the correct number of mip
levels. You can set how many. It could be 1 mip. It could be a a bunch but they each
have to be the correct size and format. Let's say you have a 8x8 texture and you want to do the
standard thing (not setting any other texture or sampler parameters). You'll also need
a 4x4 mip, a 2x2 mip, an 1x1 mip. You can get those automatically by uploading the
level 0 8x8 mip and calling glGenerateMipmap
.
Those 4 mip levels need to copied to the GPU, ideally without wasting a lot of memory. But look at the API. There's nothing in that says I can't do this
glTexImage2D(GL_TEXTURE_2D, 0, 8, 8, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixelData8x8); glTexImage2D(GL_TEXTURE_2D, 1, 20, 40, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixelData40x20); glTexImage2D(GL_TEXTURE_2D, 2, 10, 20, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixelData20x10); glTexImage2D(GL_TEXTURE_2D, 3, 5, 10, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixelData10x5); glTexImage2D(GL_TEXTURE_2D, 4, 2, 5, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixelData5x2); glTexImage2D(GL_TEXTURE_2D, 5, 1, 2, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixelData2x1); glTexImage2D(GL_TEXTURE_2D, 6, 1, 1, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixelData1x1);
If it's not clear what that code does a normal mipmap looks like this
but the mip chain above looks like this
Now, the texture above will not render but the code is valid, no errors, and, I can fix it by adding this line at the bottom
glTexImage2D(GL_TEXTURE_2D, 0, 40, 80, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixelData80x40);
I can even do this
glTexImage2D(GL_TEXTURE_2D, 6, 1000, 1000, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixelData1000x1000); glTexImage2D(GL_TEXTURE_2D, 6, 1, 1, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixelData1x1);
or this
glTexImage2D(GL_TEXTURE_2D, 6, 1000, 1000, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixelData1000x1000); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAX_LEVEL, 5);
Do you see the issue? The API can't actually know anything about what you're trying to do until you actually draw. All the data you send to each mip just has to sit around until you call draw because there's no way for the API to know beforehand what state all the mips will be until you finally decide to draw with that texture. Maybe you supply the last mip first. Maybe you supply different internal formats to every mip and then fix them all later.
Ideally you'd specify the level 0 mip and then it would be an error to specify any other mip that does not match. Same internal format, correct size for the current level 0. That still might not be perfect because on changing level 0 all the mips might be the wrong size or format but it could be that changing level 0 to a different size invalidates all the other mip levels.
This is specifically why TexStorage2D
was added but TexImage2D
is pretty much ingrained
at this point
All else being equal I prefer smaller libraries to larger ones and I prefer to glue libraries together rather than take a library that tries to combine them for me. So, looking around I found chalk, colors, and ansi-colors. All popular libraries to provide colors in the terminal.
chalk is by far the largest with 5 dependencies totaling 3600 lines of code.
Things it combines
It combines checking whether or not the your output stream supports colors. Because of this it has to add way to tell it don't check for me because I'll do the checking myself
It peeks into your application's command line arguments magically looking for
--color
or --no-color
so without modifying your app or documenting what
arguments are valid it will look at these arguments. If your app uses those
arguments for something else you lose.
It combines all the named colors from HTML even though they are of questionable usefulness in a terminal.
It includes 800 lines of code for color conversions so you can use rgb or hsl or lab or cmyk etc.
Next up is colors. It's about 1500 lines of code.
It hacks the string prototype. The author seems to think this is not an issue.
It has a theme generator which works something like this
colors.setTheme({ cool: 'green', cold: 'blue', hot: 'red', });
And you can now do
colors.hot('the sun');
Like chalk it also spies on your command line arguments.
Next up is ansi-color. It's about 900 lines of code. It claims to be a clone of colors without the excess parts. No auto detecting support. No spying on your command line. It does include the theme function if only to try to match colors API.
Starting with themes. chalk gets this one correct. They don't do anything. They just show you that it's trivial to do it yourself.
const theme = { cool: chalk.green, cold: chalk.blue, hot: chalk.red, }; console.log(theme.hot('on fire'));
Why add a function setTheme
just to do that? What happens if I go
colors.theme({ red: 'green', green: 'red', });
Yes you'd never do that but an API shouldn't be designed to fail. What was the point of cluttering this code with this feature when it's so trivial to do yourself?
It would arguably be better to just have them as separate libraries. Let's
assume the color libraries have a function rgb
that takes an array of 3 values.
Then you can do this:
const pencil = require('pencil'); const webColors = require('color-name'); pencil.rgb(webColors.burlywood)('some string');
vs
const chalk = require('chalk'); chalk.keyword('burlywood')('some-string');
In exchange for breaking the dependency you gain the ability to take the newest color set anytime color-name is updated rather than have to wait for chalk to update its deps. You also don't have 150 lines of unused JavaScript in your code if you're not using the feature which you weren't.
As above the same is true of color conversions
const pencil = require('pencil'); const hsl = require('color-convert').rgb.hsl; pencil.rgb(hsl(30, 100, 50))('some-string');
vs
const chalk = require('chalk'); chalk.hsl(30, 100, 50)('some-string');
Breaking the dependency 1500 lines are removed from the library that you probably weren't using anyway. You can update the conversion library if there are bugs or new features you want. You can also use other conversions and they won't have a different coding style.
As mentioned above chalk looks at your command line behind the scenes. I don't know how to even describe how horrible that is.
A library peeking at your command line behind the scenes seems like a really bad
idea. To do this not only is it looking at your command line it's including
another library to parse your command line. It has no idea how your command line
works. Maybe you're shelling to another program and you have a —-
to separate
arguments to your program from arguments meant for the program you spawn like
Electron and npm. How would chalk know this? To fix this you have to hack around
chalk using environment variables. But of course if the program you're shelling
to also uses chalk it will inherit the environment variables requiring yet more
workarounds. It's just simply a bad idea.
Like the other examples, if your program takes command line arguments it's
literally going to be 2 lines to do this yourself. One line to add --color
to
your list of arguments and one line to use it to configure the color library.
Bonus, your command line argument is now documented for your users instead of
being some hidden secret.
This is another one where the added dependency only detracts, not adds.
We could just do this:
const colorSupport = require('color-support'); const pencil = require('pencil'); pencil.enabled = colorSupport.hasBasic;
Was that so hard? Instead it chalk tries to guess on its own. There are plenty of situations where it will guess wrong which is why making the user add 2 lines of code is arguably a better design. Only they know when it's appropriate to auto detect.
There are more issues with dependencies than just aesthetics and bloat though.
The library has chosen specific solutions. If you need different solutions you now have to work around the hard coded ones
Every dependency adds risks.
Risk by expanding the number of people you have to trust.
You need to trust every contributor of every dependencies. A library with 5 dependencies probably has between 5 and 25 contributors. Assuming the high end that's 25 people you're trusting to always do the right thing each time the library is updated. Maybe they got angry today and decided to take their ball home or burn the world. Maybe they got offered $$$$$$$ to help hack someone and needed the money for their sick mom. Maybe they introduced a bug by accident or wrote a vulnerability by accident. Each dependency you add adds a larger surface area for these issues.
Every dependency a library uses is one more you have to deal with. Library A gets discontinued. Library B has a security bug. Library C has a data leak. Library D doesn't run in the newest version of node, etc…
If the library you were using didn't depend on A, B, C, and D all of those issues disappear. Less work for you. Less things to monitor. Less notifications of issues.
I picked on chalk and colors here because they're perfect examples of a poor tradeoffs. Their dependencies take at most 2 lines of code to provide the same functionality with out the dependencies so including them did nothing but add all the issues and risks listed above.
It made more work for every user of chalk since they have to deal with the issues above. It even made more work for the developers of chalk who have to keep the dependencies up to date.
Just like they have a small blurb in their readme on how to implement themes they could have just as easily shown how to do all the other things without the dependencies using just 2 lines of code!
I'm not saying you should never have dependencies. The point is you should evaluate if they are really needed. In the case of chalk it's abundantly clear they were not. If you're adding a library to npm please reduce your dependencies. If it only takes 1 to 3 lines to reproduce the feature without the dependency then just document what to do instead of adding a dep. Your library will be more flexible. You'll expose your users to less risks. You'll make less work for yourself because you won't have to keep updating your deps. You'll make less work for your users because they won't have to keep updating your library just to get new deps.
Less dependencies = Everyone wins!
]]>So today I needed to copy a file in a node based JavaScript build step.
Background: For those that don't know it node has a package manager called npm (Node Package
Manager). Packages have a package.json
file that defines tons of things and
that includes a "scripts" section which are effectively just tiny command line strings
associated with a keyword.
Examples
"scripts": { "build": "make -f makefile", "test": "runtest-harness" }
So you can now type npm run build
to run the build script and it will run just
as if you had typed make -f makefile
.
Other than organizational the biggest plus is that if you have any development dependencies npm will look in those locally installed dependencies to run the commands. This means all your tools can be local to your project. If this project needs lint 1.6 and some other project needs lint 2.9 no worries. Just add the correct version of lint to your development dependencies and npm will run it for you.
But, the issue comes up, I wanted to copy a file. I could use a bigger build system
but for small things you can imagine just wanting to use cp
as in
"scripts": { "build": "make -f makefile && cp a.out dist/MyApp", ...
The problem is cp
is mac/linux only. If you care about Windows devs being able
to build on Windows then you can't use cp
. The solution is to add a node based
copy command to your development dependencies and then you can use it cross
platform
So, I go looking for copy commands. One of the most popular is [cpy-cli
].
Here's its dependency tree
└─┬ cpy-cli@2.0.0 ├─┬ cpy@7.3.0 │ ├── arrify@1.0.1 │ ├─┬ cp-file@6.2.0 │ │ ├── graceful-fs@4.2.3 │ │ ├─┬ make-dir@2.1.0 │ │ │ ├── pify@4.0.1 deduped │ │ │ └── semver@5.7.1 deduped │ │ ├── nested-error-stacks@2.1.0 deduped │ │ ├── pify@4.0.1 │ │ └── safe-buffer@5.2.0 │ ├─┬ globby@9.2.0 │ │ ├─┬ @types/glob@7.1.1 │ │ │ ├── @types/events@3.0.0 │ │ │ ├── @types/minimatch@3.0.3 │ │ │ └── @types/node@12.11.6 │ │ ├─┬ array-union@1.0.2 │ │ │ └── array-uniq@1.0.3 │ │ ├─┬ dir-glob@2.2.2 │ │ │ └─┬ path-type@3.0.0 │ │ │ └── pify@3.0.0 │ │ ├─┬ fast-glob@2.2.7 │ │ │ ├─┬ @mrmlnc/readdir-enhanced@2.2.1 │ │ │ │ ├── call-me-maybe@1.0.1 │ │ │ │ └── glob-to-regexp@0.3.0 │ │ │ ├── @nodelib/fs.stat@1.1.3 │ │ │ ├─┬ glob-parent@3.1.0 │ │ │ │ ├─┬ is-glob@3.1.0 │ │ │ │ │ └── is-extglob@2.1.1 deduped │ │ │ │ └── path-dirname@1.0.2 │ │ │ ├─┬ is-glob@4.0.1 │ │ │ │ └── is-extglob@2.1.1 │ │ │ ├── merge2@1.3.0 │ │ │ └─┬ micromatch@3.1.10 │ │ │ ├── arr-diff@4.0.0 │ │ │ ├── array-unique@0.3.2 │ │ │ ├─┬ braces@2.3.2 │ │ │ │ ├── arr-flatten@1.1.0 │ │ │ │ ├── array-unique@0.3.2 deduped │ │ │ │ ├─┬ extend-shallow@2.0.1 │ │ │ │ │ └── is-extendable@0.1.1 │ │ │ │ ├─┬ fill-range@4.0.0 │ │ │ │ │ ├─┬ extend-shallow@2.0.1 │ │ │ │ │ │ └── is-extendable@0.1.1 deduped │ │ │ │ │ ├─┬ is-number@3.0.0 │ │ │ │ │ │ └─┬ kind-of@3.2.2 │ │ │ │ │ │ └── is-buffer@1.1.6 │ │ │ │ │ ├── repeat-string@1.6.1 │ │ │ │ │ └─┬ to-regex-range@2.1.1 │ │ │ │ │ ├── is-number@3.0.0 deduped │ │ │ │ │ └── repeat-string@1.6.1 deduped │ │ │ │ ├── isobject@3.0.1 │ │ │ │ ├── repeat-element@1.1.3 │ │ │ │ ├── snapdragon@0.8.2 deduped │ │ │ │ ├─┬ snapdragon-node@2.1.1 │ │ │ │ │ ├─┬ define-property@1.0.0 │ │ │ │ │ │ └─┬ is-descriptor@1.0.2 │ │ │ │ │ │ ├─┬ is-accessor-descriptor@1.0.0 │ │ │ │ │ │ │ └── kind-of@6.0.2 deduped │ │ │ │ │ │ ├─┬ is-data-descriptor@1.0.0 │ │ │ │ │ │ │ └── kind-of@6.0.2 deduped │ │ │ │ │ │ └── kind-of@6.0.2 deduped │ │ │ │ │ ├── isobject@3.0.1 deduped │ │ │ │ │ └─┬ snapdragon-util@3.0.1 │ │ │ │ │ └─┬ kind-of@3.2.2 │ │ │ │ │ └── is-buffer@1.1.6 deduped │ │ │ │ ├─┬ split-string@3.1.0 │ │ │ │ │ └── extend-shallow@3.0.2 deduped │ │ │ │ └── to-regex@3.0.2 deduped │ │ │ ├─┬ define-property@2.0.2 │ │ │ │ ├─┬ is-descriptor@1.0.2 │ │ │ │ │ ├─┬ is-accessor-descriptor@1.0.0 │ │ │ │ │ │ └── kind-of@6.0.2 deduped │ │ │ │ │ ├─┬ is-data-descriptor@1.0.0 │ │ │ │ │ │ └── kind-of@6.0.2 deduped │ │ │ │ │ └── kind-of@6.0.2 deduped │ │ │ │ └── isobject@3.0.1 deduped │ │ │ ├─┬ extend-shallow@3.0.2 │ │ │ │ ├── assign-symbols@1.0.0 │ │ │ │ └─┬ is-extendable@1.0.1 │ │ │ │ └─┬ is-plain-object@2.0.4 │ │ │ │ └── isobject@3.0.1 deduped │ │ │ ├─┬ extglob@2.0.4 │ │ │ │ ├── array-unique@0.3.2 deduped │ │ │ │ ├─┬ define-property@1.0.0 │ │ │ │ │ └─┬ is-descriptor@1.0.2 │ │ │ │ │ ├─┬ is-accessor-descriptor@1.0.0 │ │ │ │ │ │ └── kind-of@6.0.2 deduped │ │ │ │ │ ├─┬ is-data-descriptor@1.0.0 │ │ │ │ │ │ └── kind-of@6.0.2 deduped │ │ │ │ │ └── kind-of@6.0.2 deduped │ │ │ │ ├─┬ expand-brackets@2.1.4 │ │ │ │ │ ├── debug@2.6.9 deduped │ │ │ │ │ ├─┬ define-property@0.2.5 │ │ │ │ │ │ └── is-descriptor@0.1.6 deduped │ │ │ │ │ ├─┬ extend-shallow@2.0.1 │ │ │ │ │ │ └── is-extendable@0.1.1 deduped │ │ │ │ │ ├── posix-character-classes@0.1.1 │ │ │ │ │ ├── regex-not@1.0.2 deduped │ │ │ │ │ ├── snapdragon@0.8.2 deduped │ │ │ │ │ └── to-regex@3.0.2 deduped │ │ │ │ ├─┬ extend-shallow@2.0.1 │ │ │ │ │ └── is-extendable@0.1.1 deduped │ │ │ │ ├── fragment-cache@0.2.1 deduped │ │ │ │ ├── regex-not@1.0.2 deduped │ │ │ │ ├── snapdragon@0.8.2 deduped │ │ │ │ └── to-regex@3.0.2 deduped │ │ │ ├─┬ fragment-cache@0.2.1 │ │ │ │ └── map-cache@0.2.2 │ │ │ ├── kind-of@6.0.2 │ │ │ ├─┬ nanomatch@1.2.13 │ │ │ │ ├── arr-diff@4.0.0 deduped │ │ │ │ ├── array-unique@0.3.2 deduped │ │ │ │ ├── define-property@2.0.2 deduped │ │ │ │ ├── extend-shallow@3.0.2 deduped │ │ │ │ ├── fragment-cache@0.2.1 deduped │ │ │ │ ├── is-windows@1.0.2 │ │ │ │ ├── kind-of@6.0.2 deduped │ │ │ │ ├── object.pick@1.3.0 deduped │ │ │ │ ├── regex-not@1.0.2 deduped │ │ │ │ ├── snapdragon@0.8.2 deduped │ │ │ │ └── to-regex@3.0.2 deduped │ │ │ ├─┬ object.pick@1.3.0 │ │ │ │ └── isobject@3.0.1 deduped │ │ │ ├─┬ regex-not@1.0.2 │ │ │ │ ├── extend-shallow@3.0.2 deduped │ │ │ │ └─┬ safe-regex@1.1.0 │ │ │ │ └── ret@0.1.15 │ │ │ ├─┬ snapdragon@0.8.2 │ │ │ │ ├─┬ base@0.11.2 │ │ │ │ │ ├─┬ cache-base@1.0.1 │ │ │ │ │ │ ├─┬ collection-visit@1.0.0 │ │ │ │ │ │ │ ├─┬ map-visit@1.0.0 │ │ │ │ │ │ │ │ └── object-visit@1.0.1 deduped │ │ │ │ │ │ │ └─┬ object-visit@1.0.1 │ │ │ │ │ │ │ └── isobject@3.0.1 deduped │ │ │ │ │ │ ├── component-emitter@1.3.0 deduped │ │ │ │ │ │ ├── get-value@2.0.6 │ │ │ │ │ │ ├─┬ has-value@1.0.0 │ │ │ │ │ │ │ ├── get-value@2.0.6 deduped │ │ │ │ │ │ │ ├─┬ has-values@1.0.0 │ │ │ │ │ │ │ │ ├── is-number@3.0.0 deduped │ │ │ │ │ │ │ │ └─┬ kind-of@4.0.0 │ │ │ │ │ │ │ │ └── is-buffer@1.1.6 deduped │ │ │ │ │ │ │ └── isobject@3.0.1 deduped │ │ │ │ │ │ ├── isobject@3.0.1 deduped │ │ │ │ │ │ ├─┬ set-value@2.0.1 │ │ │ │ │ │ │ ├─┬ extend-shallow@2.0.1 │ │ │ │ │ │ │ │ └── is-extendable@0.1.1 deduped │ │ │ │ │ │ │ ├── is-extendable@0.1.1 deduped │ │ │ │ │ │ │ ├── is-plain-object@2.0.4 deduped │ │ │ │ │ │ │ └── split-string@3.1.0 deduped │ │ │ │ │ │ ├─┬ to-object-path@0.3.0 │ │ │ │ │ │ │ └─┬ kind-of@3.2.2 │ │ │ │ │ │ │ └── is-buffer@1.1.6 deduped │ │ │ │ │ │ ├─┬ union-value@1.0.1 │ │ │ │ │ │ │ ├── arr-union@3.1.0 deduped │ │ │ │ │ │ │ ├── get-value@2.0.6 deduped │ │ │ │ │ │ │ ├── is-extendable@0.1.1 deduped │ │ │ │ │ │ │ └── set-value@2.0.1 deduped │ │ │ │ │ │ └─┬ unset-value@1.0.0 │ │ │ │ │ │ ├─┬ has-value@0.3.1 │ │ │ │ │ │ │ ├── get-value@2.0.6 deduped │ │ │ │ │ │ │ ├── has-values@0.1.4 │ │ │ │ │ │ │ └─┬ isobject@2.1.0 │ │ │ │ │ │ │ └── isarray@1.0.0 │ │ │ │ │ │ └── isobject@3.0.1 deduped │ │ │ │ │ ├─┬ class-utils@0.3.6 │ │ │ │ │ │ ├── arr-union@3.1.0 │ │ │ │ │ │ ├─┬ define-property@0.2.5 │ │ │ │ │ │ │ └── is-descriptor@0.1.6 deduped │ │ │ │ │ │ ├── isobject@3.0.1 deduped │ │ │ │ │ │ └─┬ static-extend@0.1.2 │ │ │ │ │ │ ├─┬ define-property@0.2.5 │ │ │ │ │ │ │ └── is-descriptor@0.1.6 deduped │ │ │ │ │ │ └─┬ object-copy@0.1.0 │ │ │ │ │ │ ├── copy-descriptor@0.1.1 │ │ │ │ │ │ ├─┬ define-property@0.2.5 │ │ │ │ │ │ │ └── is-descriptor@0.1.6 deduped │ │ │ │ │ │ └─┬ kind-of@3.2.2 │ │ │ │ │ │ └── is-buffer@1.1.6 deduped │ │ │ │ │ ├── component-emitter@1.3.0 │ │ │ │ │ ├─┬ define-property@1.0.0 │ │ │ │ │ │ └─┬ is-descriptor@1.0.2 │ │ │ │ │ │ ├─┬ is-accessor-descriptor@1.0.0 │ │ │ │ │ │ │ └── kind-of@6.0.2 deduped │ │ │ │ │ │ ├─┬ is-data-descriptor@1.0.0 │ │ │ │ │ │ │ └── kind-of@6.0.2 deduped │ │ │ │ │ │ └── kind-of@6.0.2 deduped │ │ │ │ │ ├── isobject@3.0.1 deduped │ │ │ │ │ ├─┬ mixin-deep@1.3.2 │ │ │ │ │ │ ├── for-in@1.0.2 │ │ │ │ │ │ └─┬ is-extendable@1.0.1 │ │ │ │ │ │ └── is-plain-object@2.0.4 deduped │ │ │ │ │ └── pascalcase@0.1.1 │ │ │ │ ├─┬ debug@2.6.9 │ │ │ │ │ └── ms@2.0.0 │ │ │ │ ├─┬ define-property@0.2.5 │ │ │ │ │ └─┬ is-descriptor@0.1.6 │ │ │ │ │ ├─┬ is-accessor-descriptor@0.1.6 │ │ │ │ │ │ └─┬ kind-of@3.2.2 │ │ │ │ │ │ └── is-buffer@1.1.6 deduped │ │ │ │ │ ├─┬ is-data-descriptor@0.1.4 │ │ │ │ │ │ └─┬ kind-of@3.2.2 │ │ │ │ │ │ └── is-buffer@1.1.6 deduped │ │ │ │ │ └── kind-of@5.1.0 │ │ │ │ ├─┬ extend-shallow@2.0.1 │ │ │ │ │ └── is-extendable@0.1.1 deduped │ │ │ │ ├── map-cache@0.2.2 deduped │ │ │ │ ├── source-map@0.5.7 │ │ │ │ ├─┬ source-map-resolve@0.5.2 │ │ │ │ │ ├── atob@2.1.2 │ │ │ │ │ ├── decode-uri-component@0.2.0 │ │ │ │ │ ├── resolve-url@0.2.1 │ │ │ │ │ ├── source-map-url@0.4.0 │ │ │ │ │ └── urix@0.1.0 │ │ │ │ └── use@3.1.1 │ │ │ └─┬ to-regex@3.0.2 │ │ │ ├── define-property@2.0.2 deduped │ │ │ ├── extend-shallow@3.0.2 deduped │ │ │ ├── regex-not@1.0.2 deduped │ │ │ └── safe-regex@1.1.0 deduped │ │ ├─┬ glob@7.1.5 │ │ │ ├── fs.realpath@1.0.0 │ │ │ ├─┬ inflight@1.0.6 │ │ │ │ ├── once@1.4.0 deduped │ │ │ │ └── wrappy@1.0.2 │ │ │ ├── inherits@2.0.4 │ │ │ ├─┬ minimatch@3.0.4 │ │ │ │ └─┬ brace-expansion@1.1.11 │ │ │ │ ├── balanced-match@1.0.0 │ │ │ │ └── concat-map@0.0.1 │ │ │ ├─┬ once@1.4.0 │ │ │ │ └── wrappy@1.0.2 deduped │ │ │ └── path-is-absolute@1.0.1 │ │ ├── ignore@4.0.6 │ │ ├── pify@4.0.1 deduped │ │ └── slash@2.0.0 │ └── nested-error-stacks@2.1.0 └─┬ meow@5.0.0 ├─┬ camelcase-keys@4.2.0 │ ├── camelcase@4.1.0 │ ├── map-obj@2.0.0 │ └── quick-lru@1.1.0 ├─┬ decamelize-keys@1.1.0 │ ├── decamelize@1.2.0 │ └── map-obj@1.0.1 ├─┬ loud-rejection@1.6.0 │ ├─┬ currently-unhandled@0.4.1 │ │ └── array-find-index@1.0.2 │ └── signal-exit@3.0.2 ├─┬ minimist-options@3.0.2 │ ├── arrify@1.0.1 deduped │ └── is-plain-obj@1.1.0 ├─┬ normalize-package-data@2.5.0 │ ├── hosted-git-info@2.8.5 │ ├─┬ resolve@1.12.0 │ │ └── path-parse@1.0.6 │ ├── semver@5.7.1 │ └─┬ validate-npm-package-license@3.0.4 │ ├─┬ spdx-correct@3.1.0 │ │ ├── spdx-expression-parse@3.0.0 deduped │ │ └── spdx-license-ids@3.0.5 │ └─┬ spdx-expression-parse@3.0.0 │ ├── spdx-exceptions@2.2.0 │ └── spdx-license-ids@3.0.5 deduped ├─┬ read-pkg-up@3.0.0 │ ├─┬ find-up@2.1.0 │ │ └─┬ locate-path@2.0.0 │ │ ├─┬ p-locate@2.0.0 │ │ │ └─┬ p-limit@1.3.0 │ │ │ └── p-try@1.0.0 │ │ └── path-exists@3.0.0 │ └─┬ read-pkg@3.0.0 │ ├─┬ load-json-file@4.0.0 │ │ ├── graceful-fs@4.2.3 deduped │ │ ├─┬ parse-json@4.0.0 │ │ │ ├─┬ error-ex@1.3.2 │ │ │ │ └── is-arrayish@0.2.1 │ │ │ └── json-parse-better-errors@1.0.2 │ │ ├── pify@3.0.0 │ │ └── strip-bom@3.0.0 │ ├── normalize-package-data@2.5.0 deduped │ └── path-type@3.0.0 deduped ├─┬ redent@2.0.0 │ ├── indent-string@3.2.0 │ └── strip-indent@2.0.0 ├── trim-newlines@2.0.0 └─┬ yargs-parser@10.1.0 └── camelcase@4.1.0 deduped
Yea, what the actually Effing F!?
197 dependencies, 1170 files, 47000 lines of JavaScript to copy files.
I ended up writing my own. There's the entire program
const fs = require('fs'); const src = process.argv[2]; const dst = process.argv[3]; fs.copyFileSync(src, dst);
And I added it to my build like this
"scripts": { "build": "make -f makefile && node copy.js a.out dist/MyApp", ...
So, my first reaction was, yea, something is massively over engineered. Or maybe that's under engineered if by under engineered it means "made without thinking".
You might think so what, people have large hard drives, fast internet, lots of memory. Who cares about dependencies? Well, the more dependencies you have the more you get messages like this
found 35 vulnerabilities (1 low, 2 moderate, 31 high, 1 critical) in 1668 scanned packages
You get more and more and more maintenance with more dependencies.
Not only that, you get dependent, not just on the software but on the people maintaining that software. Above, 197 dependencies also means trusting none of them are doing anything bad. As far as we know one of those dependencies could easily have a time bomb waiting until some day in the future to pown your machine or server.
On the other hand my copy copies a single file. cpy-cli
copies similar to cp
.
It can copy multiple files and whole trees.
I started wondering what it would take to add the minimal features to reproduce
a functional cp
clone. Note: not a full clone, a functional clone I'm sure
cp
has a million features but in my entire 40yr career I've only used about 2
of those features. (1) copying using wildcard as in cp *.txt dst
which honestly
is handled by the shell, not cp
. (2) copying recursively cp -R src dst
.
The first thing I did was look at a command line argument library. I've used
one called optionator
in the past and it's fine. I check and it has several
dependencies. 2 that stick out are:
a wordwrap library.
This is used to make your command's help fit the size of the terminal you're in. Definitely a useful feature. I have terminals of all difference sizes. I default to having 4 open.
a levenshtein distance library.
This is used so that if you specify a switch that doesn't exist it can try to suggest the correct one. For example might type:
my-copy-clone --src=abc.txt -destinatoin=def.txt
and it would says something like
no such switch: 'destinatoin' did you mean 'destination'?`.
Yea, that's kind of useful too.
Okay so my 4 line copy.js just got 3500 lines of libraries added. Or maybe I should look into another library that uses less deps while getting "woke" about dependencies.
Meh, I decide to parse my own arguments rather that take 3500 lines of code and 7 dependencies. Here's the code
#!/usr/bin/env node 'use strict'; const fs = require('fs'); const ldcp = require('../src/ldcp'); const args = process.argv.slice(2); const options = { recurse: false, dryRun: false, verbose: false, }; while (args.length && args[0].startsWith('-')) { const opt = args.shift(); switch (opt) { case '-v': case '--verbose': options.verbose = true; break; case '--dry-run': options.dryRun = true; options.verbose = true; break; case '-R': options.recurse = true; break; default: console.error('illegal option:', opt); printUsage(); } } function printUsage() { console.log('usage: ldcp [-R] src_file dst_file\n ldcp [-R] src_file ... dst_dir'); process.exit(1); } let dst = args.pop(); if (args.length < 1) { printUsage(); }
Now that the args are parsed we need a function to copy the files
const path = require('path'); const fs = require('fs'); const defaultAPI = { copyFileSync(...args) { return fs.copyFileSync(...args) }, mkdirSync(...args) { return fs.mkdirSync(...args); }, statSync(...args) { return fs.statSync(...args); }, readdirSync(...args) { return fs.readdirSync(...args); }, log() {}, }; function ldcp(_srcs, dst, options, api = defaultAPI) { const {recurse} = options; // check if dst is or needs to be a directory const dstStat = safeStat(dst); let isDstDirectory = false; let needMakeDir = false; if (dstStat) { isDstDirectory = dstStat.isDirectory(); } else { isDstDirectory = recurse; needMakeDir = recurse; } if (!recurse && _srcs.length > 1 && !isDstDirectory) { throw new Error('can not copy multiple files to same dst file'); } const srcs = []; // handle the case where src ends with / like cp for (const src of _srcs) { if (recurse) { const srcStat = safeStat(src); if ((needMakeDir && srcStat && srcStat.isDirectory()) || (src.endsWith('/') || src.endsWith('\\'))) { srcs.push(...api.readdirSync(src).map(f => path.join(src, f))); continue; } } srcs.push(src); } const srcDsts = [{srcs, dst, isDstDirectory, needMakeDir}]; while (srcDsts.length) { const {srcs, dst, isDstDirectory, needMakeDir} = srcDsts.shift(); if (needMakeDir) { api.log('mkdir', dst); api.mkdirSync(dst); } for (const src of srcs) { const dstFilename = isDstDirectory ? path.join(dst, path.basename(src)) : dst; if (recurse) { const srcStat = api.statSync(src); if (srcStat.isDirectory()) { srcDsts.push({ srcs: api.readdirSync(src).map(f => path.join(src, f)), dst: path.join(dst, path.basename(src)), isDstDirectory: true, needMakeDir: true, }); continue; } } api.log('copy', src, dstFilename); api.copyFileSync(src, dstFilename); } } function safeStat(filename) { try { return api.statSync(filename.replace(/(\\|\/)$/, '')); } catch (e) { // } } }
I made it so you pass an optional API of all the external functions it calls. That way you can pass in for example functions that do nothing if you want to test it. Or you can pass in graceful-fs if that's your jam but in the interest of NOT adding dependencies if you want that that's on you. Simple!
All that's left is using it after parsing the args
const log = options.verbose ? console.log.bind(console) : () => {}; const api = options.dryRun ? { copyFileSync(src) { fs.statSync(src) }, mkdirSync() { }, statSync(...args) { return fs.statSync(...args); }, readdirSync(...args) { return fs.readdirSync(...args); }, log, } : { copyFileSync(...args) { return fs.copyFileSync(...args) }, mkdirSync(...args) { return fs.mkdirSync(...args); }, statSync(...args) { return fs.statSync(...args); }, readdirSync(...args) { return fs.readdirSync(...args); }, log, }; ldcp(args, dst, options, api);
Total lines: 176 and 0 dependencies.
It's here if you want it.
]]>Apple products are designed to protect your privacy.
At Apple, we believe privacy is a fundamental human right.
Here's 10 things they could do to actually honor that mission.
This one is problematic but ... the majority of apps that ask to use your camera do not actually need access to your camera. Examples are the Facebook App, The Messenger App, the Twitter App. Even the Instagram App. Instead Apple could change their APIs such that the app asks for a camera picture and the OS takes the picture.
This removes the need to for those apps to have access to the camera at all. The only thing the app would get is the picture you took using the built in camera functionality controlled by the OS itself. If you don't take a picture and pick "Use Picture" then the app never sees anything.
As it is now you really have no idea what the app is doing. When you are in the Facebook app, once you've given the app permission to use the camera then as far as you know the app is streaming video, or pictures to Facebook constantly. You have absolutely no idea.
By changing the API so that the app is required to ask the OS for a photo that problem would be solved.
The problem with this solution is it doesn't cover streaming video since in that case the app needs the constant video. It also doesn't cover unique apps that do special things with the camera.
One solution to the unique camera feature issue would be app store rules. Basically only "camera" apps would be allowed to use the camera directly. SNS apps and other apps that just need a photo would be rejected if they asked for camera permission instead of asking the OS for a photo.
Another solution might be that the OS always ask the user for permission to use the camera (or at least provide the option). In other words if you are in some app like the Instagram app and you click the "take a photo" image the OS asks you "Allow App To Use The Camera?" each and every time. As it is now it only asks once. For those people that are privacy conscious being able to give the app each and every time would prevent spying.
See previous paragraph just replace every instance of "camera" with "mic"
This is similar to the two above but, as it is now apps like the Facebook App, Twitter, etc will ask for permission to access your photos. They do this so they can provide an interface to let you choose photos to post on facebook or tweet on twitter.
The problem is the moment you give them permission they can immediately look at ALL of your photos. All of them!
It would be better if Apple changed the API so the app asks the OS to ask you to choose 1 or more photos. The OS would then present an interface to choose 1 or more photos at which point only those photos you chose are given to the app.
That way apps could not read all of your photos.
Note that I get that some apps also want permission to read all your photos to enable to upload all of them automatically as you take them. That fine, it should just be a separate permission and Apple should enforce that features that let you choose photos to upload go through the OSes photo chooser and that apps that want full permission to access all photos for things like backup must also function without that permission when selecting photos for other purposes.
There are 3 options for GPS currently
There needs to a 4th
As it is, basically if you give an app permission to use GPS at all then every time you access that app it gets to know where you are.
It would be much more privacy oriented if you could choose to only give it GPS access for a moment, next 5 minutes, next 30 minutes, etc...
As it is now if you're privacy conscious you have to dig deep into the settings app for the privacy options. Give an app permission for GPS, then remember to dig through those options again to turn GPS permission back off a few minutes later.
That's not a very privacy oriented design.
Many apps show links to websites. For example Twitter or Facebook or the Google Maps app. When you click the links those apps open a web browser directly inside their app.
This means they can spy on everything you do in that web browser. That's not privacy oriented.
Apple should disallow having an internal web browser. They could do this by enforcing a policy that you can only make an app that can access all websites if that app is a web browser app. Otherwise you have to list the sites your app is allowed to access and that list has to be relatively small.
Many apps are actually just an app that goes directly to some
company's website which is fine. The app can list company.com
or *.company.com
as the sites it accesses. Otherwise it's
not allowed to access any other websites.
This would force apps to launch the user's browser when they click a link which would mean the apps could no longer spy on your browser activity. The most the could do is know the link you clicked. The couldn't know every link you click after that nor could the log everything you enter on every website you visit while in their app as they can do now.
Note that this would also be better user experience IMO. Users are used to the features available in their browser. For example being able to search in a page. Being able to turn on reader mode. Being able to bookmark and have those bookmarks sync. Being able to use an ad blocker. Etc... As it is when an app uses an internal web browser all of these features are not available. It's inconsistent and inconvenient for the user. By forcing apps to launch the user's browser all of that is solved.
Note: Apple should also allow setting a default browser so that users can choose Firefox or Brave or Chrome or whatever browser the choose for the features they want. If I use Firefox on my Mac I want to be able to bookmark things on iOS and have those bookmarks synced to my Mac but that becomes cumbersome if the OS keeps launching Safari instead of Firefox or whatever my browser of choice is.
In Japan there is a law that phone cameras must make a shutter noise. I actually despise that law. I want to be able to take pictures of my delicious gourmet meal in a quiet fancy restaurant without alerting and annoying all the other guests that I'm doing so. Japan claims this is to prevent perverts from taking up skirt pictures but perverts can just buy non-phone cameras and they can use an app because apps are not bound by the same laws so in effect this law does absolutely nothing except make it annoying and embarrassing to take pictures in quiet places.
On the other hand, if there was a small green, or orange light next to the camera that was physically connected to the camera's power so that it came on when the camera is on then I'd know when the camera was in use which would at least be a privacy oriented feature and so unlike the law above it would have a point.
If they wanted to be cute they could use a multi-color LED where red = camera is on, green = mic is on, yellow = both are on.
Let me add, I wish Apple devices had a built in camera cover or at least the Macs. I know you can buy a 3rd party one but adding a built in cover would show Apple is serious above Privacy.
AFAIK any app can scan WiFi and or bluetooth. Apps can use this info to know your location even if you have GPS off.
Basically there are databases of every WiFi's SSID (the name you pick to connect to a WiFi hotspot/router) and the databases also have recorded that WiFi's GPS so if they know which WiFis are near then they basically know where you are.
Here's a website where you can see what I'm talking about. Zoom in anywhere in the world and it will show the known WiFi hotspots / routers.
Why do most apps need this ability? They don't? Why doesn't Apple disallow it for most apps?
There are exceptions. I have a Wifi scanner app and a WiFi signal strength app and even a Bluetooth scanner and testing app that are very useful but Apple could easily have an App Store policy that only network utilities are allowed to use this powerful spying feature.
There is absolutely no reason the Twitter app or the Facebook app need to be able to see WiFi SSIDs nor local bluetooth devices.
Apple could easily add a permission requirement to use these features and only allow select apps have them. OR they could add it as yet another per app privacy setting.
This one is probably the most controversial suggestion here. The reasoning though goes like this
Safari is not even remotely the most secure browser.
This is provable by looking through the National Vulnerability Database (NVD) run by the National Institute of Standards and Technology (NIST)
In it you can see that while all browsers have around the same amount of vulnerabilities the types of vulnerabilities are different. Some browsers are designed to be more secure and so are less likely to have vulnerabilities that compromises your device and therefore your privacy. To put it slight more concretely 2 browsers might both have 150 vulnerabilities a year but one might have 90% code execution vulnerabilities (your device and data are compromised) and the other might have 90% DOS vulnerabilities (your device slows down or freezes but no data is compromised). If you check the database you'll find it's true that some browsers have orders of magnitude more code execution vulnerabilities than others.
By allowing competing browser engines users would have the choice to run those empirically more secure browser engines.
As it is now Safari has zero competition on iOS. A developer can make a new browser but it's really just Safari with a skin. That means Apple has less competition and so there is less pressure to make Safari better.
Allowing competing browsers engines would both be win for privacy and encourage faster and more development of Safari.
The number 1 objection I hear is that allowing other engines is a security issue but that is also provably false. See the NVD above. Other engines are more secure. By disallowing other engines you prevent users from protecting themselves from being hacked and therefore having their privacy invaded.
Another objection I hear is JITing, turing JavaScript into code, is something only Apple should be able to do. That argument basically boils down to Apple's app sandbox is insecure and that all apps must be 100% perfect or else they can escape the sandbox. You can't have it both ways. Either Apple's app sandbox is insecure and therefore the whole product is insecure OR Apple's app sandbox is secure and therefore allowing JITing doesn't affect that security. Now of course Apple's app sandbox could have bugs but those bugs can be exploited by any app. The solution is for Apple to be diligent and fix the bugs quickly and timely. The solution is not to make up some bogus JIT restriction.
To make an analogy if a product advertises as waterproof then it better actually waterproof. It can't come with some disclaimer that says "waterproof to 100meters but don't actually put this product in water as it might break".
The JIT argument is basically the same. "Our app sandbox is secure but don't actually run any code". It's clear the JIT argument is bogus. It's exists only to allow Apple a monopoly on browsers on iOS so they don't have to compete and so they can wield veto power over all browser standards. Since only they can make new browser features available to their 1.4 billion iOS devices if they don't support a feature it might as well not exist. Since devs can't use the feature with those 1.4 billion devices they generally just avoid the feature altogether even on non iOS devices.
All that is the long way of saying users would be more secure and get better privacy if they could run more secure and more privacy oriented browsers.
Apple fans won't like this reason. I don't consider myself an Apple fan and yet I own a Macbook Air, a Macbook Pro, a Mac Mini, an iPad Air 3rd Generation, an iPhone6+, an iPhoneX, an Apple TV 4 and at one point I also owned late 2018 iPad Pro and 4th Gen Apple Watch so clearly I also like Apple even if I don't consider myself a fanatic.
The thing is Apple is expensive. People will argue Apple's quality is high and worth the price and that might be true but it's kind of beside the point. You could make the argument a BMW or Mercedes Benz is a higher quality car than a Kia or a Hyundai but someone who only has a budget for a used Kia or Hyundai it's not realistic to ask them to buy an BMW or Mercedes
Similarly if you have a family of 4 and you want to give everyone in the family their own laptop computer you can buy 4 Windows laptops for the price of the cheapest Mac laptop. Sure those $200-$300 laptops are not nearly as nice as a Macbook Air but just like a Kia will still get you to your job a $250 Windows laptop will still let you browse the internet, run Microsoft Word, Illustrator, Photoshop, listen to music, watch youtube, edit a blog, read reddit, learn to program, etc.... It's unrealistic to ask a family of four to spend $4400 for 4 mac laptops instead of $1200 for 4 windows laptops.
Now you might be thinking so what… people who can afford should be able to spend their money on whatever they want. That's no different than anything else. Rich people buy penthouses just off Central Park and poor people live in trailer parks. The difference though is for most expensive things there are functionally equivalent inexpensive alternatives. A Kia will get you to work just as well as a BMW. Cheap clothing from Old Navy or Uniqlo or H&M will cloth you just as well as clothing from Versace or Prada or Louis Vuitton or pick you favorite but expensive brand. The food at Applebees will feed you just as well as the food from French Laundry. A $250 Vizio TV will let you watch TV just as functionally as a $4000 Sony.
But, if Apple really is the only privacy oriented option, if Android and Windows don't take your privacy seriously, then Apple being out of reach of so many people is … well I don't know what words to use basically say that people that can't afford Apple don't deserve privacy.
Of course that's not Apple's fault. Microsoft for Windows and Google for Android could step up and start making their OSes stop sucking for privacy.
My only point is if Apple is "the privacy company" then at the moment they are really "the privacy company for non-poor people" only and that they could be the privacy company for everyone if they offered some more affordable alternatives.
If you take your Apple device into repair they will ask you for your password or passcode. What the actual Effing Eff!??? Privacy? What? What's that? No, give us the password that unlocks all of your bank accounts, shopping accounts, bitcoin accounts, etc. Give us the password that lets us look at all your photos and videos. Give us the password that gives us access to the email on your device so that we can use that to open all other accounts by asking for password resets. Give us the password for the device that has all your two-factor codes and apps that confirm login on various services.
This is Apple's default stance. If you take a device in for service they will ask you for your password or passcode. That is not the kind of policy a privacy first company would have!
If you object they might tell you to change your password to something else and then change it back after you've gotten the repaired device back. That helps them not to know your normal password. It doesn't prevent all the stuff above.
If it's a Mac they'll give you the option to either turn on the guest account or add another account for them to login. Unfortunately that's really no better. If you're actually privacy oriented you'll have encrypted the hard drive. Giving them a password that unlocks the drive effectively gives them access to all your data whether or not a particular account has access to that data.
You can opt out of that too in which case they'll basically throw up their hands and say "In that case we may not be able to confirm the repairs". Another option is you can format the drive before giving it to them. Is that really the only option a privacy orient company should give you?
Now I get it, I'm sympathetic to the fact that it's harder for them if you don't give them the password. Still, for a Mac they can plug in an external drive and boot off that and at least confirm the machine itself is fine. For an iOS device, if they really are a "Privacy First" company then they need to find another way. They need to design a way to service the products that doesn't risk your privacy and risk exposing all your data.
Do I trust Apple as a company? Mostly. Do I trust every $15 an hour employee at the store like the one asking for password? No! Do I even trust some repair technician making more money but who may be getting paid on the side to scoop up login credentials? No! Do I know they destroy the info when the repair is finished? Nope! They ask you to write it down. As far a I know I could go dig through the trash behind an Apple store and find tons of credentials. Also as far as I know it's all stored in their service database ready to be stolen or hacked.
A privacy first company would do something different. They might for example backup your entire hard drive or iDevice, then reformat it, work on it, then restore. They might put it all on a USB drive, and hand the drive to you, you bring it back when they're done with any physical repairs and they restore it then and reformat the drive. If that's too slow then that's just incentive for them to make it faster. The might add some special service partition or service mode they can boot devices into.
The point is, a company that claims to take privacy seriously shouldn't be asking you to tell them the single most important password in your life. The password that unlocks all other passwords.
I'm not really hopeful Apple will make these changes but I'd argue if they don't make them then their statements of
Apple products are designed to protect your privacy.
At Apple, we believe privacy is a fundamental human right.
Is really just marketing and not at all real. Let's hope it is real and they take more steps to increase user privacy.
]]>