What if good code colorization came before naming standards?

2022-06-04

Related to this post on time wasted because of naming standards, I just ran into this 2018 talk about tree-sitter. A fast language parser for code colorization written by Max Brunsfeld at github.

It seems pretty clear something like this had to be developed later than naming standards but its at least interesting to imagine what if it came first? Would we even need naming standards?

In the talk they point out that most editors use a complex set of regular expressions to guess how to color things.

Here's a pretty typical example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
class Foo {
 public:
  static Foo* create(int bar);
  int getBar();

 private:
  explicit Foo(int bar);
  int bar;
};

Foo::Foo(int bar) : bar(bar) {};

Foo* Foo::create(int bar) {
  return new Foo(bar);
}

int Foo::getBar() {
  return bar;
}

What to notice:

This is because most of the colorizers have no actual knowledge of the language. They just have a list of known language keywords and some regular expressions to guess at what is a type, a function, a string, a comment.

What if the colorizer actually understood the language?

For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
class Foo {
 public:
  static Foo* create(int bar);
  int getBar();

 private:
  explicit Foo(int bar);
  int bar;
};

Foo::Foo(int bar) : bar(bar) {};

Foo* Foo::create(int bar) {
  return new Foo(bar);
}

int Foo::getBar() {
  return bar;
}

What to notice: Every type is green, every function is yellow, every bar is red when it's a member of a class. This means we don't need to name it _bar or mBar or bar_ as many style guides would suggest because the editor knows what it is and shows us by color.

We could also distinguish between member functions and global functions

void Foo::someMethod() {
  doThis();  // is this a member function or a global function?
  doThat();  // is this a member function or a global function?
}

Some of these issues go away by language design. In Python and JavaScript a member function and a property both have to be accessed by self / this so yes, there are other solutions than just coloring and naming conventions to help make code more understandable at a glance.

I haven't used tree-sitter directly (apparently it's used on Github for colorization though). I just found the idea that a language parsing colorizer could help make code more readable and help distinguish between things that naming conventions are often used for. I get that color isn't everywhere so it's maybe not a solution but it's still fun to think about what other ways we could make it easier to grok the code.

PS: The coloring above is hand-written and not via tree-sitter.

Comments

How many man years are wasted with western naming convensions?

2022-06-02

In most (all?) western languages there's the concept of UPPER case and lower case. I'm only guessing this is one reason why we have different styles of naming convensions. Commonly we have things like

ALL_UPPER_CASE_SNAKE_CASE_IS_A_CONSTANT
CaptializedCamelCaseAreClassNames
lower_case_snake_case_is_a_variable_or_member
lowerCaseCamelCaseIsAVariableOrMember

We often bake these into coding standards. Google's, Mozilla's, Apple's, Microsoft's

Being a person that grew up with a native language that has the concept of UPPER/lower case some of this seems normal but several languages, off the top of my head (Japanese, Chinese, Korean, Arabic, Hindi) have no UPPER/lower case. If programming had been invented or made popular by people whose native language was one of those would we even have the concept of camelCase?

In any case, the fact that we have these styles often leads to extra work.

For Example, I'm implementing OpenGL which has constants like GL_MAX_TEXTURE_SIZE. In the code, our style guide uses mCamelCase for class members so we have something like this

struct Limits {
  unsigned mMaxTextureSize;
  ...

There's then code that is effectively

void glGetIntegerv(GLenum pname, GLint* data) {
  const Limits& limits = getCurrentContext()->getLimits();
  switch (pname) {
    case GL_MAX_TEXTURE_SIZE:
      *data = limits.mMaxTextureSize;
      return;
    ...

Notice all this busy work. Someone had to translate GL_MAX_TEXTURE_SIZE into mMaxTextureSize. We could have instead done this

struct Limits {
  unsigned GL_MAX_TEXTURE_SIZE;
  ...
};

void glGetIntegerv(GLenum pname, GLint* data) {
  const Limits& limits = getCurrentContext()->getLimits();
  switch (pname) {
    case GL_MAX_TEXTURE_SIZE:
      *data = limits.GL_MAX_TEXTURE_SIZE;
      return;
    ...

In this second case, seaching for GL_MAX_TEXTURE_SIZE will find all references to the concept/limit we're interested in. In the previous case things are separated and we either have to search for each individually or we have write some more complex query. Further, we need to be aware of the coding style. Maybe in a different code base it's like this

struct Limits {
  unsigned max_texture_size;
  ...
};

void glGetIntegerv(GLenum pname, GLint* data) {
  const Limits& limits = getCurrentContext()->getLimits();
  switch (pname) {
    case GL_MAX_TEXTURE_SIZE:
      *data = limits.max_texture_size;
      return;
    ...

In fact I've worked in projects that are made from multiple parts where different parts have their own coding standards and yet more work is require to translate between the different choices.

The point is, time is spent translating to <-> from one form GL_MAX_TEXTURE_SIZE to another maxTextureSize.

You can automatic this process. Maybe you auto generate struct Limits above but you still ended up having to write code to do the translation from one form to another, you still have the search problem, and you still need to know which to reference.

It just had me wondering, how many man years of work would be saved if we didn't have this translation step which arguably only exists because of man made style guides, arguably influenced by the fact that western languages have the concept of letter case? I suspect, over all of the millions of programmers in the world, it's 100s or 1000s of man years of work per year possibly wasted just because of the effort of converting these forms.

Comments

Randomly Selected Music

2021-01-01

I don't think this will interest anyone but me but ...

I've been listening to music via my iPhone for many years playing my collection of mp3s. I guess that dates me as I'm not using Spotify or Apple Music or Youtube Music but, I haven't had any luck using any of those services.

As some examples, Spotify, I picked "Caro Emerald" Radio. I'd classify her as modern Swing

and Spotify played "How Would You Feel" by Kzezip who I'd classify as pop.

Another example I put in Prince Radio

And Spotify played rap. Prince had nothing to do with rap. The list from when I pasted it above is basically "Hits from the 80s" but that's not what I want if pick "Prince Radio". I want music that sounds similar to Prince. Maybe Windy and Lisa, or maybe Rick James? or maybe some bands I never heard of. Checking the list though Spotify will give me Huey Lewis & The News. I have nothing against them, I like their songs, but they aren't similar to Prince.

An example from Youtube Music, I put in "Fuck you till your Groovy" by Jill Jones and pick "Radio"

And it played "All Night Long" by Lionel Richie. WAT!???!

Note: I got the Jill Jones recommendation from doing the same thing on Google Play Music who's radio feature actually seemed to work, or rather actually played music similar to the artist and not just music popular by people who like that artist.

Another Youtube Music Example, I put in "Swingrowers" radio which is an electro swing band.

And youtube played "Bliss on Mushrooms" by Infected Mushroom, an Industrial Band

WTF!!

Anyway, all this means I sadly I keep going back to just my own collection of mp3s because trying any of the other services means hitting "no" or "don't like" for 9 of 10 songs.

All that was really beside the point though. What I wanted to write about was I'd been listening to music via my iPhone for years and recently went through my entire list of ~8500 songs trying to make a playlist and that's when it became clear to me,

iPhone SUCKS at playing music!!! ๐Ÿ˜ญ

In particular I noticed lots of songs I never heard my iPhone play and conversely there were some albums it would seem to play way too often. One example is I have this album called "Pure Sugar" by Pure Sugar. It's house music from 1998

As far I know it was some album I bought at a record store,probably in 1998, because on a short listen it sounded ok and back then buying CD was the only way to add to your music collection. I had over 1100 CDs at the time.

I have no particular affinity for this CD. I wouldn't add any song on the album to a playlist but if you like house music as like background music it's fine. I can listen to the entire album which is better than most.

In any way, out of ~8500 songs my iPhone seemed to pick songs from this album all the friggen time?!?! I never really gave it much thought because I just assumed it was bad luck or one of this weird artifacts of random selection but then, when I was going through the entire list of songs and seeing all the stuff not being played I started to be clear something was broken.

I didn't actually figure out what the problem was. I've never rated any albums or tracks. iTunes/Music apparently auto rates albums. No idea what it does to do that. If it's by the number of times played that would suck because it would re-enforce its bad choices. If it's by looking up on the net other people's opinions that would suck too as I don't want other people choosing music for me from my own collection. I also have no idea if the rating are used to pick random tracks.

If shuffling is related to rating, apparently the solution is to set all the ratings to 1. That way the app will assume you set it and won't auto rate.

I actually have no idea if that works. Instead I switched to using a different music app and suddenly I'm hearing much more of my collection than I was on the built in app.

We'll see how it goes. I have no idea how the new app chooses songs though. I can think of lots of algorithms. The simplest would just be to pick a random track from all tracks.

I'm pretty confident the app isn't doing that because it's also played too many tracks from the same albums. In other words lets say it played a song from "Unbreakable" by Janet Jackson. Within about 10 songs I'd hear another song from the same album, and 10 songs later yet another. I'm not sure what the odds of that are but I think they are low for 8500 tracks. I might guess that it picks a random album and then a random track. Would that be more likely to hear songs from the same albums? Or maybe it picks N albums and then picks random tracks from just those albums trying to make a theme? I have no idea.

I wrote some code to try just picking songs at random and to see how often it picks a song from some album of the last 20 albums.

It just keeps picking songs at random until it's played every song at least once. Using JavaScript's Math.random() function, for ~8500 tracks it would have to play around 80k tracks until it's played every track once. During that time at least one track would have been played ~25 times. Also about one out of 36 tracks will be from the same album as one of the last 20 albums played. That wouldn't remotely explain getting 3+ songs from the same album in say 60 songs. Yes I know random = random but in my own tests that situation rarely comes up.

Apparently the there's also a difference between "random" and "shuffle". "Shuffle" is supposed to by like a deck of card. You put take all the tracks and "shuffle" them, then play the tracks in the shuffled order. I suppose I should check that.

Well, according to that on average every ~40 tracks I'll get a track from the same album. Maybe that's what I'm experiencing.

It's ridiculous how much of my collection I'm being re-introduced to since I switched players

In any case, within the first 60 tracks on the new app it played 2 songs from "Pure Sugar"!!! ๐Ÿ˜ญ๐Ÿ˜…๐Ÿคฃ๐Ÿคฏ

Comments

Rethinking UI APIs

2017-09-14

A few weeks ago I wrote a rant, "React and Redux are a joke right?". While many of my friends got the point most of the comments clearly responded mostly to the title without really getting the point. That's my fault for burying the point instead of making it clear, and for picking a clickbait title although that did get people to look.

I wrote a response back then but wordpress ate it. Since then I spent 2 weeks redoing my blog to be a static blog and now I'm finally getting around to writing what I hope will clear up my point.

Maybe my point can best be summed up as "Let's rethink the entire UI paradigm. Let's consider alternatives to the standard retained mode GUI. Those APIs are making us write way too much code".

If you're not familiar with what a "retained mode GUI" is it's any graphic user interface that requires a tree of objects that represent the UI. You build up the tree of objects. It sticks around "is retained" even if your code is not running. The UI system looks at the tree of objects to show the UI. This is the single most common paradigm for UIs. Examples include the DOM and all the DOM elements. On iOS all the UI objects like UIButton, UILabel, UITextField. Android, MacOS, Windows etc all work this way.

There maybe many reasons for doing it that way and I'm not arguing to stop. Rather I'm just pointing out there's another style, called "Immediate Mode GUI", and if you go try and use it for a complex UI you'll find you probably need to write 1/5th as much code to get things done as you do with a retained mode GUI.

The issue with a retained mode GUI is having to manage all of those objects. You need some way to associate your data with their representation in the UI. You also need to marshal data between your copy of the data and the UI's copy of the data. On top of that most retained mode GUIs are not that fast so you have to minimize churn. You need to try to manipulate, create, and destroy as few of these GUI objects as possible otherwise your app will be unresponsive.

Contrast that with an Immediate Mode GUI. In an Immediate Mode GUI there are no objects, there is no (or nearly no) UI state at all. The entire UI is recreated every time it's rendered. This is no different than many game engines at a low level. You have a screen of pixels. To make things appear to move over time, to animate them, you erase the entire screen and re-draw every single pixel on the entire screen. An Immediate Mode GUI is similar, there's nothing to delete because nothing was created. Instead the UI is drawn by you by calling the functions in the Immediate Mode GUI API. This draws the entire UI and with the state of the mouse and keyboard also handles the one single UI object that's actually being interacted with.

What happens when you do this, all the code related to creating, deleting, and managing GUI objects disappears because there are no GUI objects. All the code related to marshaling data into and out of the GUI system disappears because the GUI system doesn't retain any data. All the code related to try to touch as few GUI objects as possible disappears since there are no objects to touch.

And so, that's what struck me when I was trying to write some performant code in React and some site suggested Redux. React as a pattern is fine. It or something close to it would work well in an Immediate Mode GUI. But in practice part of React's purpose is to minimize creation, deletion, and manipulation of GUI objects (DOM Elements for ReactDOM, native elements for React Native). To do that it tracks lots of stuff about those objects. It also has integrated functions to try to compare your data to the GUI's data. It has its this.setState system that gets more and more convoluted with requiring you not to set state directly and not even to inspect state directly as a change might be pending. All of that disappears in an Immediate Mode GUI. Redux is one of many suggested ways to take all that tracking a step further, to make it work across branches of your GUI object hierarchy. All of that disappears in an Immediate mode GUI.

Now I'm not saying there aren't other reason you might want command patterns. Nor am I saying you don't want more structure to the ways you manipulate your data. But, those ways should be 100% unrelated to your UI system. Your UI system should not be influencing how you use and store your data.

When I wrote the title of my rant "React and Redux are a joke right?" my mindset was realizing that the entire construct of React and Redux are their to manage a retained mode GUI. If we were using an Immediate Mode GUI we wouldn't need them. The joke is on us thinking we're making our lives easier when in reality we're just making it more complex by piling on solution on top of solution on top of solution when the real problem is below all of that.

Now, I'm not suggesting we all switch to Immediate Mode GUIs. Rather, I'm suggesting that experience writing a complex UI with an immediate mode GUI might influence ways to make things better. Maybe there's some new design between retained mode GUIs and Immediate Mode GUIs that would be best. Maybe a different API or paradigm all together. I don't know if it would lead anywhere. But, I think there's room for research. As a more concrete example the entire Unity3D UI is basically an immediate mode GUI. All of their 1000s of plugins use it. Their API could probably use a refactoring but Unity3D is clearly a complex UI and it's running using an Immediate Mode style system so it's at least some evidence that such a system can work.

There are lots of objections to Immediate Mode GUIs. The biggest is probably it's assumed they are CPU hogs. I have not written any benchmarks and it would be hard as the 2 paradigms are so different it would be like those benchmarks where a C++ person translates their benchmark to Haskell not really getting Haskell end up making a very inefficient benchmark.

That said, I think the CPU objection might be overblown and might possibly be the complete opposite with Immediate Mode GUIs using less CPU than their retained mode counterparts. The reason I think this is true is that Immediate Mode GUIs almost always run at 60fps (or smooth) and retained mode GUIs almost never run smooth. To make a retained mode GUI run smooth for a complex UI requires tons and tons of code. Code like React's checks for changes, like the state tracking stuff, all the componentDidMount, componentWillUnmount, shouldComponentMount stuff, various kinds of caches etc. All that code, possibly 5x to 10x the code of an Immediate Mode GUI is code that is using the CPU. And when I say 5x to 10x the code I don't necessarily mean just written code. I mean executed code. A loop comparing if properties changed is code that's running even if the code implementing the loop is small. Some languages and or system generate tons of code for you to help with these issues. You didn't have to personally write the code but it is there bloating your app, using CPU. When you see a retained mode GUI like a webpage not be entirely responsive or take more than 16ms to update that's because the CPU was running 100% and could not keep up. That situation rarely happens with an Immediate Mode GUI. That suggests to me it's possible the Immediate Mode GUI is actually using less CPU.

Other than Unity3D which is probably the most used Immediate Mode GUI in use but is tied to their editor the most popular Immediate Mode GUI is Dear ImGUI. Yes, it's only C++. Yes, it's not designed for the Web (or phones). Again, for like the 1000th time, that is not the point. The point is not Dear ImGUI as a solution. The point is Dear ImGUI as inspiration. In realizing how much extra code we're being made to write and or execute to deal with retained mode APIs. The point is to take a step back and consider that just maybe this whole ubiquitous retained mode GUI API should be revisited.

Comments

Trying to help noobs is SOOOO FRUSTRATING!

2016-12-14

I often wonder if I'd like to teach. It's certainly fun to teach when the students are easy to teach ? But where's the challenge in that?

I wrote webglfundamentals.org (and webgl2fundamentals.org) and I answer tons of WebGL questions on stackoverflow but sometimes it's sooooooooooooooo frustrating.

I'm trying to take those frustrations as an opportunity to learn better how to teach, how to present things, how to be patient, etc but still...

(more...)
Comments

Fun (or not so fun) Chrome ... Issue

2016-07-01

Wanna lose 5 to 20 minutes of your day depending on how well you know your computer? Visit this webpage in Chrome 51 (the current version as of 6/23) on an Windows 8.1 (or maybe Windows 10) machine. Might require NVidia drivers, not sure. Tell me what happens.

(more...)
Comments

Saving and Loading Files in a Web Page

2016-06-09

This article is targeted at people who've started learning web programming. They'd made a few web pages with JavaScript. Maybe they've made a paint program using 2d canvas or a 3d scene using three.js. Maybe it's an audio sound maker, maybe it's a tile map editor. At some point they wonder "how do I save files"?

Maybe they have a save button that just puts all the data into a textarea and presents it to the user and says "copy this and paste it into notepad to save".

Well, the way you save in a web page is you save to a web server. OH THE HORROR! I hear you screaming WHAT? A server? Why would I want to install some giant server just to save data?

I'm here to show you a web server is not a giant piece of software. In fact it's tiny. The smallest web server in many languages can be written in a few lines of code.

For example node.js is a version of JavaScript that runs outside the browser. If you've ever used Perl or Python it works exactly the same. You install it. You give it files to run. It runs them. Perl you give perl files, python you give python files, node you give JavaScript files.

So using node.js here is the smallest web server

const http = require('http');
function handleRequest(request, response){
    response.end('Hello World: Path = ' + request.url);
}
http.createServer(handleRequest).listen(8000, function() { });

JUST 5 LINES OF CODE!!!

Now, all these 5 lines do is return "Hello World:: Path = <path>" for every page but really that's the basics of a web server. Looking at the code above without even explaining it you could imagine looking at request.url and deciding to do different things depending on what the url is. One URL might save, one might load, one might login, etc. That's really it.

Let's explain these 5 lines of code

const http = require('http');

require is the equivalent of import in python or #include in C++ or using in C# or in JavaScript using <script> src="..."></script>. It's loading the module 'http' and referencing it by the variable http.

function handleRequest(request, response){
    response.end('Hello World: Path = ' + request.url);
}

This is a function that will get called when we get a request from a browser. The request holds data about what the browser requested like the URL for the request and all the headers sent by the browser including cookies, what language the user's browser is set to, etc...

response is an object we can use to send our response back to the browser. As you can see here we're sending a string. We could also load a file and send the contents of that file. Or we would query a database and send back the results. But everything starts here.

const server = http.createServer(handleRequest);
server.listen(8000, function() { 
  console.log("Listening at http://localhost:8000");
});

The last line I expanded a little. First it calls http.createServer and passes it the function we want to be called for all requests.

Then it calls server.listen which starts it listening for requests from the browser. The 8000 is which port to listen on and the function is a callback to tell us when the server is up and running.

TRY IT OUT!

To run this server install node.js. Don't worry it's not some giant ass program. It's actually rather small. Much smaller than python or perl or any of those other languages.

Now open a terminal on OSX or on windows open a "Node Command Prompt" (node made this when you installed it).

Make a folder somewhere and cd to it in your terminal / command prompt

Make a file called index.js and copy and paste the 5 lines above into it. Save it.

Now type node index.js

In your browser open a new tab/window and go to http://localhost:8000 It should say Hello World: Path = \. If you type some other URL like http://localhost:8000/foo/bar?this=that you'll see it returns that back to you.

Congratulations, you just wrote a web server!

Let's add serving files

You can imagine the code to serve files. You'd parse the URL to get a path, read the corresponding file, call response.end(contentsOfFile). It's literally that easy. But, just to make it less code (and cover more cases) there's a library that does it for us and it's super easy to use.

Press Ctrl-C to stop your server if you haven't already. Then type

npm install express

It will download a bunch of files and put them in a subfolder called "node_modules". It will also probably print an error about no "package.json" which you can ignore (google package.json later)

Now let's edit our file again. We're going to replace the entire thing with this

"use strict";
const express = require('express');
const baseDir = 'public';

let app = express();
app.use(express.static(baseDir));
app.listen(8000, function() {
    console.log("listening at http://localhost:8000");
});

Looking at the last 2 lines you see app.listen(8000... just like before. That's because express is the same http server we had before. It's just added some structure we'll get to in a bit.

The cool part here is the line

app.use(express.static(baseDir));

It says "serve all the files from baseDir".

So, make a subfolder called "public". Inside make a file called test.html and inside that file put O'HI You. Save it. Now run your server again with node index.js

Go to http://localhost:8000/test.html in your browser. You should see "O'HI You" in your browser.

Congratulations. You have just made a web server that will serve any files you want all in 9 lines of code!

Let's Save Files

To save files we need to talk about HTTP methods. It's another piece of data the browser sends when it makes a request. Above we saw the browser sent the URL to the server. It also sends a method. The default method is called GET. There's nothing special about it. it's just a word. You can make up any words you want but there are 7 or 8 common ones and GET means "Get resource".

If you've ever made an XMLHttpRequest (and I hope you have because I'm not going to explain that part), you specify the method. Back on the server we could look at request.method to see what you specified and use that as yet another piece of data to decide what to do. If the method is GET we do one thing. If the method is BANANAS we do something else.

express has wrapped that http object from our first example and it adds a few major things.

(1) it does more parsing of request.url for us so we don't have to do it manually.

(2) it routes. Routing means we can tell it for any of various paths what function to call. For example we could say if the path starts with "/fruit" call the function HandleFruit and if the path starts with "/purchase/item/:itemnumber" then call HandleItemPurchase etc.. In our case we're going to just say we want all routes to call our function.

(3) it can route based on method. That way we don't have to check if the method was "GET" or "PUT" or "DELETE" or "BANANAS". We can just tell it to only call our handler if the path is XYZ and the method is ABC.

So let's update the code. Ctrl-C your server if you haven't already and edit index.js and update it to this

"use strict";
const express = require('express');
*const path = require('path');
*const fs = require('fs');
const baseDir = 'public';

let app = express();
*app.put('*', function(req, res) {
*    console.log("saving:", req.path);
*    let body = '';
*    req.on('data', function(data) { body += data; });
*    req.on('end', function() {
*        fs.writeFileSync(path.join(baseDir, req.path), body);
*        res.send('saved');
*    });
*});
app.use(express.static(baseDir));
app.listen(8000, function() {
    console.log("listening at http://localhost:8000");
});

The first 2 added lines just reference more built in node libraries. path is a library for manipulating file paths. fs stands for "file system" and is a library for dealing with files.

Next we call app.put which takes 2 arguments. The first is the route and '*' just means "all routes". Then it takes a function to call for this route. app.put only routes "PUT" method requests so this line effectively says "Call our function for every route when the method is "PUT".

The function adds a tiny event handler to the data event that reads the data the browser is sending by adding it to a string called body. It adds another tiny event handler to the end event that then writes out the data to a file and sends back the message 'saved'.

And that's it! We'd made a server that saves and loads files. It's very insecure because it can save and load any file but if we're only using it for local stuff it's a great start.

Loading And Saving From the Browser

The final thing to do is to test it out by writing the browser side. I'm going to assume if you've already made some web pages and you're at the point where you want to load and save that you probably have some idea of what XMLHttpRequest is and how to make forms and check for users clicking on buttons etc. So with that in mind here's the new test.html

<html>
<head>
    <style>
    textarea {
        display: block;
    }
    </style>
</head>
<body>

<h1>Saving</h1>
<label for="savefilename">filename:</label>
<input id="savefilename" type="text" value="myfile.txt" />
<textarea id="savedata">
this is some test data
</textarea>
<button id="save">Save</button>

<h1>Loading</h1>
<label for="loadfilename">filename:</label>
<input id="loadfilename" type="text" value="myfile.txt" />
<textarea id="loaddata">
</textarea>
<button id="load">Load</button>

</body>
<script>
// make $ a shortcut for document.querySelector
const $ = document.querySelector.bind(document);

// when the user clicks 'save'
$("#save").addEventListener('click', function() {

    // get the filename and data
    const filename = $("#savefilename").value;
    const data = $("#savedata").value;

    // save
    saveFile(filename, data, function(err) {
        if (err) {
            alert("failed to save: " + filename + "\n" + err);
        } else {
            alert("saved: " + filename);
        }
    });
});

// when the user clicks load
$("#load").addEventListener('click', function() {

    // get the filename
    const filename = $("#loadfilename").value;

    // load 
    loadFile(filename, function(err, data) {
        if (err) {
            alert("failed to load: " + filename + "\n" + err);
        } else {
            $("#loaddata").value = data;
            alert("loaded: " + filename);
        }
    });
});

function saveFile(filename, data, callback) {
    doXhr(filename, 'PUT', data, callback);
}

function loadFile(filename, callback) {
    doXhr(filename, 'GET', '', callback);
}

function doXhr(url, method, data, callback) {
  const xhr = new XMLHttpRequest();
  xhr.open(method, url);
  xhr.onload = function() {
      if (xhr.status === 200) {
          callback(null, xhr.responseText);
      }  else {
          callback('Request failed.  Returned status of ' + xhr.status);
      }
  };
  xhr.send(data);
}
</script>
</html>

If you now save that and run your server then go to http://localhost:8000/test.html you should be able to type some text in the savedata area and click "save". Afterwords click "load" and you'll see it got loaded. Check your hard drive and you'll see a file has been created.

Now of course again this is insecure. For example if you type in "text.html" in the save filename and pick "save" it's going to overwrite your "text.html" file. Maybe you should pick a different route instead of "*" in the app.put('*', ... line. Maybe you want to add a check to see if the file exists with another kind of method and only update if the user is really sure.

The point of this article was not to make a working server. It was to show you how easy making a server is. A server like this that saves local files is probably only useful for things like an internal tool you happened to write in JavaScript that you and/or your team needs. They could run a small server like this on their personal machines and have your tool load, save, get folder listings, etc.

But, seeing how easy it is also hopefully demystifies servers a little. You can start here and then graduate to whole frameworks that let users login and share files remotely.

I should also mention you can load and save files to things like Google Drive or Dropbox. Now you know what they're basically doing behind the scenes.

Comments

More NPM BS

2016-04-18

Sorry to rant on this more but sheesh! How much does it take?

So I want to watch the filesystem for changes. Unfortunately node's fs.watch isn't OS independant. Some OSes return individual files. Other OSes just return the parent folder changed. I'm sure there's some legit reason like perf not to fix this for all devs but whatever. I assume there must be a library that fixes it.

So I search NPM. Maybe I should be using Google to search NPM because NPMs results are usually pretty poor but after searching a while I find this watch module. It's got nearly 700k downloads in the last month and tons of dependents so it must be good.

Given my prior experience the first thing I check is how many dependencies does it list?

Okay only 2 but they are both related to command line stuff. WTF does a library have command line dependencies? Oh, I see this library is conflated with a utility that watches for changes and launches a specified program anytime a file changes. Sigh... ?

Okay well let's look through the dependencies. It's not too deep. Fine, maybe I'll live with it if it works.

Ok I file an issue "Separate library from command line?" and move on.

A quick glance at the code and it doesn't appear to abstract the issues I need fixed but it's easy to test. So I write some test code

var watch = require('watch');

var dir = process.argv[2];
console.log("watching: ", dir);

var options = {
  ignoreDotFiles: true, // - When true this option means that when the file tree is walked it will ignore files that being with "."
  filter: function() { return true; }, // - You can use this option to provide a function that returns true or false for each file and directory to decide whether or not that file/directory is included in the watcher.
  ignoreUnreadableDir: true, // - When true, this options means that when a file can't be read, this file is silently skipped.
  ignoreNotPermitted: true, // - When true, this options means that when a file can't be read due to permission issues, this file is silently skipped.
  ignoreDirectoryPattern: /node_modules/, // - When a regex pattern is set, e.g. /node_modules/, these directories are silently skipped.
};

watch.createMonitor(dir, options, function(monitor) {
  monitor.on('created', function(f, s)     { show("created", f, s    ); });
  monitor.on('removed', function(f, s, s2) { show("removed", f, s, s2); });
  monitor.on('changed', function(f, s)     { show("changed", f, s    ); });
});

function show(event, f, s, n) {
  console.log(event, f);
}

I run it and pass it my temp folder which has a ton of crap in it including lots of sub folders.

I edit temp/foo.txt and see it print changed: temp/foo.txt. I delete temp/foo.txt and see it print removed: temp/foo.txt. I create a temp/foo.txt and see it print created: temp/foo.txt.

So far so good. Let's test one more thing.

I look for a subfolder since I've only been changing stuff in the root of the folder I'm watching. I just happen to pick temp/delme-eslint/node_modules/foo. BOOM! It prints

created temp/delme-eslint/node_modules/abab
created temp/delme-eslint/node_modules/abbrev
created temp/delme-eslint/node_modules/acorn
created temp/delme-eslint/node_modules/acorn-globals
created temp/delme-eslint/node_modules/acorn-jsx
created temp/delme-eslint/node_modules/amdefine
created temp/delme-eslint/node_modules/ansi-escapes
created temp/delme-eslint/node_modules/ansi-regex
created temp/delme-eslint/node_modules/ansi-styles
created temp/delme-eslint/node_modules/argparse
created temp/delme-eslint/node_modules/array-differ
created temp/delme-eslint/node_modules/array-union
created temp/delme-eslint/node_modules/array-uniq
created temp/delme-eslint/node_modules/arrify
created temp/delme-eslint/node_modules/asn1
created temp/delme-eslint/node_modules/assert-plus
created temp/delme-eslint/node_modules/async
created temp/delme-eslint/node_modules/aws-sign2
created temp/delme-eslint/node_modules/aws4
created temp/delme-eslint/node_modules/balanced-match
created temp/delme-eslint/node_modules/bl
created temp/delme-eslint/node_modules/bluebird
created temp/delme-eslint/node_modules/boolbase
created temp/delme-eslint/node_modules/boom
created temp/delme-eslint/node_modules/brace-expansion
created temp/delme-eslint/node_modules/caller-path
created temp/delme-eslint/node_modules/callsites
created temp/delme-eslint/node_modules/caseless
created temp/delme-eslint/node_modules/chalk
created temp/delme-eslint/node_modules/cheerio
created temp/delme-eslint/node_modules/cli-cursor
created temp/delme-eslint/node_modules/cli-width
created temp/delme-eslint/node_modules/code-point-at
created temp/delme-eslint/node_modules/coffee-script
created temp/delme-eslint/node_modules/color-convert
created temp/delme-eslint/node_modules/colors
created temp/delme-eslint/node_modules/combined-stream
created temp/delme-eslint/node_modules/commander
created temp/delme-eslint/node_modules/concat-map
created temp/delme-eslint/node_modules/concat-stream
created temp/delme-eslint/node_modules/core-util-is
created temp/delme-eslint/node_modules/cryptiles
created temp/delme-eslint/node_modules/css-select
created temp/delme-eslint/node_modules/css-what
created temp/delme-eslint/node_modules/cssom
created temp/delme-eslint/node_modules/cssstyle
created temp/delme-eslint/node_modules/d
created temp/delme-eslint/node_modules/dashdash
created temp/delme-eslint/node_modules/dateformat
created temp/delme-eslint/node_modules/debug
created temp/delme-eslint/node_modules/deep-is
created temp/delme-eslint/node_modules/del
created temp/delme-eslint/node_modules/delayed-stream
created temp/delme-eslint/node_modules/doctrine
created temp/delme-eslint/node_modules/dom-serializer
created temp/delme-eslint/node_modules/domelementtype
created temp/delme-eslint/node_modules/domhandler
created temp/delme-eslint/node_modules/domutils
created temp/delme-eslint/node_modules/ecc-jsbn
created temp/delme-eslint/node_modules/entities
created temp/delme-eslint/node_modules/es5-ext
created temp/delme-eslint/node_modules/es6-iterator
created temp/delme-eslint/node_modules/es6-map
created temp/delme-eslint/node_modules/es6-set
created temp/delme-eslint/node_modules/es6-symbol
created temp/delme-eslint/node_modules/es6-weak-map
created temp/delme-eslint/node_modules/escape-string-regexp
created temp/delme-eslint/node_modules/escodegen
created temp/delme-eslint/node_modules/escope
created temp/delme-eslint/node_modules/eslint
created temp/delme-eslint/node_modules/espree
created temp/delme-eslint/node_modules/esprima
created temp/delme-eslint/node_modules/esrecurse
created temp/delme-eslint/node_modules/estraverse
created temp/delme-eslint/node_modules/esutils
created temp/delme-eslint/node_modules/event-emitter
created temp/delme-eslint/node_modules/eventemitter2
created temp/delme-eslint/node_modules/exit
created temp/delme-eslint/node_modules/exit-hook
created temp/delme-eslint/node_modules/extend
created temp/delme-eslint/node_modules/extsprintf
created temp/delme-eslint/node_modules/fast-levenshtein
created temp/delme-eslint/node_modules/figures
created temp/delme-eslint/node_modules/file-entry-cache
created temp/delme-eslint/node_modules/find-up
created temp/delme-eslint/node_modules/findup-sync
created temp/delme-eslint/node_modules/flat-cache
created temp/delme-eslint/node_modules/foo
created temp/delme-eslint/node_modules/forever-agent
created temp/delme-eslint/node_modules/form-data
created temp/delme-eslint/node_modules/generate-function
created temp/delme-eslint/node_modules/generate-object-property
created temp/delme-eslint/node_modules/getobject
created temp/delme-eslint/node_modules/glob
created temp/delme-eslint/node_modules/globals
created temp/delme-eslint/node_modules/globby
created temp/delme-eslint/node_modules/graceful-fs
created temp/delme-eslint/node_modules/graceful-readlink
created temp/delme-eslint/node_modules/grunt
created temp/delme-eslint/node_modules/grunt-eslint
created temp/delme-eslint/node_modules/grunt-legacy-log
created temp/delme-eslint/node_modules/grunt-legacy-log-utils
created temp/delme-eslint/node_modules/grunt-legacy-util
created temp/delme-eslint/node_modules/har-validator
created temp/delme-eslint/node_modules/has-ansi
created temp/delme-eslint/node_modules/hoek
created temp/delme-eslint/node_modules/hawk
created temp/delme-eslint/node_modules/hooker
created temp/delme-eslint/node_modules/htmlparser2
created temp/delme-eslint/node_modules/http-signature
created temp/delme-eslint/node_modules/iconv-lite
created temp/delme-eslint/node_modules/ignore
created temp/delme-eslint/node_modules/inflight
created temp/delme-eslint/node_modules/inherits
created temp/delme-eslint/node_modules/inquirer
created temp/delme-eslint/node_modules/is-fullwidth-code-point
created temp/delme-eslint/node_modules/is-my-json-valid
created temp/delme-eslint/node_modules/is-path-cwd
created temp/delme-eslint/node_modules/is-path-in-cwd
created temp/delme-eslint/node_modules/is-path-inside
created temp/delme-eslint/node_modules/is-property
created temp/delme-eslint/node_modules/is-resolvable
created temp/delme-eslint/node_modules/is-typedarray
created temp/delme-eslint/node_modules/isarray
created temp/delme-eslint/node_modules/isstream
created temp/delme-eslint/node_modules/jodid25519
created temp/delme-eslint/node_modules/js-yaml
created temp/delme-eslint/node_modules/jsbn
created temp/delme-eslint/node_modules/jsdom
created temp/delme-eslint/node_modules/json-schema
created temp/delme-eslint/node_modules/json-stable-stringify
created temp/delme-eslint/node_modules/json-stringify-safe
created temp/delme-eslint/node_modules/jsonify
created temp/delme-eslint/node_modules/jsonpointer
created temp/delme-eslint/node_modules/jsprim
created temp/delme-eslint/node_modules/levn
created temp/delme-eslint/node_modules/load-grunt-tasks
created temp/delme-eslint/node_modules/lodash
created temp/delme-eslint/node_modules/lru-cache
created temp/delme-eslint/node_modules/mime-db
created temp/delme-eslint/node_modules/mime-types
created temp/delme-eslint/node_modules/minimatch
created temp/delme-eslint/node_modules/minimist
created temp/delme-eslint/node_modules/mkdirp
created temp/delme-eslint/node_modules/ms
created temp/delme-eslint/node_modules/multimatch
created temp/delme-eslint/node_modules/mute-stream
created temp/delme-eslint/node_modules/node-uuid
created temp/delme-eslint/node_modules/nopt
created temp/delme-eslint/node_modules/nth-check
created temp/delme-eslint/node_modules/number-is-nan
created temp/delme-eslint/node_modules/nwmatcher
created temp/delme-eslint/node_modules/oauth-sign
created temp/delme-eslint/node_modules/object-assign
created temp/delme-eslint/node_modules/once
created temp/delme-eslint/node_modules/onetime
created temp/delme-eslint/node_modules/optionator
created temp/delme-eslint/node_modules/os-homedir
created temp/delme-eslint/node_modules/parse5
created temp/delme-eslint/node_modules/path-exists
created temp/delme-eslint/node_modules/path-is-absolute
created temp/delme-eslint/node_modules/path-is-inside
created temp/delme-eslint/node_modules/pify
created temp/delme-eslint/node_modules/pinkie
created temp/delme-eslint/node_modules/pinkie-promise
created temp/delme-eslint/node_modules/pkg-up
created temp/delme-eslint/node_modules/pluralize
created temp/delme-eslint/node_modules/prelude-ls
created temp/delme-eslint/node_modules/process-nextick-args
created temp/delme-eslint/node_modules/progress
created temp/delme-eslint/node_modules/pseudomap
created temp/delme-eslint/node_modules/qs
created temp/delme-eslint/node_modules/read-json-sync
created temp/delme-eslint/node_modules/readable-stream
created temp/delme-eslint/node_modules/readline2
created temp/delme-eslint/node_modules/request
created temp/delme-eslint/node_modules/require-uncached
created temp/delme-eslint/node_modules/resolve
created temp/delme-eslint/node_modules/resolve-from
created temp/delme-eslint/node_modules/resolve-pkg
created temp/delme-eslint/node_modules/restore-cursor
created temp/delme-eslint/node_modules/rimraf
created temp/delme-eslint/node_modules/run-async
created temp/delme-eslint/node_modules/rx-lite
created temp/delme-eslint/node_modules/sax
created temp/delme-eslint/node_modules/shelljs
created temp/delme-eslint/node_modules/sigmund
created temp/delme-eslint/node_modules/slice-ansi
created temp/delme-eslint/node_modules/sntp
created temp/delme-eslint/node_modules/source-map
created temp/delme-eslint/node_modules/sprintf-js
created temp/delme-eslint/node_modules/sshpk
created temp/delme-eslint/node_modules/string-width
created temp/delme-eslint/node_modules/string_decoder
created temp/delme-eslint/node_modules/stringstream
created temp/delme-eslint/node_modules/strip-ansi
created temp/delme-eslint/node_modules/strip-json-comments
created temp/delme-eslint/node_modules/supports-color
created temp/delme-eslint/node_modules/symbol-tree
created temp/delme-eslint/node_modules/table
created temp/delme-eslint/node_modules/text-table
created temp/delme-eslint/node_modules/through
created temp/delme-eslint/node_modules/tough-cookie
created temp/delme-eslint/node_modules/tr46
created temp/delme-eslint/node_modules/tryit
created temp/delme-eslint/node_modules/tunnel-agent
created temp/delme-eslint/node_modules/tv4
created temp/delme-eslint/node_modules/tweetnacl
created temp/delme-eslint/node_modules/type-check
created temp/delme-eslint/node_modules/typedarray
created temp/delme-eslint/node_modules/underscore
created temp/delme-eslint/node_modules/underscore.string
created temp/delme-eslint/node_modules/user-home
created temp/delme-eslint/node_modules/util-deprecate
created temp/delme-eslint/node_modules/verror
created temp/delme-eslint/node_modules/webidl-conversions
created temp/delme-eslint/node_modules/whatwg-url-compat
created temp/delme-eslint/node_modules/which
created temp/delme-eslint/node_modules/wordwrap
created temp/delme-eslint/node_modules/wrappy
created temp/delme-eslint/node_modules/write
created temp/delme-eslint/node_modules/xml-name-validator
created temp/delme-eslint/node_modules/xregexp
created temp/delme-eslint/node_modules/xtend
created temp/delme-eslint/node_modules/yallist

BUG!!! Really, I used it for 5 minutes ran into bug and yet 700k downloads and a ton of dependents are using this buggy library. Sigh....?

Now the question is (a) fix it or 🍺 write my own.

Yes I know there are good arguments for fixing it. Just last month 700k downloads = 700k fixed downloads next month probably. But of course it possible means arguing with the author, jumping through their hoops. And, I still don't know if it solves the problem I want solved.

On top of that I there's at least one thing I wanted to re−design I think. Not only do I want the tree to be watched, I want to know what's in the tree. It kind of seems like that should be one operation. In other words I don't want to call some other function to scan the tree to get the list of all things in it and then separately call another function to watch the tree. If I do that then a file that's created between the first scan and the watch will never show up. It will also be twice as slow AFAICT as the the watch has to scan as well. So, now yet another discussion to be haggled over if I decide to fix vs do it myself.

The worst part is the thing I wanted to do I thought would take all of 20 minutes if that functionality I was looking for existed. Now will also probably take 8x as long to fix vs do it myself on top of writing the feature I was originally setting out to d. Probably not fair but (a) no negociation and 🍺 I feel more obligated to right tests for fixing but not for my own code ?

−−−

Update:

I filed a bug on that issue but I didn't purse fixing it. Instead I wrote my own. It took about 20hrs of solid work. Probably only a 2 hours to write but 18 hours to test across 3 platforms and learn new issues. I found out ubuntu, osx, windows, and travis−ci's containers all behave differently in this area. I'm sure I didn't catch all the stuff and I still would like to write another 4−8 hours of tests but I need to move on and it's working so far.

Some things that were interesting is where to make the divisions, what features to put in. For example the most used library I linked to above, the buggy one, has options for ignoring dot files (files that start with .) as well as an option to give it a regular expression to filter (the one that didn't actually work). After writing mine I found another library that takes complex globs and has debouncing features.

I feel like those features should be layers. It's probably as little as one line of code to filter. Example:

  var watcher = new SimpleTreeWatcher(pathToWatch, {
    filter: (filepath) => { return path.basename(filepath)[0] !== '.'); },
  });

So why clutter the library with options when doing it this way keeps the library simple and let's people be as complex or simple as they want. It seems better to just provide a few examples of how to filter than have a bunch of options.

Similarly the debouncing can relatively easily be layered on top. I should add examples to the docs but at least I have something that's working for me so far.

Comments

Dynamic Typing > Static Typing?

2016-01-19

I stumbled upon this amazing video called "The Unreasonable Effectiveness of Dynamic Typing for Practical Programs"

It's about 50mins long and unfortunately vimeo has no fast playback options like youtube so I'd suggest you download it and view it in VLC then click Ctrl+ or Cmd+ to speed up

In any case he makes a very compelling argument that static typing isn't really all that. His argument is LOOK AT THE DATA!!! Everyone has opinions. People believe static typing catches bugs. People believe static typing helps document code. People believe static typing makes IDEs work better and therefore save time, etc. BUT ... those are all just beliefs not backed up by any data. He points out that so far all the studies suggest that's not true. He claims 5 years ago he was a static type believer but after looking up the research and doing some of his own he's convinced static typing is a net negative.

You should really watch the talk but I'll try to sum it up here

(more...)
Comments

Writing Good Documentation and Examples

2015-12-02

I was going to start this as a rant for how bad most docs are but instead I'll just list what I think are some best practices. I know no one will read this but what else can I do 😉

(more...)
Comments
older