More NPM BS

2016-04-18

Sorry to rant on this more but sheesh! How much does it take?

So I want to watch the filesystem for changes. Unfortunately node's fs.watch isn't OS independant. Some OSes return individual files. Other OSes just return the parent folder changed. I'm sure there's some legit reason like perf not to fix this for all devs but whatever. I assume there must be a library that fixes it.

So I search NPM. Maybe I should be using Google to search NPM because NPMs results are usually pretty poor but after searching a while I find this watch module. It's got nearly 700k downloads in the last month and tons of dependents so it must be good.

Given my prior experience the first thing I check is how many dependencies does it list?

Okay only 2 but they are both related to command line stuff. WTF does a library have command line dependencies? Oh, I see this library is conflated with a utility that watches for changes and launches a specified program anytime a file changes. Sigh... ?

Okay well let's look through the dependencies. It's not too deep. Fine, maybe I'll live with it if it works.

Ok I file an issue "Separate library from command line?" and move on.

A quick glance at the code and it doesn't appear to abstract the issues I need fixed but it's easy to test. So I write some test code

var watch = require('watch');

var dir = process.argv[2];
console.log("watching: ", dir);

var options = {
  ignoreDotFiles: true, // - When true this option means that when the file tree is walked it will ignore files that being with "."
  filter: function() { return true; }, // - You can use this option to provide a function that returns true or false for each file and directory to decide whether or not that file/directory is included in the watcher.
  ignoreUnreadableDir: true, // - When true, this options means that when a file can't be read, this file is silently skipped.
  ignoreNotPermitted: true, // - When true, this options means that when a file can't be read due to permission issues, this file is silently skipped.
  ignoreDirectoryPattern: /node_modules/, // - When a regex pattern is set, e.g. /node_modules/, these directories are silently skipped.
};

watch.createMonitor(dir, options, function(monitor) {
  monitor.on('created', function(f, s)     { show("created", f, s    ); });
  monitor.on('removed', function(f, s, s2) { show("removed", f, s, s2); });
  monitor.on('changed', function(f, s)     { show("changed", f, s    ); });
});

function show(event, f, s, n) {
  console.log(event, f);
}

I run it and pass it my temp folder which has a ton of crap in it including lots of sub folders.

I edit temp/foo.txt and see it print changed: temp/foo.txt. I delete temp/foo.txt and see it print removed: temp/foo.txt. I create a temp/foo.txt and see it print created: temp/foo.txt.

So far so good. Let's test one more thing.

I look for a subfolder since I've only been changing stuff in the root of the folder I'm watching. I just happen to pick temp/delme-eslint/node_modules/foo. BOOM! It prints

created temp/delme-eslint/node_modules/abab
created temp/delme-eslint/node_modules/abbrev
created temp/delme-eslint/node_modules/acorn
created temp/delme-eslint/node_modules/acorn-globals
created temp/delme-eslint/node_modules/acorn-jsx
created temp/delme-eslint/node_modules/amdefine
created temp/delme-eslint/node_modules/ansi-escapes
created temp/delme-eslint/node_modules/ansi-regex
created temp/delme-eslint/node_modules/ansi-styles
created temp/delme-eslint/node_modules/argparse
created temp/delme-eslint/node_modules/array-differ
created temp/delme-eslint/node_modules/array-union
created temp/delme-eslint/node_modules/array-uniq
created temp/delme-eslint/node_modules/arrify
created temp/delme-eslint/node_modules/asn1
created temp/delme-eslint/node_modules/assert-plus
created temp/delme-eslint/node_modules/async
created temp/delme-eslint/node_modules/aws-sign2
created temp/delme-eslint/node_modules/aws4
created temp/delme-eslint/node_modules/balanced-match
created temp/delme-eslint/node_modules/bl
created temp/delme-eslint/node_modules/bluebird
created temp/delme-eslint/node_modules/boolbase
created temp/delme-eslint/node_modules/boom
created temp/delme-eslint/node_modules/brace-expansion
created temp/delme-eslint/node_modules/caller-path
created temp/delme-eslint/node_modules/callsites
created temp/delme-eslint/node_modules/caseless
created temp/delme-eslint/node_modules/chalk
created temp/delme-eslint/node_modules/cheerio
created temp/delme-eslint/node_modules/cli-cursor
created temp/delme-eslint/node_modules/cli-width
created temp/delme-eslint/node_modules/code-point-at
created temp/delme-eslint/node_modules/coffee-script
created temp/delme-eslint/node_modules/color-convert
created temp/delme-eslint/node_modules/colors
created temp/delme-eslint/node_modules/combined-stream
created temp/delme-eslint/node_modules/commander
created temp/delme-eslint/node_modules/concat-map
created temp/delme-eslint/node_modules/concat-stream
created temp/delme-eslint/node_modules/core-util-is
created temp/delme-eslint/node_modules/cryptiles
created temp/delme-eslint/node_modules/css-select
created temp/delme-eslint/node_modules/css-what
created temp/delme-eslint/node_modules/cssom
created temp/delme-eslint/node_modules/cssstyle
created temp/delme-eslint/node_modules/d
created temp/delme-eslint/node_modules/dashdash
created temp/delme-eslint/node_modules/dateformat
created temp/delme-eslint/node_modules/debug
created temp/delme-eslint/node_modules/deep-is
created temp/delme-eslint/node_modules/del
created temp/delme-eslint/node_modules/delayed-stream
created temp/delme-eslint/node_modules/doctrine
created temp/delme-eslint/node_modules/dom-serializer
created temp/delme-eslint/node_modules/domelementtype
created temp/delme-eslint/node_modules/domhandler
created temp/delme-eslint/node_modules/domutils
created temp/delme-eslint/node_modules/ecc-jsbn
created temp/delme-eslint/node_modules/entities
created temp/delme-eslint/node_modules/es5-ext
created temp/delme-eslint/node_modules/es6-iterator
created temp/delme-eslint/node_modules/es6-map
created temp/delme-eslint/node_modules/es6-set
created temp/delme-eslint/node_modules/es6-symbol
created temp/delme-eslint/node_modules/es6-weak-map
created temp/delme-eslint/node_modules/escape-string-regexp
created temp/delme-eslint/node_modules/escodegen
created temp/delme-eslint/node_modules/escope
created temp/delme-eslint/node_modules/eslint
created temp/delme-eslint/node_modules/espree
created temp/delme-eslint/node_modules/esprima
created temp/delme-eslint/node_modules/esrecurse
created temp/delme-eslint/node_modules/estraverse
created temp/delme-eslint/node_modules/esutils
created temp/delme-eslint/node_modules/event-emitter
created temp/delme-eslint/node_modules/eventemitter2
created temp/delme-eslint/node_modules/exit
created temp/delme-eslint/node_modules/exit-hook
created temp/delme-eslint/node_modules/extend
created temp/delme-eslint/node_modules/extsprintf
created temp/delme-eslint/node_modules/fast-levenshtein
created temp/delme-eslint/node_modules/figures
created temp/delme-eslint/node_modules/file-entry-cache
created temp/delme-eslint/node_modules/find-up
created temp/delme-eslint/node_modules/findup-sync
created temp/delme-eslint/node_modules/flat-cache
created temp/delme-eslint/node_modules/foo
created temp/delme-eslint/node_modules/forever-agent
created temp/delme-eslint/node_modules/form-data
created temp/delme-eslint/node_modules/generate-function
created temp/delme-eslint/node_modules/generate-object-property
created temp/delme-eslint/node_modules/getobject
created temp/delme-eslint/node_modules/glob
created temp/delme-eslint/node_modules/globals
created temp/delme-eslint/node_modules/globby
created temp/delme-eslint/node_modules/graceful-fs
created temp/delme-eslint/node_modules/graceful-readlink
created temp/delme-eslint/node_modules/grunt
created temp/delme-eslint/node_modules/grunt-eslint
created temp/delme-eslint/node_modules/grunt-legacy-log
created temp/delme-eslint/node_modules/grunt-legacy-log-utils
created temp/delme-eslint/node_modules/grunt-legacy-util
created temp/delme-eslint/node_modules/har-validator
created temp/delme-eslint/node_modules/has-ansi
created temp/delme-eslint/node_modules/hoek
created temp/delme-eslint/node_modules/hawk
created temp/delme-eslint/node_modules/hooker
created temp/delme-eslint/node_modules/htmlparser2
created temp/delme-eslint/node_modules/http-signature
created temp/delme-eslint/node_modules/iconv-lite
created temp/delme-eslint/node_modules/ignore
created temp/delme-eslint/node_modules/inflight
created temp/delme-eslint/node_modules/inherits
created temp/delme-eslint/node_modules/inquirer
created temp/delme-eslint/node_modules/is-fullwidth-code-point
created temp/delme-eslint/node_modules/is-my-json-valid
created temp/delme-eslint/node_modules/is-path-cwd
created temp/delme-eslint/node_modules/is-path-in-cwd
created temp/delme-eslint/node_modules/is-path-inside
created temp/delme-eslint/node_modules/is-property
created temp/delme-eslint/node_modules/is-resolvable
created temp/delme-eslint/node_modules/is-typedarray
created temp/delme-eslint/node_modules/isarray
created temp/delme-eslint/node_modules/isstream
created temp/delme-eslint/node_modules/jodid25519
created temp/delme-eslint/node_modules/js-yaml
created temp/delme-eslint/node_modules/jsbn
created temp/delme-eslint/node_modules/jsdom
created temp/delme-eslint/node_modules/json-schema
created temp/delme-eslint/node_modules/json-stable-stringify
created temp/delme-eslint/node_modules/json-stringify-safe
created temp/delme-eslint/node_modules/jsonify
created temp/delme-eslint/node_modules/jsonpointer
created temp/delme-eslint/node_modules/jsprim
created temp/delme-eslint/node_modules/levn
created temp/delme-eslint/node_modules/load-grunt-tasks
created temp/delme-eslint/node_modules/lodash
created temp/delme-eslint/node_modules/lru-cache
created temp/delme-eslint/node_modules/mime-db
created temp/delme-eslint/node_modules/mime-types
created temp/delme-eslint/node_modules/minimatch
created temp/delme-eslint/node_modules/minimist
created temp/delme-eslint/node_modules/mkdirp
created temp/delme-eslint/node_modules/ms
created temp/delme-eslint/node_modules/multimatch
created temp/delme-eslint/node_modules/mute-stream
created temp/delme-eslint/node_modules/node-uuid
created temp/delme-eslint/node_modules/nopt
created temp/delme-eslint/node_modules/nth-check
created temp/delme-eslint/node_modules/number-is-nan
created temp/delme-eslint/node_modules/nwmatcher
created temp/delme-eslint/node_modules/oauth-sign
created temp/delme-eslint/node_modules/object-assign
created temp/delme-eslint/node_modules/once
created temp/delme-eslint/node_modules/onetime
created temp/delme-eslint/node_modules/optionator
created temp/delme-eslint/node_modules/os-homedir
created temp/delme-eslint/node_modules/parse5
created temp/delme-eslint/node_modules/path-exists
created temp/delme-eslint/node_modules/path-is-absolute
created temp/delme-eslint/node_modules/path-is-inside
created temp/delme-eslint/node_modules/pify
created temp/delme-eslint/node_modules/pinkie
created temp/delme-eslint/node_modules/pinkie-promise
created temp/delme-eslint/node_modules/pkg-up
created temp/delme-eslint/node_modules/pluralize
created temp/delme-eslint/node_modules/prelude-ls
created temp/delme-eslint/node_modules/process-nextick-args
created temp/delme-eslint/node_modules/progress
created temp/delme-eslint/node_modules/pseudomap
created temp/delme-eslint/node_modules/qs
created temp/delme-eslint/node_modules/read-json-sync
created temp/delme-eslint/node_modules/readable-stream
created temp/delme-eslint/node_modules/readline2
created temp/delme-eslint/node_modules/request
created temp/delme-eslint/node_modules/require-uncached
created temp/delme-eslint/node_modules/resolve
created temp/delme-eslint/node_modules/resolve-from
created temp/delme-eslint/node_modules/resolve-pkg
created temp/delme-eslint/node_modules/restore-cursor
created temp/delme-eslint/node_modules/rimraf
created temp/delme-eslint/node_modules/run-async
created temp/delme-eslint/node_modules/rx-lite
created temp/delme-eslint/node_modules/sax
created temp/delme-eslint/node_modules/shelljs
created temp/delme-eslint/node_modules/sigmund
created temp/delme-eslint/node_modules/slice-ansi
created temp/delme-eslint/node_modules/sntp
created temp/delme-eslint/node_modules/source-map
created temp/delme-eslint/node_modules/sprintf-js
created temp/delme-eslint/node_modules/sshpk
created temp/delme-eslint/node_modules/string-width
created temp/delme-eslint/node_modules/string_decoder
created temp/delme-eslint/node_modules/stringstream
created temp/delme-eslint/node_modules/strip-ansi
created temp/delme-eslint/node_modules/strip-json-comments
created temp/delme-eslint/node_modules/supports-color
created temp/delme-eslint/node_modules/symbol-tree
created temp/delme-eslint/node_modules/table
created temp/delme-eslint/node_modules/text-table
created temp/delme-eslint/node_modules/through
created temp/delme-eslint/node_modules/tough-cookie
created temp/delme-eslint/node_modules/tr46
created temp/delme-eslint/node_modules/tryit
created temp/delme-eslint/node_modules/tunnel-agent
created temp/delme-eslint/node_modules/tv4
created temp/delme-eslint/node_modules/tweetnacl
created temp/delme-eslint/node_modules/type-check
created temp/delme-eslint/node_modules/typedarray
created temp/delme-eslint/node_modules/underscore
created temp/delme-eslint/node_modules/underscore.string
created temp/delme-eslint/node_modules/user-home
created temp/delme-eslint/node_modules/util-deprecate
created temp/delme-eslint/node_modules/verror
created temp/delme-eslint/node_modules/webidl-conversions
created temp/delme-eslint/node_modules/whatwg-url-compat
created temp/delme-eslint/node_modules/which
created temp/delme-eslint/node_modules/wordwrap
created temp/delme-eslint/node_modules/wrappy
created temp/delme-eslint/node_modules/write
created temp/delme-eslint/node_modules/xml-name-validator
created temp/delme-eslint/node_modules/xregexp
created temp/delme-eslint/node_modules/xtend
created temp/delme-eslint/node_modules/yallist

BUG!!! Really, I used it for 5 minutes ran into bug and yet 700k downloads and a ton of dependents are using this buggy library. Sigh....?

Now the question is (a) fix it or 🍺 write my own.

Yes I know there are good arguments for fixing it. Just last month 700k downloads = 700k fixed downloads next month probably. But of course it possible means arguing with the author, jumping through their hoops. And, I still don't know if it solves the problem I want solved.

On top of that I there's at least one thing I wanted to re−design I think. Not only do I want the tree to be watched, I want to know what's in the tree. It kind of seems like that should be one operation. In other words I don't want to call some other function to scan the tree to get the list of all things in it and then separately call another function to watch the tree. If I do that then a file that's created between the first scan and the watch will never show up. It will also be twice as slow AFAICT as the the watch has to scan as well. So, now yet another discussion to be haggled over if I decide to fix vs do it myself.

The worst part is the thing I wanted to do I thought would take all of 20 minutes if that functionality I was looking for existed. Now will also probably take 8x as long to fix vs do it myself on top of writing the feature I was originally setting out to d. Probably not fair but (a) no negociation and 🍺 I feel more obligated to right tests for fixing but not for my own code ?

−−−

Update:

I filed a bug on that issue but I didn't purse fixing it. Instead I wrote my own. It took about 20hrs of solid work. Probably only a 2 hours to write but 18 hours to test across 3 platforms and learn new issues. I found out ubuntu, osx, windows, and travis−ci's containers all behave differently in this area. I'm sure I didn't catch all the stuff and I still would like to write another 4−8 hours of tests but I need to move on and it's working so far.

Some things that were interesting is where to make the divisions, what features to put in. For example the most used library I linked to above, the buggy one, has options for ignoring dot files (files that start with .) as well as an option to give it a regular expression to filter (the one that didn't actually work). After writing mine I found another library that takes complex globs and has debouncing features.

I feel like those features should be layers. It's probably as little as one line of code to filter. Example:

  var watcher = new SimpleTreeWatcher(pathToWatch, {
    filter: (filepath) => { return path.basename(filepath)[0] !== '.'); },
  });

So why clutter the library with options when doing it this way keeps the library simple and let's people be as complex or simple as they want. It seems better to just provide a few examples of how to filter than have a bunch of options.

Similarly the debouncing can relatively easily be layered on top. I should add examples to the docs but at least I have something that's working for me so far.

Comments
Crunch in Games
CAs now get to decide who's on the Internet