Sorry to rant on this more but sheesh! How much does it take?
So I want to watch the filesystem for changes. Unfortunately node's fs.watch
isn't OS independant. Some OSes return individual files. Other OSes just
return the parent folder changed. I'm sure there's some legit reason like perf
not to fix this for all devs but whatever. I assume there must be a library
that fixes it.
So I search NPM. Maybe I should be using Google to search NPM because NPMs
results are usually pretty poor but after searching a while I find this watch
module. It's got nearly 700k downloads in the last month and tons of
dependents so it must be good.
Given my prior experience the first thing I check is how many dependencies does it list?
Okay only 2 but they are both related to command line stuff. WTF does a library have command line dependencies? Oh, I see this library is conflated with a utility that watches for changes and launches a specified program anytime a file changes. Sigh... ?
Okay well let's look through the dependencies. It's not too deep. Fine, maybe I'll live with it if it works.
Ok I file an issue "Separate library from command line?" and move on.
A quick glance at the code and it doesn't appear to abstract the issues I need fixed but it's easy to test. So I write some test code
var watch = require('watch'); var dir = process.argv[2]; console.log("watching: ", dir); var options = { ignoreDotFiles: true, // - When true this option means that when the file tree is walked it will ignore files that being with "." filter: function() { return true; }, // - You can use this option to provide a function that returns true or false for each file and directory to decide whether or not that file/directory is included in the watcher. ignoreUnreadableDir: true, // - When true, this options means that when a file can't be read, this file is silently skipped. ignoreNotPermitted: true, // - When true, this options means that when a file can't be read due to permission issues, this file is silently skipped. ignoreDirectoryPattern: /node_modules/, // - When a regex pattern is set, e.g. /node_modules/, these directories are silently skipped. }; watch.createMonitor(dir, options, function(monitor) { monitor.on('created', function(f, s) { show("created", f, s ); }); monitor.on('removed', function(f, s, s2) { show("removed", f, s, s2); }); monitor.on('changed', function(f, s) { show("changed", f, s ); }); }); function show(event, f, s, n) { console.log(event, f); }
I run it and pass it my temp folder which has a ton of crap in it including lots of sub folders.
I edit temp/foo.txt
and see it print changed: temp/foo.txt
. I delete temp/foo.txt
and see it print removed: temp/foo.txt
. I create a temp/foo.txt
and see it print created: temp/foo.txt
.
So far so good. Let's test one more thing.
I look for a subfolder since I've only been changing stuff in the root of the
folder I'm watching. I just happen to pick temp/delme-eslint/node_modules/foo
. BOOM! It prints
created temp/delme-eslint/node_modules/abab created temp/delme-eslint/node_modules/abbrev created temp/delme-eslint/node_modules/acorn created temp/delme-eslint/node_modules/acorn-globals created temp/delme-eslint/node_modules/acorn-jsx created temp/delme-eslint/node_modules/amdefine created temp/delme-eslint/node_modules/ansi-escapes created temp/delme-eslint/node_modules/ansi-regex created temp/delme-eslint/node_modules/ansi-styles created temp/delme-eslint/node_modules/argparse created temp/delme-eslint/node_modules/array-differ created temp/delme-eslint/node_modules/array-union created temp/delme-eslint/node_modules/array-uniq created temp/delme-eslint/node_modules/arrify created temp/delme-eslint/node_modules/asn1 created temp/delme-eslint/node_modules/assert-plus created temp/delme-eslint/node_modules/async created temp/delme-eslint/node_modules/aws-sign2 created temp/delme-eslint/node_modules/aws4 created temp/delme-eslint/node_modules/balanced-match created temp/delme-eslint/node_modules/bl created temp/delme-eslint/node_modules/bluebird created temp/delme-eslint/node_modules/boolbase created temp/delme-eslint/node_modules/boom created temp/delme-eslint/node_modules/brace-expansion created temp/delme-eslint/node_modules/caller-path created temp/delme-eslint/node_modules/callsites created temp/delme-eslint/node_modules/caseless created temp/delme-eslint/node_modules/chalk created temp/delme-eslint/node_modules/cheerio created temp/delme-eslint/node_modules/cli-cursor created temp/delme-eslint/node_modules/cli-width created temp/delme-eslint/node_modules/code-point-at created temp/delme-eslint/node_modules/coffee-script created temp/delme-eslint/node_modules/color-convert created temp/delme-eslint/node_modules/colors created temp/delme-eslint/node_modules/combined-stream created temp/delme-eslint/node_modules/commander created temp/delme-eslint/node_modules/concat-map created temp/delme-eslint/node_modules/concat-stream created temp/delme-eslint/node_modules/core-util-is created temp/delme-eslint/node_modules/cryptiles created temp/delme-eslint/node_modules/css-select created temp/delme-eslint/node_modules/css-what created temp/delme-eslint/node_modules/cssom created temp/delme-eslint/node_modules/cssstyle created temp/delme-eslint/node_modules/d created temp/delme-eslint/node_modules/dashdash created temp/delme-eslint/node_modules/dateformat created temp/delme-eslint/node_modules/debug created temp/delme-eslint/node_modules/deep-is created temp/delme-eslint/node_modules/del created temp/delme-eslint/node_modules/delayed-stream created temp/delme-eslint/node_modules/doctrine created temp/delme-eslint/node_modules/dom-serializer created temp/delme-eslint/node_modules/domelementtype created temp/delme-eslint/node_modules/domhandler created temp/delme-eslint/node_modules/domutils created temp/delme-eslint/node_modules/ecc-jsbn created temp/delme-eslint/node_modules/entities created temp/delme-eslint/node_modules/es5-ext created temp/delme-eslint/node_modules/es6-iterator created temp/delme-eslint/node_modules/es6-map created temp/delme-eslint/node_modules/es6-set created temp/delme-eslint/node_modules/es6-symbol created temp/delme-eslint/node_modules/es6-weak-map created temp/delme-eslint/node_modules/escape-string-regexp created temp/delme-eslint/node_modules/escodegen created temp/delme-eslint/node_modules/escope created temp/delme-eslint/node_modules/eslint created temp/delme-eslint/node_modules/espree created temp/delme-eslint/node_modules/esprima created temp/delme-eslint/node_modules/esrecurse created temp/delme-eslint/node_modules/estraverse created temp/delme-eslint/node_modules/esutils created temp/delme-eslint/node_modules/event-emitter created temp/delme-eslint/node_modules/eventemitter2 created temp/delme-eslint/node_modules/exit created temp/delme-eslint/node_modules/exit-hook created temp/delme-eslint/node_modules/extend created temp/delme-eslint/node_modules/extsprintf created temp/delme-eslint/node_modules/fast-levenshtein created temp/delme-eslint/node_modules/figures created temp/delme-eslint/node_modules/file-entry-cache created temp/delme-eslint/node_modules/find-up created temp/delme-eslint/node_modules/findup-sync created temp/delme-eslint/node_modules/flat-cache created temp/delme-eslint/node_modules/foo created temp/delme-eslint/node_modules/forever-agent created temp/delme-eslint/node_modules/form-data created temp/delme-eslint/node_modules/generate-function created temp/delme-eslint/node_modules/generate-object-property created temp/delme-eslint/node_modules/getobject created temp/delme-eslint/node_modules/glob created temp/delme-eslint/node_modules/globals created temp/delme-eslint/node_modules/globby created temp/delme-eslint/node_modules/graceful-fs created temp/delme-eslint/node_modules/graceful-readlink created temp/delme-eslint/node_modules/grunt created temp/delme-eslint/node_modules/grunt-eslint created temp/delme-eslint/node_modules/grunt-legacy-log created temp/delme-eslint/node_modules/grunt-legacy-log-utils created temp/delme-eslint/node_modules/grunt-legacy-util created temp/delme-eslint/node_modules/har-validator created temp/delme-eslint/node_modules/has-ansi created temp/delme-eslint/node_modules/hoek created temp/delme-eslint/node_modules/hawk created temp/delme-eslint/node_modules/hooker created temp/delme-eslint/node_modules/htmlparser2 created temp/delme-eslint/node_modules/http-signature created temp/delme-eslint/node_modules/iconv-lite created temp/delme-eslint/node_modules/ignore created temp/delme-eslint/node_modules/inflight created temp/delme-eslint/node_modules/inherits created temp/delme-eslint/node_modules/inquirer created temp/delme-eslint/node_modules/is-fullwidth-code-point created temp/delme-eslint/node_modules/is-my-json-valid created temp/delme-eslint/node_modules/is-path-cwd created temp/delme-eslint/node_modules/is-path-in-cwd created temp/delme-eslint/node_modules/is-path-inside created temp/delme-eslint/node_modules/is-property created temp/delme-eslint/node_modules/is-resolvable created temp/delme-eslint/node_modules/is-typedarray created temp/delme-eslint/node_modules/isarray created temp/delme-eslint/node_modules/isstream created temp/delme-eslint/node_modules/jodid25519 created temp/delme-eslint/node_modules/js-yaml created temp/delme-eslint/node_modules/jsbn created temp/delme-eslint/node_modules/jsdom created temp/delme-eslint/node_modules/json-schema created temp/delme-eslint/node_modules/json-stable-stringify created temp/delme-eslint/node_modules/json-stringify-safe created temp/delme-eslint/node_modules/jsonify created temp/delme-eslint/node_modules/jsonpointer created temp/delme-eslint/node_modules/jsprim created temp/delme-eslint/node_modules/levn created temp/delme-eslint/node_modules/load-grunt-tasks created temp/delme-eslint/node_modules/lodash created temp/delme-eslint/node_modules/lru-cache created temp/delme-eslint/node_modules/mime-db created temp/delme-eslint/node_modules/mime-types created temp/delme-eslint/node_modules/minimatch created temp/delme-eslint/node_modules/minimist created temp/delme-eslint/node_modules/mkdirp created temp/delme-eslint/node_modules/ms created temp/delme-eslint/node_modules/multimatch created temp/delme-eslint/node_modules/mute-stream created temp/delme-eslint/node_modules/node-uuid created temp/delme-eslint/node_modules/nopt created temp/delme-eslint/node_modules/nth-check created temp/delme-eslint/node_modules/number-is-nan created temp/delme-eslint/node_modules/nwmatcher created temp/delme-eslint/node_modules/oauth-sign created temp/delme-eslint/node_modules/object-assign created temp/delme-eslint/node_modules/once created temp/delme-eslint/node_modules/onetime created temp/delme-eslint/node_modules/optionator created temp/delme-eslint/node_modules/os-homedir created temp/delme-eslint/node_modules/parse5 created temp/delme-eslint/node_modules/path-exists created temp/delme-eslint/node_modules/path-is-absolute created temp/delme-eslint/node_modules/path-is-inside created temp/delme-eslint/node_modules/pify created temp/delme-eslint/node_modules/pinkie created temp/delme-eslint/node_modules/pinkie-promise created temp/delme-eslint/node_modules/pkg-up created temp/delme-eslint/node_modules/pluralize created temp/delme-eslint/node_modules/prelude-ls created temp/delme-eslint/node_modules/process-nextick-args created temp/delme-eslint/node_modules/progress created temp/delme-eslint/node_modules/pseudomap created temp/delme-eslint/node_modules/qs created temp/delme-eslint/node_modules/read-json-sync created temp/delme-eslint/node_modules/readable-stream created temp/delme-eslint/node_modules/readline2 created temp/delme-eslint/node_modules/request created temp/delme-eslint/node_modules/require-uncached created temp/delme-eslint/node_modules/resolve created temp/delme-eslint/node_modules/resolve-from created temp/delme-eslint/node_modules/resolve-pkg created temp/delme-eslint/node_modules/restore-cursor created temp/delme-eslint/node_modules/rimraf created temp/delme-eslint/node_modules/run-async created temp/delme-eslint/node_modules/rx-lite created temp/delme-eslint/node_modules/sax created temp/delme-eslint/node_modules/shelljs created temp/delme-eslint/node_modules/sigmund created temp/delme-eslint/node_modules/slice-ansi created temp/delme-eslint/node_modules/sntp created temp/delme-eslint/node_modules/source-map created temp/delme-eslint/node_modules/sprintf-js created temp/delme-eslint/node_modules/sshpk created temp/delme-eslint/node_modules/string-width created temp/delme-eslint/node_modules/string_decoder created temp/delme-eslint/node_modules/stringstream created temp/delme-eslint/node_modules/strip-ansi created temp/delme-eslint/node_modules/strip-json-comments created temp/delme-eslint/node_modules/supports-color created temp/delme-eslint/node_modules/symbol-tree created temp/delme-eslint/node_modules/table created temp/delme-eslint/node_modules/text-table created temp/delme-eslint/node_modules/through created temp/delme-eslint/node_modules/tough-cookie created temp/delme-eslint/node_modules/tr46 created temp/delme-eslint/node_modules/tryit created temp/delme-eslint/node_modules/tunnel-agent created temp/delme-eslint/node_modules/tv4 created temp/delme-eslint/node_modules/tweetnacl created temp/delme-eslint/node_modules/type-check created temp/delme-eslint/node_modules/typedarray created temp/delme-eslint/node_modules/underscore created temp/delme-eslint/node_modules/underscore.string created temp/delme-eslint/node_modules/user-home created temp/delme-eslint/node_modules/util-deprecate created temp/delme-eslint/node_modules/verror created temp/delme-eslint/node_modules/webidl-conversions created temp/delme-eslint/node_modules/whatwg-url-compat created temp/delme-eslint/node_modules/which created temp/delme-eslint/node_modules/wordwrap created temp/delme-eslint/node_modules/wrappy created temp/delme-eslint/node_modules/write created temp/delme-eslint/node_modules/xml-name-validator created temp/delme-eslint/node_modules/xregexp created temp/delme-eslint/node_modules/xtend created temp/delme-eslint/node_modules/yallist
BUG!!! Really, I used it for 5 minutes ran into bug and yet 700k downloads and a ton of dependents are using this buggy library. Sigh....?
Now the question is (a) fix it or 🍺 write my own.
Yes I know there are good arguments for fixing it. Just last month 700k downloads = 700k fixed downloads next month probably. But of course it possible means arguing with the author, jumping through their hoops. And, I still don't know if it solves the problem I want solved.
On top of that I there's at least one thing I wanted to re−design I think. Not only do I want the tree to be watched, I want to know what's in the tree. It kind of seems like that should be one operation. In other words I don't want to call some other function to scan the tree to get the list of all things in it and then separately call another function to watch the tree. If I do that then a file that's created between the first scan and the watch will never show up. It will also be twice as slow AFAICT as the the watch has to scan as well. So, now yet another discussion to be haggled over if I decide to fix vs do it myself.
The worst part is the thing I wanted to do I thought would take all of 20 minutes if that functionality I was looking for existed. Now will also probably take 8x as long to fix vs do it myself on top of writing the feature I was originally setting out to d. Probably not fair but (a) no negociation and 🍺 I feel more obligated to right tests for fixing but not for my own code ?
−−−
Update:
I filed a bug on that issue but I didn't purse fixing it. Instead I wrote my own. It took about 20hrs of solid work. Probably only a 2 hours to write but 18 hours to test across 3 platforms and learn new issues. I found out ubuntu, osx, windows, and travis−ci's containers all behave differently in this area. I'm sure I didn't catch all the stuff and I still would like to write another 4−8 hours of tests but I need to move on and it's working so far.
Some things that were interesting is where to make the divisions, what features
to put in. For example the most used library I linked to above, the buggy one,
has options for ignoring dot files (files that start with .
) as well as an option to give it a regular expression to filter (the one that
didn't actually work). After writing mine I found another library that takes complex globs and has debouncing features.
I feel like those features should be layers. It's probably as little as one line of code to filter. Example:
var watcher = new SimpleTreeWatcher(pathToWatch, { filter: (filepath) => { return path.basename(filepath)[0] !== '.'); }, });
So why clutter the library with options when doing it this way keeps the library simple and let's people be as complex or simple as they want. It seems better to just provide a few examples of how to filter than have a bunch of options.
Similarly the debouncing can relatively easily be layered on top. I should add examples to the docs but at least I have something that's working for me so far.