Create immutable directory pipelines.
easy, one-off immutable directories!
Pure functions are a powerful concept. They allow you to, given an input, produce the same deterministic output, without side effects.
On the filesystem, this is hard to achieve. Filesystems are all about side effects! Consider creating a directory as a function, and then creating a file within it:
$ mkdir foobar
$ touch foobar/quux
If this was part of a script you used in, say, a build process, you might run into some problems:
mkdir foobar
is notfoobar/quux
for a different purpose? Each scriptmkdir foo; sleep 10; touch foo/quux
and, during those 10 seconds, another process didrm -rf foo
, the result would be different than if they hadn't.From this, we can say that a better solution would have the three inverse properties:
Enter ice-box: a module that manages a store of uniquely-named, immutable directories, and makes it easy to create new ones.
Let's say we have a build system that takes a directory and puts its contents
into a tarball. What might a script look like to do that, so we could invoke it
using node make-tar.js some-directory/
?
var icebox = require('ice-box')()
var fs = require('fs')
var path = require('path')
var tar = require('tar-fs')
var src = process.argv[2]
icebox(function (dst, done) {
tar
.pack(src)
.pipe(fs.createWriteStream(path.join(dst, 'result.tar')))
.on('finish', done)
}, function (err, finalDir) {
console.log(finalDir)
})
Running node make-tar.js some-directory/
will output
/home/sww/ice-box/8755ce4b-9ab0-c667-ea28-1f36bd0c8512
which contains the output file, result.tar
.
Much like UNIX pipes, this enables the creation of UNIX-like pipes: programs that consume a directory can produce a new immutable directory and output that.
Imagine we had a program that took a directory of JS files and packaged them for Electron before the tarball step:
var icebox = require('ice-box')('./ice-box')
var packager = require('electron-packager')
var src = process.argv[2]
icebox(function (dst, done) {
packager({
dir: src, // use the input dir, 'src'
arch: 'x64',
platform: 'linux',
out: dst, // use the output dir, 'dst'
tmpdir: false,
prune: true,
overwrite: true,
}, done)
}, function (err, finalDir) {
console.log(finalDir)
})
Now we could run this as just
$ node build-electron.js .
/home/sww/ice-box/8e3a47f8-f91d-a70b-692f-d0f54b730fb2
to get the electron-ready output, or it can be piped into make-tar.js
from
the above section to produce the final .tar
file!
$ node build-electron.js . | node make-tar.js
/home/sww/ice-box/a5339569-ae8f-4430-2dc1-a1a55340ea67
Now we have a directory with a tarball of the electron package!
Bonus: all intermediate steps are permanently cacheable, since they're immutable and permanent!
var iceBox = require('ice-box')
Creates a new function for adding new directories to an icebox. Both parameters are optional, and default to sane values.
outDir
(string) - The location to place the immutable output directories../ice-box
.tmpDir
(string) - The temporary location to create in-progress directoriesoutDir
.Creates a new directory for writing to.
work
is a function of the form function (dir, done) { ... }
. dir
is the
absolute path to the in-progress temporary directory. It has full write
permissions. done
is a function to call once you are done writing, to signify
that the directory can be "frozen" and placed in the icebox. If you pass in an
error (done(err)
) then the entire operation will abort cleanly.
done
is a function of the form function (dir) { ... }
. It is called once the
newly-frozen output directory is placed in the ice-box (outDir
from the above
section). path
is a string containing the absolute path to the frozen,
immutable, unique directory.
With npm installed, run
$ npm install ice-box
I was inspired by looking at how many codebases will use a many-step build process that involves transforming directories (source dir -> build dir -> packaged dir -> windows installer program), but suffer from side effects and shared global state. If build steps were interrupted the series of output directories would be inconsistent, hard to track down, etc. I really wanted to be able to make build and release pipelines that were as easy to reason about as UNIX pipes.
ISC