Purely functional build system and package manager
MIT License
Bramble is a work-in-progress functional build system inspired by Nix.
Bramble is a functional build system that intends to be a user-friendly, robust, and reliable way to build software. Here are some if the ideas behind it:
bramble.toml
and bramble.lock
file that track dependencies and other metadata needed to build the project reliably.sandbox-exec
is used on macOS. Unclear how this will work beyond Linux and macOS.load("github.com/maxmcd/busybox")
in a build file or bramble build bitbucket.org/maxm/foo:foo
from the command line. Dependencies are project specific.Many things are broken, would not expect this to work or be useful yet. The list of features seems to be solidifying so hopefully things will be more organized soon. If you have ideas or would like to contribute please open an issue.
Install with go get github.com/maxmcd/bramble
or download a recent binary release. Linux is the only supported OS at the moment. macOS support should be coming soon, others much later.
In order for rootless/userspace sandboxing to work "User Namespaces" must be compiled and enabled in your kernel:
CONFIG_USER_NS=y
is set in your kernel configuration (normally found in /proc/config.gz
)
$ cat /proc/config.gz | gzip -d | grep CONFIG_USER_NS
CONFIG_USER_NS=y
echo 1 > /proc/sys/kernel/unprivileged_userns_clone
echo 28633 > /proc/sys/user/max_user_namespaces
Here's an example project that downloads busybox and uses it to create a script that says "Hello world!".
./bramble.toml
[package]
name = "github.com/maxmcd/hello-example"
version = "0.0.1"
./default.bramble
def fetch_url(url):
"""
fetch_url is a handy wrapper around the built-in fetch_url builder. It just
takes the url you want to fetch.
"""
return derivation(name="fetch-url", builder="fetch_url", env={"url": url})
def fetch_busybox():
return fetch_url("https://brmbl.s3.amazonaws.com/busybox-x86_64.tar.gz")
def busybox():
"""
busybox downloads the busybox binary and copies it to an output directory.
Symlinks are then created for every command busybox supports.
"""
return derivation(
name="busybox",
builder=fetch_busybox().out + "/busybox-x86_64",
args=["sh", "./script.sh"],
sources=files(["./script.sh"]),
env={"busybox_download": fetch_busybox()},
)
def hello_world():
bb = busybox()
PATH = "{}/bin".format(bb.out)
return derivation(
"say_hello_world",
builder=bb.out + "/bin/sh",
env=dict(PATH=PATH, busybox=bb.out),
args=[
"-c",
"""set -e
mkdir -p $out/bin
touch $out/bin/say-hello-world
chmod +x $out/bin/say-hello-world
echo "#!$busybox/bin/sh" > $out/bin/say-hello-world
echo "$busybox/bin/echo Hello World!" >> $out/bin/say-hello-world
# try it out
$out/bin/say-hello-world
""",
],
)
./script.sh
set -e
$busybox_download/busybox-x86_64 mkdir $out/bin
$busybox_download/busybox-x86_64 cp $busybox_download/busybox-x86_64 $out/bin/busybox
cd $out/bin
for command in $(./busybox --list); do
./busybox ln -s busybox $command
done
If you copy these files into a directory you can build it like so:
$ bramble build ./:hello_world
bramble path directory doesn't exist, creating
✔ busybox-x86_64.tar.gz - 332.830943ms
✔ busybox - 88.136237ms
✔ url_fetcher.tar.gz - 424.225793ms
✔ url_fetcher - 46.129651ms
✔ fetch-url - 313.461369ms
✔ busybox - 152.799168ms
✔ say_hello_world - 29.742436ms
Huh, what's confusing. What are all these builds in our output? Bramble needs to pull some dependencies itself in order to fetch and unpack the url provided. This is so that you can pin the internal functionality of bramble to a specific version. This isn't all hooked up yet, but for the moment you can see the internals being built in the build output.
Now that the build is complete you'll see that a bramble.toml
file has been written to the project directory.
./bramble.lock
[URLHashes]
"basic_fetch_url https://brmbl.s3.amazonaws.com/busybox-x86_64.tar.gz" = "uw5ichj6dhcccmcts6p7jq6etzlh5baf"
"basic_fetch_url https://brmbl.s3.amazonaws.com/url_fetcher.tar.gz" = "p2vbvabkdqckjlm43rf7bfccdseizych"
"fetch_url https://brmbl.s3.amazonaws.com/busybox-x86_64.tar.gz" = "uw5ichj6dhcccmcts6p7jq6etzlh5baf"
These are the three archives we had to download in order for the build to run. This will ensure that if we ever download these files again, the contents will match what we expect them to.
We can use bramble run
to run the resulting script.
$ bramble run ./:hello_world say-hello-world
Hello World!
That's it! Your first build and run of a Bramble derivation.
This is a reference manual for Bramble. Bramble is a work-in-progress. I started writing this spec to solidify the major design decisions, but everything is still very much in flux. There are scattered notes in the notes folder as well.
Bramble is a functional build system and package manager. Bramble is project-based, when you run a build or run a build output it must always be done in the context of a project.
Here are three example use-cases that Bramble hopes to support and support well.
bramble build
or bramble run
within a project will use the bramble.toml
, bramble.lock
and source files to fetch dependencies from a cache or build them from source. Additionally bramble run
commands are sandboxed by default, so Bramble should be a good choice to run unfamiliar software.bramble run github.com/username/project:function binary
will pull down software from that repo, build it, and run it within a sandbox on a local system. bramble run
is sandboxed by default and aims to provide a safe and reproducible way to run arbitrary software on your system.bramble run
call can be packaged up into a container containing only the bare-minimum dependencies for that program to run.Every Project has a bramble.toml
file that includes configuration information and a bramble.lock
file that includes hashes and other metadata that are used to ensure that the project can be built reproducibly.
[package]
name = "github.com/maxmcd/bramble"
A project must include a module name. If it's expected that this project is going to be importable as a module then the module name must match the location of the repository where the module is stored.
[URLHashes]
"https://brmbl.s3.amazonaws.com/busybox-x86_64.tar.gz" = "2ae410370b8e9113968ffa6e52f38eea7f17df5f436bd6a69cc41c6ca01541a1"
The bramble.lock
file stores hashes so that "fetch" builders like "fetch_url" and "fetch_git" can ensure the contents they are downloading have the expected content. This file will also include various hashes to ensure dependencies and sub-dependencies can be reliably re-assembled.
bramble build
bramble build [options] <module or path>:<function>
bramble build [options] <path or path>
bramble build [options] <path>/...
The build
command is used to build derivations returned by bramble functions. Calling build
with a module location and function will call that function, take any derivations that are returned, and build that derivation and its dependencies.
Here are some examples:
bramble build ./tests/basic:self_reference
bramble build github.com/maxmcd/bramble:all github.com/maxmcd/bramble:bash
bramble build github.com/username/repo/subdirectory:all
bramble build github.com/maxmcd/bramble/lib
bramble build github.com/maxmcd/bramble/...
bramble build github.com/maxmcd/bramble/tests/...
bramble build ./...
bramble run
bramble run [options] <module or path>:<function> [args...]
bramble ls
bramble ls <path>
Calls to ls
will search the current directory for bramble files and print their public functions with documentation. If an immediate subdirectory has a default.bramble
documentation will be printed for those functions as well.
$ bramble ls
Module: github.com/maxmcd/bramble/
def print_simple()
def bash()
def all()
Module: github.com/maxmcd/bramble/lib
"""
Lib provides various derivations to help build stuff
"""
def cacerts()
"""cacerts provides known certificate authority certificates to verify TLS connections"""
def git()
def git_fetcher()
def git_test()
def zig()
def busybox()
bramble repl
repl
opens up a read-eval-print-loop for interacting with the bramble config language. You can make derivations and call other built-in functions. The repl has limited use because you can't build anything that you create, but it's a good place to get familiar with how the built-in modules and functions work.
bramble shell
shell
takes the same arguments as bramble build
but instead of building the final derivation it opens up a terminal into the build environment within a build directory with environment variables and dependencies populated. This is a good way to debug a derivation that you're building.
bramble gc
gc
searches for all known projects (TODO: link to what "known projects" means), runs all of their public functions and calculates what derivations and configuration they need to run. All other information is deleted from the store and project configurations.
Bramble uses starlark for its configuration language. Starlark generally a superset of Python, but has some differences that might trip up more experienced Python users. When in doubt would be sure to check out the lamnguage spec.
Here is a typical bramble file:
# Load statements
load("github.com/maxmcd/bramble/lib/stdenv")
load("github.com/maxmcd/bramble/lib")
load("github.com/maxmcd/bramble/lib/std")
def fetch_a_url():
return std.fetch_url("https://maxmcd.com/")
def step_1():
bash = "%s/bin/bash" % stdenv.stdenv()
# A derivation, the basic building block of our builds
return derivation(
"step_1",
builder=bash,
env=dict(bash=bash),
# Use of the `files()` builtin
sources=files(["./step1.sh"]),
args=["./step1.sh"],
)
Bramble source files are stored in files with a .bramble
file extension. Files can reference other bramble files by using their module names. This project has the module name github.com/maxmcd/bramble
so if I want to access a file at ./tests/basic.bramble
I can import it with load("github.com/maxmcd/bramble/tests/basic")
. Relative imports aren't supported.
The default.bramble
filename is special. If a directory has a default.bramble
in it then we can import that directory as a package and all functions in the default.bramble
. In the above example, if the file was called default.bramble
instead of basic.bramble
we could import it with load("github.com/maxmcd/bramble/tests")
.
If you call load("github.com/maxmcd/bramble/tests")
at the top of a bramble file a new global variable named tests
will be loaded into the program context. tests
will have an attribute for all global variables in the default.bramble
file unless they begin with an underscore.
# ./tests/default.bramble
def foo():
print("hi")
def _bar():
print("hello")
# In a `bramble repl`
>>> load("github.com/maxmcd/bramble/tests")
>>> tests
<module "github.com/maxmcd/bramble/tests">
>>> dir(tests)
["foo"]
>>> tests.foo()
hi
>>> tests._bar()
Traceback (most recent call last):
<stdin>:1:6: in <expr>
Error: module has no ._bar field or method
derivation(name, builder, args=[], sources=[], env={}, outputs=["out"], platform=sys.platform)
Derivations are the basic building block of a bramble build. Every build is a graph of derivations. Everything that is built has a derivation and has dependencies that are derivations.
A derivation name
is required to help with visibility and debugging. It's helpful to have derivation names be unique in a project, but this is not an enforced requirement.
A builder
can be on of the default built-ins: ["fetch_url", "fetch_git", "derivation_output"]
or it can point to an executable that will be used to build files.
args
are the arguments that as passed to the builder. env
defines environment variables that are set within the build environment. Bramble detects dependencies by scanning for derivations referenced within a derivation. builder
, args
and env
are the only parameters that can reference other derivations, so you can be sure that if a derivation isn't referenced in one of those parameters that it won't be available to the build.
sources
contains references to source files needed for this derivation. Use the files
builtin to populate sources for a given derivation.
outputs
defines this Derivation's outputs. The default is for a derivation to have a single output called "out", but you can have one or more output with any name. After a derivation is created, you can reference it's outputs as attributes. If you cast a derivation to a string it returns a reference to the default output.
>>> b = derivation("hi", "ho")
>>> b
{{ soen6obfffrahna6ojyc2cgxjx7jcmhv:out }}
>>> b.out
"{{ soen6obfffrahna6ojyc2cgxjx7jcmhv:out }}"
>>> c = derivation("hi", "ho", outputs=["a", "b", "c"])
>>> c
{{ lvliebpnk6lcalc3sdsvfbrzwlamb4qo:a }}
>>> c.a
"{{ lvliebpnk6lcalc3sdsvfbrzwlamb4qo:a }}"
>>> c.b
"{{ lvliebpnk6lcalc3sdsvfbrzwlamb4qo:b }}"
>>> c.c
"{{ lvliebpnk6lcalc3sdsvfbrzwlamb4qo:c }}"
>>> "{}/bin/bash".format(c)
"{{ lvliebpnk6lcalc3sdsvfbrzwlamb4qo:a }}/bin/bash"
>>> "{}/bin/bash".format(c.b)
"{{ lvliebpnk6lcalc3sdsvfbrzwlamb4qo:b }}/bin/bash"
platform
denotes what platform this derivation can be built on. If the specific platform is available on the current system the derivation will be built.
The run function defines the attributes for running a program from a derivation output. If a call to a bramble function returns a run command that run command and parameters will be executed.
run(derivation, cmd, args=[], paths=[], read_only_paths=[], hidden_paths=[], network=False)
The test command creates a test. Any call to the test function will register a test that can be run later. Calls to bramble test
will run all tests in that directory and it's children. Calls to a specific bramble function like bramble test ./tests:first
will run any test functions that are called during the function call.
test(derivation, args=[])
>>> sys
<module "sys">
>>> dir(sys)
["arch", "os", "platform"]
>>> sys.arch
"amd64"
>>> sys.os
"linux"
>>> sys.platform
"linux-amd64"
>>> dir(assert)
["contains", "eq", "fail", "fails", "lt", "ne", "true"]
>>> assert.contains("hihihi", "hi")
>>> assert.contains("hihihi", "how")
Traceback (most recent call last):
<stdin>:1:16: in <expr>
assert.star:30:14: in _contains
Error: hihihi does not contain how
files(include, exclude=[], exclude_directories=True, allow_empty=True)
files
searches for source files and returns a mutable list.
Bramble builds all derivations within a sandbox. There are OS-specific sandboxes that try and provide similar functionality.
A tree of derivations is assembled to build. The tree is walked, compiling dependencies first, until all derivations are built. If a derivation has already been built (TODO: or is available in a remote store) it is skipped.
When building a specific derivation the steps are as follows:
/home/maxm/bramble/bramble_store_padding/bramble_/bramble_build_directory941760171
./home/maxm/bramble/bramble_store_padding/bramble_/bramble_build_directory451318742/
./home/bramble/bramble/bramble_store_padding/bramb/
, so we must replace it with the store path (of equal length) used in this system.builder
, args
, and env
attributes are taken and used to run a sandbox. The builder
program is run and args
are passed to that program. env
values are loaded as environment variables in alphabetical order.$out
might have value /home/maxm/bramble/bramble_store_padding/bramble_/bramble_build_directory451318742/
./home/bramble/bramble/bramble_store_padding/bramb/
. This helps ensure outputs hash the same on different systems.The derivation_output
derivation outputs a new derivation graph. This graph will be merged with the existing build graph and the build will continue. There are two rules with this builder:
derivation_output
. If a derivation uses the builder derivation_output
it must not output any derivations that use that builder. This will likely be supported in the future but is currently disallowed out of caution.derivation_output
must only have the default output "out" and the updated derivation graph must also only have a single outputted derivation that has a single default output. When derivation_output
is built it replaces a node in the build graph with a new graph. Any references to that old node must be overwritten with references to the new output derivation. In order to ensure that replacement is trivial we must ensure that the old node and the new node have identical output structure.When a derivation_output
is called the resulting derivation graph is written to bramble.lock
so that the output is not rebuilt on other systems.