Any code submitted to this project is checked with the pre-commit framework. To make sure that your code will pass the checks, you can execute the pre-commit checks locally before "git pushing" your code.
You will need the following:
Then you should be able to setup your environment with:
make install
source .venv/bin/activate
make install-pre-commit
make pre-commit-run
You can also install the pre-commit tool so that any commit will be checked automatically.
buildbot.mariadb.org is our continuous integration testing platform, based on the Buildbot.net open-source testing framework.
This installation uses a custom version 2.7.1 at time of initial deployment running with Python 3 on a Ubuntu 18.04 decently tuned (performance wise) machine for a more speedy, modern and responsive UI in contrast to the existing Buildbot v0.8.8 running at buildbot.askmonty.org.
This directory contains all the configuration needed to deploy a new installation of Buildbot master to a new machine, as well as all the information needed to add new builder configurations, extra worker machines to extend our building capability and platform support and specific steps you can take to reproduce the exact environment in which a build failure occurred so you can debug without much effort.
Quick layout of the main components of the current directory structure:
buildbot.mariadb.org/
├── buildbot.tac
├── dockerfiles
│ ├── buildbot.tac
│ └── ...
│ └── ecofiles
│ ├── installdb.sh
│ ├── test-php.sh
│ └── test-pymysql.sh
├── master.cfg
├── master-private.cfg-sample
├── master-web
│ ├── buildbot.tac
│ ├── master.cfg
│ ├── static
│ │ └── (sponsor logos)
│ ├── templates
│ │ ├── console_view
│ │ │ └── console.jade
│ │ ├── grid.html
│ │ ├── grid_view
│ │ │ └── grid.jade
│ │ ├── home.jade
│ │ └── sponsor.html
│ └── twistd.pid
├── README.md
├── sponsor.py
├── static
└── util
├── buildbot-master.service
├── buildbot-worker.service
└── nginx.conf
Using Buildbot's DockerLatentWorker to connect to docker instances, the worker runs inside the Docker image, and on a build trigger, master connects to Docker first via docker-py , instantiates the image and then connects using Buildbot usual Worker API to do the heavy lifting. Upon build completion, the image is stopped and no left-overs are left behind.
Starting with version 1.0, Buildbot interface changed in a significant and positive manner. The new UI is a fresh rewrite with responsive, lazy loading and modern web conveniences that makes it easy to follow running builds as well as convenient to browse build history.
Grid View and MTRLogObserver plugin are usable out of the box and behave in a similar fashion as in the current Buildbot instance.
Debugging build failures with Docker is facilitated by the ease of replicating the actual build environment where a particular builder failed. First step is to identify on which platform the failure occurred and what's the associated Dockerfile. If you're not familiar with Docker, start by reading the Docker overview and Docker getting started documentation.
To repeat a build on buildbot, first identify the platform and look for the associated dockerfile in the dockerfiles/ directory. To create a container for Ubuntu 18.04 for e.g. you would use the docker build command like this:
docker build -t ubuntu-1804 --file ubuntu1804.dockerfile .
After a successful image build, you can run a container and execute bash using that image with docker run:
docker run -it ubuntu-1804 bash
After this step you can follow the usual MariaDB development setup instructions, i.e. clone the repo and start a build. To specifically reproduce what a particular Buildbot worker did, look into the master.cfg configuration file what addStep() has been defined for that worker, and copy paste the contents of those into the session you are running in the Docker image.
Deploying a new master (since Buildbot > 1.0 supports multi-master configuration):
/srv/buildbot/master
buildbot create-master -r /srv/buildbot/master
buildbot upgrade-master /srv/buildbot/master
buildbot start /srv/buildbot/master
To upgrade Buildbot to latest version:
sudo systemctl stop buildbot-master
sudo pip3 install -U buildbot buildbot-grid-view buildbot-waterfall-view buildbot-console-view buildbot-wsgi-dashboards buildbot-www
buildbot upgrade-master /srv/buildbot/master
buildbot cleanupdb /srv/buildbot/master
sudo systemctl start buildbot-master
To upgrade Buildbot-worker to latest version:
sudo pip3 install -U buildbot-worker
The master configuration can be site specific and should be customized according to the needs and purpose for a particular master. What follows is a description of our current master configuration.
Master.cfg is a Python script and defines the behavior of Buildbot. These are the main sections of interest:
Also, master.cfg sources a private config file that is not included in this repo, namely master-private.cfg. This file should include a Python dictionary with private information like worker passwords, database url and anything else not deemed for public disclosure.
For changes to the master.cfg to take effect, the server needs to be reloaded. While doing this, please keep an eye on the logs with tail -f /srv/buildbot/master/twistd.log
. The Buildbot master is managed by our own rolled systemd file. To invoke a reload run sudo systemctl reload buildbot-master
. The other usual systemctl are also available (status
, restart
). Before the reload, consider testing the config with buildbot checkconfig
.
There's currently two types of workers that we use. Normal Buildbot workers, running bare-bones builds and can be added using c['workers'].append(mkWorker("bm-bbw1-ubuntu1804"))
. The password entry needs to be defined in master-private.cfg
. Note that the worker name cannot be a hostname with dots. Only alphanumeric and dashes are allowed in the worker name.
For instructions on setting up a Buildbot Worker see http://docs.buildbot.net/latest/manual/installation/worker.html.
To add a buildbot worker use the following commands:
useradd buildbot
su - buildbot
python3 -m venv buildbot-worker
source buildbot-worker/bin/activate
pip install buildbot-worker
curl -o buildbot.tac https://raw.githubusercontent.com/MariaDB/buildbot/main/dockerfiles/buildbot.tac
#set buildmaster, port and worker user and password
buildbot-worker start
The other type of worker we use is DockerLatentWorker. It is useful for running tests inside single use, disposable container Linux instances. We are currently aiming to use it for most of our Linux builds.
Adding a Docker based worker is as simple as providing the IP address to the Docker server and everything else can be handled on the master side.
We currently have 2 Docker instances:
do-ubuntu-1804-bbw1-docker
: DigitalOcean worker 1 running Debian/Ubuntu (including main tarbake) buildsdo-ubuntu-1804-bbw2-docker
: DigitalOcean worker 2 running Fedora, CentOS, OpenSuSe docker buildsThis separation is purely conventional and can be arbitrarily implemented subject to available resources on each Docker host.
Sample DockerLatentWorker configuration:
c['workers'].append(worker.DockerLatentWorker("do-bbw1-docker-tarball", None,
docker_host='tcp://10.0.0.46:2375',
dockerfile=open("dockerfiles/debian.dockerfile").read(),
masterFQDN='buildbot.mariadb.org',
hostconfig={ 'shm_size':'500M' },
volumes=['/srv/buildbot/ccache:/mnt/ccache'],
properties={ 'jobs':'2' }))
We currently define a build matrix via supportedPlatforms which specified which builders should we run for a particular change after figuring out which branch it is based on. For this particular reason, all branch names should follow the following naming scheme: <version>-<suffix>
or <prefix>-<version>-<suffix>
.
Examples
10.4
- main development branch that becomes the next 10.4.x release10.0-galera
- main development branch that becomes the next 10.0.x Galera releasebb-10.3-feature
- feature branch targeting next 10.3.x release, tested by Buildbot normallyhf-5.5-fixbug
- hotfix branch targeting next 5.5.x release, not tested by Buildbot as it lacks bb-*
This project would not have gotten off the ground without the help and support of Rasmus Johansson. We thank him for his many contributions to the community, and remember him for his kindness, level headedness, and as an example for us all.