LaMachine - A software distribution of our in-house as well as some 3rd party NLP software - Virtual Machine, Docker, or local compilation/installation script
GPL-3.0 License
IMPORTANT NOTE: LaMachine is end-of-life and deprecated. There will be no further development and its usage is no longer recommended. See this post for reasons and alternative solutions
LaMachine is a unified software distribution for Natural Language Processing. We integrate numerous open-source NLP tools, programming libraries, web-services, and web-applications in a single Virtual Research Environment that can be installed on a wide variety of machines.
The software included in LaMachine tends to be highly specialised and generally depends on a lot of other interdependent software. Installing all this software can be a daunting task, compiling it from scratch even more so. LaMachine attempts to make this process easier by offering pre-built recipes for a wide variety of systems, whether it is on your home computer or whether you are setting up a dedicated production environment, LaMachine will safe you a lot of work.
We address various audiences; the bulk of the software is geared towards data scientists who are not afraid of the command line and some programming. We give you the instruments and it is up to you to yield them. However, we also attempt to accommodate researchers that require more high-level interfaces by incorporating webservices and websites that expose some of the functionality to a larger audience.
To build your own LaMachine instance, in any of the possible flavours, or to download a pre-built image, open a terminal on your Linux, BSD or MacOS system and run the following command:
bash <(curl -s https://raw.githubusercontent.com/proycon/LaMachine/master/bootstrap.sh)
This will prompt you for some questions on how you would like your LaMachine installation and allows you to include precisely the software you want or need and ensures that all is up to date. A screenshot is shown at the end of this subsection.
Are you on Windows 10 or 2016? Then you need to run this command in the Windows Linux subsystem, we do not support Windows natively. To do this you must first install the Linux Subsystem with a distribution of your choice (we recommend Ubuntu) from the Microsoft Store. Follow the instructions here. Alternatively, you may want to choose for a pre-built Virtual Machine image as explained in installation path C.
Building LaMachine can take quite some time, depending also on your computer's resources, internet connection, and the amount of software you selected to install. Half an hour to an hour is a normal build time. The bootstrap script alternatively also offers the option to download pre-built images (installation path B & C).
We regularly build a basic LaMachine image and publish it to Docker Hub. The above installation path A also offers access to this, but you may opt to do it directly:
To download and use it, run:
docker pull proycon/lamachine
docker run -p 8080:80 -h latest -t -i proycon/lamachine
This requires you to already have Docker installed and running on your system.
The pre-built image contains the stable version with only a basic set of common software rather than the full set, run lamachine-add
inside the container to select extra software to install. Alternatively, other specialised LaMachine builds may be available
on Docker Hub.
If you want another release, specify its tag explicitly:
docker pull proycon/lamachine:develop
docker run -p 8080:80 -h develop -t -i proycon/lamachine:develop
We regularly build a basic LaMachine image and publish it to the Vagrant Cloud. The above installation path A also offers (simplified) access to this (except on Windows), but you may opt to do it directly.
To download and use a LaMachine prebuilt image:
cd
); this will be the base folder, files inside will be shared within the VM under/vagrant
wget https://raw.githubusercontent.com/proycon/LaMachine/master/Vagrantfile.prebuilt.erb
vagrant init --template Vagrantfile.prebuilt.erb proycon/lamachine
from the terminal.Vagrantfile
in a text editor and change the memory and CPU options to suit your system (the more resourcesvagrant up
from the terminal to boot your VMvagrant ssh
from the terminal to connect to the VMThe pre-built image contains only a basic set of common software rather than the full set, run lamachine-stable-update --edit
inside the virtual machine to select extra software to install.
To stop the VM when you're done, run: vagrant halt
. Next time, navigate to the same base folder in your terminal and run vagrant up
and vagrant ssh
again.
LaMachine includes a wide variety of open-source NLP software. You can select which software you want to include during the installation procedure (or any subsequent update).
Note that some software may not be available on certain platforms/distributions (most notably macOS).
For a verbose list of installed software and its metadata, run lamachine-list
once you are inside your LaMachine
installation. For more information regarding software metadata, check the corresponding section in the the contributor
documentation.
If you enabled and started the webserver in LaMachine, then you have access to a rich portal page giving an overview of all installed software and providing access to any software with a web-based interface. This portal is powered by Labirinto.
LaMachine is open for contributions by other software projects, please read the contributor documentation.
LaMachine can be installed in multiple flavours:
In all cases, the installation is mediated through Ansible, providing a level of abstraction over whatever underlying technology is employed. Containerisation uses Docker or LXD. Virtualisation is made possible through Vagrant and VirtualBox. The local installation variant uses virtualenv with some custom extensions.
Initially, the user executes a bootstrap.sh
script that acts as a single point of entry for all flavours. It will
automatically download LaMachine and create the necessary configuration files for your LaMachine build, guiding you
through all the options. It will eventually invoke a so-called ansible playbook that executes installation steps for all
of the individual software projects included in LaMachine, depending on your distribution and flavour.
LaMachine uses Debian as primary Linux distribution (for virtualisation and containerisation), we support the distributions/platforms listed below for a native installation of LaMachine (i.e. compiled against the libraries of that distribution). We distinguish three categories of support (and for all we only support the x86-64 architecture):
Gold support - All participating software should work on these platforms and things are tested frequently.
Silver support - Most software should work.
Bronze support - Certain software is known not to work and/or things are more prone to breakage and have not been tested.
Deprecated:
Unsupported (not exhaustive!) - We can not support these because our effort reached its limits:
Note that this concerns the platforms LaMachine runs on natively or on which you can bootstrap your own build (installation path A). The options for host platforms for simply running a pre-built LaMachine Virtual Machine or Docker container, are much larger, and also include Windows (see installation paths B & C).
In addition to a flavour, users can opt for one of three versions of LaMachine:
Read more about the technical details in the contributor documentation.
How to start LaMachine differs a bit depending on your flavour.
Run the generated activation script to activate the local environment (here we assume your LaMachine VM is called stable!):
source lamachine-stable-activate
or lamachine-stable-activate
, this script should be located in your~/bin
directory.If you built your own LaMachine you have various scripts at your disposal (here we assume your LaMachine VM is called stable! The script names will be different for other names, replace as needed):
lamachine-stable-start
to start the VMlamachine-stable-connect
to connect to a running VM and obtain a command line shell (over ssh)lamachine-stable-stop
to stop the VMlamachine-stable-destroy
to completely delete the VM againlamachine-stable-activate
is a shortcut that starts the VM and connects automatically, and stops the VM when youIf you used a prebuilt image you have to invoke vagrant
yourself from the proper directory where you did vagrant init proycon/lamachine:stable
:
vagrant up
to start the VMvagrant halt
to stop the VMvagrant ssh
to connect to the VM and obtain a command line shellvagrant destroy
to remove the VMCommand line access to your LaMachine Virtual Machine through vagrant or lamachine-*-connect
should be passwordless, other methods
may require a login; use username vagrant
and password vagrant
. The root password is also vagrant
. Change
these in any exposed environments!
If you enabled a webserver in your LaMachine build, you can connect your web browser to http://127.0.0.1:8080 after having started the VM.
If you want to connect to a particular special-purpose server (not a webservice) inside the VM from your host system, then you often need to forward a port from your host system into the LaMachine VM, as for all intents and purposes, they should be considered two separated systems. This applies for instance when you want to use the server mode offered by software such as Frog or Alpino (again, this is completely different and independent from the webservices that LaMachine also offers).
From LaMachine 2.6.2 onward, the port 9999 is forwarded by default for the VM, meaning that if you connect to port 9999 on your local machine (IP 127.0.0.1), it will be forwarded to port 9999 in the LaMachine VM.
If you want to open any additional ports, you need to do so in Virtualbox for your LaMachine VM. Consult this guide for easy and illustrated instructions on how to set this up in the VirtualBox interface, or alternatively consult the relevant chapter in the Virtualbox Manual itself.
If you used the LaMachine bootstrap script, you will have several scripts at your disposition (we assume that your
LaMachine VM is called stable, adapt the script names to your own situation). If you instead issued a docker pull proycon/lamachine
manually you will need to run the docker commands yourself:
lamachine-stable-activate
to start a new interactive container
docker run -i -t proycon/lamachine
lamachine-stable-run
to start the command specified as a parameter in a new container (e.g. lamachine-stable-run frog
)
docker run -i -t proycon/lamachine:latest frog
-i
flag if the tool is not an interactive tool that reads from standard input (i.e. keyboard input).lamachine-stable-start
to start a webserver and all enabled webservices in a new LaMachine container:
docker run -p 8080:80 -h hostname -t proycon/lamachine:latest lamachine-start-webserver -f
-p
are the port numbers on the host side and on the container side respectively, the latter must always match with the http_port
setting LaMachine has been built with (defaults to 80).-h
with the desired hostname, this too must match the setting LaMachine has been built with!-f
argument to lamachine-start-webserver
ensures the script waits in the foreground and doesn't exit after starting. In a docker context,The scripts will automatically share your designated data directory (your home directory by default) with the container, mounted at /data
by default. To manually make persistent storage available in the container, e.g. for sharing data, use docker parameters like: --mount type=bind,source=/path/on/host,target=/data
If you use LaMachine with docker, we expect you to actually be familiar with docker and understand the non-persistent nature of containers, understand the difference between images and containers. Be aware that new containers are created every time you run any of the above commands. If you want a more VM-like container experience, you can consider LXD instead of Docker.
When you are inside LaMachine, you can update it by running lamachine-update
, if you want to add
extra software packages to your installation, run lamachine-add
first (add --list
for a list of installable packages).
You can also edit LaMachine's settings and/or directly edit the list of packages to be installed/updated with lamachine-update --edit
. Do note that
this can't be used to uninstall software.
The update normally only updates what has changed, if you want to force an update of everything instead, run
lamachine-update force=1
. You can also use the even stronger force=2
, which will forcibly remove all downloaded sources
and start from scratch.
For Docker and the Virtual Machine flavours, when a SUDO password is being asked by the update script, you can simply press ENTER and leave it empty, do not run the entire script with escalated privileges.
Updating everything can be a time-consuming endeavour. If you know what you are doing then you can limit your update to
certain packages, you can specify these packages (as a comma separated list) to the --only
parameter, e.g:
lamachine-update --only python-core,java-core
. Do be aware that this could result in your LaMachine ending up in an
unusable state (in which case a normal update should remedy the problem again).
If you want to view the LaMachine configuration, simply issue a lamachine-config
from within. If you want to edit it
interactively, add the --edit
flag. Always run lamachine-update
afterwards to apply the new configuration.
The lamachine-config
tool can also be used to quickly edit a configuration setting through the command line, see
lamachine-config --help
for details.
LaMachine comes with several webservices and web applications out of the box. Most are RESTful webservices served using CLAM, which also offer a generic web-interface for human end-users. The webserver provides a generic portal to all available services, powered by Labirinto, as shown in the screenshot below:
To start (or restart) the webserver and webservices, run lamachine-start-webserver
from within your LaMachine
installation. You can then connect your browser (on the host system) to http://localhost:8080 (the port may differ if
you changed the default value). On virtual machines, the webserver will be automatically started at boot. For
docker you can do: docker run -p 8080:80 -h hostname -t proycon/lamachine:latest lamachine-start-webserver -f
Warning: There is currently no or poor authentication enabled on the webservices, so do not expose them to the outside world!
LaMachine comes with an installation of Jupyter Lab, which provides an excellent entry-point to LaMachine as it provides a web-based scripting environment or IDE (for Python and R), web-based terminal access, and especially access to the ubiquitous Jupyter Notebooks that enjoy great popularity in data science and beyond.
You can access your Jupyter Lab installation from the portal website of your LaMachine installation. By default LaMachine also preinstalls the interactive Python Course for the Humanities for you, so you can get started right away.
The default password for the Lab environment is lamachine, you can change this with lamachine-passwd lab
.
Warning: Do not expose this service to the world without a strong customised password as it allows arbitrary code execution and full access to your system!
Unless you explicitly opt-out, LaMachine sends a few details to us regarding your installation of LaMachine whenever you build a new one or update an existing one. This is to help us keep track of its usage and improve it.
The following information is sent:
Your IP address will only be used to identify your country and not used in any other way. No personally identifiable information whatsoever will be included in any reports we generate from this and it will never be used for advertisement purposes.
To opt-out of this behaviour, set private: true
in your LaMachine settings.
During build and upgrade, LaMachine downloads software from a wide variety of external sources.
For a secure experience using LaMachine, take all of the following into account:
(vagrant:vagrant)
, the lamachine user in Docker containers haslamachine
, you will need to change this.lamachine-update
for updates! Bugs andLaMachine supports OpenID Connect, which is an extension on top of OAuth2, as a means to authenticate against an external single-sign-on authentication provider. You can configure OpenID Connect in the LaMachine configuration and LaMachine will attempt to propagate these parameters to all underlying software that supports OpenID Connect.
Please consult the LaMachine as a service documentation for further instructions.
LaMachine comes in three versions, stable installs the latest stable releases of all software, development installs the latest development releases and custom installs explicitly specified versions. This section is about the latter and is for advanced users.
LaMachine itself also carries a version number, this number corresponds to the version of all the installation scripts that make up LaMachine. It is not tied to the versions of any underlying software.
In any LaMachine installation (v2.3.0 and above), you can do lamachine-list -v
to obtain a customversions.yml
file that explicitly states what software versions are installed. When bootstrapping a new LaMachine
installation, you can place this customversions.yml
file in the directory where you run the bootstrap, and opt for
the custom version. LaMachine will then install the exact versions specified.
You can edit this customversions.yml
file if you have good reason to opt for very specific versions of certain
packages. Instead of an appropriate version number, you can also use the strings. Do be be aware that choosing version
numbers that do not exist or combining versions of different packages that are not compatible will surely break things.
If things fail, most software providers, us included, will not deliver support on older software versions.
The purpose of this custom versioning feature of LaMachine is to aid scientific reproducibility, with it you can build
an environment consisting of older software, corresponding to the versions at the time you ran your experiments. In such
cases you should publish a version of customversions.yml
along with your data (and a copy of the installation
manifest ideally).
This custom versioning is limited, it only pertains to software that is 1) not provided by the linux distribution itself, and 2) explicitly installed by LaMachine, rather than dependencies that are pulled in automatically by package managers. Even then, certain sofware is excluded from this scheme as the upstream provider does not provide the necessary facilities for obtaining older versions, LaMachine should output a warning in the log if that is the case. It is also not supported on MacOS.
If a strict reproduction environment is desired, we strongly recommend to use the docker or virtual machine flavour of LaMachine and archive the entire resulting image.
A: This depends on the software you are interested in and the kind of system you are on. LaMachine is offered as a convenience but draws from other software repositories which you can also access directly.
You may want to first check if our software packages are available for your Linux distribution. For C++ software such as Frog, ucto and Timbl, we provide packages for:
Python software is generally provided through the Python Package Index and can be installed
using pip install
.
LaMachine shines as it combines a lot of software, includes complex set-ups, and handles some default configuration.
A LaMachine installation quickly reaches 6GB, and even more if you enable software that is not enabled by default. LaMachine is a research environment that can be used in a wide variety of ways which we can't predict in advance, so we by default include a lot of popular software for maximum flexibility. When building your LaMachine, you can disable software groups you don't want and save space, or opt for extra dikspace (see the next question).
You can also limit the size somewhat by setting minimal: true
in your LaMachine configuration, but this may mean that
certains tools don't fully work.
Disk space is also, by far, the cheapest resource, in contrast to memory or CPU.
This means the virtual disk used by the virtual machine or container is full. This may especially occur if you select some of the larger optional software packages. There is only limited space available in the VM or Docker container (roughly 9GB). For the VM, when you bootstrap your own LaMachine image from scratch (an option currently not available for Windows users though), you can opt to create extra diskspace (an extra volume).
For Docker, you may need to increase the base size of your containers (depending on the storage driver you use for docker). Consult the docker documentation at https://docs.docker.com/storage/storagedriver/ and do so now if you need this.
Advanced VM users can resolve the problem on their existing LaMachine VM by adding another virtual disk and moving some of the data, but this requires a fair amount of Linux administration expertise on their part. The procedure is roughly as follows:
fdisk
or parted
)mkfs.ext4
)/etc/fstab
/usr/local
(which is where most of LaMachine is installed) to the new disk/usr/local
to the new path on the new diskNo
No
No, your Linux distribution needs to be up to date and supported.
Yes! See the contribution guidelines
Though LaMachine does not provide this out-of-the-box, you can easily install a fully fledged desktop environment as follows (do make sure you opted for extra diskspace during the bootstrap):
apt-get install task-gnome-desktop
(See https://wiki.debian.org/DesktopEnvironment)
To access the graphical desktop you will want to start LaMachine from the VirtualBox interface.
Yes, see [docs/kubernetes/README.md](these instructions) and templates.
Your Docker is too old, upgrade to at least 1.9
This error may appear when LaMachine updates from ansible 2.7 to 2.8, if this occurs, simply rerun the update.
This is a Vagrant box file. You will need to follow the instruction as specified in Installation section C, with the following differences:
vagrant init
, you will need to run vagrant box add --name custom-lamachine /path/to/your/image.box
(adapt the path tocustom-lamachine
to anything you like to identifyvagrant init --template Vagrantbox.prebuilt.erb proycon/lamachine
, do vagrant init --template Vagrantbox.prebuilt.erb custom-lamachine
(or another name if you changed it in the first step)Yes! Please report it in our Issue Tracker after checking that the problem has not already been reported (and solved perhaps) by someone else. Note that this is only for problems relating to the installation and availability of the software; for bugs or feature requests on any of the participating software (including our own), you should use the issue trackers pertaining to those software projects.