LaMachine end-of-life release

LaMachine is now deprecated and will not be further developed. Please see https://github.com/proycon/LaMachine/issues/214 for an explanation and alternative solutions we will actively work on.

• LaMachine is end-of-life now; added deprecation notices #214
• setting priority of git_branch over version #217
• attempting a fix for keeping codemetapy at previous version (#217)
• force codemetapy v0.3.5 because newer versions are not yet compatible
• adapted for Kaldi_NL v0.4
• forgot to rename this oralhistory -> asr_nl
• more retries when doing npm install
• removed old forcedalignment service
• renamed oralhistory to asr_nl

• Fix for python 3.10 compatibility, dropped 3.4 support. Considering 3.5 deprecated as well (might still work though), but please ensure 3.6 or higher.
• Fixed moving the www-data path to an externally mounted volume in docker; refactored www-data dir handling.

• Switched docker base image to Ubuntu Linux 20.04 LTS rather than Debian stable (consistency with VMs)
• Use python's built-in venv instead of virtualenv
• Fixes for npm and labirinto
• Get ansible from distribution repos for VM/container rather than the PPA

This is a major update:

• Implement OAuth2/OpenID Connect authentication solution (works for CLAM & FLAT, further testing in order for Jupyter Hub) #171
• Stripped all Python 2.7 support from the bootstrap script
• Migrated to Jupyter Hub
• Vagrant base box and Docker base image now use Ubuntu 20.04 LTS rather than Debian 10
• Use LTS release of nodejs on ubuntu/debian/redhat/centos
• Fixed issue in installation of labirinto (the portal website) #198
• Better fallbacks for installation of python bindings
• Less output clutter: hide skipped tasks (e.g. where condition is false)
• Added analiticcl to the languagemachines-rust package
• DeepFrog compilation may be broken currently
• force_ssl option now forces HTTP_X_FORWARDED_PROTO header in cases where the reverse proxy propagates the wrong headers
• Improvements to lamachine-config
  • Added a --remoteservice parameter for easy configuration editing #188
• Various fixes in kaldi/kaldi_nl

(some of the changes were already in v2.24)

• Added rust #181
  • Added rust-based software sesdiff, ssam
  • Added deepfrog
• Kaldi_NL, g2pservice and forcedalignment2 are obtained from new upstream source
  • various fixes for those webservices
• added asr4J, spreek2schrijf
• Various fixes and improvements for container deployment behind a reverse proxy
  • added custom_flat_settings option
  • use a default CLAM base configuration
  • added force_https option
• Refactoring: Implemented a languagemachine-web-install role and a language-python-link role to refactor some things and remove a lot of unnecessary duplication
• better argument parsing in lamachine-update
• explicitly set ansible python interpeter when running lamachine update
• various fixes

• integrated lamastats
• When calling the virtualenv activate script, an important part of activation wasmissed. Try to automatically call the primary activation script in such a case.
• Adding mitlm to phonetisaurus installation (phonetisaurus-train depends on it), and some CentOS 8 changes
• Docker fix, default interactive shell now starts with non-root and in the proper homedir

• Important fixes for macOS: install ansible through homebrew rather than through pip. This hopefully fixes some recent failures to install on at least macos catalina.
• Java fix: ensure Java is version 11 (versions 13 and 14 are too new for nextflow and probably also for various other tools)
• Automatically start background servers for valkuil and tscan (using uwsgi attach-daemon), and some fixes for existing background servers
• Fixed port number clash when both PICCL and T-scan were enabled
• Adding a sites-extra directory for non-LaMachine managed nginx configuration

• fixes for prebuilt VM flavour
• allow setting ssh key filename rather than forcing id_rsa

• LaMachine Release after final releases by @kosloot of full frog, ucto, libfolia, timbl, mbt software stack
• Fix for alpino
• Added fame_align (frisian aligner) to LaMachine
• Added forced alignment service
• Allow python-timbl to fail on macOS for now #175

• Added huggingface transformers library
• Fixed kaldi lib to atlas3 by default instead of openblas #172
• Various fixes for speech webservices (oral history and others)
• Allow setting ssh keypairs from the LaMachine configuration (needed for restricted access to e.g. our gitlab server)
• Added an option to skip nvm activation things
• Make lxc profile configurable in bootstrap.sh

• ansible deprecated some features, adapting accordingly (#167)
• explicitly set ALPINO_HOME and alpino-specific lib path in alpino/tscan uwsgi #153 (should already be set properly in venv though)
• completed tscan integration #153
• Rerun all activation scripts after update so the shell is up to date for any added environment variables immediately #153
• some documentation updates

• Added julia #165
• Added spaCy webservice (a CLAM webservice with spacy2folia for FoLiA output support)
• Docker: expose more ports by default (8080, 8888 and 9999 are now exposed by default in the Dockerfile)
• Adding kaldi-nl resource (#106) and oral history (+webservice) (#140) [requires private access]
• Minor updates to contributing guidelines

v2.13:

• Fix for alpino, it is linked against extra libraries which are in a non-default location
• Continue even if shared data path can't be created (due to permission issues sometimes)
• Various fixes for improved jupyter lab integration
  • auto-starting it
  • store jupyter notebooks in www_data_path and allow this to be stored on the shared data volume (relevant for docker/VM)
• Implemented new www_data_path option and a shared_www_data boolean that by defaults sets the data path to be on the shared volume (for VM/container)
  • explicitly ask during boostrap whether to put web data on the shared volume
• LaMachine now specifically provides support status indications for different Linux distributions/operating systems, we distinguish four categories (gold, silver, bronze and deprecated). Support indications are given both in the bootstrap strict as in the documentation.
• CentOS 8 support (silver support status), CentOS 7 support is being deprecated.
• Added clam_include directive to include a base CLAM configuration from the configuration of each clam service
• adding various "maintainted by lamachine" comments to configuration files, warning the user not to edit them.
• Fixes and updated instructions for for LXD containers
• Fixes in lamachine-start-webserver and cleanup of lamachine-start-webserver output

• Vagrant base box updated to Debian 10 (buster)
• Changing the way metadata is collected, adapting to codemetapy 0.3, making it easier to regenerate metadata, and removing some complexity
• Added piereling (and installing poppler and pandoc by default)
• Allowing quicker updates with the lamachine-update --only parameter no longer process all dependencies if it is set
• Remove labirinto sources before update because it often fails to upgrade nicely
• Allow LaMachine to build against a custom Python 3, by setting python_bin and pip_bin (building the custom Python is not included in LaMachine) #159
• Fixes for Arch Linux #161
• adding hdf5 to base packages (required by keras)
• adding new translitteration service (t2t)
• Various smaller fixes

Various fixes:

• lamachine-list: get $LM_PREFIX dynamically
• lamachine-list update
• metadata fixes for glem and g2pservice
• adding valkuil frontend installation and configuration (not done)
• allow setting ignore_errors in package for lamachine-package-install, as it does not propagate to tasks within ignore_role
• disabling cask installation for homebrew, they say it comes by default

• use latest available java version
• version bump because of other releases (FLAT, libfolia, ticcutils)

• Added colibri-utils (except on mac)
• Adding user to vboxsf group for shared folder access (if it exists)
• ignore errors on creating webserver dirs (in case they already exist or are links)
• don't let nvm problems prevent lamachine activation

• extract list of packages and ask to open installation manifest in editor or not (relates to #63), same thing for the configuration file
• Removing some packages from the default installation to conserve space:
  • Removing cltl-vu-python from the default packages (the folia-naf converter is not really in use nor maintained well nowadays and the git repo comes with some big files that take up too much space)
  • exclude xgboost from the default installation as it takes up a lot of space and is not a direct dependency
• add extra user input stage with updated instruction prior to ansible start. This ensures people know to expect a lot of output they can ignore and a long wait whilst things are installing, and so understand what to do when ansible asks for the BECOME password (addressed an issue raised by @pboot)
• fix for failure to start nginx #151
• adapting the strict python check and making the bootstrap also work when python 2 is absent in some cases
• ensured ~/bin is in $PATH immediately after bootstrap
• added more feedback to lamachine-start-webserver script
• implemented CSV export ability for lamachine-list
• do linux distribution detection and warn users if they run on an unsupported or outdated platform
• [documentation] updated FAQ

• Added aNtiLoPe and nederlab-pipeline, (both split off from PICCL), they are nabled by default now as part of the languagemachines-pipelines package
• Adding DBPedia spotlight #143
• Added langid.py
• Added fasttext
• Added WikiEnte
• Fix for spacy #148
• Fixed proper validation in case disk size wasn't a number
• opening port 9999 by default in vagrant for user-defined purposes

v2.7.0 (forgot to release formally here):

• New FoLiA v2 based releases for Ucto, frog, libfolia, FoLiA-utils
• [CentOS7] compile SQLite from scratch because system version is too old (django complains, affects FLAT)
• attempted fix for redhat (CentOS)
• fix: flair requires python >3.6, do not install on older distributions
• ansible 2.8 compatibility; added note for ansible upgrade error #144

• Added spacy2folia
• Added Freeling #123
• Added eng_ASR #139 (still to be properly tested)
• Adapted for FLAT v0.8.0 and foliadocserve v0.7.0
• Fix in npm upgrade
• This version also holds the final camera ready paper on LaMachine as published in the CLARIN 2018 proceedings

Adds support for FoLiA 2.0, see also the release plan in proycon/folia#68

• Added new FoLiApy
• FoLiA-tools is now split from the central FoLiA repository and LaMachine is adapted accordingly. The central repository containing examples and specifications is still supplied with LaMachine, but usually not needed for most end-users.