Index music album from the MusicBrainz open music encyclopedia into Elasticsearch
APACHE-2.0 License
Like freedb, MusicBrainz is an open music encyclopedia that collects music metadata and makes it available to the public.
The musicbrainz-elasticsearch project is a java batch that indexes release groups of the MusicBrainz database into an Elasticsearch index. From release groups, only "real" Album are indexed. Single, EP and Broadcast are not indexed. And from Album release group primary type, neither Compilation, Live, Remix or Soundtrack secondary types are indexed.
This project depends on several other open source projects:
A MusicBrainz database and an Elasticsearch cluster are the 2 pre-requisites in order to execute the batch. You have the choice by setting by yourself a MusicBrainz database and an Elasticsearch cluster or to use Docker.
Use Docker Compose to set up both a PostgreSQL database and an Elasticsearch cluster and import the musicbrainz database.
If you are on MacOS or Windows, you have to install Boot2docker in order to user Docker and Docker Compose. You will have to increase the DiskSize up to 100 Gb.
Command lines to start PostgreSQL and Elasticsearch:
git clone https://github.com/arey/musicbrainz-elasticsearch.git
cd musicbrainz-elasticsearch/docker
docker-compose up -d
docker-compose run postgresql /create-database.sh
boot2docker ip
es-musicbrainz-batch.properties
file and replace localhost with the IP in the es.host and db.musicbrainz.url properties.The last command line creates the database, downloads the latest dumps then populates the database. Depending your bandwidth, downloading of the mbdump.tar.bz2 could be take more than hour.
To index MusicBrainz data, the batch requires a connection to the MusicBrainz PostgreSQL relational database. Musicbrainz.org does not provide a public access to its database. Thus you have to install your own database. There are a two different methods to get a local database up and running, you can either:
For my part, before using Docker, I have chosen to download the MusicBrainz Server virtual machine. Available in Open Virtualization Archive (OVA), I have deployed it into Oracle VirtualBox but you may prefer VMWare. Once finished the MusicBrainz Server setup guide, you have to follow the below two final steps in order the PostgreSQL database be accessible to your host:
Configuring port forwarding with NAT
Port forwarding enables VirtualBox to listen to certain ports on the host and resends all packets which arrive there to the guest, on the same or a different port. You may used same port on host and guest. Configure two rules (the second is optional):
Configuring PostgreSQL
To enable remote access to the PostgreSQL database server, you may follow those instructions. Log into the VM (credentials: vm / musicbrainz) and edit the two configuration files pg_hba.conf and postgresql.conf.
Once steps done, you may connect to the database with any JDBC clients (ie. SQuireL):
Before launching the batch, you have to download Elasticsearch v1.7.1 and unarchived it. You may want to change the default elasticsearch cluster name from the config/elaticsearch.yml configuration file and change the name in the es-musicbrainz-batch.properties configuration file.
git clone https://github.com/arey/musicbrainz-elasticsearch.git
mvn install
mvn exec:java
(execute the IndexBatchMain main class)On a Macbook Pro, the batch takes less than 3 minutes to build the Elasticsearch.
MusicBrainz database searching with Elasticsearch : http://musicsearch.javaetmoi.com/
For command line testing, you could execute the two following curl scripts: musicbrainz_autocomplete_u2.sh and musicbrainz_fulltext_u2_war.sh
Download the code with git: git clone git://github.com/arey/musicbrainz-elasticsearch.git
Compile the code with maven:
mvn clean install
If you're using an IDE that supports Maven-based projects (InteliJ Idea, Netbeans or m2Eclipse), you can import the project directly from its POM. Otherwise, generate IDE metadata with the related IDE maven plugin:
mvn eclipse:clean eclipse:eclipse
French articles on the javaetmoi.com blog:
Travis :