🔖 public release

deezer · Oct 8, 2020 · ba74eff · ba74eff
1 parent 3f480a4
commit ba74eff
Show file tree

Hide file tree

Showing 56 changed files with 315,357 additions and 17 deletions.
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
@@ -0,0 +1 @@
+* @deezer @Faylixe @GuillaumeSalha @elenavepure @romi1502
diff --git a/.github/ISSUE_TEMPLATE/bug.md b/.github/ISSUE_TEMPLATE/bug.md
@@ -0,0 +1,44 @@
+---
+name: Bug
+about: Report a bug
+title: "[Bug] name your bug"
+labels: bug, invalid
+---
+
+<!-- PLEASE READ THIS CAREFULLY :
+- Any issue which does not respect following template or lack of information will be considered as invalid and automatically closed
+- First check FAQ from wiki to see if your problem is not already known
+-->
+
+## Description
+
+<!-- Give us a clear and concise description of the bug you are reporting. -->
+
+## Step to reproduce
+
+<!-- Indicates clearly steps to reproduce the behavior: -->
+
+1. Installed using `...`
+2. Run as `...`
+3. Got `...` error
+
+## Output
+
+```bash
+Share what your terminal says when you run the script (as well as what you would expect).
+```
+
+## Environment
+
+<!-- Fill the following table -->
+
+|                   |                                 |
+| ----------------- | ------------------------------- |
+| OS                | Windows / Linux / MacOS / other |
+| Installation type | Conda / pip / other             |
+| RAM available     | XGo                             |
+| Hardware spec     | GPU / CPU / etc ...             |
+
+## Additional context
+
+<!-- Add any other context about the problem here, references, cites, etc.. -->
diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml
@@ -0,0 +1 @@
+blank_issues_enabled: false
diff --git a/.github/ISSUE_TEMPLATE/discussion.md b/.github/ISSUE_TEMPLATE/discussion.md
@@ -0,0 +1,8 @@
+---
+name: Discussion
+about: Ideas sharing or theorical question solving 
+labels: question
+title: "[Discussion] your question"
+---
+
+<!-- Please respect the title [Discussion] tag. -->
diff --git a/.github/ISSUE_TEMPLATE/feature.md b/.github/ISSUE_TEMPLATE/feature.md
@@ -0,0 +1,14 @@
+---
+name: Feature request
+about: Submit idea for new feature
+labels: feature, enhancement
+title: "[Feature] your feature name"
+---
+
+## Description
+
+<!-- Describe your feature request here. -->
+
+## Additional information
+
+<!-- Add any additional description -->
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
@@ -0,0 +1,13 @@
+# [Pull request title]
+
+## Description
+
+A few sentences describing the overall goals of the pull request's commits.
+
+## How this patch was tested
+
+You tested it, right?
+
+## Documentation link and external references
+
+Please provide any info that may help us better understand your code.
diff --git a/Makefile b/Makefile
@@ -0,0 +1,65 @@
+###########################################################################################################################
+# Make framework that provides generic docker based function
+# and project values.
+#
+# @author Félix Voituret <fvoituret@deezer.com>
+# @version 1.0.0
+###########################################################################################################################
+
+SHELL 			= /bin/bash -x
+PROJECT			= muzeeglot
+COMPOSE			= docker-compose -p $(PROJECT)
+COMPOSE_FILE 	= docker-compose.yaml
+COMPOSE_OPTS	=
+DOCKER			= docker
+DOMAIN			=
+CERTCOMMAND		= certonly --webroot --webroot-path=/data/letsencrypt -d $(DOMAIN) -d www.$(DOMAIN)
+CERTVOLUMES		= -v $(PROJECT)-certificates:/etc/letsencrypt
+CERTVOLUMES		+= -v $(PROJECT)-certificates-data:/data/letsencrypt
+BUILDERVOLUMES  = $(CERTVOLUMES) -v frontend/nginx/certificate-builder.conf:/etc/nginx/nginx.conf:ro
+
+.PHONY: api frontend
+
+usage:
+	$(eval SHELL=/bin/bash)
+	@echo "Supported goals:"
+	@echo "	* make [no-cache] api|frontend"
+	@echo "	* make [ssl] run|start|stop"
+	@echo "	* make clean|letsencrypt|logs"
+
+no-cache:
+	$(eval COMPOSE_OPTS=$(COMPOSE_OPTS) --no-cache)
+
+ssl:
+	$(eval COMPOSE_FILE=docker-compose-ssl.yaml)
+
+letsencrypt:
+	-@$(DOCKER) volume rm $(PROJECT)-certificates
+	-@$(DOCKER) volume rm $(PROJECT)-certificates-data
+	@$(DOCKER) volume create $(PROJECT)-certificates
+	@$(DOCKER) volume create $(PROJECT)-certificates-data
+	@$(DOCKER) run -d $(BUILDERVOLUMES) -p 80:80 -p 443:443 --name $(PROJECT)-certificate-builder nginx
+	@$(DOCKER) run -it --rm $(CERTVOLUMES) deliverous/certbot $(CERTCOMMAND)
+	@$(DOCKER) kill $(PROJECT)-certificate-builder
+	@$(DOCKER) rm $(PROJECT)-certificate-builder
+
+api:
+	@$(COMPOSE) build $(COMPOSE_OPTS) api
+
+frontend:
+	@$(COMPOSE) build $(COMPOSE_OPTS) frontend
+
+run: api frontend
+	@$(COMPOSE) up
+
+start: api frontend
+	@$(COMPOSE) -f $(COMPOSE_FILE) up -d
+
+stop:
+	@$(COMPOSE) -f $(COMPOSE_FILE) down
+
+logs:
+	@$(COMPOSE) -f $(COMPOSE_FILE) logs --follow
+
+clean: stop
+	@docker volume rm $$(docker volume ls | grep muzeeglot | tr -s ' ' | cut -d' ' -f2)
diff --git a/README.md b/README.md
@@ -1,42 +1,140 @@
 # Muzeeglot
 
-In this repository, we present [Muzeeglot](https://research.deezer.com/muzeeglot), a propotype aiming at illustrating how
-multilingual music genre embedding space representations can be leveraged to
-**generate cross-lingual music genre annotations** for
-[DBpedia](https://wiki.dbpedia.org/) music entities (artists, albums, tracks...).
+In this repository, we present [Muzeeglot](https://research.deezer.com/muzeeglot),
+a propotype aiming at illustrating how multilingual music genre embedding space
+representations can be leveraged to **generate cross-lingual music genre annotations**
+for [DBpedia](https://wiki.dbpedia.org/) music entities (artists, albums, tracks, etc ...).
 
 <div align="center">
     <img
-        src="https://github.com/deezer/muzeeglot/blob/master/screenshot.png?raw=true"
+        src="https://github.com/deezer/muzeeglot/blob/master/docs/screenshot.png?raw=true"
         width="70%">
     <p>
         <a href="https://research.deezer.com/muzeeglot">Muzeeglot</a> includes a web interface to visualize these
         multilingual music genre embeddings.
     </p>
 </div>
 
+- :question: [How it works](#how-it-works)
+- :gear: [Architecture](#architecture)
+- :rocket: [Deployment](#deployment)
+    * :lock: [SSL support](#ssl-support)
+- :computer: [Development](#development)
+- :book: [Cite](#cite)
+
+## How it works
+
 Based on annotations from one or several sources languages, our system automatically
 predicts the corresponding annotations in a target language.
 
 Languages supported:
 
-- French (fr)
-- English (en)
-- Spanish (es)
-- Dutch (nl)
-- Czech (cs)
-- Japanese (ja)
+- :fr: French
+- :gb: English
+- :es: Spanish
+- :netherlands: Dutch
+- :czech_republic: Czech
+- :jp: Japanese
+
+You will find more information about application usage [here](https://github.com/deezer/muzeeglot/blob/master/docs/presentation.pdf).
+
+## Architecture
+
+[Muzeeglot](https://research.deezer.com/muzeeglot) is based on a classic _N-tier_ architecture
+including :
+
+- A [Redis](https://redis.io) instance as storage engine.
+- A REST API developed in Python with [FastAPI](https://fastapi.tiangolo.com).
+- A frontend developed with [VueJS](https://vuejs.org), as a SPA (Single Page Application).
+
+The overall stack is loadbalanced using [Nginx](https://www.nginx.com) webserver :
+
+<div align="center">
+    <img
+        src="https://github.com/deezer/muzeeglot/blob/master/docs/architecture.png?raw=true">
+</div>
+
+Data such as entities, tags, and languages are stored into the [Redis](https://redis.io)
+instance. Additionnally, a text search index based on [Whoosh](https://github.com/mchaput/whoosh)
+is maintained using _ngram_ tokenization on entity names.
+
+## Deployment 
+
+Deploying [Muzeeglot](https://research.deezer.com/muzeeglot) requires the following tools to be
+installed :
+
+- [Git](https://git-scm.com)
+- [GNU make](https://www.gnu.org/software/make/)
+- [Docker](https://www.docker.io)
+- [Docker compose](https://docs.docker.com/compose/)
+
+You can then clone this repository and start [Muzeeglot](https://research.deezer.com/muzeeglot)<sup>1</sup> :
 
-## Technical Details
+```bash
+git clone https://github.com/deezer/muzeeglot
+cd muzeeglot
+make start
+```
+
+Behind the scene it will build the required [docker](https://www.docker.io) images and run a
+[compose](https://docs.docker.com/compose/) file with everything required locally in daemon mode.
+
+> <sup>1</sup> first deployment will be long as it requires data ingestion and indexing.
+
+### SSL support
+
+In case you want to deploy [Muzeeglot](https://research.deezer.com/muzeeglot) with SSL
+using [LetsEncrypt](https://letsencrypt.org), you need to first create certificate using
+the provided bot challenge. Start by editing the following configuration files to add your
+target domain :
+
+- `frontend/nginx/certificate-builder.conf`
+- `frontend/nginx/muzeeglot-ssl.conf`
+
+Once you did so, you can run the following command to generate SSL certificates:
+
+```bash
+make letsencrypt DOMAIN=mydomain.tld
+```
+
+It will create a [docker](https://www.docker.io) volume and provision it with certificate.
+Then you can run [Muzeeglot](https://research.deezer.com/muzeeglot) as follows:
+
+```bash
+make ssl start
+```
 
-[Muzeeglot](https://research.deezer.com/muzeeglot) will be presented as a demonstration at [JEP-TALN-RECITAL 2020](https://jep-taln2020.loria.fr).
+## Development
 
-This [extended abstract](https://jep-taln2020.loria.fr/wp-content/uploads/JEP-TALN-RECITAL-2020_paper_156.pdf) (in French) from the conference provides more information on [Muzeeglot](https://research.deezer.com/muzeeglot). Feel free to also check out those [slides](https://github.com/deezer/muzeeglot/blob/master/presentation.pdf) for more information in English.
+Project can be managed using `GNU Make` through the following goals :
 
-## Demonstration
+| Goal          | Description                                    |
+| ------------- | ---------------------------------------------- |
+| _api_         | Build api image                                |
+| _frontend_    | Build frontend image                           |
+| _run_         | Start the entire stack using docker-compose    |
+| _start_       | Start the entire stack in daemon mode          |
+| _stop_        | Stop the entier stack using docker-compose     |
+| _logs_        | Display stack logs when running in daemon mode |
+| _clean_       | Clean docker volume for storage and indexes    |
+| _letsencrypt_ | Generate certificate volume                    |
 
-In this repository, we will also soon release the source code of Muzeeglot.
+Additional goals can be used to provide extra parameters:
 
+| Goal       | Description                          |
+| ---------- | ------------------------------------ |
+| _no-cache_ | Build images using `--no-cache` flag |
+| _ssl_      | Enable SSL support                   |
+
+If you want to use your own data, please provide the following files into `api/data` directory<sup>2</sup>:
+
+- Embeddings for mapping are expected through `embeddings.csv` CSV file.
+- Reduced embeddings for display are expected through `embeddings_reduced.csv` CSV file.
+- Supported language are expected through `languages.csv` CSV file.
+- Indexed entities are expected through `entites.csv` CSV file.
+- Genre mapping corpus is expected through `corpus.csv` CSV file.
+
+> <sup>2</sup> you need to clean the data storage and index to force data ingestion when you redeploy.
 
 ## Cite
 
@@ -49,4 +147,4 @@ In this repository, we will also soon release the source code of Muzeeglot.
   year={2020},
   organization={ATALA}
 }
-```
+```