๐ A python library for distributed training of a Transformer neural network across the Internet to solve the Running Key Cipher, widely known in the field of Cryptography.
APACHE-2.0 License
The main goal of the project is to study the possibility of using Transformer neural network to โreadโ meaningful text in columns that can be compiled for a Running Key Cipher. You can read more about the problem here.
In addition, the second rather fun ๐ goal is to train a large enough model so that it can handle the case described below. Let there be an original sentence:
Hello, my name is Zendaya Maree Stoermer Coleman but you can just call me Zendaya.
The columns for this sentence will be compiled in such a way that the last seven contain from ten to thirteen letters of the English alphabet, and all the others from two to five. Thus, the last seven characters will be much harder to "read" compared to the rest. However, we can guess from the meaning of the sentence that this is the name Zendaya. In other words, the goal is also to train a model that can understand and correctly โreadโ the last word.
Trecover requires Python 3.8 or higher and supports both Windows and Linux platforms.
git clone https://github.com/alex-snd/TRecover.git && cd trecover
Create a virtual environment:
python -m venv venv
python3 -m venv venv
Activate the virtual environment:
venv\Scripts\activate.bat
source venv/bin/activate
Install the package inside this virtual environment:
pip install -e ".[demo]"
pip install -e ".[train]"
pip install -e ".[dev]"
Initialize project's environment:
trecover init
For more options use:
trecover init --help
๐ค Hugging Face You can play with a pre-trained model hosted here.
๐ณ Docker Compose
docker-compose -f docker/compose/scalable-service.yml up
trecover download artifacts
docker-compose -f docker/compose/scalable-service-build.yml up
๐ป Local (requires docker)
trecover download artifacts
trecover up
The WikiText and WikiQA datasets were used to train the model, from which all characters except English letters were removed. You can download the cleaned dataset:
trecover download data
To quickly start training the model, open the Jupyter Notebook .
trecover train \
--project-name {project_name} \
--exp-mark {exp_mark} \
--train-dataset-size {train_dataset_size} \
--val-dataset-size {val_dataset_size} \
--vis-dataset-size {vis_dataset_size} \
--test-dataset-size {test_dataset_size} \
--batch-size {batch_size} \
--n-workers {n_workers} \
--min-noise {min_noise} \
--max-noise {max_noise} \
--lr {lr} \
--n-epochs {n_epochs} \
--epoch-seek {epoch_seek} \
--accumulation-step {accumulation_step} \
--penalty-coefficient {penalty_coefficient} \
--pe-max-len {pe_max_len} \
--n-layers {n_layers} \
--d-model {d_model} \
--n-heads {n_heads} \
--d-ff {d_ff} \
--dropout {dropout}
For more information use trecover train local --help
TODO: what was done, tech stack.
Contributions, issues and feature requests are welcome. Feel free to check issues page if you want to contribute.
Please don't hesitate to โญ๏ธ this repository if you find it cool!
Copyright ยฉ 2022 Alexander Shulga. This project is Apache 2.0 licensed.