A small C++ implementation of LSTM networks, focused on OCR.
APACHE-2.0 License
CLSTM is an implementation of the LSTM recurrent neural network model in C++, using the Eigen library for numerical computations.
CLSTM is mainly in maintenance mode now. It was created at a time when there weren't a lot of good LSTM implementations around, but several good options have become available over the last year. Nevertheless, if you need a small library for text line recognition with few dependencies, CLSTM is still a good option.
You can train and run clstm without installation to the local machine using the docker image, which is based on Ubuntu 16.04. This is the best option for running clstm on a Windows host.
You can either run the last version of the clstm
image from Docker Hub or build the Docker
image from the repo (see ./docker/Dockerfile
).
The command line syntax differs from a native installation:
docker run --rm -it -e [VARIABLES...] kbai/clstm BINARY [ARGS...]
is equivalent to
[VARIABLES...] BINARY [ARGS...]
For example:
docker run --rm -it -e ntrain=1000 kbai/clstm clstmocrtrain traininglist.txt
is equivalent to
ntrain=1000 clstmocrtrain traininglist.txt
# Ubuntu 15.04, 16.04 / Debian 8, 9
sudo apt-get install scons libprotobuf-dev protobuf-compiler libpng-dev libeigen3-dev swig
# Ubuntu 14.04:
sudo apt-get install scons libprotobuf-dev protobuf-compiler libpng-dev swig
The Debian repositories jessie-backports and stretch include sufficiently new libeigen3-dev packages.
It is also possible to download Eigen with Tensor support (> v3.3-beta1)
and copy the header files to an include
path:
# with wget
wget 'https://github.com/RLovelett/eigen/archive/3.3-rc1.tar.gz'
tar xf 3.3-rc1.tar.gz
rm -f /usr/local/include/eigen3
mv eigen-3.3-rc1 /usr/local/include/eigen3
# or with git:
sudo git clone --depth 1 --single-branch --branch 3.3-rc1 \
"https://github.com/RLovelett/eigen" /usr/local/include/eigen3
To use the visual debugging methods, additionally:
# Ubuntu 15.04:
sudo apt-get install libzmq3-dev libzmq3 libzmqpp-dev libzmqpp3 libpng12-dev
For HDF5, additionally:
# Ubuntu 15.04:
sudo apt-get install hdf5-helpers libhdf5-8 libhdf5-cpp-8 libhdf5-dev python-h5py
# Ubuntu 14.04:
sudo apt-get install hdf5-helpers libhdf5-7 libhdf5-dev python-h5py
To build a standalone C library, run
scons
sudo scons install
There are a bunch of options:
debug=1
build with debugging options, no optimizationdisplay=1
build with display support for debugging (requires ZMQ, Python)prefix=...
install under a different prefix (untested)eigen=...
where to look for Eigen include files (should contain Eigen/Eigen
)openmp=...
build with multi-processing support. Set theOMP_NUM_THREADS
hdf5lib=hdf5
what HDF5 library to use; enables HDF5 command linehdf5_serial
in some environments)After building the executables, you can run two simple test runs as follows:
run-cmu
will train an English-to-IPA LSTMrun-uw3-500
will download a small OCR training/test set and train an OCR LSTMThere is a full set of tests in the current version of clstm; just run them with:
./run-tests
This will check:
To build the Python extension, run
python setup.py build
sudo python setup.py install
(this is currently broken)
You can find some documentation and examples in the form of iPython notebooks in the misc
directory
(these are version 3 notebooks and won't open in older versions).
You can view these notebooks online here: http://nbviewer.ipython.org/github/tmbdev/clstm/tree/master/misc/
The clstm
library operates on the Sequence type as its fundamental
data type, representing variable length sequences of fixed length vectors.
The underlying Sequence type is a rank 4 tensor with accessors for
individual rank-2 tensors at different time steps.
Networks are built from objects implementing the INetwork
interface.
The INetwork
interface contains:
struct INetwork {
Sequence inputs, d_inputs; // input sequence, input deltas
Sequence outputs, d_outputs; // output sequence, output deltas
void forward(); // propagate inputs to outputs
void backward(); // propagate d_outputs to d_inputs
void update(); // update weights from the last backward() step
void setLearningRate(Float,Float); // set learning rates
...
};
Network structures can be hierarchical and there are some network implementations whose purpose it is to combine other networks into more complex structures.
struct INetwork {
...
vector<shared_ptr<INetwork>> sub;
void add(shared_ptr<INetwork> net);
...
};
At its lowest level, layers are created by:
make_layer
ninput
and noutput
) assub
vectorinitialize()
There are three different functions for constructing layers and networks:
make_layer(kind)
looks up the constructor and gives you an uninitialized layerlayer(kind,ninput,noutput,args,sub)
performs all initialization steps in sequencemake_net(kind,args)
initializes a whole collection of layers at oncemake_net_init(kind,params)
is like make_net
, but parameters are given in string formThe layer(kind,ninput,noutput,args,sub)
function will perform
these steps in sequence.
Layers and networks are usually passed around as shared_ptr<INetwork>
;
there is a typedef of this calling it Network
.
This can be used to construct network architectures in C++ pretty easily. For example, the following creates a network that stacks a softmax output layer on top of a standard LSTM layer:
Network net = layer("Stacked", ninput, noutput, {}, {
layer("LSTM", ninput, nhidden,{},{}),
layer("SoftmaxLayer", nhidden, noutput,{},{})
});
Note that you need to make sure that the number of input and output units are consistent between layers.
In addition to these basic functions, there is also a small implementation of CTC alignment.
The C++ code roughly follows the lstm.py implementation from the Python version of OCRopus. Gradients have been verified for the core LSTM implementation, although there may be still be bugs in other parts of the code.
There is also a small multidimensional array class in multidim.h
; that
isn't used in the core LSTM implementation, but it is used in debugging
and testing code, for plotting, and for HDF5 input/output. Unlike Eigen,
it uses standard C/C++ row major element order, as libraries like
HDF5 expect. (NB: This will be replaced with Eigen::Tensor.)
LSTM models are stored in protocol buffer format (clstm.proto
),
although adding new formats is easy. There is an older HDF5-based
storage format.
The clstm.i
file implements a simple Python interface to clstm, plus
a wrapper that makes an INetwork mostly a replacement for the lstm.py
implementation from ocropy.
There are several command line drivers:
clstmfiltertrain training-data test-data
learns text filters;
clstmfilter
applies learned text filtersclstmocrtrain training-images test-images
learns OCR (or image-to-text) transformations;
.gt.txt
fileclstmocr
applies learned OCR modelsIn addition, you get the following HDF5-based commands:
Note that most parameters are passed through the environment:
lrate=3e-5 clstmctc uw3-dew.h5
See the notebooks in the misc/
subdirectory for documentation on the parameters and examples of usage.
(You can find all parameters via grep 'get.env' *.cc
.)