Pre-trained model for English -> Vietnamese NMT

Intro.md

Datasets

I had such a bad time trying to create english-vietnamese parallel corpus from bilingual stories, but it sucks. It just wastes a lot of time. So I try to find out as much corpora as possible throughout the internet. My final dataset consists of about 2.5M pair of sentences. You can find all corpora here: link

Model

I use OpenNMT to train my nmt model. Thanks Systran and HavardNLP for open source this project. It will help me and many others to understand how a industral translation system might work. The parameters of my model are as follow:

• Preprocesssing: Using aggressive tokenizer provided by OpenNMT
• Vocabulary size: 40k words
• 6 layers of LSTM
• Using BILSTM encoder
• Th size of LSTM hidden layer: 500
• Using residual connections between layers to enhance gradient flow (like in Google NMT system).

I also provide pre-trained model for English -> Vietnamese translation for anyone who is interested in NMT. In the future, I might update some more better models. A contributor from OpenNMT team said they will support cudnn in near future so it will speedup a lot in training time and I can do more experiments.

Usage

Follow link for guide how to translate texts using a pre-trained model.

Hi Tuan,

My name is An and I obtained your en-vi model built by NMT and released it for running on CPU server.
However, I always get the following errors which I have no idea how to resolve. I also asked another member in NMT github to help and he wanted to know which NMT version you used to build your model (OpenNMT/OpenNMT#521). Could you please help? We can discuss via my email (hoangyenan@yahoo.com). Thank you!

[icuser@xxx OpenNMT-Torch]$ th translate.lua -model model_en_vi_epoch19_9.12_release.t7 -src train_snli.txt_textSingles -output train_snli.txt_textSingles_VN
[02/15/18 10:31:57 INFO] Loading 'model_en_vi_epoch19_9.12_release.t7'...
/home/icuser/torch/install/bin/luajit: /home/icuser/torch/install/share/lua/5.1/torch/File.lua:343: unknown Torch class <torch.CudaTensor>
stack traceback:
[C]: in function 'error'
/home/icuser/torch/install/share/lua/5.1/torch/File.lua:343: in function 'readObject'
/home/icuser/torch/install/share/lua/5.1/torch/File.lua:369: in function 'readObject'
/home/icuser/torch/install/share/lua/5.1/torch/File.lua:369: in function 'readObject'
/home/icuser/torch/install/share/lua/5.1/torch/File.lua:369: in function 'readObject'
/home/icuser/torch/install/share/lua/5.1/torch/File.lua:369: in function 'readObject'
/home/icuser/torch/install/share/lua/5.1/torch/File.lua:409: in function 'load'
./onmt/translate/Translator.lua:200: in function '__init'
/home/icuser/torch/install/share/lua/5.1/torch/init.lua:91: in function 'new'
translate.lua:80: in function 'main'
translate.lua:365: in main chunk
[C]: in function 'dofile'
...user/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x004064f0