Initial version

README.md

@@ -1,3 +1,58 @@
----
-license: mit
----
+---
+language: 
+- ca
+- jp
+
+tags:
+- translation
+
+library_name: opennmt
+license: mit
+metrics:
+- bleu
+
+inference: false
+---
+
+### Introduction
+
+Catalan - Japanese translation model for OpenNMT. These are the same models that we have in production at https://www.softcatala.org/traductor/.
+The models are quantified for low latency.
+
+### Usage
+
+Install the necessary dependencies:
+
+
+```bash
+pip3 install ctranslate2 pyonmttok
+```
+
+
+Simple tokenization & translation using Python:
+
+
+```python
+import ctranslate2
+import pyonmttok
+from huggingface_hub import snapshot_download
+model_dir = snapshot_download(repo_id="softcatala/translate-cat-jpn", revision="main")
+
+tokenizer=pyonmttok.Tokenizer(mode="none", sp_model_path = model_dir + "/sp_m.model")
+tokenized=tokenizer.tokenize("Hola amics")
+
+translator = ctranslate2.Translator(model_dir)
+translated = translator.translate_batch([tokenized[0]])
+print(tokenizer.detokenize(translated[0][0]['tokens']))
+```
+
+## Benchmarks
+
+| testset                               | BLEU  |
+|---------------------------------------|-------|
+| test dataset (from train/dev/test)   	| 21.3	|
+| Flores200 dataset                     | 19.8	|
+
+## Additional information
+* https://github.com/Softcatala/nmt-models
+* https://github.com/Softcatala/parallel-catalan-corpus

config.json

@@ -0,0 +1,2 @@
+{
+}

model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4e1a5087e77cabf47474c57fdf96a41a0e5215cfe53b5585ee3cf1d48a091fa5
+size 70727761

model_description.txt

@@ -0,0 +1,9 @@
+Model description: cat-jpn
+Date: 2023-02-18
+TF version 2.10.0, OpenNMT version 2.29.1, CTranslate2 version 2.24.0
+Test data set
+BLEU|nrefs:1|case:mixed|eff:no|tok:ja-mecab-0.996-IPA|smooth:exp|version:2.1.0 = 21.3 50.4/29.3/20.1/15.2 (BP = 0.821 ratio = 0.835 hyp_len = 86497 ref_len = 103594)
+             chrF2|nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0 = 25.7
+Flores data set
+BLEU|nrefs:1|case:mixed|eff:no|tok:ja-mecab-0.996-IPA|smooth:exp|version:2.1.0 = 19.8 59.7/30.3/17.2/10.1 (BP = 0.836 ratio = 0.848 hyp_len = 28227 ref_len = 33286)
+             chrF2|nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0 = 28.3

sp_m.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50d319430280ec73ccdf61e4b258af5905368105f64165d8e896f19c8985ab66
+size 1159941