<p>I don't why this RNN k-mer Pytorch model is not producing an output. I do know that,</p>
<pre><code> &quot;rnn_dropout&quot;: 0.0,
</code></pre>
<p>... is bad news for any ANN (artificial neural network) let alone LSTM RNN, which is a pretty complex model.</p>
<p>Firstly, you have to assign a meaningful drop out rate and 0.5 is okay.</p>
<p>If dropout rate = 0 as an output then the model hasn't worked. A dropout layer is essential to present overfitting. Its a foundational part of deep learning. No-one emperically knows quite why it works, there's a few approximate good guesses, but a dropout is essential.</p>
<hr />
<p>The other thing to note is what does &quot;n_process&quot; mean? If you are parallelising a GPU across 4-cores, okay its got a lot of layers but even so thats alot of cores. It ain't a CPU and I'd get it working on a single GPU core first.</p>

CC BY-SA 4.0
2020-11-10T01:54:52.497Z
I see your points. However, this should work presumably out of the box. I have tried to use google colab too, but it was too complicated there compared to SageMaker.
0x90
CC BY-SA 4.0
2020-11-10T02:00:47.430Z
Yep thats Google! Its deep learning is always geeky  :-)  I'm surprised to hear this about colab because it is supposed to be set up for DL, but can vouch 100% thats what  Google developers do. Tensorflow (Google) vs PyTorch (FaceBook) ... is similar, PyTorch is much less geekified.
M__
CC BY-SA 4.0
2020-11-10T02:02:31.553Z
I was intending to give DNABERT a try as it was published on GitHub. Sadly the developers haven’t replied back to me. Feel free to give it a try and see if it works for you as-is.
0x90
CC BY-SA 4.0
2020-11-10T02:06:24.640Z
I don't know, because I don't know how they are vectorising their k-mers. In my opinion converting DNA to numbers for DL is key to the results. You are using a solid model with a crazy number of layers (12 is alot). Anyway, it was fun to chat.
M__
CC BY-SA 4.0
2020-11-10T13:51:13.320Z
They use BERT which is a "crazy" model by definition. Here is the paper in case someone wants to see it biorxiv.org/content/10.1101/2020.09.17.301879v1
0x90
CC BY-SA 4.0
2020-11-10T13:58:37.057Z
Thanks @0x90. Sorry I said 'crazy' what I meant was the number of layers was alot. This is a good thing and was (perhaps not now) the basic method for increasing the accuracy of the model in any field of DL. This approach was famously deployed by one big player (was it Google?) to win an image training competition with the other big players. Its just in bioinformatics its the most layers I've seen in any model.
M__
CC BY-SA 4.0
2020-11-10T01:46:32.797Z
2020-11-10T01:51:57.023Z
2020-11-10T01:51:57.023Z
M__
<p>I was curious to give <a href="https://github.com/jerryji1993/DNABERT" rel="nofollow noreferrer">DNA BERT</a> a try. This is a <a href="https://en.wikipedia.org/wiki/BERT_(language_model)" rel="nofollow noreferrer">BERT</a> (Bidirectional Encoder Representations from Transformers) model that was trained on short (<code>k=3,4,5, or 6</code>) k-mers of DNA. Overall, It is exciting work since it bridges NLP approaches and genetics data.</p>
<p>I span up a pytroch3.6 machine on SageMaker with a GPU and tried to assess the prediction of its pre-trained DNABERT model:</p>
<p>Preparing the data:</p>
<pre><code>!git clone https://github.com/jerryji1993/DNABERT
%cd DNABERT
!python -m pip install --editable .
%cd examples
!python -m pip install -r requirements.txt
!curl -L https://northwestern.app.box.com/shared/static/g8m974tr86h0pvnpymxq84f1yxlhnvbi --output a6-new-12w-0.zip
!unzip a6-new-12w-0.zip
!cp ./sample_data/ft/prom-core/6/dev.tsv /home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0
</code></pre>
<ol start="2">
<li>Running <a href="https://github.com/jerryji1993/DNABERT#4-prediction" rel="nofollow noreferrer">prediction</a> based on the pre-trained data:</li>
</ol>
<pre><code>!python run_finetune.py \
   --model_type=dna \
   --tokenizer_name=dna6 \
   --model_name_or_path=/home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0 \
   --task_name=dnaprom \
   --do_predict \
   --data_dir=/home/ec2-user/SageMaker/DNABERT/examples/sample_data/ft/prom-core/6  \
   --max_seq_length=75 \
   --per_gpu_pred_batch_size=128   \
   --output_dir=/home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0/out \
   --predict_dir=/home/ec2-user/SageMaker/DNABERT/examples/result \
   --n_process=4
</code></pre>
<p>Output:</p>
<pre><code>11/09/2020 04:35:57 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 1, distributed training: False, 16-bits training: False
11/09/2020 04:35:57 - INFO - transformers.configuration_utils -   loading configuration file /home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0/config.json
11/09/2020 04:35:57 - INFO - transformers.configuration_utils -   Model config BertConfig {
  &quot;architectures&quot;: [
    &quot;BertForMaskedLM&quot;
  ],
  &quot;attention_probs_dropout_prob&quot;: 0.1,
  &quot;bos_token_id&quot;: 0,
  &quot;do_sample&quot;: false,
  &quot;eos_token_ids&quot;: 0,
  &quot;finetuning_task&quot;: &quot;dnaprom&quot;,
  &quot;hidden_act&quot;: &quot;gelu&quot;,
  &quot;hidden_dropout_prob&quot;: 0.1,
  &quot;hidden_size&quot;: 768,
  &quot;id2label&quot;: {
    &quot;0&quot;: &quot;LABEL_0&quot;,
    &quot;1&quot;: &quot;LABEL_1&quot;
  },
  &quot;initializer_range&quot;: 0.02,
  &quot;intermediate_size&quot;: 3072,
  &quot;is_decoder&quot;: false,
  &quot;label2id&quot;: {
    &quot;LABEL_0&quot;: 0,
    &quot;LABEL_1&quot;: 1
  },
  &quot;layer_norm_eps&quot;: 1e-12,
  &quot;length_penalty&quot;: 1.0,
  &quot;max_length&quot;: 20,
  &quot;max_position_embeddings&quot;: 512,
  &quot;model_type&quot;: &quot;bert&quot;,
  &quot;num_attention_heads&quot;: 12,
  &quot;num_beams&quot;: 1,
  &quot;num_hidden_layers&quot;: 12,
  &quot;num_labels&quot;: 2,
  &quot;num_return_sequences&quot;: 1,
  &quot;num_rnn_layer&quot;: 1,
  &quot;output_attentions&quot;: false,
  &quot;output_hidden_states&quot;: false,
  &quot;output_past&quot;: true,
  &quot;pad_token_id&quot;: 0,
  &quot;pruned_heads&quot;: {},
  &quot;repetition_penalty&quot;: 1.0,
  &quot;rnn&quot;: &quot;lstm&quot;,
  &quot;rnn_dropout&quot;: 0.0,
  &quot;rnn_hidden&quot;: 768,
  &quot;split&quot;: 10,
  &quot;temperature&quot;: 1.0,
  &quot;top_k&quot;: 50,
  &quot;top_p&quot;: 1.0,
  &quot;torchscript&quot;: false,
  &quot;type_vocab_size&quot;: 2,
  &quot;use_bfloat16&quot;: false,
  &quot;vocab_size&quot;: 4101
}

============================================================
&lt;class 'transformers.tokenization_dna.DNATokenizer'&gt;
11/09/2020 04:35:57 - INFO - filelock -   Lock 140334164036072 acquired on /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e.lock
11/09/2020 04:35:57 - INFO - transformers.file_utils -   https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt not found in cache or force_download set to True, downloading to /home/ec2-user/.cache/torch/transformers/tmpa9hkn2q1
Downloading: 28.7kB [00:00, 12.5MB/s]                                           
11/09/2020 04:35:57 - INFO - transformers.file_utils -   storing https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt in cache at /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
11/09/2020 04:35:57 - INFO - transformers.file_utils -   creating metadata file for /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
11/09/2020 04:35:57 - INFO - filelock -   Lock 140334164036072 released on /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e.lock
11/09/2020 04:35:57 - INFO - transformers.tokenization_utils -   loading file https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt from cache at /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
11/09/2020 04:35:57 - INFO - transformers.modeling_utils -   loading weights file /home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0/pytorch_model.bin
</code></pre>
<p>However, the <code>result</code> directory is empty and I am not sure what did I miss?</p>
<pre><code>!ls -lh /home/ec2-user/SageMaker/DNABERT/examples/result
total 0


!ls -l  /home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0/out
total 0
</code></pre>
<hr />
<p><strong>EDIT:</strong>
I have tried also to run this on another ec2 machine and apparently it gets a segmentation fault, so my conclusion is that it's unstable. I will train a RoBERTAa model or similar by myself:</p>
<p>SageMaker doesn't propagate the segmentation fault to the notebook which is another weird thing about it.</p>
<pre><code>11/12/2020 14:01:18 - INFO - filelock -   Lock 139866003860056 acquired on /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e.lock
11/12/2020 14:01:18 - INFO - transformers.file_utils -   https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt not found in cache or force_download set to True, downloading to /root/.cache/torch/transformers/tmpwpxfkxlm
Downloading: 28.7kB [00:00, 20.2MB/s]
11/12/2020 14:01:18 - INFO - transformers.file_utils -   storing https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt in cache at /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
11/12/2020 14:01:18 - INFO - transformers.file_utils -   creating metadata file for /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
11/12/2020 14:01:18 - INFO - filelock -   Lock 139866003860056 released on /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e.lock
11/12/2020 14:01:18 - INFO - transformers.tokenization_utils -   loading file https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt from cache at /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
11/12/2020 14:01:18 - INFO - transformers.modeling_utils -   loading weights file /home/ec2-user/DNABERT/examples/6-new-12w-0/pytorch_model.bin
Segmentation fault
</code></pre>

CC BY-SA 4.0
2020-11-12T07:58:50.053Z
I wonder if the data science stackexchange would end up being better for this question. I'd want to remove Michaels answer first though, which would be a shame. Could you maybe just repost this over there and provide a link back for context?
Devon Ryan
<deep-learning><pytorch><python>
How to predict with the pre-trained DNABERT model?
# How to predict with the pre-trained DNABERT model?

(Asked by: username_0 on 2020-11-10)

I was curious to give DNA BERT a try. This is a BERT (Bidirectional Encoder Representations from Transformers) model that was trained on short (k=3,4,5, or 6) k-mers of DNA. Overall, It is exciting work since it bridges NLP approaches and genetics data.
I span up a pytroch3.6 machine on SageMaker with a GPU and tried to assess the prediction of its pre-trained DNABERT model:
Preparing the data:
!git clone https://github.com/jerryji1993/DNABERT
%cd DNABERT
!python -m pip install --editable .
%cd examples
!python -m pip install -r requirements.txt
!curl -L https://northwestern.app.box.com/shared/static/g8m974tr86h0pvnpymxq84f1yxlhnvbi --output a6-new-12w-0.zip
!unzip a6-new-12w-0.zip
!cp ./sample_data/ft/prom-core/6/dev.tsv /home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0


Running prediction based on the pre-trained data:

!python run_finetune.py \
   --model_type=dna \
   --tokenizer_name=dna6 \
   --model_name_or_path=/home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0 \
   --task_name=dnaprom \
   --do_predict \
   --data_dir=/home/ec2-user/SageMaker/DNABERT/examples/sample_data/ft/prom-core/6  \
   --max_seq_length=75 \
   --per_gpu_pred_batch_size=128   \
   --output_dir=/home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0/out \
   --predict_dir=/home/ec2-user/SageMaker/DNABERT/examples/result \
   --n_process=4

Output:
11/09/2020 04:35:57 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 1, distributed training: False, 16-bits training: False
11/09/2020 04:35:57 - INFO - transformers.configuration_utils -   loading configuration file /home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0/config.json
11/09/2020 04:35:57 - INFO - transformers.configuration_utils -   Model config BertConfig {
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "do_sample": false,
  "eos_token_ids": 0,
  "finetuning_task": "dnaprom",
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "is_decoder": false,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "layer_norm_eps": 1e-12,
  "length_penalty": 1.0,
  "max_length": 20,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_beams": 1,
  "num_hidden_layers": 12,
  "num_labels": 2,
  "num_return_sequences": 1,
  "num_rnn_layer": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pad_token_id": 0,
  "pruned_heads": {},
  "repetition_penalty": 1.0,
  "rnn": "lstm",
  "rnn_dropout": 0.0,
  "rnn_hidden": 768,
  "split": 10,
  "temperature": 1.0,
  "top_k": 50,
  "top_p": 1.0,
  "torchscript": false,
  "type_vocab_size": 2,
  "use_bfloat16": false,
  "vocab_size": 4101
}

============================================================

11/09/2020 04:35:57 - INFO - filelock -   Lock 140334164036072 acquired on /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e.lock
11/09/2020 04:35:57 - INFO - transformers.file_utils -   https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt not found in cache or force_download set to True, downloading to /home/ec2-user/.cache/torch/transformers/tmpa9hkn2q1
Downloading: 28.7kB [00:00, 12.5MB/s]                                           
11/09/2020 04:35:57 - INFO - transformers.file_utils -   storing https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt in cache at /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
11/09/2020 04:35:57 - INFO - transformers.file_utils -   creating metadata file for /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
11/09/2020 04:35:57 - INFO - filelock -   Lock 140334164036072 released on /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e.lock
11/09/2020 04:35:57 - INFO - transformers.tokenization_utils -   loading file https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt from cache at /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
11/09/2020 04:35:57 - INFO - transformers.modeling_utils -   loading weights file /home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0/pytorch_model.bin

However, the result directory is empty and I am not sure what did I miss?
!ls -lh /home/ec2-user/SageMaker/DNABERT/examples/result
total 0


!ls -l  /home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0/out
total 0


EDIT:
I have tried also to run this on another ec2 machine and apparently it gets a segmentation fault, so my conclusion is that it's unstable. I will train a RoBERTAa model or similar by myself:
SageMaker doesn't propagate the segmentation fault to the notebook which is another weird thing about it.
11/12/2020 14:01:18 - INFO - filelock -   Lock 139866003860056 acquired on /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e.lock
11/12/2020 14:01:18 - INFO - transformers.file_utils -   https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt not found in cache or force_download set to True, downloading to /root/.cache/torch/transformers/tmpwpxfkxlm
Downloading: 28.7kB [00:00, 20.2MB/s]
11/12/2020 14:01:18 - INFO - transformers.file_utils -   storing https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt in cache at /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
11/12/2020 14:01:18 - INFO - transformers.file_utils -   creating metadata file for /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
11/12/2020 14:01:18 - INFO - filelock -   Lock 139866003860056 released on /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e.lock
11/12/2020 14:01:18 - INFO - transformers.tokenization_utils -   loading file https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt from cache at /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
11/12/2020 14:01:18 - INFO - transformers.modeling_utils -   loading weights file /home/ec2-user/DNABERT/examples/6-new-12w-0/pytorch_model.bin
Segmentation fault

**Tags:** deep-learning, pytorch, python
> **Comments:**
> **username_1**: I wonder if the data science stackexchange would end up being better for this question. I'd want to remove Michaels answer first though, which would be a shame. Could you maybe just repost this over there and provide a link back for context?  

---

## 1 answers

### Answer by username_2 on 2020-11-10. Score: 1

I don't why this RNN k-mer Pytorch model is not producing an output. I do know that,
 "rnn_dropout": 0.0,

... is bad news for any ANN (artificial neural network) let alone LSTM RNN, which is a pretty complex model.
Firstly, you have to assign a meaningful drop out rate and 0.5 is okay.
If dropout rate = 0 as an output then the model hasn't worked. A dropout layer is essential to present overfitting. Its a foundational part of deep learning. No-one emperically knows quite why it works, there's a few approximate good guesses, but a dropout is essential.

The other thing to note is what does "n_process" mean? If you are parallelising a GPU across 4-cores, okay its got a lot of layers but even so thats alot of cores. It ain't a CPU and I'd get it working on a single GPU core first.

> **Comments:**
> **username_0**: I see your points. However, this should work presumably out of the box. I have tried to use google colab too, but it was too complicated there compared to SageMaker.  
> **username_2**: Yep thats Google! Its deep learning is always geeky  :-)  I'm surprised to hear this about colab because it is supposed to be set up for DL, but can vouch 100% thats what  Google developers do. Tensorflow (Google) vs PyTorch (FaceBook) ... is similar, PyTorch is much less geekified.  
> **username_0**: I was intending to give DNABERT a try as it was published on GitHub. Sadly the developers haven’t replied back to me. Feel free to give it a try and see if it works for you as-is.  
> **username_2**: I don't know, because I don't know how they are vectorising their k-mers. In my opinion converting DNA to numbers for DL is key to the results. You are using a solid model with a crazy number of layers (12 is alot). Anyway, it was fun to chat.  
> **username_0**: They use BERT which is a "crazy" model by definition. Here is the paper in case someone wants to see it biorxiv.org/content/10.1101/2020.09.17.301879v1  
> **username_2**: Thanks @0x90. Sorry I said 'crazy' what I meant was the number of layers was alot. This is a good thing and was (perhaps not now) the basic method for increasing the accuracy of the model in any field of DL. This approach was famously deployed by one big player (was it Google?) to win an image training competition with the other big players. Its just in bioinformatics its the most layers I've seen in any model.  

---

