Supervised Project

Summary

NLG Task: 
 
The task is to provide a pre-train T5 model for a multitude of languages, there is 101 languages for this model
 
Training Data: 
Data comes from Common Crawl (mC4) it is in more than 100 Languages. There is about 1 trillion token of data used for pre-training the model who is about 450 GB of data from C4 (Common Crawl web space). For non- English data, the cld3 model is used to determine the language of the page.
 
Model Description: 
A Transformer is employed (The T5 transformer is used). It can be used with 60 Million to 11 Billion parameters. The vocabulary used is composed of 250,000 wordpiece. The model use 1 million steps on 1024 length-1024 input
 
Key Contribution: 
This model demonstrates that the T5 model works for multilingual and can perform it quickly. Also this model can detect illegal prediction so it can be removed automatically. 
All the code and models used can be found on their github at the link : <a href="https://github.com/google-research/multilingual-t5">https://github.com/google-research/multilingual-t5</a>

Result: 
F1 scores for every model compared to mT5
<style type="text/css">
.tg {border-collapse:collapse;border-spacing:0;margin:0px auto;}
.tg td{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;
 overflow:hidden;padding:10px 5px;word-break:normal;}
.tg th{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;
 font-weight:normal;overflow:hidden;padding:10px 5px;word-break:normal;}
.tg .tg-cly1{text-align:left;vertical-align:middle}
.tg .tg-1wig{font-weight:bold;text-align:left;vertical-align:top}
.tg .tg-wa1i{font-weight:bold;text-align:center;vertical-align:middle}
</style>
<table class="tg">
<thead>
 <tr>
 <th class="tg-wa1i">Model</th>
 <th class="tg-wa1i">XNLI</th>
 <th class="tg-wa1i">PAWS-X</th>
 <th class="tg-wa1i">WikiAnn-NER</th>
 <th class="tg-wa1i">XQuAD</th>
 <th class="tg-wa1i">MLQA</th>
 <th class="tg-wa1i">TyDiQA-GoldP</th>
 </tr>
</thead>
<tbody>
 <tr>
 <td class="tg-cly1">mBERT</td>
 <td class="tg-cly1">65.4</td>
 <td class="tg-cly1">81.9</td>
 <td class="tg-cly1">62.2</td>
 <td class="tg-cly1">64.5</td>
 <td class="tg-cly1">61.4</td>
 <td class="tg-cly1">59.7</td>
 </tr>
 <tr>
 <td class="tg-cly1">XLM</td>
 <td class="tg-cly1">69.1</td>
 <td class="tg-cly1">80.9</td>
 <td class="tg-cly1">61.2</td>
 <td class="tg-cly1">59.8</td>
 <td class="tg-cly1">48.5</td>
 <td class="tg-cly1">43.6</td>
 </tr>
 <tr>
 <td class="tg-cly1">InfoXLM</td>
 <td class="tg-cly1">81.4</td>
 <td class="tg-cly1">-</td>
 <td class="tg-cly1">-</td>
 <td class="tg-cly1">-</td>
 <td class="tg-cly1">73.6</td>
 <td class="tg-cly1">-</td>
 </tr>
 <tr>
 <td class="tg-cly1">X-STILTs</td>
 <td class="tg-cly1">80.4</td>
 <td class="tg-cly1">87.7</td>
 <td class="tg-cly1">64.7</td>
 <td class="tg-cly1">77.2</td>
 <td class="tg-cly1">72.3</td>
 <td class="tg-cly1">76.0</td>
 </tr>
 <tr>
 <td class="tg-cly1">XLM-R</td>
 <td class="tg-cly1">79.2</td>
 <td class="tg-cly1">86.4</td>
 <td class="tg-cly1">65.4</td>
 <td class="tg-cly1">76.6</td>
 <td class="tg-cly1">71.6</td>
 <td class="tg-cly1">65.1</td>
 </tr>
 <tr>
 <td class="tg-cly1">VECO</td>
 <td class="tg-cly1">79.9</td>
 <td class="tg-cly1">88.7</td>
 <td class="tg-cly1">65.7</td>
 <td class="tg-cly1">77.3</td>
 <td class="tg-cly1">71.7</td>
 <td class="tg-cly1">67.6</td>
 </tr>
 <tr>
 <td class="tg-cly1">RemBERT</td>
 <td class="tg-cly1">80.8</td>
 <td class="tg-cly1">87.5</td>
 <td class="tg-1wig">70.1</td>
 <td class="tg-cly1">79.6</td>
 <td class="tg-cly1">73.1</td>
 <td class="tg-cly1">77.0</td>
 </tr>
 <tr>
 <td class="tg-cly1">mT5-Small</td>
 <td class="tg-cly1">67.5</td>
 <td class="tg-cly1">82.4</td>
 <td class="tg-cly1">50.5</td>
 <td class="tg-cly1">58.1</td>
 <td class="tg-cly1">54.6</td>
 <td class="tg-cly1">36.4</td>
 </tr>
 <tr>
 <td class="tg-cly1">mT5-Base</td>
 <td class="tg-cly1">75.4</td>
 <td class="tg-cly1">86.4</td>
 <td class="tg-cly1">55.7</td>
 <td class="tg-cly1">67.0</td>
 <td class="tg-cly1">64.6</td>
 <td class="tg-cly1">59.1</td>
 </tr>
 <tr>
 <td class="tg-cly1">mT5-Large</td>
 <td class="tg-cly1">81.1</td>
 <td class="tg-cly1">88.9</td>
 <td class="tg-cly1">58.5</td>
 <td class="tg-cly1">77.8</td>
 <td class="tg-cly1">71.2</td>
 <td class="tg-cly1">68.4</td>
 </tr>
 <tr>
 <td class="tg-cly1">mT5-XL</td>
 <td class="tg-cly1">82.9</td>
 <td class="tg-cly1">89.6</td>
 <td class="tg-cly1">65.5</td>
 <td class="tg-cly1">79.5</td>
 <td class="tg-cly1">73.5</td>
 <td class="tg-cly1">77.8</td>
 </tr>
 <tr>
 <td class="tg-cly1">mT5-XXL</td>
 <td class="tg-1wig">85.0</td>
 <td class="tg-1wig">90.0</td>
 <td class="tg-cly1">69.2</td>
 <td class="tg-1wig">82.5</td>
 <td class="tg-1wig">76.0</td>
 <td class="tg-1wig">82.0</td>
 </tr>
</tbody>
</table>