OPI-Galactica-6.7B / README.md

Update README.md

2a299f2 verified 9 months ago

6.38 kB

	---
	license: apache-2.0
	datasets:
	- BAAI/OPI
	language:
	- en
	pipeline_tag: text-generation
	tags:
	- Life Science
	- AI4Science
	- Biology
	- Protein
	- LLM
	- Instruction
	base_model: facebook/galactica-6.7b
	---
	![OPI_logo](demo_figures/OPI_logo.png)

	# Github:
	https://github.com/baaihealth/opi
	# Paper:
	[OPI: An Open Instruction Dataset for Adapting Large Language Models to Protein-Related Tasks](https://neurips.cc/virtual/2024/105921) has been accepted by [NeurIPS 2024 Workshop: Foundation Models for Science: Progress, Opportunities, and Challenges](https://neurips.cc/virtual/2024/workshop/84714).

	# Model Card of OPI-Galactica-6.7B
	OPI-Galactica-6.7B was fine-tuned from the Galactica-6.7B model using the complete OPI training set (i.e.,[OPI_full_1.61M_train.json](https://huggingface.co/datasets/BAAI/OPI/blob/main/OPI_DATA/OPI_full_1.61M_train.json)).
	For more details of training and testing, please visit [https://github.com/baaihealth/opi](https://github.com/baaihealth/opi).

	![Overview](demo_figures/OPI_experiment_outline.png)

	# Evaluation of OPI-Galactica-6.7B model on 9 tasks
	Each testing result is derived from the Galactica-6.7B model that has been fine-tuned using [OPI_full_1.61M.json](https://huggingface.co/datasets/BAAI/OPI/blob/main/OPI_DATA/OPI_full_1.61M_train.json) and subsequently evaluated on the respective testing set for each specific task.

	<table border="1" style="text-align:center; border-collapse:collapse; width: 100%;">
	<thead>
	<tr>
	<th style="text-align:center;">Task Type</th>
	<th style="text-align:center;">Task Name</th>
	<th style="text-align:center;">Testing file</th>
	<th style="text-align:center;">Accuracy</th>
	<th style="text-align:center;">Precision</th>
	<th style="text-align:center;">Recall</th>
	<th style="text-align:center;">F1</th>
	<th style="text-align:center;">Rouge-L</th>
	</tr>
	</thead>
	<tbody>
	<tr>
	<td rowspan="6">Sequence Understanding</td>
	<td rowspan="2">EC Number Prediction (split100)</td>
	<td>CLEAN_EC_number_new_test</td>
	<td>-</td>
	<td>0.2700</td>
	<td>0.2663</td>
	<td>0.2596</td>
	<td>-</td>
	</tr>
	<tr>
	<td>CLEAN_EC_number_price_test</td>
	<td>-</td>
	<td>0.0268</td>
	<td>0.0268</td>
	<td>0.0268</td>
	<td>-</td>
	</tr>
	<tr>
	<td rowspan="3">Fold Type Prediction</td>
	<td>fold_type_test_Fold_Holdout</td>
	<td>0.0808</td>
	<td>-</td>
	<td>-</td>
	<td>-</td>
	<td>-</td>
	</tr>
	<tr>
	<td>fold_type_test_Superfamily_Holdout</td>
	<td>0.1348</td>
	<td>-</td>
	<td>-</td>
	<td>-</td>
	<td>-</td>
	</tr>
	<tr>
	<td>fold_type_test_Family_Holdout</td>
	<td>0.4854</td>
	<td>-</td>
	<td>-</td>
	<td>-</td>
	<td>-</td>
	</tr>
	<tr>
	<td>Subcellular Localization Prediction</td>
	<td>subcell_loc_test</td>
	<td>0.7771</td>
	<td>-</td>
	<td>-</td>
	<td>-</td>
	<td>-</td>
	</tr>
	<tr>
	<td rowspan="9">Annotation Prediction</td>
	<td>Function Keywords Prediction</td>
	<td>CASPSimilarSeq_keywords_test</td>
	<td>-</td>
	<td>0.8120</td>
	<td>0.7360</td>
	<td>0.7643</td>
	<td>-</td>
	</tr>
	<tr>
	<td>Function Keywords Prediction</td>
	<td>IDFilterSeq_keywords_test</td>
	<td>-</td>
	<td>0.8377</td>
	<td>0.8019</td>
	<td>0.8070</td>
	<td>-</td>
	</tr>
	<tr>
	<td>Function Keywords Prediction</td>
	<td>UniProtSeq_keywords_test</td>
	<td>-</td>
	<td>0.8596</td>
	<td>0.8196</td>
	<td>0.8276</td>
	<td>-</td>
	</tr>
	<tr>
	<td>Gene Ontology (GO) Terms Prediction</td>
	<td>CASPSimilarSeq_go_terms_test</td>
	<td>-</td>
	<td>0.7613</td>
	<td>0.7492</td>
	<td>0.7476</td>
	<td>-</td>
	</tr>
	<tr>
	<td>Gene Ontology (GO) Terms Prediction</td>
	<td>IDFilterSeq_go_terms_test</td>
	<td>-</td>
	<td>0.7404</td>
	<td>0.7274</td>
	<td>0.7207</td>
	<td>-</td>
	</tr>
	<tr>
	<td>Gene Ontology (GO) Terms Prediction</td>
	<td>UniProtSeq_go_terms_test</td>
	<td>-</td>
	<td>0.7638</td>
	<td>0.7373</td>
	<td>0.7358</td>
	<td>-</td>
	</tr>
	<tr>
	<td>Function Description Prediction</td>
	<td>CASPSimilarSeq_function_test</td>
	<td>-</td>
	<td>-</td>
	<td>-</td>
	<td>-</td>
	<td>0.7430</td>
	</tr>
	<tr>
	<td>Function Description Prediction</td>
	<td>IDFilterSeq_function_test</td>
	<td>-</td>
	<td>-</td>
	<td>-</td>
	<td>-</td>
	<td>0.7014</td>
	</tr>
	<tr>
	<td>Function Description Prediction</td>
	<td>UniProtSeq_function_test</td>
	<td>-</td>
	<td>-</td>
	<td>-</td>
	<td>-</td>
	<td>0.7133</td>
	</tr>
	<tr>
	<td rowspan="3">Knowledge Mining</td>
	<td>Tissue Location Prediction from Gene Symbol</td>
	<td>gene_symbol_to_tissue_test</td>
	<td>-</td>
	<td>0.3917</td>
	<td>0.9077</td>
	<td>0.5303</td>
	<td>-</td>
	</tr>
	<tr>
	<td>Cancer Prediction from Gene Symbol</td>
	<td>gene_symbol_to_cancer_test</td>
	<td>-</td>
	<td>0.3555</td>
	<td>0.3189</td>
	<td>0.3229</td>
	<td>-</td>
	</tr>
	<tr>
	<td>Cancer Prediction from Gene Name</td>
	<td>gene_name_to_cancer_test</td>
	<td>-</td>
	<td>0.2728</td>
	<td>0.2554</td>
	<td>0.2533</td>
	<td>-</td>
	</tr>
	</tbody>
	</table>

	# Prediction comparison with SOTA mdoels

	![model_compare](model_compare/task1_EC_number.png)
	![model_compare](model_compare/task2_fold_type.png)
	![model_compare](model_compare/task3_subcell_loc.png)
	![model_compare](model_compare/task4_keywords.png)
	![model_compare](model_compare/task5_GO.png)
	![model_compare](model_compare/task6_function.png)
	![model_compare](model_compare/task7_gsymbol2tissue.png)
	![model_compare](model_compare/task8_gsymbol2cancer.png)
	![model_compare](model_compare/task9_gname2cancer.png)


	# Demo
	We use the [FastChat](https://github.com/lm-sys/FastChat) platform to visually demonstrate the ability of OPI-Galactica-6.7B model on various evaluation tasks.

	![OPI Demo](./demo_figures/OPI_demo.gif)