21×fewer parameters than the prior state of the art, at comparable performance
92×less GPU time for pre-training
36datasets across 9 tasks in GUE
23 of 28datasets where it beats DNABERT
What it is
DNABERT-2 learns the language of DNA from genomes of many species. Byte pair encoding lets the model build its own vocabulary of sequence pieces, making it smaller and faster than earlier genome language models without losing accuracy.
Role in BODHI
DNABERT-2 is co-developed with the MAGICS Lab and is the starting point for several BODHI models, including HViLM and DNABERT-MB.
Papers
Team
Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana V. Davuluri, Han Liu
Cite
@inproceedings{zhou2024dnabert2,
title = {DNABERT-2: Efficient Foundation Model and Benchmark for Multi-Species Genomes},
author = {Zhou, Zhihan and Ji, Yanrong and Li, Weijian and Dutta, Pratik and
Davuluri, Ramana V. and Liu, Han},
booktitle = {International Conference on Learning Representations (ICLR)},
year = {2024}
}