Download PDF

Evaluating SciBERT and Large Language Models for Joint Scientific Entity and Relation Extraction

Abdelhal EL OMARI, Jilali ANTARI
Laboratory of Computer Systems Engineering- Mathematics and Applications (ISIMA), Polydisciplinary Faculty of Taroudant, Ibn Zohr University, 80000, Agadir, Morocco.
abdelhal.elomari.19@edu.uiz.ac.ma, jantari@uiz.ac.ma
How to cite: Abdelhal EL OMARI, Jilali ANTARI, "Evaluating SciBERT and Large Language Models for Joint Scientific Entity and Relation Extraction", Sciences Methods and Technologies International Journal (SciMeTech), Vol 2, Issue 2, p 1-6
Abstract
The automatic extraction of structured knowledge from scientific publications re- quires both robust recognition of entities and accurate identification of semantic relations. Traditional domain- specific models such as SciBERT have shown promising results, while recent large language models (LLMs) raise questions about their adaptability to specialized scientific tasks. In this study, we perform a comparative evaluation of joint entity and relation extraction using SciBERT, LLaMA- 2, GPT- 3.5- turbo, and GPT- 4o- mini on the SciERC corpus, which contains 450 annotated scientific abstracts. Experiments are conducted on annotated scientific corpora covering four entity types (Material, Method, Metric, Task) and two relation types (USED- FOR, EVALUATE- FOR). According to the results, SciBERT is still the best model for relation extraction (F1 = 0.674), whereas GPT- 4o- mini is best for named entity recognition (F1 = 0.623). In particular, LLaMA- 2 and GPT- 3.5- turbo show little to no success. This shows a contrast: for relation extraction, domain- specific models outperform general purpose large language models (LLMs), and for named entity recognition, the reverse is true. We suggest mixed approaches, which combine the best of both worlds, as the way forward for building scientific knowledge graphs.
Keywords: Named entity recognition, Relation extraction, SciBERT, Large Language Models

References

  1. Z. Hong, L. Ward, K. Chard, B. Blaiszik, and I. Foster, "Challenges and advances in information extraction from scientific literature: a review," JOM, vol. 73, no. 11, pp. 3383- 3400, 2021.
  2. M. H. A. Abdullah, N. Aziz, S. J. Abdulkadir, H. S. A. Alhussain, and N. Talpur, "Systematic literature review of information extraction from textual data: recent methods, applications, trends, and challenges," IEEE Access, vol. 11, pp. 10535- 10562, 2023.
  3. J. Li, A. Sun, J. Han, and C. Li, "A survey on deep learning for named entity recogni- tion," IEEE Trans. Knowl. Data Eng., vol. 34, no. 1, pp. 50- 70, 2020.
  4. K. Detroja, C. K. Bhsndadia, and B. S. Bhatt, "A survey on relation extraction," Intelli- gent Systems with Applications, vol. 19, p. 200244, 2023.
  5. Z. Yan, Z. Jia, and K. Tu, "An empirical study of pipeline vs. joint approaches to entity and relation extraction," in Proceedings of the 2nd Conference of the Asia- Pacific Chap- ter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), 2022, pp. 437- 443.
  6. S. Jain, M. Van Zuylen, H. Hajishirzi, and I. Beltagy, "SciREX: A challenge dataset for document- level information extraction," arXiv preprint arXiv:2005.00512, 2020.
  7. I. Beltagy, K. Lo, and A. Cohan, "SciBERT: A pretrained language model for scientific text," arXiv preprint arXiv:1903.10676, 2019.
  8. H. Touvron et al., "Llama 2: Open foundation and fine- tuned chat models," arXiv pre- print arXiv:2307.09288, 2023.
  9. K. Abramski, S. Citraro, L. Lombardi, G. Rossetti, and M. Stella, "Cognitive network science reveals bias in GPT- 3, GPT- 3.5 Turbo, and GPT- 4 mirroring math anxiety in high- school students," Big Data and Cognitive Computing, vol. 7, no. 3, p. 124, 2023.
  10. Y. Luan, L. He, M. Ostendorf, and H. Hajishirzi, "Multi- task identification of entities, relations, and coreference for scientific knowledge graph construction," arXiv preprint arXiv:1808.09602, 2018.
SciMeTech · Vol 2, Issue 2 · p 1-6