Semantic clustering of scientific abstracts with transformer embeddings and traditional text representations

Abstract

The large and diverse quantity of scientific documents in the world, and specifically in Indonesia, makes the process of processing scientific document data an interesting study. One that represents the entire scientific document is through abstracts. The approach that can be done for the process of processing and grouping documents is to apply clustering. In this case, text-based clustering is currently heavily influenced by the feature representation of the text data used. Some popular representations of features are term frequency-inverse document frequency (TF-IDF) and count vectorizer, but they still have significant weaknesses in the context of understanding the meaning of natural language. To cover the drawbacks, it can use transformers or embedding types. In this study, several test scenarios will be carried out to obtain information and an overview of how to compare the effectiveness of the traditional TF-IDF model and the bidirectional encoder representations from transformers (BERT) and sentence bidirectional encoder representations from transformers (SBERT) embedding models in Indonesian-language scientific abstract clustering with several clustering models, such as k-means and agglomerative. The results of the study showed that the most effective clustering obtained was by using an embedding model of a combination of BERT and k-means, which was the most consistent with the most optimal number of clusters being 2 clusters.