Analysis on the Implementation of an Indonesian Automatic Short Answer Grading System

Abstract

In the education field, many types of questions have been developed to measure the students understanding for the material that has been given, such as multiple choice, short answer, essay, and others. Assessment for essay-type questions often takes up a lot of assessors’ time. A solution to overcome this problem is the development of an automatic essay assessment system. This system is developed in various literature on Automatic Essay Scoring and Automatic Short Answer Grading. This study uses a Multiclass Support Vector Machine (SVM) model in building an automatic assessment system. There are several findings from the results of this research. For the dataset used in this research, a combination of unigram and bigram cosine similarity, type-token ratio, and word count ratio features implemented together with RBF kernel with γ = 100 produces the highest precision value at the validation stage. At the evaluation stage, with a precision metric value of 0.49 and RMSE of 2.77, this model is considered less accurate. This is because the KNN and Logistic Regression models have higher evaluation metric values. The Logistic Regression model is more recommended for this automatic short answer grading system, because this model can provide more balanced and accurate predictions based on the lowest RMSE value and the precision, recall, and f1-score values that tend to be stable.