Effectiveness Analysis of Multimodal Feature Fusion in Herbal Leaf Classification

Abstract

This study aims to evaluate the performance of leaf image classification models based on feature fusion strategies that integrate shape, texture, and semantic representations. Three feature extraction methods were employed: Histogram of Oriented Gradients (HOG) for shape, Gabor Filter for texture, and Convolutional Neural Network (CNN) using MobileNetV2 for semantic features. Each feature type was tested using three classification algorithms: Support Vector Machine (SVM), K-Nearest Neighbors (KNN), and Random Forest (RF). Experimental results show that CNN features consistently outperformed the others, achieving the highest accuracy and F1-score, with a peak accuracy of 91.0% using CNN+SVM. In contrast, HOG and Gabor features resulted in significantly lower performance. Feature fusion—such as HOG+CNN and HOG+Gabor+CNN—did not improve performance and instead caused a notable decline, primarily due to the high dimensionality of HOG features, leading to the curse of dimensionality. Confusion matrix and ROC curve analyses confirmed that the CNN-based model achieved high inter-class separability, while models with fused features produced near-random predictions in several classes. These findings suggest that feature fusion does not inherently lead to better classification performance, particularly when dimensional imbalance is not addressed. The study recommends the use of single semantic features extracted from CNN for efficient and accurate leaf image classification, while also encouraging future research into adaptive fusion strategies such as feature weighting or multimodal integration.