Adaptive sentiment analysis for stock markets using deep learning

Abstract

Stock markets are highly volatile, making price prediction very difficult. One of the factors influencing the volatility of financial markets is rapidly changing news sentiment. This study presents a novel adaptive deep learning (DL) framework for sentiment analysis with concept drift capabilities. The proposed model combines convolutional neural networks (CNN), bidirectional long short-term memory (BiLSTM), and attention mechanisms in its processing architecture. The model inputs preprocessed news headlines into both the CNN and BiLSTM-Attention networks to extract local features, model contextual dependencies, and prioritizes important sentiment cues in its prediction mechanism. We use FastText and Word2Vec for word embeddings, while incremental learning is used to manage concept drift. One key advantage of handling concept drift is that the model can continuously learn new patterns in data streams without needing to fully retrain the model. The model is validated on a curated dataset from various sources with superior performance across all metrics, like accuracy (0.9753) and an F1-score (0.98). It significantly outperforms benchmarks like distilled bidirectional encoder representations from transformers (DistilBERT), LSTM, and valence aware dictionary and sentiment reasoner (VADER). A run of ten iterations validated that the real-time pipeline did not exceed 200 ms in processing and classifying headlines. This signifies the practical viability of our model in fintech applications such as algorithmic trading and risk management.