Prediction of the Stock Market From Linguistic Phrases: A Deep Neural Network Approach

Automation of financial data collection, generation, accumulation, and interpretation for decision making may reduce volatility in the stock market and increase liquidity occasionally. Thus, future markets’ prediction factoring in the sentiment of investors and algorithmic traders is an exciting area for research with deep learning techniques emerging to understand the market and its future direction. The paper develops two FINBERT deep neural network models pre-trained on the financial phrase dataset, the first one to extract sentiment from the NSE market news. The second model is adopted to predict the stock market movement of NSE with the above sentiment, historical stock prices, return on investment, and risk as predictors. The accuracy is compared with RNN and LSTM and baseline machine learning classifiers like naïve bayes and support vector machine (SVM). The accuracy of the FINBERT model is found to out-perform the deep learning algorithms and above baseline machine learning classifiers thus justifying the importance of the FINBERT model in stock market prediction.

• Develop a methodology (Zeng et al.,2018;Zhang et al.,2020;Milecki et al.,2021) to extract market sentiment from a pre-trained FINBERT model (Yang, UY, and Huang,2020) to predict the market prices/index. • To adopt a deep learning model for predicting the stock price movement of NSE NIFTY50 and compare with state-of-the-art baseline techniques like Naïve Bayes and Support Vector Machines to validate its incremental contribution to accuracy in the prediction of the stock price and direction.
There is a need to perform sentiment analysis differently for financial jargon and non-financial jargon in news articles. This would help give a more authentic sentiment score to financial indicators quantified in the text and improve the prediction performance of the stock market for which the paper proposes a FINBERT model pretrained on a financial phrase bank which can perform authentic sentiment extraction from financial news and accurately predict the stock market direction.
The rest of the paper is structured as follows: the literature review is expounded below. This is followed by the methodology adopted in the paper. The results and inferences drawn from the study are illustrated. The implications of the research (both theoretical and managerial) are discussed. Finally, the conclusion and directions for future research are provided. The references are then stated.

LITeRATURe ReVIew
The twin forces of competition and innovation and the drive to make it speedier, cheaper, and better always rule the markets all over the Globe. In Kirilenko and Lo (2013), the authors cited what a great physicist Richard Feynman once said: "Imagine how much harder physics would be if electrons had feelings." Fin-Tech seems to benefit from Moore's law with caution to traders in the high-frequency financial markets, "whatever can go wrong will go wrong faster and bigger when computers are 3 involved." It was estimated as per Tabb (2012) cited in the above article that high-frequency trading (HFT) activity was 60% though in Aldridge and Krawciw (2015), it was stated that HFT might touch 100% trading activity in short term trading volume. According to a World Economic Forum report, 10% of trades are made by humans, 40% invest based on decisions to invest in stock market/index funds/ETFs, and the balance 50% on automatic trading. Thus, trade has become a mix of various instruments' trades on borrowed funds (Pearlstein, 2018).
The rise of algorithmic trading in financial markets owes its development to the factors of growing complexities in financial systems and abrupt changes in regulatory compliance, the breakthrough experiments in financial and statistical modeling using quantitative algorithms, and the rapid changes in technological field-computational, informational and communication sciences, cloud computing. Hendershott, Jones & Menkveld (2011) shows that increase in AT activity can increase liquidity and market quality due to higher trade volumes and speedier delivery mechanisms (Chordia, Roll & Subrahmanyam, 2011). Nevertheless, in AT, a stock's fundamentals or the firm's fortunes are ignored and agnostic to its price levels. Trades on HFT eliminate 'transitory pricing errors' (price changes that reverse after a while) according to Hendershott and Riordan (2012) and enable the "price discovery process' (Brogaard,2010). In an HFT environment, human traders can err due to adverse stock selection (Biais et al., 2010). By increasing order cancellation rates and injecting multiple quotes ('quote stuffing') to confuse the market, some HFT players may create instability Madhavan (2012). Hendershott et al. opine that circuit breakers be used with caution without impacting the market systems' positive role in price discovery and ease of operations. In Subrahmanyam (2013), the authors reviewed the literature on the effectiveness of 'circuit-breakers' during excessive price fluctuations in the market and consequent 'flash crash' to study its impact on financial markets. The regulators impose circuit breakers, and no sooner they are relaxed, the market bounces back, thus proving that the concept of circuit breakers is not evolved to regulate market trade perennially. The preclusion of trade in one market by imposing circuit breaks alerts traders to search for more lucrative markets, impacting the growth cycle. This means coordinated efforts are needed to ensure the order is restored in all markets.
It was documented in Baron et al. (2012) that high-frequency traders earn large persistent returns relative to the risk involved, and their contribution to market liquidity is in the least due to rapidity in order initiation, communication, and execution. However, trading errors can accumulate losses rapidly at the same pace before humans initiate corrective steps. In Europe, for example, a share-by-share volatility safeguard regime exists to safeguard High-frequency trading (HFT) to reduce HFT problems. HFT environment can pay dividends in varying degrees depending on safeguards taken/imposed in systems used, transparency in trading, and regulatory transformation in tune with trading practices. In Gomber and Haferkorn (2015), the paper emphasizes risk management in an HFT environment. They must ensure that the market operation system is robust to withstand speed and trade volume when it reaches the peak needed for the players' algorithms to perform.
In Mukerji et al. (2019), the authors have studied the impact of algorithmic trading in asset markets where human and algorithmic traders conduct trades using technical and fundamental analysis, apart from statistical arbitrage strategies. The objectives of this study include explaining the impact of AT on market quality. The authors have also tested the sensitivity of the findings to variations in the market. They observed that as AT's share in their simulation model reaches 10% of market trades, initially, liquidity increases. Nevertheless, if the share of trades using AT crosses 10%, liquidity increases marginally.
Further, statistical arbitrage appears to guide the trade outcomes without anchoring to fundamentals. Thus, at the estimated current level of AT in NSE crossing 46% of trades, noise trading behavior and its impact appear unavoidable as traders tend to ignore Fundamental strengths of stock and would be gravitated to market sentiment. These findings confirm that AT does not erase market sentiment, though human intervention in the retail trades may reduce due to AT's advent. In such cases, sentiment measures may be aggregated weekly and indexes developed in future studies on AT by factoring in the unincorporated Sentiment. Weller (2018) demonstrates that "AT may reduce price informativeness despite its importance for translating available information into prices". The time lag in information acquisition and incorporating the information acquired at market prices cause market tension.
A microstructure is responsible partly for price discovery and formation. In the following discussion, the research findings of current studies have been highlighted. Tetlock et al. (2008) and several other research work investigated the relevance of information in unstructured data in finance. Whereas Li (2006) analyzed the "tone of qualitative information using objective word counts from corporate annual reports and focused on the predictive ability of qualitative information." Davis et a. (2006) analyzed earnings press releases to examine the effect of qualitative information on earnings, returns. Thus, related studies concentrated on firms' stock returns (Antweiler & Frank, 2004). Das (2007) developed algorithms to quantify human interactions into positive, neutral, or negative internet chats and news ratings. A study by Antweiler (2006) uses an algorithm to identify news by the subject rather than their tone and predict return and reported significant return reversals in the ten days due to the news effect implying overreaction. When information covers negative news on firms' fundamentals, they were found useful predictors of both earnings and returns and that the hard-to-value stocks are prone to Sentiment. They observed that the market under-reacts to negative words immediately after news arrival but adjusts later.

Importance of Social Media in Investor Sentiment Analysis
Information with pessimistic content predicts market activity trends, as shown in studies (De Long, 1990;Campbell, 1993) modeling noise and liquidity traders. A reliable source of pessimism in media leads to lowering market prices, and when pessimism persists, even investors react, evidencing an increase in trade volume. Furthermore, when pessimism peaks impacting prices, its impact is realized for a long in small stocks. Shareholding patterns of a firm are essential to understand a stock's response to media content. Sprenger (2014) shows that the returns before positive news are more visible than negative news stories and confirm that news events' impact on stock returns differs considerably. Hong and Stein (1999) show that the effect of analyst opinions is visible for past losers than for past winners, and this confirms their hypothesis that firm-specific bad news penetrates slowly across retail traders.
In Tetlock (2010), the impact of media news is studied, and it is observed that when media is pessimistic, market prices plummet, and a return to the fundamentals-estimated price. However, extreme pessimism levels lead to high trading volume. These results are in congruence with theories of noise and liquidity-based trading. The findings do not support that media content is a proxy for new information on intrinsic asset values or market volatility. In Tetlock (2010), the authors used data on financial news stories to "test four predictions from an asymmetric information model of a firm's stock price. Accordingly, some traders use private information and transact in the market. Later as the news spreads in the public domain, other investors show a willingness to 'accommodate a persistent liquidity shock. The author measures public information sentiment based on stock returns on "news days in the Dow Jones archive" and finds four return predictability and trade volume patterns consistent with the asymmetric information model predictions. The authors conclude that "some evidence is, moreover, inconsistent with alternative theories in which traders have different interpretations of news for rational or behavioral reasons." However, in a later discussion (Tetlock, 2016), the authors expressed that news has the potential to explain market price variances from time to time, leading to market action, as stock prices adjust readily to commonly known information.
Nevertheless, news sentiment is a proxy indicator of a stock's future performance, and hence, market response to positive and negative news is asymmetric (Tetlock et al., 2008;Barber, 2008;Bhattacharyya et al., 2009;Davis & Tama-Sweet, 2012;Tetlock, 2016;Wamba et al., 2016;DeVault, 2019). Moreover, others have done extensive studies on the news sentiment and effect of media news on stock returns. Heston and Sinha (2017) have measured news sentiment using the Reuters Neural network and found that "daily news predicts stock returns for one or two days" consistent with prior works. "Weekly news predicts returns for one quarter." They assert that much of the unassimilated news gets adjusted around the announcement of subsequent earnings news. Thus, un-incorporated news gets incorporated near the announcement of subsequent earnings news. Uhl (2014Uhl ( , 2015 found that sentiments extracted from Reuter news can explain and predict stock returns variations more accurately than sentiments from macroeconomic indicators. They find that Reuters' negative Sentiment can be better than positive Reuter's Sentiment using vector auto-regression and error correction models. Trading strategies based on Reuter's sentiment showed high success rates and high Sharpe ratios and found that VIX changes are more considerable when negative news emerges. In Forss (2017), the authors developed an algorithm for measuring sentiment-driven network risk to understand connectivity among firms and consequent risk dispersion. For this, they rank firms in networks of co-variance due to sentiment and capture sentiment-linked risk by computing individual firm risks as well as "aggregated network risks." They extracted relative sentiment for firms to develop a measure of individual firm risk and input it into the risk model along with co-occurrences of firms extracted from news quarterly. In this study, the authors have shown different results of which, when the risk model shows the highest quarterly risk value, the probable decline in the firm stock price in 70 days after a quarterly risk measurement is prominent.
In Allen (2019), the authors have used financial news sentiment values, as shown in Thomson Reuters News Analytics (TRNA), to examine the effects of a series constructed by them for daily aggregated market news sentiment values on stock returns of Dow Jones Industrial Average (DJIA). A daily sentiment score was built, and its simple moving average was used to quantify its effects on returns were measured and evaluated two regression models. They concluded that economic news sentiment generated during the trading day significantly influences stock prices and further that "there is a more significant β i, effect on the lower stock returns than on the higher stock returns," which is consistent with the mean sentiment score being negative. Thus the financial news sentiment polarity is to be factored in the traditional asset pricing model and considered relevant in asset pricing. She and Zhang (2019) investigate the importance of sentiment and its impact on the stock market through the social media app 'WeChat'. Further, the impact of certain non-financial news like the spread of "Coronavirus" across various continents as lead articles have also recently shown significant effects on the relationship between sentiment and attraction to headlines. In Huang et al. (2019), a literature survey focusing on the latest developments in textual analysis on China's financial markets is highlighted, showing its distinction from US-based similar studies. The study concludes that there is "limited evidence on the association between sentiment and other contemporaneous or future returns." Krishnamoorthy (2018) opined that dictionary-based methods might not be suitable for extracting sentiment polarity from finance information. The paper is aimed at financial statement analysis by extracting Sentiment polarities using performance indicators. Therefore, the author adopted an "association rule mining-based hierarchical sentiment classifier model" to predict positive, negative, and neutral polarities from the text. A higher associate weightage was attached to words with a finance connotation than non-financial words. This method makes it simpler to explain predictions based on generated rules. The author finds that even domain-specific dictionaries in place of generic dictionaries are inadequate. The author has cited different works in financial sentiment analysis as under: Current works in the area are essentially classified into those who use generic dictionaries, domain-specific dictionaries, and statistical or machine learning models to extract polarity in the text. Some other works utilize financial entities like custom words to bring improved polarity out of the text but with false polarity signals. Antweiler and Frank (2004) states that Internet message postings were used to predict trading signals in their article. Naïve Bayes classifiers were used for extracting sentiment, as shown in Huang et al. (2014). In another case, a Naive Bayes classifier model with "bag-of-words" was adopted to predict the undercurrent in the prospects embedded in the tone of corporate filings (Li, 2010). The authors show that dictionary-based methods are not useful for analyzing corporate information disclosures. Gao and Liu (2020) investigated the relation of intra-day investor sentiment to near-term returns in both the stock market and futures market. It is found that sentiment is a strong predictor of subsequent stock market returns. Balasubramanian (2019) can see a similar finding where the authors capture a complex network machine learning-based model to understand the U.S. stock market dynamics. They show that the clustering of "co-moving" stocks and even breaking existing clusters cause "regime changes" in the market. The authors also present basic ideas on incorporating news and announcements on the stock price dynamics. In Das (2017), similar attempts were made by the authors to model risk based on linguistic analysis of textual information found in corporate emails and news analytics. According to Marty et al. (2020), most of the intraday literature revolved around the Machine learning approach for prediction due to high-frequency news analysis found in such cases. The authors followed Hagenau (2013) and Nassirtoussi (2014) and adopted a methodology consisting of quantifying textual information through 'feature extraction,' 'feature selection,' 'feature representation,' and 'document scoring' and 'classification.' As we propose to use the deep learning approach over traditional methods, we discuss the deep learning method and its edge over other methods with specific reference to the choice of a method among the mix of various approaches discussed in deep learning.

Deep Learning (wang & Siau, 2019)
Deep learning, a subset of machine learning, simulates the neural networks' working by adding multiple layers of feature processing through hidden layers that act on the trained input, build the context, and provide predictions and patterns based on the input. Deep Learning algorithms can be supervised (with known target output) and un-supervised (unknown output). Machine Learning techniques incredibly deep learning help in multi-input processing, i.e., processing of the historical financial data and the textual information in news and disclosures to generate predictions of the stock market movement (Guida, 2017;Coqueret & Guida, 2020;Bullock, 2017;Financial Stability Board, 2020;Kearns & Nevmyvaka, 2013). Similarly, to know factors that place a stock over a benchmark index, a neural network is given a price target and provided market data and raw company details. The artificial neurons with statistical functions demystify the factors to the users. Currently, algorithmic trading strategies involving the integration of traditional data with non-traditional data (e.g., sentiment data from social networks) use R and Python libraries for ultra-high frequency data processing on highsecurity systems. Netezza, Green plum, and Resonance are specific tools used by stock exchanges for their Big Data and analytics needs.
Textual information processing techniques like NLP and Machine learning have entered mainstream research among practitioners in finance. These include a choice of stocks (see Duhigg, 2006), sentiment analysis from news, and prediction of earnings and stock returns (Wong & Thomas, 2004;Tetlock, 2007;Ghosh et al., 2016;Ahmar, 2017), bankruptcy prediction (Atiya, 2001) and corporate diagnosis (Marco & Varetto, 1994). Data scientists can initiate an interactive analysis of structured and unstructured data that is sought in finance applications. Thus, structured and unstructured data is stored in data lakes in a single repository and use rapidly with flexibility in analytics. This system enables users to run Adhoc queries, perform cross-source navigation, and make analytical decisions in real-time.
Developments in mobile technology have enabled trading on interfaces/ terminals like Bloomberg, Reuters, in personal devices (mobiles), making trading more dynamic by using chatbots that perform automated trading based on the market scenario.
Big data analytics (Ram et al., 2016) deals with processing large datasets to demystify hidden patterns, correlations, and other required information on technologies and platforms like No-SQL databases, Hadoop, and Map-Reduce. The works focus on data-mining algorithms to capture, analyze, and visualize big data using these technologies. For extracting value in real-time data streams, IOT/ Machine learning/Deep learning and Artificial Intelligence methods are adopted in some cases. Algorithmic trading and bots for automated trading decision. Such models can be simulated and recalibrated on live market data by generating automated "bots" that run these simulations and make 'buy or sell' decisions, irrespective of whether the traders are active on the stock terminal the accuracy drastically.
The above existing studies highlight the importance of deep learning. Further, the rationale for adopting deep neural networks over traditional approaches like 'Bag-of-Words' approaches is elucidated below:

Rationale for Deep Learning and FINBERT Model
Investors, algorithmic traders, and financial analysts use company disclosures to form an opinion about the firm's prospects before making investment decisions. Cautious investors or fund managers with cognitive knowledge can correctly interpret the content in the disclosures. However, due to complex context-specific language in disclosure texts, the text's real intent needs to be carefully interpreted.
There are traditional techniques like bag-of-words, which only treat each document as a set of individual words and focus on each word's relative frequency without considering the inherent context. This limits the forecasting capability of the stock price movement from financial sources like news or disclosures. As an alternative to the bag-of-words approach, there is a need to adopt advanced techniques and consider the document's inherent context. This is the main advantage of deep neural networks (DNN). The DNN iterates over the document and scrutinizes each word while simultaneously converting it into a lower-dimensional representation of dimension 'n.' This ensures that the document's contextual meaning is preserved and correctly interpreted. Further, the rationale for adopting the specific proposed FINBERT pre-trained model and other deep learning architectures like RNN and LSTM is detailed below:

Rationale for Adopting FINBERT Pre-Trained Model and Specific Deep Learning Architectures
The FINBERT model is adopted due to its inherent architectural capability of performing a differentiated sentiment analysis for financial jargon and non-financial jargon in news articles. The text in news articles is classified and analyzed with a higher weightage to financial jargon. A financial dictionary is used to assign a polarity score to the financial jargon by mapping a particular jargon to the context in financial dictionary which boosts its significance in stock market prediction. This would help give a more authentic sentiment score to financial indicators quantified in the text and improve the prediction performance of the stock market.
The most popular deep neural network architectures for time series data are recurrent neural networks (RNNs) and the long short-term memory (LSTM) model (Kraus & Feuerriegel, 2017).
The RNN adopts a sequential approach to analyzing the document features, which helps build the context word to word. However, the RNN is used to a limited extent in real scenarios due to its vanishing gradient problem. This drawback is addressed by implementing LSTM architecture. The LSTM architecture has an in-built 'forget fate' to prevent deep gradients, and this "memory"-like feature extends its capability to analyze long-term contexts and is thus, adopted in our study on stock market prediction.
Existing studies ) which adopted CNN for stock market prediction in addition to RNN and LSTM are now updated in the paper. However, the same study confirmed that RNN and LSTM outperformed CNN in terms of stock market prediction accuracy.
The reason for this finding is that RNN adopts a sequential approach to analyzing the document features, which helps build the context word to word. This sequential approach is more appropriate for a stock market context where current prices are also driven by historical stock prices. Further, LSTM also has an in-built 'forget fate' to prevent deep gradients, and this "memory"-like feature extends its capability to analyze long-term contexts like stock market.
However, the CNN architecture is more appropriate for image recognition and classification problems (Qianwen et al., 2021) which makes it less effective in a stock market prediction context and hence not adopted in the paper.

Limitations in existing Studies
First, the existing studies, like Uhl (2015), have investigated the impact of market sentiment news on stock price independently but not in combination with historical stock prices. Historical stock prices can help estimate the price trend but, by itself, is not sufficient to accurately forecast the future price trend. There is also a need to incorporate investor sentiment extracted from market news to understand the future price movement's relative movement. This is because stock prices are not governed by a single factor but by different forces acting from different directions. The net impact of these different market forces determines the accurate stock price movement. Thus, market news sentiment incorporated with historical prices helps provide a better idea of the future direction of stock price movement and boosts the predictive accuracy in determining the correct trend.
Second, though Wolfe et al. (1994) used artificial neural networks for stock market forecasting in the Indian stock market, deep neural networks and the comparison of their predictive power viz-viz machine learning techniques were not performed in the Indian stock market context. The predictive and decision-making power of deep neural networks from historical data and oscillator history has not been established.
Thirdly, there is a need to perform sentiment analysis differently for financial jargon and nonfinancial jargon in news articles. This would help give a more authentic sentiment score to financial indicators quantified in the text and improve the prediction performance of the stock market. Performing sentiment analysis uniformly for all the words may not capture the words' true meaning, which may not help provide an accurate stock market analysis.
Lastly, the variable importance has not been computed across machine learning models in all the above existing studies (Madge & Bhatt, 2015) to determine market news's significance for stock market movement. Figure 1 illustrates the limitations in existing studies. For overcoming the above limitations, a stock price prediction model using machine learning techniques trained on historical data and technical indicators is proposed in Figure 1 to compare the predictive power of state-of-the-art machine learning techniques and variable importance.

DATA CoLLeCTIoN AND ReSeARCH MeTHoDoLoGy
The disclosure information comprises quantitative data (evidencing corporate action) and textual information (financial news). Thus, the user may need to perform cross-sectional verification of quantitative and textual averments for extracting embedded sentiment. The data sources and structure of the dataset and the studies undertaken in the paper are discussed below.

Data Sources and Implementation of the Studies
The first stage is to collect the relevant data for the specified period-over ten years from 2010 to 2020 at a monthly interval. The data collection is two-fold: First, the "structured" stock price monthly values for India's National Stock Exchange (NSE-NIFTY50 Index) from 2010 to 2020 were collected from Bloomberg.
Secondly, 1,00,000 financial market news articles were collected from the Emerging Markets Information System (EMIS) database from 2011 to 2020 by specifying the "NSE" keyword as the search term. It is ensured that duplicate news (same news with different timestamp) and news of small market capitalization firms listed on NSE are eliminated for a more accurate prediction result. For the analysis, no social media channels have been adopted since the FINBERT model is trained on financial news corpus articles and there is no mechanism of performing sentiment analysis of financial terms differently from non-financial jargon.
The dataset is split into training and a test set with an 80-20% split ratio. This splits the dataset into 80000 news for training and 20000 for testing.
The paper preferred to adopt a deep learning approach, as shown below. Figure 2 illustrates the proposed deep learning framework. A pre-trained FINBERT model is adopted with finetuning of hyperparameters on the integrated dataset of sentiment extracted from NIFTY50 market news (using another pre-trained FINBERT sentiment classification model illustrated below in Figure 3) in combination with the historical stock prices of the companies, return on investment and risk (beta) to predict the stock price direction of the companies(+1,-1 and 0 representing the upward, neutral and downward direction of stock prices) as depicted above in Figure 2 (Kraus & Feuerriegel, 2017). The same features are also fed to already established deep learning algorithms Recurrent Neural Networks (RNN) and Long Short-Term Neural Networks (LSTM) to compare the predictive accuracy of stock market direction. The accuracy is also compared with baseline machine learning models like Naïve Bayes and Support Vector Machine.

Proposed Model
The pre-trained financial sentiment analysis model FINBERT 1 is used as a base architecture for the deep neural network models inspired by the transfer learning conceptualization manifested in Kraus and Ferriguel (2017) in order to reduce the algorithm time complexity and to provide a base architecture that can be tuned for better accuracy since building a machine learning model from scratch is both space and time complex. This model is trained on the benchmark financial phrase dataset defined in Malo (2014) and reused in Krishnamoorthy (2018) and is initially used for feature extraction and sentiment analysis and later for the stock market direction prediction. Figure 3 illustrates the news sentiment extraction process through pre-trained FINBERT model neural network architecture.
Initially, the Financial Phrase bank benchmark dataset is taken as input to the FINBERT model. The FINBERT-large architecture consists of 24 encoder layers, hidden size of 1024, 16 multi-head attention heads, and 340M parameters trained on the above Financial Phrase bank dataset to compute a sentiment score from the corpus.
The FIN-BERT model performs language modelling and sentiment classification with that set of Transformer encoders stacked on each other. The language modeling algorithm differs from RNN and LSTM. Instead of predicting the next word given previous ones, BERT "masks" a randomly selected 15% of all tokens.
With a SoftMax activation function layer over vocabulary on top of the last encoder layer, the masked tokens are predicted from the phrase bank input. A second task BERT is trained on is "next sentence prediction". Given two sentences, the model predicts whether or not these two follow each other.
The SoftMax function is used as the activation function in the output layer of neural network models since it is most suited for multi-class classification problems where class membership is required on more than two class labels. In this scenario, since sentiment polarity belongs to 5 different grades (positive, very positive, negative, very negative, neutral), this is appropriate.
The input sequence (of financial phrases) is represented with token and position embeddings. Two tokens denoted by [CLS] and [SEP] (word separator token) are added to the beginning and end of the sequence, respectively. For all classification tasks, including the next sentence prediction, [CLS] token is used. Further, the pre-trained model would now be leveraged through transfer learning on any financial news corpus.
The input to the FINBERT sentiment classification neural network model is the market news from the EMIS database (1,00,000 financial market news articles collected from the Emerging Markets Information System (EMIS) database from 2011 to 2020). The data is split into training and testing set. The training set data is then trained on the pre-trained FINBERT model (left half of the figure) and the hyperparameters are tuned on the final sentiment prediction layer with the first 23 encoder layers (24 encoder layers in total) directly reused for reducing the algorithm running time. The sentiment polarity of the news (output layer of first FINBERT sentiment classification neural network) is generated which is then adopted as one of the predictors of stock prices to another FINBERT model and to deep learning models (LSTM and RNN) and baseline machine learning models.
For the deep learning models (LSTM and RNN), hyper-parameters were optimized using a Grid search algorithm with cross-validation of 5 folds applied on the training set. This is done since a model trained on financial and linguistic phrases can assign a more reliable sentiment polarity to the text of the market news statements due to assigning a higher weightage to words found in the financial phrase dataset than non-financial indicators. The sentiment attached to the financial jargon found in the market news can be better quantified and enhance the model's ability to accurately predict the stock market direction.
The FINBERT model is also compared with the deep learning algorithms, the RNN and LSTM architectures' parameters are further optimized by Adam optimizer, and the learning rate is varied in steps of 0.001 from a minimum of 0.0005 to a maximum of 0.02. Further, the Nesterov (NADAM) optimizer is also used for the same learning rate range (0.0005-0.02) to finetune the parameters.
Based on the collected data and the aggregated dataset, at the aggregate market level, the market news of the National Stock Exchange (NSE), historical market prices of NSE NIFTY 50 from 2010 to 2020, return on investment and risk (quantified by beta) for the same period are also taken as input to the pretrained FINBERT model, both the machine learning baseline models (Naïve Bayes and Support Vector Machine) and the deep learning models RNN and LSTM to compute the feature importance and to predict the direction of the stock market, which is the output of the stock price prediction FINBERT neural network. The results are enclosed below.

ReSULT AND DISCUSSIoN
This section compares the predictive performance of machine learning baseline techniques like Naïve Bayes and Support Vector Machines with the FINBERT pretrained model and other established deep learning architectures (RNN and LSTM) on financial market news and fully connected neural networks. Other deep learning techniques like Bi-directional variants of RNN and LSTM (Bi-RNN, Bi-LSTM) and more advanced RNNs like Gated Recurrent Network (GRU) and Bi-directional GRU are also compared.
Further, a sensitivity analysis is performed by varying the learning rates of the algorithms. The accuracy is compared across these different learning rates to determine the deep learning algorithms' predictive behavior vis-à-vis the baseline machine learning approaches.
The classification algorithms were implemented in Python using the 'sci-kit-learn' library predefined for implementing machine learning algorithms in Python, while 'Tensor-Flow' with 'Theano' as backend was implemented for the deep learning simulation. The resulting neural networks, which were already implemented by the Prosus AI team, were reused for implementation purposes since the running time of these algorithms is more than 30 hours and is variable based on the hardware requirements of the computing platform, and this would reduce the running time to order of 10 minutes considering the hardware limitations.
The accuracy of stock price direction classification is compared with baseline machine learning classifiers like Support Vector Machine, Naïve Bayes (baseline stock prediction models) to examine whether the FINBERT pretrained model is a better predictor than traditional machine learning techniques. The training set, testing set, feature importance, and balanced accuracy measures have been reported below in Tables 2-3. Balanced accuracy is reported to account for the unbalanced cases (cases that are not skewed to the majority class) in the dataset. Table 2 compares the stock market direction predictive accuracy of proposed FINBERT model Deep Learning with RNN, LSTM, Bi-RNN, Bi-LSTM, GRU, Bi-GRU and baseline machine learning techniques like Naïve Bayes and Support Vector Machine (SVM). FINBERT model is found to outperform the other techniques with a balanced accuracy of 83%. This indicates that from an unbalanced dataset of positive records (those which classify the stock market direction as +1) and negative records (classifying direction as -1), the number of true positive instances out of total classified instances (say,100) is 83. Figure 4 compares the methods (machine learning) and deep learning in terms of predictive accuracy and is illustrated below: The pre-trained FINBERT is a better predictor of the NIFTY market movement (85% balanced accuracy) followed by Deep Learning with LSTM (73%), Bi-directional GRU (72%), GRU (70%), and Deep Learning with RNN (70%). These techniques outperform the Bi-directional RNN, Bi-directional LSTM, fully connected neural networks and baseline machine learning classifiers. Figure 5 plots and compares the running time (time complexity) of the algorithms implemented. The baseline machine learning classifiers (Naïve Bayes and Support Vector Machines) have a running time of 5 minutes, while the deep neural networks i.e., Gated Recurrent Neural Network (GRU), LSTM and RNN have running times of 250 minutes,120 minutes and 150 minutes, respectively.
However, the pretrained FINBERT has a minimal time complexity of 10 minutes, implying that using a pre-trained neural network saves the implementation time of complex deep learning algorithms to a great extent. Figure 6 illustrates the variable importance computed by the machine learning and deep learning classifiers.
It is observed from Figure 6 that sentiment extracted from market news is considered more significant by the FINBERT and deep neural network models, while baseline techniques consider historical stock prices with a higher weightage. This is because deep neural network models are more effective in extracting the context and significance of the linguistic content in market news, and it is the market sentiment that predicts the future market path more effectively in combination with historical stock prices. This indicates that company managers must keep track of market sentiment news and announcements to analyze its impact. Further, return on investment is also a significant factor for investors and managers in stock market performance evaluation since investors proceed to invest in a stock if it provides more returns at a future date. Further, risk on investment is also important factor since investors desire to opt for a stock with minimum risk and guaranteed returns. The results are validated for the top three and bottom two performing algorithms (FINBERT, Deep Learning with Long term Short Neural network, Deep Learning with Recurrent Neural Networks in top three and least performing being Support Vector Machine and Naïve Bayes Baseline) by a sensitivity analysis performed by varying the learning rates from 0.001 to 0.009 in steps of 0.002. Figure 7 plots the balanced accuracy of the classifiers for each learning rate below.

Comparison of methods for prediction of NIFTY market movement
The FINBERT, Deep Learning with LSTM, and Deep Learning with RNN algorithms perform optimally at a learning rate of 0.009 with an accuracy of 85%, 80%, and 75%, respectively, while Naïve Bayes is optimal at a learning rate of 0.001 and SVM at a learning rate of 0.007. Figure 8 illustrates the variation of accuracy with optimizer. NADAM optimizer is more effective in achieving higher accuracy than ADAM optimizer for all the algorithms. The results show that of the various methods adopted to capture sentiment from corporate disclosures, the FINBERT model could explain upto 85% of the NISE NIFTY50 market movement while the Deep Learning methods (Recurrent Neural Networks and Long Short-Term Neural Networks) could explain 70% and 73%, respectively, overall NSE NIFTY50 market. The remaining unexplained component may be attributed to the fact that there is an asymmetry in the market speed of information feeds and information absorption. This information asymmetry means that all the sentiment extracted from market news is not reflected in the market direction as the market adjustment is influenced by market adjustment to information. The market is selective in its information assimilation, and its speed of absorption irrespective of the speed of flow of information feeds, leaving unincorporated information for the market's future direction. This unincorporated information is a valuable input for predicting probable future market path.

Theoretical Implications
The study investigates the impact of financial market news on the stock market movement. This is accomplished by incorporating both the news text and historical data as features in baseline machine learning models (Naïve Bayes and Support Vector Machine) and deep learning models (Recurrent Neural Network and Long Short-Term Neural Network). The model classifies the stock market movement (up, down, or no movement), and it is observed that the deep learning model, particularly the Long Short-Term Neural network model, outperforms the deep learning model of Recurrent Neural Network and the baseline machine learning models for the market level study. This theoretically establishes the effectiveness of deep learning architectures in finance for stock market direction prediction.

Practical Implications for Investors, Limitations, and Future Directions for Research
The paper highlights the significance of market news to estimate the future stock price movement. The study assumes significance as it attempts to construct specific sentiment measures from disclosures at the aggregate market level using advanced deep learning algorithms like FINBERT, RNN and LSTM. The FINBERT model is found to be an effective method which can be leveraged and recalibrated in different stock market contexts to perform authentic sentiment analysis of the financial content words which would help in more accurate stock market prediction.
However, one of the limitations of adopting the pre-trained FINBERT model is that finetuning the parameters is highly computationally intensive. Further, the neutral statements are not effectively interpreted by this model.
There is scope to extend this research by leveraging this theoretical contribution in practical industry scenarios by simulating and recalibrating the deep learning model on other countries' live market data. Thus, by generating automated "bots" that run these simulations and makes 'buy or sell' decisions, irrespective of whether the traders are active on the stock terminal, the study can contribute to the realistic corporate situation and would be a useful tool for companies to restore efficiency in the market.
Future researchers can also exploit the advantages of transfer learning (a process of applying the same deep learning algorithm on different stock market scenarios even outside the Indian stock market context) and generate insights for each of these scenarios in the direction of future research.

CoNCLUSIoN
The paper proposes a deep neural network approach to predict the future direction of NSE NIFTY prices. Ten years of historical NIFTY50 data with market news were factored in to train the model for the same period. The pretrained FINBERT model followed by the deep neural network with a long short-term neural network approach were found to outperform the deep learning with the Recurrent Neural Network approach and the baseline machine learning classifiers. The variable importance computed also validated that market news, combined with historical indicators, predicts the stock market trend closer to the actual trend.
The paper thus establishes the supremacy of deep learning over traditional machine learning techniques in the finance domain.

CoNFLICT oF INTeReST
The authors of this publication declare there is no conflict of interest.

FUNDING AGeNCy
This research received no specific grant from any funding agency in the public, commercial, or notfor-profit sectors.