Testing the accuracy of the stock market forecasting model

By Riya Jain & Priya Chetty on August 5, 2023

The stock market is an uncertain market consisting of opportunities to gain and lose. The risk presence in the market creates the need for a stock market forecasting model for the movements of stocks and an understanding of the possible position of stocks in position. Information on future stock movements can enable the investor to take profitable decisions (Patil & Bagodi, 2021).

The previous article, considering the stock market’s uncertain nature; is focused on building a forecasting model. Such a model shall help to understand the stock movements based on a time series dataset from 2000 to 2020. Using the ARIMA model, the most optimal model was identified, and based on it predictions were made for the next 5 years i.e. from 1st April 2020 to 31st March 2026.

However, before putting the stock market forecasting model into decision-making, there is a dire need to know the accuracy of the model (Guo, 2022). This article focuses on the comparison of the predicted data based on the model for the first 3 years i.e. from 1st April 2020 to 31st March 2023, with the actual Bombay Stock Exchange’s data for the same period.

The examination of model accuracy would be based on 2 rules (Kompella & Chilukuri, 2019) i.e.:

  • The value of the Mean square error (MSE), and Root mean square error (RMSE) should be closer to 0 as the indicators are used for recording the deviation of the actual value from the predicted values.
  • The paired t-test null hypothesis of having no difference in the actual and predicted value should not be rejected.

The hypothesis considered for the testing is:

H0: There is no difference in the average return for the predicted and observed values.

HA: There is the presence of a significant difference between predicted and observed average return values.

The stated hypothesis will be tested at a 5% level of significance (Artman & Artman, 2014; Mindrila & Balentyne, 2013). The model that fulfils both conditions would be accepted as the accurate model and could be used further by the investors for predicting stock market performances.

Accuracy check for the growth stocks

The average return model in the case of growth stocks is formulated using the (3,2,1) model of ARIMA for forecasts. The model enabled the computation of daily average return from 1st April 2020 to 31st March 2025. As currently the actual data is until 2023, thus; data from 1st April 2020 to 31st March 2023 is taken to have a comparison of 3 years’ performance of growth stocks.

For this two indicators are selected i.e. MSE and RMSE. MSE is the square of the average difference between the actual and predicted value for the given dataset. The measure is used to measure the variance of the residuals (Varshini et al., 2021). However, RMSE defined MSE square root by stating in residual standard deviation (Hodson, 2022). The results of both measures are shown below.

MSE for an average returnRMSE
0.00010.0085
Table 1: Accuracy Indicators for growth stocks

The above table shows that the value of MSE for the average return daily data is 0.0001. This is close to 1 and the RMSE value is 0.0085. The value for each of the indicators is close to 0. This shows that there is a very slight difference between the values predicted and the actual observed value of the average return.

These accuracy indicator values verify that the model is effective and could be used for forecasting the average returns for the growth stocks. The results are further validated by hypothesis testing.

VariableMeanStd ErrorStd devt-statp-value
Actual-0.00210.00030.0092-0.37000.7114
Predicted-0.00200.00020.0044  
Table 2: T-test results for growth stocks forecasting model

The above table represents that the mean value of the actual average return of growth stocks is -0.0021, while of predicted is -0.0020; which closes with a difference of -0.0001. Even, the standard error values are close and the presence of deviation recorded in the predicted values is less than the variation of the actual value.

The statistical examination of the difference in actual and observed values is done using a t-stat. The p-value for the test is 0.7114. This is more than the significance level of 0.05. The null hypothesis of having no difference in the actual and predicted average return values of the growth stocks is not rejected. Hence, there is no difference in the predicted and actual values and the model is accurate in predicting the average return for growth stocks.

Accuracy of the stock market forecasting model for income stocks

The income stock model is developed using (3,0,3) ARIMA form wherein; based on the stated values, the forecasts for the average return values have been made. The comparison needs to be made for checking the accuracy of the model and as the data is available only till 2023, thus, the data is considered only till 31st March 2023. For making the comparison value of MSE and RMSE is computed and the results of the measures are presented below.

MSE for an average returnRMSE
0.00010.0099
Table 3: Accuracy check for income stocks

The above table 3, shows that the value for both indicators is close to 0; the MSE value is 0.0001 and RMSE is 0.0099. Here the measure values are within the required range of an accurate model. Thus, the model developed for forecasting the average return in the case of income stocks is appropriate and could be used for predictions.

Though the accuracy measures help in providing an overview of the data, statistical testing also is essential to know if there exists any difference between the values. The results of the paired t-test for comparison are shown below

VariableMeanStd ErrorStd devt-statp-value
Actual-0.00230.00040.0118-1.54270.1231
Predicted-0.00170.00010.0018
Table 4: Paired t-test for income stocks accuracy check

The above table 4 reveals that the value of the mean is -0.0023, while the predicted one is -0.0017. The difference between both the values is -0.0005; revealing that there is a small difference between both values.

The standard error in both cases is different with a difference of 0.0003. Even, the standard deviation of actual and predicted ones are different as compared to the actual data which has more variation. While predicted values have less variation in their dataset.

To understand the difference, a t-test for the data is done and the results reveal that the p-value is 0.1231 > 0.05. The null hypothesis of having no difference in the values of actual and predicted average return for the income stocks is not rejected. As the hypothesis is not rejected, this shows the presence of no difference in the values making the predicted value close to the actual one. Hence, the stock market forecasting model to predict income stocks’ average return, is accurate.

Testing the accuracy of the value stock forecasting model

The value stock forecasting model is developed using ARIMA with values of (5,0,12). The forecasted average return values depict the 5 years performance of the selected value stocks. But, as comparison can only be done for the data available till 2023; thus, the 3-year-based assessment has been done using MSE and RMSE measures. The results of RMSE and MSE measures are shown below.

MSE for an average returnRMSE
0.00010.0117
Table 5: Accuracy indicators for the value stocks model

Each of the mentioned indicator values from the above table is close to 0. This shows that the required criteria of having less of a difference between the actual and predicted values are fulfilled.

A value close to 0 represents the existence of less difference and hence the model is accurate. These findings are further statistically verified by using paired t-tests. The results from the tests are given below.

VariableMeanStd ErrorStd devt-statp-value
Actual-0.00230.00050.01390.94110.3468
Predicted-0.00270.00000.0016  
Table 6: Paired t-test for the value stocks prediction model accuracy

The above table shows that the mean value of the predicted data is -0.0027 while of actual one is -0.0023. The values are different with a difference of -0.0004. The existence of close values represents that there is less variation in the predicted and observed values.

However, the presence of standard error can be observed in the actual data. Furthermore, the standard error for the predicted values is 0.0000. The variation in the average return values daily has been more in actual data as compared to the predicted values.

The mean, standard error, and deviation present an overview of the data. But for statistical comparison of both data, the t-test is performed. The p-value of 0.3468 is more than the required significance level of 0.05.

Furthermore, this shows that the null hypothesis of having no difference in the average return of predicted and actual value for value stocks is present. Hence, the model is effective in measuring the average return for value stocks and the predictions could be made for the future to make effective investment decisions.

The stock market forecasting model developed using ARIMA is accurate

In the stock market, the optimisation of the forecasting model is essential to make accurate forecasts and take profitable investment decisions. This article overcame this hindrance by having the comparison of the actual and predicted values using different accuracy measures and statistical hypothesis testing. As the value of the MSE and RMSE were close to 0 for all stocks, the models are accurate.

Furthermore, to support this conclusion the hypothesis was not rejected. This shows that there is no difference in the actual and predicted values. Hence, there is no variation in the prediction and the actual values and the investors can consider the defined model for predictions.

References

  • Artman, M., & Artman, M. (2014). An empirical study for testing the stock market efficiency and identifying abnormal return opportunities.
  • Guo, Y. (2022). Stock Price Prediction Using Machine Learning. In Södertörn University | School of Social ScienceMaster.
  • Kompella, S., & Chilukuri, K. C. (2019). Stock market prediction using Machine learning methods.
  • International Journal of Computer Engineering & Technology (IJCET), 10(3), 20–30.
  • Mindrila, D., & Balentyne, P. (2013). Tests of Significance. In The Basic Practice of Statistics.
  • Patil, S., & Bagodi, V. (2021). A study of factors affecting investment decisions in India: The KANO way.
  • Asia Pacific Management Review, 26(4), 197–214. https://doi.org/10.1016/j.apmrv.2021.02.004

I am a management graduate with specialisation in Marketing and Finance. I have over 12 years' experience in research and analysis. This includes fundamental and applied research in the domains of management and social sciences. I am well versed with academic research principles. Over the years i have developed a mastery in different types of data analysis on different applications like SPSS, Amos, and NVIVO. My expertise lies in inferring the findings and creating actionable strategies based on them. 

Over the past decade I have also built a profile as a researcher on Project Guru's Knowledge Tank division. I have penned over 200 articles that have earned me 400+ citations so far. My Google Scholar profile can be accessed here

I now consult university faculty through Faculty Development Programs (FDPs) on the latest developments in the field of research. I also guide individual researchers on how they can commercialise their inventions or research findings. Other developments im actively involved in at Project Guru include strengthening the "Publish" division as a bridge between industry and academia by bringing together experienced research persons, learners, and practitioners to collaboratively work on a common goal. 

 

I am a master's in Economics from Amity University. Having a keen interest in Econometrics and data analysis, I was a part of the Innovation Project of Daulat Ram College, Delhi University. My core expertise and interest are in environment-related issues. Apart from academics, I love music and exploring new places.

Discuss