Markov chain and its use in solving real world problems

Markov chain is one of the most important tests in order to deal with independent trials processes. There are two major principal theorems for these processes. The first one is the ‘Law of Large Numbers’ and the second one is the ‘Central Limit Theorem’. When probability experiments create independent trial processes, the outcomes for all iterations are same. Also, they increase or decrease with the same probability. Markov chain is the process where the outcome of a given experiment can affect the outcome of the future experiments. It consists of finite number of states and some known probabilities, where probability of changing from state j to state i. For instance, there are two sectors; government and private. The probability here is the likelihood of moving from state j to state i in a given time for an individual.

Below are some example situations showing the application of Markov chain.

Markov chain application example 1

RA Howard explained Markov chain with the example of a frog in a pond jumping from lily pad to lily pad with the relative transition probabilities. Lily pads in the pond represent the finite states in the Markov chain and the probability is the odds of frog changing the lily pads.

Markov chain application example 2

Manager in a company state their intention to get a promotion by the end of next month to her employee P. And P forwards the same message to Q, who further gives the news to R and so on. In such scenario there is a probability x that a person will replace the answer from yes to no while conveying the message to the next person and a probability y that the person will change the answer from no to yes.

Markov chain application example 3

Minister X has 55% votes and Minister Y has the rest 45% votes in one state. However, probability of current followers voting Minister X staying with Minister X in future is 70% and switching to Minister Y is 30%. And probability of the present followers voting Minister Y staying with Minister Y is 90% and switching to Minister X is 10%. Even though Minister X has more votes than Y in the current situation, but in the next election Minister Y will have more votes with 57% and X will be left with only 43% votes.

Simple business case for application of Markov chain

The case supposes that there are 2 types of weathers in a particular area, ‘sunny’ and ‘cloudy’. A news channel wants to broadcast their prediction about the next week’s weather.

The news channel hires a weather forecast company to find out the next week’s weather and also for the following weeks. Currently there is ‘sunny’ weather in that area.

  • Probability of the weather staying ‘sunny’ the following week is 80%.
  • Probability of the weather changing from ‘sunny’ to ‘cloudy’ over a week is 20%
  • Probability of the weather staying ‘cloudy’ the following week is 70%
  • Probability of the weather changing from ‘cloudy’ to ‘sunny’ over a week is 30%

Although the weather is predicted to be ‘sunny’ the whole week, one cannot be sure about the next week without making some transition calculations. The below matrix is to explain the transition:

Transition matrix
Transition matrix

Transition diagram

Transition diagram
Transition diagram

The diagram simply shows the transitions and the weather conditions same as the above matrix. The calculations can be done by the following matrix multiplication:

Current State * Transition Matrix = Final State

S=Sunny; C= Cloudy

It can be seen that there is 80% chance that the next whole week will also be ‘sunny’. But, there is 20% chance that the weather might change to ‘cloudy’ next week. This calculation is called Markov chain. If the transition matrix doesn’t change with time, one can also predict weather for the further weeks using the same equation. Calculations for the weather forecast in the two weeks’ time:

Mathematical steady state calculations

Therefore, to predict the weather for further days/weeks/months in the long run, steady state vector helps in calculating the likelihood of the weather staying ‘sunny’ or changing to ‘cloudy’ without depending on the given initial condition of weather.

Mathematical steady state calculations
Mathematical steady state calculations
Equation 1: 0.2x1 + 0.7x2 = 0 = 0.2x1 - 0.7x2
Equation 2: x1 + x2 = 1 (Probability vector)

Therefore, from Equation 1 and 2, about 78% of the weeks in the future will be ‘sunny’, while the rest will be ‘cloudy’.

Probable areas of application of Markov chain

Markov chain has many applications in the field of real world process are followings:-

  • The one of the most popular use of Markov chain is in determining page rank by Google.
  • Markov chain-based methods also used to efficiently compute integrals of high-dimensional functions.
  • This method plays an important role to allow sample from any arbitrary probability distribution.
  • It helps determine the probability of consumer switching from one brand to another.
  • Markov chains are popular in finance and economics to model different phenomena, including market crashes and asset prices.

A number of software can be used for Markov chain analysis, such as Ram Commander, SoHaR Reliability and safety, Markov Analysis software and MARCA (Markov Chain Analyzer).

How to perform and apply Monte Carlo simulation?How to perform bootstrap and jackknife analysis?

Prateek Sharma

Analyst at Project Guru
Prateek has completed his graduation in commerce with a rich experience in Telecom, Marketing and Banking domains for preparing comprehensive documents and reports while managing internal and external data analysis. He is an adaptable business-minded Data Analyst at Project Guru skilled in recording, interpreting and analysing data with a demonstrated ability to deliver valuable insights via data analytics and advanced data-driven methods. Apart from his strong passion towards data science, he finds extreme sports interesting. He keeps himself updated with the latest tech and always love to learn more about latest gadgets and technology.
Prateek Sharma