Introduction to Sentiment Analysis in natural language processing (NLP)

The evolution of human-computer interaction began in the 1940s during World War 2 when the military primarily used it for scientific calculations to decode enemy messages. Furthermore, after 2010 AI assistants like Siri, Google Assistant and Alexa introduced voice-based interaction that soon became prominent, allowing users to control devices with speech (Wikipedia contributors, 2025). “Sentiment” refers to feelings or emotions; “analysis” means examining something in detail. Sentiment analysis helps detect emotions like happiness, anger, or sadness in a phrase or text. This makes AI-driven interfaces like chatbots more conversational and intuitive.

Sentiment analysis is part of natural language processing (NLP), which helps computers understand human language (Jin et al., 2023). Based on its granularity and analytical focus, sentiment analysis can be classified into different types (Gupta et al., 2024):

• Document-Level Sentiment Analysis
• Sentence-Level Sentiment Analysis
• Aspect-Based Sentiment Analysis

In recent years, internet users have frequently shared their opinions online, and research (Wattenberg & Fisher, 2004) shows that online reviews (Aspect-based) significantly influence consumer decisions. Businesses must analyze these reviews to gain insights into customer behaviour and refine marketing strategies. This raises the questions:

How can businesses understand customer opinions from vast, informal, unstructured online text?

The sheer volume of such data poses a challenge, but text and sentiment analysis, combined with information visualization, offer effective solutions. Several organisations use sentiment analysis based on online reviews and marketplace feedback to see how people feel about their products.

So, sentiment analysis is an explicit application of NLP where the goal is to extract subjective information from several pieces of text. The process also involves algorithms that classify the text into several categories or parameters to detect specific emotions or the intensity of the emotions. The algorithms use machine learning models trained on labelled data to detect the sentiment of a text or phrase to classify it as positive, neutral or negative (Liu, 2015).

Importance of Sentiment Analysis in real-world applications

Sentiments and emotions are the core elements of human communication. They go beyond words, adding depth, context, and meaning to interactions. People connect better when emotions are recognized and reciprocated. Recognizing emotional cues also helps in resolving misunderstandings.

There is a vast possibility of applying sentiment analysis in the real world. For a product manufacturer, analyzing consumer opinions on its products and competitors’ offerings is essential for identifying strengths and weaknesses for marketing intelligence and product benchmarking (Liu et al., 2005). Beyond conventional analysis of product reviews and customer feedback, sentiment analysis can also be leveraged to solve social problems like:

• Fake news and misinformation detection: A sentiment-based credibility scoring system can analyze news articles, social media posts, and comments, detecting emotionally charged but unreliable content.
• Enhancing political campaigns & public policy sentiments: Policymakers across the world struggle to gauge public opinion effectively. A real-time sentiment dashboard can track public sentiment across social media, news, and forums regarding policies, elections, or leaders.
• Crisis management & disaster response: During natural disasters, governments struggle to assess public needs in real time. A sentiment-powered disaster response system can analyze social media, news, and emergency calls to prioritize rescue efforts.

Emotions are harder to detect in text-based communication, leading to misunderstandings. Furthermore, different cultures express emotions differently in their communication. Whether in personal interactions, business communication, marketing, or diplomacy, emotions shape the impact and effectiveness of a language.

Common challenges faced in sentiment analysis

Over the past decade, despite significant advancements in machine learning and artificial intelligence, sentiment analysis continues to grapple with several persistent challenges such as contextual understanding, detecting sarcasm and irony, handling negations, ambiguity and subjectivity, multilingual and cultural nuances and ethical considerations. Other sophisticated challenges include:

• Multimodal sentiment classification: It is the task of predicting sentiment by using information from multiple modalities, such as text, images, and audio. This approach aims to provide a more comprehensive and accurate understanding of human emotions. The main challenge is the improper correlation between the different modalities (Raghunathan & Saravanakumar, 2023).

• Cross-domain sentiment classification: It involves building a model that transfers knowledge from one domain to another (Raghunathan & Saravanakumar, 2023).

These challenges highlight the intricate nature of human language and underscore the need for ongoing research to develop more sophisticated and context-aware sentiment analysis models.

Introduction to Natural Language Processing (NLP)

Natural language processing (NLP) is a major area of artificial intelligence research and has seen a prevalence of very large-scale applications of statistical methods, such as machine learning and data mining (Gelbukh, 2005). The process of deriving computer models from natural language text requires a variety of sophisticated language processing tools. The key components of natural language processing are (Mihalcea et al., 2006) :

1. Tokenization: The process of breaking the text into individual words (tokens) for analysis.
2. Part-of-Speech (POS) Tagging: Identifying grammatical roles of words to understand sentiment emphasis.
3. Dependency Parsing: Mapping relationships between words to understand sentiment modifiers.
4. N-grams & Phrase-Based Analysis: Identifying word sequences that change sentiment.
5. Resolving ambiguity: Many words have multiple meanings depending on context, such as pronoun resolution, noun-modifier relationships, and named entity recognition.
6. Basic construct identification: To classify sentiment correctly, NLP models identify key constructs that express opinions, emotions, or intensity.
7. Sentiment Lexicons: Assigning predefined sentiment values to words.
8. Aspect-Based Sentiment Analysis: Detecting sentiment tied to specific features.
9. Emotion Detection: Going beyond polarity (positive/negative) to classify joy, anger, sadness, etc.
10. Lemmatization: It is the process of reducing a word to its base or root form while ensuring that it retains its meaning within the given context. It helps the models to generalize better by reducing word variations in datasets.

Loading a sample data set and performing basic preprocessing

Raw text data in the form of comments and reviews is usually messy. The raw data needs to be cleaned or pre-processed for noise like HTML tags, emojis, misspellings, or irrelevant punctuation before using it for sentiment analysis in Python. Preprocessing the raw data helps improve accuracy and consistency and reduces computational overhead. Common steps include:

• HTML tags and URLs: Use regex or libraries like BeautifulSoup to strip markups such as HTML/XML.
• Spelling: Use tools like Pyspellchecker or ML-based correctors to correct typos or spelling errors such as “awsum experience” → “awesome experience”.
• Lowercasing: Standardize text to lowercase to avoid treating “Happy” and “happy” as different tokens.
• Punctuation and special characters: Delete symbols like !, ?, or,.

Exercise

• Download and install prerequisite modules:
  • Pyspellchecker
  • BeautifulSoup4
  • Emoji
  • Pandas
• Emoji
• Download the dataset.
• Load it as a Panda data frame.
• Preview the dataset’s basic information.
• Start preprocessing.

References

• A. Gelbukh, “Natural language processing, ” Fifth International Conference on Hybrid Intelligent Systems (HIS’05), Rio de Janeiro, Brazil, 2005, pp. 1 pp.-, https://doi.org/10.1109/ICHIS.2005.79.
• Gupta, S., Ranjan, R., & Singh, S. N. (2024). Comprehensive Study on Sentiment Analysis: From Rule-based to modern LLM  based system. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2409.09989
• Jin, Y., Cheng, K., Wang, X., & Cai, L. (2023). A review of Text Sentiment Analysis Methods and Applications. Frontiers in Business Economics and Management, 10(1), 58–64. https://doi.org/10.54097/fbem.v10i1.10171
• Liu, B. (2015). Sentence subjectivity and sentiment classification. In Cambridge University Press eBooks (pp. 70–89).
• Liu, B., Hu, M., Cheng, J., & Department of Computer Science, University of Illinois at Chicago. (2005). Opinion Observer: Analyzing and comparing opinions on the web. In Department of Computer Science, University of Illinois at Chicago. https://www.cs.uic.edu/~liub/publications/www05-p536.pdf
• Mihalcea, R., Liu, H., & Lieberman, H. (2006). NLP (Natural Language Processing) for NLP (Natural Language Programming). In Lecture notes in computer science (pp. 319–330). https://doi.org/10.1007/11671299_34
• Raghunathan, N., & Saravanakumar, K. (2023). Challenges and Issues in Sentiment Analysis: A Comprehensive survey. IEEE Access, 11, 69626–69642.
• Wattenberg, M., & Fisher, D. (2004). Analyzing perceptual organization in information Graphics. Information Visualization, 3(2), 123–133. https://doi.org/10.1057/palgrave.ivs.9500070
• Wikipedia contributors. (2025, February 2). User interface. Wikipedia. https://en.wikipedia.org/wiki/User_interface