L.L.M 1st Unit

1. What is Natural Language Processing?

Natural Language Processing (N.L.P) is a branch of Artificial Intelligence (A.I) that enables computers to understand, process, and generate human language. It combines linguistics, machine learning (M.L), and deep learning to allow machines to interact with humans through text and speech.

Key Goals of N.L.P:

- ✓ Understanding the structure and meaning of language

- ✓ Extracting useful information from text and speech

- ✓ Enabling machines to communicate in natural language

- ✓ Automating language-related tasks such as translation, summarization, and sentiment analysis

Examples of N.L.P in Action:

- Google Search: Understanding user queries to provide relevant search results.

• Chatbots & Virtual Assistants: Siri, Alexa, and ChatGPT communicate using N.L.P.

- Machine Translation: Google Translate converts text between languages.

• Email Filtering: N.L.P helps detect spam emails.

- Customer Service Automation: A.I-powered chatbots handle customer inquiries.

2. How N.L.P Is Used?

Natural Language Processing (N.L.P) plays a crucial role in transforming industries by improving efficiency, automating processes, and enhancing user experiences. It is widely used in search engines, chatbots, finance, healthcare, and more.

A. Real-World Applications of N.L.P

1. Search Engines (Google, Bing, Yahoo, DuckDuckGo, etcetera)

✓ N.L.P helps search engines understand user queries and retrieve the most relevant results.

✓ It enables semantic search, meaning search engines understand the intent behind a query rather than just matching keywords.

✓ Features like autocomplete, spell check, and voice search rely on N.L.P.

Example:

- If you type "best places to visit in summer", N.L.P helps Google understand that you're looking for travel recommendations, not just any random information about summer.

2. Chatbots & Virtual Assistants (Siri, Alexa, Google Assistant, ChatGPT, etcetera)

- Chatbots use N.L.P to understand and generate human-like responses.

- They can handle customer service queries, answer F.A.Q's, and even process orders in e-commerce.

- Virtual assistants like Alexa and Siri rely on N.L.P for voice recognition and context-aware responses.

Example:

- If you ask Siri "Remind me to call mom at 6 P.M", N.L.P ensures that the request is processed, scheduled, and stored correctly in your reminders.

3. Machine Translation (Google Translate, DeepL, Microsoft Translator, etcetera)

- N.L.P enables automatic translation between different languages while preserving meaning.

- Advanced models like Google's Transformer-based bert and G.P.T improve accuracy by understanding context.

- Machine translation is crucial for global communication, business expansion, and accessibility.

Example:

- Translating "It's raining cats and dogs" into another language while maintaining its idiomatic meaning instead of a literal translation.

4. Speech Recognition (Voice-to-Text, Google Assistant, Dictation Software, etcetera)

- ✓ N.L.P enables speech-to-text conversion, allowing devices to understand and process spoken language.

- Used in smartphones, transcription software, accessibility tools for the disabled, and call center automation.

- ✓ Combines N.L.P with acoustic models and phonetics to improve accuracy.

Q Example:

YouTube's auto-generated subtitles use N.L.P to transcribe speech into text.

5. Sentiment Analysis (Opinion Mining, Social Media Monitoring, Customer Feedback Analysis, etcetera)

✓ N.L.P helps businesses understand customer emotions by analyzing text from reviews, social media, and surveys.

- ✓ Used by companies, politicians, brands, and marketing teams to gauge public opinion.

Q Example:

- A company analyzing Twitter posts to see if customers are happy with a new product release.

6. Text Summarization (News Aggregation, Report Summaries, Document Processing, etcetera)

- ✓ N.L.P helps summarize long articles, reports, and research papers into key points.

- Used in news apps, academic research, business intelligence reports, and legal document processing.

- ✓ Can be extractive (pulling key sentences) or abstractive (generating a summary in new words).

Q Example:

- Google News providing a summary of trending articles instead of showing full content.

7. Medical N.L.P (Healthcare Automation, Electronic Health Records, Disease Diagnosis, etcetera)

- ✓ N.L.P helps analyze medical records, assist in diagnosis, and improve clinical decision-making.

- ✓ Extracts insights from medical literature, patient reports, and clinical trial data.

- Used in chatbots for medical advice, symptom checkers, and health monitoring.

Q Example:

- I.B.M Watson Health analyzes patient symptoms and medical records to recommend possible diagnoses.

8. Finance & Banking (Fraud Detection, Automated Trading, Sentiment Analysis, etcetera)

- ✓ N.L.P helps detect fraudulent transactions by analyzing customer messages and patterns.

- ✓ Used in automated trading systems to make real-time stock market predictions.

- ✓ Helps banks automate customer support via chatbots.

Q Example:

- A banking chatbot that understands customer requests like "Show me my last five transactions" and retrieves data instantly.

B. N.L.P in Different Industries

Table summary: NLP applications and their benefits across three key industries. In Healthcare, NLP is used for medical diagnosis, analyzing patient records, and drug discovery by processing medical literature. In E-commerce, it powers chatbots and virtual shopping assistants, product recommendations, and the analysis of customer reviews to identify trends. In Finance and Banking, NLP is applied to fraud detection via transaction log scanning and automated trading through the analysis of financial news.

Table summary: Examples of NLP applications and benefits across three industries. In Education, NLP is used for automated grading of assignments, plagiarism detection through tools like Turnitin, and providing personalized lessons via AI tutors. Media and Journalism utilizes NLP for automated news generation with GPT-4, misinformation detection, and summarized content for readers. In the banking sector, NLP is applied to customer support automation via chatbots.

3. Building N.L.P Applications

Developing an N.L.P application requires a structured approach, from data collection to model deployment. Below is a detailed breakdown of each step in the process, including text preprocessing techniques, machine learning models, and popular N.L.P tools.

A. Steps to Build an N.L.P Application (Detailed Explanation)

1. Data Collection

- ✓ What It Is:

• Collecting raw textual data from different sources to train and test N.L.P models.

✓ Sources of Data:

- Public Datasets (e.g., Wikipedia, Common Crawl, Kaggle datasets)

- A.P.I's (e.g., Twitter A.P.I for sentiment analysis)

- Web Scraping (e.g., BeautifulSoup for extracting web data)

• Manual Data Entry (e.g., human-annotated datasets)

- ✓ Challenges:

- Data may contain noise (irrelevant content, duplicates)

- Privacy concerns when collecting user data

• Need for large, diverse datasets for deep learning

A chatbot for an e-commerce website needs a dataset containing customer queries, product descriptions, and past chat logs to train the model.

2. Text Preprocessing

- Cleaning and preparing raw text for analysis by removing unnecessary elements.

☑ Why It Matters:

- Raw text contains punctuation, special characters, typos, and stopwords that need to be removed for better model performance.

✓ Techniques:

- Tokenization: Breaking text into words/sentences ("I love A.I" leads to [I, love, A.I])

- Lowercasing: Converting text to lowercase "Hello" leads to "hello")

- Stopword Removal: Removing common words ("I", "am", a", "developer"] leads to ["developer"]

• Stemming: Reducing words to their root form ("running" leads to "run")

- Lemmatization: Converting words to their dictionary form ("better" leads to "good")

- Part-of-Speech (P.O.S) Tagging: Assigning grammatical labels ("A.I is powerful" to ("A.I", Noun), ("is", Verb), ("powerful", Adjective))

- Named Entity Recognition (N.E.R): Identifying proper nouns ("Tesla is a company" to ) ("Tesla", Company))

Q Example:

For a sentiment analysis model, removing stopwords ("the", "is", "and") improves efficiency, while lemmatization ("running" to "run") helps in standardizing text.

3. Feature Engineering (Converting Text into Numerical Data)

✓ What It Is:

- Converting text into a format that machine learning models can understand.

Common Techniques:

1. Bag of Words (BoW):

o Represents text as word frequency vectors.

○ Example: "1 love A.I" arrow left curly brace "I": 1, "love": 1, "A.I": 1 right curly brace

2. T.F-I.D.F (Term Frequency-Inverse Document Frequency):

○ Assigns weights to words based on importance in a document.

Common words (like "the") get lower weights, while rare words (like "blockchain") get higher weights.

3. Word Embeddings (Word2Vec, GloVe, FastText):

o Converts words into dense numerical vectors, capturing semantic meaning.

○ Example: "king" - "man" + "woman" = "queen" (word relationships in vector space).

4. Transformers (bert, G.P.T, T.5):

○ Uses deep learning to understand word context in sentences.

○ Example: "bank" (as a financial institution) vs "bank" (as a riverbank).

Q Example:

In spam detection, converting emails into T.F-I.D.F vectors helps classify them as spam or not spam.

4. Model Training

• Applying Machine Learning (M.L) or Deep Learning (D.L) models to process text data.

✓ Types of Models:

- Rule-Based Systems: Uses predefined grammar rules (good for structured texts).

- Machine Learning Models: Naïve Bayes, S.V.M, Decision Trees (used for sentiment analysis, spam detection).

- Deep Learning Models: R.N.N's, L.S.T.M's, Transformers like bert, G.P.T (used for advanced N.L.P tasks).

Q Example:

For a chatbot, using a deep learning model like G.P.T-4 improves its ability to understand and generate responses naturally.

5. Model Evaluation & Fine-Tuning

✓ What It Is:

- Assessing model performance and improving it for better accuracy.

✓ Evaluation Metrics:

• Accuracy: Percentage of correct predictions.

• Precision & Recall: Measures how well the model identifies correct vs incorrect results.

- F.1 Score: Balance between precision and recall.

✓ Fine-Tuning Techniques:

• Adjusting hyperparameters (learning rate, batch size).

• Adding more training data.

• Using pretrained models (bert, G.P.T) and transfer learning.

For sentiment analysis, evaluating the model on real customer reviews ensures its accuracy.

6. Deployment (Making the N.L.P Model Available for Users)

- Integrating the trained N.L.P model into an application for real-world use.

✓ Deployment Options:

- A.P.I's: Hosting the model as a rest A.P.I (Flask, FastAPI).

• Cloud Platforms: Google Cloud, A.W.S, Azure.

- Edge Deployment: Running models on mobile devices (TensorFlow Lite).

Q Example:

A customer support chatbot is deployed as an A.P.I, which websites and apps can integrate.

B. N.L.P Models & Techniques

Table B summary: Three types of Natural Language Processing models categorized by their technical approach and application. Rule-Based NLP uses predefined grammar and linguistic rules for text parsing and chatbots. Machine Learning-Based NLP employs statistical models like Naïve Bayes and SVM for sentiment analysis and spam detection. Deep Learning-Based NLP utilizes neural networks, including LSTMs and Transformers such as BERT and GPT, for machine translation and chatbots.

C. Common N.L.P Tools & Frameworks

A developer can use spaCy for fast text preprocessing, Hugging Face Transformers for state-of-the-art N.L.P models, and Flask/FastAPI to deploy the model as a web A.P.I.

Table C summary: A collection of NLP tools and libraries categorized by their function. These include Python libraries for basic processing like NLTK and large-scale processing with spaCy, deep learning resources such as Hugging Face Transformers for pretrained models and TensorFlow/PyTorch for training, and enterprise cloud services like Google Cloud NLP API and IBM Watson NLP.

4. Development and Structure

Natural Language Processing (N.L.P) is a subfield of artificial intelligence (A.I) and computational linguistics that focuses on the interaction between computers and human language. It involves the development of algorithms and models that enable machines to understand, interpret, generate, and respond to human language in a way that is both meaningful and useful. N.L.P has seen significant advancements over the years, driven by the increasing availability of data, improvements in computational power, and breakthroughs in machine learning and deep learning techniques.

1. Development of N.L.P

The development of N.L.P can be traced through several key phases, each marked by significant advancements in theory, methodology, and application.

1.1 Early Beginnings (1950s - 1970s)

- Rule-Based Systems: The earliest N.L.P systems were based on hand-crafted rules. These systems relied on linguistic theories and formal grammars to parse and generate sentences. For example, the Georgetown-I.B.M experiment in 1954 demonstrated the feasibility of machine translation by automatically translating Russian sentences into English.

- Chomsky's Transformational Grammar: Noam Chomsky's work in the 1950s and 1960s laid the theoretical foundation for understanding the structure of language. His theories on syntax and transformational grammar influenced early N.L.P systems, which attempted to model language using formal rules.

1.2 Statistical N.L.P (1980s - 1990s)

- Shift to Probabilistic Models: In the 1980s, there was a shift from rule-based systems to statistical methods. Researchers began to use probabilistic models to capture the uncertainty and variability inherent in natural language. This period saw the rise of Hidden Markov Models (H.M.M's), n-gram models, and probabilistic context-free grammars.

- Corpus-Based Approaches: The availability of large text corpora, such as the Brown Corpus and the Penn Treebank, enabled the development of data-driven approaches. These corpora were used to train statistical models for tasks like part-of-speech tagging, parsing, and machine translation.

1.3 Machine Learning Era (2000s - 2010s)

- Supervised Learning: The 2000s saw the widespread adoption of machine learning techniques in N.L.P. Supervised learning algorithms, such as Support Vector Machines (S.V.M's) and conditional random fields (C.R.F's), were applied to various N.L.P tasks, including named entity recognition, sentiment analysis, and text classification.

- Unsupervised and Semi-Supervised Learning: Researchers also explored unsupervised and semi-supervised learning methods, such as topic modeling (e.g., Latent Dirichlet Allocation) and word embeddings (e.g., Word2Vec, GloVe), which allowed models to learn representations of words and documents from large amounts of unlabeled text.

1.4 Deep Learning Revolution (2010s - Present)

- Neural Networks: The advent of deep learning has revolutionized N.L.P. Neural network architectures, such as Recurrent Neural Networks (R.N.N's), Long Short-Term Memory (L.S.T.M) networks, and Convolutional Neural Networks (C.N.N's), have achieved state-of-the-art performance on a wide range of N.L.P tasks.

• Transformers and Pretrained Models: The introduction of the Transformer architecture in 2017 marked a significant breakthrough. Transformers, which rely on self-attention mechanisms, have become the foundation for many state-of-the-art models, such as bert, G.P.T, and T.5. These models are pretrained on large corpora and fine-tuned for specific tasks, leading to unprecedented performance in tasks like machine translation, question answering, and text generation.

2. Structure of N.L.P

N.L.P encompasses a wide range of tasks and applications, each with its own set of challenges and techniques. The structure of N.L.P can be broadly categorized into several key areas:

2.1 Syntax

- Part-of-Speech Tagging: Assigning grammatical categories (e.g., noun, verb, adjective) to each word in a sentence.

- Parsing: Analyzing the grammatical structure of a sentence to determine its syntactic tree. This includes dependency parsing and constituency parsing.

- Morphological Analysis: Breaking down words into their constituent morphemes (e.g., prefixes, suffixes) to understand their structure and meaning.

2.2 Semantics

- Word Sense Disambiguation: Determining the correct meaning of a word in context, especially for words with multiple meanings (e.g., "bank" as a financial institution versus "bank" as the side of a river).

- Named Entity Recognition (N.E.R): Identifying and classifying entities (e.g., names of people, organizations, locations) in text.

- Semantic Role Labeling: Identifying the roles of words or phrases in a sentence (e.g., agent, patient, instrument) in relation to a predicate.

2.3 Discourse

- Coreference Resolution: Identifying when different words or phrases refer to the same entity in a text (e.g., "John" and "he" referring to the same person).

- Text Coherence and Cohesion: Understanding how sentences and paragraphs are connected to form a coherent text.

2.4 Pragmatics

- Speech Act Recognition: Identifying the intended action behind an utterance (e.g., a request, a command, a question).

• Sentiment Analysis: Determining the emotional tone or sentiment expressed in a piece of text (e.g., positive, negative, neutral).

- Pragmatic Inference: Understanding implied meanings and context-dependent interpretations of text.

2.5 Applications

- Machine Translation: Automatically translating text from one language to another (e.g., Google Translate).

- Text Summarization: Generating concise summaries of longer texts (e.g., news articles, research papers).

- Question Answering: Building systems that can answer questions posed in natural language (e.g., Siri, Alexa).

- Chatbots and Virtual Assistants: Developing conversational agents that can interact with users in natural language (e.g., customer support chatbots).

- Information Retrieval: Searching and retrieving relevant documents or information from large text collections (e.g., search engines like Google).

3. Challenges in N.L.P

Despite significant progress, N.L.P still faces several challenges:

- Ambiguity: Natural language is inherently ambiguous, with words and sentences often having multiple interpretations.

- Context Understanding: Understanding context and world knowledge is crucial for accurate language understanding, but it remains a difficult problem.

- Low-Resource Languages: Many N.L.P models require large amounts of annotated data, which is not available for all languages.

- Bias and Fairness: N.L.P models can inherit biases present in the training data, leading to unfair or discriminatory outcomes.

- Interpretability: Deep learning models, while powerful, are often seen as "black boxes," making it difficult to understand how they arrive at their predictions.

4. Future Directions

The future of N.L.P is likely to be shaped by several emerging trends:

- Multimodal N.L.P: Integrating text with other modalities, such as images, audio, and video, to enable more comprehensive understanding and generation of content.

- Few-Shot and Zero-Shot Learning: Developing models that can perform well with minimal or no labeled data, reducing the reliance on large annotated datasets.

- Ethical A.I: Addressing issues of bias, fairness, and transparency in N.L.P models to ensure they are used responsibly and ethically.

- Personalization: Creating N.L.P systems that can adapt to individual users' preferences, styles, and needs.

- Human-A.I Collaboration: Enhancing the ability of N.L.P systems to work alongside humans, augmenting human capabilities rather than replacing them.

5. Word and Document Embeddings:

1. Introduction to Embeddings

Embeddings are numerical representations of words, phrases, or entire documents in a continuous vector space. They capture the semantic meaning and relationships between words based on their usage in a given context.

Embeddings play a crucial role in Natural Language Processing (N.L.P) and machine learning tasks such as sentiment analysis, machine translation, information retrieval, and chatbot development.

2. Understanding the Building Blocks of Language

Before diving into embeddings, it's important to understand the fundamental components of language:

a. Characters

Characters are the smallest units of written language. They include:

• Letters (A-Z, a-z)

• Numbers (0 to 9)

• Special symbols (!, @, #, etcetera)

Punctuation marks (., ?, !)

In N.L.P, character-level analysis is useful for tasks such as:

• Spelling correction

Named entity recognition (handling unseen words)

• Language modeling

b. Words

Words are sequences of characters that carry meaning. They are the basic units of communication in a language.

- Example: "Apple," "run," "beautiful"

Words can be further categorized into:

- Nouns, verbs, adjectives, etcetera (Part of speech)

• Synonyms and antonyms (Lexical relationships)

c. Phrases

Phrases are combinations of words that form meaningful expressions.

- Example: "Artificial Intelligence," "deep learning," "fast runner"

Phrases often have meanings that cannot be understood from individual words alone. N.L.P models, such as word embeddings, help capture these contextual meanings.

3. What Are Word Embeddings?

Word embeddings are dense vector representations of words. They transform words into numerical values that reflect their meaning and usage in a given context.

a. How Do Word Embeddings Work?

Word embeddings are learned from large corpora using machine learning models. The idea is that words appearing in similar contexts should have similar vector representations.

For example, in the sentences:

• "The cat is sleeping on the couch."

• "The dog is resting on the sofa."

The words "cat" and "dog" appear in similar contexts, so their vector representations will be close in the embedding space.

b. Common Word Embedding Techniques

1. Word2Vec

o Uses a neural network to learn word associations from a large text corpus.

○ Two models:

cbow (Continuous Bag of Words): Predicts a word from its surrounding words.

Skip-gram: Predicts surrounding words given a word.

2. GloVe (Global Vectors for Word Representation)

○ Based on matrix factorization of word co-occurrence statistics.

o Captures semantic relationships between words.

3. FastText

o Similar to Word2Vec but considers subword (character n-gram) information.

Helps in handling out-of-vocabulary (O.O.V) words.

4. elmo (Embeddings from Language Models)

o Contextual word embeddings that capture different meanings of words based on context.

5. bert (Bidirectional Encoder Representations from Transformers)

○ Uses deep neural networks to generate contextual embeddings for words and phrases.

4. What Are Document Embeddings?

Document embeddings extend the concept of word embeddings to entire sentences, paragraphs, or documents. They capture the overall meaning of the text rather than individual words.

a. How Do Document Embeddings Work?

Instead of assigning vectors to single words, document embeddings represent an entire text body as a vector in a continuous space.

b. Common Document Embedding Techniques

1. T.F-I.D.F (Term Frequency-Inverse Document Frequency)

o Measures the importance of a word in a document relative to a corpus.

○ Does not capture semantic meaning but is useful for information retrieval.

2. Doc2Vec

o Extends Word2Vec to generate embeddings for sentences or documents.

○ Two models:

P.V-D.M (Distributed Memory Model of Paragraph Vectors)

P.V-D.B.O.W (Distributed Bag of Words)

3. Transformer-Based Models (bert, G.P.T, T.5, etcetera)

o Generate contextual document representations.

○ Useful for tasks like text summarization, classification, and question-answering.

5. Applications of Word and Document Embeddings

Word and document embeddings power many real-world N.L.P applications, such as:

• Search Engines: Understanding user queries and ranking relevant results.

• Chatbots and Virtual Assistants: Understanding natural language inputs.

- Machine Translation: Translating text across languages while preserving meaning.

• Text Summarization: Extracting key information from large documents.

• Spam Detection: Identifying spam emails based on document embeddings.

• Sentiment Analysis: Determining emotions in customer reviews and social media posts.

6. Tokenization, Stemming, and Lemmatization:

Tokenization, stemming, and lemmatization are essential preprocessing techniques in Natural Language Processing (N.L.P). They help transform raw text into a structured format that can be efficiently processed by machine learning models and linguistic analysis tools. These techniques improve text analysis, information retrieval, and sentiment analysis.

2. Tokenization

a. What is Tokenization?

Tokenization is the process of breaking down text into smaller units called tokens. These tokens can be words, phrases, sentences, or even characters, depending on the level of tokenization.

b. Types of Tokenization

1. Word Tokenization (Lexical Tokenization)

Splits text into individual words.

• Example:

○ Input: "Natural Language Processing is amazing!"

○ Output: ["Natural", "Language", "Processing", "is", "amazing", "!"]

2. Sentence Tokenization

Splits text into sentences.

• Example:

○ Input: "A.I is the future. It is transforming industries."

○ Output: ["A.I is the future.", "It is transforming industries."]

3. Subword Tokenization

- Breaks words into smaller meaningful parts (useful in models like bert and G.P.T).

• Example:

○ Input: "Unhappiness"

- Output: ["Un", "happiness"]

4. Character Tokenization

Splits text into individual characters.

• Example:

○ Input: "A.I"

○ Output: ["A", "I"]

c. Why is Tokenization Important?

- It converts unstructured text into a structured format.

- Essential for search engines, chatbots, and text analysis applications.

• Helps in handling different languages, punctuation, and word boundaries.

d. Challenges in Tokenization

• Handling contractions (e.g., "can't" becomes "can" plus "not").

• Dealing with punctuation (e.g., "Mr. John" versus "Mr" + ".").

• Managing languages without spaces (e.g., Chinese, Japanese).

3. Stemming

a. What is Stemming?

Stemming is the process of reducing a word to its root or base form by removing prefixes or suffixes. The output may not always be a valid word.

b. How Stemming Works

• It follows heuristic-based rules to remove word affixes (prefixes and suffixes).

○ Input: "running, runs, runner"

○ Output: "run"

c. Common Stemming Algorithms

1. Porter Stemmer

- One of the oldest and most widely used stemming algorithms.

• Example:

○ Input: "connection, connected, connecting"

○ Output: "connect"

2. Lancaster Stemmer

More aggressive than the Porter Stemmer, often reducing words too much.

• Example:

○ Input: "running, runner, runs"

○ Output: "run"

3. Snowball Stemmer

- An improvement over the Porter Stemmer with support for multiple languages.

d. Advantages of Stemming

• Reduces words to their base form, improving text search efficiency.

• Helps in document clustering and text classification.

e. Disadvantages of Stemming

• Stems may not always be meaningful words.

- It may over-stem (removing too many letters) or under-stem (not removing enough letters).

- Lacks context awareness.

4. Lemmatization

a. What is Lemmatization?

Lemmatization is the process of reducing words to their base form (lemma) using linguistic rules. Unlike stemming, it ensures the root form is a valid word.

b. How Lemmatization Works

- Uses vocabulary and morphological analysis to return the base form.

○ Input: "running, ran, runs"

○ Output: "run"

: Table c summary: Lemmatization is a more linguistically accurate but computationally expensive alternative to stemming. While stemming is faster and simply removes suffixes and prefixes, it lacks context awareness and may produce outputs that are not valid words. In contrast, lemmatization uses a dictionary-based approach with context awareness to ensure the output is always a meaningful word. For example, while both might reduce running to run, lemmatization provides a more robust morphological analysis.

d. Why is Lemmatization Important?

• Improves text normalization for N.L.P tasks.

• Helps in machine translation, text summarization, and chatbots.

- Used in search engines to match different forms of a word.

e. Challenges in Lemmatization

• Requires a dictionary and part-of-speech (P.O.S) tagging.

• Computationally expensive compared to stemming.

5. Applications of Tokenization, Stemming, and Lemmatization

a. Search Engines

Tokenization helps break queries into words.

• Stemming/Lemmatization ensures results include variations of a word.

b. Chatbots and Virtual Assistants

- Understands user input better by reducing words to their base form.

c. Text Classification & Sentiment Analysis

• Helps in feature extraction by normalizing words.

d. Machine Translation

- Improves language modeling by handling different word forms.

e. Speech Recognition

Tokenization and lemmatization improve the accuracy of voice-to-text applications.

7. Skip-Gram, Continuous Bag of Words (cbow), GloVe, and fastText:

These are popular word embedding techniques used in Natural Language Processing (N.L.P) to represent words as dense vectors in a continuous space. Each method has its unique approach to capturing semantic and syntactic relationships between words. Below, we explain each technique in detail, including their architectures, training processes, and use cases, with diagrams where applicable.

Skip-gram and Continuous Bag of Words (cbow)

These are two architectures introduced by the Word2Vec model, developed by Google in 2013. Both are shallow neural networks trained to predict words in a context window.

1.1 Skip-gram

- Objective: Predict context words given a target word.

• Architecture:

○ Input: A target word (one-hot encoded).

- Hidden Layer: A dense layer that projects the input into a lower-dimensional embedding space.

- o Output: A probability distribution over the vocabulary for context words.

• Training:

Example:

o For a given target word, the model predicts the surrounding context words within a fixed window size.

o The model is trained using negative sampling or hierarchical softmax to optimize the likelihood of predicting the correct context words.

• Sentence: "I love natural language processing."

• Target Word: "natural"

• Context Words: ["love", "language", "processing"]

1.2 Continuous Bag of Words (cbow)

- Objective: Predict a target word given its context words.

• Architecture:

- o Input: A set of context words (one-hot encoded).

- Hidden Layer: A dense layer that averages the embeddings of the context words.

- o Output: A probability distribution over the vocabulary for the target word.

• Training:

Example:

For a given set of context words, the model predicts the target word.

o The model is trained using negative sampling or hierarchical softmax.

• Sentence: "I love natural language processing."

• Context Words: ["love", "language", "processing"]

• Target Word: "natural"

: Table 1.3 summary: Skip-gram and CBOW differ primarily in their input-output directions and performance strengths. Skip-gram uses a target word as input to predict context words, making it better for rare words and large datasets. In contrast, CBOW uses context words as input to predict a target word, offering faster training and better performance for frequent words on smaller datasets.

2. GloVe (Global Vectors for Word Representation)

GloVe, developed by Stanford in 2014, is a global log-bilinear regression model that combines the advantages of global matrix factorization (like L.S.A) and local context window methods (like Word2Vec).

2.1 Key Idea

- GloVe constructs a word-word co-occurrence matrix from a corpus, where each entry represents how often two words appear together in a context window.

- It learns word embeddings by factorizing this matrix, minimizing the difference between the dot product of word vectors and the logarithm of their co-occurrence probability.

2.2 Training Objective

• The loss function is designed to minimize:

J equals the sum over i and j of f of X i j times open parenthesis w i transpose w j plus b i plus b j minus log X i j close parenthesis squared where:

○ Xij: Co-occurrence count of words i and j.

o wi and w~j: Word vectors for words i and j.

○ bi and b~j: Bias terms.

o f(Xij): A weighting function to reduce the influence of rare co-occurrences.

2.3 Advantages of GloVe

• Captures both global and local statistical information.

• Efficient training on large corpora.

• Produces high-quality embeddings for downstream tasks.

3. fastText

fastText, developed by Facebook in 2016, extends Word2Vec by representing words as bags of character n-grams. This allows it to handle out-of-vocabulary words and morphologically rich languages.

3.1 Key Idea

- Each word is represented as a sum of its character n-grams (e.g., for "apple" with n=3, the n-grams are " less than ap", "app", "ppl", "ple", "le}).

- The embedding for a word is the average of its n-gram embeddings.

3.2 Training

• Similar to Word2Vec, fastText uses either Skip-gram or cbow architectures.

- The model is trained to predict context words (Skip-gram) or target words (cbow) based on n-gram representations.

3.3 Advantages of fastText

• Handles out-of-vocabulary words by breaking them into n-grams.

• Effective for morphologically rich languages (e.g., German, Finnish).

• Produces subword embeddings, which are useful for rare words.

Word rightarrow Character N-grams rightarrow Embedding Sum/Average rightarrow

Table 4 summary: fastText is the only technique among those compared that handles out-of-vocabulary words and provides specific support for morphologically rich languages, achieving this by using word n-grams as input to produce word and subword embeddings. In contrast, Skip-gram/CBOW and GloVe both fail to handle out-of-vocabulary words and provide general language support. While GloVe is the fastest to train, using a co-occurrence matrix as input, Skip-gram/CBOW and fastText both have moderate training speeds.

8. Document-Level Embeddings

Word embeddings such as Word2Vec, GloVe, and fastText generate vector representations for individual words. However, many N.L.P tasks require representations of entire sentences, paragraphs, or documents—this is where document-level embeddings come into play.

Additionally, since embeddings are high-dimensional, visualizing embeddings helps understand word and document relationships, clustering, and model performance.

A. What Are Document-Level Embeddings?

Document-level embeddings are vector representations of entire documents, capturing both word meanings and document structure. These embeddings enable tasks such as document classification, clustering, and retrieval.

B. Why Use Document-Level Embeddings?

• Captures document-wide semantic meaning.

• Allows machine learning models to compare entire documents instead of just words.

Enables tasks such as text summarization, sentiment analysis, and search ranking.

C. Methods for Generating Document Embeddings

Several methods exist for generating document embeddings. Some use word embeddings, while others rely on deep learning-based architectures.

1. Averaging Word Embeddings

A simple approach is to average the word embeddings of all words in a document.

- Method: Sum all word vectors and divide by the number of words.

- Example: If "artificial" and "intelligence" have vectors, the document embedding could be:

vector artificial plus vector intelligence 2 fraction vector artificial vector intelligence over 2 vector artificial plus vector intelligence

- Limitation: This ignores word order and sentence structure.

2. T.F-I.D.F Weighted Word Embeddings

Instead of averaging, we can weight word embeddings based on their T.F-I.D.F scores.

- Method: Compute weighted average, where more important words contribute more to the document vector.

• Advantage: Common words (e.g., "the", "is") contribute less, improving representation.

3. Doc2Vec (Paragraph Vectors)

Doc2Vec is an extension of Word2Vec, developed to generate fixed-length document embeddings. It has two major models:

- Distributed Memory Model (D.M): Learns a unique vector for each document and uses it along with word vectors to predict missing words.

- Distributed Bag of Words (dbow): Learns a document vector by predicting words from a document in a Word2Vec-style model.

Advantage: Captures word order and document-specific information. Limitation: Requires large training datasets.

4. Transformer-Based Embeddings (bert, G.P.T, S.B.E.R.T)

Pre-trained transformers like bert, G.P.T, and S.B.E.R.T generate high-quality document embeddings.

• bert (Bidirectional Encoder Representations from Transformers):

○ Uses self-attention mechanisms to capture context-aware word and sentence embeddings.

- o Suitable for question answering, text classification, and named entity recognition.

- S.B.E.R.T (Sentence-bert):

- o Fine-tuned from bert to generate sentence/document-level embeddings efficiently.

○ Produces better similarity scores between documents.

- Advantage: Handles complex word relationships and long-range dependencies.

- Limitation: Computationally expensive.

5. Universal Sentence Encoder (U.S.E)

- Developed by Google, U.S.E generates fixed-length embeddings for sentences and documents.

- Advantage: Pre-trained and optimized for tasks like semantic similarity and text classification.

- Limitation: Requires high computational power.

9. Visualizing Embeddings

Since embeddings exist in high-dimensional spaces (e.g., 300D for Word2Vec, 768D for bert), they need to be reduced to 2D or 3D for visualization.

A. Why Visualize Embeddings?

• Helps understand word/document relationships.

• Identifies clusters and outliers.

Aids in debugging N.L.P models.

B. Methods for Visualizing Embeddings

1. P.C.A (Principal Component Analysis)

P.C.A is a linear dimensionality reduction technique that projects high-dimensional vectors into lower dimensions while preserving variance.

- Example: Given 300D Word2Vec embeddings, P.C.A can reduce them to 2D or 3D scatter plots.

- Limitation: P.C.A is linear, so it may miss non-linear relationships.

2. t-S.N.E (t-Distributed Stochastic Neighbor Embedding)

t-S.N.E is a non-linear dimensionality reduction method that groups similar words/documents closer together.

- Example: In a word embedding visualization, words like "king", "queen", "prince" form one cluster, while "apple", "banana" form another.

• Advantage: Preserves local relationships in data.

- Limitation: Computationally expensive for large datasets.

3. umap (Uniform Manifold Approximation and Projection)

umap is similar to t-S.N.E but is faster and preserves both local and global structures.

• Advantage: Suitable for large-scale word/document embeddings.

- Example: Can cluster articles, topics, or similar documents together.

4. TensorFlow Embedding Projector

Google's TensorFlow Embedding Projector allows interactive visualization of embeddings using P.C.A, t-S.N.E, and umap.

• Enables searching for specific words/documents.

- Useful for N.L.P research and debugging models.

4. Applications of Document-Level Embeddings

Document embeddings are widely used in real-world N.L.P applications.

• Text Classification: Spam detection, sentiment analysis, topic classification.

• Information Retrieval: Search engines, document similarity ranking.

- Machine Translation: Helps generate context-aware translations.

• Chatbots and Virtual Assistants: Enables chatbots to understand entire messages.

- Recommendation Systems: Suggests related articles based on semantic similarity.

5. Summary and Comparison

Table summary: Averaging Word Embeddings, which calculates the mean of word vectors, is simple and efficient but is limited because it ignores word order.

Table summary: A comparison of four embedding methods for document and sentence representation. TF-IDF Weighted Embeddings use a weighted average to reduce the influence of common words, but they ignore context. Doc2Vec captures document semantics but requires large amounts of training data. BERT/SBERT provide powerful contextualized embeddings at the cost of being computationally expensive. Finally, the Universal Sentence Encoder offers efficiency and high accuracy, though it requires GPU or TPU for large-scale tasks.