Using NLP, this project gauges customer sentiments online, offering customization and real-time feedback. Employing TF-BOW-LDA and ML models on train.csv dataset, it empowers e-commerce decisions, culminating in an NLP course at uOttawa in 2023.

• Required libraries: scikit-learn, pandas, matplotlib.
• Execute cells in a Jupyter Notebook environment.
• The uploaded code has been executed and tested successfully within the Google Colab environment.

Perform supervised sentiment analysis to categorize user sentiments into three classes: Positive, Negative, and Neutral.

• 'name': Name of the product.
• 'brand': Brand of the product.
• 'categories': Categories associated with the product.
• 'primaryCategories': Primary category of the product.
• 'reviews.date': Date of the review.
• 'reviews.text': Text content of the review.
• 'reviews.title': Title of the review.

• 'sentiment': Dependent variable indicating the sentiment (Positive, Negative, Neutral) of the review.

1. Data Explore:

   • The most common keywords and their counts.

   • The most common Positive words using WorldCloud.

   • The most common Negative words using WorldCloud.

   • The most common Neutral words using WorldCloud.

2. Data Preparation:

   • Data Cleaning

     • Handling Missing Data: The dataset has a very low percentage of missing cells (less than 0.1%) (10 values in reviews.title ), so we can safely drop or impute those missing values based on the specific context.
     • Handling Duplicate Rows: The dataset has 1.5% duplicate rows, which can be removed to ensure data integrity

   • Renaming and Dropping Columns:Renamed the columns 'reviews.text,' 'reviews.title,' and 'reviews.date' to 'reviews_text,' 'reviews_title,' and 'reviews_date,' respectively. Additionally, Dropped the columns 'name,' 'brand,' 'categories,' 'primaryCategories,' and 'reviews.date' from the dataset.

   • Sentiment Label Encoding: Created a mapping dictionary for sentiment labels and encoded the 'sentiment' column into numerical form (1 for 'Positive,' -1 for 'Negative,' and 0 for 'Neutral').

   • Create new Column ‘Polarity Scores’: Apply the SentimentIntensityAnalyzer to the 'reviews_text' column to calculate polarity scores for each review. Polarity scores represent the sentiment of the text as a continuous value between -1 (negative) and 1 (positive).

   • Balancing Data : The classes are imbalanced, you may consider applying techniques like SMOTE to balance the data.

3. Text Feature Engineering:

   • Normalizing Case Folding: Convert all text to lowercase to ensure consistent comparisons between words.
   • Removing Punctuation: Eliminate special characters and punctuation marks from the text to avoid any interference in analysis.
   • Removing Numbers: Exclude numerical digits from the text as they may not be relevant for certain tasks like sentiment analysis.
   • Removing Stopwords: Remove common words that do not carry much meaning (e.g., "the," "and," "is") using stopwords from the
   • English language.Remove Rare Words: Eliminate words that appear infrequently in the dataset, as they may not contribute significantly to the analysis.
   • Lemmatization: Convert words to their base or root form (lemmas) to reduce inflected words to a common base form. For example, "running," "runs," and "ran" will all be transformed to "run."

4. Text Transformations:

   • Bag-of-Words (BOW): Similar to TF, but it also ignores the frequency and considers only whether a word appears or not (binary representation).

   

   • Term Frequency-Inverse Document Frequency (TF-IDF): Convert the text data into a bag-of-words representation, where each document is represented as a vector of word frequencies in the corpus.

   

   • Latent Dirichlet Allocation (LDA): Perform topic modeling to extract latent topics from the text data. Each document is represented as a mixture of topics.

     

5. Modeling

   • Classfication (Random Forest , SVM , Logistic Regression , Gaussian Navie Bayes)

     • BOW Technique

       

     

     • TF-IDF Technique

       

     

     • LDA Technique

       

     

   • Clustering ( K-Means , Hierarchical)

     • BOW Technique

           Silhouette Score (K-Means): 81.55401438608376
           Silhouette Score (Hierarchical) : 17.925024032592773

       

     • TF-IDF Technique

           Silhouette Score (K-Means): 0.7683612431807604
           Silhouette Score (Hierarchical) : 17.966507375240326

       

     • LDA Technique

           Silhouette Score (K-Means): 81.55401438608376
            Silhouette Score (Hierarchical) 16.194509

       

6. Evaluations

   

   

7. Champion Model