Another little bit of NLP comparing Topic clustering in an unsupervised problem. Details below in the Jupyter Notebook.
Natural Language Processing in Python¶
LDA vs NMF Topic clustering¶
A quick no-nonsense overview of utilizing LDA (Latent Dirichlet Allocation) vs. NMF (Non-Negative Matrix Factorization) for unsupervised topic clustering problems in NLP.
The dataset I'm using is located here, and contains a large set of scraped news-stories from NPR (mostly political).
Through these two examples I will categorize each article into one of 8 topics that are defined through LDA/NMF.
LDA¶
Note that for LDA we need to utilize the CountVectorizer instead of TF-IDF as LDA relies on per word-count vectorization.
In [1]:
# Imports
import numpy as np
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation
In [2]:
# Data
npr = pd.read_csv('Datasets/npr.csv')
In [3]:
npr.head()
Out[3]:
In [4]:
npr.shape
Out[4]:
Form Vector Matrices¶
In [5]:
# Instantial CV object
# Words show up in a maximum of 90% of the documents in NPR
# Words MUST show up in a minimum of 2 documents
# Eliminate English stop-words
cv = CountVectorizer(max_df=0.9,min_df=2,stop_words='english')
In [6]:
# Create Document Term Matrix - fit transform to Article text in DataFrame
# Number of articles by words
dtm = cv.fit_transform(npr['Article'])
dtm.shape
Out[6]:
In [7]:
# Instantiate LDA, form 8 Topics
LDA = LatentDirichletAllocation(n_components=8,random_state=42)
In [8]:
# Fit to dtm (may take a while)
LDA.fit(dtm)
Out[8]:
Categorize Articles into Topics¶
cv --> contains vocabulary list of all words that were in all the documents
LDA.components_ --> 8 topics with array of 'probabilities' that a word exists in a specific topic
In [9]:
# List of words in cv (picked 1500 index at random)
cv.get_feature_names()[1500]
Out[9]:
In [10]:
# 8 Topics, with index positions of every word
LDA.components_.shape
Out[10]:
In [11]:
# Array of index positions, ranked least to greatest
# The greatest values are most likely to be related to the specified topic
# Code below will give the 10 most likely records to be related to Topic '0' (defined as components_[0])
top10 = LDA.components_[0].argsort()[-10:]
top10
Out[11]:
In [12]:
# Map the top 10 above for Topic 0 to English words
# These words are clustered into Topic 0
for i in top10:
print(cv.get_feature_names()[i])
In [13]:
# Print out the top 10 words for each of the 8 topics
for i,topic in enumerate(LDA.components_):
print(f"THE TOP 10 WORDS FOR TOPIC #{i}")
print([cv.get_feature_names()[index] for index in topic.argsort()[-10:]])
print("\n")
print("\n")
In [14]:
# Array of percentage of probabilities that document belongs to a topic
# We can see that the first document is most likely to be associated with Topic 1 at 90% (indexed at zero)
topic_results = LDA.transform(dtm)
topic_results[0].round(2)
Out[14]:
In [15]:
# Add Topic number to NPR DataFrame to associate story with a Topic
npr['Topic'] = topic_results.argmax(axis=1)
npr.head()
Out[15]:
NMF¶
Utilizing TF-IDF instead of CountVectorizer
Follows nearly the exact same pattern
In [16]:
# Imports
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import NMF
In [17]:
# Instantial TF-IDF object
# Words show up in a maximum of 90% of the documents in NPR
# Words MUST show up in a minimum of 2 documents
# Eliminate English stop-words
tfidf = TfidfVectorizer(max_df=0.95,min_df=2,stop_words='english')
In [18]:
# Create Document Term Matrix - fit transform to Article text in DataFrame
# Number of articles by words
dtm = tfidf.fit_transform(npr['Article'])
dtm.shape
Out[18]:
In [19]:
# Instantiate NMF, form 8 Topics
nmf_model = NMF(n_components=8,random_state=42)
In [20]:
# Fit model
nmf_model.fit(dtm)
Out[20]:
Categorize Articles into Topics¶
tfidf --> contains vocabulary list of all words that were in all the documents
nmfmodel.components --> 8 topics with array of 'probabilities' that a word exists in a specific topic
In [21]:
# List of words in tfidf (picked 1500 index at random)
tfidf.get_feature_names()[1500]
Out[21]:
In [22]:
# 8 Topics, with index positions of every word
nmf_model.components_.shape
Out[22]:
In [23]:
# Array of index positions, ranked least to greatest
# The greatest values are most likely to be related to the specified topic
# Code below will give the 10 most likely records to be related to Topic '0' (defined as components_[0])
top10 = nmf_model.components_[0].argsort()[-10:]
top10
Out[23]:
In [24]:
# Map the top 10 above for Topic 0 to English words
# These words are clustered into Topic 0
for i in top10:
print(tfidf.get_feature_names()[i])
In [25]:
# Print out the top 10 words for each of the 8 topics
for i,topic in enumerate(nmf_model.components_):
print(f"THE TOP 10 WORDS FOR TOPIC #{i}")
print([cv.get_feature_names()[index] for index in topic.argsort()[-10:]])
print("\n")
print("\n")
In [26]:
# Array of percentage of probabilities that document belongs to a topic
# We can see that the first document is most likely to be associated with Topic 1 at 12% (indexed at zero)
topic_results = nmf_model.transform(dtm)
topic_results[0].round(2)
Out[26]:
In [27]:
# Add Topic number to NPR DataFrame to associate story with a Topic
npr['Topic'] = topic_results.argmax(axis=1)
npr.head()
Out[27]: