Hi @koheiw ,
I have two issues that I hope to hear from you some advice and suggestion.
I - tokens or tokens_ngrams
Just to know and do my case correctly. Please give me any advice or suggestion.
I have a list of guided keywords which are both uni and n-grams.
So, when tokenizing my corpus, I am not sure about using tokens or tokens_ngrams is the most suitable one.
Q1: What should I do if I expect
- Input: both seeded uni and bi-grams. Output: newly explored keywords are both uni and bi-grams.
- Input: both seeded uni and bi-grams. Output: newly explored keywords are only uni grams?
Q2: If I have n-grams, should I use tokens_compound
?
Q3: do I need to remove stop words when using n-grams?
Please see my example code below.
# load the required packages
library(quanteda)
library(seededlda)
# Create a corpus from your text data
my_corpus <- corpus(c("The field of artificial intelligence is expanding with new breakthroughs.",
"Data analysis and machine learning are important in extracting insights.",
"Cybersecurity measures are crucial to protect sensitive information.",
"Health and fitness play a significant role in overall well-being.",
"The economy is affected by global market trends and trade policies.",
"Education is essential for personal growth and career opportunities.",
"Climate change and environmental sustainability are pressing issues.",
"Social media platforms are shaping communication and information sharing.",
"The entertainment industry is evolving with new digital platforms.",
"Urbanization and infrastructure development are transforming cities."))
# Define the seed words for each topic
seed_list <- list(topic1 = c("artificial intelligence", "machine learning", "data analysis"),
topic2 = c("cybersecurity", "privacy", "data protection"),
topic3 = c("climate change", "sustainability", "environmental impact"))
dict <- quanteda::dictionary(seed_list)
# Create a document-feature matrix with unigram and bigram features
toks <- tokens(my_corpus, remove_punct = TRUE, remove_symbols = TRUE, remove_number = TRUE) |>
tokens_select(min_nchar = 2) |>
tokens() |> ## I'M NOT SURE HERE ###
#tokens_ngrams(n = 1:2) |> ## I'M NOT SURE HERE ###
tokens_compound(dict) # for multi-word expressions
dfmt <- dfm(toks) |>
dfm_remove(stopwords('en')) |>
dfm_trim(min_termfreq = 0.5, termfreq_type = "quantile",
max_docfreq = 0.5, docfreq_type = "prop")
# Run seeded LDA
slda <- textmodel_seededlda(dfmt, dict, residual = TRUE)
# extract topic classification for each document
topics <- topics(slda)
# get most frequent keywords of each topic
keywords <- terms(slda, 20) |> as_tibble()
II - Use LDAvis to visualize and modify keywords, topics
As I need to do several rounds of seeded LDA to make the topics are separated clearly and combine topics if they are quite close to each other. I would like to use LDAvis to see more intuitively. I tried and got the error as below:
> # Create the interactive visualization using LDAvis
> lda_vis <- createJSON(phi = slda$phi,
+ theta = slda$theta,
+ doc_lengths = slda$doc_lengths,
+ vocab = slda$vocab,
+ term_frequency = slda$term_frequency)
Error in createJSON(phi = slda$phi, theta = slda$theta, doc_lengths = slda$doc_lengths, :
Length of doc.length not equal
to the number of rows in theta; both should be equal to the number of
documents in the data.
Q4: Is it because LDAvis does not work with seededlda output? If it works, what should I do? Is there any alternative visualization tools?
Thank you so much for your time and consideration.
Best,
HHN