Incorrect category for CNN about url-categorization-using-machine-learning HOT 2 CLOSED

Hey!

That's the very deep question that has more than one possible answers:

1. CNN home website may contain words that could make an impact for wrong category classification.
2. Most frequent word list could be outdated or need to be improved.

The solution to these answers could be different, but the first thing that comes to my mind is to expand words and get more information about websites as possible. One of the solution may be to not scrape only home website page, but also deeper levels of their content(recursively go to website hyperlinks). That would make more features for website when counting the most frequent words and get weights from it.

More information may lead to the more accurate model, so it is possible to do recursive scrapping not only for CNN, but for all training set websites then the most frequent models also would be more accurate. That would cost more time, but results of the model could be much better.

However, I did not implemented recursive scrapping since it could be more complex and this project was done for academic research purposes so recursive scrapping implementation needs to be done. Maybe someday when I got more free time, I could try to implement this feature so for now it's in my to-do plan list in the future.

from url-categorization-using-machine-learning.