Light

the-gupta / ted-scraper Goto Github PK

View Code? Open in Web Editor NEW

11.0 1.0 4.0 249 KB

Complete Web Scraping of TED.com for Metadata, Transcript, Audio, Video, Images using Parallel Programming

License: MIT License

Jupyter Notebook 100.00%

ted ted-talk tedx ted-talks ted-dataset scraper scraping scraping-websites dataset scalability

ted-scraper's Introduction

TED-Scraper

Web Scraping of TED.com for complete Metadata, Transcript, Audio, Video, Images using Parallel Programming.

Environment: Google Colab with Google Drive without any Hardware Accelerator. Python: 3.6.9

Context

I was looking for an interesting dataset for a personal Data Science project, and I'm a fan of TED. So, I looked for the TED dataset, found Rounka's but it is incomplete and outdated. Then, I scraped myself and made it super fast using Parallel Programming. Now, it downloads all Metadata along with the Transcript in 300 seconds of all 4609 Talks on the website*. This is the most comprehensive TED Talk dataset which includes media files (images, audio, and video) too!

*Scraped on 24-JUN-20. One can scrape entire TED.com using the code to get the latest dataset in 5 minutes.

Downloading media files take less than 2 hours in total - 2 minutes for photos of Speaker and Talk, 10 minutes for Audio, 1.5 hours for videos.
TED_Talk.xlsx and TED_Talk.csv contain Metadata and Transcript. Folder Names are intuitive. All media files are named by talk__id, except in PHOTO__SPEAKER files are named by speaker__id of the primary Speaker.

The code shows a way to scrape at scale.

ted-scraper's People

Contributors

Stargazers

Watchers

Forkers

horiajurcut mavicmed yuyang0828 radhikabhrara

ted-scraper's Issues

Comment count

Hi @The-Gupta , Thank you for this great scraper. I used the code and was able to successfully run it on colab, although there is one problem in the final output. After running the code there is no value for any of the videos regarding the number of comments in the videos and for finding out if the structure of the TED webpage has changed or not, I scraped the content of one sample page and searched for the 'comment' word and I couldn't find any. Do you think there is any way I can actually get this number given the fact that apparently there is nothing related to comments in the contents of the page?

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.