[TOC]
I have received the test with request build two simple pages
A web page to select a playlist (in a dropdown list) and enter a URL of the page that contains video or media item to crawl
A web page to show the playlist listing (crawler result)
I think the first page description is not really clear. It took me a few hours to think about it.
I have two ways of thinking: 1. User select the playlist, then enter url to get media data and save to this playlist; 2. Or user can select the playlist to show all media in that, then click one media to get data.
After all, I decided to choose the first one. I note here for you to understand what I did.
I have prepared a short clip demo.
-
Open a terminal and run:
docker-compose up -d
-
Run composer and database migration and seeding
docker exec -it nct_app composer install docker exec -it nct_app php artisan migrate:fresh --seed
-
Visit http://0.0.0.0:8888 on your browser.
-
To stop and remove all the containers use thedown command:
docker-compose down
I choose PHP 7 with Lumen framework and some package:
cviebrock/eloquent-sluggable: Easy creation of slugs for your Eloquent models in Laravel
fabpot/goutte: Provides a nice API to crawl websites and extract data from the HTML/XML responses
barryvdh/laravel-debugbar: Displays a debug bar in the browser with information from Laravel/Lumen
I use this package to check my query (query times, n+1,...), runtime exception,..
Could we crawl hot video items from this source https://www.tiktok.com/vi/trending in a similar way? How and/or Why?
Yes.
First, we will get html source
from this page and get json string
from "itemList":
to first }}
, then decode json
to get list videos