Functions for parsing wikipedia pages of categoryes types. You can get a dictionary wich contains {"text of link":"url behind the link"}.
We will show on page https://en.wikipedia.org/wiki/Category:Cities_in_the_United_States
from wiki_cat_parser import *
import pandas as pdSub categoryes has a triangle marker close to them:
pg_parsing_result = read_wiki_subcats(
"https://en.wikipedia.org/wiki/Category:Cities_in_the_United_States")
pd.DataFrame({'url':pg_parsing_result.values()},
index = pg_parsing_result.keys())| url | |
|---|---|
| Cities in the United States by county | /wiki/Category:Cities_in_the_United_States_by_... |
| Cities in the United States by metropolitan area | /wiki/Category:Cities_in_the_United_States_by_... |
| Cities in the United States by state | /wiki/Category:Cities_in_the_United_States_by_... |
| Lists by city in the United States | /wiki/Category:Lists_by_city_in_the_United_States |
| Lists of cities in the United States | /wiki/Category:Lists_of_cities_in_the_United_S... |
| Cities of Appalachia | /wiki/Category:Cities_of_Appalachia |
| Capitals of political divisions in the United States | /wiki/Category:Capitals_of_political_divisions... |
| Categories by city in the United States | /wiki/Category:Categories_by_city_in_the_Unite... |
| Consolidated city-counties | /wiki/Category:Consolidated_city-counties |
| Independent cities in the United States | /wiki/Category:Independent_cities_in_the_Unite... |
| Majority-minority cities and towns in the United States | /wiki/Category:Majority-minority_cities_and_to... |
| Cities in the Mojave Desert | /wiki/Category:Cities_in_the_Mojave_Desert |
| Planned cities in the United States | /wiki/Category:Planned_cities_in_the_United_St... |
| Pre-historic cities in the United States | /wiki/Category:Pre-historic_cities_in_the_Unit... |
| Urban decay in the United States | /wiki/Category:Urban_decay_in_the_United_States |
| Urban politics in the United States | /wiki/Category:Urban_politics_in_the_United_St... |
| Works about cities in the United States | /wiki/Category:Works_about_cities_in_the_Unite... |
| Images of cities in the United States | /wiki/Category:Images_of_cities_in_the_United_... |
Categories has circle marker close to them:
read_wiki_cats(
"https://en.wikipedia.org/wiki/Category:Cities_in_the_United_States")
pd.DataFrame({'url':pg_parsing_result.values()},
index = pg_parsing_result.keys())| url | |
|---|---|
| Cities in the United States by county | /wiki/Category:Cities_in_the_United_States_by_... |
| Cities in the United States by metropolitan area | /wiki/Category:Cities_in_the_United_States_by_... |
| Cities in the United States by state | /wiki/Category:Cities_in_the_United_States_by_... |
| Lists by city in the United States | /wiki/Category:Lists_by_city_in_the_United_States |
| Lists of cities in the United States | /wiki/Category:Lists_of_cities_in_the_United_S... |
| Cities of Appalachia | /wiki/Category:Cities_of_Appalachia |
| Capitals of political divisions in the United States | /wiki/Category:Capitals_of_political_divisions... |
| Categories by city in the United States | /wiki/Category:Categories_by_city_in_the_Unite... |
| Consolidated city-counties | /wiki/Category:Consolidated_city-counties |
| Independent cities in the United States | /wiki/Category:Independent_cities_in_the_Unite... |
| Majority-minority cities and towns in the United States | /wiki/Category:Majority-minority_cities_and_to... |
| Cities in the Mojave Desert | /wiki/Category:Cities_in_the_Mojave_Desert |
| Planned cities in the United States | /wiki/Category:Planned_cities_in_the_United_St... |
| Pre-historic cities in the United States | /wiki/Category:Pre-historic_cities_in_the_Unit... |
| Urban decay in the United States | /wiki/Category:Urban_decay_in_the_United_States |
| Urban politics in the United States | /wiki/Category:Urban_politics_in_the_United_St... |
| Works about cities in the United States | /wiki/Category:Works_about_cities_in_the_Unite... |
| Images of cities in the United States | /wiki/Category:Images_of_cities_in_the_United_... |
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent