glw_weather's Introduction

glw_weather

Python 3: Scrape Weather Data from HTML: urllib3, BeautifulSoup, terminaltables

Two versions of a scraping script for tabulated weather data from a Davis® Weatherlink Web page. So, if that Web page HTML changes, the scripts can easily break. The scripts have been created for testing and personal use. The following modules have been imported into one or both scripts:

urllib3 is the HTTP client, including PoolManager which can pool multiple servers.
BeautifulSoup functions as a Python wrapper for an HTML parser for traversing, searching and changing the parsed tree.
AsciiTable uses - for horizontal lines, | for vertical lines and + for intersections to construct tabular data grids.

Variables

http = urllib3.PoolManager(): urllib3's PoolManager() handles arbitrary server requests.
req = http.request("GET", "http://www.weatherlink.com/user/gooselakewx/index.php?view=summary&headers=0"): the GET reqeust.

if req.status == 200:
    blob = req.data
else:
    print("Check req.status")

the conditional statement above checks for OK connection status, or status 200.
soup = BeautifulSoup(blob, "html.parser"): for parsing HTML for text from elements. in this repo, most of which were <td> elements from tabular data.
f'{soup.select("td:nth-of-type(14)")[0].string}': in this f-string, the select() function is used to target a specific <td> element for its text .string.

table = AsciiTable(main_data)
r_table = AsciiTable(rainfall)
s_table = AsciiTable(soil)

print(table.table)
print(r_table.table)
print(s_table.table)

in the variables defined above, AsciiTable() class gets the data arrays and formats them into the three tables shown below:

glw_weather's People

Contributors

Watchers

glw_weather's Issues

Incorrect value

Noticed an incorrect temperature value. Also heat index, wind chill, dew point, and there may be others.

NameError: name 'blob' is not defined

since the request failed, blob = req.data returned a NameError, but such is the life of a Web scraper. Both glw_weather_1.py and glw_weather_2.py will be affected.

"IndexError: list index out of range"

Traceback (most recent call last):
  File "/usr/local/sbin/glw2", line 79, in <module>
    "\n\033[32m     Temp 1:\033[0m", soup.select("td:nth-of-type(155)")[0].string, "\033[31m|\033[0m",
IndexError: list index out of range

The 155th <td> container had the first soil temperature reading, and the rain data displayed correctly, so the soil temp data was possibly withheld. When I checked again, later this evening, the soil temps displayed correctly.

MIght add a try clause to handle IndexError, but not sure if missing soil temp data was actually the cause.

Recommend Projects