ispirt / sakram-analytics Goto Github PK

How does mailer.py work? It seems to require we create a dumps/<today's-date> directory. should we put something into this directory? where is the database running? seems like the database has 0 tweets for today and so it is failing.

Every few hours dns from digital ocean to api.twitter.com keeps failing. this causes the twitter analytics program to raise exception and die.
Identifying the dns issue is key but the problem can be mitigated quickly by catching the exception and trying again in an hour.

[01/06/2022 12:37:02 PM UTC] request: [DEBUG] Making API request: GET https://api.twitter.com/2/tweets/search/recent
Parameters: {'max_results': 100, 'expansions': 'attachments.media_keys,author_id,referenced_tweets.id,entities.mentions.username,geo.place_id,in_reply_to_user_id,referenced_tweets.id.author_id', 'media.fields': 'type,url', 'user.fields': 'name,id,username', 'tweet.fields': 'text,attachments,author_id,geo,id,lang,referenced_tweets,public_metrics,created_at', 'query': '@cyberdost'}
Headers: {'User-Agent': 'Python/3.8.10 Requests/2.27.1 Tweepy/4.6.0', 'Authorization': 'Bearer AAAAAAAAAAAAAAAAAAAAAHIPWgEAAAAArnzwQwkoY4vhE0HtkgbkQVGRV4w%3Dzj0EX3wlxPAybrkRMbEqKt4iMR2iR21iDeIGcF7EaVjewh8iCj'}
Body: None
[01/06/2022 12:37:18 PM UTC] _new_conn: [DEBUG] Starting new HTTPS connection (1): api.twitter.com:443
Traceback (most recent call last):
File "/home/dhruva/.virtualenvs/twitter_feed/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/home/dhruva/.virtualenvs/twitter_feed/lib/python3.8/site-packages/urllib3/util/connection.py", line 72, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/lib/python3.8/socket.py", line 918, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/dhruva/.virtualenvs/twitter_feed/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/dhruva/.virtualenvs/twitter_feed/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/home/dhruva/.virtualenvs/twitter_feed/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
conn.connect()
File "/home/dhruva/.virtualenvs/twitter_feed/lib/python3.8/site-packages/urllib3/connection.py", line 358, in connect
conn = self._new_conn()
File "/home/dhruva/.virtualenvs/twitter_feed/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f96b4830070>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/dhruva/.virtualenvs/twitter_feed/lib/python3.8/site-packages/requests/adapters.py", line 440, in send
resp = conn.urlopen(
File "/home/dhruva/.virtualenvs/twitter_feed/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen
retries = retries.increment(
File "/home/dhruva/.virtualenvs/twitter_feed/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.twitter.com', port=443): Max retries exceeded with url: /2/tweets/search/recent?max_results=100&expansions=attachments.media_keys%2Cauthor_id%2Creferenced_tweets.id%2Centities.mentions.username%2Cgeo.place_id%2Cin_reply_to_user_id%2Creferenced_tweets.id.author_id&media.fields=type%2Curl&user.fields=name%2Cid%2Cusername&tweet.fields=text%2Cattachments%2Cauthor_id%2Cgeo%2Cid%2Clang%2Creferenced_tweets%2Cpublic_metrics%2Ccreated_at&query=%40Cyberdost (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f96b4830070>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "main.py", line 42, in
main()
File "main.py", line 38, in main
do_main(args)
File "/home/webapp/sakram-analytics/extraction.py", line 23, in do_main
start_extraction(args)
File "/home/webapp/sakram-analytics/extraction.py", line 16, in start_extraction
tweets, media, users = fetch_tweets(bearer_token)
File "/home/webapp/sakram-analytics/twitter_.py", line 58, in fetch_tweets
tweets = client.search_recent_tweets(
File "/home/dhruva/.virtualenvs/twitter_feed/lib/python3.8/site-packages/tweepy/client.py", line 907, in search_recent_tweets
return self._make_request(
File "/home/dhruva/.virtualenvs/twitter_feed/lib/python3.8/site-packages/tweepy/client.py", line 118, in _make_request
response = self.request(method, route, params=request_params,
File "/home/dhruva/.virtualenvs/twitter_feed/lib/python3.8/site-packages/tweepy/client.py", line 76, in request
with self.session.request(
File "/home/dhruva/.virtualenvs/twitter_feed/lib/python3.8/site-packages/requests/sessions.py", line 529, in request
resp = self.send(prep, **send_kwargs)
File "/home/dhruva/.virtualenvs/twitter_feed/lib/python3.8/site-packages/requests/sessions.py", line 645, in send
r = adapter.send(request, **kwargs)
File "/home/dhruva/.virtualenvs/twitter_feed/lib/python3.8/site-packages/requests/adapters.py", line 519, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.twitter.com', port=443): Max retries exceeded with url: /2/tweets/search/recent?max_results=100&expansions=attachments.media_keys%2Cauthor_id%2Creferenced_tweets.id%2Centities.mentions.username%2Cgeo.place_id%2Cin_reply_to_user_id%2Creferenced_tweets.id.author_id&media.fields=type%2Curl&user.fields=name%2Cid%2Cusername&tweet.fields=text%2Cattachments%2Cauthor_id%2Cgeo%2Cid%2Clang%2Creferenced_tweets%2Cpublic_metrics%2Ccreated_at&query=%40Cyberdost (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f96b4830070>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))