borneo-bulletin-bot's Introduction

Welcome 👋

About Me

I'm a web developer with a degree in Computer Science, passionate on building beautiful and functional websites.
I specialize in using modern web technologies such as React, Tailwind, and Next.js to create fast, responsive, and user-friendly interfaces.
I'm also experienced in using data science libraries such as Pandas and Scikit-learn to extract insights and value from data.

Check out my website and blog!

I put a lot of effort into my personal and , where I share my experiences and showcase my projects. Please have a look!

Tech Stack

Interests

When I'm not coding, I enjoy watching anime, playing games, and learning Japanese.

Connect

Feel free to connect with me on

Whenever I got stuck doing programming, I just recall the existence of this image

borneo-bulletin-bot's People

Contributors

Stargazers

Watchers

borneo-bulletin-bot's Issues

Scraper breaks when article don't have image.

When a news article, for example, https://borneobulletin.com.bn/what-it-should-have-been/ don't have any image, it can't extract any image that it can use for discord 'embed' image thumbnail.

Scrape other sections eg. National, SE Asia, World, etc

This bot only scrapes headlines as of currently.

Make the bot scrapes other sections
https://borneobulletin.com.bn/category/national/
https://borneobulletin.com.bn/category/southeast/
https://borneobulletin.com.bn/category/world/
etc.
Add more discord commands based on the updated scraper

Add GPT-3.5 summarization

Use GPT-3.5 API to summarize the scraped news

Borneo Bulletin stops updating headline

Borneo Bulletin seems to stop updating its headline section. Need to scrape other sections instead eg. National.

related issue:
#4 (comment)

Optimize scraper for improved performance.

Currently, the scraper is using a manual open and close method for the Selenium driver for each new URL scraped. While this is a workaround to bypass Cloudflare, it can negatively impact performance by creating unnecessary overhead.
Additionally, many of the scraper functions are running synchronously in a loop, which can further slow down the scraping process. To improve performance, it may be beneficial to refactor the code to utilize asynchronous methods and reduce the number of synchronous loops.

Discord bot warning: "WARNING discord.gateway Shard ID None heartbeat blocked for more than 50 seconds.

I've been getting this warning message in my Discord bot's logs: "WARNING discord.gateway Shard ID None heartbeat blocked for more than 50 seconds." It seems to be related to running the scraper with async, but I'm not exactly sure how to fix it.

The bot itself works just fine, but this warning message is bothering me and I'd like to address it if possible. I suspect that I need to find a way to run the scraper asynchronously, but I'm not sure how to do that.

Any help or suggestions would be greatly appreciated. Thank you in advance!

2023-05-01 02:22:59 WARNING  discord.gateway Shard ID None heartbeat blocked for more than 50 seconds.
Loop thread traceback (most recent call last):
  File "/root/borneo-bulletin-bot/bot/bot.py", line 39, in <module>
    main()
  File "/root/borneo-bulletin-bot/bot/bot.py", line 35, in main
    client.run(TOKEN)
  File "/usr/local/lib/python3.10/dist-packages/discord/client.py", line 860, in run
    asyncio.run(runner())
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 633, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 600, in run_forever
    self._run_once()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1896, in _run_once
    handle._run()
  File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.10/dist-packages/discord/app_commands/tree.py", line 1089, in wrapper
    await self._call(interaction)
  File "/usr/local/lib/python3.10/dist-packages/discord/app_commands/tree.py", line 1248, in _call
    await command._invoke_with_namespace(interaction, namespace)
  File "/usr/local/lib/python3.10/dist-packages/discord/app_commands/commands.py", line 867, in _invoke_with_namespace
    return await self._do_call(interaction, transformed_values)
  File "/usr/local/lib/python3.10/dist-packages/discord/app_commands/commands.py", line 841, in _do_call
    return await self._callback(self.binding, interaction, **params)  # type: ignore
  File "/root/borneo-bulletin-bot/bot/cogs/cog1.py", line 31, in fetch_article
    responses = scraper.main()
  File "/root/borneo-bulletin-bot/bot/scraper/scraper.py", line 133, in main
    article_data.append(get_article_data(driver, link, download_image=True))
  File "/root/borneo-bulletin-bot/bot/scraper/scraper.py", line 65, in get_article_data
    driver.get(url)
  File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py", line 449, in get
    self.execute(Command.GET, {"url": url})
  File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py", line 438, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/remote_connection.py", line 290, in execute
    return self._request(command_info[0], url, body=data)
  File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/remote_connection.py", line 311, in _request
    response = self._conn.request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.10/dist-packages/urllib3/request.py", line 78, in request
    return self.request_encode_body(
  File "/usr/local/lib/python3.10/dist-packages/urllib3/request.py", line 170, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
  File "/usr/local/lib/python3.10/dist-packages/urllib3/poolmanager.py", line 376, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1374, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.10/socket.py", line 705, in readinto
    return self._sock.recv_into(b)

Recommend Projects