burnysc2 / monorepo Goto Github PK

Traceback (most recent call last):
  File "/app/src/telegram_downloader.py", line 583, in <module>
    asyncio.run(main(), debug=False)
  File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/app/src/telegram_downloader.py", line 556, in main
    await parse_channel_messages()
  File "/app/src/telegram_downloader.py", line 428, in parse_channel_messages
    async for message in messages_iter:
  File "/root/.cache/pypoetry/virtualenvs/transcriber-backend-9TtSrW0h-py3.10/lib/python3.10/site-packages/pyrogram/methods/messages/get_chat_history.py", line 101, in get_chat_history
    messages = await get_chunk(
  File "/root/.cache/pypoetry/virtualenvs/transcriber-backend-9TtSrW0h-py3.10/lib/python3.10/site-packages/pyrogram/methods/messages/get_chat_history.py", line 49, in get_chunk
    return await utils.parse_messages(client, messages, replies=0)
  File "/root/.cache/pypoetry/virtualenvs/transcriber-backend-9TtSrW0h-py3.10/lib/python3.10/site-packages/pyrogram/utils.py", line 101, in parse_messages
    parsed_messages.append(await types.Message._parse(client, message, users, chats, replies=0))
  File "/root/.cache/pypoetry/virtualenvs/transcriber-backend-9TtSrW0h-py3.10/lib/python3.10/site-packages/pyrogram/types/messages_and_media/message.py", line 590, in _parse
    parsed_message.pinned_message = await client.get_messages(
  File "/root/.cache/pypoetry/virtualenvs/transcriber-backend-9TtSrW0h-py3.10/lib/python3.10/site-packages/pyrogram/methods/messages/get_messages.py", line 115, in get_messages
    r = await self.invoke(rpc, sleep_threshold=-1)
...
  File "/root/.cache/pypoetry/virtualenvs/transcriber-backend-9TtSrW0h-py3.10/lib/python3.10/site-packages/pyrogram/connection/transport/tcp/tcp_abridged_o.py", line 61, in send
    payload = await self.loop.run_in_executor(pyrogram.crypto_executor, aes.ctr256_encrypt, data, *self.encrypt)
TypeError: Value after * must be an iterable, not NoneType

Except / ignore the TypeError in

I decided to switch to htmx to make my life easier as a backend dev. This means giving up on frontend frameworks and doing server side rendering instead.

Add queries as described in https://zaidesanton.substack.com/p/the-most-underrated-skill-sql-for

Affected folders: https://github.com/BurnySc2/monorepo/tree/develop/transcribe_website/transcriber_backend

Idea:
https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor
https://github.com/BurnySc2/monorepo/blob/daab0f49399bd98bb8ea9b1a15877dcf4b7703fb/python_examples/examples/other/multithreading_example_02.py

Why?
Easier to read and use than the asyncio syntax.
Instead of async for ... we just use for loops
Instead of async with semaphore we just use with semaphore
Instead of await function() we just call the function
Instead of asyncio.create_task(func) we just submit a new function call to the executor, which will only leave the with statement when all tasks are completed.
Should be easier to test/mock functions

Location of ORM: https://github.com/BurnySc2/monorepo/tree/develop/transcribe_website/transcriber_backend/src/models

Merge https://github.com/BurnySc2/SC2-Replay-Renamer into fastapi-server to create a .zip with renamed replays.
Save settings in frontend (as cookie?)

From a discussion I had:

• Resource collection (shows late worker transfers or late expanding)
• Comparison army lost vs active army
• Comparison between building-start-time, or building count, e.g. when was 6th gateway added
• Comparison between upgrade-finished-time (chrono boost)
• Spending (Econ, Tech, Army)
• Comparison of unit count
• Save function for ideal replays in local storage if possible

Add a README.md file for each project and list the following things:

• what is this project for
• how to start and run the project
• how to maintain/update the project
• how is it structured, what tools does it use or depend on

This repo contains match info and build order overlays
Rewrite them to use htmx to connect to the SC2 API

It was attempted to rewrite them in svelte here https://github.com/BurnySc2/monorepo/blob/7ba9ea45b71944ac97bba559e874854ec98ce93c/supabase_stream_scripts/src/pages/BuildOrder.svelte

Make them available for:

• StarCraft 2
• AOE4
• ZeroSpace
• Stormgate

Add multiple modules:

• Build order overlay (Choice with chat interaction?)
• Match info
• Match history
• Betting system with Chat
• Title updater

https://www.pyqtgraph.org/

Add https://www.psycopg.org/psycopg3/docs/basic/index.html to https://github.com/BurnySc2/monorepo/tree/develop/python_examples/examples/databases

Add rust example code and how to ship it to pypi as a package.
Use pyo3 and maturin according to this video
Idea: keep a rust and python source code file which are identically to be able to maintain the rust code when you only know python

Edit:
Use nim instead - seems much easier
https://github.com/Pebaz/nimporter
https://github.com/yglukhov/nimpy

Use this library https://pypi.org/project/fuzzywuzzy/#description

https://nim-lang.org/
How to set up a nim backend webserver?
How to add nim to frontend?

Also call nim code from Python https://github.com/yglukhov/nimpy

Also need to dockerize the service, and need to pull data from supabase db
https://github.com/BurnySc2/twitch-stream-announcer-webhook

Also need to dockerize the service, and need to pull data from supabase db
https://github.com/BurnySc2/twitch-sc2-irc-bot

Use https://github.com/coqui-ai/TTS to generate .mp3 files from texts, being able to convert a pdf into one or multiple .mp3 files
Will probably require frontend+backend

Change the following (and other lines):

Join irc chat anonymously instead with justinfanXXX where XXX is a number
https://discuss.dev.twitch.com/t/anonymous-connection-using-justinfan/2046
https://gist.github.com/skntxyz/833dd97620d44c345520
https://discuss.dev.twitch.com/t/bot-cant-send-messages-to-other-channel/30021/6

In bored_gems project I plan on using supabase. However I need to check if my application works but implementing some kind of integration tests, which means:

• start dev server (or build and "preview")
• go to locally hosted url
• registering a user via email (random username and password so it doesnt collide with existing users in db)
• logging in with that user
• creating a game in the game lobby
• repeat with 2nd user, join the game lobby
• play a simple game e.g. tic tac toe
• register/notice that the game has completed
• mark test as success

This way I can assure that at least basics of the application still works. However this poses a few issues:

• How to ensure that the test doesnt fail because of a username clash? Catch error on registering
• How to ensure that the supabase database models are up to date? Use prisma to get type safety and intellisense for the db model. How to integrate this into the supabase client?
• How to ensure that tests does not collide with production data? Add a new column in db with "DEV=true" for tests, and "DEV=false" for prod
• How to make sure that Dev is unable to access Prod database? Does RLS help here?

https://mermaid.js.org/intro/

Tasks:

• discord_bot commands
  • mmr command
  • reminder command
  • leaderboard command
  • twss command
  • emotes command
  • fetch aoe4 command
• burny_common library
  • available functions
• fastapi_server available routes and their purposes
  • todolist
  • chat
  • text to speech (twitch)
  • audiobook (behind logged in users)
  • TODO add transcribe site
• transcribe website
  • Upload video/audio to be queued to be transcribed, queue job (which language, which model to use to transcrible)
  • Download new video with hard-subs
  • Download .srt or other format file to have a separate subtitle file
  • Transcribe worker, how it works and interacts with db

Use Pony asyncpg (raw SQL queries) to access the supabase db directly instead of using the supabase library.
Alternatively if async/await syntax is prefered, use one the support postgresql and async/await from https://github.com/BurnySc2/monorepo/tree/develop/python_examples/examples/databases

Add Tortoise ORM example to https://github.com/BurnySc2/monorepo/tree/develop/python_examples/examples/databases to check out the syntax

Add to Earthfile on pre-commit a SQL query validation and formatting tool:

https://docs.sqlfluff.com/en/stable/
or
https://pypi.org/project/sqlvalidator/#description

Location: https://github.com/BurnySc2/monorepo/blob/develop/python_examples/examples/databases/sqlite_example.py

E.g.:
http://peter-hoffmann.com/2010/python-sqlite-namedtuple-factory.html
https://stackoverflow.com/a/3300514
Try if I can use psycopg's namedtuple_rowmaker

See also
https://docs.python.org/3/library/sqlite3.html
def namedtuple_factory(cursor, row):

Because I use supabase as a globally hosted database, and discord bot to access the database, I may as well use it to frequently run checks on the database data?

Alternatively: Find a cheap domain host to host the backend again. This requires me to update the certs every year though

Where to find models for it?