finaldie / auto-news Goto Github PK

Hi!

I love the project. I was looking to build something similar for myself and was very happy to found your work here.

I just noticed you made some really cool animations depicting the infrastructure workings of the project. I mean the animation in the video, but also saw similar animation in your profile.

How do you create such animations? Is there a tool to quickly generate these?

Thank you for your great work and sharing it open-source.
Cheers!

hi ,I want to install auto-news by docker compose.
so I check out the repo main branch and create environment file as wiki says, but when I run make deps && make deploy ,I have an error info:

cd docker && make deploy topdir=/home/ubuntu/auto-news build_dir=/home/ubuntu/auto-news/build
make[1]: Entering directory '/home/ubuntu/auto-news/docker'
Assembling Airflow ...
mkdir -p /home/ubuntu/auto-news/workspace/airflow
cd /home/ubuntu/auto-news/workspace/airflow && mkdir -p dags logs config plugins run data
cd /home/ubuntu/auto-news/workspace/airflow && chmod 777 run/ data/
Airflow deployment finished
Assembling Postgres ...
mkdir -p /home/ubuntu/auto-news/workspace/postgres/data
Postgres deployment finished
Assembling Redis ...
mkdir -p /home/ubuntu/auto-news/workspace/redis/data
Redis deployment finished
Assembling Mysql ...
mkdir -p /home/ubuntu/auto-news/workspace/mysql/data
Mysql deployment finished
Assembling Milvus ...
mkdir -p /home/ubuntu/auto-news/workspace/milvus/volumes
Milvus deployment finished
make[1]: Leaving directory '/home/ubuntu/auto-news/docker'
cp /home/ubuntu/auto-news/build/.env /home/ubuntu/auto-news/workspace/airflow/run/auto-news/src
cp: cannot create regular file '/home/ubuntu/auto-news/workspace/airflow/run/auto-news/src': No such file or directory
make: *** [Makefile:95: deploy-env] Error 1

I read the Makefile and found this
cp $(build_dir)/.env $(BOT_HOME)/src
where BOT_HOME from??

Hi, I've read all the documentation and this project seems really nice; I tried to set it up mon my computer but I got some errors during the docker build (about 5 min after it started building). But actually, I'm not sure I am doing it the right way :

• I am using a windows 11 computer with visual studio + docker desktop and WSL
• I first make the git clone on my computer
• I then copy and set up the .env file in the ./docker/ folder (because that's where I found the docker-compose.yaml file so I thought it was the right place to put it).
• and then, still in the docker folder, I run the cli : docker-compose up --build -> it pulls a lot of file, but crashes after 5 min (strange message about VM crashing)
  I do not have the opportunity to run the "make" command lines in the backend (and to be honnest, I was wondering where I should run those CLI anyway : in the docker Inguess ?)

Any help would be appreciated.

Thanks

你好，我是从git来的。我最近也在研究AI+NEWS方向，我加你微信了，方便聊聊，看看能否一起做事吗。感谢。

What would be a good starting point to integrate this using an API and for example display the result in React Native etc?

hi would it be possible for you to provide us with an example for an installation from portainer?? and moreover impossible to recover an API key from reddit since the changes occurred in 2023?

thank you your project looks nice

I wonder how to use ollama local model instead of "Cloud GPT"?

Hi,

Love your project. Would like to run it without interface, just Python based to better understand and maybe build extra features upon it. Can you advise howto do this?

Thanks,
S

i have problem to start it locally on mine macOS. The issue in container docker-airflow-init-user. And i can't see any notion tables crated.
Logs From Container:
2023-12-13 12:53:13 Initializing Notion database tables...
2023-12-13 12:53:13 Notion initialization started ...
2023-12-13 12:53:13 MySQL client initialization finished, host: mysql-db, port: 3306, user: bot
2023-12-13 12:53:13 loaded indexes: {'notion': {'entry_page_id': {'index_id': '', 'created_at': datetime.datetime(2023, 12, 12, 19, 31, 54), 'updated_at': datetime.datetime(2023, 12, 12, 19, 31, 54)}}}
2023-12-13 12:53:13 loaded notion indexes: {'entry_page_id': {'index_id': '', 'created_at': datetime.datetime(2023, 12, 12, 19, 31, 54), 'updated_at': datetime.datetime(2023, 12, 12, 19, 31, 54)}}
2023-12-13 12:53:13 [ERROR] Errors in creating notion pages: 1062 (23000): Duplicate entry 'notion-entry_page_id' for key 'index_pages.category_name'
2023-12-13 12:53:06 /home/airflow/.local/lib/python3.11/site-packages/langchain/init.py:34: UserWarning: Importing LLMChain from langchain root module is no longer supported. Please use langchain.chains.LLMChain instead.
2023-12-13 12:53:06 warnings.warn(
2023-12-13 12:53:13 Traceback (most recent call last):
2023-12-13 12:53:13 File "/home/airflow/.local/lib/python3.11/site-packages/mysql/connector/connection_cext.py", line 633, in cmd_query
2023-12-13 12:53:13 self._cmysql.query(
2023-12-13 12:53:13 _mysql_connector.MySQLInterfaceError: Duplicate entry 'notion-entry_page_id' for key 'index_pages.category_name'
2023-12-13 12:53:13
2023-12-13 12:53:13 The above exception was the direct cause of the following exception:
2023-12-13 12:53:13
2023-12-13 12:53:13 Traceback (most recent call last):
2023-12-13 12:53:13 File "/home/airflow/airflow/run/auto-news/src/patches.py", line 63, in
2023-12-13 12:53:13 apply_patches()
2023-12-13 12:53:13 File "/home/airflow/airflow/run/auto-news/src/patches.py", line 52, in apply_patches
2023-12-13 12:53:13 ok, msg = func()
2023-12-13 12:53:13 ^^^^^^
2023-12-13 12:53:13 File "/opt/airflow/run/auto-news/src/patch_0.py", line 14, in apply
2023-12-13 12:53:13 notion_init.init()
2023-12-13 12:53:13 File "/opt/airflow/run/auto-news/src/notion_init.py", line 8, in init
2023-12-13 12:53:13 op.init()
2023-12-13 12:53:13 File "/opt/airflow/run/auto-news/src/ops_notion.py", line 30, in init
2023-12-13 12:53:13 db_cli.index_pages_table_insert(
2023-12-13 12:53:13 File "/opt/airflow/run/auto-news/src/mysql_cli.py", line 110, in index_pages_table_insert
2023-12-13 12:53:13 c.execute(sql, val)
2023-12-13 12:53:13 File "/home/airflow/.local/lib/python3.11/site-packages/mysql/connector/cursor_cext.py", line 330, in execute
2023-12-13 12:53:13 result = self._cnx.cmd_query(
2023-12-13 12:53:13 ^^^^^^^^^^^^^^^^^^^^
2023-12-13 12:53:13 File "/home/airflow/.local/lib/python3.11/site-packages/mysql/connector/opentelemetry/context_propagation.py", line 86, in wrapper
2023-12-13 12:53:13 result = method(cnx, *args, **kwargs)
2023-12-13 12:53:13 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-13 12:53:13 File "/home/airflow/.local/lib/python3.11/site-packages/mysql/connector/connection_cext.py", line 641, in cmd_query
2023-12-13 12:53:13 raise get_mysql_exception(
2023-12-13 12:53:13 mysql.connector.errors.IntegrityError: 1062 (23000): Duplicate entry 'notion-entry_page_id' for key 'index_pages.category_name'
2023-12-13 12:53:14 Apply patches finished
2023-12-13 12:53:14 ----
2023-12-13 12:53:19 Enable DAGs...
2023-12-13 12:53:24 Dag: news_pulling, paused: False
2023-12-13 12:53:27 Dag: sync_dist, paused: False
2023-12-13 12:53:31 Dag: collection_weekly, paused: False
2023-12-13 12:53:34 Dag: upgrade, paused: False
2023-12-13 12:53:37 Dag: journal_daily, paused: False
2023-12-13 12:53:41 Dag: action, paused: False
2023-12-13 12:53:41 Enable DAGs finished

Great project.

Do you have an estimate about the billing usage of OpenAI Reddit and Twitter API?

I am continually running into issues upon initial startup.

startup process from clean install:
sudo rm -rf auto-news
git clone https://github.com/finaldie/auto-news/
cd auto-news
code . (update env variables)
make deps && make build && make deploy && make init
make start

The issue is in the container docker-airflow-init-user. No pages are created within notion. The notion page is blank aside from a title 'reader'.

2024-01-19 12:38:11 httpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://api.notion.com/v1/pages'
2024-01-19 12:38:11 For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400
2024-01-19 12:38:11
2024-01-19 12:38:11 During handling of the above exception, another exception occurred:
2024-01-19 12:38:11
2024-01-19 12:38:11 Traceback (most recent call last):
2024-01-19 12:38:11 File "/home/airflow/airflow/run/auto-news/src/patches.py", line 63, in
2024-01-19 12:38:11 apply_patches()
2024-01-19 12:38:11 File "/home/airflow/airflow/run/auto-news/src/patches.py", line 52, in apply_patches
2024-01-19 12:38:11 ok, msg = func()
2024-01-19 12:38:11 ^^^^^^
2024-01-19 12:38:11 File "/opt/airflow/run/auto-news/src/patch_0.py", line 14, in apply
2024-01-19 12:38:11 notion_init.init()
2024-01-19 12:38:11 File "/opt/airflow/run/auto-news/src/notion_init.py", line 8, in init
2024-01-19 12:38:11 op.init()
2024-01-19 12:38:11 File "/opt/airflow/run/auto-news/src/ops_notion.py", line 34, in init
2024-01-19 12:38:11 inbox_page = agent.createPageOfPage(entry_page_id, "Inbox")
2024-01-19 12:38:11 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-19 12:38:11 File "/opt/airflow/run/auto-news/src/notion.py", line 1921, in createPageOfPage
2024-01-19 12:38:11 return self.createPage(parent, props, None)
2024-01-19 12:38:11 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-19 12:38:11 File "/opt/airflow/run/auto-news/src/notion.py", line 1924, in createPage
2024-01-19 12:38:11 new_page = self.api.pages.create(
2024-01-19 12:38:11 ^^^^^^^^^^^^^^^^^^^^^^
2024-01-19 12:38:11 File "/home/airflow/.local/lib/python3.11/site-packages/notion_client/api_endpoints.py", line 208, in create
2024-01-19 12:38:11 return self.parent.request(
2024-01-19 12:38:11 ^^^^^^^^^^^^^^^^^^^^
2024-01-19 12:38:11 File "/home/airflow/.local/lib/python3.11/site-packages/notion_client/client.py", line 194, in request
2024-01-19 12:38:11 return self._parse_response(response)
2024-01-19 12:38:11 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-19 12:38:11 File "/home/airflow/.local/lib/python3.11/site-packages/notion_client/client.py", line 126, in _parse_response
2024-01-19 12:38:11 raise APIResponseError(response, body["message"], code)
2024-01-19 12:38:11 notion_client.errors.APIResponseError: body failed validation: body.children should be an array or undefined, instead was null.

Hi, what a great project!

I have Windows desktop and Macbook, but I want to try your project.
Does this project support mac?

I'm trying to figure out what it means, that is, what am I supposed to put into the journals? I've tried various methods, including dropping in articles, article titles, and the outcome is always different. Also, when I inserted an entire article into a journal, it generated a lot of TODOs, and I'm not entirely sure of their purpose. It would be great to get some clarification, thanks.

I want to develop a feature that would automatically research a specified topic, add it to articles, YouTube and other ToRead, and Lists, and then run dags as usual. According to your UML, sources should be provided by some search algorithm or API in addition you to a user.

Additionally, I might want to explore a particular topic; I would write this topic Title to notion page "Inbox - Research", and it would prepare a summary based on information collected across search engine.

Another suggestion is language detection. I've noticed it sometimes translates from the target language back to the target language. This issue should be addressed.

An improvement to include would be the detection of language translation, specifically preventing translations from and to the same language, to optimize costs.

I'm ready to assist with the "auto-news" project, though I haven't fully immersed myself in it yet. I'm eager to learn quickly, as the tool has already proven immensely useful to me. I aim to eliminate the tedious process of opening 20-30 sources on Google, transferring them to notion, and then waiting for summaries. I'd appreciate guidance on where to start to contribute effectively.

when running 'make deps && make build && make deploy && make init' getting the following error:

#25 36.85 ERROR: Cannot install -r requirements.txt (line 29) and -r requirements.txt (line 30) because these package versions have conflicting dependencies.
#25 36.85 
#25 36.85 The conflict is caused by:
#25 36.85     pymilvus 2.2.8 depends on grpcio<=1.53.0 and >=1.49.1
#25 36.85     chromadb 0.4.24 depends on grpcio>=1.58.0
#25 36.85     pymilvus 2.2.8 depends on grpcio<=1.53.0 and >=1.49.1
#25 36.85     chromadb 0.4.23 depends on grpcio>=1.58.0
#25 36.85     pymilvus 2.2.8 depends on grpcio<=1.53.0 and >=1.49.1
#25 36.85     chromadb 0.4.22 depends on grpcio>=1.58.0

suggestion: using pymilvus>=2.2.* solved that for me