morphy2k / rss-forwarder Goto Github PK

Hello!

Thanks for this awesome utility.

Is it possible to specify a default Sink URL? Like defining it at the top of config file instead of repeating it for 10-20 feeds?

Thank you.

By default it seems reqwest doesn't specify a correct Accept header, therefore some feeds return the incorrect format.

Example / reproduce:

• Use url = "https://github.com/onedr0p/flux-cluster-template/commits/main/.atom"
• It will fail because it returns a JSON format (blame github).

Possible fix:

• Setting .header(reqwest::header::ACCEPT, "application/atom+xml") on the request will return xml properly, yes this probably won't work with other feed types.

My rust isn't good enough for a full PR, but hopefully this might help or steer in the right direction.

Is it possible to add a new sink for RocketChat please?

RocketChat webhooks accepts the following payload example:

{
  "text": "Example message",
  "attachments": [
    {
      "title": "Rocket.Chat",
      "title_link": "https://rocket.chat",
      "text": "Rocket.Chat, the best open source chat",
      "image_url": "/images/integration-attachment-example.png",
      "color": "#764FA5"
    }
  ]
}

More info here: https://docs.rocket.chat/guides/administration/admin-panel/integrations#incoming-webhook-script

Unsure if this is anything that can be fixed on your end (unfortunately, I suspect not, but I figured I might as well submit it anyways just in case) but fetching RSS feeds from LiveJournal appears to cause it to respond with a 403. The feed is accessible like normal from a browser.

The same feeds seem to error out for the same reason when entering them into a FreshRSS instance hosted on the same server, which make me think it could possibly be some sort of issue with LJ not liking my server or my setup being off, hence why I'm not 100% sure if it's something you can fix, but there's not really anything pointing to my server/general setup being the issue either. Could just be LiveJournal being weird about automated requests for some reason.

This isn't an example of a feed I was trying to subscribe to (the ones I'm attempting to subscribe to are personal blogs hence why I don't want to include them in the issue if possible), but it does showcase the 403 issue: https://perl.livejournal.com/data/rss

hi! thanks for making this. i'm trying this out as an atom/rss -> filtering/enriching -> json feed aggregator.

The custom data format is nearly compatible with JSON Feed item. The json_feed_model crate at bluk/json-feed-model provides convenient get/set and (feed-level) validation of the schema.

How do you think about changing the serialization format for custom sink, or a new jsonfeed sink which use those field name?

This would lessen the downstream scripts' work in renaming them. In particular, non-URL item ID are not sent via custom output yet.

cheers!

Hey! Thanks for the cool tool. Trying to get this going, but I'm a bit stumped. Running with debug flag I see:

2023-05-30T14:29:30.530439Z INFO rss_forwarder: Start watcher for "github-blog" 2023-05-30T14:29:30.531683Z DEBUG reqwest::connect: starting new connection: https://blog.rust-lang.org/

Then it just hangs. I've tried different URLs, same deal. Internet connectivity is fine, I can curl those sites fine. Any suggestions?

I installed JQ and Rustinit, following all the defaults. Then, in an elevated PowerSgell window, I typed in "cargo install rss-forwarder" as mentioned on the main page. It downloads all the packages without issue. However, when it starts compiling, it runs into an issue. After compiling a few dozen things, it issues one error and then stops completely. For some reason, trying to run with RUST_BACKTRACE=1 does not work. I am new to this, so I am simply going to post the entire error log. I'm sorry it is a lot of text, but Powershell simply gave me a lot. I have tried restarting my computer, to no avail.

The following warnings were emitted during compilation:

warning: "jemalloc support for x86_64-pc-windows-msvc is untested"

error: failed to run custom build command for jemalloc-sys v0.3.2

Caused by:
process didn't exit successfully: C:\Users\monst\AppData\Local\Temp\cargo-installTidQGO\release\build\jemalloc-sys-49e3d58dfd9e62e8\build-script-build (exit code: 101)
--- stdout
TARGET=x86_64-pc-windows-msvc
HOST=x86_64-pc-windows-msvc
NUM_JOBS=20
OUT_DIR="C:\Users\monst\AppData\Local\Temp\cargo-installTidQGO\release\build\jemalloc-sys-bbffb68d22dbd6fc\out"
BUILD_DIR="C:\Users\monst\AppData\Local\Temp\cargo-installTidQGO\release\build\jemalloc-sys-bbffb68d22dbd6fc\out\build"
SRC_DIR="C:\Users\monst\.cargo\registry\src\github.com-1ecc6299db9ec823\jemalloc-sys-0.3.2"
cargo:warning="jemalloc support for x86_64-pc-windows-msvc is untested"
OPT_LEVEL = Some("3")
TARGET = Some("x86_64-pc-windows-msvc")
HOST = Some("x86_64-pc-windows-msvc")
cargo:rerun-if-env-changed=CC_x86_64-pc-windows-msvc
CC_x86_64-pc-windows-msvc = None
cargo:rerun-if-env-changed=CC_x86_64_pc_windows_msvc
CC_x86_64_pc_windows_msvc = None
cargo:rerun-if-env-changed=HOST_CC
HOST_CC = None
cargo:rerun-if-env-changed=CC
CC = None
cargo:rerun-if-env-changed=CFLAGS_x86_64-pc-windows-msvc
CFLAGS_x86_64-pc-windows-msvc = None
cargo:rerun-if-env-changed=CFLAGS_x86_64_pc_windows_msvc
CFLAGS_x86_64_pc_windows_msvc = None
cargo:rerun-if-env-changed=HOST_CFLAGS
HOST_CFLAGS = None
cargo:rerun-if-env-changed=CFLAGS
CFLAGS = None
cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
CRATE_CC_NO_DEFAULTS = None
CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
DEBUG = Some("false")
CC="C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.34.31933\bin\HostX64\x64\cl.exe"
CFLAGS="-nologo -MD -O2 -Brepro -W4"
JEMALLOC_REPO_DIR="jemalloc"
JEMALLOC_SRC_DIR="C:\Users\monst\AppData\Local\Temp\cargo-installTidQGO\release\build\jemalloc-sys-bbffb68d22dbd6fc\out\jemalloc"
cargo:rustc-cfg=prefixed
--with-jemalloc-prefix=rjem
running: "sh" "/c/Users/monst/AppData/Local/Temp/cargo-installTidQGO/release/build/jemalloc-sys-bbffb68d22dbd6fc/out/jemalloc/configure" "--disable-cxx" "--with-jemalloc-prefix=rjem" "--with-private-namespace=rjem" "--host=x86_64-pc-win32" "--build=x86_64-pc-win32" "--prefix=C:\Users\monst\AppData\Local\Temp\cargo-installTidQGO\release\build\jemalloc-sys-bbffb68d22dbd6fc\out"

--- stderr
thread 'main' panicked at 'failed to execute command: program not found', C:\Users\monst.cargo\registry\src\github.com-1ecc6299db9ec823\jemalloc-sys-0.3.2\build.rs:389:19
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: failed to compile rss-forwarder v0.5.1, intermediate artifacts can be found at `C:\Users\monst\AppData\Local\Temp\cargo-installTidQGO

hi, sorry that i can't get started. Here is an example config

# Feed 1
[feeds.github-blog]
url = "https://github.blog/all.atom"
interval = "10m"
retry_limit = 5
sink.type = "discord"
sink.url = "https://discord.com/api/webhooks/84175.../OZdejNBCL1..."

# Feed 2
[feeds.rust-blog]
url = "https://blog.rust-lang.org/feed.xml"
interval = "1m"

[feeds.rust-blog.sink]
type = "slack"
url = "https://slack/"

# testing outside docker image, to have `tee` command
[feeds.xkcd]
url = "https://xkcd.com/atom.xml"
interval = "2m"
sink.type = "custom"
sink.command = "tee"
sink.arguments = ["-a", "/tmp/xkcd.ndjson"]

I tried with 0.5.1 binary, the ghcr images for 0.5.1 and 0.6.0-alpha.0, as well as building git repo. I can see the custom command running:

rss-forwarder-l,179885 config.toml
  |-tee,179900 -a /tmp/xkcd.ndjson
  |-{rss-forwarder-l},179887
  |-{rss-forwarder-l},179888
  |-{rss-forwarder-l},179889
  |-{rss-forwarder-l},179890
  |-{rss-forwarder-l},179891
  |-{rss-forwarder-l},179892
  |-{rss-forwarder-l},179893
  |-{rss-forwarder-l},179894
  |-{rss-forwarder-l},179895
  |-{rss-forwarder-l},179896
  |-{rss-forwarder-l},179897
  `-{rss-forwarder-l},179899

The debug log shows feed fetching happening:

[2022-09-03T06:57:31Z INFO  rss_forwarder] Start watcher for "github-blog"
[2022-09-03T06:57:31Z INFO  rss_forwarder] Start watcher for "rust-blog"
[2022-09-03T06:57:31Z TRACE mio::poll] registering event source with poller: token=Token(1), interests=READABLE | WRITABLE
[2022-09-03T06:57:31Z INFO  rss_forwarder] Start watcher for "xkcd"
[2022-09-03T06:57:31Z DEBUG reqwest::connect] starting new connection: https://github.blog/
[2022-09-03T06:57:31Z DEBUG reqwest::connect] starting new connection: https://xkcd.com/
[2022-09-03T06:57:31Z DEBUG reqwest::connect] starting new connection: https://blog.rust-lang.org/
[2022-09-03T06:57:31Z TRACE mio::poll] registering event source with poller: token=Token(2), interests=READABLE | WRITABLE
[2022-09-03T06:57:31Z TRACE mio::poll] registering event source with poller: token=Token(3), interests=READABLE | WRITABLE
[2022-09-03T06:57:31Z TRACE mio::poll] registering event source with poller: token=Token(4), interests=READABLE | WRITABLE
[2022-09-03T06:57:31Z TRACE want] signal: Want
[2022-09-03T06:57:31Z TRACE want] signal found waiting giver, notifying
[2022-09-03T06:57:31Z TRACE want] poll_want: taker wants!
[2022-09-03T06:57:31Z TRACE want] signal: Want
[2022-09-03T06:57:31Z TRACE want] signal found waiting giver, notifying
[2022-09-03T06:57:31Z TRACE want] poll_want: taker wants!
[2022-09-03T06:57:31Z DEBUG reqwest::async_impl::client] response '200 OK' for https://blog.rust-lang.org/feed.xml
[2022-09-03T06:57:31Z TRACE want] signal: Want
[2022-09-03T06:57:31Z TRACE want] signal found waiting giver, notifying
[2022-09-03T06:57:31Z TRACE want] signal: Want
[2022-09-03T06:57:31Z TRACE want] poll_want: taker wants!
[2022-09-03T06:57:31Z TRACE want] signal: Want
[2022-09-03T06:57:31Z DEBUG reqwest::async_impl::client] response '200 OK' for https://github.blog/all.atom
[2022-09-03T06:57:31Z TRACE want] signal: Want
[2022-09-03T06:57:31Z TRACE want] signal found waiting giver, notifying
[2022-09-03T06:57:31Z TRACE want] poll_want: taker wants!
[2022-09-03T06:57:32Z TRACE want] signal: Want
[2022-09-03T06:57:32Z TRACE want] signal found waiting giver, notifying
[2022-09-03T06:57:32Z TRACE want] signal: Want
[2022-09-03T06:57:32Z TRACE want] poll_want: taker wants!
[2022-09-03T06:57:32Z TRACE want] signal: Want
[2022-09-03T06:57:32Z DEBUG reqwest::async_impl::client] response '200 OK' for https://xkcd.com/atom.xml
[2022-09-03T06:57:32Z TRACE want] signal: Want
[2022-09-03T06:57:32Z TRACE want] signal found waiting giver, notifying
[2022-09-03T06:57:32Z TRACE want] signal: Want
[2022-09-03T06:57:32Z TRACE want] poll_want: taker wants!
[2022-09-03T06:57:32Z TRACE want] signal: Want
[2022-09-03T06:58:31Z DEBUG reqwest::async_impl::client] response '200 OK' for https://blog.rust-lang.org/feed.xml
[2022-09-03T06:58:31Z TRACE want] signal: Want
[2022-09-03T06:58:31Z TRACE want] signal found waiting giver, notifying
[2022-09-03T06:58:31Z TRACE want] signal: Want
[2022-09-03T06:58:31Z TRACE want] poll_want: taker wants!
[2022-09-03T06:58:31Z TRACE want] signal: Want
[2022-09-03T06:58:42Z TRACE mio::poll] deregistering event source from poller
[2022-09-03T06:58:42Z TRACE want] signal: Closed

At shutdown via ctrl+c:

^C[2022-09-03T07:00:15Z TRACE mio::poll] deregistering event source from poller
[2022-09-03T07:00:15Z INFO  rss_forwarder] Watcher for "rust-blog" has stopped
[2022-09-03T07:00:15Z INFO  rss_forwarder] Watcher for "github-blog" has stopped
[2022-09-03T07:00:15Z INFO  rss_forwarder] Watcher for "xkcd" has stopped
[2022-09-03T07:00:15Z TRACE mio::poll] deregistering event source from poller
[2022-09-03T07:00:15Z TRACE mio::poll] deregistering event source from poller
[2022-09-03T07:00:15Z TRACE want] signal: Closed
[2022-09-03T07:00:15Z TRACE want] signal: Closed

Idling section for 0.6.0:

2022-09-03T07:06:45.053670Z TRACE hyper::proto::h1::io: received 4490 bytes
2022-09-03T07:06:45.053682Z DEBUG hyper::proto::h1::conn: incoming body completed
2022-09-03T07:06:45.053718Z TRACE hyper::proto::h1::conn: maybe_notify; read_from_io blocked
2022-09-03T07:06:45.053734Z TRACE want: signal: Want
2022-09-03T07:06:45.053745Z TRACE want: signal found waiting giver, notifying
2022-09-03T07:06:45.053763Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2022-09-03T07:06:45.053775Z TRACE want: signal: Want
2022-09-03T07:06:45.053781Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2022-09-03T07:06:45.053789Z TRACE want: poll_want: taker wants!
2022-09-03T07:06:45.053818Z TRACE hyper::client::pool: put; add idle connection for ("https", github.blog)
2022-09-03T07:06:45.053834Z DEBUG hyper::client::pool: pooling idle connection for ("https", github.blog)
2022-09-03T07:06:45.056467Z TRACE want: signal: Want
2022-09-03T07:06:45.056479Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }

But we can see that no sink are invoked. The wrong discord token, and the bad slack url are not causing error.

I added a tracing::info event in custom sink:

                let data: Vec<u8> = tokio::select! {
                    biased;
                    _ = kill_rx.recv() => break,
                    v = data_rx.recv() => v.unwrap(),
                };
                tracing::info!(data =? data, "flush");

but it doesn't happen.

I tried to use tokio-console (it's necessary to downgrade to tokio@=1.20.1 --features tracing due to tokio-rs/console#373), there's one warning, but i don't understand async enough to see how task notification works yet.

My environment is Fedora 36, container runtimes docker/moby-engine, podman give same result.

What are your environments? Any pointer on how to check more on my side? thanks!

Currently I see that the rss-forwarder is x86_64 only. Is there a possibility of adding an image build for arm64?