lwnfeed's Introduction

lwnfeed

lwnfeed is a simple daemon that creates a full-text RSS feed from lwn.net by scraping the content from the website.

You must have a LWN subscription to use this tool. It is intended for personal use only, please do not publish the generated feeds.

Getting started

Install Go 1.13 or higher.
Clone the repository (git clone https://github.com/tulir/lwnfeed.git) and enter the directory (cd lwnfeed).
Build the program (./build.sh).
Use ./lwnfeed login to log in and store the auth cookie into a file.
Start the server with ./lwnfeed start.
Open http://localhost:8080/feed.rss to view the feed. feed.atom and feed.json also work.

Docker

Run docker run --rm -itv $(pwd):/data dock.mau.dev/tulir/lwnfeed login to log in and store the auth cookie into a file.
Start the container normally with your preferred method (see examples below), keeping the same bind mount as before.

docker start

$ docker start -p 8080:8080 --name lwnfeed -v $(pwd):/data dock.mau.dev/tulir/lwnfeed

docker-compose

version: "3.7"

services:
  lwnfeed:
    container_name: lwnfeed
    image: dock.mau.dev/tulir/lwnfeed
    ports:
    - 8080:8080
    volumes:
    - .:/data

lwnfeed's People

Contributors

Stargazers

Watchers

lwnfeed's Issues

Slow FD leak

It seems that we leak a few FDs every time we scrape LWN. Here's lsof 12 hours after a restart:

rav@kaylee:~$ sudo lsof -p 21726 -n
COMMAND   PID    USER   FD      TYPE             DEVICE SIZE/OFF      NODE NAME
lwnfeed 21726 lwnfeed  cwd       DIR              254,0     4096    145299 /home/lwnfeed
lwnfeed 21726 lwnfeed  rtd       DIR              254,0     4096         2 /
lwnfeed 21726 lwnfeed  txt       REG              254,0 14380174    145267 /home/lwnfeed/lwnfeed
lwnfeed 21726 lwnfeed    0r      CHR                1,3      0t0      7292 /dev/null
lwnfeed 21726 lwnfeed    1u     unix 0x0000000035718e19      0t0 177232550 type=STREAM
lwnfeed 21726 lwnfeed    2u     unix 0x0000000035718e19      0t0 177232550 type=STREAM
lwnfeed 21726 lwnfeed    3u     IPv4          177232564      0t0       TCP 127.0.0.1:http-alt (LISTEN)
lwnfeed 21726 lwnfeed    4u  a_inode               0,13        0      7223 [eventpoll]
lwnfeed 21726 lwnfeed    5r     FIFO               0,12      0t0 177232554 pipe
lwnfeed 21726 lwnfeed    6w     FIFO               0,12      0t0 177232554 pipe
lwnfeed 21726 lwnfeed    7u     sock                0,9      0t0 177232572 protocol: TCPv6
lwnfeed 21726 lwnfeed    8u     sock                0,9      0t0 177232582 protocol: TCPv6
lwnfeed 21726 lwnfeed    9u     sock                0,9      0t0 177232587 protocol: TCPv6
lwnfeed 21726 lwnfeed   10u     sock                0,9      0t0 177232592 protocol: TCPv6
lwnfeed 21726 lwnfeed   11u     sock                0,9      0t0 177232597 protocol: TCPv6
lwnfeed 21726 lwnfeed   13u     sock                0,9      0t0 177285620 protocol: TCPv6
lwnfeed 21726 lwnfeed   14u     sock                0,9      0t0 177285625 protocol: TCPv6
lwnfeed 21726 lwnfeed   15u     sock                0,9      0t0 177285630 protocol: TCPv6
lwnfeed 21726 lwnfeed   16u     sock                0,9      0t0 177285635 protocol: TCPv6
lwnfeed 21726 lwnfeed   17u     sock                0,9      0t0 177285640 protocol: TCPv6
lwnfeed 21726 lwnfeed   18u     sock                0,9      0t0 177316086 protocol: TCPv6
lwnfeed 21726 lwnfeed   19u     sock                0,9      0t0 177316091 protocol: TCPv6
lwnfeed 21726 lwnfeed   20u     sock                0,9      0t0 177316096 protocol: TCPv6
lwnfeed 21726 lwnfeed   21u     sock                0,9      0t0 177316101 protocol: TCPv6
lwnfeed 21726 lwnfeed   22u     sock                0,9      0t0 177316106 protocol: TCPv6
lwnfeed 21726 lwnfeed   23u     sock                0,9      0t0 177344313 protocol: TCPv6
lwnfeed 21726 lwnfeed   24u     sock                0,9      0t0 177344318 protocol: TCPv6
lwnfeed 21726 lwnfeed   25u     sock                0,9      0t0 177344323 protocol: TCPv6
lwnfeed 21726 lwnfeed   26u     sock                0,9      0t0 177344328 protocol: TCPv6
lwnfeed 21726 lwnfeed   27u     sock                0,9      0t0 177344333 protocol: TCPv6

Each of those TCPv6 sockets was previously a connection to lwn.net:443.

(by the way, thanks for making this! It's super useful ❤️)

Recommend Projects