Giter Club home page Giter Club logo

mercury's Introduction

Planet Mercury

Build status Go Report Card

Mercury is intended a replacement for Sam Ruby's Planet Venus.

A planet is a kind of feed aggregator. It takes a list of newsfeeds (Atom, RSS, &c.), splices them together, and spits a set of HTML pages and/or a feed.

For details, please see the documentation.

If you're using Planet Mercury for any sites, I'd be happy to list them.

mercury's People

Contributors

dependabot[bot] avatar kgaughan avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

Forkers

toshywoshy

mercury's Issues

Respect the Expires and Cache-Control headers

Currently, the spider could be making unnecessary requests, as it's only does conditional GET requests. While doing conditional GETs is a good thing, it should also avoid making requests by checking the Expires and Cache-Control headers to see if it should be relying on the cached version instead.

There ought to be a flag to disable this, if necessary.

Internally, it might be good to parse them into a single structure, converting max-age into a timestamp.

Plugin system

One gap in the functionality of Planet Mercury over Planet Venus is the lack of a plugin/filter system.

I think this should be implemented in two ways:

  1. A Go-based plugin system using the standard library's plugin package. This would be the most powerful method and would allow for the extension of the template system on top of giving ways to process entries. Having experimented with this, I'm not so sure it's a good idea. It requires the use of CGo. A better alternative would be to make use of net/rpc/jsonrpc and spin up small servers that can service requests to process entries. I'll need to work on what the interface will be, but initially, it'll probably be a single method called process that takes a serialised entry and returns a result along the same lines as the pipeline-based method.
  2. An external method for processing entries. This would serialise entries as JSON to be read from stdin and read the result from stdout. The status could be used to determine whether the script was happy (0), wants the entry skipped (1), or is returning an error (2+, with the error serialised as JSON). I don't think it's a good idea to overload the semantics of the exit status like this now.

There's an opportunity for batching here, which could aid with filtering. I think the top level of the request and response should be an array of entries, even if the entries are just being passed through one-by-one. That way, if an entry should be skipped, omitting the entry will be enough.

Category selection

Give a way to specify particular categories that should be included from the source feed. This would be useful for blogs that lack per-category feeds.

Support a theme BOM

Themes should include a BOM indicating which files should be copied across to the output directory.

Test suite

This was originally written in a bit of a rush, so I skimped on the test suite. I ought to fix that.

Segfault when comparing feeds

I believe this might be something to do with some particular RSS feed not including a published date or something:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x3918bc]

goroutine 1 [running]:
main.feedQueue.Less(0x22bb200, 0x57, 0x80, 0x20f7200, 0x57, 0x80, 0x50, 0x4f, 0x64b01)
        mercury/feedqueue.go:35 +0x150
container/heap.down(0x4deed8, 0x209f6e0, 0x27, 0x57, 0x209f301)
        container/heap/heap.go:109 +0x104
container/heap.Init(0x4deed8, 0x209f6e0)
        container/heap/heap.go:46 +0x5c
main.main()
        mercury/main.go:95 +0x864

I'll investigate this, but I think the solution would be to filter out any bad feed items like this.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.