entranceplus / craw Goto Github PK

A rough idea:

This would be a synchronous process. the interface could be sth like

interface FeedGenerator {
    List<Article> genFeed(Set<String> topics)
}

This call can block for as long as it takes to construct the feed. You can use concurrency inside but the final call must block. This will be a common pattern for all the feed transformations so that we can mix and match them.

For example, let's say now we want to select this final set of article from a cluster. This cluster could pre-exist or will be created on demand (out of scope for now).

We could just implement the above interface, the topics in this case could belong to the tags of clusters produced and then the impl would supply the heuristics to do that. More importantly, we would already
know how to compose interfaces of this kind so we can have various such impls which we can then compose in different ways to improve feed gen.

• given a subreddit - get urls

• get all subreddit's { name, followers }

• given urls - get article content

• link- get current html

• given two html send diff

I think a java model return would be better since I can use that directly in jvm clients.

• https://github.com/entranceplus/craw/blob/master/src/main/java/com/ep/LinkExtractor/ExtractionInterface/LinkFunc.java#L9
  What url is expected here ? [Raghav]-Now no URL is expected just invoke the method

• https://github.com/entranceplus/craw/blob/master/src/main/java/com/ep/LinkExtractor/ExtractionInterface/LinkFunc.java#L6
  This could just be the subreddit name.[Raghav]- Just pass the subreddit Ex. soccer