Add support for filtering based on languages and time range. about devo HOT 6 CLOSED

Hey, thanks for the kind words.

It is something I've had in mind before as well, but the problem is that the trending information is not provided via an API, and I have a private ETL running somewhere that periodically fetches this page, parses it, builds the JSON, and stores it on DigitalOcean, which is what powers the extension for everyone. Providing this functionality would mean that I'd need to do the same operation for all the languages I'd like to provide, and for now, I think this would be too much of an effort to build and maintain. If there was an API for this then I'd update the extension to use the API directly in those cases, but there is none as far as I am aware.

I'll keep turning this idea in my mind to see if I can find an easy, quick, and cheap way of doing this, but for now, it seems unlikely. If you have ideas about how to get this from an API I am all ears.

from devo.

There is a similar app I'm using, it has done the same ETL processes to extract the trending list per language, maybe we can use them.

gitter
the api to get the trending list

export const getReposTrendingList = createApiAction(REPOSTRENDING, params => api.get('https://gitter-weapp.herokuapp.com/repositories', params))
export const getDevelopersTrendingList = createApiAction(DEVELOPERSTRENDING, params => api.get('https://gitter-weapp.herokuapp.com/developers', params))
export const getLanguageList = createApiAction(LANGUAGE, params => api.get('https://gitter-weapp.herokuapp.com/languages', params))

from devo.

In the case of Devo, I don't plan to depend on someone else's API to provide the basic functionality of the extension, and I don't see a strong enough value in the ability to filter languages and authors to even take the risk here. I should have said "from an official API" in my previous message, apologies.

I will close this issue until there is an easy API provided by GitHub that does this thing, and then I might consider maintaining the ETLs for this, but I am skeptical about it even if that existed, given that the same functionality exists properly on the GitHub Trending page and Devo includes a direct link to there on the page.

Feel free to re-open the issue it if something changes in GitHub API, and then I'll reconsider.

from devo.

I'm wondering why can't we do ETLs in the browser? Just curious.

from devo.

There is an unofficial but popular api.
github-trending-api

You can deploy it on your API server if don't want to use their service.

Lots of projects building on it.

• Hacker Tab (Chrome Store): Chrome extension to view GitHub trending projects on new tab.
• github-trending-plus (Live): ✨ Experimental Github trending UI.
• HitUp (Chrome Store) (Website): Help you find top things, including GitHub Trending Repositories.
• Gitter: Gitter for GitHub - 可能是目前颜值最高的 GitHub 小程序客户端.
• flutter_wanandroid:
  Flutter 完整项目，WanAndroid 客户端，BLoC、RxDart 、国际化、主题色、启动页、引导页，拥有较好的项目结构&比较规范的代码！
• vsc-github-trending: (VS Code Marketplace): Explore Github Trending repositories directly from Visual Studio Code
• GiTiny(AppStore): iOS app for GitHub with exploring trending.
• SwiftHub: GitHub iOS client written in RxSwift and MVVM clean architecture.
• GitNews: Trending repositories from GitHub, HackerNews & Reddit (Mobile & Desktop).
• mini-github: GitHub WeChat Mini Program (可能是全网功能最全的 GitHub 微信小程序).
• StarGit (Website): Find trending repositories from GitHub. Built with Angular.
• GitTouch (App Store, Google Play): Open source GitHub App built with Flutter.
• lazyhub: lazyhub - Terminal UI Client for GitHub using gocui.
• Gitrepos (Website): A simple PWA app that returns Trending Repositories & Developers from GitHub, Gitlab & HackerNews.
• Trendzz (Website): Github trending PWA focusing on usability.
• Ceelog Github trending in WeChat Official Account Platform.

from devo.

Do you mean running ETLs as background scripts in the browsers as part of the extension? If that's what you mean, I'd rather not do that because the UIs change often, and when they change it is usually a breaking change, causing no one to be able to view Devo on their computers, and any such change would require me to build and push a new version, and everybody to update their extensions. What I do instead is to run ETLs on my own side and store the final file in the structure the extension expects; if anything breaks, I can simply fix the ETL on my infra and no one needs to do any other update. In addition to this, I also wouldn't want to run scripts like this on the users' devices, no. This also has no benefits in terms of my main concern, which is maintaining those ETLs.

An additional point in favor of using ETLs is that the cost and maintenance structure is way simpler with ETLs than real-life APIs. The first version of Devo was using Lambda, and my costs were increasing with the number of users, and I was constantly hitting rate limits of APIs and services due to that. By relying on ETLs, I have a fully predictable cost structure, my compute and data transfer rates are in free tiers of the cloud providers, and I don't need to care about hitting rate limits of any API. Therefore, even if GitHub provides an official API, I'd still need to build ETLs to manage listings for every language, and I simply don't see it worth the effort, given that everything is already available on GitHub Trending UI itself. Devo doesn't need it.

As per the other API suggestion, I am still of the same opinion: I will only rely on either the official APIs, or the things I build myself. I want Devo to exist for a long time, and this means relying on stuff that is reliable, or stuff that I can maintain by myself.

from devo.