skroutz / downloader Goto Github PK
View Code? Open in Web Editor NEWAsync rate-limited downloading service
License: GNU General Public License v3.0
Async rate-limited downloading service
License: GNU General Public License v3.0
I am running a Redis server on a port other than the default (which is 6379) and the tests do not respect the configuration file and fail to connect.
tsiou@tsiou downloader >>= make
! gofmt -d -e -s *.go **/*.go 2>&1 | tee /dev/tty | read
/bin/sh: 1: read: arg count
go test -race -p 1 ./...
2019/03/28 15:03:34 Could not ping Redis Server successfully: dial tcp 127.0.0.1:6379: connect: connection refused
2019/03/28 15:03:35 Server on port 8123 not up after 10 retries
FAIL github.com/skroutz/downloader 0.522s
2019/03/28 15:03:35 dial tcp [::1]:6379: connect: connection refused
FAIL github.com/skroutz/downloader/api 0.009s
ok github.com/skroutz/downloader/job (cached)
2019/03/28 15:03:36 dial tcp [::1]:6379: connect: connection refused
FAIL github.com/skroutz/downloader/notifier 0.010s
2019/03/28 15:03:37 dial tcp [::1]:6379: connect: connection refused
FAIL github.com/skroutz/downloader/processor 0.008s
ok github.com/skroutz/downloader/processor/diskcheck (cached)
? github.com/skroutz/downloader/processor/errors [no test files]
ok github.com/skroutz/downloader/processor/mimetype 1.009s
? github.com/skroutz/downloader/stats [no test files]
2019/03/28 15:03:39 Could not ping Redis Server successfully: dial tcp [::1]:6379: connect: connection refused
FAIL github.com/skroutz/downloader/storage 0.008s
make: *** [Makefile:7: test] Error 1
This will effectively allow upstream servers to know with how many concurrency we're downloading from them. With this information, they could implement smarter rate-limiting, notice potential misconfigurations on our side or make more informed resource-planning decisions.
Currently we only display metrics from the Processor. After #5 however, Notifier also reports some metrics as well. We should display those too in the web interface.
For starters, refer to public/js/dashboard.js
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.