ipfs / infra Goto Github PK
View Code? Open in Web Editor NEWTools and systems for the IPFS community
License: MIT License
Tools and systems for the IPFS community
License: MIT License
Round-robin over the cjdns addresses:
fc98:424c:b433:d7e2:7ee3:9541:73ff:2cdb
fce3:c53b:c3c5:2f54:8bb0:b6d9:898e:f140
fcfe:eab4:e49c:940f:8b29:35a4:8ea8:b01a
fc4e:5427:3cd0:cc4c:4770:25bb:a682:d06c
fc3d:9a4e:3c96:2fd2:1afa:18fe:8dd2:b602
fcd8:a4e5:3af7:557e:72e5:f9d1:a599:e329
fc29:9fda:3b73:c1d2:9302:31e3:964c:144c
fcdf:a296:afe3:7118:4135:cc0b:ff92:4585
in ipfs/kubo#1261, Gitcop says
There were the following issues with your Pull Request
Commit: ef34768
Invalid signoff. Commit message must end with License: MIT Signed-off-by: .* <.*>
The commit message is
using multistream muxer
* ID service stream
* make the relay service use msmux
* fix nc tests
Note from jbenet: Maybe we should remove the old protocol/muxer
and see what breaks. It shouldn't be used by anything now.
License: MIT
Signed-off-by: Jeromy <[email protected]>
Signed-off-by: Juan Batiz-Benet <[email protected]>
Recap:
We should organize our testing infrastructure concerns.
Our constraints are:
Unfortunately, after trying multiple CI systems, always end up coming back to travis. (We used jenkins for a long time and it was a huge pain to manage. life is much better without it).
Close seconds are:
For testing on windows, we could use:
dist.ipfs.io and ipfs.io/dist
Requesting https://ipfs.io/ipfs/QmQQ3BUpPjgYiTdhp4H9YWSCtoFXs8t91njhpvXNNLd3yB
Returns
internalWebError: unexpected EOF
Unclear whether this is a go-ipfs issue or nginx etc.
We have a bunch of projects using CI.
It would be cool, possibly useful too, to have a single page that gathered them all together. Something like:
Which is an instance of this rails app:
https://github.com/pivotal/projectmonitor
Works with Circle CI, Travis and others. Could run on a free heroku instance (or wherever).
Alternatively, people who care about the overall health of multiple/all builds could pull down status messages to local notifications with CCTray, BuildNotify, etc ( some info on that for Circle here: https://circleci.com/docs/polling-project-status ).
In any case, we should probably add passing/failing badges to the projects' READMEs, so you could see build health there.
We could leave it as is: build failures should notify the committers, and if you want to check, you can probably peruse IPFS builds in Circle or Travis.
Thoughts? I vote for # 3 + # 1, in that order, with optional # 2 encouragement / instructions.
Currently only ipfs
and go-ipfs
are enabled.
So they don't get lost, and can be applied to other hosts as well
Right now cjdns runs in the host, and all dependent containers use the host network stack.
--cap-add NET_ADMIN
for /dev/net/tun
--net container:cjdns
Denylist daemon: https://github.com/ipfs/gateway-dmca-denylist
Notices storage: https://github.com/ipfs/gateway-dmca
We shouldn't have them in secrets.yml -- any service's OAuth token is the weakest link, right after our Github accounts.
We'll stick with the scheme of giant spherish space rocks, in order to not introduce another universe of terms (Matt Groening's janitors, in this case)
$janitor.i.ipfs.io
to $spacerock.i.ipfs.io
/etc/hostname
on each hostLooking at waffle.io. It looks very promising. I played around with the multi-repo linking which isn't perfect but it works. I'm currently waiting upon the creation of ipfs/ipfs to act as a root repo for the IPFS projects.
Tasks for https://waffle.io/ipfs/ipfs:
We should have a base nginx role, and each role that exposes something, should come with its own file for sites-enabled.
Docker currently keeps the complete log of the container, which we throw away anyhow since IPFS keeps its own logs. Docker 1.6 introduced the --log-driver
option which we can set to none
.
Ansible implements this in 2.0 which isn't released yet. Relevant commit: ansible/ansible-modules-core@718f32a
Someone needs to go through the New Issues on https://waffle.io/ipfs/ipfs and determine which ones should definitely be closed, moved to the backlog, or moved to the icebox. If the issue is missing any labels then label it. If it's questionable, like backlog vs icebox then get the relevant person involved.
An IRC bot similar to pinbot, but pins only on the storage hosts.
We'll use this issue to track objects to pin for now. I'll add a file. eventually we'll have a much cleaner way to do it.
(let's edit with a checkbox + point to the commit adding the file)
We can't bust 100 percent of existing NATs, but we can provide opt-in relaying via the bootstrap nodes.
I'm seeing the CORS:
> curl -I https://ipfs.io/ipns/ip
HTTP/1.1 200 OK
Server: nginx/1.9.3
Date: Sun, 06 Sep 2015 03:41:17 GMT
Content-Type: text/plain; charset=utf-8
Connection: keep-alive
Access-Control-Allow-Origin: https://gateway.ipfs.io
Access-Control-Allow-Origin: https://ipfs.io
Access-Control-Allow-Origin: http://gateway.ipfs.io
Access-Control-Allow-Origin: http://ipfs.io
Access-Control-Allow-Origin: http://localhost
Access-Control-Allow-Origin: http://localhost:8080
Access-Control-Allow-Origin: http://127.0.0.1
Access-Control-Allow-Origin: http://127.0.0.1:8080
Suborigin: ipfs.io
X-Ipfs-Path: /ipns/ipfs.io/
Strict-Transport-Security: max-age=15768000
On chrome i get
https://gateway.ipfs.io/ipfs/QmWCobpCNQy9SN19mcauGw4HR5p1KBcTvL6ousavdpJxqt/12/3790/2375.pbf. The 'Access-Control-Allow-Origin' header contains multiple values 'https://gateway.ipfs.io, https://ipfs.io, http://gateway.ipfs.io, http://ipfs.io, http://localhost, http://localhost:8080, http://127.0.0.1, http://127.0.0.1:8080', but only one is allowed. Origin 'https://ipfs.io' is therefore not allowed access.
Looks like we're returning it incorrectly (it should only return for the origin itself-- i.e. if the user's origin is the same one as one of these, then we return it). I think this has to be fixed in go-ipfs, but tagging here as we should make sure to fix it through to deploy.
jbenet commented 7 days ago
We want to put the bots into solarnet. these include:
sprintbot pinbot - probably should pin to the gateways directly
in the long run, pinbot maybe should be using an S3 datastore or something.
jbenet commented 3 days ago
pinbot crashed recently. we should bring it into the infra. what needs to be done to run it effectively?
maybe we can run it on jupiter or saturn.
jbenet commented 3 days ago
@whyrusleeping could you outline all the considerations necessary to running these bots?
AFAICT from https://github.com/whyrusleeping/pinbot -- it uses shell, so should "just work" with a standard daemon running in the background, right? there may be some interesting networking magic to do to wire up the containers.
https://github.com/whyrusleeping/pinbot/blob/master/main.go#L102 <-- nice ๐
whyrusleeping commented 3 days ago
Yeah, all you need to do is run the pinbot program on a machine with a daemon running. It would be best to have it on a machine with the most storage available to us. It also is capable of connection stealing updates, so if you modify the code, just run the new binary, it will grab the tcp connection from the old one and take it over to avoid having the bot disconnect and reconnect.
jbenet commented 3 days ago
- add the pinbot binary as a container somewhere
- largest storage? (ideally S3 once 0.4.0 merges)
- updates should gracefully run new bin before destroying the old one
What else?
Remaining questions:
- one machine or multiple?
- what redundancy should we have?
- we could have pinbot0..9, or we could have 1 irc bot that pins to several other nodes
We should have a calendar (or set of calendars) for important dates + community events (talks, etc).
We can setup google calendars @ipfs.io and have it listed to be publicly viewable somewhere. Does that sound good, or are there other services people want to try?
We want to use GitCop to make sure the commits have the following trailers:
License: MIT
Signed-off-by: user name <email@address>
We might also limit the subject of each commit to 80 characters.
What do you think?
We ran into the 10 person google hangouts limit today. i dont think google live hangouts helps much here.
Others have suggested:
main requirements are:
Have documentation on how the kanban board at https://waffle.io/ipfs/ipfs works.
See #7 (comment)
Notes:
@awrelll is setting up a gitter bridge. the goal is to get it to:
ipfs/ipfs
with our freenode #ipfs
channel.I regained control of github.com/ipfsbot -- but im thinking that the gitter bridge should be its own account (cause we'll use ipfsbot for other things too), maybe ipfs-gitter-bot
As a note, until those three checkboxes above are robust, i dont feel ok telling people to use gitter. the last thing i want is for people to think we're ignoring them (if their messages arent getting through). an unreliable comm link can be worse than forcing people to use another comm link.
root@sthing2:~# ipfs ping QmSoLV4Bbm51jM9C4gDYZQ9Cy3U6aXMJDAbzgu2fzaDs64
Looking up peer QmSoLV4Bbm51jM9C4gDYZQ9Cy3U6aXMJDAbzgu2fzaDs64
Peer lookup error: routing: not found
Got the ID from ipfs bootstrap list
root@sthing2:~# ipfs bootstrap list
/ip4/104.131.131.82/tcp/4001/ipfs/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ
/ip4/104.236.151.122/tcp/4001/ipfs/QmSoLju6m7xTh3DuokvT3886QRYqxAzb1kShaanJgW36yx
/ip4/104.236.176.52/tcp/4001/ipfs/QmSoLnSGccFuZQJzRadHn95W2CrSFmZuTdDWP8HXaHca9z
/ip4/104.236.179.241/tcp/4001/ipfs/QmSoLpPVmHKQ4XTPdz8tjDFgdeRFkpV8JgYq8JVJ69RrZm
/ip4/104.236.76.40/tcp/4001/ipfs/QmSoLV4Bbm51jM9C4gDYZQ9Cy3U6aXMJDAbzgu2fzaDs64
/ip4/128.199.219.111/tcp/4001/ipfs/QmSoLSafTMBsPKadTEgaXctDQVcqN88CNLHXMkTNwMKPnu
/ip4/162.243.248.213/tcp/4001/ipfs/QmSoLueR4xBeUbY9WZ9xGUUxunbKWcrNFTDAadQJmocnWm
/ip4/178.62.158.247/tcp/4001/ipfs/QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd
/ip4/178.62.61.185/tcp/4001/ipfs/QmSoLMeWqB7YGVLJN3pNLQpmmEk35v6wYtsMGLzSr5QBU3
It would be good to make a list of all available infrastructure in the readme. Currently there are only generalities listed.
What we have:
In order to help people find things we need to more narrowly define some of our repos. Some repos are very broad, overlap with others, or aren't defined at all. We also need to update the descriptions to reflect our definitions.
Some ideas:
It currently happens manually with apt-get...
lgierth commented 28 days ago
We might wanna separate the IPFS and HTTP traffic and spread it to different hosts depending on the load pattern. One the same hosts, they could still run in the same container. Need graphs to see if it's an issue, or notice when it turns into one.
jbenet commented 28 days ago
Need graphs to see if it's an issue, or notice when it turns into one.
yep, exactly agreed.
Stuff that can be hosted in IPFS
Add https://wiki.jenkins-ci.org/display/JENKINS/GitHub+pull+request+builder+plugin to jenkins to allow admins to run the full test suite on pull requests.
Right now it's publicly available at pluto.i.ipfs.io/prometheus, should hide it behind the cjdns whitelist.
Add https://github.com/ipfs/api
Can we get one with just the logo and a transparent background?
Discussion conclusion:
We can try https://github.com/ipfs/logo/blob/master/ipfs-logo-1024-ice.png, but @jbenet wasn't sold upon it looking better.
Moved from ipfs/ipfs#62
We don't need that many big hosts at the moment.
We want to have the gateways at ipfs.io
instead of gateway.ipfs.io
This requires
I've had it with jenkins. it's so painful to work with. I want to move off. It's currently broken-- it isn't building go-ipfs properly.
I really like golang's build system: http://build.golang.org -- it is just https://github.com/golang/build -- ideally we could run our own version of that, equipped with ipfs to avoid re-downloading things multiple times (i.e. so tests get even faster). This sounds non-trivial to setup...
So perhaps in the meantime we could try to use travis for everything we do in jenkins. This may not work, particularly:
3nodetest
) will work.The jenkins tests in question are:
test_sharness_expensive
ideally with fusebuild_docker_image
to check it worksbuild_all_commits
to check all commits build (probably redundant given test_all_commits)
test_go
test_3node
requires containerstest_go_race
likely to be very slowtest_all_commits
runs make test
on all commits.test_go_integration
runs some integration testsI think that -- instead of only testing make test
on all commits, we should instead:
This might be as simple as making the repo test all these pieces and using travis.
@hosh You have the most knowledge on this.
Some column names for the current sprint:
For the backlog column:
I want to have a full implementation-independent test suite, including things like the current network test, and the sharness tests. This will allow developers of other implementations to benchmark against another test suite.
This requires:
impl-tests
make test
in go-ipfs should still run these tests, so perhaps it can clone the repo locally.implementations wont have all the same sets of features, e.g. some implementations may be missing gateways, or missing the CLI altogether, or even the API (but still have the protocol implemented). It would be ideal if the test suite allowed implementors to pick the sets of test cases to run. Perhaps something like:
protocol
for full protocol testsAPI
for testing conformance to the the api speccli
for testing cli tools.Maybe we should fully decouple the cli in go-ipfs, so that it can be used against other language implementations of the API just as well.
This is a big project. Starting the conversation, as people are itching to get started with implementations in other languages.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.