bluesky-social / indigo Goto Github PK
View Code? Open in Web Editor NEWGo source code for Bluesky's atproto services.
Home Page: https://atproto.com/docs
License: Apache License 2.0
Go source code for Bluesky's atproto services.
Home Page: https://atproto.com/docs
License: Apache License 2.0
Still investigating, but can trigger errors like:
decoding xrpc response: parsing legacy blob: json: Unmarshal(nil *util.LegacyBlob)
Debugging this right now
I run into this segfault consistently when pointing bigsky at the PDS dev environment setup. I am running bigsky on 9a357fe with the following command:
./bigsky --aggregation=false --ssl-events=false --plc-host="http://localhost:2582"
Output:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x100220d86]
goroutine 63 [running]:
math/big.(*Int).Sign(...)
/usr/local/go/src/math/big/int.go:41
crypto/ecdsa.(*nistCurve[...]).pointFromAffine(0x100f86be0, 0x0, 0x0)
/usr/local/go/src/crypto/ecdsa/ecdsa.go:575 +0x46
crypto/ecdsa.verifyNISTEC[...](0xc00061ee40, 0xc000333568, {0xc000333508, 0x20, 0x20}, {0xc000616600, 0x0?, 0x0?})
/usr/local/go/src/crypto/ecdsa/ecdsa.go:494 +0xc5
crypto/ecdsa.VerifyASN1(0xc000333568, {0xc000333508, 0x20, 0x20}, {0xc000616600, 0x48, 0x60})
/usr/local/go/src/crypto/ecdsa/ecdsa.go:478 +0x285
crypto/ecdsa.Verify(0x11000371f53b00?, {0xc000333508, 0x20, 0x20}, 0xc0007366f0?, 0xc00033a7c0)
/usr/local/go/src/crypto/ecdsa/ecdsa_legacy.go:126 +0x10b
github.com/whyrusleeping/go-did.(*PubKey).Verify(0xc00033a780, {0xc0007366f0, 0x24, 0x30}, {0xc000618380, 0x40, 0x40})
/Users/devinivy/go/pkg/mod/github.com/whyrusleeping/[email protected]/key.go:158 +0x2f9
github.com/bluesky-social/indigo/indexer.(*KeyManager).VerifyUserSignature(0xc0006125f0?, {0x100f89d60, 0xc00061b020}, {0xc00062a3c0, 0x20}, {0xc000618380, 0x40, 0x40}, {0xc0007366f0, 0x24, ...})
/project/indigo/indexer/keymgr.go:37 +0x185
github.com/bluesky-social/indigo/repomgr.(*RepoManager).HandleExternalUserEvent(0xc00071ffc0, {0x100f89cf0, 0xc000120008}, 0x1, 0x1, {0xc00071c7a0, 0x20}, 0x0, {0xc0005ee000, 0x2b3, ...})
/project/indigo/repomgr/repomgr.go:548 +0x4ec
github.com/bluesky-social/indigo/bgs.(*BGS).handleFedEvent(0xc0002bf4f0, {0x100f89cf0, 0xc000120008}, 0xc000308090, 0xc0005ee001?)
/project/indigo/bgs/bgs.go:269 +0x434
github.com/bluesky-social/indigo/bgs.(*Slurper).handleConnection.func1(0xc0005d8300)
/project/indigo/bgs/fedmgr.go:139 +0x1a7
github.com/bluesky-social/indigo/events.HandleRepoStream({0x100f89cb8?, 0xc0005c2a50?}, 0xc000455e40, 0xc000333df0)
/project/indigo/events/consumer.go:63 +0x213
github.com/bluesky-social/indigo/bgs.(*Slurper).handleConnection(0xc000300990, 0xc000308090, 0x0?, 0xc000323ea8)
/project/indigo/bgs/fedmgr.go:135 +0x117
github.com/bluesky-social/indigo/bgs.(*Slurper).subscribeWithRedialer(0xc000300990, 0xc000308090)
/project/indigo/bgs/fedmgr.go:105 +0x370
created by github.com/bluesky-social/indigo/bgs.(*Slurper).SubscribeToPds
/project/indigo/bgs/fedmgr.go:70 +0x378
Cross-reference to main upstream (atproto) branches: bluesky-social/atproto#658
Things that will require new lexgen feature support:
type="cid-link"
, encodes as an object like { "$link": "<cid>" }
. For binary CIDs, aka "Link" in IPLD data model)type="bytes"
, encodes as an object like { "$bytes": "<base64-encoded>" }
for embedded bytes (which specific base64 config?)type="blob"
now. blob
gains a size
field (integer, size in bytes)type="number"
changed to type="float"
(though not yet used, and might change how that works)type="string" format="at-uri"
The string types are not strictly needed to get lexgen to run, but we probably might as well wire up placeholder validation for functions in codegen code. The formats are at least: handle
, nsid
, at-identifier
(handle or DID), uri
(generic), datetime
, at-uri
, did
, cid
. I can port over the validation functions and their tests. I guess we haven't actually implemented the other validators yet, like maxLength
. Maybe we punt on validation until a second pass (separate PR).
There are some semantics/benavioral changes, like idempotency of batch mutations and deleteRecord
not failing if record already does not exist. I don't think the existing golang PDS is complete enough to care about those.
Once lexgen runs successfully, presumably there will be a bunch of broken stuff that we'll just need to get to compile.
cc: @whyrusleeping
Eg, CLI --version
; HTTP client User-Agent
; etc.
The git invocation I have used before is:
git describe --tags --long --always
Need to take care in docker builds to grab this while .git
is available.
A couple things we should look at for bigsky, in the context of it reaching out and crawling an increasingly large number of hosts on the web.
/xrpc/...
)I want to be involved in this project, where do I start? Aren't there any good first issues?
I've got the following response on this endpoint:
{
"handle": "go-bluesky-tester.bsky.social",
"did": "did:plc:wflozfzpewefv46qof26vbzm",
"didDoc": {
"@context": [
"https://www.w3.org/ns/did/v1",
"https://w3id.org/security/suites/secp256k1-2019/v1"
],
"id": "did:plc:wflozfzpewefv46qof26vbzm",
"alsoKnownAs": [
"at://go-bluesky-tester.bsky.social"
],
"verificationMethod": [
{
"id": "#atproto",
"type": "EcdsaSecp256k1VerificationKey2019",
"controller": "did:plc:wflozfzpewefv46qof26vbzm",
"publicKeyMultibase": "zQYEBzXeuTM9UR3rfvNag6L3RNAs5pQZyYPsomTsgQhsxLdEgCrPTLgFna8yqCnxPpNT7DBk6Ym3dgPKNu86vt9GR"
}
],
"service": [
{
"id": "#atproto_pds",
"type": "AtprotoPersonalDataServer",
"serviceEndpoint": "https://bsky.social"
}
]
},
"collections": [
"app.bsky.actor.profile",
"app.bsky.feed.post",
"app.bsky.graph.follow"
],
"handleIsCorrect": true
}
The indigo parser chokes on it however because the diddoc
does not have a type
field and the parser wants one. Unsure who's at fault here, just that I don't know how to fix it.
Would be helpful for debugging. https://github.com/vi/websocat exists and is partially helpful, but dumps binary CBOR and doesn't handle "paired" objects (header and payload) in any special way.
As some example use-cases, could pipe output though | pv -l > /dev/null
to get an event rate counter; pipe through | rg whatever | jq
for filtering and extracting specific event fields; etc.
I could also implement in adenosine-cli, which i've been using for some development.
Our integration test setup (testing/) currently hardcodes a port for each service it spins up. This is cumbersome and error prone, we should instead listen on port 0 and detect the port the OS chooses for us, then set that field in the struct from there.
Can I haz:
&cli.StringFlag{
Name: "plc",
Value: "https://plc.staging.bsky.dev",
EnvVars: []string{"BSKY_PLC_URL"},
},
in https://github.com/bluesky-social/indigo/blob/main/cmd/gosky/main.go#L155, otherwise https://github.com/bluesky-social/indigo/blob/main/cmd/gosky/util/util.go#L24 is a noop and the command goes boom
$ go run ./cmd/gosky did get @karalabe.bsky.social
Get "/@karalabe.bsky.social": unsupported protocol scheme ""
exit status 1
Looking to use some of the functions via indigo but saw that GraphGetBlocks
seems to be missing. I would expect that function to use app.bsky.graph.getBlocks
(https://atproto.com/lexicons/app-bsky-graph#appbskygraphgetblocks) similar as it is done in the GraphGetMutes
-> https://github.com/bluesky-social/indigo/blob/main/api/bsky/graphgetMutes.go
I saw the GraphGetMutes
is generated based on the comment in the first line but I'm unsure if the missing function is due to cmd/lexgen not being run recently, but it also seems last changes to https://github.com/bluesky-social/atproto/blob/main/lexicons/app/bsky/graph/getBlocks.json were 3 weeks back, when it was last run. (+/- a day maybe).
Eg, if doing a sync method on a full repo download, should get HTTP transport compression. CAR files can get big, and compress quite well. IMO so well that we should do this at the BGS (app) layer, not at load-balancer level.
Might already even be doing this, for free! But need to check. In particular, mimetype "compressible" database might not be configured for the CAR mimetype.
This is an old issue, but assuming it will still be present in pulumi
.
failed to handle op: indexing post: post (41345, app.bsky.feed.post/3jvdilglu7j2n) had invalid timestamp ("2023-05-10T00:05:55.720835Z"): parsing time "2023-05-10T00:05:55.720835Z" as "2006-01-02T15:04:05.000Z": cannot parse "835Z" as "Z"
When creating timestamps we have been doing milisecond precision (three digits after "second"), but it seems reasonable to expect pulumi
to be able to handle longer timestamp precision (at least discarding the extra digits). This is a known ambiguity in the current Lexicon datetime specification.
This was an issue with the atproto
PDS as well, and was resolved by loading the dotenv
stuff before initializing the logger. Maybe we can re-initialize the logger if a dotenv file is found?
Running fakermaker
, in parallel, resulted in this SQL transaction failure:
2023/02/13 15:39:48 /home/bnewbold/bsky/indigo/repomgr/repomgr.go:166 context canceled; sql: transaction has already been committed or rolled back
[228.911ms] [rows:0] INSERT INTO `repo_heads` (`created_at`,`updated_at`,`deleted_at`,`usr`,`root`) VALUES ("2023-02-13 15:39:48.542","2023-02-13 15:39:48.542",NULL,282,"bafyreihnvmu2p27co2trp5pokjolay5x3vmacl7ttfrswe7byvfb47lwem") ON CONFLICT (`usr`) DO UPDATE SET `root`=`excluded`.`root` RETURNING `id`
Making this issue to keep track of any known event stream/sync issues.
null
prev field, despite the repo having pre-existing commitsCurrently, if you do something like getRecord
on a record that doesn't exist, or an unknown XRPC endpoint, you get an HTTP 500 error, without a properly formatted XRPC error reponse (aka, JSON with an error message).
I think the atproto Makefile has been updated to use target "run-dev-pds" for running pds locally but the integrated development section in HACKING.md still needs to be updated accordingly.
Am I missing something? can raise a quick pr for the same
This has only come up in the context of lexicon tests, not in any of our own lexicons. But it feels to me like a correctness issue.
May be partially/entirely an issue in upstream https://github.com/whyrusleeping/cbor-gen
I've seen some occasional test failures recently, both locally and in CI. Re-running usually fixes. I didn't capture a link or output until just now. The output is very verbose, below is just the snip at the end.
2023/02/11 21:16:56 /home/bnewbold/code/indigo/indexer/indexer.go:569 record not found
[0.047ms] [rows:0] SELECT * FROM `pds` WHERE id = 0 AND `pds`.`deleted_at` IS NULL ORDER BY `pds`.`id` LIMIT 1
2023-02-11T21:16:56.079-0800 ERROR indexer indexer/crawler.go:157 failed to perform repo crawl of "did:plc:4475991f29aeeeee": expected to find pds record in db for crawling one of their users: record not found
2023/02/11 21:16:56 /home/bnewbold/code/indigo/indexer/indexer.go:528 record not found
[0.313ms] [rows:0] SELECT * FROM `feed_posts` WHERE (rkey = "3jojbfhdi34m" AND author = (SELECT `id` FROM `actor_infos` WHERE did = "did:plc:5b48e5c9059a135b" AND `actor_infos`.`deleted_at` IS NULL)) AND `feed_posts`.`deleted_at` IS NULL ORDER BY `feed_posts`.`id` LIMIT 1
--- FAIL: TestBGSMultiGap (0.42s)
integ_test.go:190: record not found
FAIL
FAIL github.com/bluesky-social/indigo/testing 5.778s
There doesn't seem to be any documentation and the help is minimal.
There's a command createSession
, that looks like how you log in. So โฆ how? I tried things like
go run ./cmd/gosky createSession -u <username> -p <password>
and
go run ./cmd/gosky createSession --username <username>
but they don't work. When I do
go run ./cmd/gosky help createSession
I just get this:
NAME:
gosky createSession
USAGE:
gosky createSession [command options] [arguments...]
Using ephemeral/parallel postgresql databases (eg, ramdisk). Goal is to get additional confidence that we are testing the same configuration that we deploy in production and staging. Eg, should be easy to test against different versions of postgresql, and surface any possible differences between sqlite and postgresql DB drivers.
Should be fairly simple using containerization in CI.
A related goal is to make it easy to develop and run tests locally.
Trying to have a not-required boolean field on a record, lexgen seemed to work fine, but then I got:
# github.com/bluesky-social/indigo/api/label
api/label/cbor_gen.go:111:30: cannot use t.Neg (variable of type *bool) as bool value in argument to cbg.WriteBool
api/label/cbor_gen.go:288:13: cannot use false (untyped bool constant) as *bool value in assignment
api/label/cbor_gen.go:290:13: cannot use true (untyped bool constant) as *bool value in assignment
This might be an issue for upstream https://github.com/whyrusleeping/cbor-gen
There are a number of things in the lex refactor that we are skipping for now, but should come back around to soon.
$type
field (see below)From conversation with Devin, the times the $type
field needs to be set are:
record
type object, at the top level, with no #
in the type$type = "blob"
)#
indicating the variantThe lexicon refactor comes with better-defined string-based identifier types. These include handle
, nsid
, at-uri
, did
, and others.
These don't change the encoded formats, and the golang lex code doesn't directly need to do anything with these types directly, but it is important to not generate invalid records or other lex objects (eg, query params or request bodies), and it is an opportunity to add type info to our codebase.
Throwing out some ideas of how to work with these types:
CheckNSID(raw string) error
and similar functions in lexutil or similar packagefunc (r *Record) Check() error
style functions for all generated structs, which would call these methods on fields recursivelyParse(raw string) (Identifier, error)
and (ident *Identifier) ToString() string
helpers (though type coercion could also work) and JSON/CBOR marshaling helpers.This last option is basically what I was doing in adenosine (Rust). It makes validation clear (happens only once, when the string is converted), and to-string is cheap. It might make parsing records and handlers a bit slower though, especially in cases where validation doesn't really matter? But I think in all those cases the record just shouldn't be un-marshaled at all.
Keeping a single issue to track these small things:
repomgr.InitNewActor
does some com.atproto
things and some app.bsky
things. would be good to separate those steps. for example, labeling service is not an app.bsky
actor, doesn't need a profile record createdpackage schemagen
with actually-meaningful package names. for example, comatproto
/appbsky
, or just atproto
/bsky
util
or api
as package names, too generic. I think this is a common style guide recommendationkey
to indicate it is a did:key thing. possibly even spin out in to separate repo/packagelex/util
in to util
(or whatever renamed to)api/atproto.go
and api/bsky.go
record types should be deprecated and replaced with api/atproto/*.go
and api/bsky/*.go
types, respectivelypkg/
, and explicitly internal packages to internal/
(maybe opinionated/controversial?). feels to me that this helps with "scaling" as a monorepo (containing multiple commands and libraries) (update: why is against internal/
)cmdtest
or icmd
ATProto
(in api/atproto.go
) and BskyApp
(in api/bsky.go
). Use xrpc.Client
and lexgen types insteadParticularly for beemo, getting some XRPC ERROR 502: 502 Bad Gateway
.
eg, use of jwk
and jwa
. We currently have a mix of the two packages, should use just one.
(I didn't check if the default HTTP client config for gosky
already does this)
beemo
is failing immediately on a single 5xx error, for example with session auth. It would be nice to have a bit of shock absorber behavior where the XRPC client generically does a small number of retries on 5xx errors. Bots and things like beemo probably want to set that even longer (like up to a couple minutes of retries during planned maintenance?).
I find the current package names and layout confusing, even after working with them for a while. I'm also not always sure where to put stuff. Here is a draft proposal for how to re-organize things, particularly coming out of the recent lex refactor.
Would be good to push these through before too many folks start building on this repo.
Could do this in stages, or rip the bandaid all at once. Timing-wise, would like to get labelmaker landed before doing any of the more disruptive refactors.
bgs/
: "big graph server"; command name is bigsky
pds/
: "personal data server"; command name is laputa
labeler/
: "labeling service"; command name is labelmaker
These generally mange state, either on-disk or in a SQL database. Sometimes these align with "ecosystem roles" (a "service" might fulfill multiple "roles").
carstore/
: on-disk repo store (in CAR files), plus SQL database indexingblobstore/
: on-disk file storage, plus SQL database indexingeventmgr/
: handles event subscription from producer side, including persisting the stream (SQL database or other backens) and re-play bufferrepomgr/
: integrates a storage engine, key management, and event generationcrawlmgr/
: rename of indexer/
, though we might be able to split that functionality into eventmgr/
(for single consumption, like gosky, labelmaker, and search service) and bgs/
(for crawling multiple endpoints). might be simplest to keep a single implementation which works with multiple upstream endpointsaggrmgr/
: for parts of indexer
related to persisting notifications, backlinks, etcdidmgr/
(NEW): persisted cache of DID identifies and keys, including both local accounts and remote identities. supersedes previous keymgr
codelabelmgr/
(NEW): not needed to start, but might end up existing if PDS or other services need to persist and access labels from a SQL databaseThese are specific to atproto and might be reused by third parties.
atproto/identifiers/
(NEW): string wrapper types for DID, NSID, at-uri, handles, cid-str (as a string, not parsed) and other Lexicon-defined types. only string validation, not external code/helpers (eg, no DID resolution stuff). many other packages would import thisatproto/did/
(NEW): replaces whyrusleeping/go-did
. parse and represent only the DIDs supported by atproto (did:plc and did:web). no crypto! parse DID docs. possibly clients for doing did:web and DNS lookups. not a general-purpose DID library.atproto/crypto/
(NEW): replaces whyrusleeping/go-did
. parses/generates all the supported atproto key types in all the various formats (did:key, multibase-in-did-doc, hex for signing keys, etc). only works with the curves and formats used in atproto. clear naming for, eg, "HashAndSign" (which does SHA-256 then signs; instead of just "Sign" or "Verify" names)atproto/xrpc/
: generic XRPC client, and possibly some server-side helpers. may also include some subscription (websocket) client and helper codeatproto/repo/
and atproto/repo/mst/
: low-level types and algorithms for working with repo DAG structure. agnostic to storage details.This is kind of bike-sheddy, and i'm not really certain this is the best way to go. But the current api/atproto/
setup, resulting in package name atproto
by default, feels pretty confusing to me.
It really feels like the lexutil
stuff (LexBlob
etc) should live closer to the actual generated lexicons.
lexicon/comatproto/
: the current api/atproto/
. types for com.atproto.*
lexiconslexicon/appbsky/
: the current api/bsky/
. types for app.bsky.*
lexiconslexicon/lexutil/
: the current lex/util
.lexicon/gen
: implementation code for lexgen
fakedata/
: new package with most of fakermaker
code. should be in a non-main package for easier re-use in integration testsplc/
: did:plc
API client; types for encoding and verifying PLC operations; fake/mock/testing server implementation. could go under atproto/did/plc/
?dbmodels/
: only for database models that are actual shared across services (or service components)cmd/
: actual binaries, CLI handling, etctesting/
: any inter-package or high-level tests. open to renaming this (tests/
?)util/
: grab bag stuff that still doesn't fit elsewherecborgen/
or /gen/cbor/
: clearer name for current gen/
. alternatively, use the go:generate
functionality in golang to do this per-package instead of top-levelcmd/gosky/util/
to util/cmdutil/
version/
to util/version/
util/dbcid.go
and util/uid.go
to dbmodels/
util/time.go
to lexutil
. or maybe copy? and possibly rename to, eg, LexDatetime
util/fakekey.go
to didmgr
(or wherever keymgr
ends up)testscripts/
, or at least move under testing/
Right now, at least labelmaker will try to re-connect to the upstream server very aggressively (hundreds/thousands of times per second) if it receives a valid error. It should probably do an exponential backoff thing.
I vaguely remember the BGS doing some backoff when dialing PDS instances, but I could be confused.
cc: @whyrusleeping
Running lexgen
on current atproto lexicons fails on the com.atproto.admin
section:
failed to process schema "/home/bnewbold/bsky/atproto/lexicons/com/atproto/admin/moderationAction.json": schema "com.atproto.admin.moderationAction" doesn't have a main def
Skipping that section, it also fails in com.atproto.report
:
failed to process schema "/home/bnewbold/bsky/atproto/lexicons/com/atproto/report/subject.json": schema "com.atproto.report.subject" doesn't have a main def
These were previously segfaults, I added a check to print an error and bail instead of crashing.
The solution might be to generate types for all defs (not just main
) in some contexts, and just skip generation in other situations? I didn't really try to read the lexgen code and solve this myself yet, didn't want to submit a patch that would break things subtly without understanding.
Running fakermaker
gen-profile
, got an error:
mst.Add failed: value already set at key: app.bsky.actor.profile/self
Fix should include a regression test.
The database event persisters playback loop needs a bit of optimization.
Relevant code here:
https://github.com/bluesky-social/indigo/blob/main/events/dbpersist.go#L128
The easy first pass should be to just pipeline the database loading and the event hydration. Also some amount of parallelism and/or caching in the event hydration is likely warranted too.
Queried with limit=100, got back 108 records, while testing with fakermaker
.
Not a huge priority, but note that this would result in un-boounded return sets, and slow SQL queries, as repos get larger.
I love Cool Tools! There are some nifty existing tools for wrangling IPLD objects, CIDs, etc. It would be nice if we eventually had similar things for atproto repos, XRPC APIs, Lexicons, etc.
This issue is a wiki-like brainstorm of ideas for nifty developer tooling.
MST tree visualizer: prints out a graph of nodes, showing depth, number of entries, leading-zeros, CIDs, etc. Maybe a "compact" form and a "verbose" form? Could be horizontal (matches "classic" way of thinking about trees) or vertical with indentation (easier to dump large trees). Could also export graphviz or something. Bonus points for doing "diffs" between two trees, with some kind of color/visualization. Would be helpful for docs, interop tests, etc.
MST and/or repo to JSON object(s): I think the IPFS/IPLD tools (kubo
?) can already convert a CAR or arbitrary blockstore to JSON files on disk. A possible improvement, for debugging and testing, would be to dump a single giant JSON object, with CID "links" being nested sub-objects. There might already be a known format or best practice for this. Then could use jq
to pretty-print; would map to things like python dicts for integration testing; human-inspect-able format for test vectors. Could include actual CID values. Another mode would allow editing the JSON (invalidating CID links), and recomputing CID links. Maybe a "JSON patch" variant to show diffs between trees.
repo dump to/from folder: for well-behaved repos, a way to take a CAR file and unpack it into DAG-JSON files in filenames/folders which match the MST key names. And a way to parse such a thing, along with a DID and signing key, into a repo CAR file. Should reproduce. Useful for inspecting and sharing test vectors, also for folks to work with their exported CAR files. Maybe a variant would create a .tar.gz
in memory from the CAR, as a parallel export format. Optionally a "standard" way to store any extra metadata and keys in the base directory. Optionally a "standard" way to store blobs (images, etc) in the same directory and .tar.gz
file.
mount repo as FUSE filesystem: DAG-JSON objects at MST key paths. could be local (working on a blockstore or CAR file) or remote (PDS via XRPC). listing, reading, writing should work. Validation errors would be some IO error I guess. Could even mount the full atproto world with DID prefix directory (!). Blob support?
lexicon converters: like pandoc, but for schemas? lexicon-to-... openapi-v3; JSON schema; protobuf interface. Opinionated/controversial?
Some of these could be simple features in PDS implementations.
GraphQL proxy to XRPC: could be codegen or not
RSS feeds to/from bsky: the trivial thing is making author feeds available as RSS (and/or Atom, JSON feed). a general-purpose bot would consume an RSS feed and push to bsky (why's hnbot maybe already does this?)
Micropub interface to PDS: micropub is a relatively simple indieweb HTTP protocol for posting microblog-like content. There are several mobile apps. Much simpler than activitypub, IIRC. Might unlock a small ecosystem of bots, apps, integrations, etc? But might also be extremely niche.
Simple web interfaces to all the CLI things above, works by passing in a DID or ATURI to inspect.
Lexicon verifier: paste in a lexicon, get told if it is valid and if not why. then, in a second box, paste in arbitrary JSON and get feedback on whether it matches the lexicon or not, and if not why, in a human-readable form. Should be able to do this in-browser with typescript implementation, or compile whatever else to WASM. The Rust ecosystem has some nice validators that generate human-meaningful error messages; i'm sure similar things exist in other ecosystems.
DID+ATURI debugger: plug in a DID or ATURI, the web service live attempts to resolve the DID and connect to PDS; for ATURIs also tries to fetch and verify content. Similar to web tools like BGP looking glass; whois lookup; SSL score; "is my email setup (MX/SPF/DKIM/etc) working right". Might need some rate limits?
I'm pretty happy with the CLI "API" (argument structure) for adenosine-cli
, which is basically general purpose com.atproto + app.bsky CLI: https://gitlab.com/bnewbold/adenosine/-/blob/main/extra/adenosine.1.md
I could be wrong about this, but I think we have a problem with lexgen-erated client functions.
For example: https://github.com/bluesky-social/indigo/blob/main/api/atproto/admingetModerationReports.go#L20
func AdminGetModerationReports(ctx context.Context, c *xrpc.Client, cursor string, limit int64, resolved bool, subject string) (*AdminGetModerationReports_Output, error) {
limit
and resolved
query parameters here are optional. For resolved
specifically, "true", "false", and "no param supplied" are all distinct behaviors.
I'm not sure if just *bool
as the type would resolve this or not.
decoding xrpc response: unrecognized type: ""
Does not have to happen on every CI run, but want to start making it possible to run, eg, fakermaker
and bigsky
against current (main
branch) PDS/PLC/etc from atproto
repo.
Probably using containers. Maybe in the form of special indigo golang tests behind a flag.
getting record: XRPC ERROR 400: 400 Bad Request
Not all endpoints and features are implemented in the current PDS. Some notable missing things (turned up by fakermaker
test; you can all search for panic
under pds/handlers.go
).
Some notable/priority features:
This issue is a list of things i'm intending to review and test in the indigo MST and repo/repomgr implementations. I'll also be cross-referencing and expanding from the atproto
(Typescript) and adenosine
(Rust) implementations for cross-language interop.
Existing tests:
When receiving/validating external MST structures:
Edge cases:
//
)Mostly handled by IPLD libraries, but we have some additional constraints, and also have some CID string fields in records that could (should?) be validated.
Invalid in string form:
Places to check:
Stricter constraints on generally valid CIDs:
Presumably there is a validator function, which might just be a regex. It should check:
If we do want to support puny code and internationalized domains, will need converter to/from display form, with tests.
Generally, should verify:
And more specific constraints in atproto, for now:
Similar to handles and DIDs, should have tests for weird/mangled ATURIs.
Lexicon should catch most things, but some additional checks, specifically for post records.
Basically, just stick a bunch of interesting strings and interesting places and ensure they are handled correctly.
Probably handled at the load-balancer level:
/xrpc/../blah
(relative path chars)/xrpc//blah
(double slash)/xrpc/blah/
(trailing slash)At application layer (XRPC server generically):
indigo uses go-did. Currently, go-did use cgo module ipsn/go-secp256k1
. So this is not easy to build arm64 binary release on GitHub Actions. I created pull-request to switch another pure Go module for go-did. Could you please take a look?
With fakermaker
, got an error:
HANDLER ERROR: (/xrpc/app.bsky.notification.list) attempted to hydrate unknown notif kind: 2
The "mentioned" notification kind is commented out and un-implemented in notifs/notifs.go
.
gosky list --all
works on most of my repo, but some values don't parse:
% gosky list --all --values $(gosky handle resolve bradfitz.com) | grep -A 1 panic
panic: unreachable
bad blob: a5652474797065766170702e62736b792e6163746f722e70726f66696c6566617661746172a463726566d82a58250001551220cb04502240be648abdc77c7e9a5ef1750a2b0943ab6bb5f6b4fc7a16dc59092a6473697a651a0001afd965247479706564626c6f62686d696d65547970656a696d6167652f6a7065676662616e6e6572a463726566d82a5825000155122050509cd8bd1f3e4834eb5d369c712d6bd84475a795d5f309fbd26dabc07476386473697a651a0002b1fc65247479706564626c6f62686d696d65547970656a696d6167652f6a7065676b6465736372697074696f6e78b54920646f20636f6d7075746572732e204d61727269656420746f20407261646b61742e6669747a7061742e636f6d2e205468726565206b6964732e20586f6f676c65722e200a476f202823676f6c616e6729207465616d20323031307e323032302e204d616465204c6976654a6f75726e616c2c204f70656e49442c206d656d6361636865642e2043757272656e746c79206174205461696c7363616c65206d616b696e672057697265477561726420656173792e6b646973706c61794e616d657042726164204669747a7061747269636b
--
panic: unreachable
bad blob: a5647465787478254c6f6f6b73206c696b6520746869732077617320796f757220666c696768743f2046756e21652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a58250001551220773ea1e3c22c394429a71709b216ded26d0c939890351ddd37bbdb45dee5ad306473697a651a000f140065247479706564626c6f62686d696d65547970656a696d6167652f6a706567657265706c79a264726f6f74a263636964783b62616679726569673277653373676d3663653278356e666a75717a6171327169653571666c7963366b7068777a376368377535736632756366666163757269784661743a2f2f6469643a706c633a6966327475673562756335613363727a32643269323469332f6170702e62736b792e666565642e706f73742f336a74713371376235657a326d66706172656e74a263636964783b62616679726569673277653373676d3663653278356e666a75717a6171327169653571666c7963366b7068777a376368377535736632756366666163757269784661743a2f2f6469643a706c633a6966327475673562756335613363727a32643269323469332f6170702e62736b792e666565642e706f73742f336a74713371376235657a326d696372656174656441747818323032332d30342d32345431373a31383a32332e3739385a
--
panic: unreachable
bad blob: a4647465787464f09f9299652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a5825000155122055fcd00bd248cdc46dfed852b93aab637706e8daf716c43504da1dc02ad71ed96473697a651a000d4fa565247479706564626c6f62686d696d65547970656a696d6167652f6a706567696372656174656441747818323032332d30342d32345432323a30363a30312e3738355a
--
panic: unreachable
bad blob: a56474657874787e4f682c206e6963652c206d7920667269656e642040627261646669747a2e69732e74657374696e672e686f772e62656175746966756c2e6f662e612e757365726e616d652e796f752e63616e2e686176652e302e302e662e652e622e662e322e302e362e322e6970362e61727061206a6f696e656420426c7565736b7921652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a58250001551220526572a012f96be8743f6388062b9127a0fc332b8f461ce29c85cd42a7e32d636473697a651a0004e8e165247479706564626c6f62686d696d65547970656a696d6167652f6a7065676666616365747381a3652474797065776170702e62736b792e72696368746578742e666163657465696e646578a26762797465456e64186e696279746553746172741468666561747572657381a26364696478206469643a706c633a736637786a68736a3461666368326769366e743633636c65652474797065781f6170702e62736b792e72696368746578742e6661636574236d656e74696f6e696372656174656441747818323032332d30342d32355430333a35303a31312e3731315a
--
panic: unreachable
bad blob: a5647465787478b45475726e73206f7574206d792077696665206f66206e6561726c7920736576656e2079656172732c20407261646b61742e6669747a7061742e636f6d2c206469646e2774206b6e6f772074686174206d7920706572736f6e616c207765627369746520697320627261646669747a2e636f6d2e0a0a5468616e6b7320426c7565736b7920666f72206c657474696e67207573206c6561726e206d6f72652061626f75742065616368206f746865722120f09f9882652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a58250001551220e71409e2ab359e01241ccc7705b84ecac6ff638e11e071ed8cbd7211e61cf6516473697a651a0001362f65247479706564626c6f62686d696d65547970656a696d6167652f6a7065676666616365747382a3652474797065776170702e62736b792e72696368746578742e666163657465696e646578a26762797465456e64183c69627974655374617274182968666561747572657381a26364696478206469643a706c633a6f6b6b33656b7575736b7062736476777972616e72726c71652474797065781f6170702e62736b792e72696368746578742e6661636574236d656e74696f6ea265696e646578a26762797465456e64187269627974655374617274186668666561747572657381a2637572697468747470733a2f2f627261646669747a2e636f6d652474797065781c6170702e62736b792e72696368746578742e6661636574236c696e6b696372656174656441747818323032332d30342d32355431373a35393a34372e3739335a
--
panic: unreachable
bad blob: a4647465787468f09f9299f09f8cb8652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a58250001551220ee70559a326bc006796f3371d9dece0f31c318bf45569ad87d1445cbb9b79d486473697a651a000ee56a65247479706564626c6f62686d696d65547970656a696d6167652f6a706567696372656174656441747818323032332d30342d32355432303a35353a30382e3538395a
--
panic: unreachable
bad blob: a564746578747873546f64617920406361747a6b6f726e2e62736b792e736f6369616c20706f73746564207468697320696e206f6e65206f66206f757220766172696f757320536c61636b2073686974706f7374696e67206368616e6e656c7320617420407461696c7363616c652e636f6d202e2e2e20f09f9882652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a582500015512205b3dbac27e31007345f2b54ca68dc5b6430a245a5a814c60494c59992318cf296473697a651a0006220365247479706564626c6f62686d696d65547970656a696d6167652f6a7065676666616365747382a3652474797065776170702e62736b792e72696368746578742e666163657465696e646578a26762797465456e64181b696279746553746172740668666561747572657381a26364696478206469643a706c633a7a6e6f71776568777369616563656f757534747471366277652474797065781f6170702e62736b792e72696368746578742e6661636574236d656e74696f6ea3652474797065776170702e62736b792e72696368746578742e666163657465696e646578a26762797465456e64186a69627974655374617274185c68666561747572657381a26364696478206469643a706c633a746b633378697961346b37757978336a7365693362703635652474797065781f6170702e62736b792e72696368746578742e6661636574236d656e74696f6e696372656174656441747818323032332d30342d32365431383a34393a30392e3833375a
--
panic: unreachable
bad blob: a46474657874781a4f682c20746865206a6f7973206f6620696e66616e74732e2e2e652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a582500015512202bf480f10f5fd233413a06177473f6c4d2d6ea40a8bfbf9122b675067a24945d6473697a651a000d21f565247479706564626c6f62686d696d65547970656a696d6167652f6a706567696372656174656441747818323032332d30342d32375431373a30393a31322e3038345a
--
panic: unreachable
bad blob: a4647465787479012e4920686164206120636f6e74726163746f72206f757420646f696e672061206d61696e74656e616e6365207669736974206f6e206f757220686561742070756d70732e2048652068616420746f20676f207468726f756768206d79206f666669636520746f2067657420746f2074686520726f6f6620756e697420616e6420667265616b6564206f7574207768656e20686520736177206d79206465736b2c20667265657a696e6720616e642073746172696e67206174206d652c2061736b696e673a0a0a2257687920646f20796f7520686176652061206b6e696665206f6e20796f7572206465736b3f213f220a0a492068616420746f206578706c61696e20776861742061206c6574746572206f70656e6572206973202620686f7720697420776f726b65642e20f09f9882652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a582500015512201d115053c4bf8de9fc6bb243a5b23e82ac20d4f01dfce4ae31a71a0570752b3b6473697a651a000e383a65247479706564626c6f62686d696d65547970656a696d6167652f6a706567696372656174656441747818323032332d30342d32375431383a33373a34332e3735315a
--
panic: unreachable
bad blob: a4647465787478d1496620796f75206e65656420746f207761726e206d6520696e20616476616e6365207768656e20796f752068616e64206d652074686520515220636f6465206d656e7520746861742069742070726f6261626c7920776f6e2774207363616e2c206d6179626520697427732074696d6520666f72206e657720515220636f646573207468617420617265206d6f726520626f72696e673f0a0a427574206f6e207468652062726967687420736964652c207468697320776561746865722120416e64206265657221204f7574736964652e652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a582500015512202ba9760cd191f411c3a8f31700c4ced2cd7a0f61a4e3e0d0c74488b05b3732aa6473697a651a000eecbb65247479706564626c6f62686d696d65547970656a696d6167652f6a706567696372656174656441747818323032332d30342d32375432323a32313a32372e3036335a
--
panic: unreachable
bad blob: a5647465787479012c49207361772070656f706c6520686572652064697363757373696e6720656e6a6f79696e67206c696b696e6720227468697273742220706f737473206f6e207768696c65206c696b65732061726520707269766174652e0a0a4265206361726566756c2e2054686579277265206e6f742e20546865207765622f61707073206a75737420646f6e27742073686f7720796f752c206275742069742773207075626c69632e0a0a652e672e2048657265206172652040616e696c646173682e636f6d2773206c696b65732e204c6173742055524c2069730a0a68747470733a2f2f73746167696e672e62736b792e6170702f70726f66696c652f6469643a706c633a6b646a6d336d7776627662766171746236723369743571712f706f73742f336a75666d7276677069753270652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a582500015512200a15cfd7eb305eef376e743752c05754c6bbcabf28c801aac0d347dc49cd067e6473697a651a000ba21f65247479706564626c6f62686d696d65547970656a696d6167652f6a7065676666616365747382a3652474797065776170702e62736b792e72696368746578742e666163657465696e646578a26762797465456e6418c16962797465537461727418b468666561747572657381a26364696478206469643a706c633a73673265326b71647364703271327a6c3434747861646270652474797065781f6170702e62736b792e72696368746578742e6661636574236d656e74696f6ea265696e646578a26762797465456e6419012c6962797465537461727418d868666561747572657381a263757269785468747470733a2f2f73746167696e672e62736b792e6170702f70726f66696c652f6469643a706c633a6b646a6d336d7776627662766171746236723369743571712f706f73742f336a75666d7276677069753270652474797065781c6170702e62736b792e72696368746578742e6661636574236c696e6b696372656174656441747818323032332d30342d32385430333a32383a32362e3832355a
--
panic: unreachable
bad blob: a5647465787460652474797065726170702e62736b792e666565642e706f737465656d626564a2652474797065756170702e62736b792e656d6265642e696d6167657366696d6167657381a263616c746065696d616765a463726566d82a582500015512202751a5b99841ef99b3b195e750cb2f6e8b8f47b12f5b99781a09326991f30b076473697a651a000142b465247479706564626c6f62686d696d65547970656a696d6167652f6a706567657265706c79a264726f6f74a263636964783b6261667972656968716b62707236796171376a6c617a666637636763336863326468706a6b616b79677765723362657a676f7869786d763567356163757269784661743a2f2f6469643a706c633a377232667933623475376d6d6e68676264786e666c6f76762f6170702e62736b792e666565642e706f73742f336a75666f636a6763716e327066706172656e74a263636964783b626166797265696470656a3261736b737566716f79647668787736796a66376d33756c71616336787a6174666177666a696e377970356c6977336163757269784661743a2f2f6469643a706c633a3634727976757271777a72366c6a6e3576376c776e696e682f6170702e62736b792e666565642e706f73742f336a75666f6b32656468353270696372656174656441747818323032332d30342d32385430333a33353a32332e3136385a
We aspire to support the full IPLD data model, even if floats are considered a poor practice in content-addressed systems.
Upstream (cbor-gen) issue: whyrusleeping/cbor-gen#80
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.