matrix-org / matrix-appservice-bridge Goto Github PK

Because otherwise is breaks metrics by appending "all" to the list of periods and crashing everything.

In Discord, IRC and Gitter we have 3 separate logging systems that effectively do the same thing, and when I come to modify the slack bridge I will inevitably copy and paste the Gitter code.

We should pick one of the implementations and create a module for it in here so that we can reduce the LoC on those projects and also get more consistent logging for other bridges. We can also let people opt out of using it in favour of console if they don't want the extra headroom.

Almost all of the bridges, except IRC, support uploading images from the 3PN to Matrix. Each of these bridges copypastes the same basic code shape to upload an image to the media repo from some sort of in-memory Buffer object, obtaining the mxc:// URL, and post that into a room as an m.image message.

These could be neatened up a bit with a helper method on the Intent objects themselves.

I pulled this list of used calls from the matrix-appservice-irc. If we want to migrate over to matrix-bot-sdk, we will want to support this for an easy migration path.

.roomState()
.uploadContent()
.createAlias()
.setRoomDirectoryVisibilityAppService()
.sendStateEvent()
.getStateEvent()

So there are a few actions I can use the intent for, where the bridge will not check to see if the user exists like uploadContent. Is that by design or should this be checked for all functions. I've hit a few subtle bugs where by pure chance, the first thing a user tries to do is upload some content which catches things out.

Given the nature of bridges requiring connections to be held open in some cases, a hook for when the config changes would be very useful.

Unsure yet if we want to trigger this via filesystem watch, a http api or signals.

We only define the port that the AS listens on, and express just uses the default of 0.0.0.0

TL;DR: bridge sets up app quite late meaning it's complicated to rely on it in setup code.

I want to add a method to Bridge that constructs and returns a PrometheusMetrics instance, having called the setup code to attach it to the /metrics path in the app instance. Because that app instance doesn't exist yet before the run method is called, this can't quite be self-contained. Instead I have to store the metrics instance and run the addAppServicePath method later at run time.

This logic is further complicated by the fact I don't really know when the caller will be asking for this metrics object. Maybe run has already been called so actually I can (and should) register it with app now, because I won't get another chance later.

Given we should know whether we registered a user one time and it's unlikely to have ever changed since we shouldn't be checking all the time.

Hello, I'm using the following config to run a self-hosted matrix with postgres from https://github.com/spantaleev/matrix-docker-ansible-deploy

I would like to know if you can add a support for postgres.

I'm able to make the bridge & migrations work but nothing happen when I invite the bot @_webhook:<my_domain>

I suspect it come from the src/WebhookBridge.js where I commented userStore & roomStore because it was using a sqlite path.

        this._bridge = new Bridge({
            registration: this._registration,
            homeserverUrl: this._config.homeserver.url,
            domain: this._config.homeserver.domain,
            // userStore: process.env["WEBHOOKS_USER_STORE_PATH"] || "user-store.db",
            // roomStore: process.env["WEBHOOKS_ROOM_STORE_PATH"] || "room-store.db",

Note: I was able to invite the bot to a room using sqlite.

I can see The user store instance to use in bridge.js

 * @param {(UserBridgeStore|string)=} opts.userStore The user store instance to
 * use, or the path to the user .db file to load. A database will be created if
 * this is not specified.

What is the a user store instance? (tried with the table name: users & rooms but didn't work)

A homeserver is able to request third party lookup information without requiring an access_token.

This will break existing stuff, also a spec PR should be made first.

The port number that the bridge runs as is given, somewhat bizarrely, as a construction parameter to the Cli object. Other construction parameters here tell it how to read the config file. It's only when you get to the run function that you have access to the details given in the config file, which is somewhat too late to apply a port number configured there.

This is inconvenient.

Created by @ kegan:matrix.org.

The CLI needs to make it absolutely clear which URL is the HS and which URL is the AS. I've tried to clarify bits along the way but there's still confusion, maybe remove the shorthand flag form?

A call to unlinkRoomIds() appears to succeed, but the on-disk database hasn't been altered, and still contains the link. So on next startup of the bridge, it starts running it again.

The code sits at:

https://github.com/matrix-org/matrix-appservice-gitter/blob/master/index.js#L185

The rest of the matrix.org world is moving to buildkite, and we should move our tests there.

Which isn't true, you can be invited even if you don't exist in Synapse. Boo :(

Would be useful to get messages from before we got joined to a room, for instance.

This would be best handled by watching for tombstone events and firing a handler to deal with it, as it will likely need manual intervention by the bridges themselves.

As per #100 (review) (with surrounding context).

Created by @ matthew:matrix.org.

Currently I have some hideous monkeypatching:

matrix-org/matrix-appservice-gitter@19aaa4c#diff-73f5f6969c2293ed1df0158153678a6aR23

matrix-org/matrix-js-sdk#1000

Created by @ matthew:matrix.org.

see https://vector.im/develop/#/room/!SudviOJlimDvrGdFCY:matrix.org/$146245477289427yysKn:matrix.org

Is there a way to understand the status of user accesing from brigde and directly from IRC?

Seems unknow and offline means disconnected for IRC, but is threre any difference?

It would be very helpful if we can have TypeScript typing definitions with the npm module.

And It would be fantastic if we could be able to rewrite the whole project from JS to TS. ;-)

UPDATE July 7, 2019

@Half-Shot

I had spent around two full days to study the document and write the typings for matrix-appservice-bridge.

With the typing system, it helped me a lot to learn the API and prevent lots of potential bugs. I think it's worth to publish them, the best will be integrated into our npm module.

The typing definition is still working in progress and heavy developing, you can watch it at https://github.com/huan/matrix-appservice-wechaty/blob/master/src/typings/matrix-appservice-bridge.d.ts.

I'll create a pull request to add the typing to this repository after I get my project done, and any suggestion will be welcome by leaving comments under this issue.

Link to wechaty/matrix-appservice#13

Thanks!

It appears that this option

Some of our metrics (looking at you IRC bridge) can take some amount of ms to collect. We should optionally call this func and await the promise rather than calling synchronously.

Hey all, just installed 1.6.1 from npm and npm audit shows 19 vulnerabilities.

I'm unable to fix them automatically and they require manual attention. It appears that most if not all can be fixed with version updates from upstream modules.

Similar to #56, almost every bridge besides IRC can transport images from Matrix into the 3PN, and in each case has to copypaste the same lines of code to perform an HTTP(S) download from the media repo, by extracting bits of the mxc:// URL and turning it into a regular https:// URL for its own media repo. This could also be neatened up by putting it in the Intent object itself.

See

https://vector.im/develop/#/room/!zOmsiVucpWbRRDjSwe:matrix.org/$147438129821098UXnET:matrix.org

https://github.com/matrix-org/internal-config/commit/160598b99e4e15c5b92234e670cd54ea3af5e567

and any details about them. Essentially this is just like the other stores. We would also include a source key to tell us if the event came from the remote or matrix side.

Keys required are:

• Matrix Room Id
• Remote Room Id
• Matrix Event Id
• Remote Event Id
• Source
• Data (could be sender or something it we cared about senders)

Which leads to confusion and panic. We should probably pass on https://github.com/matrix-org/matrix-appservice-bridge/blob/master/lib/components/intent.js#L732 to the other functions

Hi,

Thanks of providing such base for creating bridge.

I tried to use Discord / Matrix bridge (https://github.com/Half-Shot/matrix-appservice-discord) but it didn't work because I have a self-signed certificate on my HS.

When I run the discord bridge, I faced to that error on the bridge side:

PUT https://exemple.org/_matrix/client/r0/presence/%40_discord_107486792604712960%3Aexemple.org/status (@_discord_107486792604712960:exemple.org) HTTP null Error: {"code":"DEPTH_ZERO_SELF_SIGNED_CERT"}

Any clue on how to by pass this check ?
Kind regards

This should obviously be configurable to what point we want to retry.

https://www.npmjs.com/package/prom-client

Seems to be

• Better engineered
• Already supports all the standard prometheus process-wide metrics, and a bunch of node-specific ones
• Supports Histograms and Summaries

This latter feature is relevant to matrix-org/matrix-appservice-irc#296 (comment)

A result of lots of debugging on my slack bridge (using current develop version of matrix-appservice-bridge), the bridge's own bot intent sometimes gets confused into sending messages out claiming to be one of its ghost users. This results in synapse errors:

[-] PUT http://localhost:7680/_matrix/client/r0/rooms/!dcwkeyJKpnOhTPFAWk%3Alocalhost%3A8080/send/m.room.message/m1473177931754.1 (@slack_leonerd:localhost:8080) Body: {"body":"@leo:localhost:8080:  -r=!vvBXjnizurxkUPkLLv:localhost:8080","msgtype":
[-] PUT http://localhost:7680/_matrix/client/r0/rooms/!dcwkeyJKpnOhTPFAWk%3Alocalhost%3A8080/send/m.room.message/m1473177931754.1 (@slack_leonerd:localhost:8080) HTTP 403 Error: {"errcode":"M_FORBIDDEN","error":"User @slack_leonerd:localhost:8080 not in room !dcwkeyJKpnOhTPFAWk:localhost:8080 (None)"}
Unhandled rejection M_FORBIDDEN: User @slack_leonerd:localhost:8080 not in room !dcwkeyJKpnOhTPFAWk:localhost:8080 (None)

Fixed by #74

This makes certain kinds of startup code trickier than they ought to be.

matrix-org/matrix-appservice-gitter@19aaa4c#diff-73f5f6969c2293ed1df0158153678a6aR40
matrix-org/matrix-appservice-gitter@19aaa4c#diff-73f5f6969c2293ed1df0158153678a6aR573

Because if it's done in the constructor then bridge.getIntent() returns null.

Because for IRC purposes, we want to track users who have been active months ago. A day isn't much use.

This is less of a bug and more of a list of things (for my own sanity if nothing else) to extract from the current near copy-pasta mess of matrix-appservice-{slack,gitter} and put in the bridge library.

Entire files that can be copied (near-)verbatim

• lib/AdminCommand.js
• lib/MatrixIdTemplate.js from matrix-appservice-gitter
• lib/MatrixUser.js
• lib/Metrics.js
• lib/Provisioning.js

Files that vary in implementation between Slack and Gitter but could share a common base class:

• lib/BridgedRoom.js
• lib/SlackGhost.js and lib/GitterUser.js

Then finally we come to the big one; all of the various parts of lib/MatrixSlackBridge.js and lib/MatrixGitterBridge.js that can be pulled out:

• automatic setup and use of StateLookup
• setup and delegation methods for metrics counters
• automatic counting of bridge "intent" objects
• storage/metric counting/lazy construction of MatrixUser and {remote}User objects
• storage/metric counting/lazy construction of BridgedRoom instances, keyed bidirectionally
• counting of Matrix client API calls
• returning a full http URL on the homeserver for a given mxc
• utility methods like listGhostUsers, drainAndLeaveMatrixRoom, listRoomsFor
• Matrix message routing and handling for admin room
• Checking user powerlevels for permission to execute a provisioning request

As per louischatriot/nedb#492.

It's receiving bug fixes, but these seem to be infrequent.