shuppet / raku-api-discord Goto Github PK

Objects need to know when they have not been fully populated. This can happen when the API sends us a subset of a hash instead of a full object hash.

This knowledge would also help us determine whether an object is new and thus needs to be created on Discord's end.

We could probably use the same or similar mechanism to know whether we need to update any fields, and which ones, thus enabling us to send PUT (or PATCH if that's an option) instead of a POST, which would help with #16.

We can also use this for lazy loading since we can populate an object with just its ID (for example) and let it fetch the rest of the data as needed.

If we ask a channel to fetch N messages it will add it to to the channel's internal array, but a user might want to just fetch a bunch of messages from the channel without caching them on the channel.

Here's a simple Discord bot. It prints out when somebody adds a reaction to the message.

#!raku

use API::Discord;

sub MAIN(Str $token) {
    my $discord = API::Discord.new(:$token);

    $discord.connect;
    await $discord.ready;

    my $channel = $discord.get-channel(465122426221756426);
    my $message = await $channel.send-message("React to me");

    react {
        whenever $message.events -> $r {
            $r.say;
        }
    }
}

And I react to it with 👌

« ♥
» ♥
All guilds ready!
{d => {channel_id => 465122426221756426, emoji => {id => (Any), name => 👌}, guild_id => 269438629775015936, member => {deaf => False, hoisted_role => 269513102712635393, joined_at => 2017-01-13T12:14:04.372000+00:00, mute => False, nick => (Any), premium_since => 2019-12-09T11:16:52.299000+00:00, roles => [269513102712635393 653555662445608969], user => {avatar => 0eabcd875d9a0fce4a194f7c58dec5a3, discriminator => 2319, id => 240882564770955275, username => altreus™}}, message_id => 693517208554569779, user_id => 240882564770955275}, op => 0, s => 4, t => MESSAGE_REACTION_ADD}

Now we have the bot react to the message

    $message.add-reaction('☺');

And we hear about the bot's reaction in the react, even though it's already happened:

« ♥
» ♥
All guilds ready!
{d => {channel_id => 465122426221756426, emoji => {id => (Any), name => ☺}, guild_id => 269438629775015936, member => {deaf => False, hoisted_role => (Any), joined_at => 2020-03-26T22:27:02.164045+00:00, mute => False, nick => (Any), premium_since => (Any), roles => [692871222450585610], user => {avatar => (Any), bot => True, discriminator => 2915, id => 692861210659389472, username => vurl}}, message_id => 693518655128076299, user_id => 692861210659389472}, op => 0, s => 4, t => MESSAGE_REACTION_ADD}
{d => {channel_id => 465122426221756426, emoji => {id => 675035435302387785, name => eng101}, guild_id => 269438629775015936, member => {deaf => False, hoisted_role => 269513102712635393, joined_at => 2017-01-13T12:14:04.372000+00:00, mute => False, nick => (Any), premium_since => 2019-12-09T11:16:52.299000+00:00, roles => [269513102712635393 653555662445608969], user => {avatar => 0eabcd875d9a0fce4a194f7c58dec5a3, discriminator => 2319, id => 240882564770955275, username => altreus™}}, message_id => 693518655128076299, user_id => 240882564770955275}, op => 0, s => 5, t => MESSAGE_REACTION_ADD}

We've tried this with sleeps in it and it still returns all historical emoji, even though the supply should be live.

I've been putting off behavioural tests - integration tests, I guess - because I don't know how to achieve it.

What we really need to be able to test is how the system behaves in various scenarios from Discord itself.

The prime target for that is connectivity. We have been having trouble with reconnecting (#41) after unknown events (net drop? forced disconnection?) and so we want to have a test library that will pretend to be Discord for the WebSocket class.

• Faux Discord server for integration tests
• E2E testing of scenarios:
  • Message sent/received
    • including embeds
  • Reactions added/removed
  • Member joins/leaves
• Unit tests for all classes

More to be added as I think of it

We should add a disconnect method which sends a 1000 close code to the websocket. This will allow for graceful shutdown of applications and prevent zombie connections while the heartbeats time out.

@gfldex asked about this in the Discord, just making a note of it here for later.

Right now it is impossible to install the module, trying out latest version locally:

➜  p6-api-discord git:(master) ✗ zef install .
===> Testing: API::Discord:ver<0.0.1>
===SORRY!===
The following packages were stubbed but not defined:
    API::Discord::Endpoints::API::Discord::HTTPResource
Other potential difficulties:
    Useless declaration of a has-scoped multi-method in module (did you mean 'my method format'?)
    at /home/koto/Devel/p6-api-discord/lib/API/Discord/Endpoints.pm6 (API::Discord::Endpoints):34
    ------> multi method⏏ format(Str:D: API::Discord::HTTPResourc

===> Testing [FAIL]: API::Discord:ver<0.0.1>
Aborting due to test failure: API::Discord:ver<0.0.1> (use --force-test to override)

Before that I also had to replace Websocket in dependency section to WebSocket(well, partially I am at fault here, because I missed it on first sight).

You can try out if the module can be installed using zef install . from repo directory. Thanks.

We need to handle events:

• New message
• Deleted message
• Edits to all types
• Other deletions
• New member
• Member leaving

This might be facilitated by creating a bunch of supplies that individual objects can tap into... see #20

So we don't have to understand the JSON structure of the messages we should transform them into meaningful objects, or at least have functions with meaningful names to process them.

the API object caches guilds and channels but these don't necessarily work correctly

  whenever $discord.events -> $event {
    if $event<t> eq 'TYPING_START' {
      my $channel = await $discord.get-channel($event<d><channel_id>);
      await $channel.trigger-typing;
    }
  }

Crashes with:

Died because of the exception:
    An operation first awaited:
      [...]                                                                     

    Died with the exception:
        Server responded with 411 Length Required
          in block  at [...]/BD9BD084131C6E7D2BF4FE09570E5EBE604DB8FA (Cro::HTTP::Client) line 410    

The problem seems to be that the request doesn't contain a Content-Length header:

The HyperText Transfer Protocol (HTTP) 411 Length Required client error response code indicates that the server refuses to accept the request without a defined Content-Length header.

I built P6 from the master using this script:

cd ~/source/rakudo && git checkout master && git pull &&\n git checkout $(git describe --abbrev=0 --tags) &&\n perl Configure.pl --gen-moar --gen-nqp --backends=moar &&\n make && make install

(Sadly I don't know when I did that last time, but it was definitely also 2018.12 built from master.) After that, my bot that worked fine before starting throwing some errors on startup:

5 (/home/throwaway/source/discord/bot.p6 line 9)
my $api = API::Discord.new(:$token)
6 (/home/throwaway/source/discord/bot.p6 line 11)
await $api.connect
Use of uninitialized value of type Any in string context.
Methods .^name, .perl, .gist, or .say can be used to stringify it to something meaningful.
  in block  at /home/throwaway/source/discord/../p6/p6-api-discord/lib/API/Discord/Connection.pm6 (API::Discord::Connection) line 146
Use of uninitialized value of type Any in string context.
Methods .^name, .perl, .gist, or .say can be used to stringify it to something meaningful.
  in method handle-message at /home/throwaway/source/discord/../p6/p6-api-discord/lib/API/Discord.pm6 (API::Discord) line 184
Use of uninitialized value of type Any in string context.
Methods .^name, .perl, .gist, or .say can be used to stringify it to something meaningful.
  in block  at /home/throwaway/source/discord/../p6/p6-api-discord/lib/API/Discord.pm6 (API::Discord) line 172

Also API::Discord.messages does not trigger anymore, no matter what's happening in the chat. Other things like .trigger-typing still seem to work, although those are triggered by a timer, not as reaction to some discord event. The whole bot can be found here: https://github.com/throwaway-30964/infinite-garbage/blob/master/bot.p6 (it basically takes an input file with text and generates random messages based on it.) Interestingly the basic example from the README (https://github.com/shuppet/p6-api-discord/blob/master/examples/echo-server.p6) seems to work just fine, so maybe some of the other things in the script might cause the problems, although I don't see how and it worked before the update, so maybe a bug in rakudo itself?

We have seen resume work once or twice but most of the time we get an error from somewhere or other. This improved when we fixed the heartbeat promise (albeit in a way that seems incorrect), but it is still unreliable. Is it us, or is it rakudo star 2018.10 at fault?

This is a test issue.

In HTTPResource, fetch behaves differently. Arguably, correctly.

I think I would prefer if all HTTPResources returned themselves rather than exposing the Cro-level HTTP communication.

Currently if you want to do stuff with member permissions you must explicitly specify use API::Discord::Permissions; in your code which seems awfully inconvenient for a feature that the majority of bots will actually want to use.

I suggest we make it an extension of API::Discord::Member, as permissions never apply to user objects, only members.

Will it supports voice chats?

Whenever we don't receive a heartbeat we should reconnect.

On the branch refactor-comms I have introduced a WebSocket class, whose job it is to marshal WS messages into the main Connection class. This is supposed to use a migrating Supply so that when a reconnection takes place, the WS class simply emits a new message supply, and the rest of the code should continue.

It appears that it:

• Fails to receive a heartbeat
• Destroys the current WS
• Creates a new WS, with the correct sequence number
• Connects to Discord and sends the resume payload
• Receives nothing from it

I am unable to prove that this is not my fault - mostly because it probably is. It's hard to believe that Discord is doing it wrong. But I cannot ascertain what part of the mechanism is failing.

It would be good to have #40 going, so that we can use a fake Discord to force the disconnection.

There are a few places where we currently just parse JSON and send the hash all the way to the user. We should make sure JSON responses are turned into meaningful types ASAP!

Check we don't have any typeless hashes lying around. We might even be able to use punning just to decorate a normal hash with a type.

This might be a bad idea but we should experiment with it.

One model for the API to update itself would be to have all objects listen to the events supply on the API object and handle those events that are relevant to them. This is not terrible but it does mean discarding lots of things lots of times, since the majority of events will not be relevant to the listener.

The other model is to have the API re-despatch the event to those objects to which it is relevant, by means of a supply on each object that the object can listen to. Essentially, whenever we create an object by inflation, we create a supply on the object and attach it to the main event supply via some magic or other. waves hands

The difficulty might be in new objects that we POST in the first place. We would have to attach this supply after we learn that the new object was successfully created, so that the object updates itself in future.

It might also be difficult to handle a delete event properly so we may have to refactor a few things.

The bot can be rate limited and Discord will tell us when we're allowed to send another request.

We don't know what to do about it because although we can probably enqueue a whole bunch of unsent messages, if we're being rate limited, this is unlikely to drain after any reasonable time.

Maybe we drop them? First up we need to honour that response and throttle ourselves.

When identifying to the gateway, you can specify an intents parameter which allows you to conditionally subscribe to pre-defined "intents", groups of events defined by Discord. If you do not specify a certain intent, you will not receive any of the gateway events that are batched into that group.

https://discord.com/developers/docs/topics/gateway#gateway-intents

Speaking to another experienced library developer, their opinion was that if not specified the default should be to request all intents. It looks like intents use a similar bitwise system like permissions - so we can figure out the mask for "all" and provide some template masks in the documentation for common use cases (i.e. messages only).

We don't want a channel to have an endless array of stored messages, especially if the array is being added to on a regular basis. We should truncate the array at various thresholds.

Rakudo now already has a Apple Silicon formula on Homebrew, Zef also compiles without any problem, and running a simple Hello World Raku program also has no problem at all on Apple M1. However, when I tried to run zef install API::Discord, the following error message appeared:

===> Searching for: API::Discord
===> Updating fez mirror: http://360.zef.pm/
===> Updating cpan mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/cpan1.json
===> Updating p6c mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/p6c1.json
===> Updated p6c mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/p6c1.json
===> Updated fez mirror: http://360.zef.pm/
===> Updated cpan mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/cpan1.json
===> Searching for missing dependencies: Cro::WebSocket, Data::Dump, Object::Delayed, URI::Encode, Subset::Helper, Test::META
===> Searching for missing dependencies: META6:ver<0.0.24+>, URI, License::SPDX, Test::Output, Object::Trampoline:ver<0.0.9>:auth<cpan:ELIZABETH>, Cro::HTTP, Base64, Digest::SHA1::Native, Crypt::Random, JSON::Fast, OO::Monitors
===> Searching for missing dependencies: JSON::Class:ver<0.0.15+>, JSON::Name, IO::Socket::Async::SSL, IO::Path::ChildSecure, HTTP::HPACK, Cro::Core, Cro::TLS, JSON::JWT, DateTime::Parse, Log::Timeline, if, InterceptAllMethods:ver<0.0.1>:auth<cpan:ELIZABETH>, LibraryMake, Shell::Command, JSON::Class:ver<0.0.14+>
===> Searching for missing dependencies: File::Which, File::Find, OpenSSL, JSON::Marshal:ver<0.0.20+>, JSON::Unmarshal:ver<0.08+>, MIME::Base64, Digest::HMAC
===> Searching for missing dependencies: Digest
===> Building: Digest::SHA1::Native:ver<0.04>
[Digest::SHA1::Native] ld: library not found for -ltommath
[Digest::SHA1::Native] clang: error: linker command failed with exit code 1 (use -v to see invocation)
[Digest::SHA1::Native] make: *** [/Users/deadshot465/.zef/store/p6-digest-sha1-native.git/e34d468341a572a7c089d672429cf88d21e07734/resources/libraries/libsha1.dylib] Error 1
[Digest::SHA1::Native] The spawned command '/usr/bin/make' exited unsuccessfully (exit code: 2, signal: 0)
[Digest::SHA1::Native]   in method build at /Users/deadshot465/.zef/store/p6-digest-sha1-native.git/e34d468341a572a7c089d672429cf88d21e07734/Build.pm line 14
[Digest::SHA1::Native]   in block <unit> at -e line 1
===> Building [FAIL]: Digest::SHA1::Native:ver<0.04>
Aborting due to build failure: Digest::SHA1::Native:ver<0.04> (use --force-build to override)

However, libtommath has already been installed via Homebrew. I think it has something to do with Digest::SHA1::Native dependency rather than raku-api-discord, but is there a workaround for it?

Currently, we hardcode the gateway URL like so;

#| Host to which to connect. Can be overridden for testing e.g.
has Str $.ws-host = 'gateway.discord.gg';

According to the documentation this behaviour is wrong. and in actuality we should be calling the Get Gateway or Get Gateway Bot endpoints for this information.

https://discord.com/developers/docs/topics/gateway#connecting-gateway-url-params

https://discord.com/developers/docs/topics/gateway#get-gateway
https://discord.com/developers/docs/topics/gateway#get-gateway-bot

I think it's still useful to be able to override the gateway URL, so we should keep that in mind when refactoring.

I wonder how can it be, but nonetheless.

I just try running the echo example to see if it was from my own script, but the example fails to work (sending a message back) and discord heartbeat is never replied after the first one.

This would hold middleware to collect information from Cro, plus procedures to access the collected data.

Actual data collected TBC. Currently we are thinking of memory usage and throughput.

API::Discord::Log - can probably be very easily based on an existing log module in the ecosystem.

Then remove all the say, note, warn etc

...

Embeds

:(

I think emoji has some special behaviour available but I don't really know what it is. Messages might contain emoji but it might not matter - it could easily just be simple text strings that Discord understands.

We might want a method on Message that looks for emoji in the message and perhaps an API call to ask Discord whether the emoji is real or not.

Seems like we never find out if the websocket was closed:

...

[TRACE(anon 3)] Cro::WebSocket::MessageSerializer EMIT WebSocket Frame - Continuation
                                                   

[TRACE(anon 3)] Cro::WebSocket::FrameSerializer EMIT TCP Message
  80 80 9a 03 dd 47                                .....G

[TRACE(anon 1)] Cro::HTTP::RequestSerializer EMIT TCP Message
  80 80 9a 03 dd 47                                .....G

« ♥
[TRACE(anon 3)] SetBodySerializers EMIT WebSocket Message - Binary

[TRACE(anon 3)] Cro::WebSocket::MessageSerializer EMIT WebSocket Frame - Text
  7b 0a 20 20 22 6f 70 22 3a 20 31 2c 0a 20 20 22  {.  "op": 1,.  "
  64 22 3a 20 6e 75 6c 6c 0a 7d                    d": null.}

[TRACE(anon 3)] Cro::WebSocket::FrameSerializer EMIT TCP Message
  01 9a 00 f0 4c 69 7b fa 6c 49 22 9f 3c 4b 3a d0  ....Li{.lI".<K:.
  7d 45 0a d0 6c 4b 64 d2 76 49 6e 85 20 05 0a 8d  }E..lKd.vIn. ...
                                                   

[TRACE(anon 1)] Cro::HTTP::RequestSerializer EMIT TCP Message
  01 9a 00 f0 4c 69 7b fa 6c 49 22 9f 3c 4b 3a d0  ....Li{.lI".<K:.
  7d 45 0a d0 6c 4b 64 d2 76 49 6e 85 20 05 0a 8d  }E..lKd.vIn. ...
                                                   

[TRACE(anon 3)] Cro::WebSocket::MessageSerializer EMIT WebSocket Frame - Continuation
                                                   

[TRACE(anon 3)] Cro::WebSocket::FrameSerializer EMIT TCP Message
  80 80 07 56 eb aa                                ...V..

[TRACE(anon 1)] Cro::HTTP::RequestSerializer EMIT TCP Message
  80 80 07 56 eb aa                                ...V..

[TRACE(anon 1)] Cro::TLS::Connector EMIT TCP Message
  88 18 0f a4 41 75 74 68 65 6e 74 69 63 61 74 69  ....Authenticati
  6f 6e 20 66 61 69 6c 65 64 2e                    on failed.

[TRACE(anon 2)] Cro::WebSocket::FrameParser EMIT WebSocket Frame - Close
  0f a4 41 75 74 68 65 6e 74 69 63 61 74 69 6f 6e  ..Authentication
  20 66 61 69 6c 65 64 2e                           failed.

[TRACE(anon 2)] Cro::WebSocket::MessageParser EMIT WebSocket Message - Close

[TRACE(anon 2)] SetBodyParsers EMIT WebSocket Message - Close

[TRACE(anon 3)] SetBodySerializers EMIT WebSocket Message - Close

[TRACE(anon 1)] Cro::TLS::Connector DONE
[TRACE(anon 3)] Cro::WebSocket::MessageSerializer EMIT WebSocket Frame - Close
  e8 03                                            ..

[TRACE(anon 2)] Cro::WebSocket::FrameParser DONE
[TRACE(anon 2)] Cro::WebSocket::MessageParser DONE
[TRACE(anon 2)] SetBodyParsers DONE
[TRACE(anon 1)] Cro::HTTP::ResponseParser DONE
[TRACE(anon 3)] Cro::WebSocket::FrameSerializer EMIT TCP Message
  88 82 6d 47 2c a9 85 44                          ..mG,..D

[TRACE(anon 3)] SetBodySerializers DONE
[TRACE(anon 3)] Cro::WebSocket::MessageSerializer DONE
[TRACE(anon 3)] Cro::WebSocket::FrameSerializer DONE
« ♥
Unhandled exception in code scheduled on thread 9
Cannot send on a closed WebSocket connection
  in method ensure-open at /home/adouglas/src/rakudo-2019.07.1/install/share/perl6/site/sources/A9296BB9102DC15B5CB04F24DCDE4D337413E54A (Cro::WebSocket::Client::Connection) line 172
  in method send at /home/adouglas/src/rakudo-2019.07.1/install/share/perl6/site/sources/A9296BB9102DC15B5CB04F24DCDE4D337413E54A (Cro::WebSocket::Client::Connection) line 106
  in method send at /home/adouglas/src/rakudo-2019.07.1/install/share/perl6/site/sources/A9296BB9102DC15B5CB04F24DCDE4D337413E54A (Cro::WebSocket::Client::Connection) line 112
  in block  at /home/adouglas/src/p6-api-discord/lib/API/Discord/Connection.pm6 (API::Discord::Connection) line 185

We don't want to supply the bot's own messages to the message supply because ignoring your own messages is tedious boilerplate. So we can add a Promise to the created Message and keep it when the API sees the message come back.

We would need to know how Discord munge messages so we can recognise that it was the "same" one. Maybe we're given a message ID on POST?

Can a bot make invitations? We should support creating them.

For example, a Channel object should update itself when we receive relevant messages from the Discord websocket.

Problem here is that if the user has intentionally changed a field, we will probably overwrite it with the data from Discord, even if that didn't change in the payload.

So the questions are:

• How do we get the update message into the various objects, given that the objects could have been generated directly via the API from an ID?
• How do we wire up the update stream to objects that get created by us?
• How do we know when a field is dirty?

I think the answers are something like:

• The base Object role should have a Supply on it. The API object will connect this supply to its own message handler, and emit relevant messages on the relevant objects' supplies when they arrive.

method handle-event($json) {
   # some sort of despatch to the right type or whatever
   # look it up by ID or something
   $discovered-object.update-supply.emit($json);
}

• Created objects will have to be recreated by anyone stubbornly holding onto them. We won't have an ID for it, so we might as well just re-fetch it and wire it up as normal.

• I think there's a trait members can have so we know if they're dirty. Really it's just a hash on the meta-object full of booleans. We can make use of this to avoid overwriting properties that have is-dirty set to true.