mobilecoinofficial / forest Goto Github PK

mainline, mainline, mainline

i've been trying to set up integration tests, but it would also be great to have unit tests.

i've done something like this here: https://github.com/technillogue/whispr/blob/main/test_whispr.py#L67

i think the same deal should work -- subclass ForestBot, cut out launch_and_connect, probably a method that calls handle_message/s and returns the contents of signalcli_input_queue. getting getting async tests might be a little finicky, as is handling more involved state with groups. can have a test db, or mock out pghelp -- it should support that

for the integration tests, this is what i have with signal-cli: https://github.com/forestcontact/forest-draft/blob/refactor/test_forest.py
and with auxin: https://github.com/forestcontact/auxin/blob/dockerized/healthcheck.sh

┆Issue is synchronized with this Asana task by Unito

datastore.py 220-248

def setup_tmpdir() -> None:
    if not utils.LOCAL:
        logging.warning("not setting up tmpdir, running on fly")
        return
    if utils.ROOT_DIR == ".":
        logging.warning("not setting up tmpdir, using current directory")
        return
    if utils.ROOT_DIR == "/tmp/local-signal/" and not utils.MEMFS:
        try:
            shutil.rmtree(utils.ROOT_DIR)
        except (FileNotFoundError, OSError) as e:
            logging.warning("couldn't remove rootdir: %s", e)
    if not utils.MEMFS:
        (Path(utils.ROOT_DIR) / "data").mkdir(exist_ok=True, parents=True)
    # assume we're running in the repo
    sigcli = utils.get_secret("SIGNAL_CLI_PATH") or "auxin-cli"
    sigcli_path = Path(sigcli).absolute()
    try:
        logging.info("symlinking %s to %s", sigcli_path, utils.ROOT_DIR)
        os.symlink(sigcli_path, utils.ROOT_DIR + "/auxin-cli")
    except FileExistsError:
        logging.info("auxin-cli's already there")
    try:
        os.symlink(Path("avatar.png").absolute(), utils.ROOT_DIR + "/avatar.png")
    except FileExistsError:
        pass
    logging.info("chdir to %s", utils.ROOT_DIR)
    os.chdir(utils.ROOT_DIR)
    return```

the culpript is in line 239
```python
 os.symlink(sigcli_path, utils.ROOT_DIR + "/auxin-cli")

Can encrypt with a PSK / symmetric cryptography. Just something at rest. Might as well compress too, even though that's sometimes-dangerous, we have a lot of whitespace in our tarball.

┆Issue is synchronized with this Asana task by Unito

I'm trying to figure out how tag the user who sent a message.

Like at the bottom of this example from Imogen:

I looked through https://github.com/mobilecoinofficial/forest/blob/main/imogen/imogen.py but not seeing anywhere in there that's doing it. It looks like Imogen's "your Imoge is done" logic is being handled in the redis callback somewhere, but didn't see where exactly.

I see Forest's Message.name and Message.source, and they're both giving me the correct string data, but just sticking @ in front of them didn't turn either into a user tag. E.g.

async def do_test(self, msg: Message) -> str:
    return f"@{msg.name} test"

447d946

pending new messages and better arg1 parsing

┆Issue is synchronized with this Asana task by Unito

To discuss and implement a paid priority queue for imogen.

I will write a possible spec for how it could work, and we can discuss it.

Will be resolved by PR #76

┆Issue is synchronized with this Asana task by Unito

two event loops need to not crash, the signal-cli event loop, and the aiohttp event loop - we need to build external services (out of auxin? simpler instances of our Bot?) that heartbeat / healtcheck our core communication channels, the HTTP server that catches events from teli, and the signal-cli session that is exposed to users.

When using self.send_reaction in DMs, it seems to work normally. However, when in groups, only the target author can see the reaction that the bot puts on their post. This is not the behavior i experience as a user of Signal Android or Desktop. Not sure if this is a bug in the Signal class, or an rpc error, or something in the underlying signal-cli.

Making an issue so we can discuss it, keep track, and get ideas.

The payments demo started life as a rewrite of echopay. The current rewrite so far does not actually implement echoing payments. It demonstrates the ability to send and receive payments, and react to sent and received payments. Without implementing echoing payments.

That brings up a couple questions:

• Should we change the payments demo in some way to implement an echopay (I don't think it's necessary, but I'm open to discussion)
• What should we call the payments demo if not echopay.py

Part of the tutorial includes modifying the code and so I want to include before and after examples. So whatever naming convention we choose we also need two names. I've been using echopay.py and echopay_final.py but that could be confusing.

https://github.com/forestcontact/forest-draft/blob/168de3f2b40a2d20ba1972f484256891699904b2/datastore.py#L241

don't hit this codepath in the wrong in the wrong current working directory!

todo: ensure diff occurs before fresh start, don't rm non-signalcli artifacts without user confirmation

no more commits that just say "black"

maybe using https://pre-commit.com/

c.f. #182

wondering if we can make a signal group and message it with tracebacks? 🤔

if we're not independently robust enough for that, we will need a basic mailbox / queue to catch and log failures.

smtp to a server we control might work, even submissions to a google form - simpler requests the better.

┆Issue is synchronized with this Asana task by Unito

When I push my branch to Github it fails checks due to a couple of #type: ignore checks in mypy:

Run pipenv run mypy sample_bots mc_util forest mobfriend contact imogen
  pipenv run mypy sample_bots mc_util forest mobfriend contact imogen
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.9.10/x64
mobfriend/mobfriend.py:[2](https://github.com/mobilecoinofficial/forest/runs/5281028606?check_suite_focus=true#step:8:2)22: error: Unused "type: ignore" comment
mobfriend/mobfriend.py:2[3](https://github.com/mobilecoinofficial/forest/runs/5281028606?check_suite_focus=true#step:8:3)[8](https://github.com/mobilecoinofficial/forest/runs/5281028606?check_suite_focus=true#step:8:8): error: Unused "type: ignore" comment
Found 2 errors in 1 file (checked 28 source files)
Error: Process completed with exit code 1.

When I run the same command locally it does not have this problem, though for some reason many other files fail?

╰─>$ pipenv run mypy sample_bots mc_util forest mobfriend contact imogen                                                                                                   (base) 
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
mc_util/printable_pb2.pyi:7: error: Library stubs not installed for "google.protobuf.descriptor" (or incompatible with Python 3.9)
mc_util/printable_pb2.pyi:7: error: Library stubs not installed for "google.protobuf" (or incompatible with Python 3.9)
mc_util/printable_pb2.pyi:8: error: Library stubs not installed for "google.protobuf.message" (or incompatible with Python 3.9)
mc_util/external_pb2.pyi:6: error: Library stubs not installed for "google.protobuf.descriptor" (or incompatible with Python 3.9)
mc_util/external_pb2.pyi:6: note: Hint: "python3 -m pip install types-protobuf"
mc_util/external_pb2.pyi:6: note: (or run "mypy --install-types" to install all missing stub packages)
mc_util/external_pb2.pyi:6: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
mc_util/external_pb2.pyi:6: error: Library stubs not installed for "google.protobuf" (or incompatible with Python 3.9)
mc_util/external_pb2.pyi:7: error: Library stubs not installed for "google.protobuf.internal.containers" (or incompatible with Python 3.9)
mc_util/external_pb2.pyi:7: error: Library stubs not installed for "google.protobuf.internal" (or incompatible with Python 3.9)
mc_util/external_pb2.pyi:8: error: Library stubs not installed for "google.protobuf.message" (or incompatible with Python 3.9)
forest/core.py:32: error: Library stubs not installed for "termcolor" (or incompatible with Python 3.9)
forest/core.py:32: note: Hint: "python3 -m pip install types-termcolor"
forest/core.py:48: error: Unused "type: ignore" comment
forest/core.py:643: error: Unused "type: ignore" comment
forest/core.py:644: error: Unused "type: ignore" comment
imogen/imogen.py:192: error: Unused "type: ignore" comment
imogen/imogen.py:206: error: Unused "type: ignore" comment
imogen/imogen.py:260: error: Item "MultipartReader" of "Union[MultipartReader, BodyPartReader]" has no attribute "name"
imogen/imogen.py:261: error: Item "MultipartReader" of "Union[MultipartReader, BodyPartReader]" has no attribute "filename"
imogen/imogen.py:268: error: Item "MultipartReader" of "Union[MultipartReader, BodyPartReader]" has no attribute "read_chunk"
mobfriend/mobfriend.py:19: error: Library stubs not installed for "google.protobuf" (or incompatible with Python 3.9)
Found 18 errors in 5 files (checked 28 source files)

I haven't edited mobfriend at all. I also see this happening in some other branches, specifically:
https://github.com/mobilecoinofficial/forest/runs/5238596966?check_suite_focus=true
https://github.com/mobilecoinofficial/forest/runs/5253050764?check_suite_focus=true
https://github.com/mobilecoinofficial/forest/runs/5262132498?check_suite_focus=true

My guess it might be local configuration environment configured differently from Github Actions?

Importing aPersistDict gives a dependency error:

ModuleNotFoundError: No module named 'Crypto'

This module should be installed with pycryptodome dependency which is already in the Pipfile. I reinstalled it with

pipenv install pycryptodome

and this seemed to work. Don't understand why it wouldn't be installed with the original Pipfile though...

Repo should be restructured. I have some ideas as to how this could be done.

We can have directories for sample bots, or each bot could have it's own directory.

Unfortunately restructuring the dir would mean rewriting some of the code. I think some ide's would do this for you. Maybe just Pycharm would do that. Anyone have pycharm?

I can submit a proposal for a potential organisation scheme. And we can make it happen.

use exquo/signal-libs-build#3

https://gitlab.com/packaging/signal-cli#support ?

idem for auxin

Write a ReadMe Walkthrough that shows users how to deploy echopay. Plus Information about Echopay and payments integration.

┆Issue is synchronized with this Asana task by Unito

I tried to run hellobot with:

poetry run python -m sample_bots.hellobot

And an exception is raised:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/qinyu/job/alf/singal_bot/forestbot-template/template.py", line 1, in <module>
    from forest.core import Bot, Message, run_bot
  File "/Users/qinyu/Library/Caches/pypoetry/virtualenvs/yourbot-1I-L8FrT-py3.9/lib/python3.9/site-packages/forest/core.py", line 43, in <module>
    from forest import autosave, datastore, payments_monitor, pghelp, utils, string_dist
  File "/Users/qinyu/Library/Caches/pypoetry/virtualenvs/yourbot-1I-L8FrT-py3.9/lib/python3.9/site-packages/forest/autosave.py", line 16, in <module>
    from forest import fuse, mem, utils
  File "/Users/qinyu/Library/Caches/pypoetry/virtualenvs/yourbot-1I-L8FrT-py3.9/lib/python3.9/site-packages/forest/fuse.py", line 56, in <module>
    raise EnvironmentError("Unable to find libfuse")
OSError: Unable to find libfuse

I failed to google a solution to this problem. Any thoughts?

^CINFO main:224: handling sigint. sigints: 0
INFO main:228: got running loop, scheduling async_shutdown
INFO main:194: starting async_shutdown
DEBUG datastore:185: ['data/+12406171657', 'data/+12406171657.d']
DEBUG datastore:202: saved 230.0 kb of tarballed datastore to supabase
INFO main:211: =============exited===================
^CException ignored in: <module 'threading' from '/usr/lib/python3.9/threading.py'>
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 1415, in _shutdown
    atexit_call()
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 31, in _python_exit
    t.join()
  File "/usr/lib/python3.9/threading.py", line 1033, in join
    self._wait_for_tstate_lock()
  File "/usr/lib/python3.9/threading.py", line 1049, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt:

If you don't create an AioQueue, this doesn't happen.

Maybe https://github.com/forestcontact/forest-draft/blob/v0.4/main.py#L208 needs to destroy the queue as well...?

┆Issue is synchronized with this Asana task by Unito

Do you have a template, could we check one into the repo?

If a list has multiple admins, the other admins should know when someone sends a blast.

/
/imagine {"text": "you best start believing in cyberpunk dystopias", "max_iterations":420}

Sorry! Command
/imagine not recognized! Try /help.

┆Issue is synchronized with this Asana task by Unito

#62

The Readme was written, tested, and merged.

┆Issue is synchronized with this Asana task by Unito

Followed the readme install instructions, up to Running Hellobot. Manually testing messages w/ ./signal-cli send worked.

Now when attempting to start hellobot with:

pipenv run python -m sample_bots.hellobot

Throws the fatal error:

Traceback (most recent call last):
  File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/dsernst/Documents/forest/sample_bots/hellobot.py", line 5, in <module>
    from forest.core import Bot, Message, run_bot
  File "/Users/dsernst/Documents/forest/forest/core.py", line 33, in <module>
    import aiohttp
ModuleNotFoundError: No module named 'aiohttp'

it would be helpful to be able to op people without having to restart

• Auxin only knows it if there's a profile key if messaged from first bot
• A challenge needs to be sent from auxin or python bot head to get user to message from primary device

Make a ReadMe for users to interact with imogen. It can have a section announcing future plans to allow people to implement their own imogen. But the main thing is documenting all the commands, the signal groups, and how to use the priority queue and donations.

This will collaborate closely with #77, but the doc strings don't have to 100% match the readme. The readme can have more information that is not printed to signal.

┆Issue is synchronized with this Asana task by Unito

when I run cargo build --release, follow error happens:

error: failed to select a version for the requirement curve25519-dalek = "^2.0.0"
candidate versions found which didn't match: 3.2.1
location searched: Git repository https://github.com/signalapp/curve25519-dalek.git?branch=lizard2
required by package zkgroup v0.9.0 (https://github.com/signalapp/zkgroup?tag=v0.9.0#e6e53bf6)
... which satisfies git dependency zkgroup of package auxin v0.2.0 (/Users/qinyu/job/alf/singal_bot/auxin/auxin)

I have tried both Mac and Linux, same error.
What can I do?

Each command needs a docstring, /help needs to show that docstring, and also list available commands. I started this here: https://github.com/forestcontact/forest-draft/tree/help

Additionally, there needs to be some user testing and general good docs written.

If internal data changes enough to warrant a restart, an automatic restart should be done by an internal daemon upon a developer command

starting from 0 with no creds

It'd be neat if you could set profile information such as the bot's name through an envvar

┆Issue is synchronized with this Asana task by Unito

redo the stuff from #26

Add bot_template.py from which people can copy and build their own bots.

While going through the README's install instructions, in the section Registering a Signal Account for your bot, running the line:

export CAPTCHA=$(curl -s --data-binary "https://signalcaptchas.org/registration/generate.html" https://human-after-all-21.fly.dev/6LedYI0UAAAAAMt8HLj4s-_2M_nYOhWMMFRGYHgY | jq -r .solution.gRecaptchaResponse)

seemed to hang. Unclear if it was broken or what.

After approx 60 seconds, it proceeded. 👍👍

Would be nice to note this in the install instructions. E.g.:

The 2nd command will take a minute to complete.

Sometimes auxin crashes crash forest bot. The following Auxin in 0.1.8 crash ended up crashing forest core.
#Auxin 0.

Guess is that when auxin fails, the following assert causes start_process to fail

And then because the task is stored within the class dict, no error is sent because the task is never garbage collected.

We should

1. Not use assert for stdin monitoring (asserts are for values always expected to be true, shouldn't be used on values likely to change), we should explicitly check its status and if we detect stderr, restart auxin
2. Monitor the start_process() task and if it is in error state, dereference and start a new task.

Have the readme instruct the reader on how to set up the repo and run any of the sample bots or deploy their own.

Auxin CLI no longer ouputs lists under params. Core.py logic should maybe filter for only datamessages?

Lines between
https://github.com/forestcontact/forest-draft/blob/78f2ddff11998741ec7d96d0039f98bd51c5f56e/forest/core.py#L190
and
https://github.com/forestcontact/forest-draft/blob/metrics/forest/core.py#L205

could be refactored to: https://gist.github.com/iamalwaysuncomfortable/42c5aeb57fa365be0018ecd7b7f6b2ad

If it's desired to only have datamessages enter the message output queue.

┆Issue is synchronized with this Asana task by Unito

here's some things I've found about Mobot whilst using it.

What memo would you like to use? ('None' for empty

extra open paren

Happy gifting!

but it was a payment request... confusing.

Taking pictures of QR codes doesn't always work, sometimes the bot things I want to save a picture for a template instead of finding a qr code. Annoyingly it scanned the code for a payment request (which helper can't fulfill) but it refuses to scan a gift code qr which only Helper can redeem. Very annoying.

it's currently dangerous to have multiple bots in the same group: they can get into loops with each other, multiple bots respond to the same command, etc. add a do_toggle_strict admin-only command that checks mentions_us before responding in that group