Describe the bug

Traceback (most recent call last):
  File "cryptoadvance/specterext/spectrum/bridge_rpc.py", line 86, in multi
  File "cryptoadvance/specterext/spectrum/bridge_rpc.py", line 86, in <listcomp>
  File "cryptoadvance/spectrum/spectrum.py", line 348, in jsonrpc
  File "cryptoadvance/spectrum/spectrum.py", line 343, in jsonrpc
  File "cryptoadvance/spectrum/spectrum.py", line 57, in wrapper
  File "cryptoadvance/spectrum/spectrum.py", line 1025, in walletcreatefundedpsbt
cryptoadvance.spectrum.spectrum.RPCError: ('Insufficient funds', -4)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "flask/app.py", line 1523, in full_dispatch_request
  File "flask/app.py", line 1509, in dispatch_request
  File "flask_login/utils.py", line 272, in decorated_view
  File "cryptoadvance/specter/server_endpoints/wallets/wallets.py", line 531, in send_new
  File "cryptoadvance/specter/commands/psbt_creator.py", line 115, in create_psbt
  File "cryptoadvance/specter/wallet.py", line 1714, in createpsbt
  File "cryptoadvance/specter/rpc.py", line 459, in fn
  File "cryptoadvance/specterext/spectrum/bridge_rpc.py", line 98, in multi
AttributeError: 'object' object has no attribute 'status_code'

To Reproduce
Steps to reproduce the behavior:

1. Create a PSBT with a VERY low fee
2. Sign PSBT
3. Send Transaction

Expected behavior
The error message needs to bubble up in the UI to the user

Desktop (please complete the following information):

• v1.14.0-pre5

Currently, in the wallet constructor we have:

        self.rpc = self.manager.rpc.wallet(
            os.path.join(self.manager.rpc_path, self.alias)
        )

This means that if the walletManager changes its rpc-instance, this will not change the rpc-class of the wallet. As this rpc-instance is used down by a lot more objects (addresses/TxList/TxItem/...) this has quite some large impact.

Fixing that is possible in these ways:

• Creating a property out of the rpc attribute and then checking for changes in the WalletManager rpc instance before returning the old instance or creating a new one
• all the other options sound unreasonable

Currently, Spectrum is not deleting the wallets on the database side.

Implementation approach:

Create a rpc call in Spectrum (sth. like "delete_wallet") that can be called from the Spectrum node on the Specter Core side.

Unknown method "gettxoutproof"

How will the address validation in Spectrum work without Bitcoin Core code in the background?

Implementing the getrawtransaction method in Spectrum should make it possible to retrieve the whitepaper from the timechain with Spectrum.

getrawtransaction is already implemented in the ElectrumX API:
https://electrumx.readthedocs.io/en/latest/protocol-methods.html#blockchain-transaction-get

Could be integrated here on this screen. "Forget node" button only if there is a Spectrum node already configured.
With sth. like:
app.specter.node_manager.delete_node(node, app.specter)

Needs this issue to be done first:
#13

The current error is:
cryptoadvance.spectrum.spectrum.RPCError: ('Method not found (utxoupdatepsbt)', -32601)

since the RPC call utxoupdatepsbt doesn't exist / isn't handled in Spectrum.

BDK could be used here.

More background in this issue:
cryptoadvance/specter-desktop#2173

Apparently, the check_utox method is not working properly in Spectrum, resulting in such errors:

This results in not being able to view the Transactions tab or the Addresses tab.

Status Quo is unknown. Needs docuementation and propably some improvements.

Connecting to a Websocket Server is quite a standard thing and done all over the place. There are good libraries doing that job and doing that job well (error handling/performance/testing/maintenance/security fixes). We're using a custom implementation in elsock.py and i pledge for replacing it with an external library.

Lately i looked into https://github.com/jeffthibault/python-nostr (which has a very nice and clean codebase btw) and it's using https://github.com/websocket-client/websocket-client which seem to be a quite mature thing.

Update: The idea is stupid as the ElectrumConnection is not a websocket connection but a custom socket-protocol.

Created a hot wallet but in the wallet table in the database it shows for the wallet private keys enabled 0 and there is no seed.
When trying to sign a tx with the hot wallet, I get:

Gathering some problems / errors of the status quo in this issue:

1) Loadwallet error

• Creating a wallet while using Spectrum generates a JSON file but doesn't save the wallet file on the Core node (obviously) since it uses the database to store the wallet data.
• Therefore, when you switch back to using the Core node you get this error:

One of the main reasons specter doesn't show the correct amounts is a not-synced node. For Spectrum, this might also be the case but there is another reason on top: Spectrum just booted up and connected to the electrum server but have not yet sunscribed to all the scripthashes it needs to. A scripthash is basically an address and this looks like this in the log:

[2023-01-23 12:07:39,618] INFO in spectrum: Now subscribed to 2900 scripthashes
[2023-01-23 12:07:44,015] INFO in spectrum: Now subscribed to 3000 scripthashes
[2023-01-23 12:07:49,189] INFO in spectrum: Now subscribed to 3100 scripthashes
[2023-01-23 12:07:53,949] INFO in spectrum: Now subscribed to 3200 scripthashes
...

Potentially, this could be shown as a percentage in the logs as the potential scripthashes are all in the DB. Simplified code from spectrum.py sync-method:

        for sc in Script.query.all():
            # ignore external scripts (labeled recepients)
            if sc.index is None:
                continue
            subscription_logging_counter += 1
            if subscription_logging_counter % 100 == 0:
                logger.info(
                    f"Now subscribed to {subscription_logging_counter} scripthashes"
                )
            res = self.sock.call("blockchain.scripthash.subscribe", [sc.scripthash])
            if res != sc.state:
                self.sync_script(sc, res)

So unless this loop is finished, specter might be outdated. This should clearly be reflexted in the UI.
The current way of showing that looks like this:

If you click on the node, you get some details of that node which currently looks like this:

This is not optimal. I think it could be shown how much blocks are still too sync but both of the screenshots don't show that.

Anyway, this is about spectrum. Here, we don't have the warning, but you can only see the blockcount of the electrum-server:

So in order to give better feedback:

• We need a warning if the underlying timechain is not in sync OR spectrum has not yet finished subscribing to all Scripthashes
• Also some better statistics on that page would be good. E.g. Number of (synced-) scripthashes, wallets, Utxos, Txs and Descriptors

So how should that be implemented? I would suggest the following:

• Change the loop above to calculate percentages on top of absolute numbers
• The warning-sign is based on the result of initialblockdownload
• This is not properly implemented in [spectrum)(https://github.com/cryptoadvance/spectrum/blob/master/src/cryptoadvance/spectrum/spectrum.py#L383). I hope that electrum has a way to figure it out. Can't imagine that this is not obtainable via electrum.
• As the effect on a spectrum not being DB-synced is the exact same than a Core being in initialblockdownload=True, i'd suggest that we implement it so that spectrum is not giving back false until the DB-sync is completed as well.
• However, it'd be beneficial to also know the exact sync status. For that, i'd suggest to create an own non-core-rpi-call getscripthashsyncinfo which returns the status (which can be set as an attribute on the spectrum-object while syncing and the rpc-call reads it from there).
• This change might be a good opportunity to also refactor all the rpc-calls in three files (spectrum.py is way too long), like: core_rpc.py, wallet_rpc.py and spectrum_rpc.py (with getscripthashsyncinfo as the currently only call in there)
• If this is done, it should be easy to call that info somewhere in the spectrum_info.jinja.

If you click the regular delete wallet button you get this:

We know that there are issues with running several nodes on the same network. E.g. a spectrum_node and a Bitcoin core node on mainnet. There might be bugs but there are also conceptual issues. Let's focus on the conceptual issues:

• A wallet needs to exist on the specter-side as json file in the .specter-folder. (for mainnet e.g. ~/.specter/wallets/main/mywallet.json)
• However, as we use the core-API for heavy lifting, the wallet also needs to exist behind the core API. There its name would be e.g. spectera8d0c2bcfa1dd45c/mywallet
• If the wallet exists on the API-side but not on the specter-side, it will never be shown on specter. It practically disappeared
• If the wallet exits on the specter side but not the API side, you will get something like this:
  
• So you will be able to "recreate" the wallet at the core side

So now it's possible to misuse this mechanism having more than one Node on the same network and then simply recreate the wallets on the API-side. There might be bugs:

• #35

So the switch_node method might not be working properly. This might be much more severe in this cases as it is in cases where more than one node but on different networks (e.g. liquid).

Should we allow that? I think we should, but we should setup a huge warning sign.

It'd not yet possible to connect to a electrum via Tor. We basically need something very similiar than specter.requests_session:

    def requests_session(self, force_tor=False):
        requests_session = requests.Session()
        if self.only_tor or force_tor:
            proxy_url = self.proxy_url
            proxy_parsed_url = urlparse(self.proxy_url)
            proxy_url = proxy_parsed_url._replace(
                netloc="{}:{}@{}".format(
                    str(random.randint(10000, 0x7FFFFFFF)),
                    "random",
                    proxy_parsed_url.netloc,
                )
            ).geturl()
            requests_session.proxies["http"] = proxy_url
            requests_session.proxies["https"] = proxy_url
        return requests_session

It's currently not possible to delete the Bitcoin Core node if it is the only node. So, perhaps better to write ""Disconnect the Bitcoin Core node"?

This is needed for the whitepaper functionality in Specter.
cryptoadvance/specter-desktop#2065

The extension-generation process is currently assuming you need encrypted userdata. That should get adjusted like this:

• Add an option to the click annotations like --encrypted-userdata here: https://github.com/cryptoadvance/specter-desktop/blob/master/src/cryptoadvance/specter/cli/cli_ext.py#L39 (including adding it as a param to the method signature)
• Add a section explaining what it means and that the extension should decide between encrypted/unencrypted here: https://github.com/cryptoadvance/specter-desktop/blob/master/src/cryptoadvance/specter/cli/cli_ext.py#L111-L112
• Add the variable to the ExtGen instantiation and there ...
• The service and the controller would needs some adjustments. They are rendered here: https://github.com/cryptoadvance/specter-desktop/blob/master/src/cryptoadvance/specter/cli/cli_ext.py#L150-L160 however, the code for the templates is living here:
  • https://github.com/cryptoadvance/specterext-dummy/blob/master/src/dummyorg/specterext/dummy/service.py#L12
  • https://github.com/cryptoadvance/specterext-dummy/blob/master/src/dummyorg/specterext/dummy/controller.py#L53

Mind the different templating syntax. Those are "metatemplates" as they are creating templates which themself are later templates.

You can test it via using --tmpl-fs-source and checkout the dummy-project somewhere. Also i've created a small section about https://docs.specter.solutions/desktop/extensions/intro/#virtualenv-management

This is necessary to implement a delete functionality.