Signing witness inputs: builtins.ValueError: witness scritpubkey is found for non-witness UTXO at index 1 about python-bitcointx HOT 24 CLOSED

The utxo is taken like this:

So it should be CTxOut if witness_utxo is present

from python-bitcointx.

Did you look at the decoded PSBT in question in python shell ? Does input at index 1 has witness_utxo field ?

Maybe add debug prints into stream_deserialize() for PSBT_Inputs after read_psbt_keymap() (or inside read_psbt_keymap() itself to see what is decoded

from python-bitcointx.

Hi, sorry was afk for a bit. So first: yes, silly reading fail on my part, as to the origin of the utxo variable in that function.

Second, the PSBT concerned is like this: it has an input which was taken from an earlier PSBT object, and that input had a NONWITNESS_UTXO field only (i.e. it didn't have a WITNESS_UTXO field). So that input has utxo set as a CTransaction but no witness_utxo data.
So it seems that's my basic problem, that Wasabi reasons as follows: we need the NONWITNESS_UTXO field to address the recent PSBT changes, and we don't need additional redundant information; then I take that PSBT_Input.utxo field and add it into the newly constructed coinjoin transaction, and it gets rejected when I'm going through the inputs in the call to sign().

Unfortunately I cannot look directly at the data of this transaction, I only have the stack trace as per here) since I'm not running these two apps on testnet, but I do know that Wasabi sends its original PSBT with only NONWITNESS_UTXO data attached, see the example base64 here. (That is finalized, but it is the pre-payment in payjoin bip78, we are signing a new psbt with that input inserted but of course un-finalized).

Now that I understand it, I could manually insert the witness_utxo into the inputs as needed, but I'd rather do whatever is actually correct here.

(new coinjoin psbt creation happens here:

https://github.com/JoinMarket-Org/joinmarket-clientserver/blob/c75a5b10de30ade18ea0916176c6ae86384e5aa3/jmclient/jmclient/payjoin.py#L997-L998

in case that small bit of context helps).

...

from python-bitcointx.

The example base64-encoded psbt that you linked to contains both WITNESS_UTXO and NON_WITNESS_UTXO fields (and also FINAL_SCRIPTWITNESS) field in its only input (and has witness_utxo property populated when decoded)

it has an input which was taken from an earlier PSBT object, and that input had a NONWITNESS_UTXO field only (i.e. it didn't have a WITNESS_UTXO field). So that input has utxo set as a CTransaction but no witness_utxo data.

The encoded transaction can have only NONWITNESS_UTXO in its encoding and still have witness_utxo field in PSBT_Input, if other PSBT fields are present in the encoding that allow to say that this is indeed a witness UTXO. If there's no such data, we and no WITNESS_UTXO in encoded transaction, it will be deemed as non-witness.

You said that that input has utxo set as CTransaction but no witness_utxo. That means that the encoding for that input did not contain any of WITNESS_SCRIPT, FINAL_SCRIPT_WITNESS,WITNESS_UTXO. The only way to know if this is a witness input is then out-of-band.

from python-bitcointx.

I could manually insert the witness_utxo into the inputs as needed

you could do psbt.set_utxo(None, i, force_witness_utxo=True) to forcefully "witness-ize" the input.

from python-bitcointx.

the 'None' means just use self._utxo.

Now that I think about this, maybe not the most intuitive thing, and this actually changed the semantics, as previously this would un-set the utxo, and now it won't have this effect. This seems like a mistake, but is an off-topic in this issue

from python-bitcointx.

The example base64-encoded psbt that you linked to contains both WITNESS_UTXO and NON_WITNESS_UTXO fields (and also FINAL_SCRIPTWITNESS) field in its only input (and has witness_utxo property populated when decoded)

Right, my mistake. It is yet another instance of the logic failing because I originally assumed only one of the two were included; my human_readable_psbt function made this assumption; when I corrected it I see it.

You said that that input has utxo set as CTransaction but no witness_utxo. That means that the encoding for that input did not contain any of WITNESS_SCRIPT, FINAL_SCRIPT_WITNESS,WITNESS_UTXO. The only way to know if this is a witness input is then out-of-band.

This part I only partially understand, if the utxo is type CTransaction then presumably it's possible to deduce that the input is segwit type if the scriptPubKey of the referenced output is not p2sh, even without the other fields. But that's academic (I know that this "corner case" is addressed in other parts of the recent patch), since as you point out the witness_utxo was included here (edit: wait, even if it's p2sh, it is possible right, because utxo would be a fully signed transaction, so would have the witness).

from python-bitcointx.

the witness_utxo was included here.

But the error is also about input at index 1, while the linked base64-encoded psbt had only one input

from python-bitcointx.

the witness_utxo was included here.

But the error is also about input at index 1, while the linked base64-encoded psbt had only one input

That's a detail of the complexity of this case; we have that psbt as "input" (a fully signed pre-payment); and then we grab that input and include it in the new payjoin PSBT.

from python-bitcointx.

So .. I need to include the provided witness_utxo field when I "port" this input from the old (fully signed non coinjoin) PSBT that the sender provided, to the new payjoin PSBT (to which I add my own input(s)).

from python-bitcointx.

Sorry, I do not fully understand your context.

who provides witness_utxo ?

Will supplying always_include_witness_utxo=True to serialize() somewhere could help you ?

from python-bitcointx.

(oh, please ignore last message, always_include_witness_utxo is true by default)

from python-bitcointx.

OK, I realise the problem comes in here:

https://github.com/JoinMarket-Org/joinmarket-clientserver/blob/c75a5b10de30ade18ea0916176c6ae86384e5aa3/jmclient/jmclient/wallet.py#L1203-L1208

... basically, same issue: I was assuming only one of the two, and call set_utxo with whichever one is given. As per docstring of function, caller is supposed to provide one of CTxOut or CTransaction in spent_outs[i].

Context:
sender sends a finalized psbt which is checked to be a valid payment of the chosen amount.
receiver (in this case, JM code) after checking that, takes the input(s) from that tx, and creates a brand new psbt with those inputs, and his own inputs, to recreate the payment as a coinjoin. There are a bunch of rules (BIP78) but that's a sidetrack. Is it clear now?

from python-bitcointx.

The problem is that the call to input.set_utxo() does not supply unsigned_tx. Without it, there is nowhere to extract the witness utxo from

from python-bitcointx.

Or rather, no way to locate the output to extract the witness utxo from supplied CTransaction

from python-bitcointx.

Ah yes, that makes perfect sense, I see. If I read the code correctly, if I provide that, it will populate witness_utxo (due to call of check_witness_utxo_spk).

from python-bitcointx.

btw, I see that you actually have unsigned_tx earlier in the code (new_psbt = btc.PartiallySignedTransaction(unsigned_tx=tx)) so you can just supply it, or use new_psbt.set_utxo(spent_outs[i], i)

from python-bitcointx.

if I provide that, it will populate witness_utxo (due to call of check_witness_utxo_spk)

yes

from python-bitcointx.

btw, I see that you actually have unsigned_tx earlier in the code (new_psbt = btc.PartiallySignedTransaction(unsigned_tx=tx)) so you can just supply it, or use new_psbt.set_utxo(spent_outs[i], i)

But that is unsigned_tx of this PSBT, whereas we need the unsigned_tx of the input being spent, right? I think that's why I didn't do it originally.

Oh no, I see that's wrong now.

from python-bitcointx.

re unexpectedly changed set_utxo(None, ...) semantics, I opened separate issue #47

from python-bitcointx.

But that is unsigned_tx of this PSBT, whereas we need the unsigned_tx of the input being spent, right? I think that's why I didn't do it originally.

These unsigned_tx's should be in-sync, no ?

from python-bitcointx.

Yeah, see above, just got confused.

from python-bitcointx.

wait, even if it's p2sh, it is possible right, because utxo would be a fully signed transaction, so would have the witness

In this case we are interested in the output of the CTransaction that resides in the NON_WITNESS_UTXO, and the input witnesses are irrelevant. This output is what is spend by the input that we want to know the witness status of. If it is a p2sh output, then the only way we can infer that it is was p2sh_p2wsh is if our psbt input also has witness_script or final_script_witness.

And by the way, if your new_psbt inputs can be missing witness_script or final_script_witness and be spending p2sh outputs, you can hit this corner case. If this is possible, then maybe you want to store that 'witness-ness' flags somewhere along spent_outs and supply it to force_witness_utxo kwarg of set_utxo().

OTOH, I think it is likely that correctly-constructed inputs will bear witness_script, so probably this corner case will not emerge.

from python-bitcointx.

In this case we are interested in the output of the CTransaction that resides in the NON_WITNESS_UTXO, and the input witnesses are irrelevant

Yes, it's clear. Sorry for confusion.

from python-bitcointx.