Hey,
I found a bug in the generation of the bundle hash which leads me to general questions regarding conversion and padding.

Overview

When the hashed Essence returns an insecure bundle hash, the obsolete tag is incremented and a new bundle hash is generated, such that the obsolete tag functions as a nonce.
In this process the obsolete tag isn’t padded correctly after incrementing, here’s an example:

Example

Essence (before incrementing): c2eb2d5297f4e70f3e40e3d7aa3f5c1d7405264aeb72232d06776605d8b6121100000000000000350000000000000000000000000000000000000000000000000000000000000000000000005d303e2500000000000000000000000000000002

Essence (after incrementing): c2eb2d5297f4e70f3e40e3d7aa3f5c1d7405264aeb72232d06776605d8b61211000000000000003501000000005d303e250000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000

Solution / Questions

When I tried to use padHBits() I noticed the padding function is reversed (pads from right). Is there any reason for that?
To fix this I rewrote padHBits like this:

export const padHBits = (length: number) => (hbits: Int8Array) =>
  hbits.length < length
    ? new Int8Array(length).map((n, i) => (i > (length - hbits.length) ? hbits[i - (length - hbits.length)] : 0))
    : hbits;

and padded obsolete tag to the size of 256 bits after incrementing (finalizeBundle() line 202):

obsoleteTagHBits[0] = padHBits(TRANSACTION_OBSOLETE_TAG_BITS_SIZE) (hbits(hex(add(toHBytes(hbytes(obsoleteTagHBits[0])), 1)))

I noticed a lot of conversion being done here, which might not be needed. As I understand, correctly we are converting everything to bits and then converting them to bytes when using functions with bytes input (like hHash.absorb() or add()), but because there isn’t a function that converts bits to bytes we are converting bits to hex (hbytes()) and then hex to bytes (toHBytes()). So I wondered why bits are needed at all?

To reduce the amount of conversion in this line I would suggest the following:

obsoleteTagHBits[0] = padHBits(TRANSACTION_OBSOLETE_TAG_BITS_SIZE)(hbits(value(obsoleteTagHBits[0]) + 1));

Expected Behavior

Please describe the behavior you are expecting

Current Behavior

What is the current behavior?

Failure Information (for bugs)

Please help provide information about the failure if this is a bug. If it is not a bug, please remove the rest of this template.

Steps to Reproduce

Please provide detailed steps for reproducing the issue.

1. step 1
2. step 2
3. you get it...

Context

Please provide any relevant information about your setup. This is important in case the issue is not reproducible except for under certain conditions.

• Version:
• Operating System:

Failure Logs

Please include any relevant log snippets or files here.

Expected Behavior

(1) When getAccountData(...) is passed a seed that has an address containing balance, the reported balance should be shown (and summed across all addresses belonging to the seed if necessary).

(2) When passed a value containing address from a seed, getBalances(...) should return the correct balance for that address.

Current Behavior

Currently, getBalances(...) returns the correct balance for this address,

6f3b6f0d3d7a15dca8935a0bec7bfb0aee1edc2de319eb16f6cd71892d40f8892fd05e28

but when the seed,

d2ebccb3cff4ea6a6c7ba4d2528cf60c14e0f4d1af762bdda9442efb94cdc857

is passed to getAccountData(...), the return balance is 0.

Steps to Reproduce

Download the Wallet.

Create a new account with your own seed:

Send some value to this 0 valued account.

Wait for status of receiving account to clear pending transaction and update balance.

Now, obtain the address that has new value. Go to settings > view addresses, and copy the address with balance.

Paste the address with balance into the getBalances API method. Check the result.

Now check the return results of getAccountData with the seed of the account you created and just funded.

Version:

• Operating System: Ubuntu bionic beaver / MacOS
• Clean install of helix-lib master branch, Wallet downloaded on 10th Sept 2019.

Expected Behavior

After the transaction is broadcasted, the transaction hash should get referenced under the milestone issued by the coordinator after some time and getInclusionState with the transaction hash should resolve to true

Current Behavior

The transaction doesn't get referenced to milestone even after several hours

Steps to Reproduce

1. Create any random transaction in the network
2. Call the getInclusionState API with the transaction hash
3. It should resolve to true after some time but currently its not.

• Version: ^1.0.0-alpha.2
• Operating System: Ubuntu 18.04

The docs attached to the packages still contain wrong information.

Hey Cristina,

I tested your code from here:
https://github.com/netobjex/helix.api/commit/f858ab1db25ab340067f337b279818f054815dad

there were two things going wrong. The tag in TransactionConverter.asTransactionHBytes() (line 60) was padded to wrong size:

padHBytes(OBSOLETE_TAG_BYTE_SIZE)(
        transaction.tag || transaction.obsoleteTag
      )

obsolete tag has a size of 256 bits and tag has a size of 64 bits. This resulted in different bytes and in a different transaction hash.
I fixed it with

1. removing tag padding and take the tag as it is
2. pad zeros at the end to reach the full transaction size

export function asTransactionHBytes(
  transactions: Transaction | ReadonlyArray<Transaction>
): HBytes | ReadonlyArray<HBytes> {
  const txHBytes = asArray(transactions).map(transaction =>
    [
      transaction.signatureMessageFragment,
      transaction.address,
      hBitsToHBytes(hbits(transaction.value)),
      padHBytes(OBSOLETE_TAG_BYTE_SIZE)(transaction.obsoleteTag),
      hBitsToHBytes(hbits(transaction.timestamp)),
      hBitsToHBytes(hbits(transaction.currentIndex)),
      hBitsToHBytes(hbits(transaction.lastIndex)),
      transaction.bundle,
      transaction.trunkTransaction,
      transaction.branchTransaction,
      transaction.tag,
      hBitsToHBytes(hbits(transaction.attachmentTimestamp)),
      hBitsToHBytes(hbits(transaction.attachmentTimestampLowerBound)),
      hBitsToHBytes(hbits(transaction.attachmentTimestampUpperBound)),
      transaction.nonce,
      "000000000000000000000000000000000000000000000000"
    ].join("")
  );

  return Array.isArray(transactions) ? txHBytes : txHBytes[0];
}

When building the hash with Transaction.transactionHash() we need bytes as input and because asTransactionHBytes() returns a hex string we have to convert it before:

.then(results => {
    console.log("--------------------response from tangle ------------------");

    console.log("export const bundle: Transaction[] = ");
    console.log(results);

    console.log("export const bundleHBytes: HBytes[] = ");
    const hbytesResult = TransactionConverter.asTransactionHBytes(results);
    console.log(hbytesResult);

    const bundleFromTangle = Array.from(results);
    const transactions = Array.from(hbytesResult);
    for (var i = 0; i < transactions.length; i++) {
      console.log("Transaction " + i);
      console.log(
        "Transaction hash from tangle: " + bundleFromTangle[i]["hash"]
      );
      let computedTransactionHash = Transaction.transactionHash(
        Converter.toHBytes(transactions[i])
      );
      console.log(
        "New computed hash (in helix.lib): " + computedTransactionHash
      );
      console.log(
        "Equal hashes? " + bundleFromTangle[i]["hash"] ==
          computedTransactionHash
      );
    }
  })
      );

Expected Behavior

I expect the example/tx.js from the dev branch to work with a zero value transaction with the helix-1.0/dev branch node running on localhost.

Current Behavior

The client returns an Internal Server Error.

Steps to Reproduce

1. Fork helix-lib, clone, and checkout dev branch set to track upstream dev on local computer.

npm init
npm i @helixnetwork/core

1. Fork helix-1-0, clone, and checkout dev branch set to track upstream dev on local computer.
2. Put the private key for the coordinator into helix-1.0/resources
3. Start helix-1.0 with default configurations

java -jar target/helix*.jar -p 14700 

4. Run the example/tx.js from the client directory.

Version:

• Operating System: Ubuntu bionic beaver
• Clean installs of helix-1.0 dev branch, Helix-lib dev branch.

Failure Logs

The steps result in the helix-1.0 log issuing,

07/22 14:20:24.238 [XNIO-1 task-4] ERROR net.helix.hlx.service.API - API Validation failed: Invalid parameters

And the client console issuing,

Request error: Internal Server Error

Note

I think this is related to the refactor of the helix-1.0 node:

HelixNetwork/pendulum@023efc3

IOTA model

IOTA uses a balanced ternary model, which is based on trits (they can contain 3 values: -1, 0, 1). A tryte consists of 3 trits and it can be in one of 27 states (3^3), one state is represented using one of (26) uppercase letters or number 9. e.g 9ABCDEFGHIJKLMNOPQRSTUVWXYZ

Trits in Iota are represented using Int8Array (8 bytes are used for 1 trit), trytes are represented using string. In Iota model most of the members are represented in trytes and computation is done in trytes when this thing is possible otherwise trytes are converted to trists.

• Example of places where trytes are used:
  Bundle creation - concatenation is done in trytes (is plain string concatenation after computation which is done in trits)
  Padding - some variable should have fix length, for this cases some padding with ‘9’ are done. Etc
  Attach to tangle, broadcast bundle

• Examples of places where trits are used: (conversion from trytes is needed):
  Hashing algorithms (kerl, curl) their input is a list of bytes (Int8Arrays)
  Signing - subseed computation
  Bundle creation, validation, etc

When trytes are used, string operations are performed: e.g. concatenation, when trits are used bytes array operation are performed.

Converting trytes => trits, each tryte is converted to 3 trits, results for 1 char => 3 bytes
Converting trits => trytes, 3 trits will be converted in 1 tryte, from 3 bytes => 1 char

Problem is when we need to convert a list of bytes (which represent a big integer) to a list of trits, here the conversion is not straightforward and more calculation needs to be done (conversion from base 2 to base 3 for a big integer) - this part is time consuming. (it’s used every time kerl function is called e.g. (signing, bundle validation etc)

Because trytes is a string, model is readable, seed, keys, addresses, bundles etc.

Helix conceptual model

Helix aims to work based on a binary model, based on bit (0,1), one byte contains 8 bits, and it can hold 256 states (2^8).

In this model there is no need to change base for numbers resulted in hash algorithms (improvement comparing with trinary model).
It’s easier to work with bytes comparing with trytes, current processors are using binary logic, and binary model fits better on them.
Iota is a project in progress, every update / bugfix, should be replicated and converted for Helix.
Curl algorithm can not be used, it uses ternary logic which will not make sense in binary model.

Byte & bit in the converted model

What is the best way to represent a Byte & bit in converted model?

There are 2 possibilities:

1. String representation for Byte with UInt8Array for bits:

export type Bytes = string;
Bits = Int8Array
Pro:
fast & easier conversion implementation model from trinary to binary
Easier to compare 2 values
For 1 byte information, it will be 64 bits input for hash algorithm (1 byte real information, but 8 bytes list)
Cons:
We still have conversions from string to byte arrays
For each store bytes we use 8 times more space

2. Uint8Array representation for Bytes (no specific representation for bits)

export type Bytes = Uint8Array;
bit = 1 bit (⅛ part of a byte) (limitation, bits length should be multiple of 8)
Pro:
less consumed space, but faster computation (no string conversion)
Hashing algorithms (e.g. kerl, sha256, except curl) are using bytes / words (conversion is easier from bytes to wordlist comparing with conversion using string.
There is no need to convert bytes to bit.
Cons:
It will increase model conversion time implementation
More modification are required comparing with previous approach
For 1 byte information, it will be 1 byte input for hash algorithm
There is still need to have a string representation for trytes (e.g addresses)

Conclusion

What is the best method to deal with the above conversions in the presented context?