Fetch compilers from list.json, not the github releases about blockscout-rs HOT 3 CLOSED

The format

Current format

Currently ethereum/solc-bin repository uses the following format for list.json file:

{
  "builds": [
    {
      "path": "solc-linux-amd64-v0.8.14+commit.80d49f37",
      "version": "0.8.14",
      "build": "commit.80d49f37",
      "longVersion": "0.8.14+commit.80d49f37",
      "keccak256": "0x73c9a363ad79aa396a706175521b9ccb41dce2a2eee9f04b58dbac87b1fc8614",
      "sha256": "0xd5b027c86c0f8fecc024d5d4f95d8ea48d8a942d79970310e342370532b502f0",
      "urls": [
        "bzzr://a46ac2b6fa96c949a6f70c05b07f378d565ae51f0de1b6bfb5ee4dbd13820019",
        "dweb:/ipfs/QmPpPcCq91ufSe3ZCNjWs26nuGEuuUNLcgX5Sa3y3wPiPk"
      ]
    },
    {
      "path": "solc-linux-amd64-v0.8.15+commit.e14f2714",
      "version": "0.8.15",
      "build": "commit.e14f2714",
      "longVersion": "0.8.15+commit.e14f2714",
      "keccak256": "0x3811602585e07c5404735f5bd329b11619a855f40ff2e1428bf95cb10c1a57ff",
      "sha256": "0x5189155ce322d57fb75e8518d9b39139627edea4fb25b5f0ebed0391c52e74cc",
      "urls": [
        "bzzr://151d8aa0efc4c02e7d5656b9d2b926dedd3207ca010f787bf2ecc0564eb3c3e4",
        "dweb:/ipfs/QmWdaqCBgtJyXJS5R5CbUYzxqbBuPxkKnKNxVmUtFGf2Zd"
      ]
    }
  ],
  "releases": {
    "0.8.15": "solc-linux-amd64-v0.8.15+commit.e14f2714",
    "0.8.14": "solc-linux-amd64-v0.8.14+commit.80d49f37"
  },
  "latestRelease": "0.8.15"
}

Url to download the compiler is obtained by replacing list.json suffix in the current url for the value of the path field. While it is not seen from the example, builds may contain not only release versions but nightly versions as well. Thus, releases key contains the map from each version to the path which corresponds to the corresponding release version. latestRelease is used to easily obtain the latest released version.

Proposal

Our format should be backwards compatible with the format specified above. That way users may use compilers available from official repository just by specifying the path to corresponding list.json.

So I propose the following format for the verification service:

{
  "builds": [
    {
      "pathPrefix": "https://github.com/blockscout/solc-bin/releases/download"
      "path": "solc-v0.8.14+commit.80d49f37/solc",
      "longVersion": "0.8.14+commit.80d49f37",
      "sha256": "0xd5b027c86c0f8fecc024d5d4f95d8ea48d8a942d79970310e342370532b502f0",
    },
    {
      "path": "solc-linux-amd64-v0.8.15+commit.e14f2714",
      "longVersion": "0.8.15+commit.e14f2714",
      "sha256": "0x5189155ce322d57fb75e8518d9b39139627edea4fb25b5f0ebed0391c52e74cc",
    }
  ]
}

Here we remove all other keys except build (and we may add them in the future if they would be required) and leave only path, longVersion and sha256 keys inside. That way we may easily parse current official list.json format by ignoring fields that we are not interested in:

1. path - used as a suffix instead of list.json for the url list.json was downloaded from.
2. longVersion - used to parse a corresponding compiler version (and we can make a distinction whether the version is release or nightly based on that field).
3. sha256 - a checksum used to verify compiler after download.

An important addition, though, is a new pathPrefix key, which is optional, and only added for compilers which should be downloaded from another url prefix list.json is located at. The application would use that prefix to create an URL where the compiler should be downloaded from. If the key is absent though, current prefix for list.json file is used.

From a user's perspective, they should only specify for a path there list.json is located at through the configuration file. We must require that the file with information about compilers is named list.json and will be looking for it inside the path specified. The user may easily change one list.json to another just by modifying the path inside configuration file (e.g. change from https://solc-bin.ethereum.org/linux-amd64 to https://solc-bin.ethereum.org/macos-amd64 in order to run the service on Mac).

from blockscout-rs.

Updates

After discussion we decided to leave only the path field which will be used for both relative and absolute addresses. So that the final structure will remain the same as in ethereum/solc-bin:

{
  "builds": [
    {
      "path": "https://github.com/blockscout/solc-bin/releases/download/solc-v0.8.14+commit.80d49f37/solc",
      "longVersion": "0.8.14+commit.80d49f37",
      "sha256": "0xd5b027c86c0f8fecc024d5d4f95d8ea48d8a942d79970310e342370532b502f0",
    },
    {
      "path": "solc-linux-amd64-v0.8.15+commit.e14f2714",
      "longVersion": "0.8.15+commit.e14f2714",
      "sha256": "0x5189155ce322d57fb75e8518d9b39139627edea4fb25b5f0ebed0391c52e74cc",
    }
  ]
}

The only difference would be that it would be in how we interpret that value. If the path starts with some scheme prefix (e.g. https://) we interpret it as an absolute path and are not add any prefix when trying to download corresponding compiler. When the path does not start with scheme prefix, we interpret it the way it is currently interpreted in ethereum/solc-bin. That is, we prepend the path with a prefix url list.json is located at.

We decided to go this way as it seems to be more intuitive than adding additional fields which complements or exclude (e.g. absolutePath which is used instead of path if present) the path field. That way, a user may just add a full URL to download the compiler and not bother themselves with any additional details. In a rare circumstances if that becomes a problem, they may go to the source code (or documentation, if any) and look for the difference between how paths with and without schema prefixes are handled.

from blockscout-rs.

Implemented in #41

from blockscout-rs.