What needs to be built?

We need to expand the validation of packages to look at the build_dependencies key.

For the initial implementation add the following functionality to the code which validates the package during instantiation of the Package object.

• if the build_dependencies key is present in the package and it is non-empty, raise a NotImplementedError indicating that validation and loading of build dependencies has not yet been implemented.
• This validation should occur after the schema validation step.

What was wrong?

Being able to deal with individual packages (and their dependencies) is good, but a higher level of abstraction is needed for this library to be truly useful and allow packaging to be a black box for other libraries that use this tool.

How can it be fixed?

Implement a new PackageManager class. This class will be responsible for the following things.

• Given a set of packages, install them.
• Managing the set of currently installed packages.
• Backend style API for sourcing packages to be installed (like from a package index contract).

Low level package installation

Assuming we have a collection of the package data for a number of packages, installation is the process of writing those to disk in some structured way. This should be written as a simple pluggable backend system so that we can support multiple installation schemes.

Our initial default installation scheme will be derived from the code found at populus/packages/installation.py.

TODO: translate that code to a spec here or in a dedicated issue linked from here.

Managing packages

This is a slightly higher level interface that should cover installation, upgrading, and removal of packages.

TODO: do we need to support a package.json document of some sort?

In addition, we should support pip freeze style exporting of currently installed packages.

Package Backends

The heavy lifting is going to exist in the various package backends. A package backend is responsible for sourcing package data to be used for package installation.

We need to support the following use cases across some minimal set of default backends.

• Human readable strings to URI: owned==0.1.0 => ipfs://Qm...
• IPFS URI to package data: ipfs://Qm... > package_data
• Filesystem paths to package data: /path/to/package.json => package_data

It's worth noting that ultimately, we need to have everything reduce down to package_data but some sources like a package index may only do a partial reduction from owned==0.1.0 > URI and then we'll need another backend to do the reduction from URI > package_data. This structure is already in place in the populus codebase and should be used as a starting point for this functionality.

TOOD: new ticket with thorough spec on how this API should work.

What was wrong?

Packages can contain the addresses of deployed contract instances. We need an API for retrieving these.

How can it be fixed?

Since packages can contain deployments across multiple chains we'll need an API that allows us to access these. I propose the following two APIs

• Package.deployments
• Package.get_deployments(w3)

The Package.deployments API would use the default web3 instance that the Package is currently configured with. If the Package has no web3 instance accessing this property should throw an exception indicating that a an instance of web3 is reuired.

The Package.get_deployments(w3) allows a user to specify a web3 instance.

Both of these APIs should return a Deployments object which exposes a subset of the normal dictionary interface.

• Deployments.get('MyContract')
• Deployments['MyContract']
• 'MyContract' in Deployments
• Deployments.keys()
• Deployments.items()
• Deployments.values()

TODO: figure out what to do about validation and what that API looks like. Does this issue need to stub out any of that functionality.

We need an API for fetching the various contract types that are available in a given package.

What should be done

Make the following API modifications:

• Change the constructor of the Package object to accept an optional Web3 instance.

class Package:
    def __init__(self, package_identifier, web3=None):
        if web3 is not None:
            self.web3 = web3
        ...

Add a new API: Package.set_default_web3(web3)

The set_default_web3(...) function can be used to set the default web3 instance that the Package will use.

Add a new API: Package.get_contract_type(name, web3=None)

• The name should be validated to be a valid contract name matching the regex: [a-zA-Z][-a-zA-Z0-9_]{0, 255}
• If the web3 argument is not None then it should be used as the web3 instance used to generate the contract factory. otherwise it should use the web3 instance located at self.web3. If self.web3 is None then an exception should be raised.
• If a key by the given name is found in the contract_types of the package, it's data should be passed to the web3.eth.contract API to generate a contract factory class which should be used as the return value for get_contract_type function.

What was wrong?

If there is no Infura environment variable set, web3 v5 tests can't run because ethpm is loading Infura here: https://github.com/ethpm/py-ethpm/blob/master/ethpm/backends/registry.py#L5

How can it be fixed?

@pipermerriam suggested a few things:
Since we need to get this out before 3/27, we can back out ethpm from v5 for a minute, and then add it back in after we've made "the fix"
"The fix" might look like:

• Passing a web3 instance to ethpm (least user friendly option, but least error prone)
• Check for the environment variable and auto connect to infura if the environment variable is set
• Something else...?

Probably all of these options require some documentation updates

Note: I'm not sure if this is the right repo in which to record this thought. Let's move this issue ticket once we figure that out.

What's wrong?

It seems possible that our pinning service, which will be listening for registry events and is supposed to pin package manifests and assets, might be down or not able to pin resources for some reason. In the event that the service misses the opportunity to pin a required resource, the package author would want or need to re-publish the package and associated resources.

How can we address this?

We need to make sure that authors have a way to rectify this kind of situation or have a way of telling if their package was successfully published and pinned. Maybe some features such as the following would be useful:

• The service should have some kind of view that shows if a package is fully pinned.
• Perhaps the registry API should include some kind of contact information for package owners? This could allow registry supervisor services such as ours to notify authors if any issues come up with their packages.

Need to setup travis-ci. Mirroring the .travis.yml and tox.ini files from https://github.com/pipermerriam/ethereum-utils should be a good starting point. we can drop testing against 2.7 and 3.4. Would be good to add testing against 3.6

What was wrong?

Right now we have to copy and paste updates to V2 schema as the master version gets updated in ethpm-spec

How can it be fixed?

Remove assets folder, include the ethpm-spec repo as a submodule here and directly refer to it's schema AND it's packages

Currently blocked (as of May 4) - schema in ethpm-spec is still on V1, but is going through the process of updating to V2 - should be updated by May 14

What was wrong?

STUB

validate that within the deployments of a package, all of the contracts types are present in the contract_types.

What was wrong?

It's still kind of hard for people to work with ethpm packages.

1. Vyper might add support for using EthPM for contract interfaces
2. Web3.py supports working with packages but right now you have to do a bit of configuration to make it work.
3. Maybe we get Solidity onboard at some point, or web3.js, or brownie, or ....

But other than the first two which are snake charmer projects, it's likely that the 3rd party projects end up implementing support in their own

How can it be fixed?

This has me thinking about the concept of filesystem installs for ethpm packages. I implemented this once in populus a long time ago:

https://github.com/ethereum/populus/blob/4e9cc1efcea2333f48ceec9081ef5392d87608e8/populus/packages/installation.py

I'm wondering if there's utility in shipping a CLI for installing packages to disk as well as going ahead and defining the specification for the layout of packages on disk so that 3rd party integration is well supported. This would lower barrier to using EthPM based features for both Vyper and Web3.py use case and should reduce some barriers to adding support to some of those other 3rd party frameworks.

Here's the general layout as I remember it from the Populus implementation.

ROUGH Filesystem Installation Spec

The below is written using UNIX environment variable syntax for simplicity sake.

• let $CWD be the current working directory.
• let $PACKAGES_DIR be the directory that packages are installed to. This defaults to the relative path ./ethpm_packages
  • If the path is relative it is resolved with respect to $CWD

To install a package a user must provide the following information.

• PACKAGE_IDENTIFIER: specifies the package to install
• ALIAS: an optional alias name to install the package under
  • This is needed to allow installation of multiple versions of the same package under the same the same namespace.
  • Subject to the same name validity rules as package_name from the EthPM spec

The PACKAGE_IDENTIFIER concept should be fully fleshed out, potentially using a URI scheme, however for this spec, I'll define two identifier types which can generally be mapped to URI schemes.

• By Content Hash (such as an IPFS or SWARM URI)
• A Registry URI:
  • NAME: The name of the package (as it exists in the registry)
  • REGISTRY_URI: the package registry to install the package from.
  • VERSION: The version identifier to use to resolve the package.
    • Things like =1.2.3, >=1.2.3,<1.8, etc

Installation of packages is a recursive operation, with the top level namespace being comprised of user specified packages and deeper name spacing being the dependencies of the user specified packages. Here is what a single package namespace looks like.

|--- CWD
  |---PACKAGES_DIR
    |--- ethpm.lock
    |--- <package-name>
    | |--- package.json
    | |--- SRC/<package-source-tree>
    | |--- PACKAGES_DIR  (same directory format containing installed dependencies for this package.
    |   |--- <...>
    |--- <...>
```

When a user installs a package, the directory structure above is lazily created.  Each top level package is installed to it's own directory as follows.

- `$PACKAGES_DIR/<package-name>/`

Within that directory, the following files and directories are created.

- `package.json`: The package manifest for the package
- A directory named `./src` which contains the fully resolved package source tree.
- another `$PACKAGES_DIR` which contains all of the installed dependencies for the package.
    - This could probably be omitted for packages without dependencies.

Each `$PACKAGES_DIR` contains a top level file named `ethpm.lock`.  The purpose of this file is track what packages are installed and what mechanism they were installed with.  This is necessary to preserve the information of where packages came from in for packages installed via mechanisms like registry URIs.  The file structure is out of scope for this spec but it would contain thing like the following.

- The registry address
- The package name specified by the user
- The alias for the package
- The version identifier specified by the user
- The version that was resolved (and installed)
- The resolved content hash (probably normalized)

What was wrong?

A util function would be useful that accepts a valid github uri pointing to a manifest json file, and returns the uri as defined here.

How can it be fixed?

The utility function should be written in https://github.com/ethpm/py-ethpm/blob/master/ethpm/utils/uri.py. It should validate that the given uri leads to a json file hosted on github, validate the json file against the manifest json schema, and return a uri with the specified encoding here. Tests should be written here

Why Optional? Assuming this is to take into account packages initialized without a URI, what if we just generated a data URI for those? Data URIs could always be valid since they contain their content.

Originally posted by @pipermerriam in https://github.com/ethpm/py-ethpm/pull/153/files

What was wrong?

Right now we only support fetching packages on IPFS, and only through the INFURA gateway.

Also extend backend so it can swap in a dummy backend that will serve content rather than monkeypatching http requests. (see piper's comment here . . . #46)

How can it be fixed?

Add configurable option to allow fetching packages from other gateways & local nodes

What was wrong?

I'm a bit in love with the "builder" concept so take this with a grain of salt. It may not be a good idea.

How can it be fixed?

Tools for functional compisition based manifest building.

manifest = build_manifest(
    setup_manifest(),  # stubs out some initial data to seed the manifest.
    set_meta(author=...),  # injects the `meta` information into the manifest.
    set_meta(license=...),
    set_contract_types(
        ERC20Token=extract_contract_type('ERC20Token', solc_compiler_output),
        owned=extract_contract_type('owned', solc_compiler_output),
    ),
    set_deployments(
        ...
    ),
    prettify(),
)

Rather than try to build something that is infinitely flexible, build a lot of small tools that are designed to be chained together to build a manifest... We could pair on this.

What was wrong?

Valid manifests must be packed according to specific rules.

The document must be tightly packed, meaning no linebreaks or extra whitespace.
The keys in all objects must be sorted alphabetically.
Duplicate keys in the same object are invalid.
The document must use UTF-8 encoding.
The document must not have a trailing newline.

We need to enforce that manifests used to instantiate packages follow these rules

How can it be fixed?

Fill this section in if you know how this could or should be fixed.

What is wrong?

It would be good if we could standardize our tests into a set of portable JSON documents which describe the inputs and expected results for things like loading and validating packages and their internal data. This would serve as a framework for other languages which implement a similar library to ensure that everyone is developing against the same spec.

How can it be fixed?

Similar to how the ethereum/tests repository contains JSON based tests for EVM implementations, we should develop the same. We can start with them living in this repository and then move them to their own repository in due time.

Something I've been thinking about since the library got into link reference land is whether we can come up with less confusing names for things.

We don't have to use the same names for things as the spec does....

I'm thinking about this in the context of link_references and link_dependencies. I pretty much always have to think about which is which and even then, I'm regularly not 100% sure I'm right. Maybe we could rename those to something like:

• Contract.unlinked_references
• Contract.linked_references

Probably some better names out there if any other wordsmiths want to take a stab at this.

See the conversation here

Rename the base module to ethpm (by renaming the ./erc190 directory to be ./ethpm instead.

This change should be reflected in setup.py, tox.ini, and really anywhere else that you can find in the files.

• Version: x.0.1.0a17
• Python: 3.6.5
• OS: linux

What was wrong?

w3 = Web3(Web3.IPCProvider("/home/bryant/.ethereum/testnet/geth.ipc"))
from web3.pm import PM
PM.attach(web3, 'pmp')
owned_package = w3.pmp.get_package_from_manifest({
    ...:   "manifest_version": "2",
    ...:   "version": "1.0.0",
    ...:   "package_name": "owned",
    ...:   "meta": {
    ...:     "license": "MIT",
    ...:     "authors": [
    ...:       "Piper Merriam <[email protected]>"
    ...:     ],
    ...:     "description": "Reusable contracts which implement a privileged 'owner' model for authorization.",
    ...:     "keywords": [
    ...:       "authorization"
    ...:     ],
    ...:     "links": {
    ...:       "documentation": "ipfs://QmUYcVzTfSwJoigggMxeo2g5STWAgJdisQsqcXHws7b1FW"
    ...:     }
    ...:   },
    ...:   "sources": {
    ...:     "./contracts/Owned.sol": "ipfs://Qme4otpS88NV8yQi8TfTP89EsQC5bko3F5N1yhRoi6cwGV"
    ...:   }
    ...: })
owned_package.get_contract_type('owned')
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-55-a85b3dd83952> in <module>()
----> 1 owned_package.get_contract_type('owned')

~/.pyenv/versions/3.6.5/envs/pytest-ethereum/lib/python3.6/site-packages/ethpm/package.py in get_contract_type(self, name, w3)
     95 
     96         validate_contract_name(name)
---> 97         validate_w3_instance(current_w3)
     98 
     99         if name in self.package_data['contract_types']:

~/.pyenv/versions/3.6.5/envs/pytest-ethereum/lib/python3.6/site-packages/ethpm/utils/contract.py in validate_w3_instance(w3)
     46 def validate_w3_instance(w3: Web3) -> None:
     47     if w3 is None or not isinstance(w3, Web3):
---> 48         raise ValueError("Package does not have valid web3 instance.")
     49 
     50 

ValueError: Package does not have valid web3 instance.

What was wrong?

Somewhat confusing validation util with an even more confusing flag (allow_missing_data). Improvement would be helpful. See conversation in pr #138

What was wrong?

No type hints

How can it be fixed?

Add type hints

• Version: 0.1.0a17
• Python: 3.6.5 (Not in template)
• OS: linux

What was wrong?

Unhandled KeyError:

w3 = Web3(Web3.IPCProvider("/home/bryant/.ethereum/testnet/geth.ipc"))
pm = PM(w3)  # `PM.attach(w3, 'pmp')` didn't seem to do anything
package = {                                                                                                               
    ...:   "manifest_version": "2",                                                                                                      
    ...:   "version": "1.0.0",                                                                                                           
    ...:   "package_name": "owned",                                                                                                      
    ...:   "meta": {                                                                                                                     
    ...:     "license": "MIT",                                                                                                           
    ...:     "authors": [                                                                                                                
    ...:       "Piper Merriam <[email protected]>"                                                                                  
    ...:     ],                                                                                                                          
    ...:     "description": "Reusable contracts which implement a privileged 'owner' model for authorization.",                          
    ...:     "keywords": [                                                                                                               
    ...:       "authorization"                                                                                                           
    ...:     ],                                                                                                                          
    ...:     "links": {                                                                                                                  
    ...:       "documentation": "ipfs://QmUYcVzTfSwJoigggMxeo2g5STWAgJdisQsqcXHws7b1FW"                                                  
    ...:     }                                                                                                                           
    ...:   },                                                                                                                            
    ...:   "sources": {                                                                                                                  
    ...:     "./contracts/Owned.sol": "ipfs://Qme4otpS88NV8yQi8TfTP89EsQC5bko3F5N1yhRoi6cwGV"
    ...:   }
    ...: }
owned_package = pm.get_package_from_manifest(owned_package)
owned_package.set_default_w3(w3)  # Not set when initialized
owned_package.get_contract_type('owned')
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-44-a85b3dd83952> in <module>()
----> 1 owned_package.get_contract_type('owned')

~/.pyenv/versions/3.6.5/envs/pytest-ethereum/lib/python3.6/site-packages/ethpm/package.py in get_contract_type(self, name, w3)
     97         validate_w3_instance(current_w3)
     98
---> 99         if name in self.package_data['contract_types']:
    100             contract_data = self.package_data['contract_types'][name]
    101             validate_minimal_contract_data_present(contract_data)

KeyError: 'contract_types'

What is wrong

I stumbled across ethereum/web3.py#1242 and though: "Wouldn't it be neat if we could distribute smart contract packages in a first class way from using python packaging."

Which made me think of how pytest does it for plugins.

https://docs.pytest.org/en/latest/writing_plugins.html#making-your-plugin-installable-by-others

So suppose that we used the same API. Here's a sketch of how it might work.

Suppose we create a new project, vyper-registry. It's a python package that contains the various manifests for the vyper implementation of the registry contract. That package would ship with those JSON documents as part of the files it installs. Here's a simple example source tree.

vyper_registry
└── assets
    ├── 1.0.0.json
    └── 1.1.0.json

So this is nice and all, and from another library like web3.py we could get at the by 1) knowing where they are and 2) loading them from the filesystem. This works for individual cases, but it fails as a general solution since you have to know how the files are laid out and each project does this differently.

This is where setuptools entry points come in. Here's a simple idea of what this might looks like in the setup.py of the vyper-registry package.

#  ./setup.py file
from setuptools import setup

setup(
    name="vyper_registry",
    packages=["vyper_registry"],
    # the following is what would make the plugin available to py-ethpm
    entry_points={"ethpm11": ["./assets/1.0.0.json", "./assets/1.1.0.json"]},
)

The specifics of the various strings don't matter much at this point. ethpm11 could really be any string as long as we are pretty sure it isn't going to accidentally collide with some other project. The file paths could also be any strings we wanted as long as they allowed py-ethpm to find the manifests.

I haven't dug into the distutils APIs to figure out how we get at these entry points but... lets pretend it looks something like this.

# this isn't real code, I don't know what the actual `distutils` APIs are.
from distutils import get_entry_points

def get_from_python_package(identifier):
    all_entry_points = get_entry_points("ethpm11")
    manifest = ... # filter and find the *right* one.
    return manifest

With this, a project like web3 could then do the following.

from ethpm import get_package

manifest = get_from_python_package('vyper-registry>=1.1.0,<2')

# do things with it...

Whatcha think?

What is needed.

Create a new class Package that has the following functionality.

• Can be instantiated with a single argument, the package_identifier which is required.
• if the package_identifier argument is a filesystem path, that file is loaded and parsed as JSON. Once parsed, it should be validated against the JSON-schema for a package
• if the package_identifier is not a filesystem path or there is no file at the given filesystem path or the file fails validation, instantaition throws an error.
• if the parsed JSON of the package passes validation the parsed json should be stored locally on the Package instance.

More details

• The Package class should reside in ethpm/package.py.
• The Package class should be importable as from ethpm import Package

What was wrong?

from #152

Allow 3rd party URIs to be registered and accepted. Bitbucket, gitlab, gists, .... ipfs-gateway uris (there can be more than one gateway, cloudflair has one). The EthPM spec is intentionally open about the definition of a content addressable URI and thus, there are tons of options out there for people to include the content hash in different URI schemes.

How can it be fixed?

Probably a similar mechanism to the eth-abi library with some kind of Registry implementation that allows new ones to be added with the library supporting the common use case with a default registry with the built-in URIs pre-registered.

What needs to be done?

Packages have a few simple properties:

• package_name
• version

These properties should be exposed on the Package object through the Package.name and Package.version properties.

def name(self):
   ... # return the package name pulled from the package info

Additionally, implement a __repr__ function on the Package object which returns a string formatted like <Package {package_name}=={version})>.

The package named tokens at version 1.2.3 would return the string "<Package owned==1.2.3>" when repr is called on it.

What was wrong?

Right now the Package.__init__ expects to be given a path to a lockfile. Ideally, we'd like to be able to instantiate Package objects with any of:

• Filesystem path
• Actual lockfile data (the parsed JSON data)
• lockfile URI
• other things we haven't thought of yet.

How can it be fixed?

We could make the Package.__init__ method accept any of the formats and do content negotiation there. However this approach is likely to result in complex initialization logic that is hard to test and isolate.

Instead lets do the following.

• Modify the Package object to only accept the package data.
• Add the following classmethod methods on the Package object:
  • Package.from_file
  • Package.from_uri

Both of the from_* methods will retrieve the lockfile JSON from their respective sources and then initialize a new Package object.