WARNING!
py-ethpm
is deprecated, and no longer maintained.
Please use the ethpm module in web3.py instead.
This library is deprecated. ethPM python tooling is now located in web3.py
License: MIT License
WARNING!
py-ethpm
is deprecated, and no longer maintained.
Please use the ethpm module in web3.py instead.
We need to expand the validation of packages to look at the build_dependencies
key.
For the initial implementation add the following functionality to the code which validates the package during instantiation of the Package
object.
build_dependencies
key is present in the package and it is non-empty, raise a NotImplementedError
indicating that validation and loading of build dependencies has not yet been implemented.Being able to deal with individual packages (and their dependencies) is good, but a higher level of abstraction is needed for this library to be truly useful and allow packaging to be a black box for other libraries that use this tool.
Implement a new PackageManager
class. This class will be responsible for the following things.
Assuming we have a collection of the package data for a number of packages, installation is the process of writing those to disk in some structured way. This should be written as a simple pluggable backend system so that we can support multiple installation schemes.
Our initial default installation scheme will be derived from the code found at populus/packages/installation.py.
TODO: translate that code to a spec here or in a dedicated issue linked from here.
This is a slightly higher level interface that should cover installation, upgrading, and removal of packages.
TODO: do we need to support a package.json
document of some sort?
In addition, we should support pip freeze
style exporting of currently installed packages.
The heavy lifting is going to exist in the various package backends. A package backend is responsible for sourcing package data to be used for package installation.
We need to support the following use cases across some minimal set of default backends.
owned==0.1.0 => ipfs://Qm...
ipfs://Qm... > package_data
/path/to/package.json => package_data
It's worth noting that ultimately, we need to have everything reduce down to package_data
but some sources like a package index may only do a partial reduction from owned==0.1.0 > URI
and then we'll need another backend to do the reduction from URI > package_data
. This structure is already in place in the populus codebase and should be used as a starting point for this functionality.
TOOD: new ticket with thorough spec on how this API should work.
Packages can contain the addresses of deployed contract instances. We need an API for retrieving these.
Since packages can contain deployments across multiple chains we'll need an API that allows us to access these. I propose the following two APIs
Package.deployments
Package.get_deployments(w3)
The Package.deployments
API would use the default web3 instance that the Package
is currently configured with. If the Package
has no web3 instance accessing this property should throw an exception indicating that a an instance of web3 is reuired.
The Package.get_deployments(w3)
allows a user to specify a web3 instance.
Both of these APIs should return a Deployments
object which exposes a subset of the normal dictionary interface.
Deployments.get('MyContract')
Deployments['MyContract']
'MyContract' in Deployments
Deployments.keys()
Deployments.items()
Deployments.values()
TODO: figure out what to do about validation and what that API looks like. Does this issue need to stub out any of that functionality.
We need an API for fetching the various contract types that are available in a given package.
Make the following API modifications:
Package
object to accept an optional Web3 instance.class Package:
def __init__(self, package_identifier, web3=None):
if web3 is not None:
self.web3 = web3
...
Add a new API: Package.set_default_web3(web3)
The set_default_web3(...)
function can be used to set the default web3
instance that the Package
will use.
Add a new API: Package.get_contract_type(name, web3=None)
name
should be validated to be a valid contract name matching the regex: [a-zA-Z][-a-zA-Z0-9_]{0, 255}
web3
argument is not None
then it should be used as the web3
instance used to generate the contract factory. otherwise it should use the web3
instance located at self.web3
. If self.web3
is None
then an exception should be raised.name
is found in the contract_types
of the package, it's data should be passed to the web3.eth.contract
API to generate a contract factory class which should be used as the return value for get_contract_type
function.If there is no Infura environment variable set, web3 v5 tests can't run because ethpm is loading Infura here: https://github.com/ethpm/py-ethpm/blob/master/ethpm/backends/registry.py#L5
@pipermerriam suggested a few things:
Since we need to get this out before 3/27, we can back out ethpm from v5 for a minute, and then add it back in after we've made "the fix"
"The fix" might look like:
Probably all of these options require some documentation updates
Note: I'm not sure if this is the right repo in which to record this thought. Let's move this issue ticket once we figure that out.
It seems possible that our pinning service, which will be listening for registry events and is supposed to pin package manifests and assets, might be down or not able to pin resources for some reason. In the event that the service misses the opportunity to pin a required resource, the package author would want or need to re-publish the package and associated resources.
We need to make sure that authors have a way to rectify this kind of situation or have a way of telling if their package was successfully published and pinned. Maybe some features such as the following would be useful:
Need to setup travis-ci. Mirroring the .travis.yml
and tox.ini
files from https://github.com/pipermerriam/ethereum-utils should be a good starting point. we can drop testing against 2.7
and 3.4
. Would be good to add testing against 3.6
Right now we have to copy and paste updates to V2 schema as the master version gets updated in ethpm-spec
Remove assets
folder, include the ethpm-spec
repo as a submodule here and directly refer to it's schema AND it's packages
Currently blocked (as of May 4) - schema in ethpm-spec
is still on V1, but is going through the process of updating to V2 - should be updated by May 14
STUB
validate that within the deployments
of a package, all of the contracts types are present in the contract_types
.
It's still kind of hard for people to work with ethpm packages.
But other than the first two which are snake charmer projects, it's likely that the 3rd party projects end up implementing support in their own
This has me thinking about the concept of filesystem installs for ethpm packages. I implemented this once in populus a long time ago:
I'm wondering if there's utility in shipping a CLI for installing packages to disk as well as going ahead and defining the specification for the layout of packages on disk so that 3rd party integration is well supported. This would lower barrier to using EthPM based features for both Vyper and Web3.py use case and should reduce some barriers to adding support to some of those other 3rd party frameworks.
Here's the general layout as I remember it from the Populus implementation.
The below is written using UNIX environment variable syntax for simplicity sake.
$CWD
be the current working directory.$PACKAGES_DIR
be the directory that packages are installed to. This defaults to the relative path ./ethpm_packages
$CWD
To install a package a user must provide the following information.
PACKAGE_IDENTIFIER
: specifies the package to installALIAS
: an optional alias name to install the package under
package_name
from the EthPM specThe PACKAGE_IDENTIFIER
concept should be fully fleshed out, potentially using a URI scheme, however for this spec, I'll define two identifier types which can generally be mapped to URI schemes.
NAME
: The name of the package (as it exists in the registry)REGISTRY_URI
: the package registry to install the package from.VERSION
: The version identifier to use to resolve the package.
=1.2.3
, >=1.2.3,<1.8
, etcInstallation of packages is a recursive operation, with the top level namespace being comprised of user specified packages and deeper name spacing being the dependencies of the user specified packages. Here is what a single package namespace looks like.
|--- CWD
|---PACKAGES_DIR
|--- ethpm.lock
|--- <package-name>
| |--- package.json
| |--- SRC/<package-source-tree>
| |--- PACKAGES_DIR (same directory format containing installed dependencies for this package.
| |--- <...>
|--- <...>
```
When a user installs a package, the directory structure above is lazily created. Each top level package is installed to it's own directory as follows.
- `$PACKAGES_DIR/<package-name>/`
Within that directory, the following files and directories are created.
- `package.json`: The package manifest for the package
- A directory named `./src` which contains the fully resolved package source tree.
- another `$PACKAGES_DIR` which contains all of the installed dependencies for the package.
- This could probably be omitted for packages without dependencies.
Each `$PACKAGES_DIR` contains a top level file named `ethpm.lock`. The purpose of this file is track what packages are installed and what mechanism they were installed with. This is necessary to preserve the information of where packages came from in for packages installed via mechanisms like registry URIs. The file structure is out of scope for this spec but it would contain thing like the following.
- The registry address
- The package name specified by the user
- The alias for the package
- The version identifier specified by the user
- The version that was resolved (and installed)
- The resolved content hash (probably normalized)
A util function would be useful that accepts a valid github uri pointing to a manifest json file, and returns the uri as defined here.
The utility function should be written in https://github.com/ethpm/py-ethpm/blob/master/ethpm/utils/uri.py. It should validate that the given uri leads to a json file hosted on github, validate the json file against the manifest json schema, and return a uri with the specified encoding here. Tests should be written here
Why Optional
? Assuming this is to take into account packages initialized without a URI
, what if we just generated a data URI for those? Data URIs could always be valid since they contain their content.
Originally posted by @pipermerriam in https://github.com/ethpm/py-ethpm/pull/153/files
Right now we only support fetching packages on IPFS, and only through the INFURA gateway.
Also extend backend so it can swap in a dummy backend that will serve content rather than monkeypatching http requests. (see piper's comment here . . . #46)
Add configurable option to allow fetching packages from other gateways & local nodes
I'm a bit in love with the "builder" concept so take this with a grain of salt. It may not be a good idea.
Tools for functional compisition based manifest building.
manifest = build_manifest(
setup_manifest(), # stubs out some initial data to seed the manifest.
set_meta(author=...), # injects the `meta` information into the manifest.
set_meta(license=...),
set_contract_types(
ERC20Token=extract_contract_type('ERC20Token', solc_compiler_output),
owned=extract_contract_type('owned', solc_compiler_output),
),
set_deployments(
...
),
prettify(),
)
Rather than try to build something that is infinitely flexible, build a lot of small tools that are designed to be chained together to build a manifest... We could pair on this.
Valid manifests must be packed according to specific rules.
The document must be tightly packed, meaning no linebreaks or extra whitespace.
The keys in all objects must be sorted alphabetically.
Duplicate keys in the same object are invalid.
The document must use UTF-8 encoding.
The document must not have a trailing newline.
We need to enforce that manifests used to instantiate packages follow these rules
Fill this section in if you know how this could or should be fixed.
It would be good if we could standardize our tests into a set of portable JSON documents which describe the inputs and expected results for things like loading and validating packages and their internal data. This would serve as a framework for other languages which implement a similar library to ensure that everyone is developing against the same spec.
Similar to how the ethereum/tests
repository contains JSON based tests for EVM implementations, we should develop the same. We can start with them living in this repository and then move them to their own repository in due time.
Something I've been thinking about since the library got into link reference land is whether we can come up with less confusing names for things.
We don't have to use the same names for things as the spec does....
I'm thinking about this in the context of link_references
and link_dependencies
. I pretty much always have to think about which is which and even then, I'm regularly not 100% sure I'm right. Maybe we could rename those to something like:
Contract.unlinked_references
Contract.linked_references
Probably some better names out there if any other wordsmiths want to take a stab at this.
See the conversation here
Rename the base module to ethpm
(by renaming the ./erc190
directory to be ./ethpm
instead.
This change should be reflected in setup.py
, tox.ini
, and really anywhere else that you can find in the files.
w3 = Web3(Web3.IPCProvider("/home/bryant/.ethereum/testnet/geth.ipc"))
from web3.pm import PM
PM.attach(web3, 'pmp')
owned_package = w3.pmp.get_package_from_manifest({
...: "manifest_version": "2",
...: "version": "1.0.0",
...: "package_name": "owned",
...: "meta": {
...: "license": "MIT",
...: "authors": [
...: "Piper Merriam <[email protected]>"
...: ],
...: "description": "Reusable contracts which implement a privileged 'owner' model for authorization.",
...: "keywords": [
...: "authorization"
...: ],
...: "links": {
...: "documentation": "ipfs://QmUYcVzTfSwJoigggMxeo2g5STWAgJdisQsqcXHws7b1FW"
...: }
...: },
...: "sources": {
...: "./contracts/Owned.sol": "ipfs://Qme4otpS88NV8yQi8TfTP89EsQC5bko3F5N1yhRoi6cwGV"
...: }
...: })
owned_package.get_contract_type('owned')
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-55-a85b3dd83952> in <module>()
----> 1 owned_package.get_contract_type('owned')
~/.pyenv/versions/3.6.5/envs/pytest-ethereum/lib/python3.6/site-packages/ethpm/package.py in get_contract_type(self, name, w3)
95
96 validate_contract_name(name)
---> 97 validate_w3_instance(current_w3)
98
99 if name in self.package_data['contract_types']:
~/.pyenv/versions/3.6.5/envs/pytest-ethereum/lib/python3.6/site-packages/ethpm/utils/contract.py in validate_w3_instance(w3)
46 def validate_w3_instance(w3: Web3) -> None:
47 if w3 is None or not isinstance(w3, Web3):
---> 48 raise ValueError("Package does not have valid web3 instance.")
49
50
ValueError: Package does not have valid web3 instance.
Somewhat confusing validation util with an even more confusing flag (allow_missing_data
). Improvement would be helpful. See conversation in pr #138
No type hints
Add type hints
Unhandled KeyError
:
w3 = Web3(Web3.IPCProvider("/home/bryant/.ethereum/testnet/geth.ipc"))
pm = PM(w3) # `PM.attach(w3, 'pmp')` didn't seem to do anything
package = {
...: "manifest_version": "2",
...: "version": "1.0.0",
...: "package_name": "owned",
...: "meta": {
...: "license": "MIT",
...: "authors": [
...: "Piper Merriam <[email protected]>"
...: ],
...: "description": "Reusable contracts which implement a privileged 'owner' model for authorization.",
...: "keywords": [
...: "authorization"
...: ],
...: "links": {
...: "documentation": "ipfs://QmUYcVzTfSwJoigggMxeo2g5STWAgJdisQsqcXHws7b1FW"
...: }
...: },
...: "sources": {
...: "./contracts/Owned.sol": "ipfs://Qme4otpS88NV8yQi8TfTP89EsQC5bko3F5N1yhRoi6cwGV"
...: }
...: }
owned_package = pm.get_package_from_manifest(owned_package)
owned_package.set_default_w3(w3) # Not set when initialized
owned_package.get_contract_type('owned')
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-44-a85b3dd83952> in <module>()
----> 1 owned_package.get_contract_type('owned')
~/.pyenv/versions/3.6.5/envs/pytest-ethereum/lib/python3.6/site-packages/ethpm/package.py in get_contract_type(self, name, w3)
97 validate_w3_instance(current_w3)
98
---> 99 if name in self.package_data['contract_types']:
100 contract_data = self.package_data['contract_types'][name]
101 validate_minimal_contract_data_present(contract_data)
KeyError: 'contract_types'
I stumbled across ethereum/web3.py#1242 and though: "Wouldn't it be neat if we could distribute smart contract packages in a first class way from using python packaging."
Which made me think of how pytest does it for plugins.
https://docs.pytest.org/en/latest/writing_plugins.html#making-your-plugin-installable-by-others
So suppose that we used the same API. Here's a sketch of how it might work.
Suppose we create a new project, vyper-registry
. It's a python package that contains the various manifests for the vyper implementation of the registry contract. That package would ship with those JSON documents as part of the files it installs. Here's a simple example source tree.
vyper_registry
└── assets
├── 1.0.0.json
└── 1.1.0.json
So this is nice and all, and from another library like web3.py
we could get at the by 1) knowing where they are and 2) loading them from the filesystem. This works for individual cases, but it fails as a general solution since you have to know how the files are laid out and each project does this differently.
This is where setuptools entry points come in. Here's a simple idea of what this might looks like in the setup.py
of the vyper-registry
package.
# ./setup.py file
from setuptools import setup
setup(
name="vyper_registry",
packages=["vyper_registry"],
# the following is what would make the plugin available to py-ethpm
entry_points={"ethpm11": ["./assets/1.0.0.json", "./assets/1.1.0.json"]},
)
The specifics of the various strings don't matter much at this point.
ethpm11
could really be any string as long as we are pretty sure it isn't going to accidentally collide with some other project. The file paths could also be any strings we wanted as long as they allowedpy-ethpm
to find the manifests.
I haven't dug into the distutils APIs to figure out how we get at these entry points but... lets pretend it looks something like this.
# this isn't real code, I don't know what the actual `distutils` APIs are.
from distutils import get_entry_points
def get_from_python_package(identifier):
all_entry_points = get_entry_points("ethpm11")
manifest = ... # filter and find the *right* one.
return manifest
With this, a project like web3 could then do the following.
from ethpm import get_package
manifest = get_from_python_package('vyper-registry>=1.1.0,<2')
# do things with it...
Whatcha think?
Create a new class Package
that has the following functionality.
package_identifier
which is required.package_identifier
argument is a filesystem path, that file is loaded and parsed as JSON. Once parsed, it should be validated against the JSON-schema for a packagepackage_identifier
is not a filesystem path or there is no file at the given filesystem path or the file fails validation, instantaition throws an error.Package
instance.More details
Package
class should reside in ethpm/package.py
.Package
class should be importable as from ethpm import Package
from #152
Allow 3rd party URIs to be registered and accepted. Bitbucket, gitlab, gists, .... ipfs-gateway uris (there can be more than one gateway, cloudflair has one). The EthPM spec is intentionally open about the definition of a content addressable URI and thus, there are tons of options out there for people to include the content hash in different URI schemes.
Probably a similar mechanism to the eth-abi library with some kind of Registry implementation that allows new ones to be added with the library supporting the common use case with a default registry with the built-in URIs pre-registered.
Packages have a few simple properties:
These properties should be exposed on the Package
object through the Package.name
and Package.version
properties.
def name(self):
... # return the package name pulled from the package info
Additionally, implement a __repr__
function on the Package
object which returns a string formatted like <Package {package_name}=={version})>
.
The package named
tokens
at version1.2.3
would return the string"<Package owned==1.2.3>"
when repr is called on it.
Right now the Package.__init__
expects to be given a path to a lockfile. Ideally, we'd like to be able to instantiate Package
objects with any of:
We could make the Package.__init__
method accept any of the formats and do content negotiation there. However this approach is likely to result in complex initialization logic that is hard to test and isolate.
Instead lets do the following.
Package
object to only accept the package data.classmethod
methods on the Package
object:
Package.from_file
Package.from_uri
Both of the from_*
methods will retrieve the lockfile JSON from their respective sources and then initialize a new Package
object.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.