openmined / syft-proto Goto Github PK

The JVM stubs aren't currently published by the publish action. We'll probably need to add credentials to the repo secrets in order to make this work.

This issue doesn't need to live in syft-proto, but it must exist as some sort of a Github issue somewhere. This project is probably the best central location for it for now.

This issue describes how we will need to develop a testing suite that can run span multiple repositories, languages, and environments. We want to ensure that our worker libraries (syft.js, KotlinSyft, SwiftSyft, and PySyft's FL Worker) are always working properly with PyGrid. Likewise, we'll need to test this on various versions of syft-proto and Threepio to ensure that our assisting libraries pair nicely with the other various projects. We need to do roughly the following:

• Create a central Dockerfile to provision and organize all our necessary resources
• Create model, training plan, and averaging plan and upload them to PyGrid instance
• Allow syft.js to participate in a round of training
• Write tests to ensure that PyGrid is updating the model from syft.js round
• Allow PySyft FL Worker to participate in a round of training
• Write tests to ensure that PyGrid is updating the model from PySyft FL Worker round

We should ideally cover the following cases, at the minimum:

• Is the round able to start?
• Is the round able to take reports?
• How do we manage a variety of different server_config's?
• ... others we should have? Let's open a discussion in the comments below.

Update 29/05/2020 - We do not currently have any plans to include KotlinSyft or SwiftSyft in our end-to-end testing because of their dependence on a live device or an emulator. Until we have the ability to implement these libraries in a "headless" context, they will not be practical to test end-to-end. We're open to challenging this thought and would love to implement them, but don't see a clear path at the moment.

I installed syft-proto from source.

Traceback (most recent call last): File "run_websocket_client.py", line 10, in <module> import syft as sy File "/Users/hrishikesh/opt/anaconda3/envs/py3_6_3/lib/python3.6/site-packages/syft/__init__.py", line 15, in <module> import syft.frameworks.torch.hook.hook_args File "/Users/hrishikesh/opt/anaconda3/envs/py3_6_3/lib/python3.6/site-packages/syft/frameworks/torch/hook/hook_args.py", line 6, in <module> from syft.frameworks.torch.tensors.interpreters.native import TorchTensor File "/Users/hrishikesh/opt/anaconda3/envs/py3_6_3/lib/python3.6/site-packages/syft/frameworks/torch/tensors/interpreters/native.py", line 13, in <module> from syft.frameworks.torch.tensors.interpreters.crt_precision import _moduli_for_fields File "/Users/hrishikesh/opt/anaconda3/envs/py3_6_3/lib/python3.6/site-packages/syft/frameworks/torch/tensors/interpreters/crt_precision.py", line 7, in <module> from syft.frameworks.torch.tensors.interpreters.precision import FixedPrecisionTensor File "/Users/hrishikesh/opt/anaconda3/envs/py3_6_3/lib/python3.6/site-packages/syft/frameworks/torch/tensors/interpreters/precision.py", line 7, in <module> from syft.generic.pointers.multi_pointer import MultiPointerTensor File "/Users/hrishikesh/opt/anaconda3/envs/py3_6_3/lib/python3.6/site-packages/syft/generic/pointers/multi_pointer.py", line 11, in <module> from syft.workers.base import BaseWorker File "/Users/hrishikesh/opt/anaconda3/envs/py3_6_3/lib/python3.6/site-packages/syft/workers/base.py", line 21, in <module> from syft.generic.pointers.object_pointer import ObjectPointer File "/Users/hrishikesh/opt/anaconda3/envs/py3_6_3/lib/python3.6/site-packages/syft/generic/pointers/object_pointer.py", line 15, in <module> from syft.messaging.message import ForceObjectDeleteMessage File "/Users/hrishikesh/opt/anaconda3/envs/py3_6_3/lib/python3.6/site-packages/syft/messaging/message.py", line 14, in <module> from syft_proto.messaging.v1.message_pb2 import OperationMessage as OperationMessagePB ImportError: cannot import name 'OperationMessage'

Question

Hi, how could I generate the protobuf/msgpack customized Message class file for the websocketAPI in PySyft, where are these tutorials? Thank you~

The action isn't required to pass before merging so that it serves as a warning but doesn't block anything. However, it isn't failing when it should for some reason.

The stub-building script currently runs the buf linter, compiles the Protobuf schemas, generates Python and Java stubs, builds JS stubs, and (tries to) publish a JVM package. The script can probably be modularized somewhat, so that it's possible to run the linter (and maybe breaking change detection?) independently, build the stubs without publishing a package, etc

Several versions of 0.1.1a1 that are in pypi are missing. The versions in question have a .post## naming convention. The 0.1.1a1.post20 and 0.1.1a1 versions have significant differences between them. Could tags be added for these missing releases?

Where?

Main readme

Who?

All contributors

What?

Users of this library should have a good idea of where to get support within Slack so that the general channel does not become a place for people dropping software implementation problems. Please add the following section somewhere in your readme:

## Support
For support in using this library, please join the **#lib_syft-proto** Slack channel. If you’d like to follow along with any code changes to the library, please join the **#code_syft-proto** Slack channel. [Click here to join our Slack community!](https://slack.openmined.org)

When finished, it should look exactly as it does here: https://github.com/OpenMined/.github/blob/master/README-TEMPLATE.md#support

Additional Context

None

With JS and JVM generated code already in their own folder, it would be convenient to have Python in a directory too.

As you can see by the screenshots attached below, we're duplicating codes for some of the keys. I don't know why this was ever merged in the first place, but this shouldn't be allowed.

The breaking change detection Github action "works", in that it runs and detects breaking changes. However, the Github action doesn't fail when there are breaking changes.

(#77 was an attempt to fix this, but didn't work as expected.)

The publish action is failing with this error:

FAILURE: Build failed with an exception.

* What went wrong:
Task 'bintray' is ambiguous in root project 'syft-proto'. Candidates are: 'bintrayPublish', 'bintrayUpload'.

Description

We need to rename this project slightly to fit our new project naming standards.

The new name for this repo should be: SyftProtobuf
The release name depends on the language:

• Python release name: openmined.syftprotobuf
• Javascript release name: @openmined/syft-protobuf.js
• Kotlin release name: org.openmined.kotlin-syft-protobuf
• Swift release name: OpenMinedSwiftSyftProtobuf

This is the one repo I will not be renaming myself due to potential unknown breaking changes in doing so.

If you have any questions about this issue, please comment here or message @cereallarceny on Slack.

Placing on this repo since there's not a good repo to encompass all other repos...

In theory, the syft-proto project will drive a lot of class creation in PySyft when those classes will inevitably be used by a worker library (syft.js, KotlinSyft, or SwiftSyft). In order to avoid issues of subtleties existing between various versions of OpenMined projects, it may be best to centralize all continuous integration, end-to-end testing, and version release management into one place. This isn't an issue that necessitates a code change (but if it does, it should be fairly minimal), but more a designation that we will need to hook up all the projects together into a CI pipeline that is run on an interval and runs our end-to-end tests (#37). This ensures that all versions of the workers are compatible with PySyft and PyGrid. This will allows to almost catch issues before our users do.

The following are classes that exist in PySyft that we have no yet migrated to syft-proto, or classes that are in the middle of a refactor and will need to be modified upon completion:

(items in bold are expected to be needed for mobile)

PyTorch

• memory_format

Numpy

• ndarray
• float32
• float64
• int32
• int64
• ...any other dtypes...

PySyft Basics

• BaseWorker
• VirtualWorker (can rely on BaseWorker serialization for this)
• TrainConfig (deprecated training method should be removed from PySyft)
• String

PySyft Pointers and Wrappers

• MultiPointerTensor
• ObjectPointer
• ObjectWrapper
• PointerPlan
• PointerProtocol

PySyft Tensors

• LoggingTensor
• AutogradTensor
• CRTPrecisionTensor
• FixedPrecisionTensor
• GradFunc
• PrivateTensor
• PlaceholderTensor

PySyft Messaging

• Message (generic message type should not be serialized to Protobuf)
• Plan as a list of operations
• Plan as Torchscript for inference
• Plan as Torchscript for training
• State
• Protocol

PySyft Messages

• ForceObjectDeleteMessage
• GetShapeMessage
• IsNoneMessage
• ObjectMessage
• ObjectRequestMessage
• Operation
• PlanCommandMessage
• SearchMessage

PySyft Exceptions

• GetNotPermittedError
• ResponseSignatureError

Note we haven't documented it yet, but buf protoc is now integrated into buf, and would replace much of your logic piping the output of buf image build -o - to protoc --descriptor_set_in - would recommend checking it out! It's a drop-in replacement to protoc, but does use protoc for the builtin plugins java, cpp, etc.

Based on this readme template, we should improve our readmes across all OpenMined projects.

Please consult the following document for our desired best practices.