Bug:

When checked out on public_python (f550ca1 at time of writing) or master (0bf29e1 at time of writing) both tox and setup.py are broken.

Output:

GLOB sdist-make: /home/cgmcintyre/devel/metaparticle/package/python/setup.py
ERROR: invocation failed (exit code 1), logfile: /home/cgmcintyre/devel/metaparticle/package/python/.tox/log/tox-0.log
ERROR: actionid: tox
msg: packaging
cmdargs: ['/home/cgmcintyre/.virtualenvs/METAPARTICLE/bin/python3', local('/home/cgmcintyre/devel/metaparticle/package/python/setup.py'), 'sdist', '--formats=zip', '--dist-dir', local('/home/cgmcintyre/devel/metaparticle/package/python/.tox/dist')]

Traceback (most recent call last):
  File "setup.py", line 4, in <module>
    config = json.loads('./metaparticle_pkg/version.json')
  File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

ERROR: FAIL could not package project - v = InvocationError('/home/cgmcintyre/.virtualenvs/METAPARTICLE/bin/python3 /home/cgmcintyre/devel/metaparticle/package/python/setup.py sdist --formats=zip --dist-dir /home/cgmcintyre/devel/metaparticle/package/python/.tox/dist (see /home/cgmcintyre/devel/metaparticle/package/python/.tox/log/tox-0.log)', 1)

Traceback (most recent call last):
  File "setup.py", line 4, in <module>
    config = json.loads('./metaparticle_pkg/version.json')
  File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Causes:

setup.py is not loading python/metaparticle_pkg/version.json.
python/metaparticle_pkg/version.json has a trailing comma.
python/metaparticle_pkg/version.json is not included in the setup.py's files to distribute.

Related testing issue: tox.ini's second coverage command is not pointing at source metaparicle_pkg but is trying to report on coverage of non-existent metaparticle python package.

I've implemented a fix in cgmcintyr@6c5bae4

I'm happy to submit a pull request if this looks okay :)

If you're using Google Cloud Platform (GCP) and/or Kubernetes Engine and/or prefer to use Google Container Registry (GCR), absent a specific GCR builder, it is possible to authenticate a docker client|CLI against GCR and then use GCR (gcr.io) transparently instead of docker.io

For example, my Docker Hub is docker.io/dazwilkin and customarily:

echo $PASSWORD | docker login --username=dazwilkin --password-stdin https://docker.io

GCR is GCP-project specific. You can authenticate the docker client to your project's GCR with:

PASSWORD=$(gcloud auth application-default print-access-token)
docker login \
  --username=oauth2accesstoken \
  --password=${PASSWORD} \
  https://gcr.io

Then docker registry-related commands are effective against the specified (see below) GCP-project registry. For example, I then replace:

repository: "docker.io/dazwilkin"

With the value of my desired GCP Project ID [[PROJECT-ID]]:

repository: "gcr.io/[[PROJECT-ID]]"

If you're using containers on GCP, this provides simpler security and better network performance.

See:
https://cloud.google.com/container-registry/docs/advanced-authentication#using_an_access_token
https://medium.com/google-cloud/google-container-registry-4aca1fc6cf74

It would be great to have a ruby implementation of Metaparticle/Package

For dotnet, image publish is not enforced. AciExecutor won't work if the image is not published.

An important (and open) question.

Proposals wanted!

I followed the instructions (honest):

virtualenv venv
source venv/bin/activate
pip install metaparticle_pkg

pip freeze | grep metaparticle
metaparticle-pkg==0.6.3

I tried the Python sample as-is:

from metaparticle import containerize

This returns:

Traceback (most recent call last):
  File "web.py", line 5, in <module>
    from metaparticle import containerize
ImportError: No module named metaparticle

I then tried:

from metaparticle_pkg import containerize

This returns:

Traceback (most recent call last):
  File "web.py", line 27, in <module>
    'name': 'test',
TypeError: 'module' object is not callable

I briefly explored the package but it's not immediately obvious to me what needs to be done :-(

At the top of this page...

https://metaparticle.io/tutorials/javascript-sharding/

... is a sentence...

Note, this is an advanced tutorial, please start with the initial tutorial

...where initial tutorial is a link. The link 404s.

The generated link is...

https://metaparticle.io/tutorials/javascript-sharding/tutorial.md

... when it should be...

https://metaparticle.io/tutorials/javascript/

The same is true for all languages and for each section, so all links back from the "advanced" tutorial to the introduction are broken.

The whole library is quite poorly tested, we need better unit testing.

Trying to figure out the strategy for updating a service beyond, deleting or manually using kubectl.

If the docker tag updates when I use something like npm start shouldn't the deployment also get updated?

(preface: totally understand if this is out of scope but could be a cool feature)

Invisible Sandboxing of Applications

One of the fun and innovative things that could be done since metaparticle is in charge of handling the users code and running the specific function passed, is an automatic generation of a seccomp profile for their application/function being run.

Background

Seccomp is "secure computing with filters." It allows developers to write BPF programs that determine whether a given system call will be allowed or not.

It has support in container runtimes as well as k8s.

Integration with metaparticle

Since metaparticle knows the go code that it needs to run, it could generate a list of syscalls required for that, and then automatically apply it to the kubernetes config.

In laymans terms, metaparticle would automatically perfectly sandbox your application so even if a malicious individual cracked the application running, they would only be allowed to execute the syscalls required by the application in the container. This would reduce the attack surface substantially.

Go makes parsing the syscalls easy because of the design. I had personally made a POC of this with the go compiler in the past :)

Just an idea. You could do it with the other languages as well, but I don't know enough about their runtime internals to know how difficult or complex it would be.

Let me preface all of this with: I like the idea behind this project. Any time I like something, I try to poke holes into it because I believe knowing the weaknesses sooner, rather than later, will only make it stronger.

With that said, I wrote an article on metaparticle, playing the devil's advocate. I came up with three main arguments against it: leaky abstractions, pennies vs. quarters, and questioning the philosophical choice of this abstraction layer's location, aka separation of duties. @brendanburns suggested I take this discussion to github.

Because leaky abstractions is just a fact of life, and something that will have to be dealt with, and because I'm not sure if the philosophical debate belongs in here (if you think it should be, I can add it as a separate issue), I'll just focus on the pennies vs. quarters argument for now.

The pennies vs. quarters idea stems from the saying "would you rather be holding a 100 pennies or 4 quarters?". Namely, because the abstraction layer is in the code, managing that abstraction layer becomes many orders of magnitude harder than an centralized abstraction layer that sits outside of the application code. Specifically, I'm referring to three scenarios:

• Changes to the underlying infrastructure, especially changes that touch many applications, will require a lot of redeploy/recompiles of application code due to the tight coupling (also arguably chewing up needless cycles of the developer--but this leads into the separation of duties debate).
• Changes to the underlying packaging/clustering technologies (Docker and Kubernetes in this case) that could break the abstraction layer. This introduces a dependency matrix nightmare.
• Changes to the underlying packaging/clustering technologies that would add new feature/functionality that the abstraction layer could leverage... which requires updates to numerous libraries and redeploy/recompile of many applications.

I look for to the discussion below.

We should detect when code uses metaparticle locks or elections and automaticallly include the elector side car.

Write unit tests for the python codebase.

Currently python/metaparticle_pkg/builder/docker_builder.py does not have any tests for build or publish when docker.APIClient.build or docker.APIClient.push returns an error.

The lines which need to be tested are:

Using release 0.4.0 on MacOS.

The Maven file package/java/pom.xml is missing a couple of Unit test dependencies. Without them running the mvn install command in package/java results in a build failure because package/java/src/test/java/io/metaparticle/test/MetaparticleTest.java can't compile.
The missing dependencies are for JUnit Jupiter, Mockito, and the PowerMock Mockito API.

go build fails with the following:

# github.com/metaparticle-io/package/go/metaparticle
/Users/yuvas/go/src/github.com/metaparticle-io/package/go/metaparticle/docker_impl.go:57:22: cannot use dockerClient (type *client.Client) as type dockerImageClient in field value:
	*client.Client does not implement dockerImageClient (wrong type for ImageBuild method)
		have ImageBuild("github.com/docker/docker/vendor/golang.org/x/net/context".Context, io.Reader, types.ImageBuildOptions) (types.ImageBuildResponse, error)
		want ImageBuild("context".Context, io.Reader, types.ImageBuildOptions) (types.ImageBuildResponse, error)
/Users/yuvas/go/src/github.com/metaparticle-io/package/go/metaparticle/docker_impl.go:57:36: cannot use dockerClient (type *client.Client) as type dockerContainerRunner in field value:
	*client.Client does not implement dockerContainerRunner (wrong type for ContainerCreate method)
		have ContainerCreate("github.com/docker/docker/vendor/golang.org/x/net/context".Context, *container.Config, *container.HostConfig, *network.NetworkingConfig, string) (container.ContainerCreateCreatedBody, error)
		want ContainerCreate("context".Context, *container.Config, *container.HostConfig, *network.NetworkingConfig, string) (container.ContainerCreateCreatedBody, error)
/Users/yuvas/go/src/github.com/metaparticle-io/package/go/metaparticle/docker_impl.go:245:98: cannot use exposedPorts (type "github.com/docker/go-connections/nat".PortSet) as type "github.com/docker/docker/vendor/github.com/docker/go-connections/nat".PortSet in field value
/Users/yuvas/go/src/github.com/metaparticle-io/package/go/metaparticle/docker_impl.go:246:37: cannot use portBindings (type "github.com/docker/go-connections/nat".PortMap) as type "github.com/docker/docker/vendor/github.com/docker/go-connections/nat".PortMap in field value
  

Typically, when we develop a project, we reference libraries and other projects into our source code and then compile them. In dotnet land, the references are located in the csproj file and we pull them from nuget sources.

Now in the distributed computing world, we ideally have loosely coupled services without direct dependencies to other services. However there could be those occasions where one service does depend on another service. If we use the kubernetes services example, we may write a service that depends on nginx being available in the cluster as it may wish to offload some requests to that service.

This brings up the question as to whether there should be a need for metaparticle to have some smarts to enable users to build up their service dependencies in code, or is it wrong architecturally to have this dependency in the first place? If the former is true, should we be checking whether a service/collection of services is reachable when you run your application?

Using JavaScript|Node.js

The "Replicated" example works for me both locally and deployed to a GKE cluster.

The "Sharding" example does not (appear to) work correctly when deployed to a GKE cluster.

I can access it locally running as a single Docker container.

It appears to npm start correctly:

sharding-0:sharding-0 
sharding-0:sharding-0 > [email protected] start /sharding
sharding-0:sharding-0 > node ./index.js
sharding-0:sharding-0 
sharding-0:sharding-0 server up on 8080
sharding-1:sharding-0 
sharding-1:sharding-0 > [email protected] start /sharding
sharding-1:sharding-0 > node ./index.js
sharding-1:sharding-0 
sharding-1:sharding-0 server up on 8080
sharding-2:sharding-0 
sharding-2:sharding-0 > [email protected] start /sharding
sharding-2:sharding-0 > node ./index.js
sharding-2:sharding-0 
sharding-2:sharding-0 server up on 8080

From within the cluster, I can access the StatefulSet pods:

curl sharding-0.sharding:8080
Hello Henry: hostname: sharding-0

But there's no LoadBalancer created as there was this with "Replicated" example. And I'm unable to access the service, ReplicaSet or pods representing the router from within the cluster. I tried port-forwarding to one of the sharding-sharder (RS) pods but it does not respond on the port:

kubectl port-forward sharding-sharder-56dc4995db-hl89w 8080:8080

Flummoxed.

kubectl get deployments
NAME               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
sharding-sharder   3         3         3            3           2m

kubectl get services
NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
sharding           ClusterIP   None            <none>        8080/TCP   2m
sharding-sharder   ClusterIP   10.43.255.169   <none>        8080/TCP   2m

kubectl get rs
NAME                          DESIRED   CURRENT   READY     AGE
sharding-sharder-56dc4995db   3         3         3         2m

kubectl get statefulsets
NAME       DESIRED   CURRENT   AGE
sharding   3         3         3m

kubectl get pods
NAME                                READY     STATUS    RESTARTS   AGE
sharding-0                          1/1       Running   0          4m
sharding-1                          1/1       Running   0          4m
sharding-2                          1/1       Running   0          4m
sharding-sharder-56dc4995db-2hdzb   1/1       Running   0          4m
sharding-sharder-56dc4995db-6dkpw   1/1       Running   0          4m
sharding-sharder-56dc4995db-kbc8h   1/1       Running   0          4m

The size of the Golang Docker image is large, very large (this is for a very simple application, no dependencies):

This makes the pushing/pulling of the image take a significant amount of time - is there any area we might improve here?

(Is it the same with other languages?)

[Am a Googler ;-)] Google Container Registry (GCR) provides Google Cloud Platform (GCP) users with a convenient private Docker registry. The GCP command-line aka gcloud provides convenience wrappers for docker push|pull so it's trivial to extend metaparticle to support GCR with a slightly revised 'push' function.

Here for JavaScript but other languages are same-same...

Add e.g. gcr-builder.js:

(function () {
    var shell = require('shelljs');

    module.exports.build = (img) => {
        shell.exec(`docker build -t ${img} .`);
    };

    module.exports.publish = (img) => {
        shell.exec(`gcloud docker -- push ${img}`);
    };
})();

Augment index.js:

    selectBuilder = (buildSpec) => {
        switch(buildSpec) {
            case 'docker':
                return require('./docker-builder');
            case 'gcr':
                return require('./gcr-builder');
            default:
                throw `Unknown builder: ${buildSpec}`;
        }
    }

Then, reflect the new config in mp.containerize:

const projectID = [PROJECT-ID]
const GCR = `gcr.io/${projectID}`
...
mp.containerize(
	{
		builder: "gcr",
		ports: [8080],
		repository: GCR,
		publish: true,
		public: true,
	},
	() => {
		server.listen(port, (err) => {
			if (err) {
				return console.log("server startup error: ", err);
			}
			console.log(`server up on ${port}`);
		});
	}
);

Then npm start:

npm start

> [email protected] start /.../metaparticle/package/tutorials/javascript
> node ./index.js

Sending build context to Docker daemon  670.2kB
Step 1/4 : FROM node:6-alpine
 ---> 29428556c0cb
Step 2/4 : COPY ./ /metaparticle-example/
 ---> bfa18eb16959
Step 3/4 : RUN npm --prefix /metaparticle-example/ install
 ---> Running in 8343230e3a75
npm WARN [email protected] No repository field.
 ---> a122411d31fd
Removing intermediate container 8343230e3a75
Step 4/4 : CMD npm --prefix /metaparticle-example/ start
 ---> Running in 37356ae1e80d
 ---> e27343f47400
Removing intermediate container 37356ae1e80d
Successfully built e27343f47400
Successfully tagged gcr.io/[PROJECT-ID]/metaparticle-example:latest
The push refers to a repository [gcr.io/[PROJECT-ID]/metaparticle-example]
0163fc9a0d9d: Preparing
d8b55f59c7cc: Preparing
06a3955bac48: Preparing
8aebbd0f4dee: Preparing
52a5560f4ca0: Preparing
8aebbd0f4dee: Layer already exists
06a3955bac48: Layer already exists
52a5560f4ca0: Layer already exists
0163fc9a0d9d: Pushed
d8b55f59c7cc: Pushed
latest: digest: sha256:2a17daae09a4e3b7fc76b7f861fb734aca28f4c423792f65a3a05ed6c79bf25e size: 1368
3bcddd5dce12db8e7079996794112169a437bdb5095c0874dc56971387098040

> [email protected] start /metaparticle-example
> node ./index.js

server up on 8080

and:

It would be great to have a go implementation of Metaparticle/Package

I try the Golang tutorial to learn the Metaparticle. However, it doesn't work. I can fix several issues however, I can't solve an error. I can run through the tutorial, I'd happy to send a pull request for the doc.

1. There is no sample under tutorial/go

It is on the tutorial web page. so no problem.

2. go get fails

go get https://github.com/metaparticle-io/package/go/metaparticle

fails. It is the same error as this issue.

#62

We can solve the same solution (install dep and solve the dependency)

3. The second sample didn't work.

This sample cause an error now, metaparticle.Containerize need &metaparticle.Runtime to run it.
To solve this, just add Runtime. It is added the next sample.

func main() {
	metaparticle.Containerize(
		&metaparticle.Package{
			Name:       "metaparticle-web-demo",
			Repository: "your-docker-user-goes-here",
			Builder:    "docker",
			Verbose:    true,
			Publish:    true,
		},
		func() {
			log.Println("Starting server on :8080")
			http.HandleFunc("/", handler)
			err := http.ListenAndServe(fmt.Sprintf(":%d", port), nil)
			if err != nil {
				log.Fatal("Couldn't start the server: ", err)
			}
		})
}

4. repository name must be canonical error

When I tried

go run main.go

It cause this error. metaparticle already has created the same image, when I try docker push tsuyoshiushio/mdemo by myself, it works.

Successfully built 92f2ff7c4e5d
Successfully tagged tsuyoshiushio/mdemo:latest
panic: Could not push image "tsuyoshiushio/mdemo" to the repository: Error sending push request to docker: repository name must be canonical

goroutine 1 [running]:
github.com/metaparticle-io/package/go/metaparticle.Containerize(0xc4201346c0, 0xc4205a90c0, 0x144d350)
	/Users/ushio/Codes/metaparticle/experiment/src/github.com/metaparticle-io/package/go/metaparticle/metaparticle.go:161 +0x651
main.main()
	/Users/ushio/Codes/metaparticle/experiment/src/github.com/TsuyoshiUshio/MetaparticleSpike/main.go:21 +0x146
exit status 2

This error might be related this issue.

ImagePull in golang client v17.03.1-ce fails with 'repository name must be canonical' #32372

The guy is just down grade docker package, however, I guess it is not proper way to solve this error. Any ideas?

It would be great to have a Python implementation of the metaparticle/package library.

Change docker's builder and executor to use docker's go client API.

Waiting on merge of PR #20 / completion of issue #5

I tried using metaparticle to containerize a simple Spring Boot application but it is not running. I am not sure if maybe I am not doing something correctly or if it is a mismatch between what metaparticle expects and the conventions of Spring Boot. Here is my basic Boot app

package com.example.demo;

import io.metaparticle.annotations.Package;
import io.metaparticle.annotations.Runtime;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import static io.metaparticle.Metaparticle.Containerize;

@SpringBootApplication
@RestController
public class DemoApplication {

	private static final int port = 8080;

	@Runtime(ports = {port})
	@Package(repository="docker.io/ryanjbaxter",
			jarFile="target/demo-0.0.1-SNAPSHOT.jar")
	public static void main(String[] args) {
		Containerize(() -> {
			SpringApplication.run(DemoApplication.class, args);
		});
	}

	@RequestMapping
	public String index() {
		return "Hello Metaparticle!";
	}
}

I then run ./mvnw clean package and it all works fine. But then I run the app using the traditional java -jar target/demo-0.0.1-SNAPSHOT.jar, (This is how you would normally run a Boot app) but it fails with

	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)
Caused by: java.lang.NoClassDefFoundError: io/metaparticle/Util$1
	at io.metaparticle.Util.once(Util.java:45)
	at io.metaparticle.Metaparticle.Containerize(Metaparticle.java:115)
	at com.example.demo.DemoApplication.main(DemoApplication.java:23)
	... 8 more
Caused by: java.lang.ClassNotFoundException: io.metaparticle.Util$1
	at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:94)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 11 more
Caused by: java.io.EOFException: Unexpected end of ZLIB input stream
	at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:240)
	at org.springframework.boot.loader.jar.ZipInflaterInputStream.fill(ZipInflaterInputStream.java:62)
	at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
	at org.springframework.boot.loader.jar.ZipInflaterInputStream.read(ZipInflaterInputStream.java:52)
	at sun.misc.Resource.getBytes(Resource.java:124)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:462)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
	... 17 more

Maintaining many libraries is hell on wheels, especially if they are feature-rich and/or rapidly changing. It would be ideal to only maintain one copy of language-agnostic logic, without forcing the developer to have additional toolchains.

Shared logic can have a single implementation, such as an API server (distinct or extended from k8s) for Metaparticle.

For example, a given library could create the server, then make a single call to create a deployment with specific parameters. The API server could then build and make the Kubernetes API calls, figure out bad/missing parametrs, etc. This leaves the library only responsible for capturing those parameters, making a simple API call, and reporting the result. If Metaparticle winds up adding support for something more complex like a CI/CD pipeline, canary releasing, etc, this would be especially valuable.

I saw that the use of docker's client API was reverted for pushing the images because of lack of authentication support.

Reading docker's client API I found that auth.go has the type AuthConfig which is used to set authentication options on API requests.

So I wanted to get an opinion before I start implementing it.

Adding either the token or the user/password to the metaparticle "annotation" is obviously a bad idea, we don't want anyone commiting secrets to the repositories.

So I was thinking that environment variables are the best (maybe only?) option, so I was thinking that we could define that some environment variables can be used for authenticating with the api, as such:

METAPARTICLE_REGISTRY_USER
METAPARTICLE_REGISTRY_PASSWORD
and
METAPARTICLE_REGISTRY_TOKEN

So, we could check if a token env variable is defined, and use it, otherwise fallback to the user/password env variables. And if none are defined just let it try without authentication and if it fails for auth reasons, suggest setting these variables.

What does everyone think?

I keep running into the following error trying to get the python example to run. Any guidance would be appreciated!

  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 357, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/docker/transport/unixconn.py", line 46, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory
 During handling of the above exception, another exception occurred:
 Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 357, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.6/site-packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 357, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)```

Setting METAPARTICLE_IN_CONTAINER to false in a container has no effect:

~/Repos/metaparticle-io
❯ docker run -itv /var/run/docker.sock:/var/run/docker.sock --rm -v $PWD:/app -w /app docker ash
/app #
/app # apk --update add nodejs
fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz
(1/7) Installing libcrypto1.0 (1.0.2m-r0)
(2/7) Installing libgcc (6.3.0-r4)
(3/7) Installing http-parser (2.7.1-r1)
(4/7) Installing libssl1.0 (1.0.2m-r0)
(5/7) Installing libstdc++ (6.3.0-r4)
(6/7) Installing libuv (1.11.0-r1)
(7/7) Installing nodejs (6.10.3-r1)
Executing busybox-1.26.2-r9.trigger
OK: 26 MiB in 19 packages
/app #
/app # cd testlocaljs/
/app/testlocaljs # node index.js
server up on 8080
^C
/app/testlocaljs # METAPARTICLE_IN_CONTAINER=false node index.js
server up on 8080
^C

Bug

Python 2.7 tests are failing due to import error when trying to import unittest.mock

(. venv/bin/activate; \
tox; \
)
GLOB sdist-make: /home/cgmcintyre/devel/metaparticle/package/python/setup.py
py2 inst-nodeps: /home/cgmcintyre/devel/metaparticle/package/python/.tox/dist/metaparticle_pkg-0.6.4.zip
py2 installed: attrs==17.4.0,backports.ssl-match-hostname==3.5.0.1,certifi==2018.1.18,chardet==3.0.4,configparser==3.5.0,coverage==4.5.1,docker==2.7.0,docker-pycreds==0.2.2,enum34==1.1.6,flake8==3.5.0,funcsigs==1.0.2,idna==2.6,ipaddress==1.0.19,mccabe==0.6.1,metaparticle-pkg==0.6.4,mock==2.0.0,pbr==3.1.1,pluggy==0.6.0,py==1.5.2,pycodestyle==2.3.1,pyflakes==1.6.0,pytest==3.4.1,pytest-cov==2.5.1,pytest-mock==1.7.0,requests==2.18.4,six==1.11.0,urllib3==1.22,websocket-client==0.47.0
py2 runtests: PYTHONHASHSEED='1510474863'
py2 runtests: commands[0] | coverage erase
py2 runtests: commands[1] | coverage run --source metaparticle_pkg -m py.test
============================= test session starts ==============================
platform linux2 -- Python 2.7.14+, pytest-3.4.1, py-1.5.2, pluggy-0.6.0
rootdir: /home/cgmcintyre/devel/metaparticle/package/python, inifile:
plugins: mock-1.7.0, cov-2.5.1
collected 0 items / 5 errors

==================================== ERRORS ====================================
____ ERROR collecting metaparticle_pkg/builder/test/test_docker_builder.py _____
ImportError while importing test module '/home/cgmcintyre/devel/metaparticle/package/python/metaparticle_pkg/builder/test/test_docker_builder.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
metaparticle_pkg/builder/test/test_docker_builder.py:6: in <module>
    from unittest.mock import patch
E   ImportError: No module named mock
_____ ERROR collecting metaparticle_pkg/runner/test/test_docker_runner.py ______
ImportError while importing test module '/home/cgmcintyre/devel/metaparticle/package/python/metaparticle_pkg/runner/test/test_docker_runner.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
metaparticle_pkg/runner/test/test_docker_runner.py:6: in <module>
    from unittest.mock import patch, MagicMock
E   ImportError: No module named mock
______ ERROR collecting metaparticle_pkg/runner/test/test_metaparticle.py ______
ImportError while importing test module '/home/cgmcintyre/devel/metaparticle/package/python/metaparticle_pkg/runner/test/test_metaparticle.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
metaparticle_pkg/runner/test/test_metaparticle.py:5: in <module>
    from unittest.mock import patch, MagicMock, mock_open
E   ImportError: No module named mock
_________ ERROR collecting metaparticle_pkg/test/test_containerize.py __________
ImportError while importing test module '/home/cgmcintyre/devel/metaparticle/package/python/metaparticle_pkg/test/test_containerize.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
metaparticle_pkg/test/test_containerize.py:5: in <module>
    from unittest.mock import patch, mock_open, MagicMock
E   ImportError: No module named mock
____________ ERROR collecting metaparticle_pkg/test/test_option.py _____________
ImportError while importing test module '/home/cgmcintyre/devel/metaparticle/package/python/metaparticle_pkg/test/test_option.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
metaparticle_pkg/test/test_option.py:6: in <module>
    from unittest.mock import patch, MagicMock
E   ImportError: No module named mock
!!!!!!!!!!!!!!!!!!! Interrupted: 5 errors during collection !!!!!!!!!!!!!!!!!!!!
=========================== 5 error in 0.31 seconds ============================
ERROR: InvocationError: '/home/cgmcintyre/devel/metaparticle/package/python/.tox/py2/bin/coverage run --source metaparticle_pkg -m py.test'
py3 inst-nodeps: /home/cgmcintyre/devel/metaparticle/package/python/.tox/dist/metaparticle_pkg-0.6.4.zip
py3 installed: attrs==17.4.0,certifi==2018.1.18,chardet==3.0.4,coverage==4.5.1,docker==2.7.0,docker-pycreds==0.2.2,flake8==3.5.0,idna==2.6,mccabe==0.6.1,metaparticle-pkg==0.6.4,mock==2.0.0,pbr==3.1.1,pluggy==0.6.0,py==1.5.2,pycodestyle==2.3.1,pyflakes==1.6.0,pytest==3.4.1,pytest-cov==2.5.1,pytest-mock==1.7.0,requests==2.18.4,six==1.11.0,urllib3==1.22,websocket-client==0.47.0
py3 runtests: PYTHONHASHSEED='1510474863'
py3 runtests: commands[0] | coverage erase
py3 runtests: commands[1] | coverage run --source metaparticle_pkg -m py.test
============================= test session starts ==============================
platform linux -- Python 3.6.4, pytest-3.4.1, py-1.5.2, pluggy-0.6.0
rootdir: /home/cgmcintyre/devel/metaparticle/package/python, inifile:
plugins: mock-1.7.0, cov-2.5.1
collected 26 items

metaparticle_pkg/builder/test/test_docker_builder.py ..                  [  7%]
metaparticle_pkg/runner/test/test_docker_runner.py ...                   [ 19%]
metaparticle_pkg/runner/test/test_metaparticle.py .....                  [ 38%]
metaparticle_pkg/test/test_containerize.py ............                  [ 84%]
metaparticle_pkg/test/test_option.py ....                                [100%]

========================== 26 passed in 0.38 seconds ===========================
py3 runtests: commands[2] | coverage report
Name                                                   Stmts   Miss  Cover
--------------------------------------------------------------------------
metaparticle_pkg/__init__.py                               2      0   100%
metaparticle_pkg/builder/__init__.py                       5      3    40%
metaparticle_pkg/builder/docker_builder.py                32     19    41%
metaparticle_pkg/builder/test/__init__.py                  0      0   100%
metaparticle_pkg/builder/test/test_docker_builder.py      16      1    94%
metaparticle_pkg/containerize.py                          56      3    95%
metaparticle_pkg/option.py                                33      8    76%
metaparticle_pkg/runner/__init__.py                        8      5    38%
metaparticle_pkg/runner/docker_runner.py                  24      1    96%
metaparticle_pkg/runner/metaparticle.py                   25      0   100%
metaparticle_pkg/runner/test/__init__.py                   0      0   100%
metaparticle_pkg/runner/test/test_docker_runner.py        31      1    97%
metaparticle_pkg/runner/test/test_metaparticle.py         58      1    98%
metaparticle_pkg/test/__init__.py                          0      0   100%
metaparticle_pkg/test/test_containerize.py                97      1    99%
metaparticle_pkg/test/test_option.py                      44      3    93%
--------------------------------------------------------------------------
TOTAL                                                    431     46    89%
___________________________________ summary ____________________________________
ERROR:   py2: commands failed
  py3: commands succeeded
Makefile:4: recipe for target 'test' failed
make: *** [test] Error 1

Cause

All python unit tests import functions from unittest.mock, however unittest.mock is only included in the standard library from python version 3.3 onwards (see https://pypi.python.org/pypi/mock).

Fix

Use https://www.python.org/dev/peps/pep-0508/#environment-markers in requirements.txt and setup.py to install mock if python version < 3.3

Change imports to use unittest.mock if python version >= 3.3, otherwise use mock package.

Briefly mentioned this in #50, but I think it deserves its own issue.

Right now as I looked around the packaging, when writing an application you cannot customize the Dockerfile in any way - would it make sense to allow it as a parameter?

I am not saying it should be default, but once the needs of the service overcome what is included in the basic Dockerfile for each runtime, then it might be useful to allow (either in code or as separate file) to come with your own Dockerfile.

Looking at #2 - does it only refer to adding files from the project?
How would you handle external deps?

Radu M

As a cloud-developer,
I would like mount a volume into my container,
So I can use it with my uservice.

Ex. I would like to have a /data or /tmp directory on my container so my code can use to store some data.

My motivation is to include support for Windows containers. I think all that's required is making the following values configurable. At least for dotnet.

This was mentioned in #53 and #51. I don't think #51 addresses this sufficiently. If I have to maintain a separate Dockerfile, then I lose much of the value of Metaparticle.

At the moment, all builder implementation require Docker client locally.

Has it been discussed the possibility of having something similar to Draft that builds and pushes your image without you having anything locally?

Thanks!

To handle batch workloads.

It cause /proc/1/cgroup not found. The issue is fixed alread on this issue/pull request.

#9

However, the npm package seems not updated. Someone who is doing this tutorial might have a problem on Mac until now.

https://www.npmjs.com/package/@metaparticle/package

I think this belongs more in this repo - metaparticle-io/metaparticle-ast#2 based on https://github.com/metaparticle-io/package/blob/master/javascript/metaparticle-package/index.js#L5-L18

following is the exception.
Exception in thread "main" java.lang.NullPointerException at io.metaparticle.Metaparticle.inDockerContainer(Metaparticle.java:19) at io.metaparticle.Metaparticle.Containerize(Metaparticle.java:78) at io.metaparticle.tutorial.Main2.main(Main2.java:24)

It is because of METAPARTICLE_IN_CONTAINER is not set as env variable.
once i set the env variable to false, it is working, i think info regarding this variable has to added to tutorial(tutorial) like exporting the variable , if it will be further used as environment variable.

can't we use System.getProperty("METAPARTICLE_IN_CONTAINER"); instead of System.getenv("METAPARTICLE_IN_CONTAINER"); in metaparticle.java
if we use property, we can pass it as args or we can also set using System.setProperty();
or am i making wrong assumptions.

As it stands, we have nice toy code for metaparticle that runs. Next step is to begin making that useful by building a python package that can be installed, imported and used. I'm currently working on that using these docs: https://packaging.python.org/tutorials/distributing-packages/

Right now the packaging just packages code. We should extend the annotations to include arbitrary files via a FileSet annotation or somesuch.

Currently you need to run the given example.py as root, or else you'll see a permission error:

Traceback (most recent call last):
  File "example.py", line 16, in <module>
    hihi()
  File "/home/wagoodman/github/package/python/metaparticle.py", line 70, in wrapper
    if is_in_docker_container():
  File "/home/wagoodman/github/package/python/metaparticle.py", line 13, in is_in_docker_container
    with open('/proc/1/cgroup', 'r+t') as f:
PermissionError: [Errno 13] Permission denied: '/proc/1/cgroup'

It looks like when you use the 'metaparticle' runner that it assumes a service port will be provided when creating the pods:

Successfully tagged timfpark/metaparticle-example:latest
2017/12/08 15:33:11 Service "metaparticle-example" is invalid: spec.ports: Required value
==> Detected container metaparticle-example-3000602432-cqpx8:metaparticle-example-0

and also that a service exists when you exit:

^Cexec: internal error
F1208 15:40:41.905327 31945 mp-compiler.go:122] services "metaparticle-example" not found
goroutine 1 [running]:
github.com/golang/glog.stacks(0xc42045ce00, 0xc4201e00b0, 0x5c, 0xa9)
/home/bburns/gopath/src/github.com/golang/glog/glog.go:769 +0xa7
github.com/golang/glog.(*loggingT).output(0x20655a0, 0xc400000003, 0xc4201e0000, 0x1fef790, 0xe, 0x7a, 0x0)
/home/bburns/gopath/src/github.com/golang/glog/glog.go:720 +0x36b
github.com/golang/glog.(*loggingT).printf(0x20655a0, 0xc400000003, 0xc4204c6330, 0x29, 0x0, 0x0, 0x0)
/home/bburns/gopath/src/github.com/golang/glog/glog.go:655 +0x14f
github.com/golang/glog.Fatalf(0xc4204c6330, 0x29, 0x0, 0x0, 0x0)
/home/bburns/gopath/src/github.com/golang/glog/glog.go:1148 +0x67
main.main()
/home/bburns/gopath/src/github.com/metaparticle-io/metaparticle-ast/cmd/compiler/mp-compiler.go:122 +0x438

Is it possible to make this an optional element?

PS. Happy to provide a PR if you can point me in the right direction.

Context: #82

It's unclear to me where I should specific `'public': True' in the Python library in order to get a network LB programmed.

Using the Python sample summarized in that issue, succeeds with a K8s Deployment but the Service created exposes a ClusterIP. Per @brendanburns suggestion, there appears no support for "public: True" in the Python code but I am likely wrong.

Including

runtime={
  ...
  'public': True
  ...
}

results:

Unexpected option(s) provided: __new__() got an unexpected keyword argument 'public'

There are 2 Python implementations of metaparticle in the repo.

I'm using the one under package\python\metaparticle_pkg and the options.py specifies:

class RuntimeOptions(namedtuple('Runtime', 'executor replicas ports publicAddress shardSpec jobSpec')):

This appears (!?) to exclude an option for public.

I tried publicAddress with a static IP but that does not work either.

The Python code does generate 3 metaparticle configuration files:

[[NAME]]-deploy.json
[[NAME]]-load-balancer.json
spec.json

The -load-balancer.json omits type: LoadBalancer probably due to the fact that I'm not able to specify 'public': True in the runtime config :

{
  "metadata": {
    "name": "[[NAME]]",
    "creationTimestamp": null
  },
  "spec": {
    "ports": [
      {
        "protocol": "TCP",
        "port": 8080,
        "targetPort": 0
      }
    ],
    "selector": {
      "app": "mp-python"
    }
  },
  "status": {
    "loadBalancer": {}
  }
}

Should the ACI executors use Azure SDK instead of azure cli?

Job specification model is missing in java package. Is this yet to be coded?? Its causing compilation issues.

Thought of opening a new thread rather than expanding on #49

Assuming that the metaparticle is to have a set of tests associated with it, how should they be included? Should they be a seperate project that resides in a particular folder that gets run as part of the build command? Or should they be included along side the code?

Would be interesting to hear your thoughts @brendanburns and others

Metaparticle can provide a good story for making "self-deployable" services and with replica count, we can easily make provide higher availability/performance, yet for many scenarios, specifying replica count explicitly is not enough since scale can change and it can lead to frequent code-changes/redeployment. Providing an auto-autoscale policy that the orchestrator respects can solve this problem.

Kubernetes HPA (https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) might be suited for such cases and it should be fairly easy to configure and implement. (at least for cpu-based scaling)