In Ubuntu 16.04.2, using tensorflow 1.3.0, when "import polyaxon", this error occurs.

# python -c "import polyaxon"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/polyaxon/__init__.py", line 6, in <module>
    from . import models
  File "/usr/local/lib/python2.7/dist-packages/polyaxon/models/__init__.py", line 6, in <module>
    from polyaxon.models.base import BaseModel
  File "/usr/local/lib/python2.7/dist-packages/polyaxon/models/base.py", line 8, in <module>
    from polyaxon.estimators.estimator_spec import EstimatorSpec
  File "/usr/local/lib/python2.7/dist-packages/polyaxon/estimators/__init__.py", line 7, in <module>
    from polyaxon.estimators.estimator import Estimator
  File "/usr/local/lib/python2.7/dist-packages/polyaxon/estimators/estimator.py", line 19, in <module>
    from tensorflow.python.estimator.model_fn import MetricKeys
ImportError: cannot import name MetricKeys

I am using minikube to set up a local kubernetes single node cluster. I have set up helm as described in the docs. But when I try to deploy polyaxon by following the docs, I get an error.

temp-training:~ shivam.m$ helm install --wait polyaxon/polyaxon Error: release rousing-peahen failed: clusterroles.rbac.authorization.k8s.io "rousing-peahen-polyaxon-ingress-clusterrole" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["configmaps"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["configmaps"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["endpoints"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["endpoints"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["nodes"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["nodes"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["secrets"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["secrets"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["nodes"], APIGroups:[""], Verbs:["get"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["get"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["ingresses"], APIGroups:["extensions"], Verbs:["get"]} PolicyRule{Resources:["ingresses"], APIGroups:["extensions"], Verbs:["list"]} PolicyRule{Resources:["ingresses"], APIGroups:["extensions"], Verbs:["watch"]} PolicyRule{Resources:["events"], APIGroups:[""], Verbs:["create"]} PolicyRule{Resources:["events"], APIGroups:[""], Verbs:["patch"]} PolicyRule{Resources:["ingresses/status"], APIGroups:["extensions"], Verbs:["update"]}] user=&{system:serviceaccount:kube-system:tiller 8e197f15-1373-11e8-9b02-080027bbca2c [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]} ownerrules=[] ruleResolutionErrors=[clusterroles.rbac.authorization.k8s.io "cluster-admin" not found]

I tried disabling the rbac and running it again but then I get an error related to port allocation.
temp-training:~ shivam.m$ helm install --set=rbac.enabled=false polyaxon/polyaxon Error: release mortal-gorilla failed: Service "mortal-gorilla-docker-registry" is invalid: spec.ports[0].nodePort: Invalid value: 31813: provided port is already allocated

The scheduler build and repush the image even though the image already exist. This behaviour can be improved by checking if the image exists, and skip the build in that case.

Hi,

We have a kubernetes setup on Google Cloud and it´s seems we did everything correctly (no warnings or errors). After depolying polyaxon we get

1. Get the application URL by running these commands:
    export POLYAXON_IP=*******
    export POLYAXON_HTTP_PORT=*******
    export POLYAXON_WS_PORT=*******

    echo http://$POLYAXON_IP:$POLYAXON_HTTP_PORT

2. Setup your cli by running theses commands:

  polyaxon config set --host=$POLYAXON_IP --http_port=$POLYAXON_HTTP_PORT  --ws_port=$POLYAXON_WS_PORT


3. Log in with superuser

  USER: root
  PASSWORD: *******

But when trying out the simple example I notice there are no nodes visible:

> polyaxon cluster

Cluster info:

--------------  ----------------------------------------
build_date      2018-01-31T22:30:55Z
major           1
go_version      go1.9.2b4
git_version     v1.9.2-gke.1
platform        linux/amd64
git_commit      4ce7af72d8d343ea2f7680348852db641ff573af
git_tree_state  clean
minor           9+
compiler        gc
--------------  ----------------------------------------

And after submitting a job, the job does not execute and remains at "building".

Do you have maybe a hint, what´s going wrong?

Thanks.
Vitali

Monitor plugin jobs (tensorboard, notebooks) statuses.
This is related to the issue with multiple instances of tensorboard in #51

#I installed polyaxon and started it but I am not able to get the POLYAXON_IP now.
i think it is the cause of external-ip is pending

Add config tabs to view and edit experiment/job/plugin configs

Allow users to specify the termination policy for distributed training

After examining documentation I don't see possibility to mount actually existing volume with data to newly created experiment's pod.
In the polyaxon-examples I see data prefetched from the Internet. It won't work for multi-gigabyte datasets (real-world scenarios).

I imagine it would be possible to specify persistent storage in polyaxonfile that would be attached to pods. Correct me if I am wrong. Then for first experiment data from one place is mounted and for the second one completely different source of data is specified.

I see a way to specify data share via configuration here but as I understand it is not easily changeable after deployment. It can be workaround to use just one volume that holds entire data for now.
For the long run I think I would be better to make it configurable per job/experiment.

@mouradmourafiq It is actually up to you to decide which way to go of course. This issue is just my suggestion.

I want to deploy polyaxon in CPU，But there has always been amistake？The polyaxon requie the GPU，what should I do？

Reorganizing tasks in DAGs would create deterministic execution workflows, it will allow to avoid issues like #51

Currently, Polyaxon supports an early stopping criterion that stops experiments as soon as the user-defined metric validates/exceeds a certain threshold.

Allow user to define other early stopping strategies.

Hi,

what is the difference between you and:

• http://clipper.ai
• https://www.acumos.org
• https://predictionio.apache.org

you all seem to run on k8s and in one way or another provide some easy way to deploy models, also might offer some turn key models.

Excerpt from polyaxon-postgres deployment:

...
        volumeMounts:
        - mountPath: /var/lib/postgresql/data/pgdata
          name: data
          subPath: postgresql-db
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir: {}
        name: data
...

Although I used persistent volumes in polyaxon configuration, database is not held on them. As I see in above deployment there is just directory on node mounted for postgres.
After resizing my GKE cluster to 0 and back to positive value I have database related errors. Now I even can't purge my polyaxon deployment because polyaxon-clean-experiments fails with following error:

django.db.utils.ProgrammingError: relation "experiments_experiment" does not exist
LINE 1: ...etric_id", "experiments_experiment"."commit" FROM "experimen...

Using this config to run an experiment, then getting experiment via CLI or dashboard will fail.

---
version: 1

project:
  name: pytorch-demo

environment:
  resources:
    gpu:
      requests: 1
      limits: 1

run:
  image: my-image
  cmd: python test.py

Error traceback

...
  486.             response =3D handler(request, *args, **kwargs)
 File "/usr/local/lib/python3.5/dist-packages/rest_framework/generics.py" in=
 get
  284.         return self.retrieve(request, *args, **kwargs)
 File "/usr/local/lib/python3.5/dist-packages/rest_framework/mixins.py" in r=
etrieve
  58.         return Response(serializer.data)
 File "/usr/local/lib/python3.5/dist-packages/rest_framework/serializers.py"=
 in data
  533.         ret =3D super(Serializer, self).data
 File "/usr/local/lib/python3.5/dist-packages/rest_framework/serializers.py"=
 in data
  262.                 self._data =3D self.to_representation(self.instance)
 File "/usr/local/lib/python3.5/dist-packages/rest_framework/serializers.py"=
 in to_representation
   487.                 attribute =3D field.get_attribute(instance)
 File "/usr/local/lib/python3.5/dist-packages/rest_framework/fields.py" in g=
et_attribute
  440.             return get_attribute(instance, self.source_attrs)
 File "/usr/local/lib/python3.5/dist-packages/rest_framework/fields.py" in g=
et_attribute
  99.                 instance =3D getattr(instance, attr)
 File "/usr/local/lib/python3.5/dist-packages/django/utils/functional.py" in=
 __get__
  35.         res =3D instance.__dict__[self.name] =3D self.func(instance)
 File "/polyaxon/web/../polyaxon/experiments/models.py" in resources
  112.         return self.compiled_spec.total_resources
 File "/usr/local/lib/python3.5/dist-packages/polyaxon_schemas/polyaxonfile/=
utils.py" in __get__
  44.         res =3D instance.__dict__[self.name] =3D self.func(instance)
 File "/usr/local/lib/python3.5/dist-packages/polyaxon_schemas/polyaxonfile/=
specification/experiment.py" in total_resources
  150.         cluster, is_distributed =3D self.cluster_def
 Exception Type: TypeError at /api/v1/root/pytorch-demo/experiments/1
Exception Type: TypeError at /api/v1/root/pytorch-demo/experiments/1
Exception Value: 'NoneType' object is not iterable
...

I found this bug in version 0.0.6. Seems like it still exists in latest master.

I have fixed it in my fork. And I will post a merge request. Please check it.

do we need any seed data when install polyaxon ? I meet error in postgresql console:

ERROR: relation "clusters_cluster" does not exist at character 164
STATEMENT: SELECT "clusters_cluster"."id", "clusters_cluster"."created_at", "clusters_cluster"."updated_at", "clusters_cluster"."uuid", "clusters_cluster"."version_api" FROM "clusters_cluster" WHERE "clusters_cluster"."id" = 1

Hi, great project, do you plan support OAuth or external authenticators?

In a similar way JupyterHub does?

So far I only see one would need to deal with user management and passwords within this tool, which normally is better to outsource this to an external auth provider.

Polyaxon currently supports 3 main algorithms for hyperparams tuning, these algorithms require that the user create an experiment group specification each time they want to explore a hyperparams space.

By implementing an approach similar to Vizier, we can provide the users a service to store possible studies to get suggestions from and start experiment groups for trials without the need to create a new polyaxonfile/specification.

How to compile polyaxon? where is the dockerfile for each individual component, such as : api, event, beat, resource and so on? where is the doc show me to build all stuff?

despite being correctly logged in (checked with: polyaxon whoami) and having already successfully uploaded the same code I have the error:

Request to http://35.195.144.125:80/api/v1/root/my_project/repo/upload/ failed with status code 405 Reason: [================================] 18062/18062 - 00:00:00 Error while updating project repo: None
Could not upload code for project "my_project" Error message "405 on http://35.195.144.125:80/api/v1/root/my_project/repo/upload/.".

and at the address the error is Authentication credentials were not provided

Any pointers?

Polyaxon relies heavily on Tensorboard for visualisation, but some frameworks and libraries do not support Tensorboard.

Polyaxon should provide basic visualisation for all metrics collected during the training.

Some search algorithms run in multiple iteration/stages and create multiple experiments in every iteration, e.g. Hyperband, Bayesian optimization... This information is currently stored in the backend, but the end user must be able to tell which experiments were create at which iteration, and if the experiments were ever resumed because they were reselected.

Currently, Polyaxon restart experiments based on their fingerprints (i.e. code commit, params, data, and a copy of last outputs).

There should be two different use cases:

• Restart: the experiment should restart based on the parameters of the previous experiments and possibly an updated version of these params.
• Resume: the experiment should restart based on the same environment.

Add bayesian global optimization with gaussian processes as a search algorithm for experiment groups.

Allow users to override TIME_ZONE when deploying the platform.

• Execution on particular date/time
• Execution on particular date/time range
• Execution on schedule

Current behavior of polyaxon experiment logs don't show logs after experiment is finished, it just show Session ended merely.

$ polyaxon experiment -xp 2 logs
Session ended

Is it a expected behavior? Or something went wrong for me?

In current docs, user.password - 'root' in example config.yaml cannot pass validation, it will fail admin-user job.

Logs from admin-user-job

Traceback (most recent call last):
  File "polyaxon/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/polyaxon/polyaxon/users/management/commands/createuser.py", line 105, in handle
    validate_password(password, self.UserModel(**user_data))
  File "/usr/local/lib/python3.5/dist-packages/django/contrib/auth/password_validation.py", line 56, in validate_password
    raise ValidationError(errors)
django.core.exceptions.ValidationError: ['The password is too similar to the username.', 'This password is too short. It must contain at least 6 characters.']

Many people will copy and paste when trying Polyaxon. I suggest using 'hello%1234' as password in config.yaml in docs to improve user experience.

Add experiment and job log views to the dashboard

Hello there,
when I want to execute the notebook on time series, I get the error

KeyError Traceback (most recent call last)
in () ---> 16 predicted = [i['results'] for i in exp.estimator.predict(get_test_inputs(), hooks=[])]

The key 'results' does not seem to exist and I tried to solve this problem, but it did not work out.
Thanks for your help!

Extend the experiment group scheduler with hyperband hyperparameter optimization.

Link to paper: Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization

All summaries operations should be created in the correct scope to have cleaner tensorboar graph.

During polyaxon tests I spotted problematic place in code:
https://github.com/polyaxon/polyaxon-chart/blob/master/polyaxon/templates/_helpers.tpl#L297

This helper config entry is used by scheduler deployment to spot network address of docker-registry.
For my GKE polyaxon deployment it turned out blocker when value held localhost:31813. Scheduler was not operable. Every call to docker-registry timed-out. I used almost default GKE cluster.

Workaround:
I manually changed POLYAXON_REGISTRY_HOST env value of scheduler deployment to CLUSTER-IP of service and port 5000 and it worked.

To make it working you should ensure config works for multi-node clusters.

Hi!

During my experiments with polyaxon I found following problem.

kubectl logs polyaxon-polyaxon-resources-j9tr4
...
...
...
ERROR 2018-02-22 13:51:14,200 monitor_resources 1 140178581817088 Unhandled exception occurred invalid literal for int() with base 10: '3920m'
Traceback (most recent call last):
  File "/polyaxon/polyaxon/events/management/commands/monitor_resources.py", line 36, in handle
    resources.run(containers, node, persist)
  File "/polyaxon/polyaxon/events/monitors/resources.py", line 158, in run
    update_cluster(gpu_resources)
  File "/polyaxon/polyaxon/events/monitors/resources.py", line 136, in update_cluster
    update_system_nodes()
  File "/usr/local/lib/python3.5/dist-packages/celery/local.py", line 191, in __call__
    return self._get_current_object()(*a, **kw)
  File "/usr/local/lib/python3.5/dist-packages/celery/app/task.py", line 380, in __call__
    return self.run(*args, **kwargs)
  File "/polyaxon/polyaxon/clusters/tasks.py", line 50, in update_system_nodes
    node_dict = ClusterNode.from_node_item(node)
  File "/polyaxon/polyaxon/clusters/models.py", line 120, in from_node_item
    'n_cpus': nodes.get_n_cpus(node),
  File "/polyaxon/polyaxon/spawner/utils/nodes.py", line 21, in get_n_cpus
    return int(node.status.allocatable['cpu'])
ValueError: invalid literal for int() with base 10: '3920m'

This piece of code is not able to work with kubernetes millicpu way of describing CPU resources
https://github.com/polyaxon/polyaxon/blob/master/polyaxon/spawner/utils/nodes.py#L21

Steps to reproduce:

1. Create a project.
2. Start tensorboard with polyaxon CLI.
   polyaxon project start_tensorboard
3. Use polyaxon CLI again to start another tensorboard within the same project.
4. Delete tensorboard twice.
   polyaxon project stop_tensorboard

Expected results:
It is up to authors how can it be handled.
Either not allowing creation of second tensorboard or possibility to choose which tensorboard to stop would be ok.

Actual results:
Only one tensorboard is stopped. Second remains running. Data scientists would be forced to approach k8s cluster and delete tensorboard manually.

Note: This occurs for version 0.0.3 released as chart.

My experiment stuck on building status. I checked logs, found that image was built and tagged successfully, but was not pushed to registry.

(polyaxon) gzc@ubuntu:~$ tail -n 3 /logs/root/cifar10/independents/3
2018-03-28 03:16:31,591 INFO {'aux': {'ID': 'sha256:eca15e3fdea00dbe5cc4367c8134d99197e193e3bb72a58049da845e81384b63'}}
2018-03-28 03:16:31,690 INFO {'stream': 'Successfully built eca15e3fdea0\n'}
2018-03-28 03:16:31,821 INFO {'stream': 'Successfully tagged 10.109.14.143:5000/cifar10:2873c5f4e75e08f3a719f43a8dd8373370822a51\n'}

I am using Polyaxon on minikube with none vm driver on ubuntu.

minikube start command

sudo minikube start --vm-driver=none --feature-gates=Accelerators=true

config.yaml

polyaxonSecret: "RANDOM_STRING"

user:
  username: "root"
  email: "[email protected]"
  password: "hello%1234"

isMinikube: true

rbac:
  enabled: false

ingress:
  enabled: false
serviceType: LoadBalancer

dirs:
  nvidia:
    lib: '/usr/lib/nvidia-375'
    bin: '/usr/lib/nvidia-375/bin'
    libcuda: '/usr/lib/x86_64-linux-gnu/libcuda.so.1'

polyaxon version: 0.0.6
kubernetes version: 1.8.0

I suspect there may be a network issue. But I don't know how to investigate deeper.

Add possibility to persist the monitored resources of jobs and experiments

Since Polyaxon is now supporting more algorithms for hyperparameters search, we need to include more metadata about the search algorithm used, its parameters, and if applicable the iterations.

This annotation should be used both by the CLI and UI.