ciscocloud / mantl-api Goto Github PK

Mantl API

License: Apache License 2.0

Go 98.63% Shell 0.21% Makefile 1.16%

mantl-api's Issues

No way to install local packages.

Currently there is no possibility to specify local repo/package for Mantl API to install it.
I'm currently working on adding this functionality.

@ryane do you have any comments about this?

remove drone badge from README, add Travis

install supported package when version not requested

Installing a package without a specific version requested should install the latest supported version of the package.

Currently, it will attempt to install the latest version of a package even if it is not supported.

New Upstream Repository

Please submit new PR's to https://github.com/mantl/mantl-api as it is the new upstream for this repository.

improve drone build

Automated build
Automatically push tagged images to docker hub

separate install/uninstall from /packages endpoints

It might make sense to move package installations to another endpoint (/installs?) instead of /packages.

Expand environment variables when looking for configuration file

Expand environment variables in the path specified for the config-file argument. So that you can specify something like:

$MESOS_SANDBOX/config.toml

Add /health endpoint for marathon health check

Inconsistencies with \install or \installs

To install a package is the uri /install or installs ? I see both on this page

https://github.com/CiscoCloud/mantl-api#usage

but neither work

mantl-api panics when framework is nil

Hi @ryane

When packages.json does not contain a value for "framework":, mantl-api panics with the following message:

2016/05/06 19:48:34 http: panic serving 10.192.71.109:57460: interface conversion: interface is nil, not bool
goroutine 5249 [running]:
net/http.(*conn).serve.func1(0xc8200ca160, 0x7f6c035380c0, 0xc8201142e0)
    /usr/lib/go/src/net/http/server.go:1287 +0xb5
github.com/CiscoCloud/mantl-api/install.packageCatalog.packagesIndex(0xc8201d8240, 0xc820192048, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/CiscoCloud/mantl-api/install/catalog.go:82 +0x631
github.com/CiscoCloud/mantl-api/install.(*Install).getPackages(0xc8204db1a0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/CiscoCloud/mantl-api/install/package.go:324 +0xb2
github.com/CiscoCloud/mantl-api/install.(*Install).getPackageByName(0xc8204db1a0, 0xc820136110, 0x7, 0x0, 0x0, 0x0)
    /go/src/github.com/CiscoCloud/mantl-api/install/package.go:328 +0x42
github.com/CiscoCloud/mantl-api/install.(*Install).Package(0xc8204db1a0, 0xc820136110, 0x7, 0x9a1f98, 0x0, 0x0)
    /go/src/github.com/CiscoCloud/mantl-api/install/install.go:62 +0x41
github.com/CiscoCloud/mantl-api/api.(*Api).describePackage(0xc8201db600, 0x7f6c035940d0, 0xc8200ca210, 0xc8204f42a0, 0xc820136180, 0x1, 0x1)
    /go/src/github.com/CiscoCloud/mantl-api/api/api.go:99 +0xe4
github.com/CiscoCloud/mantl-api/api.(*Api).(github.com/CiscoCloud/mantl-api/api.describePackage)-fm(0x7f6c035940d0, 0xc8200ca210, 0xc8204f42a0, 0xc820136180, 0x1, 0x1)
    /go/src/github.com/CiscoCloud/mantl-api/api/api.go:62 +0x5c
github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc820011a80, 0x7f6c035940d0, 0xc8200ca210, 0xc8204f42a0)
    /go/src/github.com/julienschmidt/httprouter/router.go:344 +0x195
github.com/CiscoCloud/mantl-api/api.logHandler.func1(0x7f6c035940d0, 0xc8200ca210, 0xc8204f42a0)
    /go/src/github.com/CiscoCloud/mantl-api/api/api.go:43 +0x1a4
net/http.HandlerFunc.ServeHTTP(0xc8201db760, 0x7f6c035940d0, 0xc8200ca210, 0xc8204f42a0)
    /usr/lib/go/src/net/http/server.go:1422 +0x3a
net/http.serverHandler.ServeHTTP(0xc8204ee060, 0x7f6c035940d0, 0xc8200ca210, 0xc8204f42a0)
    /usr/lib/go/src/net/http/server.go:1862 +0x19e
net/http.(*conn).serve(0xc8200ca160)
    /usr/lib/go/src/net/http/server.go:1361 +0xbee
created by net/http.(*Server).Serve
    /usr/lib/go/src/net/http/server.go:1910 +0x3f6

on the customer end of the connection this appears as a 502 Bad Gateway error.

The problem mesosphere universe repos are:
./repo/packages/M/mysql/0/package.json
./repo/packages/M/mysql/1/package.json
./repo/packages/M/marathon-lb/0/package.json
./repo/packages/M/marathon-lb/1/package.json
./repo/packages/M/marathon-lb/4/package.json
./repo/packages/M/marathon-lb/2/package.json
./repo/packages/M/marathon-lb/3/package.json
./repo/packages/M/marathon/1/package.json
./repo/packages/M/marathon/2/package.json
./repo/packages/L/linkerd/0/package.json
./repo/packages/L/linkerd/1/package.json
./repo/packages/O/openvpn/0/package.json
./repo/packages/O/openvpn-admin/0/package.json
./repo/packages/N/nginx/0/package.json
./repo/packages/N/nginx/1/package.json
./repo/packages/N/namerd/0/package.json
./repo/packages/N/namerd/1/package.json
./repo/packages/W/weavescope/0/package.json
./repo/packages/W/weavescope-probe/0/package.json

I'm on rev 8783d7621439a3211d55ce5e22c128ee04a91d47 of mesosphere/universe and rev 1abaf95 of mantl-api.

I'd be happy to submit a PR for this, but before I do, would you prefer setting meta["framework"] to false if it is nil (with a warning) by wrapping https://github.com/CiscoCloud/mantl-api/blob/master/install/catalog.go#L82 inside an if block, OR should I wrap the whole of the package processing inside a test on err that warns on error but continues to the next package? I think the 2nd option might be more useful and would probably help close out #41 as well.

packages missing if not in base repository

If a full package exists in a layered repository but not in the base repository, it does not appear in the package list

vendoring

No way to install non-DCOS packages

By now Mantl-API leverages only one /repo/meta/index.json in the very last repo in it's repos list. I guess to have the possibility to create Mantl-specific packages we need to create our own index file in repo and slightly rewrite Mantl-API to merge indexes from all the repos.

Also it's a good idea to somehow generalize repo configuration for Mantl-API, make it configurable via some cli argument.

I'm working of both points if @ryane doesn't mind.

Add version command to mantl-api

Run mantl-api version to output the current version information
Log version at startup

Design Document defining integration

As a developer, I know how to package my framework so it can be easily deployed so that the extended teams can add frameworks for the API

@cdmitri @stevendborrelli Please add additional info

Only clone single git branch/commit

The mesosphere universe repo is huge (110mb), so we should only clone the most recent commit of master.

crash when attempting to delete zk node that does not exist

time="2015-10-08T16:04:09Z" level=fatal msg="zk: node does not exist"

Endpoint for documenting options

Options should be returned somewhere logical (perhaps on a call to get the package details)

Framework API Does Not Retreive Active Frameworks

When cleaning up a Cassandra un-install, the framework ID in Mesos is required.

But, using the Mantl-API, an empty JSON is returned, though I can see four active frameworks in the Mesos UI.

curl -sku username:password https://<my-mantl-control-01/api/1/frameworks

Further, when copying the Active Framework ID manually out of the Mesos UI and attempting to delete the framework using that ID via the API, this fails as the framework corresponding to the ID is reported as 'not found'.

curl -X DELETE -sku username:password https://<my-mantl-control-01/api/1/frameworks/{My_Framework_ID}
Cloud not shutdown framework {ID}: 400 No framework found with specified ID

Is this a real issue, or user error in some way?

I'm running Mantl 1.1.0 on Openstack.

cassandra installation fails

I am trying to run cassandra on Marathon using mantl-api.
When I run this command, I dont get any error messages and the Marathon UI always shows "waiting" status for Cassandra app -
curl --insecure -X POST -u admin -d "{"name": "cassandra"}" https://192.168.100.101/api/1/install

Can I get handle to the .json file which is being used for installation on Marathon?
That might give some visibility.
What's the best method to debug such kind of issues in mantl-api?

support arbitrary labels for marathon apps

It should be possible to specify arbitrary labels when installing packages.

Update README to reflect current state of Mantl integration

The README currently describes how to install mantl-api on Mantl, without discussion of how it currently works, i.e. not on port 4001, but behind :80/api/.

support for version-2.x repo schema

The default mesosphere repo schema has changed slightly in version-2.x. mantl-api is not currently compatible.

no way to tell what frameworks are running from the API

mesos framework is not always tore down successfully

When uninstalling hdfs, for example, we often see an error like

time="2015-09-29T15:46:05Z" level=debug msg="POST http://leader.mesos.service.consul:15050/master/teardown" method=POST status="400 Bad Request" statusCode=400 url="http://leader.mesos.service.consul:15050/master/teardown"

And the framework has to be removed manually

sometimes fails to remove frameworks

Intermittently, mantl-api fails to remove a framework during an uninstall. You may see an error like this:

time="2016-06-07T14:09:40Z" level=debug msg="GET http://lb0-control-02:15050/master/state.json" 
time="2016-06-07T14:09:40Z" level=debug msg="Framework mantl/elasticsearch not active"

but, the framework is active.

mantl-api error

This happens whenever i run mantl-api docker container after i provide force-sync option from marathon via environment variable. When mantl-api is first deployed, everything works but after force-sync i have not gotten it to work.

curl -k -u admin:xxx https://xxx/api/1/packages
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.6.2</center>
</body>
</html>
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->

here are the logs from mesos:

time="2016-04-25T04:51:34Z" level=debug msg="GET /1/packages" 
2016/04/25 04:51:35 http: panic serving xxx.xxx.xxx.xx:xxx: interface conversion: interface is nil, not bool
goroutine 28 [running]:
net/http.(*conn).serve.func1(0xc820134c60, 0x7f5eaca5c0e8, 0xc82011e810)
    /usr/lib/go/src/net/http/server.go:1287 +0xb5
github.com/CiscoCloud/mantl-api/install.packageCatalog.packagesIndex(0xc8203a0b40, 0xc82011e060, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/CiscoCloud/mantl-api/install/catalog.go:82 +0x631
github.com/CiscoCloud/mantl-api/install.(*Install).getPackages(0xc8201c6c30, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/CiscoCloud/mantl-api/install/package.go:324 +0xb2
github.com/CiscoCloud/mantl-api/install.(*Install).Packages(0xc8201c6c30, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/CiscoCloud/mantl-api/install/install.go:58 +0x3e
github.com/CiscoCloud/mantl-api/api.(*Api).packages(0xc8203a04e0, 0x7f5eaca65048, 0xc820134d10, 0xc8203ac380, 0x0, 0x0, 0x0)
    /go/src/github.com/CiscoCloud/mantl-api/api/api.go:90 +0x86
github.com/CiscoCloud/mantl-api/api.(*Api).(github.com/CiscoCloud/mantl-api/api.packages)-fm(0x7f5eaca65048, 0xc820134d10, 0xc8203ac380, 0x0, 0x0, 0x0)
    /go/src/github.com/CiscoCloud/mantl-api/api/api.go:68 +0x5c
github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc82037fe80, 0x7f5eaca65048, 0xc820134d10, 0xc8203ac380)
    /go/src/github.com/julienschmidt/httprouter/router.go:344 +0x195
github.com/CiscoCloud/mantl-api/api.logHandler.func1(0x7f5eaca65048, 0xc820134d10, 0xc8203ac380)
    /go/src/github.com/CiscoCloud/mantl-api/api/api.go:46 +0x1a4
net/http.HandlerFunc.ServeHTTP(0xc820326980, 0x7f5eaca65048, 0xc820134d10, 0xc8203ac380)
    /usr/lib/go/src/net/http/server.go:1422 +0x3a
net/http.serverHandler.ServeHTTP(0xc8202d3e60, 0x7f5eaca65048, 0xc820134d10, 0xc8203ac380)
    /usr/lib/go/src/net/http/server.go:1862 +0x19e
net/http.(*conn).serve(0xc820134c60)
    /usr/lib/go/src/net/http/server.go:1361 +0xbee
created by net/http.(*Server).Serve
    /usr/lib/go/src/net/http/server.go:1910 +0x3f6

consul acl support

Ability to use a token to access the mantl-install/apps key path.

UI for installing / removing packages

The UI should:

show health (/health)
present a list of packages, listing installed state for each (/1/packages)
allow filtering unsupported packages (/1/packages)
install a package (POST /1/packages)
uninstall a package (DELETE /1/packages)
get all the details about a package (/1/packages/{name})

hdfs fail to launch after mantl-api DELETE & POST

hi,
I am running a mantl cluster with 5 workers on AWS.
I managed to launch HDFS using mantl-api with the below mesas-site.xml and I can see that the marathon health checks pass.

when I call mantl-api DELETE to remove the HDFS cluster, it seams that every thing is removed and deleted.

the issue is when I call mantl-api POST again to relaunch the HDFS, the marathon task stuck in "deploying" state. and the HDFS is not working properly.

Thanks
Itay

here is my mesas-site.xml:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at
    http://www.apache.org/licenses/LICENSE-2.0
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
    <name>mesos.hdfs.data.dir</name>
    <description>The primary data directory in HDFS</description>
    <value>/var/lib/hdfs/data</value>
  </property>

  <!-- Uncomment this to enable secondary data dir
  <property>
    <name>mesos.hdfs.secondary.data.dir</name>
    <description>The secondary data directory in HDFS</description>
    <value>/var/lib/hdfs/data2</value>
  </property>
  -->

  <property>
    <name>mesos.hdfs.domain.socket.dir</name>
    <description>The location used for a local socket used by the data nodes</description>
    <value>/var/run/hadoop-hdfs</value>
  </property>

  <!-- Uncomment this to enable backup
  <property>
    <name>mesos.hdfs.backup.dir</name>
    <description>Backup dir for HDFS</description>
    <value>/tmp/nfs</value>
  </property>
  -->

  <property>
    <name>mesos.hdfs.native-hadoop-binaries</name>
    <description>Mark true if you have hadoop pre-installed on your host machines (otherwise it will be distributed by the scheduler)</description>
    <value>false</value>
  </property>

  <property>
    <name>mesos.hdfs.framework.mnt.path</name>
    <description>Mount location (if mesos.hdfs.native-hadoop-binaries is marked false)</description>
    <value>/opt/mesosphere</value>
  </property>

  <property>
    <name>mesos.hdfs.state.zk</name>
    <description>Comma-separated hostname-port pairs of zookeeper node locations for HDFS framework state information</description>
    <value>zookeeper.service.consul:2181</value>
  </property>

  <property>
    <name>mesos.master.uri</name>
    <description>Zookeeper entry for mesos master location</description>
    <value>zk://zookeeper.service.consul:2181/mesos</value>
  </property>

  <property>
    <name>mesos.hdfs.zkfc.ha.zookeeper.quorum</name>
    <description>Comma-separated list of zookeeper hostname-port pairs for HDFS HA features</description>
    <value>zookeeper.service.consul:2181</value>
  </property>

  <property>
    <name>mesos.hdfs.framework.name</name>
    <description>Your Mesos framework name and cluster name when accessing files (hdfs://YOUR_NAME)</description>
    <value>hdfs</value>
  </property>

  <property>
    <name>mesos.hdfs.mesosdns</name>
    <description>Whether to use Mesos DNS for service discovery within HDFS</description>
    <value>false</value>
  </property>

  <property>
    <name>mesos.hdfs.mesosdns.domain</name>
    <description>Root domain name of Mesos DNS (usually 'mesos')</description>
    <value>mesos</value>
  </property>

  <property>
    <name>mesos.native.library</name>
    <description>Location of libmesos.so</description>
    <value>/usr/local/lib/libmesos.so</value>
  </property>

  <property>
    <name>mesos.hdfs.journalnode.count</name>
    <description>Number of journal nodes (must be odd)</description>
    <value>3</value>
  </property>

  <!-- Additional settings for fine-tuning -->

  <property>
    <name>mesos.hdfs.jvm.overhead</name>
    <description>Multiplier on resources reserved in order to account for JVM allocation</description>
    <value>1.35</value>
  </property>
  <property>
    <name>mesos.hdfs.hadoop.heap.size</name>
    <value>512</value>
  </property>
  <property>
    <name>mesos.hdfs.namenode.heap.size</name>
    <value>512</value>
  </property>
  <property>
    <name>mesos.hdfs.datanode.heap.size</name>
    <value>512</value>
  </property>
  <property>
    <name>mesos.hdfs.executor.heap.size</name>
    <value>256</value>
  </property>
  <property>
    <name>mesos.hdfs.executor.cpus</name>
    <value>0.1</value>
  </property>
  <property>
    <name>mesos.hdfs.namenode.cpus</name>
    <value>0.25</value>
  </property>
  <property>
    <name>mesos.hdfs.journalnode.cpus</name>
    <value>0.25</value>
  </property>
  <property>
    <name>mesos.hdfs.datanode.cpus</name>
    <value>0.25</value>
  </property>
  <property>
    <name>mesos.hdfs.user</name>
    <value>root</value>
  </property>
  <property>
    <name>mesos.hdfs.role</name>
    <value>*</value>
  </property>
   <property>
    <name>mesos.hdfs.ld-library-path</name>
    <value>/usr/local/lib</value>
  </property>
  <property>    
    <name>mesos.hdfs.datanode.exclusive</name>
    WARNING-It is not advisable to run the datanode on same slave because of performance issues
    <description>Whether to run the datanode on slave different from namenode and journal nodes</description>
    <value>true</value>
  </property>
</configuration>

Smarter uninstall

If you are running a package with a different app id, the uninstall has trouble finding it. We need to be able to support multiple instances of each package with the ability to uninstall specific instances

additional uninstall options

some frameworks and applications leave data behind on the hosts. We should (optionally) support removing this when frameworks are removed.

hdfs
kafka
cassandra
probably others

Verify installation when mesos authentication is disabled

configurable CA cert for verifying Consul, Marathon, Mesos with TLS

There should be three new options:

--consul-ca-cert
--marathon-ca-cert
--mesos-ca-cert

All being the CA cert used to verify the Consul, Marathon, and Mesos servers' certificates. This will allow us to turn off consul-no-verify-ssl, marathon-no-verify-ssl, and mesos-no-verify-ssl in Mantl, since we now have valid self-signed certs 🎢

ability to specify load balancing mode

By default, mantl packages are exposed via the traefik load balancer on edge nodes. It should be possible to control this when installing packages (should it be off by default?). When mantl supports internal load balancing, it should be possible to configure load balancing to something like off | edge | internal.

ping @emdem

ciscocloud / mantl-api Goto Github PK

mantl-api's Issues

Recommend Projects

Recommend Topics

Recommend Org