openziti / ziti Goto Github PK

This will also allow ziti fabric pieces to be plugged into the ziti shell.

Ziti CLI was treating only UUIDs as IDS and everything else as name. Now that we're using short ids, that breaks.

I created an identity and a service and wish to host the service with Reflect. I've enrolled the identity "kentest" for service "kentest", but Reflect server can not "see" the service.

❯ reflect server --identity ~/Downloads/kentest.json --serviceName kentest --verbose
INFO    attempting to authenticate
DEBUG   logged in as kentest/3pYyg34GR                apiSession=zs9pkqVGg
DEBUG   using apiSession apiSession token 438efa33-5afd-4948-ac7a-dd43d49de8f2
DEBUG   started
PANIC   service 'kentest' not found in ZT
panic: (*logrus.Entry) (0x9f6ce0,0xc0000204d0)

goroutine 1 [running]:
github.com/sirupsen/logrus.Entry.log(0xc0000200e0, 0xc00009fef0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/kbingham/gopath/src/github.com/sirupsen/logrus/entry.go:255 +0x313
github.com/sirupsen/logrus.(*Entry).Log(0xc000020700, 0x0, 0xc0001e3ce8, 0x1, 0x1)
        /home/kbingham/gopath/src/github.com/sirupsen/logrus/entry.go:283 +0xeb
github.com/sirupsen/logrus.(*Entry).Panic(0xc000020700, 0xc0001e3ce8, 0x1, 0x1)
        /home/kbingham/gopath/src/github.com/sirupsen/logrus/entry.go:321 +0x55
github.com/openziti/sdk-golang/example/reflect/cmd.Server(0xc000188700, 0x7fff2d4fe8d7, 0x7)
        /home/kbingham/gopath/src/github.com/openziti/sdk-golang/example/reflect/cmd/server.go:16 +0x14e
main.main.func2(0xc0000ef340, 0xc000094280, 0x0, 0x5)
        /home/kbingham/gopath/src/github.com/openziti/sdk-golang/example/reflect/main.go:41 +0x75
github.com/spf13/cobra.(*Command).execute(0xc0000ef340, 0xc000094230, 0x5, 0x5, 0xc0000ef340, 0xc000094230)
        /home/kbingham/gopath/src/github.com/spf13/cobra/command.go:846 +0x29d
github.com/spf13/cobra.(*Command).ExecuteC(0xe61b40, 0xc0000dff68, 0x2, 0x2)
        /home/kbingham/gopath/src/github.com/spf13/cobra/command.go:950 +0x349
github.com/spf13/cobra.(*Command).Execute(...)
        /home/kbingham/gopath/src/github.com/spf13/cobra/command.go:887
main.main()
        /home/kbingham/gopath/src/github.com/openziti/sdk-golang/example/reflect/main.go:54 +0x431

❯ ziti edge list services 'name="kentest"'
id: Wcrkzq4GR    name: kentest    terminator strategy: smartrouting    role attributes: ["kentest"]
results: 1-1 of 1

❯ ziti edge version
Version     : v0.15.2
GIT revision: 2d886ff19c7a
Build Date  : 2020-07-21 15:16:47
Runtime     : go1.14.6

{
  "data": [
    {
      "_links": {
        "configs": {
          "href": "./services/Wcrkzq4GR/configs"
        },
        "self": {
          "href": "./services/Wcrkzq4GR"
        },
        "service-edge-router-policies": {
          "href": "./services/Wcrkzq4GR/service-edge-router-policies"
        },
        "service-policies": {
          "href": "./services/Wcrkzq4GR/service-policies"
        },
        "terminators": {
          "href": "./services/Wcrkzq4GR/terminators"
        }
      },
      "createdAt": "2020-08-03T22:59:07.779Z",
      "id": "Wcrkzq4GR",
      "tags": {
        "mopServiceId": "2b85cabc-01ff-45cc-8530-053d33569f71",
        "networkId": "e534d99b-d64b-4a89-a450-c7ab229e2a1d"
      },
      "updatedAt": "2020-08-03T22:59:07.779Z",
      "config": {},
      "configs": [
        "-K9zk3VMg",
        "iK9kkqVMg"
      ],
      "name": "kentest",
      "permissions": [
        "Bind",
        "Dial"
      ],
      "roleAttributes": [
        "kentest"
      ],
      "terminatorStrategy": "smartrouting"
    }
  ],
  "meta": {
    "filterableFields": [
      "terminatorStrategy",
      "id",
      "createdAt",
      "updatedAt",
      "name"
    ],
    "pagination": {
      "limit": 10,
      "offset": 0,
      "totalCount": 1
    }
  }
}

Allow identities that use CA auto-enrollment to have templates that can alter identities post-enrollment in a re-usable way:

• identity-service configs assigned to the identity when it is created
• tags created

The --resolver option is passed directly to url.Parse, which doesn't do what you'd expect if you specify e.g. a plain filename "file.txt" or an IP address "127.0.0.10" without the scheme ("file://", "udp://")

PATCH https://3.211.201.105:443/identities/1b7b2b9c-ea35-49eb-92b9-0371eaaffaf4

{
	"id": "1b7b2b9c-ea35-49eb-92b9-0371eaaffaf4",
	"name": "kenneth_bingham-laptop",
	"roleAttributes": [
		"sandbox"
	],
	"tags": null
}

responded with status 400 BAD_REQUEST and response body

{
	"error": {
		"args": {
			"urlVars": {
				"id": "1b7b2b9c-ea35-49eb-92b9-0371eaaffaf4"
			}
		},
		"cause": {
			"message": "name is must be unique",
			"field": "name",
			"value": "kenneth_bingham-laptop"
		},
		"causeMessage": "the value 'kenneth_bingham-laptop' for 'name' is invalid: name is must be unique",
		"code": "INVALID_FIELD",
		"message": "The field contains an invalid value",
		"requestId": "b3e120fc-d673-4746-8955-9025c2f20060"
	},
	"meta": {
		"apiEnrolmentVersion": "0.0.1",
		"apiVersion": "0.0.1"
	}
}

❯ ziti edge list identities
id: 1b7b2b9c-ea35-49eb-92b9-0371eaaffaf4    name: kenneth_bingham-mobile    type: Device    role attributes: ["sandbox"]
id: 7882724c-bd9c-455a-ae59-dddc4f10b16a    name: Default Admin    type: User    role attributes: {}
id: c2b4dcd4-7295-4932-b293-5af06d2e0c14    name: kenneth_bingham-laptop    type: Device    role attributes: ["defaultRouters"]
results: 1-3 of 3

--

Edge Router's need to be able to listen on

• multiple NICs, all NICs
• multiple ports
• multiple protocols (TCP, UDP, QUICK, WW, etc.)

Right now the only tunneler that requires the enroller is the linux-based tunnel. Add enroll capability to the ziti-tunnel and possibly deprecate the enroller - maybe

Ken was using the kubernetes quickstart and is getting a new error now:
tun initialization failed: failed to open tun interface (name='', mtu=65535): open /dev/net/tun: no such file or directory

I'll ask him to put exact steps to reproduce here.

Link to the discourse submissions: https://netfoundry.discourse.group/t/sidecar-quickstart-deployment-invalid/70/4

this fails in zac and via cli

eugene@scruffy:~/work/scruffy-env$ ziti edge controller list identities
id: 1fc6b5aa-bc13-450b-beb5-b47d7068457c    name: hermes    type: Device    role attributes: {}                                                                                              
id: b9fba8ad-262c-4169-b4bb-658121bb7454    name: Default Admin    type: User    role attributes: {}                                                                                         
results: 1-2 of 2

eugene@scruffy:~/work/scruffy-env$ ziti edge controller create service-policy hermes-bind1 Bind -i @hermes -r '#all'                                                                         
panic: error creating service-policies instance in Ziti Edge Controller at https://scruffy:1280. Status code: 400 Bad Request, Server returned: {"error":{"args":{"urlVars":{}},"cause":{"message":"no identities found with the given ids","field":"identityRoles","value":["hermes"]},"causeMessage":"the value '[hermes]' for 'identityRoles' is invalid: no identities found with the given ids","code":"INVALID_FIELD","message":"The field contains an invalid value","requestId":"b9624172-6e08-4438-a703-92bc63f02dcb"},"meta":{"apiEnrolmentVersion":"0.0.1","apiVersion":"0.0.1"}}

goroutine 1 [running]:
github.com/netfoundry/ziti-cmd/ziti/cmd/ziti/cmd/edge_controller.runCreateServicePolicy(0xc000297540, 0xc000167d60, 0xc000167d38)                                                            
        /home/eugene/work/github/ziti-cmd/ziti/cmd/ziti/cmd/edge_controller/create_service_policy.go:84 +0x606                                                                               
github.com/netfoundry/ziti-cmd/ziti/cmd/ziti/cmd/edge_controller.newCreateServicePolicyCmd.func1(0xc00029e780, 0xc0002b1c20, 0x2, 0x6)                                                       
        /home/eugene/work/github/ziti-cmd/ziti/cmd/ziti/cmd/edge_controller/create_service_policy.go:54 +0x66                                                                                
github.com/spf13/cobra.(*Command).execute(0xc00029e780, 0xc0002b1bc0, 0x6, 0x6, 0xc00029e780, 0xc0002b1bc0)                                                                                  
        /home/eugene/go/pkg/mod/github.com/spf13/[email protected]/command.go:830 +0x29d
github.com/spf13/cobra.(*Command).ExecuteC(0xf8a0a0, 0x1006998, 0x0, 0x0)
        /home/eugene/go/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
        /home/eugene/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
github.com/netfoundry/ziti-cmd/ziti/cmd/ziti/cmd.Execute()
        /home/eugene/work/github/ziti-cmd/ziti/cmd/ziti/cmd/cmd.go:77 +0x51
main.main()
        /home/eugene/work/github/ziti-cmd/ziti/cmd/ziti/main.go:24 +0x20

eugene@scruffy:~/work/scruffy-env$ ziti edge controller create service-policy hermes-bind1 Bind -i @1fc6b5aa-bc13-450b-beb5-b47d7068457c -r '#all'                                         
0b31d769-8e3a-4e38-a368-e2e5474c21b3

eugene@scruffy:~/work/scruffy-env$ 

The Linux deployment files (.e.g. systems. sysvinit) are in the ziti-edge repo, but they are probably more relevant in ziti-cmd.

Can you please add the details as to 'how' the cleanup works as some doc somewhere?

Originally posted by @dovholuknf in openziti/edge#24 (comment)

All ziti edge commands currently interact with controller. Move commands under controller up a level to ziti edge

Add the ability to generate ECDSA certs in the ziti CLI's pki subcmd. We currently only have the ability to generate RSA certs.

Right now it is not possible to use the CLI w/ certificate authentication. Make it so.

If a controller has more than 1 default page of identities, the CLI's create functionality fails w/ "not found" errors.

Experienced

When creating a CA, identity roles are required. The identity roles flag, -a is optional and when it is not specified the controller will return an error stating that identityRoles is required.

{"error":"validation failure list:\nvalidation failure list:\nidentityRoles in body is required"}

Expected

If -a is no specified an empty array is used for identityRoles when a CA is created and no error is output.

Reproduction

1. Create a CA via the CLI w/o the -a option

"Provided for backwards compatibility with scripts that were coded around older ziti-tunnel versions."

As endpoint's interact with the controller and edge routers:

• log last known public IP address for the:
  -- identity
  -- API session
  -- session

ziti-tunnel currently sets up a local route for the IP address of each intercepted service. The local route is being used to cause locally generated packets to traverse the PREROUTING iptables chain (where TPROXY can be applied), but there's another way to push local packets through PREROUTING:

1. create a custom routing table
2. add a default route to the custom table, sending all traffic to dev lo
3. add an ip rule that sends all packets marked "nf-intercept" to the custom routing table.
4. for each intercepted service, create an iptables rule on the OUTPUT chain marks any packets destined for the intercept ip:port:proto with the "nf-intercept" mark

I downloaded Ziti CLI 0.15.2 for Linux from here.

❯ ziti edge list edge-router-policies
panic: interface conversion: interface {} is nil, not []interface {}

goroutine 1 [running]:
github.com/openziti/ziti/ziti/cmd/ziti/cmd/edge_controller.mapRoleIdsToNames(0xc0002a8140, 0xad54ce, 0xd, 0xad3a14, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/travis/gopath/src/github.com/openziti/ziti/ziti/cmd/ziti/cmd/edge_controller/list.go:612 +0x4aa
github.com/openziti/ziti/ziti/cmd/ziti/cmd/edge_controller.outputEdgeRouterPolicies(0xc0003ae690, 0xc000298008, 0x1, 0x1, 0xc0002882d0, 0x1, 0xc0002882d0)
        /home/travis/gopath/src/github.com/openziti/ziti/ziti/cmd/ziti/cmd/edge_controller/list.go:423 +0x17a
github.com/openziti/ziti/ziti/cmd/ziti/cmd/edge_controller.runListEdgeRouterPolicies(0xc0003ae690, 0xc00019dd60, 0xc00019dd38)
        /home/travis/gopath/src/github.com/openziti/ziti/ziti/cmd/ziti/cmd/edge_controller/list.go:411 +0x9d
github.com/openziti/ziti/ziti/cmd/ziti/cmd/edge_controller.newListCmdForEntityType.func1(0xc0003b8dc0, 0x1223c18, 0x0, 0x0)
        /home/travis/gopath/src/github.com/openziti/ziti/ziti/cmd/ziti/cmd/edge_controller/list.go:174 +0x6a
github.com/spf13/cobra.(*Command).execute(0xc0003b8dc0, 0x1223c18, 0x0, 0x0, 0xc0003b8dc0, 0x1223c18)
        /home/travis/gopath/pkg/mod/github.com/spf13/[email protected]/command.go:842 +0x29d
github.com/spf13/cobra.(*Command).ExecuteC(0x11a7100, 0x1223c18, 0x0, 0x0)
        /home/travis/gopath/pkg/mod/github.com/spf13/[email protected]/command.go:943 +0x317
github.com/spf13/cobra.(*Command).Execute(...)
        /home/travis/gopath/pkg/mod/github.com/spf13/[email protected]/command.go:883
github.com/openziti/ziti/ziti/cmd/ziti/cmd.Execute()
        /home/travis/gopath/src/github.com/openziti/ziti/ziti/cmd/ziti/cmd/cmd.go:77 +0x51
main.main()
        /home/travis/gopath/src/github.com/openziti/ziti/ziti/cmd/ziti/main.go:24 +0x20

❯ ziti edge list edge-router-policies --help
lists edge-router-policies managed by the Ziti Edge Controller

Usage:
  ziti edge list edge-router-policies <filter>? [flags]

Flags:
  -h, --help                  help for edge-router-policies
  -j, --output-json           Output the full JSON response from the Ziti Edge Controller
      --output-request-json   Output the full JSON request to the Ziti Edge Controller

❯ ziti edge list edge-routers
id: -XV--AVMg    name: lorens edge router attr 8    isOnline: false    role attributes: ["testattr"]
id: 6mDEt04Gg    name: kbRt29a    isOnline: false    role attributes: ["defaultRouters"]
id: 7S-DUA4GR    name: kentest1    isOnline: false    role attributes: ["defaultRouters"]
id: IDndh0VMg    name: kbRt29b    isOnline: false    role attributes: ["defaultRouters"]
id: IcAAU04Gg    name: kentest2    isOnline: true    role attributes: ["defaultRouters"]
id: NyFiaA4GR    name: lorens test router    isOnline: false    role attributes: ["testattr"]
id: rtuOB04Gg    name: kbRt29c    isOnline: false    role attributes: ["defaultRouters"]
results: 1-7 of 7

❯ ziti edge list service-edge-router-policies
id: 8W9iu04GR    name: kentest2_ServiceEdgeRouterPolicy    edge router roles: [#all]    service roles: [@kentest2]
results: 1-1 of 1

❯ ziti version
NAME             VERSION
ziti             v0.15.2
ziti-controller  v0.15.2
ziti-enroller    v0.15.2
ziti-fabric      v0.15.2
ziti-fabric-gw   v0.15.2
ziti-fabric-test v0.15.2
ziti-proxy       0.6.0-16
ziti-prox-c      0.15.2-145
ziti-router      v0.15.2
ziti-tunnel      v0.15.2

The /enroll endpoint can return application/x-pem-file and application/json. If a client specifies an accept header of only application/x-pem-file there is no meaningful way to return API errors. The application/json content type covers are situations and should be the only response type.

Right now certificates for SDK identities and edge routers have a life-span defined during enrollment. There is currently no way to extend this life-span, roll keys, etc.

• support certificate extensions for identities/edge routers
• support key rolling for identities/edge routers
• document how to extend CAs/Intermediates

I enrolled ziti-tunnel 0.15.1 on Linux/x86 with ziti-controller 0.14.9 and the identity metadata sdkInfo and environmentInfo are still empty. The same objects are populated with expected values in another identity that was enrolled with Android Tunneler.

❯ ziti edge list identities --output-json 'name="kenneth_bingham-laptop"' | jq '.data[]|{sdkInfo:.sdkInfo,envInfo:.envInfo}'

{
  "sdkInfo": {
    "branch": "",
    "revision": "",
    "type": "",
    "version": ""
  },
  "envInfo": {
    "arch": "",
    "os": "",
    "osRelease": "",
    "osVersion": ""
  }
}

❯ ziti edge list identities --output-json 'name="kenneth_bingham-mobile"' | jq '.data[]|{sdkInfo:.sdkInfo,envInfo:.envInfo}'

{
  "sdkInfo": {
    "branch": "master",
    "revision": "942aa9d",
    "type": "ziti-sdk-android",
    "version": "0.5.15-211"
  },
  "envInfo": {
    "arch": "arm64-v8a",
    "os": "android-29",
    "osRelease": "10",
    "osVersion": "6392402"
  }
}

Ziti Edge needs journal access information:

• public IP address
• host IP addresses/NICs from SDK
• services accessed
• identity used
• policies that granted access
• etc.

This should rely on: openziti/fabric#106

With ziti-tunnel proxy on Windows and MacOS it is necessary to provide at least one service name on the command line like ziti-tunnel proxy "Gettin' Ziggy Wit It":54321. If my goal is merely to host a service with tunneler then there's no motivation for me to bind a service by name to a local port. It might make more sense to add a mode like ziti-tunnel host that simply doesn't require or allow binding a service, and so will only host services.

If only for consistency w/ ziti-router and ziti CLI,

❯ ./ziti-tunnel enroll

should be equivalent to

❯ ./ziti-tunnel enroll --help

optionally after complaining about the missing required flag --jwt on stderr

❯ ./ziti-tunnel version
v0.15.2

If I create or authorize a service while ziti-tunnel tproxy is running there are no intercepts in IPtables until I restart the tunneler.

the process could be suspended and then terminated without giving you the chance to clear the routes. It is probably rare set of circumstances but it seems that it won't be hard to handle for completeness

Originally posted by @ekoby in openziti/edge#24

add option to read config and config-type (other entities as appropriate) payloads from a file

something like this
ziti edge controller create config-type super-config -f super-config.json

Enable commands to list, update and remove identity level service config overrides.

$ ziti edge controller list identity service-configs --help
lists service-configs related to a identities instanced managed by the Ziti Edge Controller

Usage:
  ziti edge controller list identity service-configs <id or name> [flags]

Flags:
  -h, --help          help for service-configs
  -j, --output-json   Output the full JSON response from the Ziti Edge Controller

$ ziti edge controller update identity-configs --help
for the specified identity, use the given config for the given service

Usage:
  ziti edge controller update identity-configs <identity id or name> <service id or name> <config id or name> [flags]

Flags:
  -h, --help          help for identity-configs
  -j, --output-json   Output the full JSON response from the Ziti Edge Controller
  -r, --remove        Remove the sevice config override

right now any trailing whitespace from the jwt (like LF) will cause a cryptographic issue with the jwt. simply trimming the file should resolve this issue

When using ziti one cannot supply a 'limit'. This means when using ziti against an instance with > 10 objects one is trying to query results are missing. Also since one cannot specify offset - when used against really really large number of objects those also will not be findable.

Description:

ziti-router enrollment returns successful exit status when errors occur

How to replicate issue:

• Create a new edge-router via cli: ./ziti edge create edge-router test-router1
• Get jwt from edge router via cli: ./ziti edge list edge-routers "true limit 100" -j | jq '.data[] | select(.name=="test-router1") | .enrollmentJwt'
• Enroll the router: ./ziti-router enroll config.yml -j registration.jwt
• Enroll again to generate a error output: ./ziti-router enroll config.yml -j registration.jwt
• Check exit status: echo $?

Environment:

Centos 7.8
ziti-router versions tested: 15.2, 15.1, 14.13

trivial issue with output using ERROR level when testing and not updating before pushing.

Allow updating terminator attributes including

• router
• binding
• address
• precedence
• static cost

$ ziti edge controller update terminator --help
updates a service terminator

Usage:
  ziti edge controller update terminator <id> [flags]

Flags:
      --address string      Set the terminator address
      --binding string      Set the terminator binding
  -c, --cost int32          Set the terminator cost
  -h, --help                help for terminator
  -j, --output-json         Output the full JSON response from the Ziti Edge Controller
  -p, --precedence string   Set the terminator precedence ('default', 'required' or 'failed')
      --router string       Set the terminator router