Sometimes we want to retain the double quotes within a Path key. e.g. "\"randomkey\""

Current way of doing the update/replace operation requires user to provide the payload in the json format. While this seems ok for update operation, for replace operation user would have to provide the entire payload, though that is not being replaced.

For example, if user needs to replace an ip address on an interface on NX-OS, and if if the box had 48 interfaces, the user would have to create the payload which would include all the interfaces on the device with their corresponding info, even though the intent is to replace ip address for only interface.

Providing a mechanism to replace based on xpath would help resolve this.
something of the below format
--xpath_update /interfaces/interface[name=eth1/47]/@ip.json

On NX-OS, if we do not provide the entire payload, we would hit the below error. I believe, it could be the same for other platforms too.

File "/ws/mgangaia-sjc/pyats-latest/lib/python3.6/site-packages/cisco_gnmi/nx.py", line 194, in set_json
return self.set(prefix=prefix, updates=updates, replaces=replaces)
File "/ws/mgangaia-sjc/pyats-latest/lib/python3.6/site-packages/cisco_gnmi/client.py", line 223, in set
response = self.service.Set(request, metadata=self.default_call_metadata)
File "/ws/mgangaia-sjc/pyats-latest/lib/python3.6/site-packages/grpc/_channel.py", line 549, in call
return _end_unary_response_blocking(state, call, False, None)
File "/ws/mgangaia-sjc/pyats-latest/lib/python3.6/site-packages/grpc/_channel.py", line 466, in _end_unary_response_blocking
raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.INTERNAL
details = "descr:internal processing error [error:operation-failed msg:Cannot delete object of class:l1PhysIf path:/config/interfaces][error:operation-failed msg:Cannot delete object of class:l1PhysIf Commit Failed path:/config/interfaces]"
debug_error_string = "{"created":"@1595898351.637576743","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"descr:internal processing error [error:operation-failed msg:Cannot delete object of class:l1PhysIf path:/config/interfaces][error:operation-failed msg:Cannot delete object of class:l1PhysIf Commit Failed path:/config/interfaces]","grpc_status":13}"

Thanks,
Mahendra

Trying to reconcile how to use certs with gRPC and IOS XE specifically. I believe the use case presented with IOS XE is more comprehensive than IOS XR and NX-OS thus this is a good exercise if not frustrating.

Following IOS XE 16.12.x gNMI documentation.

>>> client = ClientBuilder('x').set_secure_from_file('../scripts/certs/client.crt', '../scripts/certs/client.key', '../scripts/certs/rootCA.pem').set_ssl_target_override().set_call_authentication('x', 'x').construct()
WARNING:root:Overriding SSL option from certificate could increase MITM susceptibility!
>>> print(client.capabilities())
E1008 14:49:11.385280000 4671555008 ssl_transport_security.cc:690]     Invalid private key.
E1008 14:49:11.385306000 4671555008 ssl_security_connector.cc:112]     Handshaker factory creation failed with TSI_INVALID_ARGUMENT.
E1008 14:49:11.385314000 4671555008 secure_channel_create.cc:132]      Failed to create secure subchannel for secure name 'x:9339'
E1008 14:49:11.385322000 4671555008 secure_channel_create.cc:50]       Failed to create channel args during subchannel creation.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/remcampb/Development/projects/cisco-gnmi-python/src/cisco_gnmi/client.py", line 112, in capabilities
    response = self.service.Capabilities(message)
  File "/Users/remcampb/.local/share/virtualenvs/cisco-gnmi-python-QTeA_bEB/lib/python3.7/site-packages/grpc/_channel.py", line 604, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/Users/remcampb/.local/share/virtualenvs/cisco-gnmi-python-QTeA_bEB/lib/python3.7/site-packages/grpc/_channel.py", line 506, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "Empty update"
	debug_error_string = "{"created":"@1570571351.385334000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3876,"referenced_errors":[{"created":"@1570571351.385331000","description":"Empty update","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":200,"grpc_status":14}]}"
>

Potentially helpful: grpc/grpc#9593

Blocks #9

Currently ignores (unwillingly) XPath filters which could otherwise be expressed in Path and will ultimately break if filters are provided.

Ideally we can check that __version__ variable is being incremented per PR.

This is mostly applicable for IOS XR but applies to all OS implementations, are PathElems expected to have augmentation prefixes?
Brought up per #21

Other deployment methods that would save the user from the gRPC compilation process?

An empty XPath should not result in a Path with any PathElems.

cat get_all.txt
path: <
>
type: CONFIG
encoding: JSON_IETF
I think you can remove elem entirely

Following documentation at https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/prog/configuration/1610/b_1610_programmability_cg/gnmi_protocol.html#id_67108

Using the generated certificates yields:

python test_xe.py
Traceback (most recent call last):
  File "test_xe.py", line 14, in <module>
    print(client.capabilities())
  File "/home/remcampb/development/cisco-gnmi-python/src/cisco_gnmi/client.py", line 93, in capabilities
    response = self.service.Capabilities(message, metadata=self._gen_metadata())
  File "/home/remcampb/.local/share/virtualenvs/cisco-gnmi-python-z1MRTrKn/local/lib/python2.7/site-packages/grpc/_channel.py", line 565, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/remcampb/.local/share/virtualenvs/cisco-gnmi-python-z1MRTrKn/local/lib/python2.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1569363736.047551305","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3818,"referenced_errors":[{"created":"@1569363736.047542320","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":395,"grpc_status":14}]}"
>

Attempting to download and use certificate from device:

python test_xe.py
Traceback (most recent call last):
  File "test_xe.py", line 16, in <module>
    client = XEClient('...', attempt_implicit_secure=True).with_authentication('...', '...')
  File "/home/remcampb/development/cisco-gnmi-python/src/cisco_gnmi/client.py", line 78, in __init__
    self.as_secure(root_from_target=True, target_name_from_root=True)
  File "/home/remcampb/development/cisco-gnmi-python/src/cisco_gnmi/base.py", line 113, in as_secure
    root_certificates = get_cert_from_target(self.target_netloc)
  File "/home/remcampb/development/cisco-gnmi-python/src/cisco_gnmi/base.py", line 80, in get_cert_from_target
    (self.target_netloc.hostname, self.target_netloc.port)
  File "/usr/lib/python2.7/ssl.py", line 1007, in get_server_certificate
    with closing(context.wrap_socket(sock)) as sslsock:
  File "/usr/lib/python2.7/ssl.py", line 353, in wrap_socket
    _context=self)
  File "/usr/lib/python2.7/ssl.py", line 601, in __init__
    self.do_handshake()
  File "/usr/lib/python2.7/ssl.py", line 830, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:590)

OpenSSL too has issues.

openssl s_client -state -connect $IP:$PORT
CONNECTED(00000003)
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:unknown state
depth=1 CN = rootCA
verify error:num=19:self signed certificate in certificate chain
SSL_connect:unknown state
SSL_connect:unknown state
SSL_connect:unknown state
SSL_connect:unknown state
SSL_connect:unknown state
SSL_connect:unknown state
SSL_connect:unknown state
SSL_connect:unknown state
SSL_connect:unknown state
SSL_connect:failed in unknown state
139800832919192:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:

Might be related to SNI?

Let's pin grpcio to a specific version in setup.py. We can monitor it for important fixes or new features and then upgrade after analysis of backward compatibility and stability.

When utilizing the -no_stop option to continually dump, the RPC is terminating with EOF.

python3 subscribe_dump.py -xpath '/access-points/access-point[hostname="albano-house.example.net"]/radios/radio[id="0"]/state/rx-noise-channel-utilization/' '192.168.0.31:8080' -os None
Username: admin
Password:
INFO:root:Connecting to 192.168.0.31:8080 as None ...
WARNING:root:Overriding SSL option from certificate could increase MITM susceptibility!
INFO:root:Subscribing to /access-points/access-point[hostname="albano-house.example.net"]/radios/radio[id="0"]/state/rx-noise-channel-utilization/ ...
INFO:root:{'update': {'timestamp': '1587091884807946240', 'update': [{'path': {'elem': [{'name': 'access-points'}, {'name': 'access-point', 'key': {'hostname': 'albano-house.example.net'}}, {'name': 'radios'}, {'name': 'radio', 'key': {'id': '0'}}, {'name': 'state'}, {'name': 'rx-noise-channel-utilization'}]}, 'val': {'uintVal': '0'}}]}}
ERROR:root:Stopping due to exception!
Traceback (most recent call last):
  File "subscribe_dump.py", line 56, in main
    for message in client.subscribe_xpaths(**sub_args):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grpc/_channel.py", line 364, in __next__
    return self._next()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grpc/_channel.py", line 347, in _next
    raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "EOF"
	debug_error_string = "{"created":"@1587091885.760523000","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1036,"grpc_message":"EOF","grpc_status":2}"
>
INFO:root:Writing to gnmi_sub.json ...

And gnmi_sub.json does contain that first response as in the output above.

This particular Target is an Arista Access Point.

The modules are relatively well documented with numpy style. Should be able to use sphinx with sphinx.ext.napoleon to autogen more readable docs.

Related to #40 but separating from examples given uncertainty.

Tracking in doc-autogen.

pipenv shell
mkdir docs
cd docs
sphinx-quickstart
cd ..
sphinx-apidoc -f -o docs/source src/cisco_gnmi
cd docs
make html
exit

It's currently difficult to determine exactly what messages are being constructed. Should develop message building infra and have clients communicate using it.

We should accept any Iterable (or iterator?).

e.g.

There might be some bugs in how we construct updates for messages from OC origin, need to validate it works with OC-based JSON configs.

There is some construction logic in the ClientBuilder set_ methods (e.g. set_insecure_from_target()) that should be in construct() instead - otherwise the call chain is dependent on previous set_s.

subscribe_xpaths is in every Client subclass, can consolidate into Client.
Raised in #46 per #46 (comment)

Per #40 XPath predicate exprs currently must be strings. Must investigate:

1. Current XML XPath regex result, does it require a string?
2. May YANG XPath predicate values be types other than strings?
3. Easy fix, then scope something more robust like a state machine. pyang just added some xpath code, might be able to disaggregate.

Reports of issues with:

1. Proto enums
2. append vs extend
3. Removing the origin from xpath

Need to validate

Should ensure that this repo points to YDK as well for its usefulness.

Currently the returns for Get (and all other methods for that matter) return the proto response classes. This has several implications - must understand the structure of the protos, must know encoding, must know what to access where... We can probably simplify this for the end user.

client.py

I believe the below code should be request.replace.extend(replaces)

if replaces:
request.replaces.extend(replaces)

Thanks,
Mahendra

I could use an example of how to formulate a Subscription.

Suggestion for documentation:
For the README or WiKi, I'd suggest something like the following:

Working example of OpenConfig Streaming Telemetry on Aceess Point.

from cisco_gnmi import ClientBuilder

builder = ClientBuilder('192.168.0.5:8080')
builder.set_secure_from_target()
builder.set_ssl_target_override('openconfig.mojonetworks.com')
builder.set_call_authentication('admin', 'admin')
client = builder.construct()

paths = [client.parse_xpath_to_gnmi_path('/access-points/access-point[hostname="test-ap.example.net"]/radios/radio[id="0"]/state/rx-noise-channel-utilization/')]
print(client.get(paths))

You may also want to provide a pointer in your documentation that list-key values must be quoted due to your xpath parsing. Even if the list-key is an int (like the radio id in the above example).

If the TSL/authentication parameters are not valid, the channel remains open retrying the authentication. The channel is not accessible from the subclasses of Client and without having access to the "Channel.close" method or having the ability to delete the channel from memory, the only way to completely shutdown is to exit the Python interpreter.

Tests will actually be helpful here.

This would simplify some aspects of workflow. Need for automation projects which are exclusively Python 2.

#63

When we perform a gnmi get request, the default size handles only 4mb response size. We need to handle datasize > 4mb in responses.

options = [('grpc.max_receive_message_length', 1610241024)]

Thanks,
Mahendra

Cannot set mode from CLI currently. Necessary to test ONCE, etc.

IOS XR/IOS XE set_json Path generation only goes one element deep, we should instead make this traverse the structure until multiple keys are identified and use that longest prefix as XPath.

Filing as a bug because in a replace operation this could create a significant impact on device.

NX-OS docstring is missing new methods.
https://github.com/cisco-ie/cisco-gnmi-python/blob/master/src/cisco_gnmi/nx.py#L165

There is Cisco-specific code within this repository, should we rename and publish as cisco_gnmi?

String comparisons are broken. Need to resolve for Python 2 usage.

Currently installs pytest et al in Pipfile.

Determine how to cleanly express the origin and "smart" determination of what the origin should be per XPath.

Cannot perform from CLI, need to check if library correctly handles as well.
https://github.com/openconfig/gnmi/blob/master/proto/gnmi/gnmi.proto#L341-L343

Path parsing e.g. /sys/intf/phys-[eth1/33] does not appropriately handle paths. Need to make Path parsing more modular to account for format differences that are not XPath based.

subscription {
  path {
    origin: "DME"
    elem {
      name: "sys"
    }
    elem {
      name: "intf"
    }
    elem {
      name: "phys-"
    }
    elem {
      key {
        key: "eth1"
        value: ""
      }
    }
  }
  mode: SAMPLE
  sample_interval: 10000000000
}

@remingtonc I need some help testing this out. I haven't had a chance.

The below format is the supported xpath to delete a leaflist entry on nxos.
Please check if we could use the below xpath to perform deletion.

/network-instances/network-instance[name=foo]/config/enabled-address-families[.=IPV4]

Thanks,
Mahendra

https://github.com/cisco-ie/cisco-gnmi-python/blob/master/src/cisco_gnmi/xr.py#L315

The "name" parameter is a hardcoded string called from create_updates so it will always go to this if:

        def check_configs(name, configs):
            if isinstance(name, string_types):
                logging.debug("Handling %s as JSON string.", name)
                try:
                    configs = json.loads(configs)
                except:
                    raise Exception("{name} is invalid JSON!".format(name=name))
                configs = [configs]

The "configs" variable can only be one dict for json.loads unless you put it in a list, but if you do, the configs list is then deposited into another list.