Giter Club home page Giter Club logo

ttp's Introduction

Downloads PyPI versions Documentation status

Template Text Parser

TTP is a Python library for semi-structured text parsing using templates.

Why?

To save ones time on transforming raw text into structured data and beyond.

How?

Regexes, regexes everywhere... but, dynamically formed out of TTP templates with added capabilities to simplify the process of getting desired outcome.

What?

In essence TTP can help to:

  • Prepare, sort and load text data for parsing
  • Parse text using regexes dynamically derived out of templates
  • Process matches on the fly using broad set of built-in or custom functions
  • Combine match results in a structure with arbitrary hierarchy
  • Transform results in desired format to ease consumption by humans or machines
  • Return results to various destinations for storage or further processing

Reference documentation for more information.

TTP Networktocode Slack channel

Collection of TTP Templates

Example - as simple as it can be

Simple interfaces configuration parsing example

Code 
from ttp import ttp
import pprint

data = """
interface Loopback0
 description Router-id-loopback
 ip address 192.168.0.113/24
!
interface Vlan778
 description CPE_Acces_Vlan
 ip address 2002::fd37/124
 ip vrf CPE1
!
"""

template = """
interface {{ interface }}
 ip address {{ ip }}/{{ mask }}
 description {{ description }}
 ip vrf {{ vrf }}
"""

parser = ttp(data, template)
parser.parse()
pprint.pprint(parser.result(), width=100)

# prints:
# [[[{'description': 'Router-id-loopback',
#     'interface': 'Loopback0',
#     'ip': '192.168.0.113',
#     'mask': '24'},
#    {'description': 'CPE_Acces_Vlan',
#     'interface': 'Vlan778',
#     'ip': '2002::fd37',
#     'mask': '124',
#     'vrf': 'CPE1'}]]]

Example - a bit more complicated

For this example lets say we want to parse BGP peerings output, but combine state with configuration data, at the end we want to get pretty looking text table printed to screen.

Code 
template="""
<doc>
This template first parses "show bgp vrf CUST-1 vpnv4 unicast summary" commands
output, forming results for "bgp_state" dictionary, where peer ip is a key.

Following that, "show run | section bgp" output parsed by group "bgp_cfg". That
group uses nested groups to form results structure, including absolute path
"/bgp_peers*" with path formatter to produce a list of peers under "bgp_peers"
path.

For each peer "hostname" and local bgp "local_asn" added using previous matches.
Additionally, group lookup function used to lookup peer state from "bgp_state"
group results, adding found data to peer results.

Finally, "bgp_peers" section of results passed via "tabulate_outputter" to
from and print this table to terminal:

hostname           local_asn    vrf_name    peer_ip    peer_asn    uptime    state    description    afi    rpl_in           rpl_out
-----------------  -----------  ----------  ---------  ----------  --------  -------  -------------  -----  ---------------  ---------------
ucs-core-switch-1  65100        CUST-1      192.0.2.1  65101       00:12:33  300      peer-1         ipv4   RPL-1-IMPORT-v4  RPL-1-EXPORT-V4
ucs-core-switch-1  65100        CUST-1      192.0.2.2  65102       03:55:01  idle     peer-2         ipv4   RPL-2-IMPORT-V6  RPL-2-EXPORT-V6

Run this script with "python filename.py"
</doc>

<vars>
hostname="gethostname"
chain_1 = [
    "set('vrf_name')",
    "lookup('peer_ip', group='bgp_state', update=True)"
]
</vars>

<group name="bgp_state.{{ peer }}" input="bgp_state">
{{ peer }}  4 65101      20      21       43    0    0 {{ uptime }} {{ state }}
</group>

<group name="bgp_cfg" input="bgp_config">
router bgp {{ asn | record(asn) }}
  <group name="vrfs.{{ vrf_name }}" record="vrf_name">
  vrf {{ vrf_name }}
    <group name="/bgp_peers*" chain="chain_1">
    neighbor {{ peer_ip }}
      {{ local_asn | set(asn) }}
      {{ hostname | set(hostname) }}
      remote-as {{ peer_asn }}
      description {{ description }}
      address-family {{ afi }} unicast
        route-map {{ rpl_in }} in
        route-map {{ rpl_out }} out
	</group>
  </group>
</group>

<output
name="tabulate_outputter"
format="tabulate"
path="bgp_peers"
returner="terminal"
headers="hostname, local_asn, vrf_name, peer_ip, peer_asn, uptime, state, description, afi, rpl_in, rpl_out"
/>
"""

data_bgp_state = """
ucs-core-switch-1#show bgp vrf CUST-1 vpnv4 unicast summary
Neighbor   V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
192.0.2.1  4 65101      32      54       42    0    0 00:12:33       300
192.0.2.2  4 65101      11      45       99    0    0 03:55:01       idle
"""

data_bgp_config = """
ucs-core-switch-1#show run | section bgp
router bgp 65100
  vrf CUST-1
    neighbor 192.0.2.1
      remote-as 65101
      description peer-1
      address-family ipv4 unicast
        route-map RPL-1-IMPORT-v4 in
        route-map RPL-1-EXPORT-V4 out
    neighbor 192.0.2.2
      remote-as 65102
      description peer-2
      address-family ipv4 unicast
        route-map RPL-2-IMPORT-V6 in
        route-map RPL-2-EXPORT-V6 out
"""

from ttp import ttp

parser = ttp()
parser.add_template(template)
parser.add_input(data=data_bgp_state, input_name="bgp_state")
parser.add_input(data=data_bgp_config, input_name="bgp_config")
parser.parse()

Contributions

Feel free to submit an issue, report a bug or ask a question, feature requests are welcomed.

Or sponsor me or buy me a coffee 😃 ☕.

Want To Hire

Reach out at [email protected] to discuss.

Additional resources

List of additional resources:

ttp's People

Contributors

akira6592 avatar danielolson13 avatar dmulyalin avatar jeannly avatar sunpoet avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

tobiasjohanssonsdnit mirzawaqasahmed danielolson13 tbotnz noifp zhutony lkx007 tongorz gwoodwa1 jl-massey th3architect nanisyako trasmontinho paramountk 0xack13 narimantos stephan1373 yuke1922 cuchulainx yijxiang cam-mu cptguo scottpeterman showipintbri rebiocoder jeannly danigm capuanob melodylail mayhemheroes frhtcck blacknon akira6592 zwnr

ttp's Issues

Parsing XML format

Hello,

First and foremost, I would like to say that this tool has been very useful for me due to it's easy implementation and versatility. I love it and it has solved a lot of my problems. Although I've been able to parse semi-structured text based files, recently I was required to parse a certain type of XML file.

When I tried designing a template for it, I couldn't get a regex to match the tags of XML. Can you please help me resolve this issue or point me to a documentation which could help me, I would be so grateful.

Thank you.

result duplicating across groups

Hi,
Great project, first of all.

I am trying to parse the below data:

data = """
/c/slb/real 3
	ena
	ipver v4
	rip 1.1.1.229
	name "vgqa6-01"
/c/slb/group 13
	ipver v4
	backup g14
	add 23
	name "appvip1-tcp"
"""

with the following template:

real_servers_ttp_template = """
<group name="nodes" default="">
/c/slb/real {{ node_seq | isdigit }}
	{{ config_state }}
	ipver {{ ipver }}
	rip {{ node_ip | is_ip }}
	inter {{ interval }}
	retry {{ retry }}
	name {{ node_name | replace('"','') }}
<group name="node_port*">
	addport {{ node_port | isdigit }}
</group>
</group>
"""

groups_ttp_template = """
<group name="pools" default="">
/c/slb/group {{ group_seq | isdigit }}
	ipver {{ ipver }}
<group name="backup_obj">
	backup {{ type | re("\w") }}{{ seq | re("\d+") }}
</group>
	metric {{ lb_method }}
	health {{ health_monitor }}
<group name="node_list*">
	add {{ node_seq | isdigit }}
</group>
	name {{ group_name | replace('"','') }}
</group>
"""

which is giving me the following output:

[
    {
        "nodes": {
            "config_state": "ena",
            "interval": "None",
            "ipver": "v4",
            "node_ip": "1.1.1.229",
            "node_name": "appvip1-tcp",
            "node_seq": "3",
            "retry": "None"
        }
    }
]
[
    {
        "pools": {
            "backup_obj": {
                "seq": "14",
                "type": "g"
            },
            "group_name": "appvip1-tcp",
            "group_seq": "13",
            "health_monitor": "None",
            "ipver": "v4",
            "lb_method": "None",
            "node_list": [
                {
                    "node_seq": "23"
                }
            ]
        }
    }
]

as you can see the node_name and group_name have the same value. I have tried multiple ways to get away with it...

  1. by keeping single template
  2. by multiple templates (as in example)
  3. by having 1 template but multiple data.
    nothing works.

but when I keep 2 templates and 2 data inputs it works.

Is this desired behavior? Can we not have single data input and single template with multiple groups?
Or at minimum, single data input.

thanks in advance.

Inconsistent return if match variable macro is executed

Running a match variable macro results in the result being a list of tuples of list of dictionaries, where as not running a macro results in the return being a list of dictionaries.

Per documentation located here the return should be a list of dictionaries.

Python 3.7.4 (default, Sep  7 2019, 18:27:02) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> 
>>> 
>>> from ttp import ttp
>>> 
>>> data = '''
...    remote         refid           st t when poll reach   delay   offset  jitter
... ===============================================================================
... *172.27.1.4       .GPS.            1 -  108 1024  377    0.977   -0.309   0.088
...  2600:6cec:1c0:2::8
...                   .INIT.          16 u    - 1024    0    0.000    0.000 4000.00
... '''
>>> 
>>> template_a = '''
... <group name="associations">
... {{ active | re("[ *]") | macro("check_active") }}{{ ip | IP | IPV6 }} {{ method }} {{ stratum | to_int }} {{ peer_type }} {{ when }} {{ poll }} {{ reach }} {{ delay }} {{ offset }} {{ jitter }} 
... </group>
... <group name="associations">
... {{ active | re("[ *]") | macro("check_active") }}{{ ip | IP | IPV6 }}
...                   {{ method }} {{ stratum | to_int}} {{ peer_type }} {{ when }} {{ poll }} {{ reach }} {{ delay }} {{ offset }} {{ jitter }} 
... </group>
... 
... <macro>
... def check_active(data):
...     if data == '*': 
...         return data, { 'type': 'our_master' }
...     else:
...         return data, { 'type': 'candidate' }
...     return data
... </macro>
... 
... <output macro="check_active"/>
... '''
>>> 
>>> template_b = '''
... <group name="associations">
... {{ active | re("[ *]") | macro("check_active") }}{{ ip | IP | IPV6 }} {{ method }} {{ stratum | to_int }} {{ peer_type }} {{ when }} {{ poll }} {{ reach }} {{ delay }} {{ offset }} {{ jitter }} 
... </group>
... <group name="associations">
... {{ active | re("[ *]") | macro("check_active") }}{{ ip | IP | IPV6 }}
...                   {{ method }} {{ stratum | to_int}} {{ peer_type }} {{ when }} {{ poll }} {{ reach }} {{ delay }} {{ offset }} {{ jitter }} 
... </group>
... 
... <macro>
... def check_active(data):
...     if data == '*': 
...         return data, { 'type': 'our_master' }
...     else:
...         return data, { 'type': 'candidate' }
...     return data
... </macro>
... '''
>>> 
>>> 
>>> parser_a = ttp(data=data, template=template_a)
>>> parser_b = ttp(data=data, template=template_b)
>>> parser_a.parse()
>>> parser_b.parse()
>>> 
>>> parser_a.result()
[([{'associations': [{'active': '*', 'type': 'our_master', 'ip': '172.27.1.4', 'method': '.GPS.', 'stratum': 1, 'peer_type': '-', 'when': '108', 'poll': '1024', 'reach': '377', 'delay': '0.977', 'offset': '-0.309', 'jitter': '0.088'}, {'method': '.INIT.', 'stratum': 16, 'peer_type': 'u', 'when': '-', 'poll': '1024', 'reach': '0', 'delay': '0.000', 'offset': '0.000', 'jitter': '4000.00', 'active': ' ', 'type': 'candidate', 'ip': '2600:6cec:1c0:2::8'}]}], {'type': 'candidate'})]
>>> 
>>> parser_b.result()
[[{'associations': [{'active': '*', 'type': 'our_master', 'ip': '172.27.1.4', 'method': '.GPS.', 'stratum': 1, 'peer_type': '-', 'when': '108', 'poll': '1024', 'reach': '377', 'delay': '0.977', 'offset': '-0.309', 'jitter': '0.088'}, {'method': '.INIT.', 'stratum': 16, 'peer_type': 'u', 'when': '-', 'poll': '1024', 'reach': '0', 'delay': '0.000', 'offset': '0.000', 'jitter': '4000.00', 'active': ' ', 'type': 'candidate', 'ip': '2600:6cec:1c0:2::8'}]}]]

Running the example in documentation works as expected except the documented return is a list of dictionaries where as the actual result is a list of list of dictionaries:

Python 3.7.4 (default, Sep  7 2019, 18:27:02) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> 
>>> from ttp import ttp
>>> 
>>> template = '''
... <input load="text">
... interface Vlan123
...  description Desks vlan
...  ip address 192.168.123.1 255.255.255.0
... !
... interface GigabitEthernet1/1
...  description to core-1
... !
... interface Vlan222
...  description Phones vlan
...  ip address 192.168.222.1 255.255.255.0
... !
... interface Loopback0
...  description Routing ID loopback
... !
... </input>
... 
... <macro>
... def check_if_svi(data):
...     if "Vlan" in data:
...         return data, {"is_svi": True}
...     else:
...        return data, {"is_svi": False}
... 
... def check_if_loop(data):
...     if "Loopback" in data:
...         return data, {"is_loop": True}
...     else:
...        return data, {"is_loop": False}
... </macro>
... 
... <macro>
... def description_mod(data):
...     # To revert words order in descripotion
...     words_list = data.split(" ")
...     words_list_reversed = list(reversed(words_list))
...     words_reversed = " ".join(words_list_reversed)
...     return words_reversed
... </macro>
... 
... <group name="interfaces_macro">
... interface {{ interface | macro("check_if_svi") | macro("check_if_loop") }}
...  description {{ description | ORPHRASE | macro("description_mod")}}
...  ip address {{ ip }} {{ mask }}
... </group>
... '''
>>> 
>>> parser = ttp(template=template)
>>> parser.parse()
>>> parser.result()
[[{'interfaces_macro': [{'ip': '192.168.123.1', 'mask': '255.255.255.0', 'description': 'vlan Desks', 'interface': 'Vlan123', 'is_svi': True, 'is_loop': False}, {'description': 'core-1 to', 'interface': 'GigabitEthernet1/1', 'is_svi': False, 'is_loop': False}, {'ip': '192.168.222.1', 'mask': '255.255.255.0', 'description': 'vlan Phones', 'interface': 'Vlan222', 'is_svi': True, 'is_loop': False}, {'description': 'loopback ID Routing', 'interface': 'Loopback0', 'is_svi': False, 'is_loop': True}]}]]
>>>

Parsing multi line strings

Hi !

I'm trying to parse multi-line paragraph and I havent been able to figure it out. I was wondering If you could help confirm if ttp can handle a template such as this:

An example text:

ECON*3400 The Economics of Personnel Management U (3-0) [0.50]
In this course, we examine the economics of personnel management in organizations.
Using mainstream microeconomic and behavioural economic theory, we will consider
such issues as recruitment, promotion, financial and non-financial incentives,
compensation, job performance, performance evaluation, and investment in personnel.
The interplay between theoretical models and empirical evidence will be emphasized in
considering different approaches to the management of personnel.
Prerequisite(s): ECON*2310 or ECON*2200
Department(s): Department of Economics and Finance

Currently my template looks like:

course_template = """
{{course}}*{{code}} {{ name | PHRASE }} {{ semester }} ({{lecture_lab_time}}) [{{weight}}]\n
{{ description | ROW }}
Prerequisite(s): {{prereqs | ORPHRASE}}\n
Department(s): {{department | PHRASE}}\n
"""
```]

However description is a list of rows, any guidance would be greatly appreciated!

Parse line with optional keyword

HI!

Please tell me.
I have intput text from device

.id=*c;export-route-targets=65001:48;65001:0;import-route-targets=65001:48;interfaces=lo-ext;vlan56;route-distinguisher=65001:48;routing-mark=VRF_EXT
.id=*10;comment=;export-route-targets=65001:80;import-route-targets=65001:80;65001:0;interfaces=lo-private;route-distinguisher=65001:80;routing-mark=VRF_PRIVATE

How i can write template which parse this string and return
id
comment
export-route-targets
import-route-targets
interfaces
route-distinguisher
routing-mark

my best solution only this

<macro>
def split_kv(data):
	result = dict()
	for kv in data:
		if '=' in kv:
			k,v = kv.split('=')
			result[k] = v
	return result
</macro>
<!-- comment -->
{{<group name="vrfs">
{{ vrf | _line_ | split(';') | macro('split_kv') }}
</group>

I'm sorry, but I don’t know where else you can ask for help with this.
Thank you for your work

Is it possible to save to existing Excel file?

Hi,

Is it possible to add tabs into an existing Excel file? I can't seem to see how to do this.
So one parser would create tab X, while another would create tab Y.

Is it also possible to use variable as part of tab name?

Many thanks
K

Template not working using netmiko but ok through ttp online

Hello,

I'm missing something that I haven't been able to catch up.

Here is the thing : I'm collecting some outputs from Fortinet devices through CLI with netmiko. When I've typed the command "get system interface physical", I haven't the output expected. Before Going through my testing with Netmiko, I use to test it online and my TTP template was working. So Basically, I'm lost with it.
I'm seeing this warning : WARNING:ttp.ttp:group.get_regexes: variable not found in line: '== onboard'
But I can't explain that as it is present in the output.

Thanks for your help.

Regards,

Sam

PPRINT output

('== [onboard]\n'
'\t==[mgmt1]\n'
'\t\tmode: static\n'
'\t\tip: 1.1.1.10 255.255.255.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: down\n'
'\t\tspeed: n/a\n'
'\t==[npu0_vlink0]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: up\n'
'\t\tspeed: n/a (Duplex: n/a)\n'
'\t==[npu0_vlink1]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: up\n'
'\t\tspeed: n/a (Duplex: n/a)\n'
'\t==[npu1_vlink0]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: up\n'
'\t\tspeed: n/a (Duplex: n/a)\n'
'\t==[npu1_vlink1]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: up\n'
'\t\tspeed: n/a (Duplex: n/a)\n'
'\t==[npu2_vlink0]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: up\n'
'\t\tspeed: n/a (Duplex: n/a)\n'
'\t==[npu2_vlink1]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: up\n'
'\t\tspeed: n/a (Duplex: n/a)\n'
'\t==[npu3_vlink0]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: up\n'
'\t\tspeed: n/a (Duplex: n/a)\n'
'\t==[npu3_vlink1]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: up\n'
'\t\tspeed: n/a (Duplex: n/a)\n'
'\t==[npu4_vlink0]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: up\n'
'\t\tspeed: n/a (Duplex: n/a)\n'
'\t==[npu4_vlink1]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: up\n'
'\t\tspeed: n/a (Duplex: n/a)\n'
'\t==[npu5_vlink0]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: up\n'
'\t\tspeed: n/a (Duplex: n/a)\n'
'\t==[npu5_vlink1]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: up\n'
'\t\tspeed: n/a (Duplex: n/a)\n'
'\t==[npu6_vlink0]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: up\n'
'\t\tspeed: n/a (Duplex: n/a)\n'
'\t==[npu6_vlink1]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: up\n'
'\t\tspeed: n/a (Duplex: n/a)\n'
'\t==[npu7_vlink0]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: up\n'
'\t\tspeed: n/a (Duplex: n/a)\n'
'\t==[npu7_vlink1]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: up\n'
'\t\tspeed: n/a (Duplex: n/a)\n'
'\t==[port1]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: down\n'
'\t\tspeed: n/a\n'
'\t==[port2]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: down\n'
'\t\tspeed: n/a\n'
'\t==[port3]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: down\n'
'\t\tspeed: n/a\n'
'\t==[port4]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: down\n'
'\t\tspeed: n/a\n'
'\t==[port5]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: up\n'
'\t\tspeed: 10000Mbps (Duplex: full)\n'
'\t==[port6]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: up\n'
'\t\tspeed: 10000Mbps (Duplex: full)\n'
'\t==[port7]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: down\n'
'\t\tspeed: n/a\n'
'\t==[port8]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: down\n'
'\t\tspeed: n/a\n'
'\t==[port9]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: up\n'
'\t\tspeed: 10000Mbps (Duplex: full)\n'
'\t==[port10]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: up\n'
'\t\tspeed: 10000Mbps (Duplex: full)\n'
'\t==[port11]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: up\n'
'\t\tspeed: 10000Mbps (Duplex: full)\n'
'\t==[port12]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: up\n'
'\t\tspeed: 10000Mbps (Duplex: full)\n'
'\t==[port13]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: up\n'
'\t\tspeed: 10000Mbps (Duplex: full)\n'
'\t==[port14]\n'
'\t\tmode: static\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: up\n'
'\t\tspeed: 10000Mbps (Duplex: full)\n'
'\t==[modem]\n'
'\t\tmode: pppoe\n'
'\t\tip: 0.0.0.0 0.0.0.0\n'
'\t\tipv6: ::/0\n'
'\t\tstatus: down\n'
'\t\tspeed: n/a\n')

TTP template used

""" == onboard ==[{{interface|re("mgmt\d+")|re("port\d+")}}] mode: {{mode}} ip: {{ipv4|IP}} {{mask_ipv4|IP}} ipv6: {{ipv6}}/{{mask_ipv6}} status: {{status}} speed: {{speed}} (Duplex: {{duplex}}) """

Expected output

[
[
{
"interfaces": [
{
"interface": "mgmt1",
"ipv4": "1.1.1.10",
"ipv6": "::",
"mask_ipv4": "255.255.255.0",
"mask_ipv6": "0",
"mode": "static",
"status": "down"
},
{
"duplex": "full",
"interface": "mgmt2",
"ipv4": "10.124.71.82",
"ipv6": "::",
"mask_ipv4": "255.255.255.0",
"mask_ipv6": "0",
"mode": "static",
"speed": "1000Mbps",
"status": "up"
},
{
"interface": "port1",
"ipv4": "0.0.0.0",
"ipv6": "::",
"mask_ipv4": "0.0.0.0",
"mask_ipv6": "0",
"mode": "static",
"status": "down"
},
{
"interface": "port2",
"ipv4": "0.0.0.0",
"ipv6": "::",
"mask_ipv4": "0.0.0.0",
"mask_ipv6": "0",
"mode": "static",
"status": "down"
},
{
"interface": "port3",
"ipv4": "0.0.0.0",
"ipv6": "::",
"mask_ipv4": "0.0.0.0",
"mask_ipv6": "0",
"mode": "static",
"status": "down"
},
{
"interface": "port4",
"ipv4": "0.0.0.0",
"ipv6": "::",
"mask_ipv4": "0.0.0.0",
"mask_ipv6": "0",
"mode": "static",
"status": "down"
},
{
"duplex": "full",
"interface": "port5",
"ipv4": "0.0.0.0",
"ipv6": "::",
"mask_ipv4": "0.0.0.0",
"mask_ipv6": "0",
"mode": "static",
"speed": "10000Mbps",
"status": "up"
},
{
"duplex": "full",
"interface": "port6",
"ipv4": "0.0.0.0",
"ipv6": "::",
"mask_ipv4": "0.0.0.0",
"mask_ipv6": "0",
"mode": "static",
"speed": "10000Mbps",
"status": "up"
},
{
"interface": "port7",
"ipv4": "0.0.0.0",
"ipv6": "::",
"mask_ipv4": "0.0.0.0",
"mask_ipv6": "0",
"mode": "static",
"status": "down"
},
{
"interface": "port8",
"ipv4": "0.0.0.0",
"ipv6": "::",
"mask_ipv4": "0.0.0.0",
"mask_ipv6": "0",
"mode": "static",
"status": "down"
},
{
"duplex": "full",
"interface": "port9",
"ipv4": "0.0.0.0",
"ipv6": "::",
"mask_ipv4": "0.0.0.0",
"mask_ipv6": "0",
"mode": "static",
"speed": "10000Mbps",
"status": "up"
},
{
"duplex": "full",
"interface": "port10",
"ipv4": "0.0.0.0",
"ipv6": "::",
"mask_ipv4": "0.0.0.0",
"mask_ipv6": "0",
"mode": "static",
"speed": "10000Mbps",
"status": "up"
},
{
"duplex": "full",
"interface": "port11",
"ipv4": "0.0.0.0",
"ipv6": "::",
"mask_ipv4": "0.0.0.0",
"mask_ipv6": "0",
"mode": "static",
"speed": "10000Mbps",
"status": "up"
},
{
"duplex": "full",
"interface": "port12",
"ipv4": "0.0.0.0",
"ipv6": "::",
"mask_ipv4": "0.0.0.0",
"mask_ipv6": "0",
"mode": "static",
"speed": "10000Mbps",
"status": "up"
},
{
"duplex": "full",
"interface": "port13",
"ipv4": "0.0.0.0",
"ipv6": "::",
"mask_ipv4": "0.0.0.0",
"mask_ipv6": "0",
"mode": "static",
"speed": "10000Mbps",
"status": "up"
},
{
"duplex": "full",
"interface": "port14",
"ipv4": "0.0.0.0",
"ipv6": "::",
"mask_ipv4": "0.0.0.0",
"mask_ipv6": "0",
"mode": "static",
"speed": "10000Mbps",
"status": "up"
}
]
}
]
]

Result Obtained

[[{}]]

Difficulty Parsing multi-line Table correctly

Hi,
I am working on the script which I am having difficulty in getting TTP to parse the correct results. It seems to work okay up to the 2nd line in the table. From there it seems overwrite the results so the line (below) gets no matches in the data. The only way to get matches is to remove the lines below and then it starts to pick up the wrong information from the line below.

{{ ignore(r"\s+") }} {{ input_byte_cnt | DIGIT }} {{ ignore("\s+") }} {{ output_byte_cnt | DIGIT }}

I tried to terminate each line in a separate group but this did not work and I am stuck for ideas on how to fix it.

I did try to use nested groups to see if that would make a difference so I am not sure if I need them or not here.

import pprint
from ttp import ttp



data_to_parse="""
R1#sh ip nbar protocol-discovery protocol 

 GigabitEthernet1 

 Last clearing of "show ip nbar protocol-discovery" counters 00:13:45


                              Input                    Output                  
                              -----                    ------                  
 Protocol                     Packet Count             Packet Count            
                              Byte Count               Byte Count              
                              5min Bit Rate (bps)      5min Bit Rate (bps)     
                              5min Max Bit Rate (bps)  5min Max Bit Rate (bps) 
 ---------------------------- ------------------------ ------------------------
 ssh                          191                      134                     
                              24805                    22072                   
                              2000                     1000                    
                              1999                     1001                    
 unknown                      172                      503                     
                              39713                    31378                   
                              0                        0                       
                              3000                     0                       
 ping                         144                      144                     
                              14592                    14592                   
                              0                        0                       
                              1000                     1000                    
 dns                          107                      0                       
                              21149                    0                       
                              0                        0                       
                              2000                     0                       
 vrrp                         0                        738                     
                              0                        39852                   
                              0                        0                       
                              0                        0                       
 ldp                          174                      175                     
                              13224                    13300                   
                              0                        0                       
                              0                        0                       
 ospf                         86                       87                      
                              9460                     9570                    
                              0                        0                       
                              0                        0                       
 Total                        874                      1781                    
                              122943                   130764                  
                              2000                     1000                    
                              8000                     2000 
"""

show_nbar = """
<template name="nbar" results="per_template">
<group name="{{ interface }}.{{ protocol }}" method="table">
{{ ignore(r"\s+") }} {{ interface | re('Gig | Ten*.+') }} 
 {{ protocol | WORD }} {{ input_packet_count | DIGIT }} {{ output_packet_count | DIGIT }}
  <group name="_" method="table">
   {{ ignore(r"\s+") }} {{ input_byte_count | DIGIT }} {{ ignore("\s+") }} {{ output_byte_count | DIGIT }} 
    <group name="5_min_stats" method="table">
      {{ ignore(r"\s+") }} {{ Five_Min_input_bitrate | DIGIT }} {{ Five_Min_output_bitrate | DIGIT }}
      {{ ignore(r"\s+") }} {{ Five_Min_input_max_bitrate | DIGIT }} {{ ignore("\s+") }} {{ Five_Min_output_max_bitrate | DIGIT }}
    </group>
  </group>
</group>
</template>  
"""
parser = ttp(template=show_nbar)
parser.add_input(data_to_parse, template_name="nbar")
parser.parse()
res = parser.result(structure="dictionary")
pprint.pprint(res, width=100)

Group in group

Hello,

I'm trying to parse this section:

    vrf2 {
        forwarding-options {
            dhcp-relay {
                server-group {
                    IN_MEDIA_SIGNALING {
                        10.154.6.147;
                    }
                    DHCP-NGN-SIG {
                        10.154.6.147;
                    }
                }
                group group2 {
                    active-server-group IN_MEDIA_SIGNALING;
                    overrides {
                        trust-option-82;
                    }
                }
                group NGN-SIG {
                    active-server-group DHCP-NGN-SIG;
                    overrides {
                        trust-option-82;
                    }
                }
            }
        }
    }

So I made this template

<group name="vrfs*">
    {{ name | _start_ }} {
        <group name="forwarding_options">
        forwarding-options { {{ _start_ }}
            <group name="dhcp_relay">
            dhcp-relay { {{ _start_ }}
                <group name="server_group">
                server-group { {{ _start_ }}
                    <group name="dhcp*">
                    {{ server_group_name | _start_ }} {
                        <group name="helper_addresses*">
                        {{helper_address}};
                        </group>
                    } {{ _end_ }}
                    </group>
                } {{ _end_ }}
                </group>
                <group name="groups*">
                group {{group_name | _start_ }} {
                    active-server-group {{server_group_name}};
                } {{ _end_ }}
                </group>
            } {{ _end_ }}
            </group>
        } {{ _end_ }}
        </group>
    } {{ _end_ }}
    </group>

But then I end up with these values inside dhcp

              'dhcp': [
                {
                  'helper_addresses': [
                    {
                      'helper_address': '10.154.6.147'
                    }
                  ],
                  'server_group_name': 'IN_MEDIA_SIGNALING'
                },
                {
                  'helper_addresses': [
                    {
                      'helper_address': '10.154.6.147'
                    }
                  ],
                  'server_group_name': 'DHCP-NGN-SIG'
                },
                {
                  'helper_addresses': [
                    {
                      'helper_address': 'trust-option-82'
                    }
                  ],
                  'server_group_name': 'overrides'
                },
                {
                  'helper_addresses': [
                    {
                      'helper_address': 'trust-option-82'
                    }
                  ],
                  'server_group_name': 'overrides'
                }
              ]

For some reason the these lines got recognised as a server group despite these 2 sections are in a different parent group

                    overrides {
                        trust-option-82;
                    }


If anything isn't clear let me know to clearify more

TTP GitHub repo recreated

TTP GitHub repository recreated due to failure to merge local changes with GitHub remote, most likely as a result of my PC local file system corruption.

Apologies for any inconvenience caused.

Ability to make a flat structure of the output

Hello!
I just started to use this library and it looks very handy
I am trying to make the output structure as flat as possible, for example I have an input

CAM Utilization for ASIC# 0                      Max            Used
                                             Masks/Values    Masks/values

 Unicast mac addresses:                        400/3200         50/300   
 IPv4 IGMP groups + multicast routes:          152/1216          7/27    
 IPv4 unicast directly-connected routes:       400/3200         50/300   
 IPv4 unicast indirectly-connected routes:    1040/8320         58/377   
 IPv4 policy based routing aces:               384/512           1/2     
 IPv4 qos aces:                                768/768         390/390   
 IPv4 security aces:                          1024/1024         39/39    

Note: Allocation of TCAM entries per feature uses
a complex algorithm. The above information is meant
to provide an abstract view of the current TCAM utilization


CAM Utilization for ASIC# 1                      Max            Used
                                             Masks/Values    Masks/values

 Unicast mac addresses:                        400/3200         50/300   
 IPv4 IGMP groups + multicast routes:          152/1216          7/27    
 IPv4 unicast directly-connected routes:       400/3200         50/300   
 IPv4 unicast indirectly-connected routes:    1040/8320         58/377   
 IPv4 policy based routing aces:               384/512           1/2     
 IPv4 qos aces:                                768/768         390/390   
 IPv4 security aces:                          1024/1024         39/39    

Note: Allocation of TCAM entries per feature uses
a complex algorithm. The above information is meant
to provide an abstract view of the current TCAM utilization

but the best I could get out of this doesn't quite suit my needs

template:
<group method="table" del="max_name, used_name, max_mask, used_mask">
CAM Utilization for {{ asic | replaceall('#') | ORPHRASE }} {{ max_name }} {{ used_name }}
 {{ cam | ORPHRASE }}: {{ max_mask }}/{{ max }} {{ used_mask }}/{{ used }}
</group>

output:
[
	[
		[
			{
				"asic": "ASIC 0"
			},
			{
				"cam": "Unicast mac addresses",
				"max": "3200",
				"used": "300"
			},
			{
				"cam": "IPv4 IGMP groups + multicast routes",
				"max": "1216",
				"used": "27"
			},
			{
				"cam": "IPv4 unicast directly-connected routes",
				"max": "3200",
				"used": "300"
			},
			{
				"cam": "IPv4 unicast indirectly-connected routes",
				"max": "8320",
				"used": "377"
			},
			{
				"cam": "IPv4 policy based routing aces",
				"max": "512",
				"used": "2"
			},
			{
				"cam": "IPv4 qos aces",
				"max": "768",
				"used": "390"
			},
			{
				"cam": "IPv4 security aces",
				"max": "1024",
				"used": "39"
			},
			{
				"asic": "ASIC 1"
			},
			{
				"cam": "Unicast mac addresses",
				"max": "3200",
				"used": "300"
			},
			{
				"cam": "IPv4 IGMP groups + multicast routes",
				"max": "1216",
				"used": "27"
			},
			{
				"cam": "IPv4 unicast directly-connected routes",
				"max": "3200",
				"used": "300"
			},
			{
				"cam": "IPv4 unicast indirectly-connected routes",
				"max": "8320",
				"used": "377"
			},
			{
				"cam": "IPv4 policy based routing aces",
				"max": "512",
				"used": "2"
			},
			{
				"cam": "IPv4 qos aces",
				"max": "768",
				"used": "390"
			},
			{
				"cam": "IPv4 security aces",
				"max": "1024",
				"used": "39"
			}
		]
	]
]
or
template:
<template results="per_template">
<group method="table" del="max_name, used_name">
CAM Utilization for {{ asic | replaceall('#') | ORPHRASE }} {{ max_name }} {{ used_name }}
 <group name="_" del="max_mask, used_mask">
 {{ cam | ORPHRASE }}: {{ max_mask }}/{{ max }} {{ used_mask }}/{{ used }}
 </group>
</group>
</template>

output:
[
	[
		{
			"asic": "ASIC 0",
			"cam": "IPv4 security aces",
			"max": "1024",
			"used": "39"
		},
		{
			"asic": "ASIC 1",
			"cam": "IPv4 security aces",
			"max": "1024",
			"used": "39"
		}
	]
]

maybe there is a way to "copy" the variable inside each object or you can advise me something.
I would like to get something like this

[
	[
		[
			{
				"asic": "ASIC 0"
				"cam": "Unicast mac addresses",
				"max": "3200",
				"used": "300"
			},
			{
				"asic": "ASIC 0"
				"cam": "IPv4 IGMP groups + multicast routes",
				"max": "1216",
				"used": "27"
			},
...
			{
				"asic": "ASIC 1"
				"cam": "IPv4 qos aces",
				"max": "768",
				"used": "390"
			},
			{
				"asic": "ASIC 1"
				"cam": "IPv4 security aces",
				"max": "1024",
				"used": "39"
			}
		]
	]
]

How to parse a list of values

I want to parse this input

vrf xyz
 address-family ipv4 unicast
  import route-target
   65000:3507
   65000:3511
   65000:5453
   65000:5535
  !
  export route-target
   65000:5453
   65000:5535
  !
 !
!

And I want to get the output like

{
        "vrfs": [
            {
                "export_rts": [
                    {"export_rt": "65000:5453"},
                    {"export_rt": "65000:5535"}
                ],
                "import_rts": [
                    {"import_rt": "65000:3507"},
                    {"import_rt": "65000:3511"},
                    {"import_rt": "65000:5453"},
                    {"import_rt": "65000:5535"}
                ],
                "name": "xyz"
            },
}

I tried this template

<group name="vrfs">
vrf {{name}}
  import route-target
  <group name="import_rts">
   {{import_rt}}
  </group>
  !
  export route-target
  <group name="export_rts">
   {{export_rt}}
  </group>
  !
 !
!
</group>

But i got this result

{
        "vrfs": [
            {
                "export_rts": [
                    {"export_rt": "65000:5453"},
                    {"export_rt": "65000:5535"},
                    {"export_rt": "65000:3507"},
                    {"export_rt": "65000:3511"},
                    {"export_rt": "65000:5453"},
                    {"export_rt": "65000:5535"}
                ]
                "name": "xyz"
            },
}

What can I do for this case?

include other templates

Is there a way to include external templates in a template in order to avoid duplication?

Hostname - Nokia-SR-OS

Hi,

Really enjoying working with TTP.
I am parsing Nokia output, and trying to use the getter for hostname and it's not working. I see it;'s not in supported list.

Would you consider adding to this list? The Nokia is simply as per below. The : is the separator before the hostname.

*A:hostname#

I added a space to my nodal config to see if it would match like a Cisco one, but still didn't seem to work.

Thanks for any help

Kieran

Return list as dict values all the time

I've run into a scenario where sometimes I get back a dict, and sometimes I get back a list of dicts. The best example I have is this:

Example template

spanning-tree vlan {{ stp_vlans }} priority {{ stp_priority }}

Returns list of dicts as a value

spanning-tree vlan 70,100-130,148,175,230,300-301,315,366,550 priority 24576
spanning-tree vlan 915,920 priority 24576

--------------------
Example of return:

'spanning_tree': [{'stp_priority': '24576','stp_vlans': '70,100-130,148,175,230,300-301,315,366,550'},
                  {'stp_priority': '24576', 'stp_vlans': '915,920'}]

Returns only a dict as a value

spanning-tree vlan 70,100,175,300,315,366,550 priority 12288

--------------------
Example of return:

'spanning_tree': {'stp_priority': '12288', 'stp_vlans': '70,100,175,300,315,366,550'}

It would be more consistent (and easier for me) if the second one returned a list of dicts like this:

'spanning_tree': [{'stp_priority': '12288', 'stp_vlans': '70,100,175,300,315,366,550'}]

Thanks for sharing this library! It's awesome.

how to put hostname in the filename output template?

hi, i have one cuestion its possible put the hostname in filename (excell template o maybe other template)?

for example:

{{interface}} {{ ip_address }} YES {{method}} {{ status}} {{protocol | line}}

<output
format="excel"
returner="file"
filename="%hostname_%Y-%m-%d_%H-%M-%S" #hostname for the device and date.
url="./Excel_Outputs/"
load="yaml"

table:

  • headers: interface, ip_address, method, status, protocol
    path: table
    key: interface
    tab_name: interfaces

thanks....

Some question about TTP

Hello,
I have a question about using of TTP and can't find out answer in documentation so could you help me

I have a template:

<group name="interfaces.{{ interface }}">
interface {{ interface }}

<group name="other">
 ip dhcp snooping trust {{ ip_dhcp_snooping | set("trust") | _start_ }}
 shutdown {{ status | set("disabled") | _start_ }}
 description {{ description | _start_ }} 
</group>

<group name="switchport">
 switchport nonegotiate {{ nonegotiate  | set("enable") | _start_ }}
 switchport trunk native vlan {{ native_vlan | _start_ }}
 switchport mode {{ mode | _start_ }} 
 switchport access vlan {{ access_vlan | _start_ }}
 switchport trunk allowed vlan {{ trunk_vlan | _start_ }}

<group name="port_security"> 
 switchport port-security {{ state | set("enable") | _start_ }}
 switchport port-security maximum {{ maximum | _start_ }}
</group>
</group>

<group name="stp">
 spanning-tree portfast {{ portfast | set("enable") | _start_ }} 
 spanning-tree portfast trunk {{ portfast_on_trunk | set("yes") | _start_ }}
 spanning-tree bpduguard enable {{ bpduguard | set("yes") | _start_ }}
 spanning-tree bpdufilter enable {{ bpdufilter | set("yes") | _start_ }}
 spanning-tree guard root {{ guard_root | set("yes") | _start_ }}
 spanning-tree guard loop {{ guard_loop | set("yes") | _start_ }}
</group>

<group name="storm-control">
 storm-control action {{ action | _start_ }}

<group name='broadcast'>
 storm-control broadcast level {{ units | _start_ }} {{ value }}
</group>

<group name='multicast'>   
 storm-control multicast level {{ units | _start_ }} {{ value }} 
</group>
</group>

<group name="neighbor_discovery">
 no lldp transmit {{ lldp_transmit | set("no") | _start_ }}
 no lldp receive {{ lldp_receive | set("no") | _start_ }}
 no cdp enable {{ cdp | set("disabled") | _start_ }}
</group>
<group name="undef"> 
 {{ undefined_conf | _line_  | joinmatches(', ') }}
 ! {{ _end_ }}
</group>
</group>

And all works well but the outcome of separate groups is a list, for example:

'GigabitEthernet0/1': 
  {'other': [{'description': '-=uplink=-'}, {'ip_dhcp_snooping': 'trust'}], 
  'switchport': {'mode': 'trunk'}}

How I can represent groups configuration as dictionary?
Or maybe there is any other way to define interface configuration as a nested dictionary ?
Thanks in advance

match within a group is repeated in subsequent group

We are trying to parse the configuration from a Nokia 7750 SR router. So far we have had good luck with ttp and are looking to use it to replace some messy and fragile collections of regexes. However we have run into a issue where we have a matched pattern that is being repeated in the next group and haven't been able to find out whether it is our template or a bug. For instance, if you look at the output the sdp with id 8035 from epipe id 103206 shows up in the next epipe group 103256.
The formatting is off below unless you open it to edit ... Is there a way to preserve the spacing?

Here is our template:

<macro>
import re
def qinq(data):
    data = re.sub(r"\*", r"qinq", data)
    return data 
</macro>

<group name="service">
    service {{ ignore }}
    <group name="epipe.{{ service_id }}"  default="none">
        epipe {{ service_id | _start_ }} customer {{ customer_id }} create
            description "{{ description | ORPHRASE | default("none") }}"
            service-mtu {{ service_mtu | default("none") }}
            service-name "{{ service_name | ORPHRASE | default("none") }}"
        <group name="endpoint"  default="none">
            endpoint {{ endpoint | _start_ }} create
                revert-time {{ revert_time | default("none") }}
            exit {{ _end_ }}
        </group>     
        <group name="sap.{{ sap_id }}"  default="none">
            sap {{ sap_id | macro("qinq") | _start_ | ORPHRASE }} create
                description "{{ description | ORPHRASE | default("none")}}"
                multi-service-site "{{ mss_name | default("none") }}"
            <group name="ingress" default="default_ingress" >
                ingress {{ _start_ }}
                    qos {{ sap_ingress | default("1") }}                 
                    scheduler-policy {{ scheduler_policy | default("none")}}
                exit {{ _end_ }}
            </group>
            <group name="egress" default="default_egress">
                egress {{ _start_ }}
                    scheduler-policy {{ scheduler_policy | default("none") }}
                    qos {{ sap_egress | default("1)") }}
                exit {{ _end_ }}
            </group>
                no shutdown {{ state | set("enabled") | default("disabled") }}
            exit {{ _end_ }}
        </group>
        <group name="pwr_sdp.{{pwr_spoke_sdp_id}}**" default="none"> 
            spoke-sdp {{ pwr_spoke_sdp_id | default("none")}}:{{vc_id | _start_ | default("none") }} endpoint {{ endpoint | default("none") }} create             
                precedence {{ precedence | default("default_precedence") }}
                no shutdown {{ state | set("enabled") | default("disabled") }}
            exit {{ _end_ }}
        </group> 
        <group name="regular_sdp.{{r_spoke_sdp_id}}**" default="none"> 
            spoke-sdp {{ r_spoke_sdp_id }}:{{vc_id | _start_ }} create             
                no shutdown {{ state | set("enabled") | default("disabled") }}
            exit {{ _end_ }}
        </group> 
            no shutdown {{ state | set("enabled") | default("disabled") }}
        exit {{ _end_ }}
    </group>
    exit {{ _end_ }}
</group>

Here is an extract of the config file which we are using:

 epipe 103076 customer 160 create
            description "vf=EWL:cn=TATA_COM:tl=2C02495918:st=act:"
            service-mtu 1588
            service-name "EPIPE service-103076 DKTN08a-D0105 (63.130.108.41)"
            sap 1/2/12:20.* create
                description "vf=EWL:cn=TATA_COM:tl=2C02495890:st=act:"
                multi-service-site "TATA_VSNL_STRAT_A206_LAN10"
                ingress
                    queue-override
                        queue 1 create
                            cbs default
                            mbs 40 kilobytes
                            rate 10000 cir 10000
                        exit
                    exit
                exit
                egress
                    queue-override
                        queue 1 create
                            cbs default
                            mbs 40 kilobytes
                            rate 10000 cir 10000
                        exit
                    exit
                exit
                accounting-policy 4
                no shutdown
            exit
            spoke-sdp 8051:103076 create
                no shutdown
            exit
            no shutdown
        exit
        epipe 103206 customer 1904 create
            description "vf=1273:cn=skanska:tl=3C02407455:st=act:no='SKANSKA UK PLC Stepney Green E1 3DG'"
            service-mtu 1988
            service-name "EPIPE service-103206 DKTN08a-D0105 (63.130.108.41)"
            sap 2/2/3:401.100 create
                description "vf=1273:cn=skanska:tl=3C02407455:st=act:no='SKANSKA UK PLC Stepney Green E1 3DG'"
                multi-service-site "SKANSKA_E13DG_A825_LAN1"
                ingress
                    qos 11010
                    queue-override
                        queue 1 create
                            cbs default
                            mbs 1188 kilobytes
                            rate max cir 47500
                        exit
                        queue 3 create
                            cbs default
                            mbs 63 kilobytes
                            rate max cir 2500
                        exit
                    exit
                exit
                egress
                    qos 11010
                    queue-override
                        queue 1 create
                            cbs default
                            mbs 1188 kilobytes
                            rate max cir 47500
                        exit
                        queue 3 create
                            cbs default
                            mbs 63 kilobytes
                            rate max cir 2500
                        exit
                    exit
                exit
                collect-stats
                accounting-policy 4
                no shutdown
            exit
            spoke-sdp 8035:103206 create
                no shutdown
            exit
            no shutdown
        exit
        epipe 103256 customer 160 create
            description "vf=EWL:cn=TATA_COMM:tl=2C02490189:st=act:"
            service-mtu 1988
            service-name "EPIPE service-103256 DKTN08a-D0105 (63.130.108.41)"
            sap 1/2/12:15.* create
                description "vf=EWL:cn=TATA_COMM:tl=2C02490171:st=act:"
                multi-service-site "TATA_VSNL_STRAT_A206_LAN5"
                ingress
                    qos 11000
                    queue-override
                        queue 1 create
                            cbs default
                            mbs 391 kilobytes
                            rate 100000 cir 100000
                        exit
                    exit
                exit
                egress
                    qos 11000
                    queue-override
                        queue 1 create
                            cbs default
                            mbs 391 kilobytes
                            rate 100000 cir 100000
                        exit
                    exit
                exit
                accounting-policy 4
                no shutdown
            exit
            spoke-sdp 8139:103256 create
                no shutdown
            exit
            no shutdown
        exit
        epipe 103742 customer 160 create
            description "vf=EWL:cn=TATA_COM:tl=2C02410363:st=act:"
            service-mtu 1588
            service-name "EPIPE service-103742 DKTN08a-D0105 (63.130.108.41)"
            sap 5/2/50:20.* create
                description "vf=EWL:cn=TATA_COM:tl=2C02410338:st=act:"
                multi-service-site "TATA_STRAT_LON_A206_LANA"
                ingress
                    qos 11000
                    queue-override
                        queue 1 create
                            cbs default
                            mbs 32 kilobytes
                            rate 8000 cir 8000
                        exit
                    exit
                exit
                egress
                    qos 11000
                    queue-override
                        queue 1 create
                            cbs default
                            mbs 32 kilobytes
                            rate 8000 cir 8000
                        exit
                    exit
                exit
                accounting-policy 4
                no shutdown
            exit
            spoke-sdp 8061:103742 create
                no shutdown
            exit
            no shutdown
        exit
        epipe 55513386 customer 4 vc-switching create
            description "vf=EAGG:cn=Bulldog:tl=VF"
            service-mtu 1526
            spoke-sdp 78:55513386 create
                control-word
                no shutdown
            exit
            spoke-sdp 8245:55513386 create
                control-word
                no shutdown
            exit
            no shutdown
        exit
        epipe 55517673 customer 4 create
            description "vf=EAGG:cn=Bulldog:tl=2C01291821:st=act:no=NGA EPIPE#BAACTQ#VLAN 901"
            service-mtu 1526
            service-name "epipe service-64585 DKTN08a-D0105 (63.130.108.41)"
            endpoint "SDP" create
                revert-time infinite
            exit
            sap 2/2/3:901.* create
                description "2_2_3,H0505824A,Bulldog,VLAN 901"
                ingress
                    scheduler-policy "NGA-LLU-300M"
                    qos 20010
                exit
                egress
                    scheduler-policy "NGA-LLU-300M"
                    qos 20010
                exit
                no shutdown
            exit
            spoke-sdp 8243:55517673 endpoint "SDP" create
                collect-stats
                precedence 1
                no shutdown
            exit
            spoke-sdp 8245:55517673 endpoint "SDP" create
                collect-stats
                precedence primary
                no shutdown
            exit
            no shutdown
        exit

What we see when we parse it is that the one

parser.py:16 <module>
    parser.result()[0][0]: {
        'service': {
            'epipe': {
                '103076': {
                    'service_name': 'EPIPE service-103076 DKTN08a-D0105 (63.130.108.41)',
                    'service_mtu': '1588',
                    'description': 'vf=EWL:cn=TATA_COM:tl=2C02495918:st=act:',
                    'customer_id': '160',
                    'state': 'disabled',
                    'sap': {
                        '1/2/12:20.qinq': {
                            'mss_name': 'TATA_VSNL_STRAT_A206_LAN10',
                            'description': 'vf=EWL:cn=TATA_COM:tl=2C02495890:st=act:',
                            'state': 'enabled',
                            'ingress': {
                                'sap_ingress': '1',
                                'scheduler_policy': 'none',
                            },
                            'egress': {
                                'scheduler_policy': 'none',
                                'sap_egress': '1)',
                            },
                        },
                    },
                    'regular_sdp': {
                        '8051': {
                            'state': 'enabled',
                            'vc_id': '103076',
                        },
                    },
                },
                '103206': {
                    'service_name': 'EPIPE service-103206 DKTN08a-D0105 (63.130.108.41)',
                    'service_mtu': '1988',
                    'description': "vf=1273:cn=skanska:tl=3C02407455:st=act:no='SKANSKA UK PLC Stepney Green E1 3DG'",
                    'customer_id': '1904',
                    'state': 'enabled',
                    'sap': {
                        '2/2/3:401.100': {
                            'mss_name': 'SKANSKA_E13DG_A825_LAN1',
                            'description': "vf=1273:cn=skanska:tl=3C02407455:st=act:no='SKANSKA UK PLC Stepney Green E1 3DG'",
                            'state': 'disabled',
                            'ingress': {
                                'sap_ingress': '11010',
                                'scheduler_policy': 'none',
                            },
                            'egress': {
                                'sap_egress': '11010',
                                'scheduler_policy': 'none',
                            },
                        },
                    },
                    **'regular_sdp': {
                        '8035': {
                            'state': 'enabled',
                            'vc_id': '103206',
                        },**
                    },
                },
                '103256': {
                    'service_name': 'EPIPE service-103256 DKTN08a-D0105 (63.130.108.41)',
                    'service_mtu': '1988',
                    'description': 'vf=EWL:cn=TATA_COMM:tl=2C02490189:st=act:',
                    'customer_id': '160',
                    'state': 'enabled',
                    'sap': {
                        '1/2/12:15.qinq': {
                            'mss_name': 'TATA_VSNL_STRAT_A206_LAN5',
                            'description': 'vf=EWL:cn=TATA_COMM:tl=2C02490171:st=act:',
                            'state': 'disabled',
                            'ingress': {
                                'sap_ingress': '11000',
                                'scheduler_policy': 'none',
                            },
                            'egress': {
                                'sap_egress': '11000',
                                'scheduler_policy': 'none',
                            },
                        },
                    },
                    'regular_sdp': {
                        '8035': {
                            'state': 'enabled',
                        },
                        '8139': {
                            'state': 'enabled',
                            'vc_id': '103256',
                        },
                    },
                },
            },
        },
    } (dict) len=1

Is there a way to handle multiple command options?

I understand that we can group lines that have the same content from this example in the docs:

<input load="text">
interface Loopback0
 description Router-id-loopback
 ip address 192.168.0.113/24
!
interface Gi0/37
 description CPE_Acces
 switchport port-security
 switchport port-security maximum 5
 switchport port-security mac-address sticky
!
</input>

<group>
interface {{ interface }}
 ip address {{ ip }}/{{ mask }}
 description {{ description }}
 ip vrf {{ vrf }}
 {{ port_security_cfg | _line_ | contains("port-security") | joinmatches }}
! {{ _end_ }}
</group>

However, in the situation where something has different options... like below , where cir may not exist, it would be nice if there was something like in regexes where you can put a ? after a token so that ttp will try to match cir and assign cir_percent only if cir is present.
Is there a way to do that without creating multiple lines beginning with percent-rate in the template?

percent-rate {{ pir_percent }} cir {{cir_percent }}

Issue tring to match quote-enclosed strings

Hi there,

I think I hit a bug in how quote-enclosed strings in consecutive lines are treated.

So, if I create this template:

<input load="text">
        one-interface "iface_one"
            description "iface-one"
            address 10.20.30.40/24
            ipv6
                address 1234:0:5678:9abc::d/128
            exit
        exit
        two-interface "iface_two_spaces"
            description "iface two spaces"
            address 10.20.30.40/24
            ipv6
                address 1234:0:5678:9abc::d/128
            exit
        exit
        three-interface "iface_three_no_quotes"
            description iface-three-desc_no-quotes
            address 10.20.30.40/24
            ipv6
                address 1234:0:5678:9abc::d/128
            exit
        exit
</input>

<group name="interfaces.{{ name }}" contains="addr">
{{ ignore('\s*') }}{{ iftype | contains_re('interface') }} "{{ name }}"
{{ ignore('\s*') }}description "{{ description }}"
{{ ignore('\s*') }}address {{ addr | joinmatches (' ; ') }}{{ ignore('.*') }}
</group>

this is what I get:

$ ttp -t ./parsing_ttp.ttp -o yaml
- - interfaces:
      iface_three_no_quotes:
        addr: 10.20.30.40/24 ; 1234:0:5678:9abc::d/128
        iftype: three-interface
      iface_two_spaces:
        addr: 10.20.30.40/24 ; 1234:0:5678:9abc::d/128
        iftype: two-interface

whereas I'd expect something like this instead:

- - interfaces:
      iface_three_no_quotes:
        addr: 10.20.30.40/24 ; 1234:0:5678:9abc::d/128
        iftype: three-interface
      iface_two_spaces:
        addr: 10.20.30.40/24 ; 1234:0:5678:9abc::d/128
        iftype: two-interface
        description: iface two spaces
      iface_one:
        addr: 10.20.30.40/24 ; 1234:0:5678:9abc::d/128
        iftype: one-interface
        description: iface_one

Am I doing something wrong or is this a bug?

Question: Handling duplicates?

Hi, Love the program. I have gotten a lot accomplished thus far but can you please point me to the documents that can solve my issue:

interface Vlan140
ip address {{ ip_addr }} {{ net mask }}
ip address {{ secondary_addr}} {{ secondary_netmask }} secondary

I am a very inexperienced programmer and I do not know what's the best practice to handle multiple secondary address entries. I am only getting the first matched. Sorry for the inconvenience and I thank you for any insight or direction you can point me to.

PS: tried and regex but to no avail, not sure if those are the correct ways to go about this issue.

Thanks!

TTP logger duplicating my project logging?

I've noticed that when I create a logger for my application, the TTP logger is also duplicating my logs, for example:

2020-07-18 10:31:20,016 DEBUG: Waiting 60s before next collection
07/18/2020 10:31:20.16 [TTP DEBUG] 165; Waiting 60s before next collection

I see from looking at the code that ttp uses logging.basicConfig

ttp/ttp/ttp.py

Line 2482 in 42cb4bc

if LOG_LEVEL.upper() in valid_log_levels:

Which by default is set at the WARNING level:

ttp/ttp/ttp.py

Line 136 in 42cb4bc

def __init__(self, data='', template='', log_level="WARNING", log_file=None, base_path='', vars={}):

So I am perplexed as to why TTP is shadowing my DEBUG messages.

As a workaround, I am setting log_level="none" explicitly when I instantiate a ttp object.

I appreciate your time to assist me with this question.

How to load CSV data from a file for Lookup

Hi, I am looking at using the lookup feature for a CSV file. I have seen the example with static data in the documentation. For example below:

<lookup name="aux_csv" load="csv">
ASN,as_name,as_description,prefix_num
65100,Subs,Private ASN,734
65200,Privs,Undef ASN,121
</lookup>

However, I would like to replace this static data with input from a CSV file. Now I believe that using include="" would work but I cannot get it to read in the same data. This is the syntax I am using below with the static data in a CSV file called "gary.csv" :

<lookup name="aux_csv" load="csv" include="./gary.csv">
</lookup>

Is it possible to read in a lookup table from a file and if so is there an example? Can I also do the same with things like vars too? I just see some flexability in having the lookups and vars in a file which can be read into TTP instead of statically defined. Thanks for your time and help on this one. I have a different use case but I need to get the example in the docs working first.

how can I fill with missing column name in previous line? 

    # show service
     Name                         Protocol     Dst-Port/Type
     DISCARD                           UDP                 9
     DNS                                   UDP                53  
                                              TCP                 53
      ECHO                               UDP                   7

how can I get json as:
[
{"Name": "DISCARD","Protocol":"UDP","Port":9},
{"Name": "DNS","Protocol":"UDP","Port":53},
{"Name": "DNS","Protocol":"TCP","Port":53},
{"Name": "ECHO","Protocol":"UDP","Port":7}
]
without missing DNS TCP 53 ??

Python3 issues deprecation warnings

My pytest based test suite streams with tonnes and tonnes of warnings, like this:

<unknown>:43: DeprecationWarning: invalid escape sequence \S
<unknown>:1: DeprecationWarning: invalid escape sequence \S
<unknown>:2: DeprecationWarning: invalid escape sequence \S
<unknown>:3: DeprecationWarning: invalid escape sequence \S
<unknown>:4: DeprecationWarning: invalid escape sequence \d
<unknown>:5: DeprecationWarning: invalid escape sequence \.
<unknown>:6: DeprecationWarning: invalid escape sequence \.
<unknown>:10: DeprecationWarning: invalid escape sequence \S
<unknown>:11: DeprecationWarning: invalid escape sequence \.
<unknown>:12: DeprecationWarning: invalid escape sequence \S
<unknown>:13: DeprecationWarning: invalid escape sequence \S
<unknown>:14: DeprecationWarning: invalid escape sequence \S
<unknown>:15: DeprecationWarning: invalid escape sequence \S
<unknown>:16: DeprecationWarning: invalid escape sequence \S
<unknown>:17: DeprecationWarning: invalid escape sequence \S
<unknown>:10: DeprecationWarning: invalid escape sequence \S

Runs with deprecation warnings unsuppressed no doubt. Reasonable enough for a test suite default.

Fixed by #11

Enhancement: Some way to easily output lines that did not get matched

When dealing with larger configs (a firewall with many NAT statements or ACLs for example) it would be helpful to know what did not actually match the template.
I've read through the docs but haven't easily found a way to do this...apologies if i missed some method to do this

Complex regex in vars are not working

Hello,

when I try to declare a complex regex in the template like below, I've got an error on the execution of the script. I soon as take of
variable regex it works fine

Vars declaration

<var>
check = r"((?<!range\s).+?)"
</var>

#Error message raised :
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/ttp/ttp.py", line 1161, in parse_template_XML
template_ET = ET.XML(template_text)
File "/usr/lib/python3.6/xml/etree/ElementTree.py", line 1314, in XML
parser.feed(text)
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 2, column 14

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/user/template/test_macro.py", line 14, in
parser = ttp(data=result, template="/home/user/template/check_macro.ttp")
File "/usr/local/lib/python3.6/dist-packages/ttp/ttp.py", line 189, in init
self.add_template(template=template)
File "/usr/local/lib/python3.6/dist-packages/ttp/ttp.py", line 311, in add_template
ttp_macro=ttp.get("custom_functions", {}).get("macro", {})
File "/usr/local/lib/python3.6/dist-packages/ttp/ttp.py", line 845, in init
self.load_template_xml(template_text)
File "/usr/local/lib/python3.6/dist-packages/ttp/ttp.py", line 1186, in load_template_xml
parse_template_XML(template_text)
File "/usr/local/lib/python3.6/dist-packages/ttp/ttp.py", line 1171, in parse_template_XML
"\n{}\n".format(template_text)
File "/usr/lib/python3.6/xml/etree/ElementTree.py", line 1314, in XML
parser.feed(text)
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 3, column 14

Python program for the testing purpose

from netmiko import ConnectHandler
from ttp import ttp
import json

check={'device_type': 'cisco_ios','host': "r1",'username': "user",'password': "password","secret":"secret",'verbose' : True,'fast_cli':True}

connect=ConnectHandler(**check)
connect.enable()
result=connect.send_command("show running partition access-list")
print(result)
parser = ttp(data=result, template="/home/scamelo/bypass_enable_nornir/template/check_macro.ttp")
parser.parse()
results = parser.result(format='json')[0]
print(results)

ttp template

<var>
check =r"((?<!range\s).+?)"
</var>
<macro>

def check_acl_rule(data):

    if data["action"]=="remark":
        data["remark_name"]=data["action_detail"]
        data.pop("action_detail")
        
    else:
        itervalue=data["action_detail"].split(" ")
        if len(itervalue)==1:
            data["src_ip"]=itervalue[0]
            data.pop("action_detail")
            
        elif len(itervalue)==2:
            if itervalue[1]=="log":
                data["src_ip"]=itervalue[0]
                data["log"]="log"
                data.pop("action_detail")
                return data
            else:
                data["src_ip"]=itervalue[0]
                data["src_wildcard"]=itervalue[1]
                data.pop("action_detail")
                return data
        else:
            
            data["protocol"]=itervalue[0]
            itervalue.pop(0)
            newparse=data["action_detail"].replace(data["protocol"],"")
            print(itervalue)
            print(newparse)
            """
            for i in range(1, len(itervalue)-1):
                if re.match(r"host",itervalue[i]) is not None:
                    host_check=True
                    any_check=False
                elif re.match(r"host",itervalue[i]) is not None:
            data["toparse"]=itervalue
            """
    return data     
        
                  
</macro>

<group name="ip.{{type}}.{{accesslist}}*">
ip access-list {{type}} {{accesslist|WORD|_start_}}
 <group name="{{rule_num}}*" macro="check_acl_rule">
 {{rule_num}} {{action}} {{action_detail|_line_}}
ip{{_end_}}
</group>
</group>

Help with indentation in templates

Hi,

I have the following data I want to parse through, to simply extract the IP address of the interface.

In [68]: print(data)

interface Loopback0
 description Fabric Node Router ID
 ip address 192.2.101.70 255.255.255.255
 ip pim sparse-mode
 ip router isis 
 clns mtu 1400
end

With the assumption that my template will be indented in python code, I have added numerous whitespaces in the template and want to regex match on it to ignore the white spaces. The template I have is:

In [75]: print(show_run_parser_template)

   {{ ignore("\s+") }}ip address {{ ip_address }} 255.255.255.255

This is unable to parse through the data and I always get an empty list back. Any idea what I am doing wrong here?

Use arbitrary text as a start filter in template.

Hello, thank you for TTP.

Could you please support with solving task i faced. My goal is to parse logfile which consists of repeating blocks: some multiline text following several lines which include values to parse. Preceding text may differ, so i need include blocks with particular preceding text.

To illustrate the idea, below is modified example from quick start (One line is added at the beginning of both blocks and template). I expect to have only first block as out put but got both. Please advise how to modify template to output only first block.

Expected:

[
    {
        "description": "Router-id-loopback",
        "interface": "Loopback0",
        "ip": "192.168.0.113",
        "mask": "24"
    }
]

Got:

[
    [
        {
            "description": "Router-id-loopback",
            "interface": "Loopback0",
            "ip": "192.168.0.113",
            "mask": "24"
        },
        {
            "description": "CPE_Acces_Vlan",
            "interface": "Vlan778",
            "ip": "2002::fd37",
            "mask": "124",
            "vrf": "CPE1"
        }
    ]
]

Code:

from ttp import ttp

data_to_parse = """
Some text which indicates that below block should be included in results
interface Loopback0
 description Router-id-loopback
 ip address 192.168.0.113/24
!
Some text which indicates that below block should NOT be included in results
interface Vlan778
 description CPE_Acces_Vlan
 ip address 2002::fd37/124
 ip vrf CPE1
!
"""

ttp_template = """
Some text which indicates that below block should be included in results
interface {{ interface }}
 ip address {{ ip }}/{{ mask }}
 description {{ description }}
 ip vrf {{ vrf }}
"""

# create parser object and parse data using template:
parser = ttp(data=data_to_parse, template=ttp_template)
parser.parse()

# print result in JSON format
results = parser.result(format='json')[0]
print(results)

Issue with parsing Multicast Template

Hi, I am having some difficulty with a multicast template as it has two permutations in the output. I cannot get this to work correctly when I have both permutations in the output. It's probably do with the hierarchy that I am using. Also I noticed that "*" is not able to be used as a sorting key and instead we get ' ' instead. I thought * was a valid dictionary key so I don't know what the issue is there. Please find my code below and let me know what you think.

import pprint
from ttp import ttp


data_to_parse="""
(*, 239.100.100.100)    
    LISP0.4200, (192.2.101.65, 232.0.3.1), Forward/Sparse, 1d18h/stopped
    LISP0.4201, (192.2.101.70, 232.0.3.1), Forward/Sparse, 2d05h/stopped

(192.2.31.3, 239.100.100.100), 6d20h/00:02:23, flags: FT
  Incoming interface: Vlan1029, RPF nbr 0.0.0.0
  Outgoing interface list:
    LISP0.4100, (192.2.101.70, 232.0.3.1), Forward/Sparse, 1d18h/stopped

"""


show_mcast1="""
<template name="mcast" results="per_template">
<group name="mcast_entries.{{ overlay_src }}">
({{ overlay_src }}, {{ overlay_grp }})
({{ overlay_src | IP }}, {{ overlay_grp }}), {{ entry_uptime }}/{{ entry_state_or_timer }}, flags: {{ entry_flags }}
  Incoming interface: {{ incoming_intf }}, RPF nbr {{ rpf_neighbor }}
</group>

<group>
    {{ outgoing_intf }}, ({{ underlay_src }}, {{ underlay_grp }}), Forward/Sparse, {{ outgoing_intf_uptime }}/stopped
    {{ outgoing_intf }}, ({{ underlay_src }}, {{ underlay_grp }}), Forward/Sparse, {{ oil_uptime }}/{{ oil_state_or_timer}}  
</group>

</template>
"""

parser = ttp(template=show_mcast1)
parser.add_input(data_to_parse, template_name="mcast")
parser.parse()
res = parser.result(structure="dictionary")

pprint.pprint(res, width=100)

Can I create logic to match next N lines?

I'd like to parse something like this:

Origin:
Some random name
Example Address, example number, example city

Into this

{
    "origin": {
        "name": "Some random name",
        "address": "Example Address, example number, example city"
    }
}

Would that be possible?
The name and address doesn't have any specific mark, the only thing for sure is that they are the next two lines after the token Origin

How to match an entire line and discard

This may end up being a feature request but I am unsure on how within TTP you match something but then tell the parser to ignore the rest of the line i.e. we don't care about anything on the rest of the line at all. I have found a way to achieve this which is using {{ line }} which is making the match but then I do del="line" in the group to remove it. I've tried using {{ ignore }} but I haven't got it to match the rest of the line. I think what is required is a very simple feature which does the equivalent of {{ line }} del="line" as a broad brush. If this was a inbuilt documented feature then it would help getting templates to match the data much more easily. Hope this makes sense and I have an example script below.

from ttp import ttp
import json

data_to_parse="""
===============================================================================
Single Fiber Mode  : No                         Min Frame Length : 64 Bytes
IfIndex            : 35848192                   Hold time up     : 5 seconds
"""


ttp_template = """
<group name="fibre_check" method="table" del="_line_">
=============================================================================== {{ _start_ }}
Single Fiber Mode  : {{ SMF }}{{_line_}}                  
IfIndex            : {{ IfIndex }}{{ _line_ }}
</group>
"""

parser = ttp(data=data_to_parse, template=ttp_template)
parser.parse()
res = json.loads(parser.result(format='json')[0])
print(res)

Maybe I am over complicating this using my method but it's a very common use case to want to match something but then ignore whatever follows after that for the whole line.

Parser not matching/returning nested text

Hi. I have this below text

19: IP4 1.1.1.1,   00:03:b2:78:04:13, vname portal, NO SERVICES UP
    Virtual Services:
    http: rport http, group 11, health http (HTTP), pbind clientip
        Real Servers:
        22: 10.10.10.10, web1, group ena, health  (runtime HTTP), 0 ms, FAILED
			Reason: N/A
        23: 10.11.11.11, web2, group ena, health  (runtime HTTP), 0 ms, FAILED
			Reason: N/A
    https: rport https, group 12, health tcp (TCP), pbind clientip
        Real Servers:
        22: 10.10.10.10, web1, group ena, health  (runtime TCP), 0 ms, FAILED
			Reason: N/A
        23: 10.11.11.11, web2, group ena, health  (runtime TCP), 0 ms, FAILED
			Reason: N/A

and below template:

<template name="VIP_cfg" results="per_template">
<group name="{{ vs_instance }}" default="">
{{ vs_instance }}: IP4 {{ vs_ip }}, {{ notneeded | ORPHRASE }}
<group name="services*" default="">
    Virtual Services:
    {{ vs_service }}: rport {{ rport }}, {{ notneeded | ORPHRASE }}
<group name="pool*" default="">
        Real Servers:{{ _start_ }}
        {{ node_id }}: {{ node_ip }}, {{ notneeded | ORPHRASE }}
			Reason: {{ reason }}
</group>
</group>
</group>
</template>

I am trying to match the last group pool...but the parser is not returning anything.

This is what is returning... I have tried to use different variations of this but it looks like its completely skipping the match and only capturing Reason

 {'VIP_cfg': [{'19': {'notneeded': '00:03:b2:78:04:13, vname portal, NO SERVICES UP',
                      'services': [{'notneeded': 'group 11, health http (HTTP), pbind clientip',
                                    'pool': [{'node_id': '',
                                              'node_ip': '',
                                              'notneeded': '',
                                              'reason': 'N/A'}],
                                    'rport': 'http',
                                    'vs_service': 'http'},
                                   {'notneeded': 'group 12, health tcp (TCP), pbind clientip',
                                    'pool': [{'node_id': '',
                                              'node_ip': '',
                                              'notneeded': '',
                                              'reason': 'N/A'}],
                                    'rport': 'https',
                                    'vs_service': 'https'}],
                      'vs_ip': '1.1.1.1'}}]}

Thanks

Inconsistent result from group

There should be a way to force a group to always return a list of dictionaries. As it stands now if a group only matches once it returns a dictionary of dictionary. In my usecase, I am writing router automated testing, the expectation of the code is to always receive the same data-structure in this example a list (even if empty).

template = '''
<group name='demo'>
<group name='audiences'>
Hello {{ audience }}
</group>
</group>
'''

input_1 = '''
Hello World
'''

input_2 = '''
Hello World
Hello Family
'''

The results:

result_1 = [
	[
		{
			"demo": {
				"audiences": {
					"audience": "World"
				}
			}
		}
	]
]

result_2 = [
	[
		{
			"demo": {
				"audiences": [
					{
						"audience": "World"
					},
					{
						"audience": "Family"
					}
				]
			}
		}
	]
]

Could u please help parse address output?

Could u please help parse address output?

here are my code, which missing some columns.

from ttp import ttp
from pprint import pprint

data= '''
SG-6000# show  address
Total entries count: 8.
================================================================================
Address-name            Member-count  Address members & Excluded members
--------------------------------------------------------------------------------
 Any                               1  address members:
                                      0.0.0.0/0
                                      excluded members:

 1.1.1.1                           0  address members:

                                      excluded members:

 192.168.0.224                     0  address members:

                                      excluded members:

 2.2.2.0/32                        0  address members:

                                      excluded members:

 3.3.3.0/24                        0  address members:

                                      excluded members:

 private_network                   5  address members:
                                      10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16
                                      excluded members:
                                      4.4.4.0/24, 1.1.1.1-1.1.2.255
 test                              1  address members:
                                      1.1.101.0/24
                                      excluded members:

 testex                            0  address members:

                                      excluded members:

================================================================================
SG-6000#
'''
template1 = """
 {{ address_name }}     {{ member_count |DIGIT }}  address members:
                                      {{ member }}
                                      excluded members:
                                      {{ excludedmembers }}{{ _end_ }}
"""

parser = ttp(data, template1)
parser.parse()
pprint(parser.result())
#[[[{'address_name': 'Any', 'member_count': '1'},
#   {'address_name': '1.1.1.1', 'member_count': '0'},
#   {'address_name': '192.168.0.224', 'member_count': '0'},
#   {'address_name': '2.2.2.0/32', 'member_count': '0'},
#   {'address_name': '3.3.3.0/24', 'member_count': '0'},
#   {'address_name': 'private_network', 'member_count': '5'},
#   {'address_name': 'test', 'member_count': '1'},
#   {'address_name': 'testex', 'member_count': '0'}]]]

Multi-line composition

Hello.
Thank you for great product.

I am stumbling upon parsing the table style text.
Probably I expect to use "joinmaches" or "method=table", but it doesn't work.
I cannot add the port numbers of VLAN50 and VLAN60 in the next line.

 VLAN Name                             Status    Ports
 ---- -------------------------------- --------- -------------------------------
 1    default                          active    Gi0/1
 10   Management                       active    
 50   VLan50                           active    Fa0/1, Fa0/2, Fa0/3, Fa0/4, Fa0/5, Fa0/6, Fa0/7, Fa0/8, Fa0/9
                                                 Fa0/10, Fa0/11, Fa0/12
 60   VLan60                           active    Fa0/13, Fa0/14, Fa0/15, Fa0/16, Fa0/17, Fa0/18, Fa0/19, Fa0/20
                                                 Fa0/21, Fa0/22, Fa0/23, Fa0/24
 1002 fddi-default                     act/unsup 
 1003 token-ring-default               act/unsup 
 1004 fddinet-default                  act/unsup 
 1005 trnet-default                    act/unsup

I'm not good at programming, but I was able to write templates, which made my work more efficient.
thx

Joining corresponding fields

Hi,

I was wondering if I could join relations from a fortinet config

For example:

FortiGate address definition:

config firewall address
    edit "Address-1"
        set subnet 192.168.2.0 255.255.255.0
    next
    edit "Address-2"
        set subnet 192.168.3.0 255.255.255.0
    next
 end

FortiGate policy definiton:

config firewall policy
    edit 1
        set name "policy-1"
        set srcintf "CORE"
        set dstintf "OFFICE"
        set srcaddr "Address-1"
        set dstaddr "Address-2"
        set action accept
    next
end

Is there a way to map this relation with TTP?

Removing _root_template_ and getting a full dictonary (without lists)

I have the following code below, which works fine but I cannot make this a full dictionary output. Instead, get a list with root_template as the key and then it is a dictionary. Is there a way to make this a proper dictionary without having any list involved? How do I remove the ['root_template'] from appearing in the result? I really like TTP by the way, I just need to figure out any small details like this. Thank you in advance for any help and hope this question makes sense

type(res_a['root_template'])
<class 'list'>

from ttp import ttp
import pprint

data_to_parse_a="""
R1#sh vrrp
GigabitEthernet1 - Group 100
DC-LAN Subnet
  State is Master
  Virtual IP address is 192.168.10.1
  Virtual MAC address is 0000.5e00.0164
  Advertisement interval is 1.000 sec
  Preemption enabled
  Priority is 120 
  VRRS Group name DC_LAN
    Track object 1 state Up decrement 30
  Authentication text "hash"
  Master Router is 192.168.1.233 (local), priority is 120 
  Master Advertisement interval is 1.000 sec
  Master Down interval is 3.531 sec
"""

data_to_parse_b="""
R2#sh vrrp
GigabitEthernet1 - Group 100
  State is Init
  Virtual IP address is 192.168.10.1
  Virtual MAC address is 0000.5e00.0164
  Advertisement interval is 1.000 sec
  Preemption enabled
  Priority is 115
  Authentication text "hash"
  Master Router is 192.168.1.233, priority is 120 
  Master Advertisement interval is 1.000 sec
  Master Down interval is 3.550 sec
"""


ttp_template = """
<group name="show_vrrp**.{{ interface }}">
{{ interface }} - Group {{ VRRP_Group | DIGIT }}
{{ VRRP_Description | ORPHRASE }}
  State is {{ VRRP_State | contains("Master") | let("VRRP Master for this Group") }}
  State is {{ VRRP_State | contains("Init") | let("VRRP Slave for this Group") }}
  Virtual IP address is {{ VRRP_Virtual_IP | IP }}
  Virtual MAC address is {{ VRRP_MAC | ORPHRASE }}
  Advertisement interval is {{ adv_interval | ORPHRASE }}sec
  Preemption {{ VRRP_Preempt | ORPHRASE }}
  Priority is {{ VRRP_Priority | ORPHRASE }} 
  VRRS Group name {{ Group_Name }}
    Track object {{ track_obj }} state {{ track_obj_status }} decrement {{ track_obj_decrement }}
  Authentication text {{ Auth_Text }}
  Master Router is {{ Master_IP }} (local), priority is {{ priority }}
  Master Router is {{ Master_IP }}, priority is {{ priority }} 
  Master Advertisement interval is {{ master_int }} sec
  Master Down interval is {{ master_down }} sec
</group>
"""



parser = ttp(data=data_to_parse_a, template=ttp_template)
parser.parse()

res_a = parser.result(structure="dictionary")
pprint.pprint(res_a)

parser = ttp(data=data_to_parse_b, template=ttp_template)
parser.parse()


res_b = parser.result(structure="dictionary")
pprint.pprint(res_b)

Documentation is Inaccurate

Per documentation located here for Output Macros, the return should be a list of list of dictionaries, instead the result is a list of list of list of dictionaries.

Example:

Python 3.7.4 (default, Sep  7 2019, 18:27:02) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> 
>>> 
>>> from ttp import ttp
>>> 
>>> template = '''
... <input load="text">
... interface Vlan778
...  ip address 2002::fd37::91/124
... !
... interface Loopback991
...  ip address 192.168.0.1/32
... !
... </input>
... 
... <macro>
... def check_svi(data):
...     # data is a list of lists:
...     # [[{'interface': 'Vlan778', 'ip': '2002::fd37::91', 'mask': '124'},
...     #   {'interface': 'Loopback991', 'ip': '192.168.0.1', 'mask': '32'}]]
...     for item in data[0]:
...         if "Vlan" in item["interface"]:
...             item["is_svi"] = True
...         else:
...             item["is_svi"] = False
... </macro>
... 
... <group>
... interface {{ interface }}
...  ip address {{ ip }}/{{ mask }}
... </group>
... 
... <output macro="check_svi"/>
... '''
>>> 
>>> parser = ttp(template=template)
>>> parser.parse()
>>> parser.result()
[[[{'ip': '2002::fd37::91', 'mask': '124', 'interface': 'Vlan778', 'is_svi': True}, {'ip': '192.168.0.1', 'mask': '32', 'interface': 'Loopback991', 'is_svi': False}]]]

Carrier return (\r) character issue

I'm reading data from telnet like this:

response = client.read_until(b"\r\n\r\n\r\n;", 10).decode("utf-8")
parser = ttp(data=response, template=open("mytemplate.ttp").read())
parser.parse()

However parser.result() returns an empty array.

I have noticed that all my lines have a trailing \r\n, so I removed the \r part changing to:

response = client.read_until(b"\r\n\r\n\r\n;", 10).replace(b"\r", b"").decode("utf-8")

and it worked!

May this be a bug in ttp ?

Named input parsed by group mapped to default input

Looks like an inputs mapping bug:

def test_input_to_groups_mapping():
    template = """
<input name="sys_host" load="text">
 Static hostname: localhost.localdomain
         Chassis: vm
      Machine ID: 2a26648f68764152a772fc20c9a3ddb3
Operating System: CentOS Linux 7 (Core)
</input>
    
<group name="system">
 Static hostname: {{ hostname }}
         Chassis: {{ chassis }}
      Machine ID: {{ machine_id }}
Operating System: {{ os | ORPHRASE }}
</group>
    """
    parser = ttp(template=template)
    parser.parse()
    res = parser.result()
    pprint.pprint(res)

gives:

[[{'system': {'chassis': 'vm',
              'hostname': 'localhost.localdomain',
              'machine_id': '2a26648f68764152a772fc20c9a3ddb3',
              'os': 'CentOS Linux 7 (Core)'}}]]

while should produce empty results, because group name="system" maps to default input and not to sys_host input

Thanks!

No issue here at all!

I just wanted to express some appreciation for your time and effort spent on this project. Your contribution has definitely improved my day to day workflow. I recommend TTP to everyone who comes to the inevitable crossroad of textfsm and regex. Kudos to you! @dmulyalin

Please let me know if there any way I can help out with the project.

Once again, Great job!
Marcus

Clearing the parser results for re-using the parser instance?

I've created a template parser and parsed some input. Now I want to "clear" the template parser so that I can feed it new input and only get the result from the new input.

I see that I can run clear_input(), but after I run that I then run parse() and result() and I see two dictionaries in the results list; the original parsed data is still there. Is there a mechanism to clear the existing parsed data?

I hacked this:

In [86]: template_parser._templates[0].results.clear()

In [87]: template_parser.result()
Out[87]: [[]]

But I could not find a public API that would achieve the same goal.

I appreciate your time to answer my question. Thank you!

Is there a way to escape xml/html tags?

Hello! I'm openning this issue because I coudn't find anything in the docs
Is there a way to escape html/xml chars?

Simplest example:
Data:

Name:Jane<br>

Template:

Name:{{name}}<br>

Expected result:

[
	[
		{
			"name": "Jane"
		}
	]
]

I tried to escape using &gt; and &lt; with no success

Create a template sources like textfsm

Hello,

It's not an issue but more an enhancement that could be cool. :-)

Is there a way that we could create a repo where we could share all the templates that we use and standardize it like textfsm ?

The strength of TTP is the fact is it more user friendly than textFSM.

We would be able to share in the community our different templates.

Regards,

Sam

null path handling issue

template:

<input load="text">
router bgp 65100
 rid 1.1.1.1
 !
 address-family ipv4 vrf vrf_name
  redistribute ospf 321 match internal external 1 external 2 route-map vrf_nameSVRF-OSPF->BGP
</input>

<group name="BGP">
router bgp 65100{{ _start_ }}
## rid {{ rid }}
<group name="_">
 address-family ipv4 vrf vrf_name {{ _start_ }}
  redistribute ospf 321 match internal external 1 external 2 route-map {{ ospf_to_bgp_redistr_rpl }}
</group>
</group>

produces:

"BGP": {
            "_": {},
            "ospf_to_bgp_redistr_rpl": "vrf_nameSVRF-OSPF->BGP"
        }

while should produce:

"BGP": {
            "ospf_to_bgp_redistr_rpl": "vrf_nameSVRF-OSPF->BGP"
        }

Help with TTP for fixed column table

Hi @dmulyalin - I am having a very challenging time trying to create a TTP template to handle this tabular input. Would you have any suggestions? My two issues is (1) handling the case where the "Name" column is empty and is instead consuming the "Status" as "Name", and (2) Handling the case for Gi0/16 - I am using ORPHRASE but TTP is only consuming the first word and then making Status "te1/1/4".

Port      Name               Status       Vlan       Duplex  Speed Type
Gi0/1     PIT-VDU213         connected    18         a-full  a-100 10/100/1000BaseTX
Gi0/2     PIT-VDU211         connected    18         a-full  a-100 10/100/1000BaseTX
Gi0/3     PIT-VDU212         notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/4                        connected    18         a-full  a-100 10/100/1000BaseTX
Gi0/5                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/6                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/7                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/8                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/9                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/10                       notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/11                       notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/12                       notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/13                       disabled     1            auto   auto 10/100/1000BaseTX
Gi0/14                       disabled     1            auto   auto 10/100/1000BaseTX
Gi0/15                       connected    trunk        full   1000 1000BaseLX SFP
Gi0/16    pitrs2201 te1/1/4  connected    trunk        full   1000 1000BaseLX SFP

I very much appreciate your help and guidance. Thank you!

Match values override one another 

data = """
/c/slb/real 3
	ena
	ipver v4
	rip 1.1.1.229
	name "vgqa6-01"
/c/slb/group 13
	ipver v4
	backup g14
	add 23
	name "appvip1-tcp"
"""

with the following template:

real_servers_ttp_template = """
<group name="nodes" default="">
/c/slb/real {{ node_seq | isdigit }}
	{{ config_state }}
	ipver {{ ipver }}
	rip {{ node_ip | is_ip }}
	inter {{ interval }}
	retry {{ retry }}
	name {{ node_name | replace('"','') }}
<group name="node_port*">
	addport {{ node_port | isdigit }}
</group>
</group>
"""

groups_ttp_template = """
<group name="pools" default="">
/c/slb/group {{ group_seq | isdigit }}
	ipver {{ ipver }}
<group name="backup_obj">
	backup {{ type | re("\w") }}{{ seq | re("\d+") }}
</group>
	metric {{ lb_method }}
	health {{ health_monitor }}
<group name="node_list*">
	add {{ node_seq | isdigit }}
</group>
	name {{ group_name | replace('"','') }}
</group>
"""

which is giving me the following output:

[
    {
        "nodes": {
            "config_state": "ena",
            "interval": "None",
            "ipver": "v4",
            "node_ip": "1.1.1.229",
            "node_name": "appvip1-tcp",
            "node_seq": "3",
            "retry": "None"
        }
    }
]
[
    {
        "pools": {
            "backup_obj": {
                "seq": "14",
                "type": "g"
            },
            "group_name": "appvip1-tcp",
            "group_seq": "13",
            "health_monitor": "None",
            "ipver": "v4",
            "lb_method": "None",
            "node_list": [
                {
                    "node_seq": "23"
                }
            ]
        }
    }
]

group_name and node_name are the same, while should be different

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.