knipknap / gelatin Goto Github PK

Any idea why this XML:

<xml>
  <vip LocalAddress="10.140.67.150" LocalPort="10002">
    <entries>
      <remoteaddress>10.57.0.168</remoteaddress>
      <remoteport>18676</remoteport>
    </entries>
  </vip>
  <vip LocalAddress="10.140.67.150" LocalPort="10183">
    <entries>
      <remoteaddress>10.57.0.168</remoteaddress>
      <remoteport>6900</remoteport>
    </entries>
    <entries>
      <remoteaddress>10.57.0.176</remoteaddress>
      <remoteport>5293</remoteport>
    </entries>
  </vip>
</xml>

translates to this Json:

{
    "vip": [
        {
            "@LocalAddress": "10.140.67.150",
            "@LocalPort": "10002",
            "entries": {
                "remoteaddress": {
                    "#text": "10.57.0.168"
                },
                "remoteport": {
                    "#text": "18676"
                }
            }
        },
        {
            "@LocalAddress": "10.140.67.150",
            "@LocalPort": "10183",
            "entries": [
                {
                    "remoteaddress": {
                        "#text": "10.57.0.168"
                    },
                    "remoteport": {
                        "#text": "6900"
                    }
                },
                {
                    "remoteaddress": {
                        "#text": "10.57.0.176"
                    },
                    "remoteport": {
                        "#text": "5293"
                    }
                }
            ]
        }
    ]
}

that is pretty ugly (why @, #text ???)
?

Thanks in advance

The main Gelatin example at https://github.com/knipknap/Gelatin/wiki has a couple of references to: out.leave()

However, the Gelatin Syntax Reference makes no reference to this function (nor does Gelatin/compiler/Context.py). I'm assuming that it no longer does anything. Perhaps references to it should be removed from the main Gelatin example?

I'm thinking of round tripping, so something that works elegantly with text->xml is a factor.

Thanks, Martin.

Hi. Installing the package dependencies and got the following error;

sudo apt-get install python-json

Package python-json is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or is only available from another source
E: Package 'python-json' has no installation candidate

If I install python-cjson and change 'python-json' to 'python-cjson' in files below, will it work?

~/knipknap-Gelatin-273d92e$ grep -irn python-json *

debian/control:13:Depends: ${python:Depends}, python-json, python-lxml, python-yaml, ${misc:Depends}
README:14:* python-json

Could you change your dependencies from 'python-json' to 'python-cjson'?

Thanks.
Andrew

I can get the gel script to run from the source dir, or in c:\Python27\Scripts, but i'd expect to be able to run "python gel -s blah.gel inputfile" from any directory, assuming the argument paths are correct and python is in the path.

I would presume that on windows, that gel might need a .py extension and an addition to PATHEXT so the windows shell will run it. Files without extensions in PATHEXT wont run at the shell at all. They need an extension eg: .cmd or .bat (which run scripts by default) to run at the shell, otherwise you need to call python.exe with gel as the first parameter (but this did not work for me).

So plain "gel" at the command line wont work, 1. as it is has no extension, and 2. that extension is not in PATHEXT.

One workaround would be rename gel to add a extension and then add it to PATHEXT so the windows shell will run it :

set PATH=C:\Python27;C:\Python27\Scripts;%PATH%
set PATHEXT=.PY;%PATHEXT%

I renamed gel to gel.py, and now i can run the test script in a non-default location:

C:\Connectivity\python\gelatin\syntax\simple>gel -s syntax.gel input1.txt


1st Ave
1978-01-01


2nd Ave
1970-01-01

Admittedly the PATH/PATHEXT should be done by the python install, but having correct file extension on windows is a key.

I'm working in Ubuntu 16.04 LTS. I cloned the repository and opened the terminal on its path.
then I created input.txt file and when I run this command :
gel -s mysyntax.gel input.txt

I get this error :

No command 'gel' found, did you mean:
 Command 'gegl' from package 'gegl' (universe)
 Command 'tel' from package 'orville-write' (universe)
 Command 'genl' from package 'iproute2' (main)
 Command 'gcl' from package 'gcl' (universe)
 Command 'gdl' from package 'gnudatalanguage' (universe)
 Command 'gle' from package 'gle-graphics' (universe)
 Command 'bel' from package 'belier' (universe)
 Command 'geo' from package 'rheolef' (universe)
 Command 'gem' from package 'ruby' (main)
 Command 'gen' from package 'multimon' (universe)
 Command 'el' from package 'oneliner-el' (universe)
 Command 'ge' from package 'pvm-examples' (universe)
gel: command not found

any help please..

If you have syntax.gel for IOS config file? Thanks in advance!

According to your post from (https://stackoverflow.com/questions/1191031/cisco-ios-show-conf-to-xml) there are .gel files for IOS. I can't find the syntax.gel for ios show configuration: syntax/ios/show_running_config/syntax.gel)

Consider this case wherein there are multiple fields with fields corresponding to each and also the xml tag which comes by default while using Gelatin has been replaced with Annotation tag:

<Annotation> #instead of xml <file>img</file> <Object> <name>cycle</name> <boundingbox> #a field nested in another field <x>2.0</x> <y>0.3</y> </boundingbox> </Object> </Annotation>

Is it possible to generate a gel file for the specified scenario?

I'd like to make a feature request to add a "do.warn" that would work just like "do.say" but write to stderr instead.

When installing on CentOS I got an error under version.sh:

tempfile: command not found

As I saw searching google, I tried to replace VERSION_FILE_TMP=mktemp under version.sh and could install with no error. But if I try to run it I get the following:

[joao@nirvana teste]$ gel -s mysyntax.gel input.txt
Traceback (most recent call last):
File "/usr/local/bin/gel", line 4, in
import pkg_resources
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2655, in
working_set.require(requires)
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 648, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 546, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: Gelatin==v1.0.17-g0e1b9c9

If I run it from the sources it works ok.

This means my output JSON has @ symbols in front of all the keys.

Hi there,

I appear to be getting some strange indenting errors. When I copy the quickstart .gel file and text and run the provided command, I get the error Exception: indent must be a multiple of 4, is 14 in line 24: u' user('

I am using Python 3.7.3, Visual Studio Code with spaces set to 4 and UTF-8 encoding.

Here is the copied quickstart syntax file:

# Define commonly used data types. This is optional, but
# makes your life a litte easier by allowing to reuse regular
# expressions in the grammar.
define nl /[\r\n]/
define ws /\s+/
define fieldname /[\w ]+/
define value /[^\r\n,]+/
define field_end /[\r\n,] */

grammar user:
    match 'Name:' ws value field_end:
        out.add_attribute('.', 'firstname', '$2')
    match 'Lastname:' ws value field_end:
        out.add_attribute('.', 'lastname',  '$2')
    match fieldname ':' ws value field_end:
        out.add('$0', '$3')
    match nl:
        do.return()

# The grammar named "input" is the entry point for the converter.
grammar input:
    match 'User' nl '----' nl:
        out.open('user')
        user()

and the copied text:

User
----
Name: John, Lastname: Doe
Office: 1st Ave
Birth date: 1978-01-01

User
----
Name: Jane, Lastname: Foo
Office: 2nd Ave
Birth date: 1970-01-01

The indentation provided for the syntax file in the docs (https://gelatin.readthedocs.io/en/latest/quick.html) is different from the Readme, but also does not work.

I'm having a problem with Gelatin failing to match on long lines. Is there a line length limit?

Case in point: given the following line

set ca "-----BEGIN CERTIFICATE-----MIIDyTCCArGgAwIBAgIQfL6Zck9KLYFGZ0kGPboOdTANBgkqhkiG9w0BAQUFADBrMRMwEQYKCZImiZPyLGQBGRYDZWR1MRwwGgYKCZImiZPyLGQBGRYMY29sb3JhZG9tZXNhMRIwEAYKCZImiZPyLGQBGRYCYWQxIjAgBgNVBAMTGUNvbG9yYWRvTWVzYVVuaXZlcnNpdHktQ0EwHhcNMTIwNDI1MTcxOTI2WhcNMjcwNDI1MTcyOTI1WjBrMRMwEQYKCZImiZPyLGQBGRYDZWR1MRwwGgYKCZImiZPyLGQBGRYMY29sb3JhZG9tZXNhMRIwEAYKCZImiZPyLGQBGRYCYWQxIjAgBgNVBAMTGUNvbG9yYWRvTWVzYVVuaXZlcnNpdHktQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCe+cda5Xk4AbJFNiN6sVXMsl5QGBcIfPnPa6ytmpSmRQD0Ce/bCBQU4mxruBczlVinJ6UR+jslrnWF7bwSHKmN8CDNYVYGnTIYOqnnx8N89d2RE00agycDHlg6pV3PB5azbo5cG1M9JmKaWlNXZDGpTurTmFE6rOr5yGPAgZ1eunZTMiHq1uQhpXF/sIEOEx/3hYkLHmFW4HGXww27i5G4/rl81qLOqevsCf85uMTrVuE9xvSU+VuabxkR1sajV+o5cMsRY84BZJToHl+ECIZP9pBzWSheVMYMg2ZgX0uNAg6OVgNJ3P2Wpi4VG3rRrSsOCZ6xRD0UdgvhAxdy5+zRAgMBAAGjaTBnMBMGCSsGAQQBgjcUAgQGHgQAQwBBMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQVC6yhHmS+36I8MNBd54scdWnCujAQBgkrBgEEAYI3FQEEAwIBADANBgkqhkiG9w0BAQUFAAOCAQEAWnQVv32b92Gbabe22YSOwZQ5uTx3mwsoiYROTjQEGoTje+F21nynmD8Qn5GgVardg2Yk2bYvDvBdCk3GxuBgGgPm4PrhcrQmrxP5KnlCwt6Co7aw6/dY7UvzX5zZSjv/EiDLa0MsfLDFf+o9EmtyeL4tel4eqSo/qpUW9A22KgULiLFnvllkdlaYLbizLSiOobvWrJ/ygto8O/RFGy3n/qLeOcpy3U1U1HhK5Cf3js/M80ADFgHqyWkycWTY6C/wJ5n8Ku1Lk6ZdQejDAFkcuTOwteUS7YA4UCx3Mxik1I+oOiTlbLy6C8UMzRwhcygne5Zw6AgLhi8tzeFZpa4mZw==-----END CERTIFICATE-----"

Fails to match the following grammar:

define set      /\s*set/
define ws /[ \t]+/
define bareword       /[^"\s]+/
define dq /"/
define eol /[ \t]*[\r\n]/
define longtext /[^"\r\n]+/

grammar input:
    match set ws bareword ws dq longtext dq eol:
        do.return()

However, if I shorten that line down considerably by truncating the double-quoted string, then it matches....

I am trying to use this pkg, but I do not find any syntax.gel file inside.... I am looking one for Cisco devices.

Get a local copy to play with in the usual python way...

• git clone Gelatin
• cd Gelatin
• python setup.py develop

Unhappy failure

    $ gel
    Traceback (most recent call last):
      File "/usr/local/bin/gel", line 7, in <module>
        execfile(__file__)
      File "/Users/fish/Projects/Gelatin/gel", line 19, in <module>
        from Gelatin      import __version__, generator
    ImportError: No module named Gelatin

This is easily fixed with ln

    ln -s src/Gelatin Gelatin

ENV

I'm using Python 2.6 on OSX 10.6

Hi I am having a very strange situation when a "description" has an & on it, it makes it fail for some reason... here an example

line to be "tokenized" and translated
ip access-list standard my_long_description_with_an_&_as_it_fails

The output in JSON is:
"@description": "\"my_long_description_with_an_",
"permit": {
"@Mask": "0.0.0.x",
"@family": "x.x.x.x"
}
The output if I remove the "&" or any other string...
"@description": "my_long_description_with_an_as_it_fails",
"permit": {
"@Mask": "0.0.0.x",
"@family": "x.x.x.x"
}

Grammar (I reduced to exemplify... the issue is at the check_list_items function... and only in that use case:
define word /\w+/
define name /[a-zA-Z0-9]+[-_a-zA-Z0-9]*/
define description /[^\r\n]+/

grammar access_list_items(default):
match /(?:permit|deny)/ fs name fs word fs word nl:
out.add('$0/$2?inbound="$4"&outbound="$6"')
do.return()

grammar check_access_list(default)
match 'ip' fs 'access-list' fs name fs description nl:
out.enter('$2/$4?description="$6"')
access_list_items()
do.return()

hallo everybody ,
I have a couple of questions about Gelatin, since I'm new to deal with it :)

first of all I have a text file, which has a lot of tables inside and I have to use gelatin to convert these tables to json .
the main goal to use json is to monitor my system by using elasticsearch data base so I can show my analysis in Grafana dashboard later .

my question is , how can I download gelatine on my remote server putty so I can use the command of converting the text to json , because when I use it directly , the system send me a message "bash:command is not found ".

thank you all for your help

There appears to be a bug or quirk about how functions are executed in a grammar match. Given the following Gelatin snippet:

grammar doset:
    match bareword:
        out.open('set?attr="$0"')
        addelement()
        do.return()

When "bareword" is matched, everything goes as expected, until the addelement() grammar returns. At the time it returns, the next thing that happens is NOT the call to do.return(), but that the doset() grammar loops and starts again.

Context: I'm trying to parse a text snippet the following:

config system interface
    edit "BYOD VLAN"
        set dhcp-relay-service "enable"
        set dhcp-relay-ip "10.90.2.101" "10.90.2.109" "10.90.2.110"
        set allowaccess "ping"
    next
end

Into the following XML:

<xml>
  <system-interface>
    <edit dqname="BYOD VLAN">
      <dhcp-relay-service>"enable"</dhcp-relay-service>
      <set attr="dhcp-relay-ip">
        <val>"10.90.2.101"</val>
        <val>"10.90.2.109"</val>
        <val>"10.90.2.110"</val>
      </set>
      <allowaccess>"ping"</allowaccess>
    </edit>
  </system-interface>
</xml>

The number of parameters after "set dhcp-relay-ip ..." can be arbitrary.

I have a gelatin file that is close to what I want:

#------------------------------------------------------------------------------------
define eol /[ \t]*[\r\n]/
define ws /[ \t]+/

define bareword       /[^"\s]+/
define dqstring        /"(?:(\\[^\r\n]|[^"\r\n])*)"/

define config   /\s*config/
define edit     /\s*edit/
define end      /\s*end/
define next     /\s*next/
define set      /\s*set/

grammar addelement:
    match ws dqstring:
        out.create('val', '$1')
    match ws bareword:
        out.add('$1', 't')
    match eol:
        do.return()

## The "do.return() in 'match bareword:' is not honored
## The "match ws:" statement below it is the hack for working around this problem.
grammar doset:
    match bareword ws dqstring eol:
        out.add('$0', '$2')
        do.return()
    match bareword:
        out.open('set?attr="$0"')
        addelement()
        do.return()
    match ws:
        do.return()

grammar parse2next:
    match next:
        do.return()
    match set ws:
        doset()

grammar parse2end:
    match end eol:
        do.return()
    match edit ws dqstring eol:
        out.open('edit?dqname=$2')
        parse2next()

grammar input:
    match config ws 'system' ws bareword eol:
        out.open('system-$4')
        parse2end()

#------------------------------------------------------------------------------------

In looking through the source, it looks like "do.return()" can take a numeric parameter to indicate how many levels back to go. However, using "do.return(2)" in the addelement() grammar doesn't seem to work either.

Thoughts?

- Daniel

Hi,

When i convert text to json using the examples you have provided in your Git Wiki, i get incomplete json output.
Also, It outputs for only the second user.

ubuntu@ubuntu-admin:~$ gel -s syntax.gel -f json sample.txt
user [] <Gelatin.generator.Json.Node instance at 0x8f6c62c>
user [] <Gelatin.generator.Json.Node instance at 0x8f6c62c>
{
    "user": {
        "birth-date": {
            "#text": "1970-01-01"
        }, 
        "@firstname": "Jane", 
        "office": {
            "#text": "2nd Ave"
        }, 
        ".": {}
    }
}

But, while converting to XML i get perfect output of both the Users.
How do i fix it ?

I installed like so: sudo easy_install pyyaml simpleparse lxml gelatin

When I got to run it, I get: gel -s xo.gel xo.txt

Traceback (most recent call last):
  File "/usr/local/bin/gel", line 5, in <module>
    pkg_resources.run_script('Gelatin==DEVELOPMENT', 'gel')
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 499, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1235, in run_script
    execfile(script_filename, namespace, namespace)
  File "/usr/local/lib/python2.7/dist-packages/Gelatin-DEVELOPMENT-py2.7.egg/EGG-INFO/scripts/gel", line 20, in <module>
    from Gelatin.parser      import Parser
  File "/usr/local/lib/python2.7/dist-packages/Gelatin-DEVELOPMENT-py2.7.egg/Gelatin/parser/__init__.py", line 15, in <module>
    from Parser import Parser
  File "/usr/local/lib/python2.7/dist-packages/Gelatin-DEVELOPMENT-py2.7.egg/Gelatin/parser/Parser.py", line 23, in <module>
    _ebnf      = open(_ebnf_file).read()
IOError: [Errno 2] No such file or directory: '/usr/local/lib/python2.7/dist-packages/Gelatin-DEVELOPMENT-py2.7.egg/Gelatin/parser/syntax.ebnf'

Hello I am not able to make the project and run it. I don't know why I am getting this error on my screen:

aims@aims:~/Gelatin$ sudo make install PREFIX=/usr/local
sudo: /etc/sudoers.d is world writable
[sudo] password for aims: 
mkdir -p /usr/local`python -c "import sys; from distutils.sysconfig import get_python_lib; print get_python_lib()[len(sys.prefix):]"`
./version.sh
Version is v1.0.20-g7865dfa
Version file unchanged.
export PYTHONPATH=/usr/local`python -c "import sys; from distutils.sysconfig import get_python_lib; print get_python_lib()[len(sys.prefix):]"`:; \
python setup.py install --prefix /usr/local \
                        --install-scripts /usr/local/bin \
                        --install-lib /usr/local`python -c "import sys; from distutils.sysconfig import get_python_lib; print get_python_lib()[len(sys.prefix):]"`
Traceback (most recent call last):
  File "setup.py", line 16, in <module>
    from setuptools import setup, find_packages
ImportError: No module named setuptools
Makefile:29: recipe for target 'install' failed
make: *** [install] Error 1

Is there any solution for this. Kindly, let me know you response. Thanks.

Hi,
Can this be used to convert running config of a Cisco device to XML?

Thanks in advance.

Hello.
I install Gel on Debian 9 with all depend on py 3.5

root@mirror:/tmp/Gelatin/demo/simple# gel --version
2.2.10
root@mirror:/tmp/Gelatin/demo/simple# 

but when i run convert, have next error

gel -s input1.txt out.txt
Traceback (most recent call last):
  File "/usr/local/bin/gel", line 4, in <module>
    __import__('pkg_resources').run_script('Gelatin==2.2.10', 'gel')
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 739, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1494, in run_script
    exec(code, namespace, namespace)
  File "/usr/local/lib/python3.5/dist-packages/Gelatin-2.2.10-py3.5.egg/EGG-INFO/scripts/gel", line 77, in <module>
    converter = compile(options.syntax)
  File "/usr/local/lib/python3.5/dist-packages/Gelatin-2.2.10-py3.5.egg/Gelatin/util.py", line 53, in compile
    encoding=encoding)
  File "/usr/local/lib/python3.5/dist-packages/Gelatin-2.2.10-py3.5.egg/Gelatin/parser/Parser.py", line 56, in parse
    return self.parse_string(string, compiler)
  File "/usr/local/lib/python3.5/dist-packages/Gelatin-2.2.10-py3.5.egg/Gelatin/parser/Parser.py", line 48, in parse_string
    error(input, end)
  File "/usr/local/lib/python3.5/dist-packages/Gelatin-2.2.10-py3.5.egg/Gelatin/parser/util.py", line 45, in error
    raise Exception(msg)
Exception: Syntax error in line 1:
'User'
 ^

File input1.txt from repo

root@mirror:/tmp/Gelatin/demo/simple# cat input1.txt 
User
----
Name: John, Lastname: Doe
Office: 1st Ave
Birth date: 1978-01-01

User
----
Name: Jane, Lastname: Foo
Office: 2nd Ave
Birth date: 1970-01-01
root@mirror:/tmp/Gelatin/demo/simple# 

On py 2.7 i have this error too.
This is bug or my py version not supported?

Hi,

Is it possible take multiple values for a field and convert it to xml
Eg.,
Name: John
Name: Doe
will beome
John
Doe

Any other ideas how i can provide mutliple values for a field?