ibm / dp-file-uploader Goto Github PK

View Code? Open in Web Editor NEW

4.0 6.0 6.0 30 KB

Tool for easily uploading files to an IBM DataPower® Gateway file system.

License: MIT License

Python 100.00%

ibmcode python datapower automation tooling requests

dp-file-uploader's Introduction

dp-file-uploader

Tool for easily uploading files to an IBM DataPower® Gateway file system.

Pre-requisites

XML Management Interface

Since this script is designed to upload files via the XML management interface the xml-mgmt object must be enabled and up in the default domain of the gateway you wish to target. You can validate this easily by logging into the CLI of the gateway and checking as follows:

idg# show xml-mgmt

xml-mgmt [up]
--------
 admin-state enabled
 ip-address 0.0.0.0
 port 5550
 acl xml-mgmt  [up]
 slm-peering 10 Seconds
 mode any+soma+v2004+amp+slm+wsrr-subscription
 ssl-config-type server

Installation

Clone or download the repository from the Releases page

$ git clone [email protected]:IBM/dp-file-uploader.git
$ cd dp-file-uploader/

Install package via pip3
```
$ pip3 install .
```
Note: Installing via pip3 adds the dp-file-uploader executable to your PATH.
Validate the installation
```
$ dp-file-uploader --version
```

Usage

This script can be used to upload a single file, or multiple files, to a target IBM DataPower® Gateway filesystem. You control the behavior of the script through command-line arguments. The minimum usage would be as follows:

$ dp-file-uploader my.datapower.com "local:///sandbox/" file.txt

This would upload the file.txt file to the IBM DataPower® Gateway at hostname (my.datapower.com) and store the file within the local:///sandbox/ directory.

Since no other arguments were provided, some defaults were used:

user defaults to admin
password defaults to admin
port defaults to 5550

You can specify each of these via command-line argument. For example:

$ dp-file-uploader \
    --user "myaccount" \
    --password "mypassword" \
    --port 9550 \
    my.datapower.com "local:///sandbox/" file.txt

You can also upload multiple files at once, using either wildcards or specifying multiple names manually.

# using wildcards
$ dp-file-uploader my.datapower.com "local:///sandbox/" file*.txt

# specifying each manually
$ dp-file-uploader my.datapower.com "local:///sandbox/" file1.txt file2.txt file3.txt

Troubleshooting

You can enable verbose output via the -V, --verbose command line argument to get a little more detail from the script as it runs. If this does not help to solve your problem, please feel free to open an issue.

dp-file-uploader's People

Contributors

Stargazers

Watchers

Forkers

bhaskers-blu-org1 greenmountainkate test-jenkins101 ghas-results

dp-file-uploader's Issues

DP file upload gets error: FileNotFoundError: [Errno 2] No such file or directory

Can you help me what I am missing?

Below is the full error:

C:\Users\200634\Downloads>dp-file-uploader --user "admin" --password "admin" --port 5550 mgmt-kxesadpwd1.isbank "local:///DEV/" file1.txt file2.txt -V
Namespace(hostname=['mgmt-kxesadpwd1.isbank'], directory=['local:///DEV/'], port=[5550], user=['admin'], password=['admin'], domain=None, verbose=True, fileName=['file1.txt', 'file2.txt'])
URL: https://mgmt-kxesadpwd1.isbank:5550/
Preparing to send file: file1.txt
Building XML request
File target: local:///DEV/file1.txt
Generated XML:
XML string length = 265

<env:Envelope xmlns:env="http://schemas.xmlsoap.org/soap/envelope/">
env:Body
<dp:request xmlns:dp="http://www.datapower.com/schemas/management">
<dp:set-file name="local:///DEV/file1.txt">ZGVuZW1lIDENCmRlbmVtZSAx</dp:set-file>
</dp:request>
</env:Body>
</env:Envelope>

Sending POST request
Traceback (most recent call last):
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 382, in _make_request
self._validate_conn(conn)
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 1010, in validate_conn
conn.connect()
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connection.py", line 416, in connect
self.sock = ssl_wrap_socket(
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\ssl.py", line 449, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\ssl.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 512, in wrap_socket
return self.sslsocket_class._create(
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 1070, in _create
self.do_handshake()
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 1341, in do_handshake
self._sslobj.do_handshake()
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\adapters.py", line 440, in send
resp = conn.urlopen(
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\retry.py", line 532, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\packages\six.py", line 769, in reraise
raise value.with_traceback(tb)
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 382, in _make_request
self._validate_conn(conn)
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 1010, in validate_conn
conn.connect()
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connection.py", line 416, in connect
self.sock = ssl_wrap_socket(
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\ssl.py", line 449, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\ssl.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 512, in wrap_socket
return self.sslsocket_class._create(
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 1070, in _create
self.do_handshake()
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 1341, in do_handshake
self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\Scripts\dp-file-uploader-script.py", line 33, in
sys.exit(load_entry_point('dp-file-uploader==0.1.4', 'console_scripts', 'dp-file-uploader')())
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\site-packages\dp_file_uploader\command_line.py", line 34, in main
dp_file_uploader.run_with_args(args)
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\site-packages\dp_file_uploader_init_.py", line 139, in run_with_args
process_file(filename, directory, url, user, password)
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\site-packages\dp_file_uploader_init_.py", line 110, in process_file
r = requests.post(url, auth=(user, password), data=xml, headers=headers, verify=False)
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\api.py", line 117, in post
return request('post', url, data=data, json=json, **kwargs)
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\sessions.py", line 529, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\sessions.py", line 645, in send
r = adapter.send(request, **kwargs)
File "C:\Users\200634\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\adapters.py", line 501, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

C:\Users\200634\Downloads>

Optionally support REST Management Interface

It may be useful to support the REST management interface in addition to the XML Management Interface. Could specify which to use via command line argument, defaulting to xml-mgmt (current behavior).

add HTTP header logging for verbose logging

Trying to upload 1Gb firmware file ends in erorr 500

I'm trying to upload a large firmware file to the DP appliance. After about 20-30 min, the program aborts with the following error:

[..]
</dp:set-file>
</dp:request>
</env:Body>
</env:Envelope>

Sending POST request
HTTP Response Code: 500
Response XML:

<env:Envelope xmlns:env="http://schemas.xmlsoap.org/soap/envelope/">
env:Body
env:Fault
env:Client
Malformed content (from client)
</env:Fault>
</env:Body>
</env:Envelope>

Is there a file transfer limit? The file I am uploading ends with .scrypt3 extension, which is the standard (binary I believe) format of IBM's firmware updates. This particular file is aprox 970MB in size.

verbose logging hangs when printing large file

Example of trying to log the XML request (when verbose logging is enabled) for a file that is very large (in this case a firmware image that's ~900 MB).

Generated XML:

^CTraceback (most recent call last):
  File "/usr/local/bin/dp-file-uploader", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/dp_file_uploader/command_line.py", line 29, in main
    dp_file_uploader.run_with_args(args)
  File "/usr/local/lib/python3.7/site-packages/dp_file_uploader/__init__.py", line 113, in run_with_args
    process_file(filename, directory, url, user, password)
  File "/usr/local/lib/python3.7/site-packages/dp_file_uploader/__init__.py", line 84, in process_file
    xml = build_xml(directory, filename, data)
  File "/usr/local/lib/python3.7/site-packages/dp_file_uploader/__init__.py", line 76, in build_xml
    print_pretty_xml(xmlRequest)
  File "/usr/local/lib/python3.7/site-packages/dp_file_uploader/__init__.py", line 54, in print_pretty_xml
    dom = xml.dom.minidom.parseString(xml_str)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xml/dom/minidom.py", line 1968, in parseString
    return expatbuilder.parseString(string)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xml/dom/expatbuilder.py", line 925, in parseString
    return builder.parseString(string)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xml/dom/expatbuilder.py", line 223, in parseString
    parser.Parse(string, True)
  File "/private/tmp/python-20190709-71298-1cfqdlu/Python-3.7.4/Modules/pyexpat.c", line 282, in CharacterData
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xml/dom/expatbuilder.py", line 285, in character_data_handler_cdata
    value = node.data + data
KeyboardInterrupt

relative file path results in absolute path appended to target

Example:

dp-file-uploader -V my.datapower.com "local:///" ~/firmware/img/idg2018418.scrypt4

Results in:

Preparing to send file: /Users/aharbis/firmware/img/idg2018418.scrypt4
Building XML request
File target: local:////Users/aharbis/firmware/img/idg2018418.scrypt4

version.py sourced via relative file path, breaks when running outside repo dir

Traceback (most recent call last):
  File "/usr/local/bin/dp-file-uploader", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/dp_file_uploader/command_line.py", line 28, in main
    args = process_args()
  File "/usr/local/lib/python3.7/site-packages/dp_file_uploader/command_line.py", line 21, in process_args
    argparser.add_argument('-v', '--version', action='version', version=get_version())
  File "/usr/local/lib/python3.7/site-packages/dp_file_uploader/command_line.py", line 7, in get_version
    with open('./dp_file_uploader/version.py') as fp:
FileNotFoundError: [Errno 2] No such file or directory: './dp_file_uploader/version.py'

No domain informed

Hi @aharbis
I see that on the SOMA request there is no domain info, and so, files are pushed to 'default' domain.
In some scenarios (specially in test envs.) users are allowed to upload files only to their own domains, and so, in the way the tool is designed it will certainly fail into an auth. error.
I would suggest adding a command line parameter to get the domain name, and, if omitted, uses the 'default' domain.

Content-Type not set on request

Logs on DataPower show that a Content-Type is not specified by the client (this tool):

20200103T164828.043Z [0x80c00004][multistep][debug] xmlfirewall(xml-mgmt): tid(161)[request][x.x.x.x] gtid(b8def43f5e0f705c000000a1): Protocol layer did not supply content-type