seattletestbed / seattlelib_v2 Goto Github PK

The base64 module is not currently being tested for proper behavior on bad input.

We want to convert the openDHTadvertise library to repy so that we can use it within repy code. r2436 is a better place to start from because the XMLRPC conversion is already done.

The imported modules should be moderately straightforward to replace with repy code. The only real addition of functionality is the one call to urllib.urlopen. This needs to know enough HTTP to get the "webpage" that is the server list.

The xmlparse module raises a xmlparse_XMLParseError exception on reading a XML comment. The comment should be ignored instead.

For example, parsing the following XML response fails:

Please add a parameter to each of the two functions exposed by openDHTadvertise.repy to specify the number of concurrent events openDHTadvertise may use.

Since send can block, it probably also should have a timeout.

In response to ticket #255. The functionality to bypass the web interface in communicating with the GENI portal requires the use of a RPC protocol, namely XML-RPC.

Add some test code for what I added in rsa.repy. It can be a separate file

The XML-RPC protocol supports date and time data, but it is not currently supported by xmlrpc_common. Adding support would allow vessels to exchange time information more efficiently.

testbase64 currently outputs nothing on success. It should notify the user that all tests have passed.

tcp_time.repy and its timeserver counterpart need to use advertise.repy, so that both centralized advertise and openDHT advertise methods will be used to ensure users have connectivity to the time_servers.

Explain why the functions are needed...

To prevent namespace collisions, change xmlrpc_common to preface all functions with xmlrpc_common.

The API for openconn doesn't take the timeout kwarg, so it needs to change to comply to the current API.

testxmlrpc_common currently outputs nothing on success. It should notify the user that all tests have passed.

To prevent namespace collisions, change xmlrpc_client to preface all functions with xmlrpc_client.

Check the return values of socket.send() for valid values so as not to go into an infinite loop on receiving bad data.

The nil value type in XML-RPC is an extension to the original protocol. The Python implementation can use it, but not by default. It would be nice to figure out how to handle nil values since functions can and should be able to return nil values.

To fit into the new time.repy framework timefromservice_updatetime() must take a port as a parameter (and not a number of retries) The port does not have to be used, but it will be passed in.

Rather than retrying a connection to the same server in the event of a connection failure another server should be tried instead.

No sleep should be performed in between connection attempts.

The xmlrpc client and server API are not consistent. The xmlrpc client should accept a url as the argument to the constructor.

Explain why the functions are needed...

Base 64 encoding is widely used as a means to encode binary data to send in text format. It is a useful feature in XML-RPC that is easy to implement.

Tom has requested synchronization primitives. I'm starting with semaphores. We'll add other things later.

Several changes need made to update the current methods used for repy programs to get ntp time.

1. Include a new file time_interface.repy
2. Change tcp_time.repy and ntp_time.repy to include time_interface.repy
3. Replace time.repy with a file that is just:
   include ntp_time.repy
   include tcp_time.repy

how to use the new version:

old programs that currently include time.repy will now try to use both the ntp and tcp_time.repy methods to update the time. (this works from python with repyhelper as well as from repy)

If you want to write a program that uses only one method you just include that method (i.e include foo_time.repy)

You do not have to change any other code, the only calls you will ever have to make regradless of what / how many implementations you are using is time_updatetime(port) and time_getttime()

If you call socket.close() more than once, it should not have an error. Right now you see:

Traceback (most recent call last):
File "centralizedputget.py", line 673, in main
File "centralizedputget.py", line 486, in centralizedadvertise_announce
File "centralizedputget.py", line 363, in close
AttributeError: 'NoneType' object has no attribute 'close'

Both the XML-RPC client and server require a method to translate between XML-RPC and Python data types.

xmlrpc_common throws an exception (xmlrpc_common_XMLParseError "No parameters found") when a function is called over XMLRPC with zero parameters; this shouldn't happen.

The xmlrpc_client module is currently tested against http://phpxmlrpc.sourceforge.net/server.php on the examples.getStateName function. Additional unit tests are needed to test for compatibility. When the xmlrpc_server module is completed, the xmlrpc_client module should be tested against it as well.

We need a program that can run to deploy an arbitrary service (and keep the service running) on an arbitrary number of nodes.

This program will use the geni-xmlrpc interface and nmclient to acquire resources, deploy services, check the status of services, and redeploy as needed.

Additionally the program will be extensible, making it easy to add an additional application specific check to ensure that the deployed service is running.

The base64 module uses str.translate() in a less than ideal way. It would be better to implement inline data transformations and to clean up the code.

In the event of a server or network failure timefromservice_updatetime() may never return.

This can be corrected by using timeoutsockets (found in seattlelib/sockettimeout.repy )

The xmlrpc_xml_to_python function is not to be used and needs to be removed.

Provide an easy to use server for RePy programs and/or libraries wishing to expose their functionality over XML-RPC.

Convert the current socket_timeout repy module in the standard library to have the openconn / waitforconn interface. This would be a relatively simple, but very useful abstraction to have. This would also be a good candidate for service composition.

This is inspired by the problems seen in ticket #129.

nmclient_createhandle sets the default socket timeout to 10 seconds.

nmAPI startvessle waits 10 seconds for a vessel to start.

If a vessel "fails to start in a timely manner" an error message indicating the failure is generated by nmAPI, BUT it is more likely that the socket will timeout before the error message is delivered.

This masking of indications can complicate troubleshooting.

We need to look into this issue and know what the reasons for the different timeouts are, and what change history has taken place, prior to fixing this.

See commented BUG just above function time_updatetime(localport) in time.repy

On slow or unreachable servers, the xmlrpc_client module should timeout to prevent possible denial of service attacks (or just slow operations). The module currently will run until Python or Repy kills the program.

The xmlrpc library is failing to close and free file descriptors. This was causing #470. It needs to be fixed so that this doesn't happen even in error cases.

When network communication times out, the error string is empty. We should fix this because this string is visible to Seash and other things.

The test cases for xmlrpc_common.repy in tests/testxmlrpc_common.repy are limited and should be expanded.

There are typos in the base 64 functions.

There is a workaround for Ticket #362 in seattlelib/xmlrpc_client.repy that should be removed when the bug is fixed.

The implementation of XML-RPC requires parsing XML data. Since REPY has no regular expressions, this must be implemented at a low level. In addition, performance may become an issue for medium-sized files.

The signed_data library currently has a signdata method that appends a timestamp, destination, public key, sequence id, and a signature to a given message, using "!" as the token separator. We should have another method in signed_data that simply takes a parameter string (with private and public keys) and returns a signature for that string. This will allow users more freedom in choosing how to compose their messages.
signeddata_create_signature(message_string, public_key, private_key)

During testing, in the execution of testprocess.py an error is thrown in emulcom.py.

The error seems to have to do with name resolution?

Traceback (most recent call last):
  File "./seattle_repy/testprocess.py", line 332, in <module>
    nmclient_rawsay(getmyip(), 1224, "GetVessels")
  File "/home/userdir/seattle_repy/emulcomm.py", line 684, in getmyip
    raise e
socket.gaierror: (-2, 'Name or service not known')

Additional information about the node:
-Version: 0.1h
-OS: fc10
-wireless connection that is definitely live as this is my box and I've got a live ssh connection.
-The computer is behind a router, no firewall.
-Error is not always reproducible, occurs seemingly randomly :). When it does occur, it takes a long time to process the testing script which would suggest some kind of timeout is triggered to prevent a hang.

Also noted to have occurred on the following planet-lab machines:
planet1.ku.edu.tr
planet2.ku.edu.tr

a2b_hex and b2a_hex have already been ported in the process of creating sshkey.repy. More detailed comments and a testbinascii will need to be put together.

parallelize.repy raises and catches ParallelizeException but only defines ParallelizeError. r2495 changed the only usage of this exception that I could see outside of parallelize.repy. Here are the ones in that parallelize.repy that need to be changed:

seattlelib/parallelize.repy:      ParallelizeException is raised if the handle is unrecognized
seattlelib/parallelize.repy:    raise ParallelizeException("Cannot abort the parallel execution of a non-existent handle:"+str(parallelizehandle))
seattlelib/parallelize.repy:      ParallelizeException is raised if the handle is unrecognized
seattlelib/parallelize.repy:    raise ParallelizeException("Cannot get status for the parallel execution of a non-existent handle:"+str(parallelizehandle))
seattlelib/parallelize.repy:      ParallelizeException is raised if the handle is unrecognized
seattlelib/parallelize.repy:    raise ParallelizeException("Cannot get results for the parallel execution of a non-existent handle:"+str(parallelizehandle))
seattlelib/parallelize.repy:      ParallelizeException is raised if there isn't at least one free event.

The constructor for xmlrpc_client_Client needs to check its arguments for validity and the send_request method needs to check that method_name is a string.

openconn should accept None for local port when the user only wants to set the timeout.

Provide a simple fallback that users can use if NTP is blocked from their donated machine.

There is a workaround for Ticket #362 in seattlelib/xmlrpc_client.repy that should be removed when the bug is fixed.