ibrewster / p4d Goto Github PK

Hi,

Here is a snippet to show the problem:

cnx = p4d.py4d_connection(...)
sql = 'SELECT * FROM _USER_TABLES;'
cur = cnx.cursor()
cur.execute(sql)
cur.close()

Running this code results in core dump with message "double free or corruption". The double free comes from the close function of py4d_cursor, ligne 218:

        if self.result is not None:
            self.lib4d_sql.fourd_free_result(self.result)
        self.connection.close()

If the cursor have some results, we free it be don't set the result field to None. The close method of the connection will try to free it again. This will fix the problem:

        if self.result is not None:
            self.lib4d_sql.fourd_free_result(self.result)
            self.result = None
        self.connection.close()

Regards,

Olivier

Hi

When a higher offset than the total row count is specified, 4D server returns "Row-Count: <total_row_count>" and "Row-Count-Sent: <total_row_count>". Because of this, lib4d_sql expects data but the 4D server never sends a reply in this case. Therefore, lib4d_sql gets stuck in frecv. Do you have an idea how to prevent / mitigate this? I thought about implementing a timeout, but this is obviously not the best solution.

Due to an apparent bug in the C code or the 4D server itself, only the first page of results can be returned on a multi-page sized query.

A workaround has been added to the code in the form of a "pagesize" parameter that is accessible in the cursor class as well as the C query functions. However, as the result set size may not be known ahead of time, or there may be other reasons to avoid large "page" sizes, this workaround is far from ideal.

I'm trying to read all data of more than 50 tables. It works for many but 12.

If I try one named "Heures" with 74 columns, a segfault occurs. If a limit my select to some columns (eg. 50 first ou 50 last) it works.

Digging into the fetchone method problem comes from lib4d_sql. I think there's a "buffer overflow" some where.

Can you give me some advice to debug lib4d_sql?

Hi,

You are creating a sub-class of str called Binary for BLOB (and IMAGE) data. Using sub-class of str implies "decoding" but this is raw data. Why not using/sub-classing bytes?

When connection error occurs, we only have a exception with Unable to connect to 4D Server. There is an error message in the C library. It can be usefull to change

to:

raise OperationalError("Unable to connect to 4D Server: {0}".format(ffi.string(self.connptr.error_string)))

I'm have a 4D table "Societe" with 66 columns. When reading data, I can SELECT all column except "OB_GestionDossAbsence". When I include this column un SELECT, this error occurs:

...
  File "/volumes/users/olivier/dev/scopuli/scopuli-cli/venv3/lib/python3.6/site-packages/p4d/p4d.py", line 382, in execute
    raise ProgrammingError(ffi.string(self.lib4d_sql.fourd_error(self.fourdconn)))
p4d.p4d.ProgrammingError: b'Failed to execute statement.'

The data type of this column is 21.

How can I have more info on this error?

Hi,

When reading on field of one row over thousand, it raise the following exception:

  ...
  File ".../p4d/p4d/p4d.py", line 600, in __next__
    result = self.fetchone()
  File ".../p4d/p4d/p4d.py", line 505, in fetchone
    raise e
  File ".../p4d/p4d/p4d.py", line 502, in fetchone
    decoded_value = output.decode('UTF-16LE')
  File ".../venv3/lib/python3.5/encodings/utf_16_le.py", line 16, in decode
    return codecs.utf_16_le_decode(input, errors, True)
UnicodeDecodeError: 'utf-16-le' codec can't decode bytes in position 86-87: illegal UTF-16 surrogate

The data to decode:

b'#\x002\x002\x001\x007\x001\x00 \x00C\x00h\x00e\x00c\x00k\x00 \x00J\x00S\x00 \x00P\x00H\x00P\x007\x00 \x00#\x002\x002\x001\x007\x001\x00 \x00C\x00h\x00e\x00c\x00k\x00 \x00J\x00S\x00 \x00P\x00H\x00P\x007\x00\r\x00 #\x8c\xd9\r\x00(\xf7\xa9\x0e\xc8\xd4H\x01'

The data as shown in 4D:

Data seems to be corrupted but do you have an idea to workaround this?

Brief:
An exception is thrown when closing an open connection, I expect the connection to be closed without a problem.

Code like this:

if conn.connected:
    conn.close()

Is throwing a ProgrammingError: b''

The traceback is:

File "p4d/p4d.py", line 651, in close
  self.__private_cursor__.execute("ROLLBACK")
File "p4d/p4d.py", line 307, in execute
  raise ProgrammingError(error)

There aren't any active transactions to be rolled back.

Hello,
I'm a happy user of p4d, and would like to thank you a lot for this very useful library. I use it quite often !

I noticed an bug when sending a request containing accented characters.
The requests always fail with error "Failed to parse statement."

Here is an example of the query failing with the above error:

select count(*) from people where first_name = 'René';

At first I thought it was an issue on 4D side so I tried to work around it.
But after much digging I found it was an issue in p4d.
This is happening in base64.c, in function base64_encode.

unsigned char *base64_encode(const char *str, size_t length, int *ret_length)
{
	const char *current = str;
	unsigned char *p;
	unsigned char *result;

	if (((length + 2) / 3) >= (1 << (sizeof(int) * 8 - 2))) {
		if (ret_length != NULL) {
			*ret_length = 0;
		}
		return NULL;
	}

	result = (unsigned char *)malloc(((length + 2) / 3) * 4* sizeof(char)+1);
	p = result;

	while (length > 2) { /* keep going until we have less than 24 bits */
		*p++ = base64_table[current[0] >> 2];                         // <-- BUG HERE !
	// ...snip...

The last line shown has the bug:
if current[0] is >=128, it is considered negative because it has the type char.
When a negative number is shifted to the right, the most significant bit of the number are set to 1 to preserve the sign. In that case, the computed index is incorrect.
This case does not happen for regular ASCII chars because they have values < 128 but it systematically happens with non ASCII chars.
To fix it, simply declare the variable current as an unsigned char pointer, so it will never be considered negative.
Here is the proposed fix:

unsigned char *base64_encode(const char *str, size_t length, int *ret_length)
{
-	const char *current = str;
+	const unsigned char*current = (const unsigned char*)str;
	unsigned char *p;
	unsigned char *result;

Strangely enough, the original library 4d/SQLlib_4D has the correct type for current...
Would it be possible to issue a new release of p4d with this fix ?

Have a nice day !
-Guillaume

Hi,

Here is a use case:

    sql = 'SELECT * FROM _USER_TABLES;'
    with closing(cnx4d.cursor()) as cur:
        cur.execute(sql)
        rows = cur.fetchall()
        print(len(rows))

    with closing(cnx4d.cursor()) as cur:
        cur.execute(sql)
        rows = cur.fetchall()
        print(len(rows))

The first cursor is okay. When running the query with the second cursor:

InternalError: Database not connected