boemska / h54s Goto Github PK

In corporate environments using IE the following tag may be necessary:

<meta http-equiv="x-ua-compatible" content="ie=edge">

This should be part of the readme

And return more meaningful error messages, maybe abort the stored process rather than erroring.

It didn't occur to me that not everyone wants to clone them along with the code

As it stands our POST requests are being sent as application/x-www-form-urlencoded. There seems to be a bug in the 9.4 STP webapp where it takes over a minute, sometimes even ten minutes, to deserialise the POST body into params / activemq messages. This also seems to depend on the length of the longest parameter - so one single 32,000 parameter takes exactly 70 seconds every time, two 16,000 parameters (so same amount of data) would take 35 seconds, 6 lots of 5,000 would take 12-13 secs.

The fix is to upload the POST body as multipart/form-data, which seems to work much better.

Example of it taking 10 mins to pass params from JVM to multibridge stp runner:

line 566 of h54s.sas wipes all the formats (providing a consistent numerical value to the js side).

data _null_;
  file &h54starget.;
  set tempOutputView end=lastrec;
  format _all_;
  ....

The issue is that the default best12. format is applied - resulting in a loss of precision (as can be evidenced by running the code below).

data _null_;
  x=3815842.11116;
  put x=;
run;

At my client (a large UK bank) we have been using h54s very successfully except for one issue we have with Internet Explorer when we call the adapter. The adapter.call is done and the timings from the network tab in the IE tools show that it waited for about 40 secs, and shows that Result as (Aborted). Occasionally this will work properly and make the call. But usually we get the abort, and our webapp gets stuck. Please provide some kind of fix. Looking at stack overflow I could see some other people having problems with IE and POST requests like this suddenly aborting for no apparent reason. So it seems more of an IE issue than a h54s issue.

The Stored Procedures that are required for the example applications to work are not included as part of the repository rendering the examples from a "seeing an functioning fully" perspective useless.

https://github.com/Boemska/h54s/blob/master/dist/h54s.js#L53

This is causing (benign) JVM warnings on any version that is not 9.3. We should remove it from the default config and test

The following attributes would be nice to surface within NPM:

• repostory
• bugs
• homepage
• keywords

This is an extension of issue #41

At the moment, SAS returns a status from the adapter, which was a placeholder that we were always going to standardise.

So how about this:

a "status": "success" is the standard response for a successful execution of whatever it is that SAS was doing. A status of anything else (an example of which is the existing "status" : "inputTableNotFound" returned by %hfsErrorCheck and already outlined in the README), should be considered an error and throw an err object that is already documented in the readme.

For this type of SAS-thrown backend error we should implement a new err.type of programError, and the err.type = 'programError' object should then include two new properties: err.status and err.message.

I'll document the mechanisms to implement this error throwing from the SAS side separately.

I have followed the instructions how to setup h54s, but cannot get authentication to work. I tried to use the h54s-angular-seed application, but after entering username / pass (in the popup), I end up in a redirect loop (status code 302). Creating a small app where I manually call adapter.login(..) I get the same.

Really cant figure out why. Currently using SAS 9.4 M3. Opening the stored process (via sas portal) using h54s works fine. Using Chrome.

Any help would be appreciated.

#29 partly depends on this. Paths for test files should be configurable via h54sconfig.json.

I'm seeing instances where ajaxTimeout: 0 is basically timing out immediately. I don't know if this is related to the fact I am calling multiple instances of the adaptor.. But setting the value back (eg to 60000) prevents the following in the console:

TypeError: res is undefined

adding my .js file and my copy of the adaptor for info.
h54s.js.txt
myJS.txt

Our tables object (add) should throw an exception if macro name has numeric value at the end.
Example:

var table = new h54s.Tables([], 'data1');

XHR retry requests need a property of 'programName' so that the program which failed/was retried can be noted in an Application Log if required

For STDOUT output when doing integration tests mostly

This breaks the tests for any data volumes that require chunking.

Ideally let's use a prompt for chunk size performance testing.

Instead of hardcoding the _sasapp parameter for the SASLogon post submit, it needs to be attained from the initial SASLogon redirect URL, as follows:

Request URL:http://server57562.uk2net.com/SASLogon/index.jsp?_sasapp=Stored+Process+Web+App+9.3&

Currently it is hardcoded to 9.3, but this version will change with the target SAS version

dictionary tables performs better on dedicated stp sessions but isn't as good when a session has (pre)assigned DBMS libs

At line 195 in the README.md there's a minor typo I suppose, DHS() should be DHMS()?
For the rest it is a great project! I'm just starting to experiment with the code.

The current version of the adaptor will attempt to convert any variables prefixed with DT_ to a javascript datetime (and vice versa when coming back to SAS).

This can feasibly cause issues / require workarounds should such variables be passed without the intention to translate them.

A mechanism should exist to allow datetime variables to be identified separately from the variable name itself.

The print output is breakingee the integration tests when outputting to stdout

Currently we're loading all macro params into a single-variable work dataset and using prx to work through one 32k chunk at a time. DS2 might give us a way of parallelising the processing of multiple frames when datasets exceed the 32k limit.

Also look at JSON package in 9.4m3

Pretty sure it's down to this:

It works fine when there are no multibyte characters involved. This should also solve the PROC JSON bug we've been seeing with the utf8 tests, as they've been failing when the last character has been a triple byte char split truncated by 2 bytes.

This substitution for " => "" should be global when escaping double quotes

Is it possible to specify somehow what encoding to use when serializing json? I mean inside config of h54s.
I have run into issue with using norwegian letters in javascript, and trying to send that over to sas.

This is not handled well by the parser yet and causes an infinite loop. Shouldn't be sent back.

Use feature branch until we have a working system.

Need to make sure that the numbers the adapter is sending back to SAS fall within the same maximum integer / precision limits imposed by SAS.

JS ref: http://stackoverflow.com/questions/307179/what-is-javascripts-highest-integer-value-that-a-number-can-go-to-without-losin

sas ref: https://v8doc.sas.com/sashtml/win/numvar.htm

Maybe use toPrecision() js method, http://www.w3schools.com/jsref/jsref_toprecision.asp

Possibly add a prxparse regex that validates the input json objects

We need conditional logic to replace a blank numeric with a period (.) in the adapter

A period should be recognised by javascript as a valid numeric character when the spec cell type is numeric.

Also, additional debug information should be thrown.

The replace here should also be global (and the typo needs to be corrected)

We also need to consider whether we need this, or if it is better to allow developers to explicitly stringify objects and have them stored as strings, as we do now where this is uesd. Either is doable but we need a solid mechanism for identifying these on the way back from SAS (eg. a column name prefix), same as we do with the dates.

Sent from SAS:

data sasparams;
  USER="Sugarhill Gang";
run;
%hfsHeader;
%hfsOutDataset(jsParams,WORK, sasparams);
%hfsFooter;

Received by browser:

So maybe USER is a reserved variable name in H54s.. If so should probably be documented somewhere? Ideally though this variable name would be available to developers, in order to avoid unexpected behaviour.

The following attributes would be helpful in the h54s response:

• &syssite (site id)
• &syshostname (server name)
• &sysscp (operating system)
• &sysvlong (full version including release number)

The adapter provides a neat way to surface messages to the user (usermessage) or specific runtime issues (logmessage). It would be nice if a similar mechanism existed for returning an error message (eg errormessage). This would allow developers to use SAS side logic to gracefully surface a user friendly error message back to the front end - such as user Jim not found or unable to access xyz.

Maybe we should add timeout in h54s config. It's hardcoded to 30 seconds.

The code for converting json to sas dataset seems to be slow, and should be looked closer at. This is the macro named hfsGetDataset. A dataset with around 4000 rows and 9 columns take around 10 sec to process.

Currently the SAS Erros array fills up with each occurrence of an error pulled out of the SAS log, with no context. What we need it to do is populate the ERRORs array in the same way that the Debug log array is handled.

So:

• When an error occurs in non-debug mode (an error being a json parsing failure), the entire HTML body that is returned should be pushed onto the SAS Errors Array.
• When in Debug mode, all logs can continue to go into debug mode using the existing mechanisms.

Less to do when registering stored processes.

%LET _RESULT=STREAM;

Need to add upcase function to left side of where clause when querying DICTIONARY.COLUMNS for memname/dsn.

This only applies when the libname option PRESERVE_COL_NAMES is used (default for SAS generated libnames), however since we cannot easily determine if this option has been set, the upcase solves the problem. i.e.:

CREATE TABLE tempcols as 
  SELECT upcase(name) as name, 
         type, 
         length 
  FROM   dictionary.columns 
  WHERE  upcase(memname) = upcase("&dsn.") 
         AND libname = upcase("&libn.");  

We don't need url encoded strings, only some special characters escaped

When a response to a request to the SASStoredProcess webapp includes the following string, regardless of debug/nondebug mode:

<h2>Stored process not found:

The adapter should throw a new class of error. The condition here is that the program has not been found, or more likely, access to it has been denied to the user in question.

The error message should include the following:

You have not been granted permission to perform this action, or the STP is missing.

This issue was discovered in a Windows 9.3 environment, and is tricky to replicate from a code sample and so a dataset is attached.
endash.zip

When returning via the adapter, the json appears valid:

{ "columnspec" : [ {"NAME":"TESTVAR", "TYPE":2, "LENGTH":2000, "LABEL":" "} ], "tabledata" : [ {"TESTVAR":"ITCO%20%96%20Data"} ], "usermessage" : "blank", "logmessage" : "blank", "requestingUser" : "Allan Bowe", "requestingPerson" : "t'other Allan Bowe", "executingPid" : 3968, "sasDatetime" : 1790069921 , "status" : "success"}

But the adapter returns an error:

Object { message: "malformed URI sequence", type: "unknownError", stack: "[7]</module.exports.unescapeValues@…" }

As per http://www.w3schools.com/tags/ref_urlencode.asp, if the UTF-8 equivalent (%E2%80%93) is returned, then the adapter works fine.

@boemska-nik has already identified a potential solution (which works fine for me) which is to apply the following option in the relevant sas session:

options URLENCODING=UTF8;

SAS 9.3 is outputting error messages event if debug is not set, but 9.4 is not with both debug settings.
You can see error messages in 9.4 starting with "ERROR" colored in red.
Can you investigate this?

The presence of particular macro variables (type and name) can cause the adapter to fail due to incorrect use of triple ampersands (&&&) - eg &&&type&colNo. In these cases, a double ampersand is needed to cause the desired resolution.