Optimize the code for converting json to dataset about h54s HOT 7 CLOSED

Hi @rohnny - we agree! If you checkout our development branch you will find a revised version that is 10,000 times faster, using the file upload mechanism (rather than creating macro variables).

You will need 9.4 to run it as it uses proc json. The macro calls are also different (eg it uses bafgetdatasets() to retrieve all datasets). example syntax:

/* get data */
%bafgetdatasets()

/* send data back */
%bafheader()
    %bafOutDataset(sasMembers, work, sasMembers)
%bafFooter()

from h54s.

also, if you are using the adapter to load data into SAS you might want to check out http://datacontroller.io - it's free for up to 10 users.

from h54s.

@allanbowe Thanks for making me aware of this! My client runs 9.4 so should be fine. I will check this out in the next coming days.

from h54s.

Hi @allanbowe, I did a checkout of the development branch and I cannot see any reference to bafgetdatasets as you mention? Also looks like the build is in the middle between grunt and gulp - what is the status of this? jshint is failing. Something I can do to help? :) Do you have any documentation regarding how to switch over to use proc json for receiving json?

from h54s.

Hi @rohnny - we did some work on this today, if you git pull the dev branch you should now be able to run gulp and gulp release however - this is only necessary if you plan to contribute to the dev build.

For general adapter usage just take the h54s.js from the dist folder, and the new h54s.sas from the sasautos directory. bafgetdatasets() is here and is now documented in the README.md.

We're glad you've raised this issue as it's triggered us to finally get (what we call) "version 1" of the adapter into master! The PR is here.

from h54s.

@rohnny For the v1 release @allanbowe is talking about, we've moved off JSON and now construct a csv payload that's presented to SAS as a file upload. Check this comment for an indication of the performance uplift.

We've been testing this release quite extensively for the last couple of years, and it's rock solid now. Just needs to be documented sufficiently and 'released'.

from h54s.

@rohnny this issue was resolved as of the v1.0 release of h54s. We've just released version 2.0 if you want to check it out.

Closing this issue.

from h54s.