Giter Club home page Giter Club logo

Comments (5)

thxmike avatar thxmike commented on August 23, 2024

Not to be a pain but is there any update on this item?

from node-datapumps.

novaki avatar novaki commented on August 23, 2024

Maybe the docs are not clear on this, so I'll only close this ticket after updating the docs. .whenFinished() promise is fulfilled when the pump is finished. A pump is considered finished when all items from its input buffer are read and it's output buffer is empty (some other pump read it, or it was not written at all).

Basically, you don't need to write the output buffer of your pump, if there isn't any pump reading from it (.from() method). Otherwise, your output buffer will not become empty and pumping will never stop.

var candidate_map = [];
data_pump
  .mixin(datapumps.mixin.RestMixin)
  .process(function (data) {
    return this.post(target_service_end_point, {
      multipart: false,
      headers: {
        'Content-Type' : 'application/json',
        'X-ApiKey' :  target_service_end_point_shared_secret
      },
      data: record
    }).then(function(response) {
      if(response.statusCode == 201) {
        candidate_map.push({ "source":data.id, "destination":response.result.id });
      }
    }.bind(this))
  })
  .logErrorsToConsole()
  .start()
  .whenFinished().then(function(data){
    console.log("The processing is finished");
    console.log(candidate_map);
  });

from node-datapumps.

thxmike avatar thxmike commented on August 23, 2024

OK, the previous example is incomplete and we do have data going into the pump in a from. Here is a more full example that shows the use of from(). I have removed some sensitive areas of the example, most of this works but we do not get to whenFinished. We can see all the data going through the pump and the execution of the code complete but we never see this line execute console.log("The processing is finished"); :

var datapumps = require('datapumps');
var mssql = require('mssql');
var candidate_map = require('../candidate/candidate_map');
var source_sql_query = '< some sql statement>';

var config = {
    user: source_database_server_user,
    password: source_database_server_password,
    server: source_server, // You can use 'localhost\\instance' to connect to named instance
    database: source_database,
    //stream: true, // You can enable streaming globally
    requestTimeout: source_server_database_query_timeout,
    options: {
        encrypt: true // Use this if you're on Windows Azure
    }
}

var connection = new mssql.Connection(config, function (err) {
    // ... error checks
    if (err) {
        console.log(err);
        return;
    }
    var request = connection.request();
    request.stream = true;
    //Parameter
    //request.input('locationid', mssql.Int, 6);
    var sql_query = source_sql_query;

    request.query(sql_query,
        function (err, recordSet) {
            // ... error checks
            if (err) {
                console.log(err);
                return;
            }
        });

    request.on('row', function(row) {
        data_pump.from().writeAsync(row);

    });

});
// Setup the data pump
var data_pump = new datapumps.Pump().from(new datapumps.Buffer());

//Process the Post operation
data_pump
    .mixin(datapumps.mixin.RestMixin)
    .process(function (data) {
        var record = '<some json looking data';

            return this.post(target_service_end_point, {

            multipart: false,

            headers: {
                'Content-Type' : 'application/json',

                'X-ApiKey' : 
 target_service_end_point_shared_secret
            },

            data: record
        }).then(function(response) {

            console.log(data.Email + '*******************************************');
            console.log(response.result);
            return this.copy(response);
        }.bind(this));

    }).logErrorsToConsole().start()
    .whenFinished().then(function(data){
     console.log("The processing is finished");    
     console.log(data);
});

Perhaps it is a misunderstanding but I am not sure

from node-datapumps.

novaki avatar novaki commented on August 23, 2024

You need to tell when the sql query is done:

    request.on('done', function(row) {
        data_pump.from().seal();
    });

The .seal() method must be called on a buffer when it will receive no more data.

So, a pump is considered finished (i.e. .whenFinished() promise is fulfilled) when its input buffer is sealed and empty and it's output buffer is empty (some other pump read it, or it was not written at all). The pump will automatically seal output buffers when its finished.

from node-datapumps.

thxmike avatar thxmike commented on August 23, 2024

Thank you. I was able to resolve this issue with the recommendation above.

from node-datapumps.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.