Comments (5)
Not to be a pain but is there any update on this item?
from node-datapumps.
Maybe the docs are not clear on this, so I'll only close this ticket after updating the docs. .whenFinished()
promise is fulfilled when the pump is finished. A pump is considered finished when all items from its input buffer are read and it's output buffer is empty (some other pump read it, or it was not written at all).
Basically, you don't need to write the output buffer of your pump, if there isn't any pump reading from it (.from()
method). Otherwise, your output buffer will not become empty and pumping will never stop.
var candidate_map = [];
data_pump
.mixin(datapumps.mixin.RestMixin)
.process(function (data) {
return this.post(target_service_end_point, {
multipart: false,
headers: {
'Content-Type' : 'application/json',
'X-ApiKey' :
target_service_end_point_shared_secret
},
data: record
}).then(function(response) {
if(response.statusCode == 201) {
candidate_map.push({ "source":data.id, "destination":response.result.id });
}
}.bind(this))
})
.logErrorsToConsole()
.start()
.whenFinished().then(function(data){
console.log("The processing is finished");
console.log(candidate_map);
});
from node-datapumps.
OK, the previous example is incomplete and we do have data going into the pump in a from. Here is a more full example that shows the use of from(). I have removed some sensitive areas of the example, most of this works but we do not get to whenFinished. We can see all the data going through the pump and the execution of the code complete but we never see this line execute console.log("The processing is finished"); :
var datapumps = require('datapumps');
var mssql = require('mssql');
var candidate_map = require('../candidate/candidate_map');
var source_sql_query = '< some sql statement>';
var config = {
user: source_database_server_user,
password: source_database_server_password,
server: source_server, // You can use 'localhost\\instance' to connect to named instance
database: source_database,
//stream: true, // You can enable streaming globally
requestTimeout: source_server_database_query_timeout,
options: {
encrypt: true // Use this if you're on Windows Azure
}
}
var connection = new mssql.Connection(config, function (err) {
// ... error checks
if (err) {
console.log(err);
return;
}
var request = connection.request();
request.stream = true;
//Parameter
//request.input('locationid', mssql.Int, 6);
var sql_query = source_sql_query;
request.query(sql_query,
function (err, recordSet) {
// ... error checks
if (err) {
console.log(err);
return;
}
});
request.on('row', function(row) {
data_pump.from().writeAsync(row);
});
});
// Setup the data pump
var data_pump = new datapumps.Pump().from(new datapumps.Buffer());
//Process the Post operation
data_pump
.mixin(datapumps.mixin.RestMixin)
.process(function (data) {
var record = '<some json looking data';
return this.post(target_service_end_point, {
multipart: false,
headers: {
'Content-Type' : 'application/json',
'X-ApiKey' :
target_service_end_point_shared_secret
},
data: record
}).then(function(response) {
console.log(data.Email + '*******************************************');
console.log(response.result);
return this.copy(response);
}.bind(this));
}).logErrorsToConsole().start()
.whenFinished().then(function(data){
console.log("The processing is finished");
console.log(data);
});
Perhaps it is a misunderstanding but I am not sure
from node-datapumps.
You need to tell when the sql query is done:
request.on('done', function(row) {
data_pump.from().seal();
});
The .seal()
method must be called on a buffer when it will receive no more data.
So, a pump is considered finished (i.e. .whenFinished()
promise is fulfilled) when its input buffer is sealed and empty and it's output buffer is empty (some other pump read it, or it was not written at all). The pump will automatically seal output buffers when its finished.
from node-datapumps.
Thank you. I was able to resolve this issue with the recommendation above.
from node-datapumps.
Related Issues (20)
- Default functionality / Run
- Run / Custom Output buffer
- Run/start HOT 1
- .to() function
- write 2 different files (question) HOT 9
- Using the group method on MongoDbMixin HOT 3
- Upgrade the mongdb driver on the MongoDbMixin HOT 1
- Copying data between two mongo servers HOT 2
- Getting "Cannot change source buffer after pumping has been started" all of a sudden HOT 6
- ExcelWriterMixin - Streaming / memory handling for very large files HOT 2
- MongoDB mixin: add `aggregate` to available methods HOT 2
- How to do multiple processBatch commands? HOT 2
- Medium severity vulnerability affecting this repo
- Error while insert item in MySql HOT 1
- Read from 2 collections and write HOT 3
- NumberLong can not be exported HOT 1
- PostgresqlMixin Process not called
- CsvWriter Mixin "TypeError: May not write null values to stream" HOT 1
- Apply sort on mongodb
- MySQL to Postgres - complex transformations HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from node-datapumps.