datafire / datafire Goto Github PK
View Code? Open in Web Editor NEWA framework for building integrations and APIs
Home Page: https://datafire.io
License: MIT License
A framework for building integrations and APIs
Home Page: https://datafire.io
License: MIT License
Hi, is this project no longer in development? there's no update since mid last year. Can anyone update the project status?
thx
Based on the documentation of the request library, you're incorrectly always defaulting to formData
when form
should be used for the content type application/x-www-form-urlencoded
.
https://www.npmjs.com/package/request#forms
https://github.com/DataFire/DataFire/blob/master/src/lib/openapi-action.js#L62
Given this schema:
"/users/login": {
"post": {
"responses": {
"200": {
"description": "Successful authentication",
"schema": {
"items": {
"properties": {
"token": {
"type": "string"
},
"id": {
"type": "string"
},
"tokenExpires": {
"type": "string"
}
}
}
}
},
"400": {
"description": "Error in authentication\n",
"schema": {
"items": {
"properties": {
"reason": {
"type": "string"
},
"error": {
"type": "number"
}
}
}
}
},
"default": {
"description": "Error in authentication\n"
}
},
"parameters": [
{
"required": true,
"type": "string",
"description": "Your username\n",
"in": "formData",
"name": "username"
},
{
"in": "formData",
"description": "Your password\n",
"format": "password",
"required": true,
"type": "string",
"name": "password"
}
],
"tags": [
"Login"
],
"operationId": "login",
"consumes": [
"application/x-www-form-urlencoded",
"application/json"
],
"summary": "Login with REST API"
}
}
}
You can see it clearly states that it only allows 2 data forms. It cannot accept multipart/form-data
, which is what formData
is used for in request
.
The docs for Swagger 2.0 also show an example that this is the way the swagger spec expects it to be: https://swagger.io/docs/specification/2-0/describing-parameters/#form-parameters
This means that the code should be checking for what the API endpoint consumes and then changing the usage of form
or formData
accordingly.
Hi,
The link #CustomIntegrationModal seems to be hidden on the Integrations page. it appears for a half a second (see image) but is hidden with CSS. I changed the CSS to:
[hidden] {
display: inline-block;
}
and the modal popped up - I assume we are not supposed to be using it at this time? (as it is clearly intentionally hidden)
Update: NB: I want to have a try at adding WooCommerce API.
e.g.
{
"page": 3,
"total": 5034,
"items": ["..."]
}
Should be able to detect new/changed items automatically by providing a local file
or a database where changes can be tracked.
datafire.monitor('issues', {
do: github.get('/user/issues'),
check: item, cb => {
cb(null, require('issues.json').indexOf(item.id) === -1)
},
stash: item, cb => {
let issues = require('issues.json')
issues.push(item.id)
fs.writeFile('issues.json', JSON.stringify(issues), cb)
}
})
Not sure if I can explain this right, much less figure it out after a day of banging away but I have an express boilerplate I've been hacking on for a few weeks and don't want to give it up but really, really like your platform and integrations so hoping to get a positive answer...
The stack is here: https://github.com/kevoj/nodetomic-api-swagger and it uses the /api' path. I've tried setting up Datafire on the '/api' path and also '/v1' and also '/df' without success. I can get it to register without any errors (I think is is registering...) but just cannot get any paths or actions to fire. Can I use Datafire in conjunction with an express server already set up for apis on another path than /api so I can add some integrations?
I currently have an issue where I need to ensure secure connections are made with no lower than TLS v1.2.
This can be set using the agentOptions parameter when making a request:
request.get({
url: 'https://api.some-server.com/',
agentOptions: {
secureProtocol: 'TLSv1_2_method'
}
});
The ability to set request options also has the added benefit of being able to add a certificate and key to make a connection. Not sure what the best way would be to implement this as certificates will vary across services.
Being able to set it for each integration would be best so something like this would be ideal:
var hackerNews = require('@datafire/hacker_news').create();
hackerNews.requestOptions({
agentOptions: {
secureProtocol: 'TLSv1_2_method'
}
});
OR
var hackerNews = require('@datafire/hacker_news').create();
hackerNews.requestOptions({
agentOptions: {
cert: fs.readFileSync(certFile),
key: fs.readFileSync(keyFile),
passphrase: 'password',
securityOptions: 'SSL_OP_NO_SSLv3'
}
});
Thoughts?
Currently hard-coded as 3333
I know this question will sound stupid, but I am developing a DataFire.Action using VSCode. How do I properly debug the handler? Thanks in advance.
Just need to resolve $refs from external integrations
e.g. zoomconnect expects both "user" and "token" headers
this code in file bin/datafire
#!/usr/bin/env node
if (process.version > 'v6') {
module.exports = require('../src/cmd');
} else {
module.exports = require('../distribution/cmd');
}
if process.version = v10
then v10 > v6 gives false . We need to fix it.
Just an idea:
# DataFire.yml
paths:
/hello:
get:
action: ./function/hello
cache: 5
Many times i find myself caching endpoint responses to prevent expense async calls / speed up responses.
I usually use this middleware module cache.js
:
var mcache = require('memory-cache');
// usage:
// var cache = require('cache')
// app.get("/foo/bar*", cache(5) ) // cache 5 seconds
module.exports = (duration) => {
return (req, res, next) => {
let key = JSON.stringify({method:req.method, url:req.url, query:req.query, body:req.body, headers:req.headers })
let cachedBody = mcache.get(key)
if (cachedBody) {
res.header(cachedBody.headers)
res.send(cachedBody.body)
return
} else {
res.sendResponse = res.send
res.send = (body) => {
var obj = {body:body, headers:res.header()._headers}
mcache.put(key, obj, duration * 1000);
res.sendResponse(body)
}
next()
}
}
}
Is this something which might be interesting to include in the datafire module?
It might also give the public datafire.io-server some headroom.
cheers
I'm getting an issue with a schema that is valid on the latest version of ajv but fails to compile on the version bundled to datafire. It looks like datafire is using ~4 and the latest is on ~6. Is there a reason this wasn't already updated?
Hi,
Is there a codesnippet to embed datafire as middleware for express?
Or the otherway around?..i really would like to add extra middleware before the datafire-middleware kicks in.
This is how i launch it now on my server:
let YAML = require('yamljs');
let path = require('path');
let fs = require('fs');
let datafire = require('datafire/src/lib');
args = {}
args.port = process.env.PORT || 3000;
args.directory = process.env.DATAFIRE_DIR || process.cwd();
let project = datafire.Project.fromDirectory(args.directory);
project.serve(args);
I tried hacking my middleware into express inside a datafire function, but without succes:
var app = require('express')()
app.use( function(req,res,next){
console.log("unfortunately im not called")
next()
})
// hack: cut the line in the middleware-chain
app._router.stack.unshift( app._router.stack.pop() )
module.exports = {
handler: (input) => { .... }
}
However, I would love to add some express middleware like so:
let project = datafire.Project.fromDirectory(args.directory);
project.middleware([
(req,res,next) => { console.log("i am custom middleware, ran before datafire endpoints") }
])
project.serve(args);
That way we can re-use existing middleware components from npm.
I know its a niche feature, and not really the point of FaaS, but it would be great to have some analytics on the side like express-status-monitor etc.
Any thoughts on this?
✗ High severity vuln found in [email protected], introduced via @datafire/[email protected]
Description: Prototype Override Protection Bypass
Info: https://snyk.io/vuln/npm:qs:20170213
From: @datafire/[email protected] > [email protected] > [email protected] > [email protected] > [email protected]
for the purpose of taking that JSON dump and rendering it with a static site generator?
Project Server
DataFire listening on port 3333
✓ should respect top-level cache option (130ms)
✓ should respect cache override for path (230ms)
✓ should respet cache disable for path
Version...
Integrate...
npm ERR! Test failed. See above for more details.
We should have a way to maintain state each time an action is run. For instance, a task might crawl a different page of a result set each time it runs, keeping track of state.pageNumber
.
It would be easy to keep a small amount of state in-memory, but if the process is restarted, we'll often want to pick up where we left off.
I think loadState
and saveState
should be optional, asynchronous functions that can be specified by the user. They can write to a local file, a remote database, an in-memory object, or whatever else. They should be able to be specified once for the project as a whole, and overridden for individual actions (maybe inside context
?)
Hi,
I tried to follow the hello world example, but I got some problems with it.
module.exports = new datafire.Action({
handler: (input, context) => {
if (input.name === 'Voldemort') {
return new datafire.Response({
statusCode: 401,
headers: {'Content-Type': 'text/html'},
body: "<h1>Nope.</h1>",
});
} else {
return {context, input};
}
}
})
In this example the input is always empty. If I check the output, I got the following result.
{
"context": {
"results": {},
"type": "http",
"accounts": {},
"startTime": "2017-06-08T08:20:04.600Z",
"request": {
"query": {
"name": "world"
},
"headers": {
"host": "localhost:3000",
"user-agent": "curl/7.49.1",
"accept": "*/*"
},
"body": {},
"path": "/hello",
"method": "GET"
}
},
"input": {}
}
node: v6.10.3
npm: 5.0.3
datafire: v2.2.0
[sqz@localhost/tmp] $ node -v
v6.9.1
[sqz@localhost/tmp] $ datafire integrate --openapi http://petstore.swagger.io/v2/swagger.json
Created integration petstore_swagger in ./integrations/petstore_swagger
path.js:7
throw new TypeError('Path must be a string. Received ' + inspect(path));
^
TypeError: Path must be a string. Received undefined
at assertPath (path.js:7:11)
at Object.join (path.js:1211:7)
at callback (/tmp/node_modules/datafire/src/commands/integrate.js:55:33)
at fs.writeFile.e (/tmp/node_modules/datafire/src/commands/integrate.js:90:11)
at /tmp/node_modules/graceful-fs/graceful-fs.js:43:10
at FSReqWrap.oncomplete (fs.js:123:15)
Is there an API for DataFire itself so CRUD operations can be performed against integrations, actions, triggers, etc? I know there's DataFire.io but say I wanted to manage these points with another interface, is there an established way of doing that?
Hopefully I didn't miss this somewhere obvious. Thanks!
I love how the YAML can reference actions from integrations, this promotes re-use of actions, instead of hardcoding methods into the integration.
paths:
"/db/:id":
get:
action: mongodb/findOne
But, how can i reference an action of my own custom integration?
paths:
"/db/:id":
get:
action: mongodb/findOne
post:
action: ./myfile.js/mypostAction
This does not work :/ Probably because it applies require()
on it.
How about an optional spaceseparated action? something like this:
paths:
"/db/:id":
get:
action: mongodb/findOne
post:
action: ./myfile.js mypostAction
or this:
paths:
"/db/:id":
get:
action: mongodb/findOne
post:
file: ./myfile.js
action: mypostAction
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.