koopjs / koop-cli Goto Github PK
View Code? Open in Web Editor NEWCLI tool to build Koop applications and plugins
License: Other
CLI tool to build Koop applications and plugins
License: Other
Running this command will throw an error in the newly created project:
koop add provider --local test-provider
The Koop CLI v1.0 aims to provide the full capability to manage every step in the app/plugin development. Beside the exising add
command, the new version should include:
remove
command that reverses the operation of the add
comamnd (#53)list
command that lists installed plugins for an app (#55)validate
command that checks whether the current plugin follows the spec (#54)koop.json
to include the plugin list for app and plugin propertiesAll development will happen within the branch 1.0.0-rc
If I add a local plugin like
$ koop add provider plugins/my-db-provider --local
and run the test, the local plugin tests will fail because the module path is incorrect.
The CLI by default uses npm install
to install dependencies, which creates an undesired lock file for projects manged by yarn
. The user should be able to specify which package manager to use in the CLI configuration.
While it has an add
command to auto-add a new plugin, there should be a remove
command to auto-remove an existing plugin for app ONLY.
koop remove <name>
For npm packages, the name is the package name, like @koopjs/provider-csv
. For local plugins, the name should be the plugin path in the src
directory. It should first look up the dependency name in the packages.json
then look up the src
directory if the name is ambiguous.
The command should remove the following thing from repo:
src
foldertest
foldersrc/plugins.js
Note that for configuration, the plugin namespace could be various depending on the plugin developer. The CLI can only try the best guess.
The command should ouput a list of delete actions to notify the changes and should not perform the removal at first. This is because older projects created by previous CLI versions may not be compatible with this command.
The user should run the command with -f
or --force
to actually execute the removal.
koop remove plugin-name -f
So the workflow would be
# the command must be used in the app directory
> cd app-dir
# try to remove a provider
> koop remove @koopjs/provider-csv
# The command ouputs
The following items will be modified:
- remove src/provider-csv
- remove test/provider-csv
- remove provider registration in src/plugins.js
- remove provider configuration in config/*.json
Please use `koop remove @koopjs/provider-csv -f` to confirm the removal.
# This command will actually execute the removal
> koop remove @koopjs/provider-csv -f
The add
and remove
command is directly updating the js files. If there is an error and the process is forced to exit, it will leave some files changed and some unchanged. This will pollute the user's codebase and it is difficult to restore if the user is unfamiliar with the internal process of Koop CLI. Ideally, the change should be generated first and then applied to the files if the first step succeeds.
In the sandbox mode, the CLI tool should generate all changed files and new files in a temporary work directory. It should take several steps:
If there is an error in step 1 to 4, or the user specifies the --sandbox
flag, it should skip the final step. This will keep the user's codebase in the original state at problematic situations.
otherwise, the provider output will not be properly cached
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.
src/utils/create-new-project/add-deployment-target/docker/Dockerfile
.github/workflows/main.yml
actions/checkout v3
actions/setup-node v3
package.json
@koopjs/koop-core ^9.1.1
dargs ^7.0.0
easy-table ^1.2.0
execa ^5.1.1
fs-extra ^11.1.1
klaw-sync ^6.0.0
latest-version ^5.1.0
lodash ^4.17.21
node-fetch ^2.6.12
nodemon ^3.0.1
recast ^0.23.4
yargs ^17.7.2
yargs-parser ^21.1.1
chai ^4.3.7
config ^3.3.9
cross-env ^7.0.3
express ^4.18.2
husky ^8.0.3
lint-staged ^14.0.0
mocha ^10.2.0
npm-check-updates ^16.10.18
proxyquire ^2.1.3
standard ^17.1.0
supertest ^6.3.3
npm 8.19.4
node 16.20.2
src/template/components/app/package.json
@koopjs/koop-core ^9.1.1
src/template/components/auth/package.json
src/template/components/output/package.json
express ^4.18.2
supertest ^6.3.3
src/template/components/provider/package.json
src/template/project/package.json
config ^3.3.9
@koopjs/cli ^1.1.7
chai ^4.3.7
mocha ^10.2.0
.travis.yml
I'm wanting to test out the koop-auth-direct-file auth plugin, and I tried to install by using the command: koop add auth @koopjs/auth-direct-file
Although the above command appeared to succeed, I am now getting an error when trying to run koop serve
on my project:
Error: Cannot find module '@koopjs/auth-direct-file'
I'm expecting I missed something for getting this set up. Can you provide documentation for how to install existing/published plugins?
npm install -g @koopjs/cli
npm ERR! code E404
npm ERR! 404 Not Found: @koopjs/cli@latest
npm ERR! A complete log of this run can be found in:
npm ERR! /Users/dani7825/.npm/_logs/2019-01-31T17_01_59_642Z-debug.log
For development workflows where you're working on a provider and using the koop-cli
to run a dev/debug environment, it would be valuable to have the option to have the server that runs when you run koop serve
to support HTTPS/SSL. This is because more places (now notably the ArcGIS Online Map Viewer) are enforcing HTTPS even for development workflows.
My own preference would be to remove most of the sample code (https://github.com/koopjs/koop-cli/blob/master/src/templates/provider/project/src/model.js#L54-L77) from model.js. It's a good illustration of a typical provider, but in practice I have to go and delete/cleanup this code when I use cli to spin up new provider boilerplate.
Can we have another provider template choice that omits this code?
Some deployment methods need some simple additional files in the project:
It can be added with an option when a new app project is created.
Since the CLI tool only supports koop >=3 and there is a koopCompatibility
field in the koop config, it should try to detect if the new plugin is compatible with the current project.
This is going to resolve #3 and #4.
Issue #3 requires the plugin list to store the registration options, along with the library instance, for each plugin. This is like
const github = require('@koopjs/provider-github');
const cache = require('koop-cache-memory');
const plugins = [
{
instance: github,
options: {
routePrefix: '/api/v1'
}
},
{
// no option for this plugin
instance: cache
}
];
module.exports = plugins;
Issue #4 requires a specific order in the plugin list, which is outputs should be ahead of other plugins in order to provider correct ouptut URLs. With such requirement, the plugin list should be refactored as
const odatat = require('@koopjs/output-odata');
const github = require('@koopjs/provider-github');
const cache = require('koop-cache-memory');
const outputs = [
{ instance: odata }
];
const plugins = [
{
instance: github,
options: {
routePrefix: '/api/v1'
}
},
{
// no option for this plugin
instance: cache
}
];
module.exports = [...outputs, ...plugins];
As the plugin list updated, the plugin registration at the Koop app should become
const plugins = require('./plugins')
plugins.forEach((plugin) => {
koop.register(plugin.instance, plugin.options);
});
As the plugin list becomes more complicated, it is difficult to continue using the simple string manipulation to modify the JavaScript source code.
A more reasonable way is to parse the source file into a JavaScript AST and insert the updates into the corresponding plugin array. The workflow can changes to:
This can be done using the recast library.
koop serve
fails when directory has spaces in name. IE /my project/koop
.
Error: Cannot find module '/my '
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:581:15)
at Function.Module._load (internal/modules/cjs/loader.js:507:25)
at Function.Module.runMain (internal/modules/cjs/loader.js:742:12)
at startup (internal/bootstrap/node.js:283:19)
at bootstrapNodeJSCore (internal/bootstrap/node.js:743:3)
Koop plugins should be registered in order otherwise they may not function correctly.
A certain rule is that output plugins should be registered before provider plugins. But the user may change the order based on a specific need. For example, an authz plugin may be ahead of a certain provider plugin for authenitication.
To implement this feature, the CLI needs to:
The init
command initializes the Koop project in a given directory. This command is similar to the add
command but without creating a new directory. It will add any necessary project files in the directory if not exists.
This is useful when the user already has a project folder, for example, a fresh clone from a new GitHub repo. The user can directly start from the repo directory.
// initialize an app in the current directory
koop init app .
The current add
command can only add Koop plugins already published at npm. If a developer wants to add a plugin that is customized and local only, the developer has to create all script files from scratch and register the plugin manually. While the Koop CLI encourages to use well-tested open-source plugins, it should also help the development of private plugins.
For example, I want to expose data from my local data store, which needs a very customized access method. As I don't want to publish any detail about my data store, I would like to directly develop a plugin within the Koop app and register it.
I should be able to use
koop add provider plugins/my-data-provider --local
In this command, the local
parameter indicates the plugin is not from npm, but from local code. The plugins/my-data-provider
is a path to the provider source code directory (in /src
), which should have the same structure as the template.
If the source code files are not found, the CLI should create the folder, then add the source code and test files from the template:
// source code
src/plugins/my-data-provider/index.js
src/plugins/my-data-provider/model.js
// test code
test/plugins/my-data-provider/index.test.js
test/plugins/my-data-provider/model.test.js
Then the command can register the plugin at the app automatically.
We should add support to the Koop authorization plugin at the CLI.
Check the specification
new
command to create an auth plugin projectadd
command to add an auth plugin (#16)v0.2.0
Koop can accept options when registering a plugin.
const provider = require('@koopjs/provider-github')
koop.register(provider, { routePrefix: '/api/v1'})
see https://koopjs.github.io/docs/usage/koop-core
We should consider a way to inject the option for the koop add
command.
The validate
command is for plugin projects only. It is used to check whether the plugin follows the plugin spec and export necessary properties.
koop validate
It will load the plugin package and return a succesful message if every check passes
> koop validate
The plugin is valid.
Otherwise, it shoud exit with an error code and print the errors
> koop valid
The plugins in invalid: the koop.json file is missing.
When I follow the instructions on https://koopjs.github.io/docs/basics/quickstart#using-the-koop-cli, it installs an app that is using koop v3:
... but v4 has been out for a month. I see you updated the master branch, but can we do a release so when I install koop-cli from npm I get the latest koop? Thanks!
Authoriztion plugins should be registered after output-services plugins and before providers. Any providers registered before an authorization plugin will not have its plugin routes secured.
The list command is for app project ONLY. This command read the plugins
list from the koop.js
file and print the plugin names:
# not specify any plugin type, then list everything
> koop list
Provider:
- geojson-provider
- csv-provider
Output:
- geoservice-output
Auth:
- koop-file-auth
# specify a type
> koop list provider
Provider
- geojson-provider
- csv-provider
These names can be used in the remove
command (#53). The remove
command and add
command should update the plugin list in the koop.js
:
{
"plugins": [
{
"name": "test-provider",
"type": "proider"
},
{
"name": "test-output",
"type": "output",
"local": true
}
]
}
It should not directly parse the src/plugins.js
file because it may be too complicated to include all variations in js.
This command works best with a project created with the latest version of CLI.
This issue is to discuss the deploy
command and the deployer. More content is to add.
This proposed command is to automate the Koop application deployment to a given cloud platform.
When used, the command reads koop.json
, the CLI configuration, at the current Koop application project and check if the project's deployer is specified.
{
"deployer": "koop-deployer-heroku"
}
The deployer
property refers to the name of a standalone Koop deployer module, which includes the actual logic of the deployment.
Alternatively, a js file path can be specified if a custom deployment is needed.
{
"deployer": "./path/to/custom/module.js"
}
If the deployer is not specified, the command will throw an deployer undefined error.
If the deployer is specified, the deployer will be loaded and deploy the application with any given arguments.
The deploy
command is actually a runner of any given deployer. This provides many advantages:
Psedocode of the src/commands/deploy.js
:
// read koop.json at the app directory
const config = readKoopConfig()
const deployModule = {
command: 'deploy',
description: 'deploy command'
}
if (config.deployer) {
// if the deployer is specified, load the deployer
const deployer = require(config.deployer)
// pass the deployer's argument builder and handler to the "deploy" command
deployModule.builder = deployer.builder
deployModule.handler = deployer.handler
} else {
// export a default deploy command module, which is just to print the
// deployer undefined error
deployModule.builder = () => {}
deployModule.handler = () => {
throw new Error('deployer undefined')
}
}
// export a yargs command module
module.exports = deployModule
A deployer is a separate nodejs module that exports a yargs command module. It is the actual implementation of the deploy
command for a specific platform.
An example:
module.exports = {
command: 'koop deploy',
description: 'my custom deploy command',
builder: (yargs) => {
// defind command options
}
handler: (argv) => {
// handle command call
}
}
The Koop CLI and the deployer module must be installed as global packages or the project's dev dependencies at the same time. Once the user specifies the deployer
at the project's koop.json
, he can use the koop deploy
command for the app.
deploy
commandnew
command to create a new deployer projectadd
command to add a deployer to the current Koop appv0.3.0
Add a dockerfile that runs a Koop server for the new or existing project. This could be useful for users who run the Koop server in a docker environemnt.
Create a new app with a dockerfile
koop new app my-app --dockerfile
Add a dockerfile to the existing app
koop add dockerfile
@haoliangyu how do you setup Visual Studio or use node --inspect --debug-brk
when debugging a new provider? I'm running koop serve
but I don't know how to debug it.
Thanks!
After installing Koopjs CLI as per instructions. 'koop' command failed with error below.
I am using node version -- v16.14.2, NPM: 8.5.0
I have tried downgrading node-fetch to 2.6.1. This had no impact.
I was looking for an answer that avoided re writing the imports. #
$ koop
/opt/nodejs/16.14.2/lib/node_modules/@koopjs/cli/src/utils/add-plugin/add-npm-plugin.js:3
const latestVersion = require('latest-version')
^
Error [ERR_REQUIRE_ESM]: require() of ES Module /opt/nodejs/16.14.2/lib/node_modules/@koopjs/cli/node_modules/latest-version/index.js from /opt/nodejs/16.14.2/lib/node_modules/@koopjs/cli/src/utils/add-plugin/add-npm-plugin.js not supported.
Instead change the require of index.js in /opt/nodejs/16.14.2/lib/node_modules/@koopjs/cli/src/utils/add-plugin/add-npm-plugin.js to a dynamic import() which is available in all CommonJS modules.
at Object.<anonymous> (/opt/nodejs/16.14.2/lib/node_modules/@koopjs/cli/src/utils/add-plugin/add-npm-plugin.js:3:23)
at Object.<anonymous> (/opt/nodejs/16.14.2/lib/node_modules/@koopjs/cli/src/utils/add-plugin/index.js:4:22)
at Object.<anonymous> (/opt/nodejs/16.14.2/lib/node_modules/@koopjs/cli/src/cli/commands/add.js:1:19)
at /opt/nodejs/16.14.2/lib/node_modules/@koopjs/cli/node_modules/require-directory/index.js:76:17
at Array.forEach (<anonymous>)
at Object.requireDirectory (/opt/nodejs/16.14.2/lib/node_modules/@koopjs/cli/node_modules/require-directory/index.js:59:24)
at M.addDirectory (/opt/nodejs/16.14.2/lib/node_modules/@koopjs/cli/node_modules/yargs/build/index.cjs:1:5492)
at Xt.commandDir (/opt/nodejs/16.14.2/lib/node_modules/@koopjs/cli/node_modules/yargs/build/index.cjs:1:32877)
at Object.<anonymous> (/opt/nodejs/16.14.2/lib/node_modules/@koopjs/cli/src/cli/index.js:9:4) {
code: 'ERR_REQUIRE_ESM'
}
Talking with Daniel at the DC conference he asked for some suggestions for items to add. I don't know if they are included in the forthcoming CLI, but yeoman generators would be nice to get up and running.
Examples:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.