koopjs / koop-cli Goto Github PK

Running this command will throw an error in the newly created project:

koop add provider --local test-provider

The Koop CLI v1.0 aims to provide the full capability to manage every step in the app/plugin development. Beside the exising add command, the new version should include:

• a remove command that reverses the operation of the add comamnd (#53)
• a list command that lists installed plugins for an app (#55)
• a validate command that checks whether the current plugin follows the spec (#54)
• update koop.json to include the plugin list for app and plugin properties
• update the logger for more structual logging

All development will happen within the branch 1.0.0-rc

If I add a local plugin like

$ koop add provider plugins/my-db-provider --local

and run the test, the local plugin tests will fail because the module path is incorrect.

The CLI by default uses npm install to install dependencies, which creates an undesired lock file for projects manged by yarn. The user should be able to specify which package manager to use in the CLI configuration.

While it has an add command to auto-add a new plugin, there should be a remove command to auto-remove an existing plugin for app ONLY.

koop remove <name>

For npm packages, the name is the package name, like @koopjs/provider-csv. For local plugins, the name should be the plugin path in the src directory. It should first look up the dependency name in the packages.json then look up the src directory if the name is ambiguous.

The command should remove the following thing from repo:

• plugin directory in the src folder
• plugin directory in the test folder
• plugin registration code in the src/plugins.js
• plugin configuration

Note that for configuration, the plugin namespace could be various depending on the plugin developer. The CLI can only try the best guess.

The command should ouput a list of delete actions to notify the changes and should not perform the removal at first. This is because older projects created by previous CLI versions may not be compatible with this command.

The user should run the command with -f or --force to actually execute the removal.

koop remove plugin-name -f

So the workflow would be

# the command must be used in the app directory
> cd app-dir

# try to remove a provider
> koop remove @koopjs/provider-csv

# The command ouputs
The following items will be modified:
- remove src/provider-csv
- remove test/provider-csv
- remove provider registration in src/plugins.js
- remove provider configuration in config/*.json

Please use `koop remove @koopjs/provider-csv -f` to confirm the removal.

# This command will actually execute the removal
> koop remove @koopjs/provider-csv -f

• neat console message and progress notification
• consistent error handling pattern
• optionally add the configuration when creating a project
• optionally add a server.js component when creating a provider project
• complete API and command documentation

The add and remove command is directly updating the js files. If there is an error and the process is forced to exit, it will leave some files changed and some unchanged. This will pollute the user's codebase and it is difficult to restore if the user is unfamiliar with the internal process of Koop CLI. Ideally, the change should be generated first and then applied to the files if the first step succeeds.

In the sandbox mode, the CLI tool should generate all changed files and new files in a temporary work directory. It should take several steps:

1. Create a temporary work directory
2. Create all new files
3. Copy all files to change
4. Change files
5. Move all changes to the actual work directory

If there is an error in step 1 to 4, or the user specifies the --sandbox flag, it should skip the final step. This will keep the user's codebase in the original state at problematic situations.

otherwise, the provider output will not be properly cached

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update dependency lint-staged to v14.0.1
• fix(deps): update dependency node-fetch to v2.6.13
• chore(deps): update dependency npm-check-updates to v16.12.3
• fix(deps): update dependency @koopjs/koop-core to v9.2.0
• fix(deps): update dependency execa to v8
• Click on this checkbox to rebase all open PRs at once

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• chore(deps): update dependency npm to v9
• chore(deps): update node.js to v18
• fix(deps): update dependency dargs to v8
• fix(deps): update dependency latest-version to v7
• fix(deps): update dependency node-fetch to v3

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

I'm wanting to test out the koop-auth-direct-file auth plugin, and I tried to install by using the command: koop add auth @koopjs/auth-direct-file

Although the above command appeared to succeed, I am now getting an error when trying to run koop serve on my project:
Error: Cannot find module '@koopjs/auth-direct-file'

I'm expecting I missed something for getting this set up. Can you provide documentation for how to install existing/published plugins?

For development workflows where you're working on a provider and using the koop-cli to run a dev/debug environment, it would be valuable to have the option to have the server that runs when you run koop serve to support HTTPS/SSL. This is because more places (now notably the ArcGIS Online Map Viewer) are enforcing HTTPS even for development workflows.

My own preference would be to remove most of the sample code (https://github.com/koopjs/koop-cli/blob/master/src/templates/provider/project/src/model.js#L54-L77) from model.js. It's a good illustration of a typical provider, but in practice I have to go and delete/cleanup this code when I use cli to spin up new provider boilerplate.

Can we have another provider template choice that omits this code?

Some deployment methods need some simple additional files in the project:

• serverless.yml
• .dockerfile

It can be added with an option when a new app project is created.

Since the CLI tool only supports koop >=3 and there is a koopCompatibility field in the koop config, it should try to detect if the new plugin is compatible with the current project.

This is going to resolve #3 and #4.

Issue #3 requires the plugin list to store the registration options, along with the library instance, for each plugin. This is like

const github = require('@koopjs/provider-github');
const cache = require('koop-cache-memory');

const plugins = [
  {
    instance: github,
    options: {
      routePrefix: '/api/v1'
    }
  },
  {
    // no option for this plugin
    instance: cache
  }
];

module.exports = plugins;

Issue #4 requires a specific order in the plugin list, which is outputs should be ahead of other plugins in order to provider correct ouptut URLs. With such requirement, the plugin list should be refactored as

const odatat = require('@koopjs/output-odata');
const github = require('@koopjs/provider-github');
const cache = require('koop-cache-memory');

const outputs = [
  { instance: odata }
];

const plugins = [
  {
    instance: github,
    options: {
      routePrefix: '/api/v1'
    }
  },
  {
    // no option for this plugin
    instance: cache
  }
];

module.exports = [...outputs, ...plugins];

As the plugin list updated, the plugin registration at the Koop app should become

const plugins = require('./plugins')

plugins.forEach((plugin) => {
  koop.register(plugin.instance, plugin.options);
});

As the plugin list becomes more complicated, it is difficult to continue using the simple string manipulation to modify the JavaScript source code.

A more reasonable way is to parse the source file into a JavaScript AST and insert the updates into the corresponding plugin array. The workflow can changes to:

1. parse the source file into an AST
2. add plugin import into the AST
3. add the plugin object into the plugin list
4. write the AST back to the source file

This can be done using the recast library.

koop serve fails when directory has spaces in name. IE /my project/koop.

Error: Cannot find module '/my '
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:581:15)
    at Function.Module._load (internal/modules/cjs/loader.js:507:25)
    at Function.Module.runMain (internal/modules/cjs/loader.js:742:12)
    at startup (internal/bootstrap/node.js:283:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:743:3)

Koop plugins should be registered in order otherwise they may not function correctly.

A certain rule is that output plugins should be registered before provider plugins. But the user may change the order based on a specific need. For example, an authz plugin may be ahead of a certain provider plugin for authenitication.

To implement this feature, the CLI needs to:

• get the type of the new plugin and register it at the right position
• be flexible for the user to customize the order

Description

The init command initializes the Koop project in a given directory. This command is similar to the add command but without creating a new directory. It will add any necessary project files in the directory if not exists.

Use Case

This is useful when the user already has a project folder, for example, a fresh clone from a new GitHub repo. The user can directly start from the repo directory.

// initialize an app in the current directory
koop init app .

The current add command can only add Koop plugins already published at npm. If a developer wants to add a plugin that is customized and local only, the developer has to create all script files from scratch and register the plugin manually. While the Koop CLI encourages to use well-tested open-source plugins, it should also help the development of private plugins.

For example, I want to expose data from my local data store, which needs a very customized access method. As I don't want to publish any detail about my data store, I would like to directly develop a plugin within the Koop app and register it.

I should be able to use

koop add provider plugins/my-data-provider --local

In this command, the local parameter indicates the plugin is not from npm, but from local code. The plugins/my-data-provider is a path to the provider source code directory (in /src), which should have the same structure as the template.

If the source code files are not found, the CLI should create the folder, then add the source code and test files from the template:

// source code
src/plugins/my-data-provider/index.js
src/plugins/my-data-provider/model.js

// test code
test/plugins/my-data-provider/index.test.js
test/plugins/my-data-provider/model.test.js

Then the command can register the plugin at the app automatically.

We should add support to the Koop authorization plugin at the CLI.

Check the specification

Plan

• add the plugin project template
• update new command to create an auth plugin project
• update add command to add an auth plugin (#16)
• release as v0.2.0

Koop can accept options when registering a plugin.

const provider = require('@koopjs/provider-github')
koop.register(provider, { routePrefix: '/api/v1'})

see https://koopjs.github.io/docs/usage/koop-core

We should consider a way to inject the option for the koop add command.

The validate command is for plugin projects only. It is used to check whether the plugin follows the plugin spec and export necessary properties.

koop validate

It will load the plugin package and return a succesful message if every check passes

> koop validate
The plugin is valid.

Otherwise, it shoud exit with an error code and print the errors

> koop valid
The plugins in invalid: the koop.json file is missing.

When I follow the instructions on https://koopjs.github.io/docs/basics/quickstart#using-the-koop-cli, it installs an app that is using koop v3:

... but v4 has been out for a month. I see you updated the master branch, but can we do a release so when I install koop-cli from npm I get the latest koop? Thanks!

Authoriztion plugins should be registered after output-services plugins and before providers. Any providers registered before an authorization plugin will not have its plugin routes secured.

The list command is for app project ONLY. This command read the plugins list from the koop.js file and print the plugin names:

# not specify any plugin type, then list everything
> koop list
Provider:
- geojson-provider
- csv-provider

Output:
- geoservice-output

Auth:
- koop-file-auth

# specify a type
> koop list provider
Provider
- geojson-provider
- csv-provider

These names can be used in the remove command (#53). The remove command and add command should update the plugin list in the koop.js:

{
  "plugins": [
    {
      "name": "test-provider",
      "type": "proider"
    },
    {
      "name": "test-output",
      "type": "output",
      "local": true
    }
  ]
}

It should not directly parse the src/plugins.js file because it may be too complicated to include all variations in js.

This command works best with a project created with the latest version of CLI.

This issue is to discuss the deploy command and the deployer. More content is to add.

deploy command

This proposed command is to automate the Koop application deployment to a given cloud platform.

When used, the command reads koop.json, the CLI configuration, at the current Koop application project and check if the project's deployer is specified.

{
  "deployer": "koop-deployer-heroku"
}

The deployer property refers to the name of a standalone Koop deployer module, which includes the actual logic of the deployment.

Alternatively, a js file path can be specified if a custom deployment is needed.

{
  "deployer": "./path/to/custom/module.js"
}

If the deployer is not specified, the command will throw an deployer undefined error.

If the deployer is specified, the deployer will be loaded and deploy the application with any given arguments.

The deploy command is actually a runner of any given deployer. This provides many advantages:

• the deployment logic for each platform is defined at a separate module
• the Koop CLI is still lean
• the app developer is able to switch deployers

Psedocode of the src/commands/deploy.js:

// read koop.json at the app directory
const config = readKoopConfig()

const deployModule = {
  command: 'deploy',
  description: 'deploy command'
}

if (config.deployer) {
  // if the deployer is specified, load the deployer
  const deployer = require(config.deployer)

  // pass the deployer's argument builder and handler to the "deploy" command
  deployModule.builder = deployer.builder
  deployModule.handler = deployer.handler
} else {
  // export a default deploy command module, which is just to print the
  // deployer undefined error
  deployModule.builder = () => {}
  deployModule.handler = () => {
    throw new Error('deployer undefined')
  }
}

// export a yargs command module
module.exports = deployModule

Deployer

A deployer is a separate nodejs module that exports a yargs command module. It is the actual implementation of the deploy command for a specific platform.

An example:

module.exports = {
  command: 'koop deploy',
  description: 'my custom deploy command',
  builder: (yargs) => {
    // defind command options
  }
  handler: (argv) => {
    // handle command call
  }
}

How to use

The Koop CLI and the deployer module must be installed as global packages or the project's dev dependencies at the same time. Once the user specifies the deployer at the project's koop.json, he can use the koop deploy command for the app.

Plan

• implement deploy command
• create an example deployer project
• update new command to create a new deployer project
• update add command to add a deployer to the current Koop app
• release as v0.3.0

Ddescription

Add a dockerfile that runs a Koop server for the new or existing project. This could be useful for users who run the Koop server in a docker environemnt.

Use Case

Create a new app with a dockerfile

koop new app my-app --dockerfile

Add a dockerfile to the existing app

koop add dockerfile

@haoliangyu how do you setup Visual Studio or use node --inspect --debug-brk when debugging a new provider? I'm running koop serve but I don't know how to debug it.

Thanks!

After installing Koopjs CLI as per instructions. 'koop' command failed with error below.

I am using node version -- v16.14.2, NPM: 8.5.0

I have tried downgrading node-fetch to 2.6.1. This had no impact.

I was looking for an answer that avoided re writing the imports. #

$ koop
/opt/nodejs/16.14.2/lib/node_modules/@koopjs/cli/src/utils/add-plugin/add-npm-plugin.js:3
const latestVersion = require('latest-version')
                      ^

Error [ERR_REQUIRE_ESM]: require() of ES Module /opt/nodejs/16.14.2/lib/node_modules/@koopjs/cli/node_modules/latest-version/index.js from /opt/nodejs/16.14.2/lib/node_modules/@koopjs/cli/src/utils/add-plugin/add-npm-plugin.js not supported.
Instead change the require of index.js in /opt/nodejs/16.14.2/lib/node_modules/@koopjs/cli/src/utils/add-plugin/add-npm-plugin.js to a dynamic import() which is available in all CommonJS modules.
    at Object.<anonymous> (/opt/nodejs/16.14.2/lib/node_modules/@koopjs/cli/src/utils/add-plugin/add-npm-plugin.js:3:23)
    at Object.<anonymous> (/opt/nodejs/16.14.2/lib/node_modules/@koopjs/cli/src/utils/add-plugin/index.js:4:22)
    at Object.<anonymous> (/opt/nodejs/16.14.2/lib/node_modules/@koopjs/cli/src/cli/commands/add.js:1:19)
    at /opt/nodejs/16.14.2/lib/node_modules/@koopjs/cli/node_modules/require-directory/index.js:76:17
    at Array.forEach (<anonymous>)
    at Object.requireDirectory (/opt/nodejs/16.14.2/lib/node_modules/@koopjs/cli/node_modules/require-directory/index.js:59:24)
    at M.addDirectory (/opt/nodejs/16.14.2/lib/node_modules/@koopjs/cli/node_modules/yargs/build/index.cjs:1:5492)
    at Xt.commandDir (/opt/nodejs/16.14.2/lib/node_modules/@koopjs/cli/node_modules/yargs/build/index.cjs:1:32877)
    at Object.<anonymous> (/opt/nodejs/16.14.2/lib/node_modules/@koopjs/cli/src/cli/index.js:9:4) {
  code: 'ERR_REQUIRE_ESM'
}

Talking with Daniel at the DC conference he asked for some suggestions for items to add. I don't know if they are included in the forthcoming CLI, but yeoman generators would be nice to get up and running.

Examples:

• Koop
  • AGOL/Portal authentication/no auth/custom authentication mocked up with comments
  • Options for what cache to use
  • Outputs
  • Providers
• Provider with comments and scaffolded name etc