In the execution popup, the user will know:

• If the output of the graph is properly connected (Blocking)
• If a DHF envelope is setup (Info)
• If DHF envelope then inputs properly set

There's a dist_user/DesignerUI directory not used anymore and is confusing.

Custom fields are not being put back into the model when a block is reloaded from the database

@danielholgate , @epoilvet , let's discuss this one, not sure if we want it and why. Just an idea.

Currently (0.8-beta.2) Pipes allows the user to download the custom step code and the user has to copy it manually (from Download or other directory designated for downloads by the browser).

The goal of the ticket is to allow the user to deploy the code directly from the UI without the need to do anything on the command line.

Graphs/Steps have a "Title" (left panel), but when you "Save graph" the dialog asks for a name to save the Graph under.

Two enhancements around this:

1. Save dialog should use the "title" as default value in the pop up. Users can then change/modify this at save time and changes in popup also update the Title.
2. Using an existing (in DB) graph name should pop up a warning before save to ensure user is aware they'll overwrite

Use labels instead to improve user understanding of the messages

i.e instead of
"Usually there is at least one dhf/envelope block in the graph."
use
"Usually there is at least one Envelope block in the graph."

Core.sjs is now duplicated 2-3 times in the project. This should be removed. For the faster engine (#40) we need to 1 core.sjs. Can you please prioritize this?

Could be used for near real-time data transformation i.e. running flows for incoming documents before they get ingested into ML.

I will add

• SafetyReport
• Chicago contractors
• Customer360

Anything you guys can add, @epoilvet , @danielholgate ?

Collections such "http://marklogic.com/entity-services/models", "http://marklogic.com/data-hub/flow" etc are used by DHF to store flows, steps and other configurations.
Currently Pipes displays every collection in drop downs (when you create a block or execute a graph) but we don't want to let users accidentally pull documents from these collections, or write into them, at the risk of corrupting a DHF project.
Solution is to filter out collection names starting with the "http://marklogic.com"

Enable a user to remove the Pipes code from their DHF.

Use cases:

• A user tries Pipes and doesn't want to keep using it
• A user wants to do a fresh install of Pipes
• A user wants to remove Pipes as part of troubleshooting their DHF code

Currently geo/GeoProject function returns hardcoded WKT LINESTRING.

Should use input geometry type as return type e.g. POINT, LINESTRING, POLYGON and MULTI... variations.

WKT geometric objects support after this improvement:
POINT
LINESTRING
POLYGON <- Only simple polygon without hole

Not supported:
POLYGON <- Polygon with hole
MULTIPOINT
MULTILINESTRING
MULTIPOLYGON <- Multipolygon without or with hole
GEOMETRYCOLLECTION
TRIANGLE
TIN
POLYHEDRALSURFACE
POINT ZM <- Point with
POINT M <- Point with
POINT EMPTY <- Empty geometry object
MULTIPOLYGON EMPTY <- Empty geometry object

Examples:

Supported:
POINT (30 10)
LINESTRING (30 10, 10 30, 40 40)
POLYGON ((30 10, 40 40, 20 40, 10 20, 30 10)) <- Only simple polygon without hole

Not supported:
POLYGON ((35 10, 45 45, 15 40, 10 20, 35 10), (20 30, 35 35, 30 20, 20 30)) <- Polygon with hole
MULTIPOINT ((10 40), (40 30), (20 20), (30 10))
MULTIPOINT (10 40, 40 30, 20 20, 30 10)
MULTILINESTRING ((10 10, 20 20, 10 40), (40 40, 30 30, 40 20, 30 10))
MULTIPOLYGON (((30 20, 45 40, 10 40, 30 20)), ((15 5, 40 10, 10 20, 5 10, 15 5)))
MULTIPOLYGON (((40 40, 20 45, 45 30, 40 40)), ((20 35, 10 30, 10 10, 30 5, 45 20, 20 35), (30 20, 20 15, 20 25, 30 20))) <- Multipolygon with hole
GEOMETRYCOLLECTION (POINT (40 10), LINESTRING (10 10, 20 20, 10 40), POLYGON ((40 40, 20 45, 45 30, 40 40)))

See https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry

Add support for javascript reprojection libraries like RDNAPTRANS and PROJ6 so you can choose between online (Esri) service or local library.
For big data sets might be better to convert outside DHF/Pipes

Possible changes:

• don't assume default values if application.properties username and password are not provided
• they can still be provided in which case the log-in form can be skipped
• log in form
• username in upper right corner (?) as information
• log out button - could be available if clicked on the username, see previous (?), regardless how user signed in (using properties or log-in form)

@danielholgate, @epoilvet , what do you think about proposed changes?
It's a request from @michelderu

Explain what they do

• Source block
• Entity block

Examples:

• listing available DBs - which role can do that?
• executing graph
  ...

And inform the user about.
Right now an exception is thrown.

Examples:

• document identifier block (URI block)
• document grouping block (Collections block)
• Envelope should produce URI automatically
  ...

@epoilvet, @danielholgate, anything to add?

Allow graph A to be represented as a block in graph B. When you click on that block in graph B you need to be able to "zoom in" graph A (the editor switches from showing B to A. Also you need to able to zoom out so return from A to B.

@danielholgate , can you please copy/paste your stack trace? Thanks!

Will use space as separator because that is an invalid URI char. Will tokenize.

Add XPath block. Widgets: XPath + Namespaces. Input: node, output: nodes.

A Gradle plugin is needed to deploy/redeploy/undeploy pipes backend. This is needed for automated production environments and CICD pipelines.

I can work on a high performance Execution Engine. I have some background in compiler/language design, so I can help with this ;-).

I first will play around with Pipes for the coming week for a POC, and we first need to collaborate on ideas/requirements.

I think the best way is to write a SJS code generator, which converts the model in native SJS code. An alternative is converting the model to an Intermediate Code which then requires an interpreter on top of SJS, which cost extra development work and does not add anything because SJS is already portable etc.

I see that execution of the code is relatively isolated, so the code generator generated code should have the same input/output to make it a (relatively easy) drop-in-replacement.

Windows 10, Java SE Runtime Environment (build 1.8.0_171-b11)

Starting Pipes throws an exception:

`java -jar marklogic-pipes-0.8.1579126868.jar

20:00:00.808 [main] ERROR o.s.boot.SpringApplication - Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'restTemplate' defined in class path resource [com/marklogic/pipes/ui/ClientConfig.class]:
Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.web.client.RestTemplate]: Factory method 'restTemplate' threw exception; nested exception is java.lang.NumberFormatException: null
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:656)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:484)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1338)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:557)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:879)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215)
at com.marklogic.pipes.ui.Proxy.main(Proxy.java:13)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52)
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.web.client.RestTemplate]: Factory method 'restTemplate' threw exception; nested exception is java.lang.NumberFormatException: null
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:651)
... 27 common frames omitted
Caused by: java.lang.NumberFormatException: null
at java.lang.Integer.parseInt(Unknown Source)
at java.lang.Integer.parseInt(Unknown Source)
at com.marklogic.pipes.ui.ClientConfig.restTemplate(ClientConfig.java:72)
at com.marklogic.pipes.ui.ClientConfig$$EnhancerBySpringCGLIB$$48defcbf.CGLIB$restTemplate$1()
at com.marklogic.pipes.ui.ClientConfig$$EnhancerBySpringCGLIB$$48defcbf$$FastClassBySpringCGLIB$$8a57712b.invoke()
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:363)
at com.marklogic.pipes.ui.ClientConfig$$EnhancerBySpringCGLIB$$48defcbf.restTemplate()
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
... 28 common frames omitted`

@stuartMoorhouse, would you still like to tackle this?

node_modules_override/litegraph.js is not used for the build which causes the issue.

MarkLogic ports 8001/8002 are hardcoded. I will create a pull request to get this externalized to application.properties (MLGradle naming).

The fields which can be selected when creating Source Blocks should be inspected from a user-selected DB & Collection (Default), or alternatively from those in a pre-existing DHF Custom Step.

Add a selector between these two options and hide/show db and collection dropdowns etc

A few problems have been reported recently which were caused by a trailing space which had been accidentally copy/pasted after either the credentials or the DHF path in application.properties.

Properties should be trimmed so Pipes still works despite these. Possibly also alert the user on the console to the problem so they can correct, but still go ahead and start up with the trimmed values

When going to FROM FILE I can select any kind of document. E.g. when I pick a PDF and PNG it looks like they get 'uploaded'. Not sure if there is any negative impact but might make sense to restrict to certain file types?

This happens when the source collection contains both json and xml. @danielholgate , can you add the source samples and the screenshot, please?

When a graph fails, I want to know which component is the culprit.

Add a button to remove Source Block from current library. Check if they're used in current graph first

PLEASE DON'T TOUCH THIS ISSUE.

The page should show how the minimum valid graph looks like and explain the blocks.

To prevent browser failing to notice changed blocks.

It would be nice if Pipes could run in Gradle properties mode eg:

java -jar *pipes.jar -gradleMode -environmentLabel=mydocker

This should read properties from gradle.properties and override by gradle-docker.properties.

I can take care of the impl.

Tx

When a block is reloaded from the database the details appear in the "Source Blocks" panel but the block does not appear in the Library (right click) within the graph

Enrich the block library metadata to generate the block documentation

At the moment, Entity blocks are fixed at creation time.

Is there a preferred way to work with Entities getting updated in Pipes? Say you add new property to an Entity, do you have to create another block first, remove existing from graph, and redo connecting new block with input output? Or are there other options? Similar question for removing or editing Entity properties.

An enhancement would be to notify users when an entity underlying a block changes. This would at least make user aware and give him possibility to act -> manually recreate it.. Auto-recreating/updating the block could be problematic because properties could disappear or be renamed and would break existing connections. But doing nothing for updated entities will break the graph too?

Ideally, Pipes should draw a warning sign on an Entity block if it notices that the Entity definition in DHF has changed. This could be triggered by re-loading the graph. Depending on change automated actions could be executed.

The block creation form needs clean up to fix a few quirky behaviors and give better feedback to users:

• Add validation of custom field names (cannot be empty/null, must be valid json property name, maximum property lengths?)
• Validation of block names (non-null, remove unsupported characters etc)
• Reset fields/form after successful save
• Currently has two "Create Block" fields and buttons at top + bottom : Just one seems cleaner
• Instead of current adding "- already saved" to block name after save, add an icon which indicates it has been saved
• Define use case for re-loading/editing blocks and change form as appropriate. Should blocks ever be editable after creation?
  Misc other small UI glitches around data entry here

Let's address the security issues.
To see them, click on the Security tab.

Seems to be random/when execute button in preview clicked very rapidly

Error is "Info: Status 500: XDMP-ARGTYPE: xdmp.documentGetCollections(null) -- arg1 is not of type String" (even though collection is set in UI)

also seen in backend MarkLogic logs

Error Handling is not very good in Pipes. The Java Middle Tier is throwing away Exceptions. I will create a branch, fix this and create a pull request.