Giter Club home page Giter Club logo

hop's Issues

[Bug]: Transform Get Subfolders is broken

Apache Hop version?

2.2.0

Java version?

openjdk version "11.0.17" 2022-10-18

Operating system

Linux

What happened?

With any folder you specify you get over 1000 rows on output until you get a stack overflow error:

2022/12/01 12:24:03 - Get subfolder names.0 - ERROR: java.lang.StackOverflowError
2022/12/01 12:24:03 - Get subfolder names.0 - 	at org.apache.hop.ui.hopgui.file.pipeline.HopGuiPipelineGraph$4.rowWrittenEvent(HopGuiPipelineGraph.java:4200)
2022/12/01 12:24:03 - Get subfolder names.0 - 	at org.apache.hop.pipeline.transform.BaseTransform.handlePutRow(BaseTransform.java:1065)
2022/12/01 12:24:03 - Get subfolder names.0 - 	at org.apache.hop.pipeline.transform.BaseTransform$DefaultRowHandler.putRow(BaseTransform.java:3794)
2022/12/01 12:24:03 - Get subfolder names.0 - 	at org.apache.hop.pipeline.transform.BaseTransform.putRow(BaseTransform.java:1010)
2022/12/01 12:24:03 - Get subfolder names.0 - 	at org.apache.hop.pipeline.transforms.getsubfolders.GetSubFolders.processRow(GetSubFolders.java:202)
2022/12/01 12:24:03 - Get subfolder names.0 - 	at org.apache.hop.pipeline.transforms.getsubfolders.GetSubFolders.processRow(GetSubFolders.java:206)
2022/12/01 12:24:03 - Get subfolder names.0 - 	at org.apache.hop.pipeline.transforms.getsubfolders.GetSubFolders.processRow(GetSubFolders.java:206)
2022/12/01 12:24:03 - Get subfolder names.0 - 	at org.apache.hop.pipeline.transforms.getsubfolders.GetSubFolders.processRow(GetSubFolders.java:206)

Issue Priority

Priority: 1

Issue Component

Component: Transforms

[Feature Request]: Have specific row listeners for putRowTo() and getRowTo()

What would you like to happen?

Some transforms produce output of varying nature. For example the Workflow and Pipeline Executor transforms can generate various types of output like the result, result rows, output rows, result files and so on.
If we attach a row listener to this transform we'll get rows with varying layout in the listeners without a way of keeping them apart by target transform.
We have the same problem for info rows entering a transform with getRowFrom().

Suggested for Hop 3.x

Issue Priority

Priority: 3

Issue Component

Component: Api

Rewrite the database dialog

Get rid of the XUL dependency.
Improves performance

Introduce extension points to allow database plugins to add extra metadata fields and options.

[Bug]: Hop Web clipboard access is not available for non-localhost servers

Apache Hop version?

2.2.0

Java version?

openjdk version "11.0.17" 2022-10-18

Operating system

Docker

What happened?

When copying to the clipboard on hop web we see the following error appear:

There was an unexpected error while copying data to the clipboard. org.eclipse.swt.SWTError: Cannot set data in clipboard at org.eclipse.swt.dnd.DND.error(DND.java:263) at org.eclipse.swt.dnd.DND.error(DND.java:219) at org.eclipse.swt.dnd.Clipboard.setContents(Clipboard.java:272) at org.apache.hop.ui.core.gui.GuiResource.toClipboard(GuiResource.java:1296) at org.apache.hop.ui.hopgui.file.pipeline.delegates.HopGuiPipelineClipboardDelegate.toClipboard(HopGuiPipelineClipboardDelegate.java:62) at org.apache.hop.ui.hopgui.file.pipeline.delegates.HopGuiPipelineClipboardDelegate.copySelected(HopGuiPipelineClipboardDelegate.java:296) at org.apache.hop.ui.hopgui.file.pipeline.HopGuiPipelineGraph.copySelectedToClipboard(HopGuiPipelineGraph.java:5134) at org.apache.hop.ui.hopgui.HopGui.menuEditCopySelected(HopGui.java:880) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.hop.ui.core.gui.GuiMenuWidgets.executeMenuItem(GuiMenuWidgets.java:174) at org.apache.hop.ui.core.gui.GuiMenuWidgets.lambda$addMenuWidgets$0(GuiMenuWidgets.java:122) at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:109) at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:687) at org.eclipse.swt.widgets.Widget.notifyListeners(Widget.java:594) at org.eclipse.swt.widgets.MenuItem.handleAcceleratorActivation(MenuItem.java:687) at org.eclipse.swt.widgets.AcceleratorBinding.handleEvent(AcceleratorBinding.java:35) at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:109) at org.eclipse.swt.widgets.Display.filterEvent(Display.java:1819) at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:685) at org.eclipse.swt.widgets.Widget.notifyListeners(Widget.java:594) at org.eclipse.swt.widgets.Display.executeNextEvent(Display.java:1216) at org.eclipse.swt.widgets.Display.runPendingMessages(Display.java:1197) at org.eclipse.swt.widgets.Display.safeReadAndDispatch(Display.java:1180) at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:1172) at org.eclipse.rap.rwt.application.AbstractEntryPoint.createUI(AbstractEntryPoint.java:69) at org.eclipse.rap.rwt.internal.lifecycle.RWTLifeCycle.createUI(RWTLifeCycle.java:177) at org.eclipse.rap.rwt.internal.lifecycle.RWTLifeCycle$UIThreadController.run(RWTLifeCycle.java:290) at java.base/java.lang.Thread.run(Thread.java:829) at org.eclipse.rap.rwt.internal.lifecycle.UIThread.run(UIThread.java:107)

Issue Priority

Priority: 2

Issue Component

Component: Hop Web

[Bug]: Workflow executor causes exception saving Execution Information

Apache Hop version?

SNAPSHOT-20221128

Java version?

openjdk version "11.0.9.1" 2020-11-04 OpenJDK Runtime Environment (build 11.0.9.1+1-post-Uos-1deb10u2) OpenJDK 64-Bit Server VM (build 11.0.9.1+1-post-Uos-1deb10u2, mixed mode, sharing)

Operating system

Linux

What happened?

The bug can be reproduced in the "samples" project by running the pipeline

transforms/workflow-executor-basic.hpl

Step to reproduce:

  1. Open project "samples"
  2. Define a Execution data profile "last_100" with sampler "last output rows" and size "100"
  3. Define a Execution information location (e.g. "neo4j" as neo4j location)
  4. Define a Pipeline run configuration (e.g. "local") choosing up the previous Execution information location "neo4j" and Execution data profile "last_100"
  5. Define a Workflow run configuration (e.g. "local") (optional: choose the previous Execution information location)
  6. Run the workflow-executor-basic pipeline

Otherwise:

  1. copy in your already configured (test) project:

transforms/workflow-executor-basic.hpl
transforms/workflow-executor-child.hwf

  1. Run the workflow-executor-basic pipeline

Note:
The child process adds to result the filename under HOP_UNIT_TESTS_FOLDER, which is somewhere hardcoded. There must be some files in that folder.

log:

2022/11/30 15:59:47 - Hop - Pipeline opened.
2022/11/30 15:59:47 - Hop - Launching pipeline [workflow-executor-basic]...
2022/11/30 15:59:47 - Hop - Started the pipeline execution.
2022/11/30 15:59:48 - workflow-executor-basic - Executing this pipeline using the Local Pipeline Engine with run configuration 'local'
2022/11/30 15:59:48 - workflow-executor-basic - Execution started for pipeline [workflow-executor-basic]
2022/11/30 15:59:48 - Data for parameters.0 - Finished processing (I=0, O=0, R=0, W=3, U=0, E=0)
2022/11/30 15:59:48 - workflow-executor-child - Start of workflow execution
2022/11/30 15:59:48 - workflow-executor-child - Starting action [Wait for]
2022/11/30 15:59:49 - workflow-executor-child - Starting action [Add filenames to result]
2022/11/30 15:59:49 - workflow-executor-child - Finished action [Add filenames to result] (result=[true])
2022/11/30 15:59:49 - workflow-executor-child - Finished action [Wait for] (result=[true])
2022/11/30 15:59:49 - workflow-executor-child - Workflow execution finished
2022/11/30 15:59:49 - workflow-executor-child.hwf.0 - Workflow duration : 1.089 seconds [  1.088" ]
2022/11/30 15:59:49 - workflow-executor-child - Workflow duration : 1.089 seconds [  1.088" ]
2022/11/30 15:59:49 - workflow-executor-child - Start of workflow execution
2022/11/30 15:59:49 - workflow-executor-child - Starting action [Wait for]
2022/11/30 15:59:50 - workflow-executor-basic - Warning: unable to register execution info (data and state) at location neo4j(non-fatal)
2022/11/30 15:59:50 - workflow-executor-child - Starting action [Add filenames to result]
2022/11/30 15:59:50 - workflow-executor-child - Finished action [Add filenames to result] (result=[true])
2022/11/30 15:59:50 - workflow-executor-child - Finished action [Wait for] (result=[true])
2022/11/30 15:59:50 - workflow-executor-child - Workflow execution finished
2022/11/30 15:59:50 - workflow-executor-child.hwf.0 - Workflow duration : 1.042 seconds [  1.042" ]
2022/11/30 15:59:50 - workflow-executor-child - Workflow duration : 1.042 seconds [  1.042" ]
2022/11/30 15:59:50 - workflow-executor-child - Start of workflow execution
2022/11/30 15:59:50 - workflow-executor-child - Starting action [Wait for]
2022/11/30 15:59:51 - workflow-executor-child - Starting action [Add filenames to result]
2022/11/30 15:59:51 - workflow-executor-child - Finished action [Add filenames to result] (result=[true])
2022/11/30 15:59:51 - workflow-executor-child - Finished action [Wait for] (result=[true])
2022/11/30 15:59:51 - workflow-executor-child - Workflow execution finished
2022/11/30 15:59:51 - workflow-executor-child.hwf.0 - Workflow duration : 1.074 seconds [  1.074" ]
2022/11/30 15:59:51 - workflow-executor-child - Workflow duration : 1.074 seconds [  1.074" ]
2022/11/30 15:59:51 - workflow-executor-child.hwf.0 - Finished processing (I=0, O=0, R=3, W=5895, U=0, E=0)
2022/11/30 15:59:51 - Results.0 - Finished processing (I=0, O=0, R=3, W=3, U=0, E=0)
2022/11/30 15:59:51 - Result files.0 - Finished processing (I=0, O=0, R=5892, W=5892, U=0, E=0)
2022/11/30 15:59:51 - workflow-executor-basic - Pipeline duration : 3.41 seconds [  3.410" ]
2022/11/30 15:59:51 - workflow-executor-basic - ERROR: Error handling writing final pipeline state to location (non-fatal)
2022/11/30 15:59:51 - workflow-executor-basic - ERROR: org.apache.hop.core.exception.HopException: 
2022/11/30 15:59:51 - workflow-executor-basic - Error getting execution from Neo4j
2022/11/30 15:59:51 - workflow-executor-basic - org.apache.hop.core.exception.HopValueException: 
2022/11/30 15:59:51 - workflow-executor-basic - Unexpected conversion error while converting value [ExecutionTime Integer(15)] to an Integer
2022/11/30 15:59:51 - workflow-executor-basic - class java.lang.String cannot be cast to class java.lang.Long (java.lang.String and java.lang.Long are in module java.base of loader 'bootstrap')
2022/11/30 15:59:51 - workflow-executor-basic - 
2022/11/30 15:59:51 - workflow-executor-basic - 
2022/11/30 15:59:51 - workflow-executor-basic - 	at org.apache.hop.neo4j.execution.NeoExecutionInfoLocation.registerData(NeoExecutionInfoLocation.java:878)
2022/11/30 15:59:51 - workflow-executor-basic - 	at org.apache.hop.pipeline.engines.local.LocalPipelineEngine.stopTransformExecutionInfoTimer(LocalPipelineEngine.java:486)
2022/11/30 15:59:51 - workflow-executor-basic - 	at org.apache.hop.pipeline.engines.local.LocalPipelineEngine.pipelineCompleted(LocalPipelineEngine.java:465)
2022/11/30 15:59:51 - workflow-executor-basic - 	at org.apache.hop.pipeline.Pipeline.firePipelineExecutionFinishedListeners(Pipeline.java:1343)
2022/11/30 15:59:51 - workflow-executor-basic - 	at org.apache.hop.pipeline.Pipeline.lambda$startThreads$0(Pipeline.java:1142)
2022/11/30 15:59:51 - workflow-executor-basic - 	at org.apache.hop.pipeline.transform.BaseTransform.fireTransformFinishedListeners(BaseTransform.java:2782)
2022/11/30 15:59:51 - workflow-executor-basic - 	at org.apache.hop.pipeline.transform.BaseTransform.markStop(BaseTransform.java:2771)
2022/11/30 15:59:51 - workflow-executor-basic - 	at org.apache.hop.pipeline.transform.RunThread.run(RunThread.java:142)
2022/11/30 15:59:51 - workflow-executor-basic - 	at java.base/java.lang.Thread.run(Thread.java:834)
2022/11/30 15:59:51 - workflow-executor-basic - Caused by: java.lang.RuntimeException: org.apache.hop.core.exception.HopValueException: 
2022/11/30 15:59:51 - workflow-executor-basic - Unexpected conversion error while converting value [ExecutionTime Integer(15)] to an Integer
2022/11/30 15:59:51 - workflow-executor-basic - class java.lang.String cannot be cast to class java.lang.Long (java.lang.String and java.lang.Long are in module java.base of loader 'bootstrap')
2022/11/30 15:59:51 - workflow-executor-basic - 
2022/11/30 15:59:51 - workflow-executor-basic - 	at org.apache.hop.neo4j.execution.NeoExecutionInfoLocation.saveNeo4jRowsAndMeta(NeoExecutionInfoLocation.java:1116)
2022/11/30 15:59:51 - workflow-executor-basic - 	at org.apache.hop.neo4j.execution.NeoExecutionInfoLocation.registerNeo4jData(NeoExecutionInfoLocation.java:945)
2022/11/30 15:59:51 - workflow-executor-basic - 	at org.apache.hop.neo4j.execution.NeoExecutionInfoLocation.lambda$registerData$11(NeoExecutionInfoLocation.java:876)
2022/11/30 15:59:51 - workflow-executor-basic - 	at org.neo4j.driver.internal.InternalSession.lambda$transaction$4(InternalSession.java:150)
2022/11/30 15:59:51 - workflow-executor-basic - 	at org.neo4j.driver.internal.retry.ExponentialBackoffRetryLogic.retry(ExponentialBackoffRetryLogic.java:103)
2022/11/30 15:59:51 - workflow-executor-basic - 	at org.neo4j.driver.internal.InternalSession.transaction(InternalSession.java:146)
2022/11/30 15:59:51 - workflow-executor-basic - 	at org.neo4j.driver.internal.InternalSession.writeTransaction(InternalSession.java:124)
2022/11/30 15:59:51 - workflow-executor-basic - 	at org.neo4j.driver.internal.InternalSession.writeTransaction(InternalSession.java:118)
2022/11/30 15:59:51 - workflow-executor-basic - 	at org.apache.hop.neo4j.execution.NeoExecutionInfoLocation.registerData(NeoExecutionInfoLocation.java:876)
2022/11/30 15:59:51 - workflow-executor-basic - 	... 8 more
2022/11/30 15:59:51 - workflow-executor-basic - Caused by: org.apache.hop.core.exception.HopValueException: 
2022/11/30 15:59:51 - workflow-executor-basic - Unexpected conversion error while converting value [ExecutionTime Integer(15)] to an Integer
2022/11/30 15:59:51 - workflow-executor-basic - class java.lang.String cannot be cast to class java.lang.Long (java.lang.String and java.lang.Long are in module java.base of loader 'bootstrap')
2022/11/30 15:59:51 - workflow-executor-basic - 
2022/11/30 15:59:51 - workflow-executor-basic - 	at org.apache.hop.core.row.value.ValueMetaBase.getInteger(ValueMetaBase.java:2347)
2022/11/30 15:59:51 - workflow-executor-basic - 	at org.apache.hop.core.row.value.ValueMetaInteger.getNativeDataType(ValueMetaInteger.java:40)
2022/11/30 15:59:51 - workflow-executor-basic - 	at org.apache.hop.neo4j.execution.NeoExecutionInfoLocation.saveNeo4jRowsAndMeta(NeoExecutionInfoLocation.java:1082)
2022/11/30 15:59:51 - workflow-executor-basic - 	... 16 more
2022/11/30 15:59:51 - workflow-executor-basic - Caused by: java.lang.ClassCastException: class java.lang.String cannot be cast to class java.lang.Long (java.lang.String and java.lang.Long are in module java.base of loader 'bootstrap')
2022/11/30 15:59:51 - workflow-executor-basic - 	at org.apache.hop.core.row.value.ValueMetaBase.getInteger(ValueMetaBase.java:2260)
2022/11/30 15:59:51 - workflow-executor-basic - 	... 18 more
2022/11/30 15:59:51 - workflow-executor-basic - Execution finished on a local pipeline engine with run configuration 'local'

Issue Priority

Priority: 2

Issue Component

Component: Execution Information

[Task]: Replace all clone() methods with copy a constructor

What needs to happen?

Sonar considers using clone() (and interface Clonable) as a blocking issue.
A lot of our code already switched over to copy constructors. Let's clean up the remainder.

Issue Priority

Priority: 3

Issue Component

Component: Api

[Bug]: Neo4j Constraint action should use 5.0 syntax

Apache Hop version?

2.2.0

Java version?

openjdk version "11.0.17" 2022-10-18

Operating system

Linux

What happened?

The Neo4j Constraint action throws an error against Neo4j 5.x databases:

2022/12/01 09:42:10 - Neo4j Constraint - ERROR: Error creating constraint with cypher [CREATE CONSTRAINT pk_person_id IF NOT EXISTS ON (n:Person) ASSERT  n.ID IS UNIQUE ]
2022/12/01 09:42:10 - Neo4j Constraint - ERROR: org.neo4j.driver.exceptions.ClientException: Invalid constraint syntax, ON and ASSERT should not be used. Replace ON with FOR and ASSERT with REQUIRE. (line 1, column 1 (offset: 0))
2022/12/01 09:42:10 - Neo4j Constraint - "CREATE CONSTRAINT pk_person_id IF NOT EXISTS ON (n:Person) ASSERT  n.ID IS UNIQUE"

Issue Priority

Priority: 1

Issue Component

Component: Graph

[Bug]: Some SWT dialogs aren't correctly parented

Apache Hop version?

2.2.0

Java version?

openjdk version "11.0.17" 2022-10-18

Operating system

Linux

What happened?

When editing an environment you can create a new file. However, it's not possible to click on a button:
image

In other metadata editing scenarios (run config ==> execution location ==> Neo4j connection...) you can notice that when you close the last dialog the wrong dialog is shown on top. There too the shell parenting is not done correctly.

Issue Priority

Priority: 2

Issue Component

Component: Hop Gui

[Bug]: Copy files action doesn't copy files from the local file system to S3

Apache Hop version?

SNAPSHOT-20221129

Java version?

openjdk version "11.0.17" 2022-10-18

Operating system

Linux

What happened?

The [https://hop.apache.org/manual/latest/workflow/actions/copyfiles.html](Copy files) action doesn't copy files from the local file system to S3. Copying files from S3 to the local file system does work.

Issue Priority

Priority: 2

Issue Component

Component: Actions

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.