apache / hop Goto Github PK
View Code? Open in Web Editor NEWHop Orchestration Platform
Home Page: https://hop.apache.org/
License: Apache License 2.0
Hop Orchestration Platform
Home Page: https://hop.apache.org/
License: Apache License 2.0
Cleanup XML of transform Mail
Priority: 3
Component: Metadata
Cleanup XML of transform UDJE
Priority: 3
Component: Metadata
Cleanup XML of transform Mail Input
Priority: 3
Component: Metadata
Cleanup XML of transform Join Rows
Priority: 3
Component: Metadata
Cleanup XML of transform Random Value
Priority: 3
Component: Metadata
Cleanup XML of WorkflowMeta
Priority: 3
Component: Metadata
Cleanup XML of transform Get table names
Priority: 3
Component: Metadata
'nough said
To get rid of a critical security issue
Cleanup XML of transform Property Input
Priority: 3
Component: Metadata
Add MDI support to Azure Event Hubs Listener
Priority: 3
Component: Metadata
Cleanup XML of transform Get Sub Folders
Priority: 3
Component: Metadata
Add MDI support to Call DB procedure
Priority: 3
Component: Metadata
Cleanup XML of transform Pipeline Executor
Priority: 3
Component: Metadata
Cleanup XML of transform UDJC
Priority: 3
Component: Metadata
Cleanup XML of transform LDAP Output
Priority: 3
Component: Metadata
Cleanup XML of transform LDAP Input
Priority: 3
Component: Metadata
Cleanup XML of transform Reservoir Sampling
Priority: 3
Component: Metadata
Cleanup XML of transform Execute Process
Priority: 3
Component: Metadata
2.2.0
openjdk version "11.0.17" 2022-10-18
Linux
With any folder you specify you get over 1000 rows on output until you get a stack overflow error:
2022/12/01 12:24:03 - Get subfolder names.0 - ERROR: java.lang.StackOverflowError
2022/12/01 12:24:03 - Get subfolder names.0 - at org.apache.hop.ui.hopgui.file.pipeline.HopGuiPipelineGraph$4.rowWrittenEvent(HopGuiPipelineGraph.java:4200)
2022/12/01 12:24:03 - Get subfolder names.0 - at org.apache.hop.pipeline.transform.BaseTransform.handlePutRow(BaseTransform.java:1065)
2022/12/01 12:24:03 - Get subfolder names.0 - at org.apache.hop.pipeline.transform.BaseTransform$DefaultRowHandler.putRow(BaseTransform.java:3794)
2022/12/01 12:24:03 - Get subfolder names.0 - at org.apache.hop.pipeline.transform.BaseTransform.putRow(BaseTransform.java:1010)
2022/12/01 12:24:03 - Get subfolder names.0 - at org.apache.hop.pipeline.transforms.getsubfolders.GetSubFolders.processRow(GetSubFolders.java:202)
2022/12/01 12:24:03 - Get subfolder names.0 - at org.apache.hop.pipeline.transforms.getsubfolders.GetSubFolders.processRow(GetSubFolders.java:206)
2022/12/01 12:24:03 - Get subfolder names.0 - at org.apache.hop.pipeline.transforms.getsubfolders.GetSubFolders.processRow(GetSubFolders.java:206)
2022/12/01 12:24:03 - Get subfolder names.0 - at org.apache.hop.pipeline.transforms.getsubfolders.GetSubFolders.processRow(GetSubFolders.java:206)
Priority: 1
Component: Transforms
To ensure that there is no accidental code inclusion and to flag absolute code incompatibility.
The goal is to re-introduce this as a metadata plugin later on.
It doesn't belong in the inner bowels of the code.
Cleanup XML of transform Multi-Merge Join
Priority: 3
Component: Metadata
Some transforms produce output of varying nature. For example the Workflow and Pipeline Executor transforms can generate various types of output like the result, result rows, output rows, result files and so on.
If we attach a row listener to this transform we'll get rows with varying layout in the listeners without a way of keeping them apart by target transform.
We have the same problem for info rows entering a transform with getRowFrom().
Suggested for Hop 3.x
Priority: 3
Component: Api
Get rid of the XUL dependency.
Improves performance
Introduce extension points to allow database plugins to add extra metadata fields and options.
Cleanup XML of transform Replace String
Priority: 3
Component: Metadata
2.2.0
openjdk version "11.0.17" 2022-10-18
Docker
When copying to the clipboard on hop web we see the following error appear:
There was an unexpected error while copying data to the clipboard. org.eclipse.swt.SWTError: Cannot set data in clipboard at org.eclipse.swt.dnd.DND.error(DND.java:263) at org.eclipse.swt.dnd.DND.error(DND.java:219) at org.eclipse.swt.dnd.Clipboard.setContents(Clipboard.java:272) at org.apache.hop.ui.core.gui.GuiResource.toClipboard(GuiResource.java:1296) at org.apache.hop.ui.hopgui.file.pipeline.delegates.HopGuiPipelineClipboardDelegate.toClipboard(HopGuiPipelineClipboardDelegate.java:62) at org.apache.hop.ui.hopgui.file.pipeline.delegates.HopGuiPipelineClipboardDelegate.copySelected(HopGuiPipelineClipboardDelegate.java:296) at org.apache.hop.ui.hopgui.file.pipeline.HopGuiPipelineGraph.copySelectedToClipboard(HopGuiPipelineGraph.java:5134) at org.apache.hop.ui.hopgui.HopGui.menuEditCopySelected(HopGui.java:880) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.hop.ui.core.gui.GuiMenuWidgets.executeMenuItem(GuiMenuWidgets.java:174) at org.apache.hop.ui.core.gui.GuiMenuWidgets.lambda$addMenuWidgets$0(GuiMenuWidgets.java:122) at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:109) at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:687) at org.eclipse.swt.widgets.Widget.notifyListeners(Widget.java:594) at org.eclipse.swt.widgets.MenuItem.handleAcceleratorActivation(MenuItem.java:687) at org.eclipse.swt.widgets.AcceleratorBinding.handleEvent(AcceleratorBinding.java:35) at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:109) at org.eclipse.swt.widgets.Display.filterEvent(Display.java:1819) at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:685) at org.eclipse.swt.widgets.Widget.notifyListeners(Widget.java:594) at org.eclipse.swt.widgets.Display.executeNextEvent(Display.java:1216) at org.eclipse.swt.widgets.Display.runPendingMessages(Display.java:1197) at org.eclipse.swt.widgets.Display.safeReadAndDispatch(Display.java:1180) at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:1172) at org.eclipse.rap.rwt.application.AbstractEntryPoint.createUI(AbstractEntryPoint.java:69) at org.eclipse.rap.rwt.internal.lifecycle.RWTLifeCycle.createUI(RWTLifeCycle.java:177) at org.eclipse.rap.rwt.internal.lifecycle.RWTLifeCycle$UIThreadController.run(RWTLifeCycle.java:290) at java.base/java.lang.Thread.run(Thread.java:829) at org.eclipse.rap.rwt.internal.lifecycle.UIThread.run(UIThread.java:107)
Priority: 2
Component: Hop Web
Cleanup XML of transform Property Output
Priority: 3
Component: Metadata
Cleanup XML of transform JavaScript
Priority: 3
Component: Metadata
SNAPSHOT-20221128
openjdk version "11.0.9.1" 2020-11-04 OpenJDK Runtime Environment (build 11.0.9.1+1-post-Uos-1deb10u2) OpenJDK 64-Bit Server VM (build 11.0.9.1+1-post-Uos-1deb10u2, mixed mode, sharing)
Linux
The bug can be reproduced in the "samples" project by running the pipeline
transforms/workflow-executor-basic.hpl
Step to reproduce:
Otherwise:
transforms/workflow-executor-basic.hpl
transforms/workflow-executor-child.hwf
Note:
The child process adds to result the filename under HOP_UNIT_TESTS_FOLDER, which is somewhere hardcoded. There must be some files in that folder.
log:
2022/11/30 15:59:47 - Hop - Pipeline opened.
2022/11/30 15:59:47 - Hop - Launching pipeline [workflow-executor-basic]...
2022/11/30 15:59:47 - Hop - Started the pipeline execution.
2022/11/30 15:59:48 - workflow-executor-basic - Executing this pipeline using the Local Pipeline Engine with run configuration 'local'
2022/11/30 15:59:48 - workflow-executor-basic - Execution started for pipeline [workflow-executor-basic]
2022/11/30 15:59:48 - Data for parameters.0 - Finished processing (I=0, O=0, R=0, W=3, U=0, E=0)
2022/11/30 15:59:48 - workflow-executor-child - Start of workflow execution
2022/11/30 15:59:48 - workflow-executor-child - Starting action [Wait for]
2022/11/30 15:59:49 - workflow-executor-child - Starting action [Add filenames to result]
2022/11/30 15:59:49 - workflow-executor-child - Finished action [Add filenames to result] (result=[true])
2022/11/30 15:59:49 - workflow-executor-child - Finished action [Wait for] (result=[true])
2022/11/30 15:59:49 - workflow-executor-child - Workflow execution finished
2022/11/30 15:59:49 - workflow-executor-child.hwf.0 - Workflow duration : 1.089 seconds [ 1.088" ]
2022/11/30 15:59:49 - workflow-executor-child - Workflow duration : 1.089 seconds [ 1.088" ]
2022/11/30 15:59:49 - workflow-executor-child - Start of workflow execution
2022/11/30 15:59:49 - workflow-executor-child - Starting action [Wait for]
2022/11/30 15:59:50 - workflow-executor-basic - Warning: unable to register execution info (data and state) at location neo4j(non-fatal)
2022/11/30 15:59:50 - workflow-executor-child - Starting action [Add filenames to result]
2022/11/30 15:59:50 - workflow-executor-child - Finished action [Add filenames to result] (result=[true])
2022/11/30 15:59:50 - workflow-executor-child - Finished action [Wait for] (result=[true])
2022/11/30 15:59:50 - workflow-executor-child - Workflow execution finished
2022/11/30 15:59:50 - workflow-executor-child.hwf.0 - Workflow duration : 1.042 seconds [ 1.042" ]
2022/11/30 15:59:50 - workflow-executor-child - Workflow duration : 1.042 seconds [ 1.042" ]
2022/11/30 15:59:50 - workflow-executor-child - Start of workflow execution
2022/11/30 15:59:50 - workflow-executor-child - Starting action [Wait for]
2022/11/30 15:59:51 - workflow-executor-child - Starting action [Add filenames to result]
2022/11/30 15:59:51 - workflow-executor-child - Finished action [Add filenames to result] (result=[true])
2022/11/30 15:59:51 - workflow-executor-child - Finished action [Wait for] (result=[true])
2022/11/30 15:59:51 - workflow-executor-child - Workflow execution finished
2022/11/30 15:59:51 - workflow-executor-child.hwf.0 - Workflow duration : 1.074 seconds [ 1.074" ]
2022/11/30 15:59:51 - workflow-executor-child - Workflow duration : 1.074 seconds [ 1.074" ]
2022/11/30 15:59:51 - workflow-executor-child.hwf.0 - Finished processing (I=0, O=0, R=3, W=5895, U=0, E=0)
2022/11/30 15:59:51 - Results.0 - Finished processing (I=0, O=0, R=3, W=3, U=0, E=0)
2022/11/30 15:59:51 - Result files.0 - Finished processing (I=0, O=0, R=5892, W=5892, U=0, E=0)
2022/11/30 15:59:51 - workflow-executor-basic - Pipeline duration : 3.41 seconds [ 3.410" ]
2022/11/30 15:59:51 - workflow-executor-basic - ERROR: Error handling writing final pipeline state to location (non-fatal)
2022/11/30 15:59:51 - workflow-executor-basic - ERROR: org.apache.hop.core.exception.HopException:
2022/11/30 15:59:51 - workflow-executor-basic - Error getting execution from Neo4j
2022/11/30 15:59:51 - workflow-executor-basic - org.apache.hop.core.exception.HopValueException:
2022/11/30 15:59:51 - workflow-executor-basic - Unexpected conversion error while converting value [ExecutionTime Integer(15)] to an Integer
2022/11/30 15:59:51 - workflow-executor-basic - class java.lang.String cannot be cast to class java.lang.Long (java.lang.String and java.lang.Long are in module java.base of loader 'bootstrap')
2022/11/30 15:59:51 - workflow-executor-basic -
2022/11/30 15:59:51 - workflow-executor-basic -
2022/11/30 15:59:51 - workflow-executor-basic - at org.apache.hop.neo4j.execution.NeoExecutionInfoLocation.registerData(NeoExecutionInfoLocation.java:878)
2022/11/30 15:59:51 - workflow-executor-basic - at org.apache.hop.pipeline.engines.local.LocalPipelineEngine.stopTransformExecutionInfoTimer(LocalPipelineEngine.java:486)
2022/11/30 15:59:51 - workflow-executor-basic - at org.apache.hop.pipeline.engines.local.LocalPipelineEngine.pipelineCompleted(LocalPipelineEngine.java:465)
2022/11/30 15:59:51 - workflow-executor-basic - at org.apache.hop.pipeline.Pipeline.firePipelineExecutionFinishedListeners(Pipeline.java:1343)
2022/11/30 15:59:51 - workflow-executor-basic - at org.apache.hop.pipeline.Pipeline.lambda$startThreads$0(Pipeline.java:1142)
2022/11/30 15:59:51 - workflow-executor-basic - at org.apache.hop.pipeline.transform.BaseTransform.fireTransformFinishedListeners(BaseTransform.java:2782)
2022/11/30 15:59:51 - workflow-executor-basic - at org.apache.hop.pipeline.transform.BaseTransform.markStop(BaseTransform.java:2771)
2022/11/30 15:59:51 - workflow-executor-basic - at org.apache.hop.pipeline.transform.RunThread.run(RunThread.java:142)
2022/11/30 15:59:51 - workflow-executor-basic - at java.base/java.lang.Thread.run(Thread.java:834)
2022/11/30 15:59:51 - workflow-executor-basic - Caused by: java.lang.RuntimeException: org.apache.hop.core.exception.HopValueException:
2022/11/30 15:59:51 - workflow-executor-basic - Unexpected conversion error while converting value [ExecutionTime Integer(15)] to an Integer
2022/11/30 15:59:51 - workflow-executor-basic - class java.lang.String cannot be cast to class java.lang.Long (java.lang.String and java.lang.Long are in module java.base of loader 'bootstrap')
2022/11/30 15:59:51 - workflow-executor-basic -
2022/11/30 15:59:51 - workflow-executor-basic - at org.apache.hop.neo4j.execution.NeoExecutionInfoLocation.saveNeo4jRowsAndMeta(NeoExecutionInfoLocation.java:1116)
2022/11/30 15:59:51 - workflow-executor-basic - at org.apache.hop.neo4j.execution.NeoExecutionInfoLocation.registerNeo4jData(NeoExecutionInfoLocation.java:945)
2022/11/30 15:59:51 - workflow-executor-basic - at org.apache.hop.neo4j.execution.NeoExecutionInfoLocation.lambda$registerData$11(NeoExecutionInfoLocation.java:876)
2022/11/30 15:59:51 - workflow-executor-basic - at org.neo4j.driver.internal.InternalSession.lambda$transaction$4(InternalSession.java:150)
2022/11/30 15:59:51 - workflow-executor-basic - at org.neo4j.driver.internal.retry.ExponentialBackoffRetryLogic.retry(ExponentialBackoffRetryLogic.java:103)
2022/11/30 15:59:51 - workflow-executor-basic - at org.neo4j.driver.internal.InternalSession.transaction(InternalSession.java:146)
2022/11/30 15:59:51 - workflow-executor-basic - at org.neo4j.driver.internal.InternalSession.writeTransaction(InternalSession.java:124)
2022/11/30 15:59:51 - workflow-executor-basic - at org.neo4j.driver.internal.InternalSession.writeTransaction(InternalSession.java:118)
2022/11/30 15:59:51 - workflow-executor-basic - at org.apache.hop.neo4j.execution.NeoExecutionInfoLocation.registerData(NeoExecutionInfoLocation.java:876)
2022/11/30 15:59:51 - workflow-executor-basic - ... 8 more
2022/11/30 15:59:51 - workflow-executor-basic - Caused by: org.apache.hop.core.exception.HopValueException:
2022/11/30 15:59:51 - workflow-executor-basic - Unexpected conversion error while converting value [ExecutionTime Integer(15)] to an Integer
2022/11/30 15:59:51 - workflow-executor-basic - class java.lang.String cannot be cast to class java.lang.Long (java.lang.String and java.lang.Long are in module java.base of loader 'bootstrap')
2022/11/30 15:59:51 - workflow-executor-basic -
2022/11/30 15:59:51 - workflow-executor-basic - at org.apache.hop.core.row.value.ValueMetaBase.getInteger(ValueMetaBase.java:2347)
2022/11/30 15:59:51 - workflow-executor-basic - at org.apache.hop.core.row.value.ValueMetaInteger.getNativeDataType(ValueMetaInteger.java:40)
2022/11/30 15:59:51 - workflow-executor-basic - at org.apache.hop.neo4j.execution.NeoExecutionInfoLocation.saveNeo4jRowsAndMeta(NeoExecutionInfoLocation.java:1082)
2022/11/30 15:59:51 - workflow-executor-basic - ... 16 more
2022/11/30 15:59:51 - workflow-executor-basic - Caused by: java.lang.ClassCastException: class java.lang.String cannot be cast to class java.lang.Long (java.lang.String and java.lang.Long are in module java.base of loader 'bootstrap')
2022/11/30 15:59:51 - workflow-executor-basic - at org.apache.hop.core.row.value.ValueMetaBase.getInteger(ValueMetaBase.java:2260)
2022/11/30 15:59:51 - workflow-executor-basic - ... 18 more
2022/11/30 15:59:51 - workflow-executor-basic - Execution finished on a local pipeline engine with run configuration 'local'
Priority: 2
Component: Execution Information
Add MDI support to Azure Event Hubs Write
Priority: 3
Component: Metadata
Cleanup XML of transform Load File Input
Priority: 3
Component: Metadata
Sonar considers using clone() (and interface Clonable) as a blocking issue.
A lot of our code already switched over to copy constructors. Let's clean up the remainder.
Priority: 3
Component: Api
Cleanup XML of transform PGP Encrypt Stream
Priority: 3
Component: Metadata
Cleanup XML of transform Row Normaliser
Priority: 3
Component: Metadata
Cleanup XML of transform ETL Metadata Inject
Priority: 3
Component: Metadata
Cleanup XML of transform HTTP
Priority: 3
Component: Metadata
Sonar identified some vulnerability/security issues in the Hop codebase which need to be fixed:
Priority: 2
Component: Api
Cleanup XML of transform Merge Rows
Priority: 3
Component: Metadata
Cleanup XML of transform MongoDB Output
Priority: 3
Component: Metadata
Cleanup XML of transform MongoDB Input
Priority: 3
Component: Metadata
Cleanup XML of transform Get Variables
Priority: 3
Component: Metadata
Cleanup XML of transform PGP Decrypt Stream
Priority: 3
Component: Metadata
2.2.0
openjdk version "11.0.17" 2022-10-18
Linux
The Neo4j Constraint action throws an error against Neo4j 5.x databases:
2022/12/01 09:42:10 - Neo4j Constraint - ERROR: Error creating constraint with cypher [CREATE CONSTRAINT pk_person_id IF NOT EXISTS ON (n:Person) ASSERT n.ID IS UNIQUE ]
2022/12/01 09:42:10 - Neo4j Constraint - ERROR: org.neo4j.driver.exceptions.ClientException: Invalid constraint syntax, ON and ASSERT should not be used. Replace ON with FOR and ASSERT with REQUIRE. (line 1, column 1 (offset: 0))
2022/12/01 09:42:10 - Neo4j Constraint - "CREATE CONSTRAINT pk_person_id IF NOT EXISTS ON (n:Person) ASSERT n.ID IS UNIQUE"
Priority: 1
Component: Graph
2.2.0
openjdk version "11.0.17" 2022-10-18
Linux
File sample.xlsx is modified during the execution of the integration tests. These only read the file.
Apparently we need to open the file as read-only in the POI API to prevent this.
https://stackoverflow.com/questions/39920260/apache-poi-reading-modifies-excel-file
Priority: 1
Component: Transforms
Cleanup XML of PipelineMeta
Priority: 3
Component: Metadata
2.2.0
openjdk version "11.0.17" 2022-10-18
Linux
When editing an environment you can create a new file. However, it's not possible to click on a button:
In other metadata editing scenarios (run config ==> execution location ==> Neo4j connection...) you can notice that when you close the last dialog the wrong dialog is shown on top. There too the shell parenting is not done correctly.
Priority: 2
Component: Hop Gui
SNAPSHOT-20221129
openjdk version "11.0.17" 2022-10-18
Linux
The [https://hop.apache.org/manual/latest/workflow/actions/copyfiles.html](Copy files) action doesn't copy files from the local file system to S3. Copying files from S3 to the local file system does work.
Priority: 2
Component: Actions
Cleanup XML of transform RegEx
Priority: 3
Component: Metadata
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.