Giter Club home page Giter Club logo

waarp-all's Introduction

Waarp All

Waarp provides a secure and efficient open source MFT solution

Waarp Platform is a set of applications and tools specialized in managing and monitoring a high number of transfers in a secure and reliable way.

It relies on its own open protocol named R66, which has been designed to optimize file transfers, ensure the integrity of the data provide ways to integrate transfers in larger business transactions.

Current applications are:

  • Waarp R66: the transfer agent that implements the R66 protocol (source)
  • Waarp Gateway FTP: a service to interconnect FTP- and R66-based file exchanges (source)
  • Waarp Password: a tool to generate the password files used by Waarp R66 and Waarp Gateway FTP (source)
  • Waarp FTP: a fast and extensible FTP server based on Netty (source)

The following applications are deprecated and won't be maintained anymore:

  • Waarp Administrator
  • Waarp XmlEditor

Features

  • Supports Java 6, 8 and 11 in different Jars (JRE 11 is compatible up to JRE 16)
  • Supports multiple databases: Postgresql (recommended), MySQL, MariaDB, Oracle DB, H2
  • Unlimited number of transfers
  • Unlimited number of connections
  • Traceability
  • End-to-end security
  • End-to-end integrity checks
  • Virtualization of access path
  • Encrypted connections with TLS
  • Partners authentication (with login/password and/or strong TLS client authentication)
  • Pre and Post actions, and Error actions, both on sender and receiver sides
  • Works in clusters
  • REST API
  • Support for ICAP servers submission (antivirus for example)
  • Support for S3 access with GET, PUT and DELETE operations
  • And much much more!

Getting Started

Build from source

Just clone the project and use Maven version 3.6.3 minimum to build it (3.8.x for JRE > 11).

Even though Java 6 is supported at runtime, Java 8 or 11 is required to build the project (JDK 11 can be replaced by a newer version, tested until 16 but with maven 3.8.x)

git clone https://github.com/waarp/Waarp-All.git
cd Waarp-All
mvn -P jre11 package

If you want to build unofficial RPM/DEV/TGZ/ZIP and documentation, you can do as the following, ensuring you have already cloned and install using pip the repo for Sphinx template for Waarp code.waarp.fr:2222/waarp/sphinx-template.git with the following packages for Sphinx:

  • sphinx
  • sphinx-autobuild
  • sphinxcontrib-httpdomain
    • Possibly fix the current version 1.6 to 1.7
      • sphinxcontrib/httpdomain.py
        • line 766
        • + app.add_domain(HTTPDomain)
  • sphinxcontrib-openapi
  • sphinx.ext.todo
mvn -P jre11,release package

You can use a JDK 11 (or higher) with jre11 profile, and a JDK 8 with jre8 or jre6 profiles.

mvn -P jre11 package also runs the full test suite, which takes quite some time (for more information about setting up your environment to run the tests, see below).

If you want to build the jars without running the tests, use the following command instead:

mvn -p jre11 -D skipTests package

After that, you will find the JARs for each module and application in their respective target directory (ex: ./WaarpR66/target/WaarpR66-*.jar)

And moreover, you will have also a shaded jar that include all dependencies, except Oracle JDBC for licence issue, under name WaarpR66-X.Y.Z-jar-with-depencencies.jar, or equivalent for all runnable jars.

A detailed documentation (in French) is also available in the directory doc/releasing.md.

Installation

Detailed instructions are provided in the documentation (in French only for now) to install Waarp R66 from portable archives ad OS packages (Deb, RPM).

Run the tests

Even though Java 6 is supported at runtime, Java 8 is required to build the project

The full test suite (including integration tests on several databases) requires Docker.

From the root of the project, run the command:

mvn -P jre11 test

Build the documentation

Documentation is managed with Sphinx. To build it, you need Python:

cd doc/waarp-r66

# create a virtual environment (you need to do this only once
python -m virtualenv .venv

# Enter the virtual environment
. .venv/bin/activate

# Install the requirements
pip install -r requirement.txt

# Build the doc. After that, it will be available in the directory build/html
make html

Note that documentations are built in final module named WaarpPackaging and WaarpPackaging6.

Support

Waarp provides professional support and services.

You can also have community support on our forum.

Related Projects

Those projects are part of the Waarp Platform, but are managed seperately:

Waarp solutions are built on the amazing Netty framework.

License

This project is distributed under the terms of the GNU GPLv3 License

waarp-all's People

Contributors

amottier avatar bcarlin avatar fredericbregier avatar jgowder avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

waarp-all's Issues

FtpClient2TLSNativeTest test failed

I'm trying to build Wraap from source for and with Java 11. I'm doing that using GitLab CI (see my gitlab-ci.yml). It basically get the branch v3.6 and try to build it by running mvn --batch-mode --errors --fail-at-end --show-version -DinstallAtEnd=true -DdeployAtEnd=true -P jre11 clean org.jacoco:jacoco-maven-plugin:prepare-agent test org.jacoco:jacoco-maven-plugin:report (with Maven 3.8.1 and OpenJDK 11).

Test failed but the logs does not give much information about the cause of the failure:

08:32:37.811 [main] WARN  org.waarp.ftp.client.AbstractFtpClient - org.waarp.ftp.client.AbstractFtpClient.stopServer(AbstractFtpClient.java:96) : Will shutdown from client
08:32:38.084 [Command-5] WARN  org.waarp.ftp.core.command.internal.INTERNALSHUTDOWN - org.waarp.ftp.core.command.internal.INTERNALSHUTDOWN.exec(INTERNALSHUTDOWN.java:82) : Shutdown...
SHUTDOWN: System shutdown
08:32:38.089 [Command-5] WARN  org.waarp.ftp.core.utils.FtpChannelUtils - org.waarp.ftp.core.utils.FtpChannelUtils.exit(FtpChannelUtils.java:343) : Exit: Give a delay of 500 ms
08:32:38.090 [main] WARN  org.waarp.ftp.client.AbstractFtpClient - org.waarp.ftp.client.AbstractFtpClient.stopServer(AbstractFtpClient.java:120) : Will stop server
ERROR Exit System
477833 [ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.953 s <<< FAILURE! - in org.waarp.ftp.client.FtpClient2TLSNativeTest
477834 [ERROR] org.waarp.ftp.client.FtpClient2TLSNativeTest.test0_0FtpApacheZ  Time elapsed: 0.373 s  <<< FAILURE!
java.lang.AssertionError: No KO expected:<0> but was:<1>
	at org.junit.Assert.fail(Assert.java:89)

You can view the CI job log here: https://gitlab.ow2.org/ow2/oscar/waarp/-/jobs/63217

Maybe this is related to #99 ?

Usability issues in R66 admin interface

In the "Transfers > Listing" and "Transfers > Cancel/restart" pages:

  • sorting does not work for the specialid column
  • The "clear" button does not work if the list has already been filtered (a click on "clear" works just after having typed text though)

Test execution hang in test1_Ftp4JSimple

In my attempt to build Waarp from sources available in https://github.com/fredericBregier/Waarp-All/tree/v3.6.0 I'm now facing an issue related to test execution.

Here is the latest logs I got when Waarp Ftp R66 client tests are executed:

Starting test: test1_Ftp4JSimple
10:29:26.614 [main] WARN  org.waarp.ftp.client.AbstractFtpClient - org.waarp.ftp.client.AbstractFtpClient.testFtp4J(AbstractFtpClient.java:172) : First connexion
10:29:26.902 [main] WARN  org.waarp.ftp.client.AbstractFtpClient - org.waarp.ftp.client.AbstractFtpClient.testFtp4J(AbstractFtpClient.java:182) : Create Dirs
.
10:29:26.923 [main] WARN  org.waarp.ftp.client.AbstractFtpClient - org.waarp.ftp.client.AbstractFtpClient.testFtp4J(AbstractFtpClient.java:193) : Feature commands
SITE: true
SITE CRC: true
CRC: true
MD5: true
SHA1: true
DIGEST: true
10:29:26.944 [main] WARN  org.waarp.ftp.client.AbstractFtpClient - org.waarp.ftp.client.AbstractFtpClient.testFtp4J(AbstractFtpClient.java:201) : Logout
10:29:26.947 [main] WARN  org.waarp.ftp.client.AbstractFtpClient - org.waarp.ftp.client.AbstractFtpClient.testFtp4J(AbstractFtpClient.java:205) : Will start 1 Threads
10:29:27.117 [pool-1-thread-1] WARN  org.waarp.ftp.client.transaction.FtpClientThread - org.waarp.ftp.client.transaction.FtpClientThread.run(FtpClientThread.java:123) : T0 change mode passive

On another attempt I also got the following logs:

15:24:33.403 [pool-1-thread-1] WARN  org.waarp.ftp.client.transaction.FtpClientThread - org.waarp.ftp.client.transaction.FtpClientThread.run(FtpClientThread.java:123) : T0 change mode passive
...
15:24:33.863 [Data-3] WARN  org.waarp.ftp.core.data.handler.DataNetworkHandler - org.waarp.ftp.core.data.handler.DataNetworkHandler.channelActive(DataNetworkHandler.java:227) : DataChannel immediately closed since RETR is not ok at startup
15:24:34.062 [Data-4] WARN  org.waarp.ftp.core.data.handler.DataNetworkHandler - org.waarp.ftp.core.data.handler.DataNetworkHandler.channelActive(DataNetworkHandler.java:227) : DataChannel immediately closed since RETR is not ok at startup
15:24:34.067 [pool-1-thread-1] ERROR org.waarp.ftp.client.WaarpFtp4jClient - org.waarp.ftp.client.WaarpFtp4jClient.transferFile(WaarpFtp4jClient.java:505) : null: Retrieve operation not allowed
15:24:35.409 [Data-3] WARN  org.waarp.ftp.core.data.FtpTransferControl - org.waarp.ftp.core.data.FtpTransferControl.abortTransfer(FtpTransferControl.java:603) : Abort problem : No Command currently running
15:24:35.412 [Data-3] WARN  org.waarp.ftp.core.data.FtpTransferControl - org.waarp.ftp.core.data.FtpTransferControl.setTransferAbortedFromInternal(FtpTransferControl.java:700) : Command cancelled
15:24:35.592 [Data-4] WARN  org.waarp.ftp.core.data.FtpTransferControl - org.waarp.ftp.core.data.FtpTransferControl.abortTransfer(FtpTransferControl.java:603) : Abort problem : No Command currently running

I guess that the hang is probably related to awaitTermination that is configured to wait for 12000 seconds (200 minutes).

Also I think the calculation System.currentTimeMillis() - 120000 * 60; might be wrong because it should be * 1000 instead of * 60 to convert from seconds to milliseconds.

If I can provide more information in order to help debugging please let me know.

[R66] Listing pages are broken in the admin interface

FollowId addition breaks the json included in the html source:

2020-07-31-114046_487x74_scrot

I can only guess it comes from DbTaskRunner.getJson() which uses arrayNode.toString() to generate JSON.

but according to jackson doc:

Method that will produce developer-readable representation of the node; which
may or may not be as valid JSON. If you want valid JSON output (or output
formatted using one of other Jackson supported data formats) make sure to use
ObjectMapper or ObjectWriter to serialize an instance, for example:

String json = objectMapper.writeValueAsString(rootNode);

I imagine it also would remove the need to do "manual" cleaning of the json generated.

The JRE never exits when a DirectTransfer receives a "HostId not allowed" error

The scenario to reproduce the error is the following :

an R66 client sends a file to an R66 server, which has no entry for the client in its partner database, with a DirectTransfer (I suppose it is the same for other types of transfers).
The client receives an error packet A, closes its network and database connections, but never exits.

I attached the debug logs for such a transfer.

Can it be included in the 3.3.2 version ?

client.log

Allow absolute path in <directory> config

Currently paths in the directory group of the config files are used as follow:
serverhome + value (value among in, out, work, arch, conf)
This prevent the use of absolute path in the value part of the configuration.
A solution would be to modify the getters of the configuration java like this:

public String getConfigPath() {
  File conf = new File(configPath);
  if (conf.isAbsolute()) {
    return configPath;
  }
  return getBaseDirectory() + configPath;
}

@fredericBregier what do you think ?

Proposition for unique transfer ID

Given two partners exchanging transfers if one of the transfer IDs is reset (DB purge, ...) new transfers cannot be added as the partners who did not reset its IDs will refuse with the reason the TransferID already exists for the given owner/requester/requested.

A solution would be to use unix epoch in nanoseconds for transfer IDs (instead of DB increments) , this would ensure uniqueness of IDs given a unique partners does not overload with transfers. The IDs would have a expiration date in year 2554 which should be a comfortable margin.

Escaping issue in rules whose path contain "\"

Steps to reproduce

  1. Go to the rule page in the admin interface and select a rule to edit.
  2. Change a path to contain \ (for example C:\Data\)
    2020-04-30-105750_264x39_scrot
  3. Update the rule.
  4. In the resulting list, the path contains \\ instead of \
    2020-04-30-105806_432x85_scrot
  5. When re-editing the rule, the \\ are kept
    2020-04-30-105839_277x49_scrot
  6. Saving the rule without fixing the paths will re-double the \, and so on
    2020-04-30-105853_412x82_scrot

The output of client does not escape values according to the target format

Some output values can contain XML-like data.
When we ask the command to format the output in XML with -xml, this produce a valid XML structure, but it does not match the intent.

For example, with the output for a failed transfer, I have the following XML (the formatting has been added by me to highlight what I mean):

<xml>
  <command>DirectTransfer</command>
  <args>etc/conf.d/server2/client.xml -file world.txt -to server1 -rule error -xml </args>
  <status>2</status>
  <statusTxt>Transfer in status: FAILED with no Transfer Id</statusTxt>
  <remote>server1</remote>
  <error>
    ErrorPacket:(3:E) PreTask in error: Runner is failed: 
    <STATUS>-1</STATUS>
    <ERROR>will fail </ERROR>
  </error>
</xml>

Though it is valid, <STATUS> and <ERROR> are part of <error> and not tags of the main XML document.
The output should use entity refs to protect the content of the <error> message:

<xml>
  <command>DirectTransfer</command>
  <args>etc/conf.d/server2/client.xml -file world.txt -to server1 -rule error -xml </args>
  <status>2</status>
  <statusTxt>Transfer in status: FAILED with no Transfer Id</statusTxt>
  <remote>server1</remote>
  <error>
    ErrorPacket:(3:E) PreTask in error: Runner is failed: 
    &lt;STATUS&gt;-1&lt;/STATUS&gt;
    &lt;ERROR&gt;will fail &lt;/ERROR&gt;
  </error>
</xml>

edit:
I realised by reviewing #67 that the same problem exists for other output formats.

Here is an example with -csv. The culprits are the semicolon (;) between "will" and "fail" and the new line which are not protected, hence adding an extra column in the first line, and a second record in the second line:

command;args;status;statusTxt;remote;error
DirectTransfer;etc/conf.d/server2/client.xml -file world.txt -to server1 -rule error -csv ;2;Transfer in status: FAILED with no Transfer Id;server1;ErrorPacket:(3:E) PreTask in error: Runner is failed: <STATUS>-1</STATUS><ERROR>will ;fail
always

The output should be:

command;args;status;statusTxt;remote;error
DirectTransfer;etc/conf.d/server2/client.xml -file world.txt -to server1 -rule error -csv ;2;Transfer in status: FAILED with no Transfer Id;server1;"ErrorPacket:(3:E) PreTask in error: Runner is failed: <STATUS>-1</STATUS><ERROR>will ;fail
always"

Error during ExplainRequestIT test execution

I'm trying to build Waarp using Maven 3.9.1 (using Maven wrapper) and OpenJDK 11.0.18 on Debian Bullseye.

I'm running ./mvnw -P jre11 clean install and get the following error in the log (I can add the full build log if useful):

11:42:51.725 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.setUp3DbBeforeClass(ExplainRequestIT.java:279) : File /home/antoine/workspace/projects/waarp/Waarp-All/WaarpR66/src/test/resources/it/scenario_big_file_limitbdw/R1/conf/server_1_SQLDB.xml exists? true
11:42:51.726 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.setUp3DbBeforeClass(ExplainRequestIT.java:283) : Dir /home/antoine/workspace/projects/waarp/Waarp-All/WaarpR66 exists? true
11:42:51.767 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.getServerConfigFile(ExplainRequestIT.java:204) : Build configFile
jdbc:postgresql://localhost:32918/test?loggerLevel=OFF
org.postgresql.Driver
postgresql
11:42:51.779 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.getServerConfigFile(ExplainRequestIT.java:256) : Config file created at /tmp/R66/scenario_big_file_limitbdw/R1/conf/server.xml
11:42:51.780 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.getServerConfigFile(ExplainRequestIT.java:260) : Copy from /tmp/R66/scenario_big_file_limitbdw/R1/conf/server.xml to /tmp/config.xml
11:42:51.781 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.getServerConfigFile(ExplainRequestIT.java:264) : Copy from /tmp/R66/scenario_big_file_limitbdw/R1/conf/server.xml to /tmp/config.xml and return /tmp/R66/scenario_big_file_limitbdw/R1/conf/server.xmlExplainRequestIT
11:42:51.781 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.setUp3DbBeforeClass(ExplainRequestIT.java:290) : Config file found /tmp/R66/scenario_big_file_limitbdw/R1/conf/server.xml
11:42:51.782 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.setUp3DbBeforeClass(ExplainRequestIT.java:308) : Could not find /tmp/config.xml or /tmp/R66/scenario_big_file_limitbdw/R1/conf/server.xml
11:42:51.782 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.initiateDb(ExplainRequestIT.java:327) : File /tmp/R66/scenario_big_file_limitbdw/R1/conf/server.xml exists? true
11:42:51.783 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.initiateDb(ExplainRequestIT.java:330) : File /home/antoine/workspace/projects/waarp/Waarp-All/WaarpR66/src/test/resources/it/scenario_big_file_limitbdw/OpenR66-authent.xml exists? true
ERROR org.waarp.openr66.server.ServerInitDatabase  /tmp/R66/scenario_big_file_limitbdw/R1/conf/server.xml -initdb -dir /home/antoine/workspace/projects/waarp/Waarp-All/WaarpR66/src/test/resources/it/scenario_big_file_limitbdw/R1/conf -auth /home/antoine/workspace/projects/waarp/Waarp-All/WaarpR66/src/test/resources/it/scenario_big_file_limitbdw/OpenR66-authent.xml return code: 1 pid: 99999999
11:42:52.304 [main] WARN  org.waarp.openr66.protocol.configuration.Configuration - org.waarp.openr66.protocol.configuration.Configuration.setTransferGuid(Configuration.java:2665) : Startup: Transfer Id will use SQL Sequence
11:42:52.334 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.initiateDb(ExplainRequestIT.java:327) : File /home/antoine/workspace/projects/waarp/Waarp-All/WaarpR66/target/test-classes/it/scenario_big_file_limitbdw/R2/conf/server_2.xml exists? true
11:42:52.335 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.initiateDb(ExplainRequestIT.java:330) : File /home/antoine/workspace/projects/waarp/Waarp-All/WaarpR66/target/test-classes/it/scenario_big_file_limitbdw/OpenR66-authent.xml exists? true
ERROR org.waarp.openr66.server.ServerInitDatabase  /home/antoine/workspace/projects/waarp/Waarp-All/WaarpR66/target/test-classes/it/scenario_big_file_limitbdw/R2/conf/server_2.xml -initdb -dir /home/antoine/workspace/projects/waarp/Waarp-All/WaarpR66/target/test-classes/it/scenario_big_file_limitbdw/R2/conf -auth /home/antoine/workspace/projects/waarp/Waarp-All/WaarpR66/target/test-classes/it/scenario_big_file_limitbdw/OpenR66-authent.xml return code: 1 pid: 99999999
11:42:52.631 [main] WARN  org.waarp.common.utility.WaarpLoggerListener - org.waarp.common.logging.AbstractWaarpLogger.log(AbstractWaarpLogger.java:168) : 
BUILD SUCCESSFUL
Total time: 0 seconds
Find server file: /home/antoine/workspace/projects/waarp/Waarp-All/WaarpR66/target/test-classes/it/scenario_big_file_limitbdw/R2/conf/server_2.xml
ERROR org.waarp.openr66.server.R66Server  /home/antoine/workspace/projects/waarp/Waarp-All/WaarpR66/target/test-classes/it/scenario_big_file_limitbdw/R2/conf/server_2.xml return code: 0 pid: 390356
11:42:52.912 [main] WARN  org.waarp.openr66.protocol.junit.TestAbstractMinimal - org.waarp.openr66.protocol.it.ExplainRequestIT.startServer(ExplainRequestIT.java:360) : Start Done: 390356
11:42:53.438 [main] ERROR org.waarp.common.database.DbRequest - org.waarp.common.database.DbRequest.select(DbRequest.java:127) : [server1] SQL Exception Request:SELECT * FROM  CONFIGURATION  ERROR: relation "configuration" does not exist
  Position : 16
11:42:53.450 [main] ERROR org.waarp.common.database.DbConstant - org.waarp.common.database.DbConstant.error(DbConstant.java:69) : [server1] SQLException: ERROR: relation "configuration" does not exist
  Position : 16 SQLState: 42P01 VendorError: 0
11:42:53.452 [main] WARN  org.waarp.openr66.configuration.FileBasedConfiguration - org.waarp.openr66.configuration.FileBasedConfiguration.loadDatabase(FileBasedConfiguration.java:1670) : [server1] Database is not yet initiated: run ServerInitDatabase -initdb first : SQL Exception Request:SELECT * FROM  CONFIGURATION 
11:42:54.272 [main] WARN  org.waarp.openr66.configuration.FileBasedConfiguration - org.waarp.openr66.configuration.FileBasedConfiguration.loadServerParam(FileBasedConfiguration.java:491) : [server1] Multiple Monitor configuration unactive
11:42:54.356 [main] ERROR org.waarp.openr66.configuration.FileBasedConfiguration - org.waarp.openr66.configuration.FileBasedConfiguration.setConfigurationServerFromXml(FileBasedConfiguration.java:2324) : [server1] Cannot find Authentication for current host
11:42:54.356 [main] ERROR org.waarp.openr66.server.R66Server - org.waarp.openr66.server.R66Server.initialize(R66Server.java:84) : [server1] Needs a correct configuration file as first argument
11:42:54.357 [main] ERROR org.waarp.openr66.server.R66Server - org.waarp.openr66.server.R66Server.main(R66Server.java:66) : [server1] Cannot start Server OpenR66 for server1
ERROR Cannot start Server OpenR66 for server1
####################### NO R66 SERVER RUNNING ###################
ERROR Exit end of Shutdown
[INFO] Running org.waarp.openr66.protocol.it.ExplainRequestIT
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.023 s <<< FAILURE! - in org.waarp.openr66.protocol.it.ExplainRequestIT
[ERROR] org.waarp.openr66.protocol.it.ExplainRequestIT  Time elapsed: 0.013 s  <<< ERROR!
java.lang.NullPointerException
	at [email protected]/org.waarp.openr66.protocol.it.ExplainRequestIT.setUpBeforeClass(ExplainRequestIT.java:189)
	at [email protected]/org.waarp.openr66.protocol.it.ExplainRequestIT.setup(ExplainRequestIT.java:152)

What is weird is that line 189 in ExplainRequestIT.java is setUpBeforeClassClient(); so I don't understand how it can trigger a NullPointerException.

[R66] XMLRuleDAO select, find and exists operations only work on internal cache

To avoid lookups on disk every time a Rule is selected, XMLRuleDAO has an internal cache, which is good.

However, select, find and exists operations only works on the internal cache (as opposed to XMLHostDAO which falls bach on disk search if nothing id found in the internal cache).

As the internal cache if only filled when getAll() is called, there are some situations where select, find and exists never find rules that do exists.
As a side effect, the file watcher is not working without database: SpooledDirectoryTransfer (and its execution tree) never runs XMLRuleDAO.getAll() durining its initialization, so the individual lookups for each spooldirectory's rule always fail.

[WaarpR66] Possible reuse of closed database connection in HttpResponsiveSslHandler

Here are the logs:

04/05/2020/16:23:00.498 ERROR [org.waarp.common.database.DbPreparedStatement] [HttpWorker-27] org.waarp.common.database.DbPreparedStatement.createPrepareStatement(DbPreparedStatement.java:235) : [MYSERVER] SQL Exception createPreparedStatement from jdbc:postgresql://127.0.0.1:5432/waarp_r66:SELECT GLOBALSTEP,GLOBALLASTSTEP,STEP,RANK,STEPSTATUS,RETRIEVEMODE,FILENAME,ISMOVED,IDRULE,BLOCKSZ,ORIGINALNAME,FILEINFO,TRANSFERINFO,MODETRANS,STARTTRANS,STOPTRANS,INFOSTATUS,UPDATEDINFO,OWNERREQ,REQUESTER,REQUESTED,SPECIALID FROM RUNNER WHERE SPECIALID >= ? AND SPECIALID <= ? AND IDRULE LIKE '%%' AND ( REQUESTED LIKE '%%' OR REQUESTER LIKE '%%' ) AND OWNERREQ = 'MYSERVER' ORDER BY STARTTRANS DESC LIMIT 48 This connection has been closed.
04/05/2020/16:23:00.498 ERROR [org.waarp.common.database.DbSession] [HttpWorker-27] org.waarp.common.database.DbSession.error(DbSession.java:263) : [MYSERVER] SQLException: This connection has been closed. SQLState: 08003VendorError: 0
04/05/2020/16:23:00.498 WARN [org.waarp.openr66.protocol.http.adminssl.HttpResponsiveSslHandler] [HttpWorker-27] org.waarp.openr66.protocol.http.adminssl.HttpResponsiveSslHandler.setDbTaskRunnerJsonData(HttpResponsiveSslHandler.java:208) : [MYSERVER] OpenR66 Web Error SQL Exception createPreparedStatement: SELECT GLOBALSTEP,GLOBALLASTSTEP,STEP,RANK,STEPSTATUS,RETRIEVEMODE,FILENAME,ISMOVED,IDRULE,BLOCKSZ,ORIGINALNAME,FILEINFO,TRANSFERINFO,MODETRANS,STARTTRANS,STOPTRANS,INFOSTATUS,UPDATEDINFO,OWNERREQ,REQUESTER,REQUESTED,SPECIALID FROM RUNNER WHERE SPECIALID >= ? AND SPECIALID <= ? AND IDRULE LIKE '%%' AND ( REQUESTED LIKE '%%' OR REQUESTER LIKE '%%' ) AND OWNERREQ = 'MYSERVER' ORDER BY STARTTRANS DESC LIMIT 48

The same error happens with rules, hosts, export, etc., always in HttpResponsiveSslHandler

[R66] the formatting of the output of the send commands is broken

It seems there is no support for the formatting options of the send comands: -xml, -json, -property, etc.

➜ ./bin/waarp-r66client.sh client1 send -property -file ./file.dat -to server1 -rule push
usage: Transfer
 -block <arg>   Specify the block size
 -delay <arg>   Specify the delay time as an epoch time or '+' a delay in
                ms
 -file <arg>    Specify the file path to operate on
 -id <arg>      Specify the id of transfer
 -info <arg>    Specify the transfer information
 -logWarn       Specify to log final result as Warn if OK
 -md5           Specify the option to have a hash computed for the
                transfer
 -nofollow      Specify if the transfer should not integrate a FOLLOW id
 -nolog         Specify to not log anything included database once the
                transfer is done
 -notlogWarn    Specify to log final result as Info if OK
 -rule <arg>    Specify the Rule
 -start <arg>   Specify the start time as yyyyMMddHHmmss
 -to <arg>      Specify the requested Host
Wrong initialization

[R66] LogExtendedExport does not work anymore

The command was functional in 3.0.x versions, but on 3.4 dev branch, it does not seem to send the request to the server.

Here are the logs when i try this command:

14:51:28.516 [main] WARN org.waarp.openr66.protocol.configuration.Configuration - org.waarp.openr66.protocol.configuration.Configuration.pipelineInit(Configuration.java:766) : [server2] Server Thread: 8 Client Thread: 80 Runner Thread: 1000
14:51:28.517 [main] INFO org.waarp.openr66.protocol.configuration.Configuration - org.waarp.openr66.protocol.configuration.Configuration.pipelineInit(Configuration.java:772) : [server2] Current launched threads: 8
14:51:28.558 [main] DEBUG org.waarp.openr66.server.LogExtendedExport - org.waarp.openr66.server.LogExtendedExport.run(LogExtendedExport.java:187) : [server2] ExtendedLogCommand: {"@class":"org.waarp.openr66.protocol.localhandler.packet.json.LogJsonPacket","comment":null,"requestUserPacket":0,"purge":false,"clean":false,"statuspending":false,"statustransfer":false,"statusdone":true,"statuserror":false,"rule":null,"request":null,"start":null,"stop":1594850400000,"startid":null,"stopid":null}
14:51:28.558 [main] INFO org.waarp.openr66.server.LogExtendedExport - org.waarp.openr66.server.LogExtendedExport.run(LogExtendedExport.java:189) : [server2] *** before tryconnect
14:51:28.559 [main] DEBUG org.waarp.openr66.protocol.networkhandler.NetworkTransaction - org.waarp.openr66.protocol.networkhandler.NetworkTransaction.closeAll(NetworkTransaction.java:1371) : [server2] close All Network Channels
14:51:28.761 [main] DEBUG org.waarp.common.database.DbAdmin - org.waarp.common.database.DbAdmin.closeAllConnection(DbAdmin.java:396) : [server2] Close (all) Db Conn: aeaaaaaap3ilnhaiplsac42xvyaqkaaaae

From what I can see, and with a few more testing, the code for the command
(LogExtendedExport.run()) is executed until line
189
:

LocalChannelReference localChannelReference =
        AbstractTransfer.tryConnect(host, newFuture, networkTransaction);

I don't know why, tryConnect does not seem to be executed (its first line is a logger call, which does not appear in the logs).

Waarp Password does not work

Hi,

On CentOS and Windows, waarp-password.sh / waarp-password.bat does not work correctly.

./bin/waarp-password.sh -ki ../waarp-certs/cryptokey.des -pwd adminpwd -po outputpwd.txt
Error: Could not find or load main class org.waarp.uip.WaarpPassword

After checking, the classe org.waarp.uip.WaarpPassword is not in share/lib/WaarpR66-3.3.0-jar-with-dependencies.jar
WaarpR66 Version : 3.3 / 3.2

Best Regards
TM

Build failed due to missing waarp theme for Sphinx

I'm trying to build (./mvnw -P jre11 -D skipTests package using Maven 3.8.1, OpenJDK 11, Debian 10) from branch v3.6.0 from fredericBregier fork (https://github.com/fredericBregier/Waarp-All/tree/v3.6.0) and I get an error related to Sphinx documentation generation:

[INFO] --- exec-maven-plugin:3.0.0:exec (SphinxHtml) @ WaarpPackaging ---
Sphinx v4.1.2 en cours d'exécution
chargement des traductions [fr]... fait
création du dossier de destinataire... fait

Theme error:
Le thème nommé 'waarp' n'a pas été trouvé (le fichier theme.conf est-il bien présent ?)
make: *** [Makefile:20: html] Error 2
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 2 (Exit value: 2)

Based on Sphinx configuration file it seems that theme waarp is referenced but could not be found.

Transfer listing does not load if a rule has been deleted

In the Admin HTTPS interface, if a transfer uses a rule that has been removed, the transfer listing does not show the transfer, but an error message.

Steps to reproduce

  • create a rule "foo"
  • do a transfer with the rule "foo"
  • delete the rule "foo"
  • go to the admin interface and navigate to the transfer listing page

Expected outcome

the transfer that used the rule "foo" shows in the list of transfers

Actual outcome

The https interface shows this error:

!ErrorCode.17!: Rule cannot be found for DbTaskRunner: {"@model":"DbTaskRunner","SPECIALID":-9223372036854775795,"RETRIEVEMODE":false,"IDRULE":"test","MODETRANS":1,"FILENAME":"/in/-9223372036854775795_3185345321839916308_foo.txt","ORIGINALNAME":"/out/foo.txt","FILEINFO":"noinfo","ISMOVED":false,"BLOCKSZ":65536,"OWNERREQ":"server","REQUESTER":"client","REQUESTED":"server","TRANSFERINFO":"{\"ORIGINALSIZE\":12}","GLOBALSTEP":4,"GLOBALLASTSTEP":4,"STEP":0,"STEPSTATUS":"O ","INFOSTATUS":"O ","RANK":1,"STARTTRANS":1587989702508,"STOPTRANS":1587989702609,"UPDATEDINFO":6,"ORIGINALSIZE":-1}

The logs contains only this warning :

28/04/2020/09:58:57.783 WARN [org.waarp.openr66.protocol.http.adminssl.HttpResponsiveSslHandler] [HttpWorker-12] org.waarp.openr66.protocol.http.adminssl.HttpResponsiveSslHandler.setDbTaskRunnerJsonData(HttpResponsiveSslHandler.java:208) : [server] OpenR66 Web Error Rule cannot be found for DbTaskRunner: {"@model":"DbTaskRunner","SPECIALID":-9223372036854775795,"RETRIEVEMODE":false,"IDRULE":"test","MODETRANS":1,"FILENAME":"/in/-9223372036854775795_3185345321839916308_foo.txt","ORIGINALNAME":"/out/foo.txt","FILEINFO":"noinfo","ISMOVED":false,"BLOCKSZ":65536,"OWNERREQ":"server","REQUESTER":"client","REQUESTED":"server","TRANSFERINFO":"{\"ORIGINALSIZE\":12}","GLOBALSTEP":4,"GLOBALLASTSTEP":4,"STEP":0,"STEPSTATUS":"O  ","INFOSTATUS":"O  ","RANK":1,"STARTTRANS":1587989702508,"STOPTRANS":1587989702609,"UPDATEDINFO":6,"ORIGINALSIZE":-1} 

[R66] ClassCastException thrown by EXECOUTPUT tasks the the command failed

Here is the full stacktrace:

30/07/2020/12:47:31.051 DEBUG [org.waarp.openr66.protocol.localhandler.LocalServerHandler] [Handler-1] org.waarp.openr66.protocol.localhandler.LocalServerHandler.exceptionCaught(LocalServerHandler.java:370) : [server1] Exception and isFinished: false
java.lang.ClassCastException: java.lang.Thread cannot be cast to org.waarp.openr66.context.task.AllLineReader
	at org.waarp.openr66.context.task.ExecOutputTask.finalizeFromError(ExecOutputTask.java:204)
	at org.waarp.openr66.context.task.AbstractExecTask$ExecuteCommand.invoke(AbstractExecTask.java:552)
	at org.waarp.openr66.context.task.ExecOutputTask.run(ExecOutputTask.java:127)
	at org.waarp.openr66.database.data.DbTaskRunner.runNextTask(DbTaskRunner.java:2718)
	at org.waarp.openr66.database.data.DbTaskRunner.runNext(DbTaskRunner.java:2796)
	at org.waarp.openr66.database.data.DbTaskRunner.run(DbTaskRunner.java:2844)
	at org.waarp.openr66.database.data.DbTaskRunner.finalizeTransfer(DbTaskRunner.java:3035)
	at org.waarp.openr66.context.R66Session.setFinalizeTransfer(R66Session.java:953)
	at org.waarp.openr66.protocol.localhandler.TransferActions.endTransferR(TransferActions.java:1168)
	at org.waarp.openr66.protocol.localhandler.TransferActions.fromEndTransferSToTransferR(TransferActions.java:1048)
	at org.waarp.openr66.protocol.localhandler.TransferActions.endTransfer(TransferActions.java:1033)
	at org.waarp.openr66.protocol.localhandler.LocalServerHandler.channelRead1(LocalServerHandler.java:265)
	at org.waarp.openr66.protocol.localhandler.LocalServerHandler.channelRead0(LocalServerHandler.java:107)
	at org.waarp.openr66.protocol.networkhandler.NetworkServerHandler.channelRead0(NetworkServerHandler.java:392)
	at org.waarp.openr66.protocol.networkhandler.NetworkServerHandler.channelRead0(NetworkServerHandler.java:63)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61)
	at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.lang.Thread.run(Thread.java:748)

Steps to reproduce :

create a rule that starts the script fail.sh, which always ends in error :

Content of the rule:

<rule>
  <idrule>error</idrule>
  <comment>The default rule send rule (extended version)</comment>
  <mode>1</mode>
  <rpretasks>
    <tasks>
    </tasks>
  </rpretasks>
  <rposttasks>
    <tasks>
      <task>
        <type>EXECOUTPUT</type>
        <path>./bin/fail.sh</path>
        <delay>60000</delay>
      </task>
    </tasks>
  </rposttasks>
  <rerrortasks>
    <tasks>
    </tasks>
  </rerrortasks>
  <spretasks>
    <tasks>
    </tasks>
  </spretasks>
  <sposttasks>
    <tasks>
    </tasks>
  </sposttasks>
  <serrortasks>
    <tasks>
    </tasks>
  </serrortasks>
</rule>

content of the script:

#!/usr/bin/env bash

echo will fail
exit 34
        

testSysmonGetLoadAverage failed

In our CI execution of testSysmonGetLoadAverage failed with error message: java.lang.AssertionError: Max > current: 1.0 >? 1.0. The CI is building the v3.6.1 branch from https://github.com/fredericBregier/
You can see the details here: https://gitlab.ow2.org/ow2/oscar/waarp/-/jobs/65315#L424
CI passed successfully on Aug 24, 2021 and failed on Aug 29, 2021 so based on the list of commits I'm assuming that the issue might have been introduced by the commit 086c6b7.

Will Waarp support the PeSIT protocol?

Hello,

I saw on Waarp-gateway's GitLab repository a branch called "Add sync & seek methods to the pipeline" created by a user who plans to add compatibility for the PeSIT protocol.

Since this branch hasn't been updated for six months, I wanted to know if the PeSIT protocol is still planned for Waarp or not. If it's the case, is there a roadmap or a release date?

[R66] Commander might be blocked in some cases

The commander lists the tansfers to start, and loops on them.
In the loop, there are two branches with continue statements, but the loop index is not incremented before the continue. This results in infinite loops when a transfer enters one of these branches.

while (i < tasks.length) {
if (WaarpShutdownHook.isShutdownStarting()) {
logger.info("Will not start transfers, server is in shutdown.");
return;
}
final DbTaskRunner taskRunner = tasks[i];
logger.debug("get a task: {}", taskRunner);
// Launch if possible this task
final String key =
taskRunner.getRequested() + ' ' + taskRunner.getRequester() +
' ' + taskRunner.getSpecialId();
if (Configuration.configuration.getLocalTransaction()
.getFromRequest(key) != null) {
// already running
continue;
}
if (taskRunner.isSelfRequested()) {
// cannot schedule a request where the host is the requested host
taskRunner.changeUpdatedInfo(UpdatedInfo.INTERRUPTED);
try {
taskRunner.update();
} catch (final WaarpDatabaseNoDataException e) {
logger.warn("Update failed, no transfer found");
}
continue;
}
internalRunner.submitTaskRunner(taskRunner);
i++;
totalRuns++;
}

Moving the i++ from line 465 to line 443 should fix this)

Build failed if no Maven profile specified on command line

I'm trying to build from source using:

  • AdoptOpenJDK 1.8.0_282
  • Maven 3.6.3
  • Debian Buster 10

mvn -P jre6 clean install lead to a successful build whereas mvn clean install failed (despite jre6 profile being enabled by default) with error:

[ERROR] Failed to execute goal org.codehaus.mojo:flatten-maven-plugin:1.2.5:flatten (flatten) on project WaarpCommon: 11 problems were encountered while building the effective model for Waarp:WaarpCommon:${revision}
[ERROR] [ERROR] 'dependencies.dependency.version' for com.h2database:h2:jar must be a valid version but is '${h2.version}'. @ 
[ERROR] [ERROR] 'dependencies.dependency.version' for org.mariadb.jdbc:mariadb-java-client:jar must be a valid version but is '${mariadb.version}'. @ 
[ERROR] [ERROR] 'dependencies.dependency.version' for mysql:mysql-connector-java:jar must be a valid version but is '${mysql.version}'. @ 
[ERROR] [ERROR] 'dependencies.dependency.version' for Waarp:WaarpDigest:jar must be a valid version but is '${waarp.version}'. @ 
[ERROR] [ERROR] 'dependencies.dependency.version' for commons-io:commons-io:jar must be a valid version but is '${commons.io.version}'. @ 
[ERROR] [ERROR] 'dependencies.dependency.version' for com.google.guava:guava:jar must be a valid version but is '${guava.version}'. @ 
[ERROR] [ERROR] 'dependencies.dependency.version' for org.dom4j:dom4j:jar must be a valid version but is '${dom4j.version}'. @ 
[ERROR] [ERROR] 'dependencies.dependency.version' for com.fasterxml.jackson.core:jackson-databind:jar must be a valid version but is '${jackson-databind.version}'. @ 
[ERROR] [ERROR] 'dependencies.dependency.version' for com.fasterxml.jackson.core:jackson-annotations:jar must be a valid version but is '${jackson.version}'. @ 
[ERROR] [ERROR] 'dependencies.dependency.version' for com.fasterxml.jackson.core:jackson-core:jar must be a valid version but is '${jackson.version}'. @ 
[ERROR] [ERROR] 'dependencies.dependency.version' for com.fasterxml.jackson.dataformat:jackson-dataformat-smile:jar must be a valid version but is '${jackson.version}'. @ 

mvn help:active-profiles and mvn -P jre6 help:active-profiles gave exactly the same output.

If I run Maven with debug enable I can see some differences in flatten maven plugin debug logs:

  • [DEBUG] Resolving dependencies of Waarp:Waarp-All:pom:3.5.2 if profile is specified as Maven command line argument
  • [DEBUG] Resolving dependencies of Waarp:Waarp-All:pom:${revision} if profile is not specified as Maven command line argument

I did a similar test with a very minimal pom.xml that define artifact <version> using a property defined in a profile enabled by default and I experience similar behavior (even with latest flatten plugin version).

It seems that flatten plugin as several issue related to profile and property usage: https://github.com/mojohaus/flatten-maven-plugin/issues?q=is%3Aissue+is%3Aopen+profile

I guess the easy fix for this issue would be to update README.md to include -P jre6 (or maybe jre11 as version 11 is maybe now the most popular one) to avoid this issue.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.