For some reason, on Java 8 in Ubuntu 18.04 running in WSL, the DS 3.x tests are failing for testGetFreePort
. It looks like the JRE or Windows are allowing Java to bind two different server sockets to the same port (port sharing), which violates the test's setup expectations.
The build should succeed with no test failures.
Full failure log
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running TestSuite
Configuring TestNG with: TestNG652Configurator
Test environment:
Java version: 1.8.0_191
Java vendor: Oracle Corporation
JVM name: OpenJDK 64-Bit Server VM
JVM version: 25.191-b12
JVM vendor: Oracle Corporation
JVM info: mixed mode
Java home: /usr/lib/jvm/java-8-openjdk-amd64/jre
OS: Linux 4.4.0-17763-Microsoft amd64
Processors: 4
Max memory: 2837446656
Total memory: 192937984
How to read the progressive status info:
Test duration status: {Total min:sec. Since last status sec.}
Test count status: {# test classes # test methods # test method invocations # test failures}.
TestClass (the class that just completed)
{ 0:00 ( 0s)} { 0c 0m 0i 0f} : starting
{ 0:00 ( 0s)} { 1c 6m 6i 0f} : CertificateTestCase
{ 0:00 ( 0s)} { 2c 9m 9i 0f} : ProductInformationTest
{ 0:00 ( 0s)} { 3c 13m 21i 0f} : SetupCliTestCase
{ 0:00 ( 0s)} { 4c 16m 24i 0f} : DataConfigurationTestCase
T E S T F A I L U R E ! ! !
Failed Test: org.forgerock.opendj.server.setup.model.ListenerSettingsTestCase#testGetFreePort
Failure Cause: java.lang.AssertionError: actual value:<61728> should not be equal to:<61728>
org.fest.assertions.Fail.failure(Fail.java:228)
org.fest.assertions.Fail.fail(Fail.java:218)
org.fest.assertions.Fail.failIfEqual(Fail.java:53)
org.fest.assertions.GenericAssert.isNotEqualTo(GenericAssert.java:228)
org.fest.assertions.IntAssert.isNotEqualTo(IntAssert.java:71)
org.forgerock.opendj.server.setup.model.ListenerSettingsTestCase.testGetFreePort(ListenerSettingsTestCase.java:80)
{ 0:00 ( 0s)} { 5c 20m 28i 1f} : ListenerSettingsTestCase
{ 0:00 ( 0s)} { 6c 38m 46i 1f} : ModelTestCase
{ 0:00 ( 0s)} { 7c 43m 51i 1f} : RuntimeOptionsTestCase
The following unit tests failed:
org.forgerock.opendj.server.setup.model.ListenerSettingsTestCase#testGetFreePort
Include the ant option '-Dtest.failures=true' to rerun only the failed tests.
Wrote full test report to:
/mnt/c/Users/guypa/Documents/WrenProjects/wrends/opendj-server/target/surefire-reports/results.txt
Test classes run interleaved: 0
Final amount of memory in use: 3.0 MB
Final number of threads: 1
Tests run: 51, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.897 sec <<< FAILURE! - in TestSuite
testGetFreePort(org.forgerock.opendj.server.setup.model.ListenerSettingsTestCase) Time elapsed: 0.06 sec <<< FAILURE!
java.lang.AssertionError: actual value:<61728> should not be equal to:<61728>
at org.fest.assertions.Fail.failure(Fail.java:228)
at org.fest.assertions.Fail.fail(Fail.java:218)
at org.fest.assertions.Fail.failIfEqual(Fail.java:53)
at org.fest.assertions.GenericAssert.isNotEqualTo(GenericAssert.java:228)
at org.fest.assertions.IntAssert.isNotEqualTo(IntAssert.java:71)
at org.forgerock.opendj.server.setup.model.ListenerSettingsTestCase.testGetFreePort(ListenerSettingsTestCase.java:80)
Results :
Failed tests:
ListenerSettingsTestCase.testGetFreePort:80 actual value:<61728> should not be equal to:<61728>
Tests run: 51, Failures: 1, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6.884 s
[INFO] Finished at: 2019-04-18T14:51:48Z
[INFO] Final Memory: 27M/252M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test (default-test) on project opendj-server: There are test failures.
[ERROR]
[ERROR] Please refer to /mnt/c/Users/guypa/Documents/WrenProjects/wrends/opendj-server/target/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Environment
- Product version (include GIT commit ID if using a dev build): 3.0.0 @ 3a60e66
- Operating system:
Ubuntu 18.04.1 LTS
Linux 4.4.0-17763-Microsoft #379-Microsoft Wed Mar 06 19:16:00 PST 2019 x86_64 x86_64 x86_64 GNU/Linux
- JRE version:
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-2ubuntu0.18.04.1-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
- Was instance upgraded from a previous version? If so, what version were you using before? No
Additional Notes
Looking into this issue revealed a separate issue with the way that the DS 3.x SDK looks for a free port. A separate issue for that is pending.
Summary
If a collection sub-resource defined in the Rest2LDAP endpoints JSON file is marked "isReadOnly": true
, that endpoint does not appear at all in the API descriptor returned by CREST.
Affected Version
master
(4.0.0-SNAPSHOT
)
Steps to Reproduce
- Alter the
example-v1.json
file, and add the following under resourceTypes/example-v1/subResources
(ensure you don't remove the existing users
and groups
sub-resource definitions):
"all-users": {
"type": "collection",
"dnTemplate": "ou=people,dc=example,dc=com",
"resource": "frapi:opendj:rest2ldap:user:1.0",
"namingStrategy": {
"type": "clientDnNaming",
"dnAttribute": "uid"
},
"isReadOnly": true
2, Alter the Rest2LdapJsonConfiguratorTest
, adding the line System.out.println(prettyPrint(api));
right after the parseJson(prettyPrint(api));
line.
3. Run the test.
4. Observe the console output from the test.
Expected results
The output includes the new all-users
sub-resource, even though it's read-only. Something like the attached expected.json
.
Actual results
The output omits all read-only collection sub-resources. See the attached actual.json
.
Attachments
actual vs. expected JSON.zip
When doing persistent search via LDAPConnectionHandler2
the result entry does not contain EntryChangeNotification
control. This causes issues in Wren:AM that uses that information in SMS component.
I have already debugged what is happening and the issue is that LDAPClientConnection2
is mapping search result entry to response using method that leads to calling incorrect overloaded constructor.
Quite obvious fix is to add overloaded method to Responses
class. However I would like to verify that it is the correct way and also possibly investigate if there is a place where we can add unit test to test this behaviour.
Affected Version
Wren:DS 3.5.1
Repro Steps
- From a checked-out copy of Wren:DS, use
mvn versions:set -DnewVersion=X.Y.Z
inside the opendj-bom
module to bump the version of the project to one that does not exist in JFrog.
- Run
mvn clean install -Ppackages -Pforgerock-release -Dignore-artifact-sigs
from the root of the Wren:DS repo.
Expected
All projects build successfully.
Actual
The "Wren:DS Debian Standard Package" module fails because it's looking for "Wren:DS Doc Generated References" in JFrog instead of using what was built in the same maven reactor / build.
Running recommended command docker run --rm --name wrends-test -p 1389:1389 -p 1636:1636 wrensecurity/wrends:latest
causes docker to crash.
opendj-setup-5163692219923348077.log
On the other hand, the docker image runs smoothly when i build it first and then run it.
docker build . -t wrends-arm:latest
docker run --rm --name wrends-test -p 1389:1389 -p 1636:1636 wrends-arm:latest
It is possible to move all repositories from WrenArchiver here?
And you can convert the WrenArchiver personal account to organization too.
the title says it all -- we need to make similar changes to the sustaining/4.0
branch as we did for sustaining/3.0
.
Summary
I would like Wren:DS distributions not to contain any reference to ForgeRock's commercial license.
Solution You'd Like to See
- The TXT files that contain the terms of the FR commercial license are no longer included in JAR packages for Wren:DS.
- The FR commercial license is no longer displayed on the first screen of the interactive installer. In its place, we should be displaying the CDDL instead.
Workarounds/Alternatives
Users currently have to ignore the commercial license during install, since it doesn't apply to them. This causes confusion.
Summary
The GeneralizedTime
component inconsistently formats / parses the date when the year of the date is greater than 9999. During serialization, the year is serialized as is (e.g. 292278994
), but the parsing process expects the year to be in YYYY
format.
Steps To Reproduce ("Repro Steps")
@Test
public void testGeneralizedTimeMaxYear() {
String serialized = GeneralizedTime.valueOf(Long.MAX_VALUE).toString();
GeneralizedTime parsed = GeneralizedTime.valueOf(serialized);
}
The preceding test ends with the following exception:
org.forgerock.i18n.LocalizedIllegalArgumentException: The provided value "2922789940817071255.807Z" is not a valid generalized time value because "78" is not a valid month specification
at org.forgerock.opendj.ldap.GeneralizedTime.valueOf(GeneralizedTime.java:221)
Expected Result (Behavior You Expected to See)
The behaviour of the GeneralizedTime
component will be consistent if the year is greater than 9999.
[WARNING] [Retry 1 of 10] Waiting 750 milliseconds before retrying key request from hkps.pool.sks-keyservers.net/192.146.137.98 after last request failed: HTTP/1.1 502 Bad Gateway
[WARNING] [Retry 2 of 10] Waiting 1500 milliseconds before retrying key request from hkps.pool.sks-keyservers.net/192.146.137.98 after last request failed: HTTP/1.1 502 Bad Gateway
[WARNING] [Retry 3 of 10] Waiting 2250 milliseconds before retrying key request from hkps.pool.sks-keyservers.net/192.146.137.98 after last request failed: HTTP/1.1 502 Bad Gateway
[WARNING] [Retry 4 of 10] Waiting 3000 milliseconds before retrying key request from hkps.pool.sks-keyservers.net/192.146.137.98 after last request failed: HTTP/1.1 502 Bad Gateway
[WARNING] [Retry 5 of 10] Waiting 3750 milliseconds before retrying key request from hkps.pool.sks-keyservers.net/192.146.137.98 after last request failed: HTTP/1.1 502 Bad Gateway
[WARNING] [Retry 6 of 10] Waiting 4500 milliseconds before retrying key request from hkps.pool.sks-keyservers.net/192.146.137.98 after last request failed: HTTP/1.1 502 Bad Gateway
[WARNING] [Retry 7 of 10] Waiting 5250 milliseconds before retrying key request from hkps.pool.sks-keyservers.net/192.146.137.98 after last request failed: HTTP/1.1 502 Bad Gateway
[WARNING] [Retry 8 of 10] Waiting 6000 milliseconds before retrying key request from hkps.pool.sks-keyservers.net/192.146.137.98 after last request failed: HTTP/1.1 502 Bad Gateway
[WARNING] [Retry 9 of 10] Waiting 6750 milliseconds before retrying key request from hkps.pool.sks-keyservers.net/192.146.137.98 after last request failed: HTTP/1.1 502 Bad Gateway
[WARNING] [Retry 10 of 10] Waiting 7500 milliseconds before retrying key request from hkps.pool.sks-keyservers.net/192.146.137.98 after last request failed: HTTP/1.1 502 Bad Gateway
Wren:DS has a plugable storage backend. Currently it primarily uses Berkeley DB Java edition as storage backend. At some point it also supported PDE however that didn't workout well so support was marked deprecated. PDE support was added due to ForgeRock's concerns about the licensing of Berkeley DB. However soon after PDE support was released the licensing of Berkeley DB was changed to a license which no longer posed a problem for ForgeRock.
While Berkeley DB was long the go to embedded storage backend new challengers have emerged. One of these is RocksDB. RocksDB was / is developed by Facebook and is based on research by Google (LevelDB).
RocksDB is supposedly, very, very fast. I think it would be interesting to do some research to see if A) RocksDB could be used as a backend for Wren:DS and B) If so, how much performance gain would that give us.
Summary
After upgrading Wren:DS to use wrensec-parent-3.0.0-SNAPSHOT
, the build fails to complete successfully when building opendj-server-legacy
.
Affects Version
4.0.0-SNAPSHOT (at 513505e
)
Environment
- Ubuntu 16.04.5 LTS
- Apache Maven 3.3.9
- Java 1.8.0_181-b13
Steps
Within the wrends
folder checked out to master
, run the following command:
mvn clean install -DskipTests -Denforcer.skip=true -DignoreArtifactSigs -DskipTests
Expected
The build succeeds without error.
Actual
The build fails during the maven-jar-plugin
execution of the opendj-server-legacy
module:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-jar-plugin:3.1.0:jar (build-bootstrap-client-jar) on project opendj-server-legacy: You have to use a classifier to attach supplemental artifacts to the project instead of replacing them. -> [Help 1]
Root Cause
The project builds many JARs with different names, but without any classifiers. In the 3.x version of the maven-jar-plugin
, classifiers are now mandatory. But, adding them will cause the JARs to be renamed.
Summary
Artifacts produced by Wren:DS end up with colons in their filenames, which is not an allowed character in Windows. This causes several issues, including build failure on Windows platforms.
Environment
Windows 10, JDK 8
Steps to Reproduce
- Checkout the Wren:DS project at the
3.0.0
tag.
- Run
mvn clean install
.
Expected Results
The build succeeds.
Actual Results
The build fails with an error about "negative time". This is actually because the JAR files don't get written to disk properly because their filenames contain characters not allowed in Windows.
Additional Notes
- This is like caused by the colon in the product name (i.e.
Wren:DS
).
- This may not be an issue in 3.5+ since in that version, there are new, separate Maven properties to control the lowercase product name that's used for archives.
- Even on Linux systems, this issue causes the names of files inside the Wren:DS zip file not to be correct when extracted.
When calling for status of DS i'm getting an exit code 6 as a return.
./status -ns
Server Run Status: Started
Open Connections: 0
Host Name: <hostname>
Administrative Users: cn=Directory Manager
Installation Path:
<installation path>/wrends/opendj-server-legacy/target/package/wrends
Version: Wren:DS Server 4.0.0-RC2-SNAPSHOT
Java Version: <not available> (*)
Administration Connector: Port 4444 (LDAPS)
Address:Port: --
Protocol: LDIF
State: Disabled
Address:Port: 0.0.0.0:636
Protocol: LDAPS
State: Disabled
Address:Port: 0.0.0.0:1389
Protocol: LDAP
State: Enabled
Address:Port: 0.0.0.0:1689
Protocol: JMX
State: Disabled
Address:Port: 0.0.0.0:8080
Protocol: HTTP
State: Disabled
Base DN: dc=example,dc=com
Backend ID: userRoot
Entries: <not available> (*)
Replication:
Get exit code in bash:
echo $?
6
It's not possible to build all of OpenDJ successfully in one pass. the opendj-dsml-servlet
project will fail because it can't find packages, but re-building just that package succeeds.
something is causing the classpath to be stale during the build.
Steps
- checkout Wren:DS from
sustaining/3.0.0
.
- run
mvn clean install
.
Expected
Currently
- the build fails because it cannot find several packages of generated source code:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) on project opendj-dsml-servlet: Compilation failure: Compilation failure:
[ERROR] opendj-dsml-servlet/src/main/java/org/opends/dsml/protocol/DSMLAddOperation.java:[37,40] package org.opends.server.protocols.ldap does not exist
- afterwards, building with
mvn clean install -pl opendj-dsml-servlet
succeeds.
Summary
HTTP endpoints do not return full JSON response.
$ curl -u admin:password http://localhost:8080/admin
{"_rev":"0000000054c141ca"
$ curl -u admin:password http://localhost:8080/api
{"_rev":"0000000054c141ca"
Steps To Reproduce ("Repro Steps")
- Enable HTTP Connector Handler
- Send GET requests to HTTP endpoints
Expected Result (Behavior You Expected to See)
Full readable JSON response.
Actual Result (Behavior You Saw)
Incomplete JSON response.
Environment
- Product version: 4.0.0-RC2
- JRE version: openjdk 17.0.6
Summary
When in multi-master replication mode a single server is unreachable (socket connection has to timeout) it will cause replication server to not be able to accept any connection due to data server's own socket timeout. This happens during the first DS-RS handshake phase and is accompanied with the following error in the log:
category=SYNC severity=ERROR msgID=178 msg=Directory server 15265 was attempting to connect to replication server 25602 but has disconnected in handshake phase
Steps To Reproduce ("Repro Steps")
- Download attached docker-compose.yml
- Start both server instances with
docker-compose up -d
- Configure replication:
docker exec -it wrends-test1 \
dsreplication enable --adminUID admin --adminPassword password --trustAll --no-prompt --baseDN dc=example,dc=com \
--host1 wrends-test1 --port1 4444 --bindDN1 "cn=Directory Manager" \
--bindPassword1 password --replicationPort1 8989 \
--host2 wrends-test2 --port2 4444 --bindDN2 "cn=Directory Manager" \
--bindPassword2 password --replicationPort2 8989
docker exec -it wrends-test1 \
dsreplication initialize-all --adminUID admin --adminPassword password --trustAll --no-prompt \
--baseDN dc=example,dc=com --hostname wrends-test1 --port 4444
- Shutdown both servers with
docker-compose stop
- Start only the first server with
docker-compose up -d wrends-test1
- Try to perform modification with
docker exec -it wrends-test1 ldapdelete -h localhost -p 1389
docker exec -it wrends-test1 ldapdelete -h localhost -p 1389 "uid=user.1,ou=People,dc=example,dc=com"
Expected Result (Behavior You Expected to See)
Server deletes the requested LDAP entry.
Actual Result (Behavior You Saw)
The following error is returned:
Processing DELETE request for uid=user.1,ou=People,dc=example,dc=com
The LDAP delete request failed: 53 (Unwilling to Perform)
Additional Information: The Replication is configured for suffix
dc=example,dc=com but was not able to connect to any Replication Server
Additional Notes
I have spent several hours trying to debug this. The underlying issue is that replication server's connection listener is trying to contact all other replication servers when accepting new connection. As the other server does not exist, this attempt timeouts with the same timeout value as is the data server's timeout for the connection handshake.
I am not sure why we need to wait for the replication domain to actually contact all servers. Simply increasing handshake timeout wouldn't work if we have multiple servers in the domain.
Creating this issue to actually track potential discussion regarding solving this problem.
Summary
It's currently not possible to build master
on Windows machines, even if all dependencies are satisfied and PGP verification is being skipped, because the opendj-doc-generated-ref
plug-in fails to generate documentation.
Affects Version
4.0.0-SNAPSHOT (at 4678646
)
Environment
- Windows 10.0.17134 Build 17134
- Apache Maven 3.3.9
- Java 1.8.0_172-b11
Steps
Within the wrends
folder checked out to master
, run the following command:
mvn clean install -DskipTests -Denforcer.skip=true -DignoreArtifactSigs -DskipTests
Expected
The build succeeds without any failures.
Actual
The build fails during the opendj-doc-generated-ref
plug-in execution:
[INFO]
[INFO] --- opendj-doc-maven-plugin:4.0.0-SNAPSHOT:generate-config-ref (generate-configuration-reference-doc) @ opendj-doc-generated-ref ---
[INFO]
[INFO] --- opendj-doc-maven-plugin:4.0.0-SNAPSHOT:generate-global-acis-table (generate-global-acis-table-and-ref-for-doc) @ opendj-doc-generated-ref ---
[INFO]
[INFO] --- opendj-doc-maven-plugin:4.0.0-SNAPSHOT:generate-schema-ref (generate-global-acis-table-and-ref-for-doc) @ opendj-doc-generated-ref ---
[INFO]
[INFO] --- opendj-doc-maven-plugin:4.0.0-SNAPSHOT:generate-xml-messages-doc (generate-log-reference-doc) @ opendj-doc-generated-ref ---
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Wren:DS Doc Helper Maven Plugin .................... SUCCESS [ 14.641 s]
[INFO] Wren:DS Doc Generated References ................... FAILURE [ 35.501 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 56.519 s
[INFO] Finished at: 2018-08-23T00:59:02-04:00
[INFO] Final Memory: 46M/556M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.forgerock.opendj:opendj-doc-maven-plugin:4.0.0-SNAPSHOT:generate-xml-messages-doc (generate-log-reference-doc) on project opendj-doc-generated-ref: NullPointerException: entry -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :opendj-doc-generated-ref
Root Cause
It looks like the issue is that the plug-in is using Paths.get()
within loadPropertiesFromJar()
to assemble the path that's passed-in to the JAR file. On Windows machines, this is problematic because the method generates a path with backslashes instead of forward slashes, and the ZIP spec requires forward slashes.
Affected Version: 3.0
Summary
The Wren:DS / OpenDJ Control Panel application does not appear to properly handle custom X-ENUM
syntaxes; when they are defined in the server, the Control Panel erroneously flags the schema as being incomplete and will not provide a listing of the directory or the schema.
Steps to Reproduce
- Install Wren:DS, being sure to create at least a root entry for the DC.
- Create a custom
X-ENUM
syntax (for example: https://docs.oracle.com/cd/E19476-01/821-0509/enumeration-syntax-extension.html). You can create the syntax through either the ldapmodify
CLI, or by editing 99-user.ldif
.
- Define an attribute type that uses the new
X-ENUM
syntax (reference the SunDS article mentioned earlier). You can create the syntax through either the ldapmodify
CLI, or by editing 99-user.ldif
(place it AFTER the X-ENUM
syntax definition).
- Restart Wren:DS.
- Attempt to connect to the server with Wren:DS / OpenDJ Control Panel.
Expected Results
- Control Panel connects to the server without error.
- If you browse the schema using the Control Panel application, you will find the OID of the
X-ENUM
syntax under "Attribute Syntaxes" in the "Manage Schema" window.
Actual Results
- Control Panel connects but displays errors.
- The error displayed is:
Error Reading Configuration
The definition for the attribute type with OID 1.3.6.1.4.1.32473.5 declared that it should have a syntax with OID 1.3.6.1.4.1.32473.4. No such syntax is configured for use in the Directory Server
Additional Info
Anecdotally, it seems like this might be fixed in the 4.0 branch, according to https://bugster.forgerock.org/jira/browse/OPENDJ-3252. However, simply cherry-picking aaf3145 (the commit cited in the comments for that ticket) on to the 3.0 SDK does NOT solve the issue for 3.x. There was likely a larger re-factor of this code that addressed it.
Summary
Wren:DS SDK defines a PORT_INCREMENT
value of 1000
, which means that if a port it wants to use is in-use, it skips 1,000 ports at a time until it can find the desired port. This can lead to failures during test runs if the port number the OS hands to the test is close to the maximum port number (e.g. max is 65535
and starting port number is 64536
).
At run-time, presumably this issue is less severe because Wren:DS has no preference for starting port number, so it has a larger pool of addresses to search 1,000 at a time. This defect only appears when the starting port # is within a few thousand port numbers of the RFC-defined max port #.
Steps To Reproduce ("Repro Steps")
NOTE: These steps do not reliably reproduce the issue because the issue depends on what ports are available on the host machine.
- Checkout Wren:DS @ 9e5a6fa.
- Build the project with
mvn clean install
.
Expected Result (Behavior You Expected to See)
The build should succeed with no test failures.
Actual Result (Behavior You Saw)
Sporadically, tests may fail with an IllegalArgumentException
and message of "Invalid port."
.
Environment
- Product version (include GIT commit ID if using a dev build): 3.0.0 @ 9e5a6fa
- Operating system:
Ubuntu 18.04.1 LTS
Linux 4.4.0-17763-Microsoft #379-Microsoft Wed Mar 06 19:16:00 PST 2019 x86_64 x86_64 x86_64 GNU/Linux
- JRE version:
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-2ubuntu0.18.04.1-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
- Was instance upgraded from a previous version? If so, what version were you using before?
No.
This one comes from an internal feature request we're working on for Rest2LDAP.
Summary
As a developer querying a deeply nested LDAP directory of users, I would like it if nested entries could be retrieved and filtered through the Rest2LDAP interface like flat collections so that I don't have to pull the full list of entries just to get information about a single entry.
Component Affected
Rest2LDAP (OpenDJ 3.5+)
Conditions of Acceptance
The Rest2LDAP module within OpenDJ is modified as follows:
- A new option is added to the Rest2LDAP configuration file to allow collection sub-resources to support sub-tree flattening, such that nested LDAP entries are included as though they were all at the same flat level in the directory.
- A new option is added to the Rest2LDAP configuration file to allow collection sub-resources to support an optional "base" search filter that can be used to restrict what entries get returned through the sub-resource. This search filter supplements any filters provided in the request.
- Validation is added to ensure that the new sub-tree flattening option is only being used if the sub-resource is marked as read-only in the config (without this, there is no way to unambiguously map attempts to create new entries to a specific DN). Any attempt to configure a writable sub-resource with sub-tree flattening turned on must result in an error.
- Sub-resources must be modified to behave as follows:
- When the sub-resource has the new subtree flattening option toggled on: it should return all LDAP entries and sub-entries under the base DN that match both search filters provided on the query string as well as the search filter specified in the configuration for the sub-resource (i.e. it expands the search scope to
SearchScope.SUBORDINATES
).
- When the sub-resource has the new subtree flattening option toggled off or has not specified whether the option is toggled on: it should return only entries that exist directly under the base DN, excluding sub-entries (i.e. the search scope is
SearchScope.SINGLE_LEVEL
), and then apply both search filters provided on the query string as well as the search filter specified in the configuration for the sub-resource (if any).
Workarounds / Other Approaches
- Set-up a singleton resource that's mapped to a virtual static group that queries the whole sub-tree and selects the desired users from everywhere under the desired base DN. By rendering the members of the virtual static group, you can access all of the users everywhere in the subtree instead of just at the base level. Major drawback: Huge performance hit, since trying to access a single record requires pulling down the whole list of users from the server.
- Don't use subtrees -- put all users under a single flat portion of the directory (not ideal for most use cases).
- Don't use Rest2LDAP -- query LDAP directly from the application (not ideal because it requires all consumers to understand LDAP).
Summary
import-ldif
crashes with OutOfMemoryError on Java 17.
Steps To Reproduce ("Repro Steps")
-
Run setup with sample data creation:
./wrends/setup \
--cli \
--backendType je \
--baseDN dc=example,dc=com \
--ldapPort 1389 \
--adminConnectorPort 4444 \
--rootUserDN cn=Directory\ Manager \
--rootUserPassword password \
--no-prompt \
--noPropertiesFile \
--doNotStart \
--sampleData 200
Expected Result (Behavior You Expected to See)
import-ldif
successfully imports data without errors.
Actual Result (Behavior You Saw)
The import stucks on message:
Importing Automatically-Generated Data (200 Entries) ........
The log in /tmp/opendj-setup-*.log
contains following error:
[24/11/2022:11:56:07 +0100] category=QUICKSETUP seq=72 severity=INFO msg=import-ldif arg list: [/tmp/wrends/bin/import-ldif, --offline, -n, userRoot, -A, /tmp/opendj-install17722320096080648957.template, -s, 0, -F, --noPropertiesFile]
[24/11/2022:11:56:11 +0100] category=QUICKSETUP seq=73 severity=INFO msg=import-ldif out log: [24/11/2022:11:56:11 +0100] category=JVM seq=0 severity=INFO msg=Installation Directory: /tmp/wrends
[24/11/2022:11:56:11 +0100] category=QUICKSETUP seq=74 severity=INFO msg=import-ldif out log: [24/11/2022:11:56:11 +0100] category=JVM seq=1 severity=INFO msg=Instance Directory: /tmp/wrends
[24/11/2022:11:56:11 +0100] category=QUICKSETUP seq=75 severity=INFO msg=import-ldif out log: [24/11/2022:11:56:11 +0100] category=JVM seq=2 severity=INFO msg=JVM Information: 17.0.5+8-Ubuntu-2ubuntu1 by Private Build, 64-bit architecture, 3856662528 bytes heap size
[24/11/2022:11:56:11 +0100] category=QUICKSETUP seq=76 severity=INFO msg=import-ldif out log: [24/11/2022:11:56:11 +0100] category=JVM seq=3 severity=INFO msg=JVM Host: foobar, running Linux 5.19.0-23-generic amd64, 15426191360 bytes physical memory size, number of processors available 16
[24/11/2022:11:56:11 +0100] category=QUICKSETUP seq=77 severity=INFO msg=import-ldif out log: [24/11/2022:11:56:11 +0100] category=JVM seq=4 severity=INFO msg=JVM Arguments: "--add-exports=java.base/sun.security.x509=ALL-UNNAMED", "--add-exports=java.base/sun.security.tools.keytool=ALL-UNNAMED", "-Dorg.opends.server.scriptName=import-ldif"
[24/11/2022:11:56:11 +0100] category=QUICKSETUP seq=78 severity=INFO msg=import-ldif out log: [24/11/2022:11:56:11 +0100] category=BACKEND seq=5 severity=FINE msg=JE backend 'userRoot' does not specify the number of cleaner threads: defaulting to 16 threads
[24/11/2022:11:56:11 +0100] category=QUICKSETUP seq=79 severity=INFO msg=import-ldif out log: [24/11/2022:11:56:11 +0100] category=BACKEND seq=6 severity=FINE msg=JE backend 'userRoot' does not specify the number of lock tables: defaulting to 37
[24/11/2022:11:56:11 +0100] category=QUICKSETUP seq=80 severity=INFO msg=import-ldif out log: [24/11/2022:11:56:11 +0100] category=BACKEND seq=24 severity=INFO msg=Wren:DS Server 4.0.0-M4-SNAPSHOT starting import (build 20221124115033, Rd7998ffe96597b13df5e2d4af125455f05fbf0da)
[24/11/2022:11:56:12 +0100] category=QUICKSETUP seq=81 severity=INFO msg=import-ldif out log: [24/11/2022:11:56:12 +0100] category=BACKEND seq=25 severity=INFO msg=The amount of free memory available to the import task is 3470996275 bytes. The number of phase one buffers required is 640 buffers
[24/11/2022:11:56:12 +0100] category=QUICKSETUP seq=82 severity=INFO msg=import-ldif out log: [24/11/2022:11:56:12 +0100] category=BACKEND seq=26 severity=INFO msg=Setting DB cache size to 33554432 bytes and phase one buffer size to 4194304 bytes
[24/11/2022:11:56:14 +0100] category=QUICKSETUP seq=84 severity=WARNING msg=import-ldif error log: Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
[24/11/2022:11:56:14 +0100] category=QUICKSETUP seq=85 severity=WARNING msg=import-ldif error log: at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:64)
[24/11/2022:11:56:14 +0100] category=QUICKSETUP seq=86 severity=WARNING msg=import-ldif error log: at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:363)
[24/11/2022:11:56:14 +0100] category=QUICKSETUP seq=87 severity=WARNING msg=import-ldif error log: at org.opends.server.backends.pluggable.OnDiskMergeImporter$BufferPool.<init>(OnDiskMergeImporter.java:2890)
[24/11/2022:11:56:14 +0100] category=QUICKSETUP seq=88 severity=WARNING msg=import-ldif error log: at org.opends.server.backends.pluggable.OnDiskMergeImporter$StrategyImpl.newHeapBufferPool(OnDiskMergeImporter.java:466)
[24/11/2022:11:56:14 +0100] category=QUICKSETUP seq=89 severity=WARNING msg=import-ldif error log: at org.opends.server.backends.pluggable.OnDiskMergeImporter$StrategyImpl.newBufferPool(OnDiskMergeImporter.java:401)
[24/11/2022:11:56:14 +0100] category=QUICKSETUP seq=90 severity=WARNING msg=import-ldif error log: at org.opends.server.backends.pluggable.OnDiskMergeImporter$StrategyImpl.importLDIF(OnDiskMergeImporter.java:210)
[24/11/2022:11:56:14 +0100] category=QUICKSETUP seq=91 severity=WARNING msg=import-ldif error log: at org.opends.server.backends.pluggable.BackendImpl.importLDIF(BackendImpl.java:660)
[24/11/2022:11:56:14 +0100] category=QUICKSETUP seq=92 severity=WARNING msg=import-ldif error log: at org.opends.server.tools.ImportLDIF.processLocal(ImportLDIF.java:853)
[24/11/2022:11:56:14 +0100] category=QUICKSETUP seq=93 severity=WARNING msg=import-ldif error log: at org.opends.server.tools.tasks.TaskTool.process(TaskTool.java:246)
[24/11/2022:11:56:14 +0100] category=QUICKSETUP seq=94 severity=WARNING msg=import-ldif error log: at org.opends.server.tools.ImportLDIF.process(ImportLDIF.java:243)
[24/11/2022:11:56:14 +0100] category=QUICKSETUP seq=95 severity=WARNING msg=import-ldif error log: at org.opends.server.tools.ImportLDIF.mainImportLDIF(ImportLDIF.java:113)
[24/11/2022:11:56:14 +0100] category=QUICKSETUP seq=96 severity=WARNING msg=import-ldif error log: at org.opends.server.tools.ImportLDIF.main(ImportLDIF.java:88)
[24/11/2022:11:56:14 +0100] category=QUICKSETUP seq=83 severity=INFO msg=import-ldif out log: [24/11/2022:11:56:14 +0100] category=BACKEND seq=27 severity=INFO msg=Import LDIF environment close took 0 seconds
[24/11/2022:11:56:14 +0100] category=QUICKSETUP seq=97 severity=INFO msg=import-ldif out log: [24/11/2022:11:56:14 +0100] category=BACKEND seq=28 severity=INFO msg=Flushing data to disk
Environment
- Product version (include GIT commit ID if using a dev build): master (d7998ff)
- Operating system: Kubuntu 22.10, Linux 5.19.0-23-generic
- JRE version: openjdk 17.0.5 2022-10-18
- Was instance upgraded from a previous version? If so, what version were you using before? No
- CPU: AMD Ryzen 7 PRO 4750U (16 threads)
- Memory: 14GB
Additional Notes
- I did not notice this problem on Java 8 neither 11.
- Manual invocation of
import-ldif
with -Xmx256m
works.
- It works on some other computers with different CPU and memory size.
When building Wren:DS in a network without public DNS available, maven fails while trying to verify the artifact signatures:
[DEBUG] (f) project = MavenProject: org.forgerock.opendj:opendj-server-parent:3.0.0 @ /home/olbohlen/git/wren/wrends/pom.xml
[DEBUG] (f) scope = test
[DEBUG] (f) session = org.apache.maven.execution.MavenSession@704b2127
[DEBUG] (f) verifyPomFiles = true
[DEBUG] -- end configuration --
[DEBUG] The resource 'http://wrensecurity.org/trustedkeys.properties' was not found with resourceLoader org.codehaus.plexus.resource.loader.JarResourceLoader.
[DEBUG] The resource 'http://wrensecurity.org/trustedkeys.properties' was not found with resourceLoader org.codehaus.plexus.resource.loader.FileResourceLoader.
[DEBUG] The resource 'http://wrensecurity.org/trustedkeys.properties' was not found with resourceLoader org.codehaus.plexus.resource.loader.ThreadContextClasspathResourceLoader.
[DEBUG] URLResourceLoader: Exception when looking for 'http://wrensecurity.org/trustedkeys.properties
java.net.UnknownHostException: wrensecurity.org
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
as a work-around, mvn -Dignore-artifact-sigs clean install
works.
From gitter chat:
The only difference between sustaining/3.5.x and the current master are the following commits:
ad669ec93a Upgrade Wren:DS to `wrensec-parent` (2.2.0)
7765a3cb9a Remove Unused Wren Deploy Values
7dd5225a57 Fix Acct. Change Notification Package Name
b210895b0e Fix file names to conform to class names in them.
47b06874c8 Fix path to Wren:DS JAR file for Doc Generation
bef165bab8 Add note about OPENDJ-3377 as OpenIG Workaround
8bef7b5dcb OPENDJ-3377 Switch OpenDJ dependency from openig-toolkit to chf-oauth2
55da63bff3 Update 3.5.1 for WrenSec repos & info
91c3b0ba77 Add Official OpenDJ 3.5.1 Sources from Backstage
I think we can go through those and apply them if necessary.
The following will be evaluated (the main idea behind this is that if there is a critical patch, it has unit tests as well):
git show 91c3b0ba77 -I'^ *([/\*]|@Override)' -- **/src/test/** > changes.diff
Recommend Projects
-
-
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
An Open Source Machine Learning Framework for Everyone
-
The Web framework for perfectionists with deadlines.
-
A PHP framework for web artisans
-
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
Some thing interesting about web. New door for the world.
-
A server is a program made to process requests and deliver data to clients.
-
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Some thing interesting about visualization, use data art
-
Some thing interesting about game, make everyone happy.
-
Recommend Org
-
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Open source projects and samples from Microsoft.
-
Google ❤️ Open Source for everyone.
-
Alibaba Open Source for everyone
-
Data-Driven Documents codes.
-
China tencent open source team.
-