In order to further limit which attributes can be set by Frank!Developers a Java annotation ProtectedAttribute has been introduced (in the Frank!Framework), see frankframework/frankframework/pull/2474 .

In order to avoid supporting duplicate code/functionality I'd like to only use the new Java annotation ProtectedAttribute.
The @ff.noAttribute tag now supports 're-introducing' setters. Since we are not using this, I suggest disabling/removing this.

The combination of PRs #41 (quoted forward/parameter names), #42 (tags) and #21 (logging) will introduce a small issue with logging. The logging PR checks whether a description is missing. It does so by splitting on spaces and counting the number of resulting substrings. This is not correct anymore when a name can have multiple words. The code needs to be refactored when the mentioned PRs will have been merged.

Describe the issue
Most Pipes derive from AbstractPipe, that derives from TransactionAttributes. That has an attribute 'transactionAttribute', that is not seen in the Frank!Doc.

Reporter
Gerrit

To Reproduce
Steps to reproduce the behavior:

1. Go to Ibis4Example
2. Click on FrankDoc
3. Select a Pipe
4. See no transactionAttribute attribute

Environment

FF! 7.7-SNAPSHOT: Ibis4Example
Running on vlapp05.interpar.nl using Apache Tomcat/7.0.90
Java Version: OpenJDK Runtime Environment (1.8.0_242-b08)
Heap size: 165M, total JVM memory: 1103M
Free disk space: 11GB, total disk space: 17GB
Up since: 2021-10-12 18:21:25 (14h)

The algorithm that checks for digester-rules.xml patterns expects a rule named "configuration" to match root class Configuration. However, we want to remove that role.

XSD generated by frank-doc version 1.1-SNAPSHOT are illegal. In both generated XSDs there is a duplicate attribute name in XComSenderDeclaredAttributeGroup.
One is from XComSenderDeclaredAttributeGroup itself, the other from SenderBaseDeclaredAttributeGroup

The schema doesn't appear to be valid by itself (as a part of another schema, it might still be OK).
	The complex type definition 'XComSenderType' already has an attribute 'name'.
		Reason: The following attribute declarations are duplicates of 'name' (see below)
			'name'
		Error location: xs:schema / xs:attributeGroup / xs:attribute
		Details
			ct-props-correct.4: The complex type definition 'XComSenderType' already has an attribute 'name'.
	Only one attribute declaration 'name' is allowed within the attribute group definition 'XComSenderCumulativeAttributeGroup'.
		Reason: The following attribute declarations are duplicates of 'name' (see below)
			'name'
		Error location: xs:schema / xs:attributeGroup / xs:attribute
		Details
			ag-props-correct.2: Only one attribute declaration 'name' is allowed within the attribute group definition 'XComSenderCumulativeAttributeGroup'.
	The complex type definition '{anonymous}' already has an attribute 'name'.
		Reason: The following attribute declarations are duplicates of 'name' (see below)
			'name'
		Error location: xs:schema / xs:attributeGroup / xs:attribute
		Details
			ct-props-correct.4: The complex type definition '{anonymous}' already has an attribute 'name'.
	The complex type definition '{anonymous}' already has an attribute 'name'.
		Reason: The following attribute declarations are duplicates of 'name' (see below)
			'name'
		Error location: xs:schema / xs:attributeGroup / xs:attribute
		Details
			ct-props-correct.4: The complex type definition '{anonymous}' already has an attribute 'name'.

Describe the issue
I think we can/should re-evaluate the log statements in the frankdoc, mainly which warnings are actually errors. There are currently no errors appearing in the log, I'm unsure if this is because they are not implemented.
I'd also like the build to fail when there are error's.

Additionally, the following error appeared in my log;
WARN FrankDocModel.createAttributes():251[] Attribute [Receiver.onError] has an invalid default value, [continue, detail Value [continue] is not allowed, expected one of [CONTINUE, RECOVER, CLOSE]]

• The default value compare appears to parse the default value as a lowercase constant.

WARN FrankDocModel.createAttributes():251[] Attribute [JdbcQuerySenderBase.useNamedParams] has an invalid default value, [null, detail Value [null] is not Boolean]

• The log reads all booleans as primitives, but there are also boolean objects (Boolean). The objects are allowed a NULL value.
• The log should not print using the object Boolean when it parses as a primitive.

WARN FrankDocModel.createAttributes():251[] Attribute [MongoDbSender.action] has an invalid default value, [, detail Value [] is not allowed, expected one of [INSERTONE, INSERTMANY, UPDATEONE, DELETEMANY, UPDATEMANY, FINDONE, FINDMANY, DELETEONE]]

• null (or empty) should be an acceptable (not-set) value when not using primitives.

WARN FrankElement.setChildrenOfKind():111[] Frank element [nl.nn.adapterframework.senders.SenderSeries] has multiple attributes / config children with key [(roleName=sender, elementType=nl.nn.adapterframework.core.ISender)]

• Assuming this is because the Frank element implements ISender as well as ISenderWithParameters this warning doesn't have to be thrown.

Optional overrides any mandatory setting.
Example use case:
AbstractPipe.setName() will get a @ff.mandatory annotation
IWrapperPipe.setName() and IValidator.setName() will get a @Optional annotation or a @ff.optional javadoc tag, to avoid having to set a name for wrappers and validators

Page https://frankdoc.frankframework.org/#!/Other in the left menu shows:

PipeLineExit
PipeLineExits
PipeForward
PipeForwards
ScheduleManager

While I would expect:

Exit
Exits
Forward
Forwards
Scheduler

According to Niels it would be a good idea to make it possible to give elements like this a "name"

To support cooperation with tools like FrankFlow, a FrankConfiguration must be able to contain data from these tools, like x,y coordinates. To handle these independently, they are placed in a specific namespace.
To support this, the generated XSDs must allow for this.
This can be done be adding the following at all places where also the 'active' is added:
<xs:anyAttribute namespace="##other"/>

For example for https://frankdoc.frankframework.org/#!/Pipes/BytesOutputPipe the code completion shows:

BytesOutputPipe - nl.nn.adapterframework.pipes.BytesOutputPipe used as Pipe Output bytes as specified by the input XML.

This is very confusing. In my opinion there's no need to add "BytesOutputPipe - nl.nn.adapterframework.pipes.BytesOutputPipe used as Pipe" at the beginning of the text. Please remove it.

with recent changes to the FrankFramework, some classes can be removed from the FrankElementFilter.
Please review each of the classes in the filter.
It should be possible to remove all SAP classes, so they are included in the FrankDoc.
A number of elements refer to non existent senders in the pipes package. These lines can be removed too.

In issue frankframework/frankframework#2877 we are renaming attribute path of element Exit to name. For
FrankConfig-strict.xsd this simply means that the path attribute needs to be renamed to name (or path removed and name added) but for FrankConfig-compatibility.xsd this is a bit more complicated as we will need to keep support for path and also allow name but name should not be mandatory as otherwise old configs with only a path still get an error.

As discussed with Martijn and Gerrit this probably means we need an extra annotation (something like "loosely mandatory").

Describe the issue
ManageFileSystem gives the following warnings in the Frank!Console:

Configuration [IAF_Util] - Validation error in [classpath:./ConfigurationManageFileSystem.xml] at line [10] when validating against schema [/xml/xsd/FrankConfig-compatibility.xsd]: cvc-complex-type.4: Attribute 'name' must appear on element 'XmlInputValidator'.

Configuration [IAF_Util] - Validation error in [classpath:./ConfigurationManageFileSystem.xml] at line [14] when validating against schema [/xml/xsd/FrankConfig-compatibility.xsd]: cvc-complex-type.4: Attribute 'name' must appear on element 'XmlOutputValidator'.

Issue frankframework/frankframework#2687 seems to report the same issue but is closed

Reporter
Jaco de Groot

To Reproduce
Steps to reproduce the behavior:

1. Start a Frank! with ManageFileSystem enabled
2. Open the console
3. See warnings at the top of the page

Screenshots
-

Configuration
-

Input
-

Environment

FF! 7.8-20220318.150450: Frank2Example3
Running on *** using Apache Tomcat/9.0.60
Java Version: OpenJDK Runtime Environment (1.8.0_292-b10)
Heap size: 348M, total JVM memory: 718M
Free disk space: 102GB, total disk space: 475GB
Up since: 2022-03-18 17:56:08 (2m) 

Additional Environment
-

Additional Context
-

To support frankframework/frankframework#2736, we would like to have version information in the XML Schema and in the json.
The version string itself (something like '7.7-RC1' or '7.7-20220209' or any other string) can be passed to the FrankDoc generator as a command line parameter at startup.
In the XML Schema's the version info can be stored in an annotation somewhere in the first lines of the file
For the JSON, a metadata field can be inserted, like:

  "metadata": { "version": "7.7-RC1" }

The metadata construct allows to add other information later

And the unit tests should be updated. Method names and Java comments will be confusing when @ff.ignoreTypeMembership does not exist. Before, @ff.ignoreTypeMembership also suppressed attribute production.

Describe the issue
The Frank!Doc is quite complicated. One of the reasons has to do with method registerChild. This method has multiple overloads, causing multiple ElementRole objects sharing the same element role. This situation requires complicated logic in the Frank!Doc. Method registerChild is deprecated however. If we remove the registerChild methods, we may be able to remove much logic from the Frank!Doc. Doing so will make future development more easy.

Reporter
Martijn Dirkse

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Screenshots
If applicable, add screenshots to help explain your problem.

Configuration

Please provide the configuration of the Pipe or Receiver with the problem.

Input

Please provide an example of the input message. 
Alternatively, provide a Ladybug report.

Environment

Please go to the console of your Frank, click on 'Information' and copy-paste the results here. 

Additional Environment

• DBMS: [e.g. Oracle, MSSQL, MariaDB, MySql, PostgreSQL]
• Browser: [e.g. Chrome, Safari, Edge, Firefox]

Additional Context
Add any other context about the problem here. (f.e. ladybug report / test adapter with larva test)

Presently the Frank!Doc front-end resides in the https://github.com/ibissource/iaf project, while this project produces the JSON file with the information it needs. This is a risk because the front-end code can become incompatible with the JSON. Therefore we want to add the frank-doc frontend to this project. Then this project becomes a multi-module project.

Describe the issue
IbisDoc or JavaDoc for ConfigChild-setters in interfaces is not seen in FrankDoc

Reporter
Gerrit

To Reproduce
Steps to reproduce the behavior:

1. Go to FrankDoc of AbstractPipe, e.g. via FixedResult
2. See that Locker has no documentation
3. See that Locker is documented in IExtendedPipe

Reporter
Diederik Kerkhof

Describe the solution you'd like
A tag attribute for pipes added in the ibisdoc. An example of this attribute is soap for a soap pipe or api for api pipes. In this 'tag' tag there can be multiple tags such as described above or a tag for difficulty for example. These tags can be useful for grouping pipes that are similar to each other or showing pipes that belong to the same group. Based on these tags we can also add icons in the frankflow.

Describe alternatives you've considered
The alternative is to make a file for every application that wants to use these kind of groups and use that to order or display that specific group.

Additional context
An example for the ibisdoc would be (tags at bottom of code)

{
            "name":"MailSender",
            "fullName":"nl.nn.adapterframework.senders.MailSender",
            "isAbstract":false,
            "isDeprecated":false,
            "parent":"nl.nn.adapterframework.senders.MailSenderBase",
            "elementNames":[
                "MailErrorSender",
                "MailSender",
                "MailSenders"
            ],
            "attributes":[
                {
                    "name":"smtpHost",
                    "isDeprecated":false,
                    "describer":"nl.nn.adapterframework.senders.MailSender",
                    "description":"name of the host by which the messages are to be send",
                    "default":""
                }
            ],
            "children":[
            ],
            "tags":[
                 {
                      "primairy":"mail"
                      "difficulty":"easy"
                      "popularity":"5"
                 }
            ]
        }

Describe the issue
The forwards table in the Frank!Doc for the XmlSwitch contains malformed forwards, I guess with escaping of some XML characters.

Reporter
Ricardo

To Reproduce
Steps to reproduce the behavior:

1. Go to https://frankdoc.frankframework.org/#!/Pipes/XmlSwitch
2. See malformed forwards

Screenshots

Configuration

Please provide the configuration of the Pipe or Receiver with the problem.

Input

Please provide an example of the input message. 
Alternatively, provide a Ladybug report.

Environment

Please go to the console of your Frank, click on 'Information' and copy-paste the results here. 

Additional Environment

• DBMS: [e.g. Oracle, MSSQL, MariaDB, MySql, PostgreSQL]
• Browser: [e.g. Chrome, Safari, Edge, Firefox]

Additional Context
Add any other context about the problem here. (f.e. ladybug report / test adapter with larva test)

With each build of the Frank!Framework the following warnings are displayed when executing the FrankDoc:

Warning:  Javadoc Warnings
Warning:  javadoc: warning - The old Doclet and Taglet APIs in the packages
Warning:  com.sun.javadoc, com.sun.tools.doclets and their implementations
Warning:  are planned to be removed in a future JDK release. These
Warning:  components have been superseded by the new APIs in jdk.javadoc.doclet.
Warning:  Users are strongly recommended to migrate to the new APIs.

Please analyze and do the needful to solve this.

Describe the issue
Sender configuration below is not properly parsed by the digester. Therefore the following configuration warning is shown in the console.

Configuration [outgoing] - Configuration [outgoing] Error when validating against schema [/xml/xsd/FrankConfig-compatibility.xsd] in [classpath:Configuration_Ibis4Integrity.xml] at line,column [46,18]: cvc-complex-type.3.2.2: Attribute 'slotId' is not allowed to appear in element 'MessageStoreSender'. Configuration [outgoing] - Configuration [outgoing] Error when validating against schema [/xml/xsd/FrankConfig-compatibility.xsd] in [classpath:Configuration_Ibis4Integrity.xml] at line,column [46,18]: cvc-complex-type.3.2.2: Attribute 'type' is not allowed to appear in element 'MessageStoreSender'.

Configuration:
<SenderPipe name="IntegrityErrorStoreSender" > <messageLog className="nl.nn.adapterframework.jdbc.DummyTransactionalStorage" datasourceName="jdbc/ibis4oma" slotId="${applicationId}/IbisForIntegrityErrorStorage" type="E" retention="7" /> <MessageStoreSender datasourceName="jdbc/ibis4oma" slotId="${applicationId}/IbisForIntegrityErrorStorage" type="E" > </MessageStoreSender> <Forward name="success" path="EXIT" /> </SenderPipe>

Reporter
Ali

When you have built the F!F and there is an error with the Frank!Doc, you do not have the trace logs. The build of the Frank!Doc hard-codes log-level INFO. You have to rebuild the Frank!Doc on your development PC with a temporary change of log4j2.xml. After that, you have to rebuild the F!F to get the trace logs.

Two improvements are possible:

• The F!F build should provide an option to adjust the log level applied by the Frank!Doc doclet.
• The F!F should have the doclet produce trace logs by default, but these trace logs should not clutter the console output. If you see from the console output that there is a Frank!Doc issue, it should not be necessary to repeat the build to have the traces. It would also be useful if the GitHub tests save the Frank!Doc trace logs as artifacts. This allows for quick analysis of failed builds, without needing an additional build on a developer computer.

The second option may be complicated, because the default output directory of the Frank!Doc is target/frankdoc as is configured in the F!F pom.xml. The frank-doc-plugin takes care of attaching the contents of target/frankdoc to the JAR of the core subproject. This logic has to change, because the Frank!Doc logs appear in target/frankdoc but should not be attached to the JAR.

@nielsm5 and @gvanbrakel, what do you think most useful and efficient?

Describe the issue
Documentation is added to the xsd elements, just like in the JSON. However in the XSD you sometimes still see {@link Receiver Receivers} it should remove the text between the {@.. } but keep the last word.
eq;
{@link Receiver Receivers} --> Receivers
{@link Receiver} --> Receiver

Each pipe prescribes what forward names are allowed. For example, the XsltPipe supports forward names "SUCCESS" and "EXCEPTION". No error is produced however if an XsltPipe has a forward with another name. This is true for every pipe: forwards with unsupported names do not produce errors.

I am sure that we can implement this in XML Schema 1.0. We can define the <Forward> element within a pipe using an <xs:element> within the complex type of the pipe. If we define the forward differently within different pipes, we can have different constraints for the name attributes of the forwards. Or we can define different types for the <Forward> element that we reference in the definitions of the different pipe elements. In any case, we also have to take care of the generic element option. A <Pipe> element with a className attribute cannot have forwards that have their name checked.

I expect that a more intelligent solution is possible with XML Schema 1.1. I think so after reading https://www.altova.com/blog/what-s-new-in-xml-schema-11/#:~:text=XML%20Schema%201.1%20allows%20you,complex%20type%20in%20the%20schema.&text=Above%20are%20some%20of%20the,%3Aerror%20datatype%2C%20and%20others., but did not investigate any further.

About 80% of our code is covered by unit tests. This looks good, but some important features of the Frank!Doc have no coverage. This should be added.

The Frank!Framework tests (including the Larva tests) hold many Frank configurations. When the language of Frank configs develops, these old configs should stay working. We can add this to the GitHub test of this project, because it does apply itself to the F!F code already (F!F is checked out during test).

There are two ways to do this:

• Feed the checked-out config files to XmlAgainstXsdValidator. Drawback: that tool is not designed thoroughly and does not implement entity references. Extending requires the tool to be unit tested.
• Run the ibis-adapterframework-core JAR headless. Then we use the existing logic of parsing Frank configs (including but not only validation against FrankConfig-compatibility.xsd).

A FrankAttribute does not have its "documented" property set if the modeled Java method has an @ff.mandatory annotation. If a Java method with an @ff.mandatory annotation is inherited from a parent class and if the parent method is not annotated, then the Frank!Doc may miss that the child attribute is mandatory. The Frank!Method may view the overriding method as a "technical override" and ignore it.

A unit test is needed to test whether this bug is present. If this is indeed a bug, then it should be fixed.

See https://frankdoc.frankframework.org/#!/Senders/IbisLocalSender, HTML h4 elements are all placed on the same line. Page source shows the content at the correct place and viewing it without style also does, so I suspect CSS is the culprit.

Right now only elements that are a child of a root element are included in the Frank!Doc. Because of this the framework now contains lots of dummy methods such as:

// Dummy setter to allow JmsRealms being added to Configurations via FrankDoc.xsd
public void registerJmsRealm(JmsRealm realm) {
	JmsRealmFactory.getInstance().registerJmsRealm(realm);
}

// Dummy method to include monitoring in the Frank!Doc.
public void registerMonitoring(MonitorManager factory) {
}

In order to avoid these methods I propose that we change the required rules in the digester-rules.xml from relative to absolute paths.

From:
<rule pattern="*/monitoring" registerMethod="registerMonitoring" factory="nl.nn.adapterframework.monitoring.MonitoringFactory"/>
To:
<rule pattern="configuration/monitoring" factory="nl.nn.adapterframework.monitoring.MonitoringFactory"/>

Note the absolute path/pattern, enforcing this element to be a child of Configuration, even though there is no setter available for this element.

Describe the issue
If enum values are deprecated, they should not appear in the FrankDoc when not explicitly selected.

Reporter
Gerrit

To Reproduce
Steps to reproduce the behavior:

1. Go to FrankDoc of Parameter
2. See values LIST and MAP
3. Go to the source of Parameter
4. See that LIST and MAP are deprecated

Environment

FF! 7.7-SNAPSHOT: Ibis4Test
Running on LPZWNL000040524 using Apache Tomcat/9.0.43
Java Version: OpenJDK Runtime Environment (1.8.0_41-b04)
Heap size: 189M, total JVM memory: 247M
Free disk space: 231GB, total disk space: 476GB
Up since: 2021-12-13 10:06:44 (17m)

According to the current FrankConfig.xsd the following root elements are allowed (screenshot of design view in Eclipse):

While the old xsd shows the only three allowed root elements:

Describe the issue
We never implemented logic about overloaded attribute setters. What if we have method public void setXyz(String arg) and also public void setXyz(int arg)? The Frank!Doc should behave the same as the digester does for these corner cases. We also should consider inheritance here. Part of this issue is to investigate what the digester does, such that the same can be implemented in the Frank!Doc.

Reporter
Martijn Dirkse

For each of the javadoc tags used by the Frank!Doc, a java annotation should exist as an alternative:
@ff.mandatory <-> @MandatoryAttribute
@ff.default <-> @DefaultValue
@ff.optional <-> @OptionalAttribute
@deprecated <-> @deprecated

This allows the parser to use these values too, and allows to override values without having to respecify the javadoc.

When using FrankConfig.xsd in VSCode with F!F 7.8-20220401.105103 the following is shown when hovering FrankConfig.xsd:

There are '2' errors in 'FrankConfig.xsd'.xml
FrankConfig.xsd(7649, 28): src-resolve: Cannot resolve the name 'MailListenerCumulativeAttributeGroup' to a(n) 'attribute group' component.

During the development of #94 I wanted to configure log level TRACE. That caused the build to fail. This failure is fixed in #94, but we do not want to allow such an issue on the master. I propose we add an extra GitHub workflow to build this project with log level TRACE. The job should clone this project, then modify log4j2.xml to configure log level TRACE and then the workflow should run the Maven build.

The strict XSD rejects a <DummyMessageLog> within a <SenderPipe>. See picture below.

It would be easier for users to always use the same root element and have no need to adjust the root element based on the use case. See description in the Frank!Runner README for the current three options:

https://github.com/ibissource/frank-runner#code-completion-with-frankconfigxsd

So I would like to suggest to remove the possibility to have Module and Adapter as root elements and allow a Configuration element in a Configuration element for the use case where one configuration file is included in another configuration file using an entity reference. Only one include level is enough in my opinion (no need to recursively include a Configuration in a Configuration).

This also means F!F needs to be adjusted to be able to parse a Configuration element in a Configuration element when initializing a configuration. Ask Gerrit and/or Niels to look into this when you change the Frank!Doc for this.

For example https://frankdoc.frankframework.org/#!/Pipes/BytesOutputPipe contains a lot of information about the pipe but using code completion you will only see: Output bytes as specified by the input XML.

It would be nice to see more information. In the case of BytesOutputPipe it would probably be to much information to add all Javadoc that is available on the class but at least a certain amount of characters would be nice and a pointer that more information is available at BytesOutputPipe. Adding the link so the user can click on it would be ideal but if not possible at least show something like ... more information available at the Frank!Doc website.

Implement @see or {@inheritdoc} annotation to replace @IbisDocRef

Presently, the Frank!Doc cannot make the difference between mandatory and optional config children. All config children are treated as optional. We may need mandatory config children, because the F!F version 7.7-20211209.115618 produces an error if a <Receiver> does not have listeners, see issue frankframework/frankframework#2524.

We distinguish config children that can appear multiple times (method name starts with add or register) from config children that can appear only once (method name starts with set). We could add other prefixes for config children that should appear exactly once or for config children that should appear at least once.

When I add links to external websites, like <a href="https://en.wikipedia.org/wiki/Representational_state_transfer">Wikipedia</a>, the links do not work in the Frank!Doc webapplication. The link is prepended by "localhost". When you click it, you get an error page.

In the FrankDoc.xsd, every element contains a reference to attribute 'active'.
It would be more efficient if child elements would inherit this from their hierarchical parent.

See for example the following pages:

https://frankdoc.frankframework.org/#!/Pipes/EscapePipe
https://frankdoc.frankframework.org/#!/Listeners/JavaListener
https://frankdoc.frankframework.org/#!/Pipes/BytesOutputPipe

In the Java source code you can see that the Javadoc of the class is divided into two paragraphs by a <p> or newline but in the browser the paragraphs are are not divided by an empty line

FrankDoc should not check on validity of default values. The value of the @Default annotation or @ff.default tag is a description, not a real value.
The warning should be removed, it clutters the list of warnings, making it less useful.

As PipeLineExit.setState() has @ff.mandatory I would expect the state attribute to appear automatically when I use code completion to create the Exit element but only the name attribute is added automatically. According to the FrankConfig.xsd it isn't mandatory, I think it should be mandatory in the XSD.

Method PipeLine.setPipeLineExits() has the following documentation:

@IbisDoc({ "PipeLine exits. If no exits are specified, a default one is created with name=\""+DEFAULT_SUCCESS_EXIT_NAME+"\" and state=\"SUCCESS\""})

But https://frankdoc.frankframework.org/#!/Other/PipeLineExits doesn't show it and the info is not available in de FrankConfig.xsd for code completion. Maybe the documentation needs to be moved to the PipeLineExits class? Or does the Frank!Doc code need to be changed? Either way the Exits needs to be documented for the end user

Jaco asked me to omit the class name in the tooltip help of elements. You now see the class name in your text editor when you want information about an element. This class name is not relevant if you are writing in syntax 2.