Orchestra2md and Md2Orchestra currently have some implementation for the metadata section, but they are not interoperable. It should be possible to round-trip between XML and markdown. All Dublin Core Terms should be be supported.

Support optional use of additional field attributes such as minimum and maximum length.

Update the wiki

Hello,
As you know I want to test md2spec, but I may have misunderstood how it has to be running. As stated in the README and requirements, I've donwloaded and installed Maven and try to build from there after cloning the project on my machine.
My questions are :
-Was it the right way to proceed?
-What I understood was we modify the md2orchestra-proto.md to get an orchestra file, am I right?

I need some clarification as I'm a beginner.

Regards,

Ndakey

Datatype patterns and Score expressions contain special characters that need to be escaped in markdown to prevent parser errors. However, the escape sequences obstruct human readability. Instead of escaping special characters, wrap the expressions as markdown literals (aka code span). The reverse flow needs to unwrap a literal for insertion in XML.

The markdown parser already supplies document position in parsing exceptions as line number plus character position in line. However, structural anomalies without a syntax error provide no document context. For example, an application may report that the name of an element is missing. There would be no way for a user to know which line in a markdown file caused the problem.

Supported by md-grammar -- see FIXTradingCommunity/md-grammar#3

Executables md2orchestra and orchestra2md (and any others to follow) should be packaged using platform-specific runtimes. The benefit is make the apps appear as native apps to users, and they need not be concerned with dependencies. Just download, install, and play.

At minimum, platforms should include Windows and MacOs. Ideally, this should be accomplished with JPackager, which is to be officially released with Java 14, as it provides a native app installer. Perhaps a shorter term goal would be to use jlink, which provides a runtime modular image but no installer.

Components can have subcomponents and those subcomponents may have a scenario specialization. The documentation doesn't show the scenario column available for components. Was wondering if this is a functionality issue or a documentation issue?

The md-grammar module was developed concurrently with md2orchestra, etc. Now, however, there are needs for markdown parsing logic in other applications, not necessarily related to Orchestra. Therefore, to keep a clean chain of dependencies, md-grammar should become an independent module.

Some of the Orchestra attributes sound technical, e.g. implMaxLength. It would be desirable to provide a more humanly readable column heading such as "Maximum length".

The problem is that all specs are subject to roundtrip between orchestra2md and md2orchestra, so there must be a way to store the alternative heading in the markdown document to substitute back to its target XML attribute name. Markdown does not officially allow for comments or hidden text, but perhaps an embedded HTML comment could be used to store such operations.

Use fix-orchestra repository module built with Java 11+. This is a step in achieving #4.
See FIXTradingCommunity/fix-orchestra#38

Currently, md2orchestra makes one pass through a markdown input file. Whether a reference to an element is processed correctly depends on order within the file. For example, a component reference will assume that its component is missing if the definition of the component is positioned after the reference.

To resolve this problem, two passes must be made. On the first pass, all missing references are collected. On the second pass, they are resolved to defined elements. Only if a reference is still missing after the second pass, is it truly a missing element.

md2orchestra supports file search using glob. For portability, support search using Unix style file separator on Windows.

md2orchestra now supports multiple input files (see #26) but they must be listed explicitly. It would be more convenient to allow a file glob pattern as in input specification, e.g. "mydirectory/*.md"

See typical pattern match capability at getPathMatcher

In an Orchestra XML file, a rule element may alter presence to tell when a field or component is required. The orchestra2md should output the member as conditionally required, and md2orchestra should recognized an easy syntax for this purpose.

Messages, codesets, components, and fields may be reused across multiple service offerings. It would be desirable to keep reusable elements in their own files and include them in rules of engagement. Therefore, md2orchestra should be capable of reading multiple markdown files to compose an Orchestra file.

In the Orchestra XML schema, the documentation element has optional attribute "purpose". Common values are "SYNOPSIS", "ELABORATION" and "EXAMPLE". Currently, orchestra2md concatenates all document elements but does not output the value of purpose. It could be done easily for paragraphs about a message or component. However, there is a style issue when outputing documentation about a message or component member with a table. Markdown tables do not support line breaks within a table cell. Purpose could be printed, it probably will not start a new line.

If purpose is included in markdown, md2orchestra should recognize that a word should be used top populate the XML attribute. One difficulty with this is that the schema allows a user to deviate from the suggested values. There would be no way for the parser to know that some other word is intended as the purpose unless we can come up with a style convention. Suggestions?

Hello I was testing the tool with your executables on jdk14 and while the program where running, It was generating empty files for md2orchestra. Any idea of what it could be? since I got no error whatsoever.

Thanks.

For md2orchestra and orchestra2md, implement translation of

• actors
• state variables of actors
• flows

The markdown parser sees blank lines at end of file as a parsing error.

Currently in md2orchestra, documentation element is populated without its purpose attribute. The Orchestra schema enumerates these values for purpose: synopsis, elaboration, example, and display. Tablature should support those keywords as column headers and use them to populate purpose in documentation. It can continue to map "documentation" column header to a <documentation> element without purpose attribute.

There remains the issue of what to do with columns that do not match an expected attribute. If the document writer thought it was important enough to add to rules of engagement, it should be saved somewhere in Orchestra. The <appinfo> element can be used for this. Like documentation, appinfo is a child of an annotation element, and it also has purpose attribute. Any unexpected column header can be mapped to the purpose attribute of appinfo element.

Atom markdown table editor writes a table delimiter row with a space on either side of each pipe character. The gfm spec makes no mention of spaces in a delimiter row in text but their examples show it. If possible, the Tablature markdown parser should be tolerant of spaces in that position.

You can add sections to markdown but they are ignored by md2orchestra if the heading doesn't contain a known keyword. However, in tables, new column headers are not ignored, they are added to annotation with purpose equal to the column header. We could do similar with added sections by adding documentation to the top level annotation element in the XML schema. This would simplify flow by not requiring users to keep text in separate files and reassembling each time they are edited.

Jim Northey has proposed changing the name of this project to Tablature. It is a term for musical notation and is consistent with FIX Orchestra and its DSL Score. Comment now if you have any objection.

Publish a user guide for md2orchestra telling how a markdown document should be formatted.

Currently, md2orchestra does not support the provenance features of Orchestra to tell when a message element was added or updated. Should it support keywords like "added", or take a different approach?

For md2orchestra and orchestra2md, implement state machines.

The Tablature tools already write errors and warnings to event logs using log4j. Additionally, they should write them in machine readable form for rendering or programmatic actions. JSON should be used for this purpose.

Some characters, especially the pipe character, are normal characters in XML but are special in markdown. It must be escaped with backslash in markdown as "\|" to be treated as a regular character.

Some examples use the pipe character to represent a field delimiter in FIX.

<fixr:documentation purpose="EXAMPLE">723 in field 21 would be mapped int as |21=723|. -723 in field 12 would be mapped int as |12=-723|.</fixr:documentation>

Another use of the pipe is for datatype patterns.

<fixr:mappedDatatype standard="XML" base="xs:string" builtin="0" pattern="\d{4}(0|1)\d([0-3wW]\d)?">

Two solutions are possible:

1. Escape pipe or other special character with backslash in markdown as \|
2. Turn the whole string into a literal with backticks like

`\d{4}(0|1)\d([0-3wW]\d)?`

Solution 1 is already done for some documentation but seems not be working for the examples. (Also, there is some reverse translation from markdown to XML in md2orchestra.)
Solution 2 seems viable for pattern and the like where special characters are likely to be used without modifying the pattern itself.

Tracked in Jira as ODOC-10

In addition to tables (see #29), also support documentation purpose for regular paragraphs. Such paragraphs will be preceded by a heading with purpose attribute, one of Synopsis, Elaboration, Example, or Display.

Additionally, in both paragraphs and tables, support headings of the form "Appinfo SomePurpose" to populate annotation/appinfo elements in Orchestra. (Since appinfo is by definition application specific, the purpose names need not conform to the enumeration for documentation.)

md2orchestra takes a filename pattern for markdown input files. This supports wildcards like *.md in a directory to read multiple files. It was reported that a relative path for the directory fails to find the desired files on Ubuntu Linux, but an absolute path does work.

In markdown, a fenced code block supports literal text of various programming languages and file types. Renderers support syntax-specific highlighting based on an infostring that tells the language or or file type. In FIX specifications, we have often used fenced code blocks for XML or raw data snippets.

The plan is to also extend code blocks for snippets to be imported from another file. See #37.

Some technical specification documents include snippets of code examples or XML elements and the like. Those snippets should remain in source files that can be validated by a language interpreter. Also, source code may be subject to improvements, changes, or corrections over time, but if they are included into a markdown file by copy and paste, they become frozen. Therefore, it would be better to provide a syntax to include snippets dynamically when a Tablature file is processed.

Typically code snippets are bound by markdown fenced code blocks using "```". Rather than literally copying the code into the fenced block, the block would contain syntax to tell a Tablature processor to include a file or section of a file. The source file would need to be accessible at processing time.

Add configuration to control order of section generation. (Currently, goes from large grained to small grained with messages at the top.)

Hello I restested tablature and I dont know why I cant get m2orchestra to work properly. Even with files generated from orchestra2md, which works perfectly.
I still have a blank file anyway.

I got this error on IntelliJ:

line 34:347 extraneous input '21=723|. -723 in field 12 would be mapped int as |12=-723|.                                                                                                                               |' expecting {<EOF>, NEWLINE}
14:49:50.135 [main] ERROR io.fixprotocol.md.event.DocumentParser.SyntaxErrorListener - Markdown parser failed at line 34 position 347 due to extraneous input '21=723|. -723 in field 12 would be mapped int as |12=-723|.                                                                                                                               |' expecting {<EOF>, NEWLINE}
line 35:0 mismatched input '| Language            ' expecting TABLEDELIMINATORCELL
14:49:50.146 [main] ERROR io.fixprotocol.md.event.DocumentParser.SyntaxErrorListener - Markdown parser failed at line 35 position 0 due to mismatched input '| Language            ' expecting TABLEDELIMINATORCELL
line 40:107 extraneous input '18=2 A F| ).                                                                                                                               |' expecting {<EOF>, NEWLINE}
14:49:50.181 [main] ERROR io.fixprotocol.md.event.DocumentParser.SyntaxErrorListener - Markdown parser failed at line 40 position 107 due to extraneous input '18=2 A F| ).                                                                                                                               |' expecting {<EOF>, NEWLINE}
line 41:0 mismatched input '| MultipleStringValue ' expecting TABLEDELIMINATORCELL
14:49:50.194 [main] ERROR io.fixprotocol.md.event.DocumentParser.SyntaxErrorListener - Markdown parser failed at line 41 position 0 due to mismatched input '| MultipleStringValue ' expecting TABLEDELIMINATORCELL
line 41:109 extraneous input '277=AV AN A| ).                                                                                                                               |' expecting {<EOF>, NEWLINE}
14:49:50.210 [main] ERROR io.fixprotocol.md.event.DocumentParser.SyntaxErrorListener - Markdown parser failed at line 41 position 109 due to extraneous input '277=AV AN A| ).                                                                                                                               |' expecting {<EOF>, NEWLINE}
line 272:0 no viable alternative at input '\n'
14:49:50.534 [main] ERROR io.fixprotocol.md.event.DocumentParser.SyntaxErrorListener - Markdown parser failed at line 272 position 0 due to no viable alternative at input '\n'
14:49:50.571 [main] FATAL io.fixprotocol.md2orchestra.Md2Orchestra - Md2Orchestra: exception occurred
java.lang.NullPointerException: null
        at io.fixprotocol.md2orchestra.RepositoryBuilder.findDatatypeByName(RepositoryBuilder.java:989) ~[md2orchestra-0.1.0-jar-with-dependencies.jar:?]
        at io.fixprotocol.md2orchestra.RepositoryBuilder.addType(RepositoryBuilder.java:501) ~[md2orchestra-0.1.0-jar-with-dependencies.jar:?]
        at io.fixprotocol.md2orchestra.RepositoryBuilder.addFieldAndType(RepositoryBuilder.java:342) ~[md2orchestra-0.1.0-jar-with-dependencies.jar:?]
        at io.fixprotocol.md2orchestra.RepositoryBuilder.addField(RepositoryBuilder.java:332) ~[md2orchestra-0.1.0-jar-with-dependencies.jar:?]
        at io.fixprotocol.md2orchestra.RepositoryBuilder.accept(RepositoryBuilder.java:111) ~[md2orchestra-0.1.0-jar-with-dependencies.jar:?]
        at io.fixprotocol.md2orchestra.RepositoryBuilder.accept(RepositoryBuilder.java:61) ~[md2orchestra-0.1.0-jar-with-dependencies.jar:?]
        at io.fixprotocol.md.antlr.MarkdownEventSource.exitTablerow(MarkdownEventSource.java:272) ~[md2orchestra-0.1.0-jar-with-dependencies.jar:?]
        at io.fixprotocol.md.antlr.MarkdownParser$TablerowContext.exitRule(MarkdownParser.java:741) ~[md2orchestra-0.1.0-jar-with-dependencies.jar:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:47) ~[md2orchestra-0.1.0-jar-with-dependencies.jar:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:30) ~[md2orchestra-0.1.0-jar-with-dependencies.jar:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[md2orchestra-0.1.0-jar-with-dependencies.jar:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[md2orchestra-0.1.0-jar-with-dependencies.jar:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[md2orchestra-0.1.0-jar-with-dependencies.jar:?]
        at io.fixprotocol.md.event.DocumentParser.parse(DocumentParser.java:48) ~[md2orchestra-0.1.0-jar-with-dependencies.jar:?]
        at io.fixprotocol.md2orchestra.Md2Orchestra.generate(Md2Orchestra.java:246) ~[md2orchestra-0.1.0-jar-with-dependencies.jar:?]
        at io.fixprotocol.md2orchestra.Md2Orchestra.generate(Md2Orchestra.java:222) ~[md2orchestra-0.1.0-jar-with-dependencies.jar:?]
        at io.fixprotocol.md2orchestra.Md2Orchestra.generate(Md2Orchestra.java:143) ~[md2orchestra-0.1.0-jar-with-dependencies.jar:?]
        at io.fixprotocol.md2orchestra.Md2Orchestra.main(Md2Orchestra.java:104) [md2orchestra-0.1.0-jar-with-dependencies.jar:?]

Move the module orchestra-common to fix-orchestra project so it can be reused for projects. It is not particular to the Tablature domain.

Generate a markdown document from an Orchestra interfaces XML file.

Tracked in Jira as ODOC-12

Parse a markdown file and convert it XML conforming to the Orchestra interfaces schema. That schema governs service offerings and session configurations.

Tracked in Jira as ODOC-11

Currently, fields and codesets are provided in the markdown file as separate chapters. This is fit for purpose for the complete FIX standard but less so for a given implementation, typically a small subset.

The current markdown output also fits the authoring requirement for markdown very well and should continue to be the standard. The requirement is different when the markdown file is used for a Rules of Engagement document (docx, pdf, html). A spec reader would like to have the following information directly integrated into the message/component/group table.

• Datatype of the field
• Supported values of the field

The majority of publicly available RoE docs integrate this information. It does not require to switch back and forth between chapters. FIXimate supports it albeit in a different way by integrating supported values into the field description in the data dictionary and displaying that below the message layout.

Field definition descriptions are separate from field usage descriptions, i.e. the compact representation would not contain field definition text in the message/component/group tables. The entire sections for fields and codesets are then obsolete unless they also need to be described in general.

A field usage entry in markdown in a compact representation should look as follows for LocateReqd(114) in NewOrderSingle(35=D):

Required for short sell orders

Supported values:
N = Indicates the broker is not required to locate
Y = Indicates the broker is responsible for locating the stock

The CR/LF should use "/P/" as usual for the multiline text. The datatype of the field (Boolean, not LocateReqdCodeSet) should be an extra column for all fields, i.e. also those that do not have a codeset, for example:

The power of this compact representation comes when using codeset scenarios. You can then immediately see which values of a field are supported in which message.

Event logs including error messages are written using log4j. By default they will be written to the console, but they may be controlled by log4j configuration supplied through various means. Since most users will not be familiar with log4j, there should at least be a command line argument to send the event log to a named file.

The XML processor generates XML prefixes like ns1, ns2, etc. Although namespace prefixes are only in effect within a file , and only the namespaces themselves should be considered portable, this may cause problems for other applications that read the XML output. Therefore, map namespace prefixes to common values used in examples.

Translating Orchestra XML to markdown and back to XML fails to preserve internal IDs of messages, groups, components, and codesets.

Currently, orchestra2md always outputs codes in a codeset table separate from a field definition. This follows the Orchestra XML schema. Alternatively, it could output a few valid values directly in the field definition or for a fieldRef within a message or component. See example below. md2orchestra already parses this format. Also, this format would be natural for many users who may not be familiar with the concept of a codeset.

Counterarguments to consider:

• How many codes can be listed within a table cell? This works ok for a few, but would be unwieldy for a long list. Markdown table cells do not support internal line breaks; a break starts a new table row.
• An advantage of a separate codeset table is it has room for documentation of each code.

To support a calling application such as a markdown document validator, allow it to receive parsing error events. Currently, they are logged but not exposed to a caller.

According to the GFM spec,

Any ASCII punctuation character may be backslash-escaped

OTOH

Backslashes before other characters are treated as literal backslashes

One motivation for this is that the Orchestra DSL symbol "||" would be recognized as markdown table column delimiters unless escaped.

Technical specifications often include snippets of data, XML, or code. If data is pasted into a text document, it becomes static and can no longer be validated by a system that managed or created it. It would be better to support import of text from external sources dynamically. Each time a Tablature process is run, it would access fresh data. Once imported, the text would be treated the same as the surrounding document.

Markdown already has a feature called fenced code block for a literal snippet, delimited by ``` (3 backticks). A Tablature processor would recognize a syntax for an import statement within such a block and pull it in.

Applications based on Tablature depend on keywords in the markdown document. For example, md2orchestra uses keywords "message, "field", "component", and so forth to identify context. A question is how far Tablature apps should go with synonyms for keywords. In md2orchestra, "description" and "documentation" are already handled as synonyms. However, alternative terms have been used in FIX that might be used by many people. For example, a code could be described as "code value" and "symbolic name" instead of just "value" and "name". To be very flexible would require a controlled vocabulary with many synonyms.

Tracked in Jira ticket ODOC-7 Markdown tools should handle workflow

The markdown tools md2orchestra and orchestra2md currently only handle message structures. They should also handle the workflow feature of Orchestra. That is, a user should be able to specify responses to a message and tell when each response is activated. Conversely, it should be possible to document in markdown the responses encoded in an Orchestra file.

A markdown feature called a code span supports literal text in which escape sequences do not apply. This would be useful for Score expressions, regex expressions, and the like that make use of special characters, including pipe and backslash.