redpen-cc / redpen Goto Github PK

Periods are used not only the end of the sentence but also the exceptional usages. The followings are the special usages.

• digits
  • 123.4893
• special words
  • Mr. Mrs.

The parsers should handle above language matters.

Generated Currently command file does not have the execute permission. The command should have the permission for user to use the DocumentValidator easily.

Current DV supports period, question mark and exclamation mark as the end of sentence character. But some users would want to add other characters as the end of character such as colon.

This issue is to add the configuration to the end of sentence characters for such users.

try and check if aspectj could be a soultion

Markdown has two styles of headers ("underlined" and "atx"). Currently DV works with atx style header and the style is tested.

We should check the the library used in the MarkdownParser supports the underline syntax or not. If we find that the library support the style, tests are added.

Configuration for Japanese could be much different from english configurations. Although default Settings of DocumentValidator should be for English, Japanese configurations should be not so difficult.

For the easy configuration for japanese, I will provide the sample setting files for Japanese documents.

In order to use DocumentValidator as a library, Main class should be excluded. The Main class would be a sub module of main project.

Check Japanese Katakana words with edit distance to extract the variation of spelling such as インデクス and インデックス.

Currently DocumentVlaidator has two configuration files (validator.conf and character-table.conf), but there should also be the configrutions for parsers. However, too many configuraiton files lead to decrease the usability.

i will add the central configuration file for the DocumentValidator which contains the sub configurations such as validator.conf and character-table-conf.txml.

I'm not sure that it's currently need but if DV has become a mutilingal app, it harms to us to add a new lang to CharacterTableLoader. So, if we have the rule that a lang 'xy' corresponds to XyDefaultSymbols, we can dynamically instantiate the class.

Support the following elements. I will support the following common elements in this issue.

• Inline Formatting (Issue #41)
  • Bold
  • Italics
  • italic and bold
  • underline text
  • strikethrough text
• Comments(Issue #39)
• Lists (Issue #35)
  • Bullet lists
  • Numbered lists
• Links (Issue #33)
  • External links
  • Internal links
  • images

add tests and implements

• indentation of parameters

DocumentValidator now always return 0 even when validation errors are found. Therefore we are not able to know if there are errors or not without the output file or terminal.

Returning the error code (1), we can see the results from shell scripts or wrapper tools.

Wikiparser support to extract links in the document which are used to implement validaotor to check the links.

Currently wikiparser extract the label of the links when there is a link block such as [label | url], however the label is not used to validator the link url. Therefore parser should extract url of the link (not the label).

StringUtils.getSentenceEndPosition method should detect exclamation mark and question mark as the end of the sentences. When the characters are not detected as the end position of the sentences. The sentences are not split with parsers.

For example, given the following string the parsers do not detect two sentences.

Is this a apple? Yes it is.

SpaceBegginingOfSentenceValidator flushes the errors alway when there are heading coments. For example, adding SpaceBeginningOfSentenceValidator in the validation-config.xml file and process the input document containing following line, SpaceBeginningOfSentenceValidator reports a error message "Space not exist the beginning of sentence".

# this is a heading

But, the heading contents may not start with space and therefore the behavior is not expected.

Currently SentenceIterator does not check sentences in headers or list elements. I will make SentenceIterator check sentences contained such tags.

Before, fixing this issue, Header elements and ListElements should be extended to have a Sentence Element.

Need SpellCheckValidator especially for English.

Markdown or Textile formats provide internal links and external urls in the contents. Sometimes links are down and user cannot got to the pages the links connect to. This validator check if the internal and external links are alive.

BlockTypes class is not used in current version.

add some tests for Markdown Parser

Currently, not support reference-link type sentence.

http://daringfireball.net/projects/markdown/syntax#link

Currently ValidationError does not have Sentence information except for the error messages created in Validators. To make the errors into more readable, i will split error message into error type and sentence information.

Specfically i will add "sentence" member in ValidationError, and error messages in validator make simple removing sentence information.

Current MarkdownParser treat the tables in a markdown syntax document. This leads the registered validators process the tables as the input sentence and flush the errors. But this behavior is not expected one, since the table is not a sentence (each elements could be sentences...).

To suppress the warnings, the changes not to add the tables as sentences would be needed.

WikiParser and MarkdownParser contain the same functional methods. This issue extract such methods into utility classes.

Parsers in DocumentValidator use String allocating many substrings. The codes are not efficient at all. The parsers could be more efficient using StringBuffer.

Document validator should have the default symbol (character) table. If there is a default value for common symbols (such as comma, colon), the configurations would be simplified since users just need to override the default value onluy if the modifications are needed
.

Current output of ValidationError does not contains the Validator's name flushed it. Users do not know which ValidationError came from.

Adding the ValidationError into ValidatorName, Users understand that which validator output it.

To shorten the processing time, DocumentValidator shouldl support multi threads.

If the left quotation mark is registered as the same as right quotation mark, they cannot be distinguished.

For example if we have the following settings, DocumentValidator thinks left quotation mark '"' needs not only before space but also after space.

<character name="LEFT_DOUBLE_QUOTATION_MARK" value="\'"  before-space="true" />
<character name="RIGHT_DOUBLE_QUOTATION_MARK" value="\'" after-space="true" />  

Validate the end hyphens of Katakana words in Japanese documents. Japanese Katakana words have variations in end hyphen. For example, "computer" is written in Katakana by "コンピュータ (without hyphen) ", and "コンピューター (with hypen) ". This validator check if Katakana words ending format is match the predefined standard.

In European languages such as English, Sentences start with capital characters. This validator check if the sentences start with capital case. Probably this validator need exception lists to handle exceptional cases.

• Enrich description in javadoc
• Follow standard Javadoc style (e.g. start with capital character)

Document validator should support semi-strructured formats such as xml or json. If the output has a semi-structured format, the output can be used another programs.

DocumentValidator has global default settings in DefaultSymbol class and users override the default values if needed. This setting is good for european languages but quite different from Japanese or Chinese documents.

I will add language dependent settings especially for asian languages.

Support comments.

The code which distributes the results is quite messy. The code can be improved with abstract class extracting common methods.