The request

NsiqCppStyle should provide a convenient and easy-to-use tool to show precisely which rule callback functions will be invoked for a given sequence of source code.

Background

Writing an NsiqCppStyle rule requires understanding which callback functions will be invoked when the engine is processing a known sequence of source code. Currently, knowing which callback functions to take advantage of can be tedious. In addition, a rule writer may not be aware of which callback functions are actually invoked, often missing an elegant method of implementing a rule.

The request is to write a tool that, given a snippet of source code, shows the rule writer the precise set of callback functions invoked, and their order. Each callback function displayed should provide important information such as the token type and value, any existing context stack, and other valuable data passed via the callback function.

The tool is written in the style of an NsiqCppStyle rule file, hooking in to the callback function system.

This tool is similar to Clang's ast-dump command line option.

Example

Suppose we want to see what callback functions NsiqCppStyle will invoke for the following C++ source code:

// Your First C++ Program

#include <iostream>

int main() {
    std::cout << "Hello World!";
    return 0;
}

Here is what the above tool's output might look like:

D:\nsiqcppstyle>python nsiqcppstyle.py -f rules\filefilter.txt "D:\Junk\NsiqCppStyle\hello-world.cpp"
nsiqcppstyle: N'SIQ Cpp Style ver 0.3.0.1

======================================================================================
=  Analyzing hello-world.cpp
======================================================================================
  -  RULE_50_99_A_test_nsiqcppstyle_callbacks is applied.
======================================================================================
LexToken contain six fields: (type, value, lineno, column, lexpos, inactive, pp)

SessionStartRule()
Filter Scope "default" is applied.
Current Filter Setting (Following is applied sequentially)
  1. \.cvs\ is excluded
  2. \.svn\ is excluded

Current File extension and Language Settings
  C/C++=cpp,mm,cc,c,m,h,hpp,cxx,hxx,hh

======================================================================================
Processing:  D:\Junk\NsiqCppStyle\hello-world.cpp
FileStartRule(lexer, filename='hello-world.cpp', dirname='D:\Junk\NsiqCppStyle')
--------------------------------------------------
LineRule(lexer, line='// Your First C++ Program', lineNumber=1)
CommentRule(lexer, token=LexToken(CPPCOMMENT,'// Your First C++ Program',1,1,0, False, None))
--------------------------------------------------
LineRule(lexer, line='#include <iostream>', lineNumber=3)
PreprocessRule(lexer,
               contextStack,
               token=LexToken(PREPROCESSOR,'#include',3,1,27, False, True)
PreprocessRule(lexer,
               contextStack,
               token=LexToken(LT,'<',3,10,36, False, True)
PreprocessRule(lexer,
               contextStack,
               token=LexToken(ID,'iostream',3,11,37, False, True)
PreprocessRule(lexer,
               contextStack,
               token=LexToken(GT,'>',3,19,45, False, True)
--------------------------------------------------
LineRule(lexer, line='int main() {', lineNumber=5)
Rule(lexer,
     contextStack,
     token=LexToken(INT,'int',5,1,48, False, None))
FunctionNameRule(lexer,
                 fullName='main',
                 decl='False',
                 contextStack,
                 context='FUNCTION_BLOCK, main, LexToken(LBRACE,'{',5,12,59, False, None), LexToken(RBRACE,'}',8,1,108, False, None)')
Rule(lexer,
     contextStack,
     token=LexToken(FUNCTION,'main',5,5,52, False, None))
Rule(lexer,
     contextStack,
     token=LexToken(LPAREN,'(',5,9,56, False, None))
     Context stack:
         PARENBLOCK, , 56, 57
Rule(lexer,
     contextStack,
     token=LexToken(RPAREN,')',5,10,57, False, None))
FunctionScopeRule(lexer, contextStack)
                  Context stack:
                      FUNCTION_BLOCK, main, LexToken(LBRACE,'{',5,12,59, False, None), LexToken(RBRACE,'}',8,1,108, False, None)
Rule(lexer,
     contextStack,
     token=LexToken(LBRACE,'{',5,12,59, False, None))
     Context stack:
         FUNCTION_BLOCK, main, LexToken(LBRACE,'{',5,12,59, False, None), LexToken(RBRACE,'}',8,1,108, False, None)
--------------------------------------------------
LineRule(lexer, line='    std::cout << "Hello World!";', lineNumber=6)
FunctionScopeRule(lexer, contextStack)
                  Context stack:
                      FUNCTION_BLOCK, main, LexToken(LBRACE,'{',5,12,59, False, None), LexToken(RBRACE,'}',8,1,108, False, None)
Rule(lexer,
     contextStack,
     token=LexToken(ID,'std',6,5,65, False, None))
     Context stack:
         FUNCTION_BLOCK, main, LexToken(LBRACE,'{',5,12,59, False, None), LexToken(RBRACE,'}',8,1,108, False, None)
FunctionScopeRule(lexer, contextStack)
                  Context stack:
                      FUNCTION_BLOCK, main, LexToken(LBRACE,'{',5,12,59, False, None), LexToken(RBRACE,'}',8,1,108, False, None)
Rule(lexer,
     contextStack,
     token=LexToken(DOUBLECOLON,'::',6,8,68, False, None))
     Context stack:
         FUNCTION_BLOCK, main, LexToken(LBRACE,'{',5,12,59, False, None), LexToken(RBRACE,'}',8,1,108, False, None)
FunctionScopeRule(lexer, contextStack)
                  Context stack:
                      FUNCTION_BLOCK, main, LexToken(LBRACE,'{',5,12,59, False, None), LexToken(RBRACE,'}',8,1,108, False, None)
Rule(lexer,
     contextStack,
     token=LexToken(ID,'cout',6,10,70, False, None))
     Context stack:
         FUNCTION_BLOCK, main, LexToken(LBRACE,'{',5,12,59, False, None), LexToken(RBRACE,'}',8,1,108, False, None)
FunctionScopeRule(lexer, contextStack)
                  Context stack:
                      FUNCTION_BLOCK, main, LexToken(LBRACE,'{',5,12,59, False, None), LexToken(RBRACE,'}',8,1,108, False, None)
Rule(lexer,
     contextStack,
     token=LexToken(LSHIFT,'<<',6,15,75, False, None))
     Context stack:
         FUNCTION_BLOCK, main, LexToken(LBRACE,'{',5,12,59, False, None), LexToken(RBRACE,'}',8,1,108, False, None)
FunctionScopeRule(lexer, contextStack)
                  Context stack:
                      FUNCTION_BLOCK, main, LexToken(LBRACE,'{',5,12,59, False, None), LexToken(RBRACE,'}',8,1,108, False, None)
Rule(lexer,
     contextStack,
     token=LexToken(STRING,'"Hello World!"',6,18,78, False, None))
     Context stack:
         FUNCTION_BLOCK, main, LexToken(LBRACE,'{',5,12,59, False, None), LexToken(RBRACE,'}',8,1,108, False, None)
FunctionScopeRule(lexer, contextStack)
                  Context stack:
                      FUNCTION_BLOCK, main, LexToken(LBRACE,'{',5,12,59, False, None), LexToken(RBRACE,'}',8,1,108, False, None)
Rule(lexer,
     contextStack,
     token=LexToken(SEMI,';',6,32,92, False, None))
     Context stack:
         FUNCTION_BLOCK, main, LexToken(LBRACE,'{',5,12,59, False, None), LexToken(RBRACE,'}',8,1,108, False, None)
--------------------------------------------------
LineRule(lexer, line='    return 0;', lineNumber=7)
FunctionScopeRule(lexer, contextStack)
                  Context stack:
                      FUNCTION_BLOCK, main, LexToken(LBRACE,'{',5,12,59, False, None), LexToken(RBRACE,'}',8,1,108, False, None)
Rule(lexer,
     contextStack,
     token=LexToken(RETURN,'return',7,5,98, False, None))
     Context stack:
         FUNCTION_BLOCK, main, LexToken(LBRACE,'{',5,12,59, False, None), LexToken(RBRACE,'}',8,1,108, False, None)
FunctionScopeRule(lexer, contextStack)
                  Context stack:
                      FUNCTION_BLOCK, main, LexToken(LBRACE,'{',5,12,59, False, None), LexToken(RBRACE,'}',8,1,108, False, None)
Rule(lexer,
     contextStack,
     token=LexToken(NUMBER,'0',7,12,105, False, None))
     Context stack:
         FUNCTION_BLOCK, main, LexToken(LBRACE,'{',5,12,59, False, None), LexToken(RBRACE,'}',8,1,108, False, None)
FunctionScopeRule(lexer, contextStack)
                  Context stack:
                      FUNCTION_BLOCK, main, LexToken(LBRACE,'{',5,12,59, False, None), LexToken(RBRACE,'}',8,1,108, False, None)
Rule(lexer,
     contextStack,
     token=LexToken(SEMI,';',7,13,106, False, None))
     Context stack:
         FUNCTION_BLOCK, main, LexToken(LBRACE,'{',5,12,59, False, None), LexToken(RBRACE,'}',8,1,108, False, None)
--------------------------------------------------
LineRule(lexer, line='}', lineNumber=8)
Rule(lexer,
     contextStack,
     token=LexToken(RBRACE,'}',8,1,108, False, None))
FileEndRule(lexer, filename='hello-world.cpp', dirname='D:\Junk\NsiqCppStyle')
ProjectRule(targetName='D:\Junk\NsiqCppStyle\hello-world.cpp')


=================================== Summary Report ===================================
 ** Total Available Rules     : 54
 ** Total Applied Rules       : 1
 ** Total Violated Rules      : 0
 ** Total Errors Occurs       : 0
 ** Total Analyzed Files      : 1
 ** Total Violated Files Count: 0
 ** Build Quality             : 100.00%

================================ Violated Rule Details ===============================

================================ Violated File Details ===============================
SessionEndRule()

A few comments:

• Each new line encountered (identified by a LineRule() callback) is preceded by a visual line of dashes. This allows the rule writer to focus on the line and all of its tokens.
• All other callbacks emanating from this new line follow in chronological order
• Each callback function displayed shows the relevant information passed to it (e.g., parameter values, the token type and value, context stacks).

Points to Consider

• Although written using NsiqCppStyle's "rule" programming style, the tool is technically not a rule - it is tool or a utility program. We should consider naming this Python script file as "TOOL_" or "UTIL_" to differentiate it from "real" rule files.
• What should the name format of this "rule" file be? Should it follow the NsiqCppStyle standard of [RULE|TOOL|UTIL]_nn_nn_text or some other standard? If the user standard, what number corresponds to NsiqCppStyle-related rule files?
• NsiqCppStyle has the command line option "-r" to list all rule files, as in nsiqcppstyle -r > filefilter.txt. We need to modify the "-r" option NOT to output tool/util scripts, as they are not rules for finding warnings in C++ source code.
• Question: where should this tool/utility script be stored? We can store it in the "rules" folder, although it is not technically a rule. Alternatively, we can create a new folder called. say, "tools" and store it there.
• To simplify running this tool script, I envision writing a shell script (i.e., both CMD batch and *nix shell) for easy invocation. The script will accept the path to the source file to process, creating a temporary filefilter.txt file containing only the tool script, running nsiqcppstyle.py passing the source file, and cleaning up when finished. Where should this script be stored: in the NsiqCppStyle home folder or in a "tools" folder?

I don't see any setuptools or autotools for this project, so I wonder if there is a preferred way to prepare a source tarball. I have packaged RPMs in the past, with a tarball made by hand and a custom spec file which added a few homegrown rules.

But now I'm also preparing debian packages, and I'd like to have a more reproducible way to go from repository to debian or RPM package.

Is your thought that nsiqcppstyle should eventualy have one of these formal distribution approaches? Would you welcome a setup.py approach? Or is there already something and I have just not found it?

nsiqcppstyle.py currently accepts INCLUDE/EXCLUDE filters via the filefilter.txt file. Using the README example, if filefilter.txt contains the following lines:

- \
+ \src\
- \src\test

NsiqCppStyle will analyze all source files under \src\ except \src\test.

The INCLUDE/EXCLUDE filters are checked against a relative-path candidate file (e.g., \src\graph-logic.h) to see if the file path starts with the filter string. So in this example, the path begins with \src\ so it will be included.

The problem is that there is no way to exclude a file name exhibiting a certain pattern. For example, I wanted to process all files under a folder, but did not want to process any file ending with .pb.h (i.e., generated by the Google Protobuf Buffer mechanism). As far as I can see, there is no way to do this using NsiqCppStyle's filters in filefilters.txt.

There are two possible solutions:

• Extend NsiqCppStyle's INCLUDE/EXCLUDE to accept regular expressions, rather than static strings that need to match the start of the path.
• Support passing a text file via the command line that contains a complete list of file paths of files to process.

Background

NsiqCppStyle is a serious Open Source software project, intended for general use. Every time a change is made to the engine code, there is a real risk of breaking the existing implementation and/or features. NsiqCppStyle requires high-quality and reliable regression tests to ensure that changes to the existing engine code base won't affect the existing features of the project.

As expected of a well-designed product, NsiqCppStyle has a dedicated unit/regression test source file: nsiqunittest/nsiqcppstyle_unittest.py. The file defines a class unitTest with member functions, each one representing a unit/regression test. This script file currently has a small number of tests:

It is clear that these tests are the proverbial "tip of the iceberg", and that many more such tests should be added to ideally provide complete coverage of internal engine code.

Request

Effort should be invested to write as many targeted unit/regression tests for NsiqCppStyle internals as possible. As more people submit code changes, having a rich set of regression tests will give us the confidence needed to make changes without breaking the existing code base.

The writing of such tests should be an ongoing task: every time any contributor (or the maintainer) touches a section of NsiqCppStyle engine code, time should be allocated for enriching the set of unit/regression tests, all related to the section/features modified.

Similarly, while implementing a new feature, should a contributor break the product during development, s/he should stop and write the necessary unit/regression tests to never allow that break to happen again.

IMHO, this should be a high-priority task as soon as the current pull requests have settled and are merged.

nsiqcppstyle_exe.py lines: 293 - 294 should have \ to continue the line. This can lead to compilation errors:

File "/usr/lib/nsiqcppstyle/nsiqcppstyle_exe.py", line 293
if fileExtension in cExtendstionSet and
^
SyntaxError: invalid syntax

Existing Code:

                 if fileExtension in cExtendstionSet and
                            basefilelist.IsNewOrChanged(eachFile) and

Suggested Replacement:
if fileExtension in cExtendstionSet and
basefilelist.IsNewOrChanged(eachFile) and \

The Goal

Prepare top-notch, visually appealing GitHub wiki documentation for NsiqCppStyle. The documentation should be as clear, simple, and appealing as possible, and should provide everything an NsiqCppStyle user needs to write style rules.

Most GitHub projects either have no wiki-based documentation, or the documentation is of poor quality. The goal is to make NsiqCppStyle attractive to potential users by providing world-class documentation.

It will take time and patience to implement this issue, given the wealth of technical information within the project. We should plan the wiki page structure before diving in. This page should serve as the whiteboard for developing the best page structure. Once decided, it will be easier to tackle the wiki pages one at a time.

Resources / References

List of excellent GitHub wiki examples

Following is a short list of GitHub projects that have attractive and high-quality wiki documentation (by all means, if you find a great example, add it to the list!). We should feel free to emulate the best of what there is out there.

• Netflix/Hystrix : Excellent example of what one can do with the GitHub wiki
• snowplow/snowplow
  • The project links to non-wiki documentation, but the topics listed there can be helpful for NsiqCppStyle
• d3/d3 : Look at the list of wiki pages for ideas

How to view a wiki's raw markdown

To see the raw markdown of a wiki page you like, slightly manipulate the project wiki page URL to generate the desired URL. Note that the URL is case-sensitive. Here are two examples to show how.

Example 1: View raw markdown for the wiki home page of Netflix/Hystrix

The URL to the GitHub wiki home page of the Netflix/Hystrix project is https://github.com/Netflix/Hystrix/wiki. Looking at that page shows the page title is "Home":

• Begin with the following domain: https://raw.githubusercontent.com/wiki/
• Append the project repository's name: https://raw.githubusercontent.com/wiki/Netflix/Hysterix/
• Append the page title: https://raw.githubusercontent.com/wiki/Netflix/Hysterix/Home
• Append .md: https://raw.githubusercontent.com/wiki/Netflix/Hysterix/Home.md

Example 2: View raw markdown of d3/d3's Tutorials wiki page

The URL to the GitHub wiki home page of the Netflix/Hystrix project is https://github.com/d3/d3/wiki. Looking at that page shows the page title is "Tutorials":

• Begin with the following domain: https://raw.githubusercontent.com/wiki/
• Append the project repository's name: https://raw.githubusercontent.com/wiki/d3/d3/
• Append the page title: https://raw.githubusercontent.com/wiki/d3/d3/Tutorials
• Append .md: https://raw.githubusercontent.com/wiki/d3/d3/Tutorials.md

Action Items

• Create a brainstormed list of topics the wiki should deal with
• Structure the list of topics into a well-ordered hierarchy
• Develop an NsiqCppStyle logo for the project and wiki pages
• The project's README.md file can reference wiki pages
• Be sure to include pages to pique readers' interests, for example:
  • Step-by-step tutorials
  • HOW-TO
  • FAQ
  • Case studies
  • History-related (e.g., how did NsiqCppStyle come about)
  • Product comparison pages (e.g., NsiqCppStyle vs. Clang-Tidy)

In function PrepareReport the open functions try to open the files as binaries. This leads to problems, when the output xml/csv is chosen. I think this is python 2 style ? Think about using csvfile = open(outputPath, "w"). This fixed it for me

The fmt library in C++ offers a streamlined implementation of the boost::format libraries. Among other things it offers an sprintf() function. This function safely generates an std::string without buffer overflow issues.

nsiqcppstyle reports:
Do not use burfferoverflow risky function(sprintf [RULE_10_1_A_do_not_use_bufferoverflow_risky_function_for_unix]

for this line of code:

string s = fmt::sprintf("%2s%012.8f", year, doy_fraction);

The simple solution might be to check if sprintf( is preceded by a "::" (two colons), although that would run in to problems if you have "using namespace fmt;" and thus drop the fmt::. Still, I think it would be quite OK.

The Problem

When NsiqCppStyle parses the following code snippet:

Event& Event::operator=(OUT Event& other)
{
}

The internal token list (accessed via the lexer.tokenlist member variable) looks like this:

tokenlist = {list: 19}
 00 = {LexToken} LexToken(ID,'Event',1,1,0, False, None)
 01 = {LexToken} LexToken(AND,'&',1,6,5, False, None)
 02 = {LexToken} LexToken(SPACE,' ',1,7,6, False, None)
 03 = {LexToken} LexToken(ID,'Event',1,8,7, False, None)
 04 = {LexToken} LexToken(DOUBLECOLON,'::',1,13,12, False, None)
 05 = {LexToken} LexToken(FUNCTION,'operator',1,15,14, False, None)
 06 = {LexToken} LexToken(EQUALS,'=',1,23,22, False, None)
 07 = {LexToken} LexToken(LPAREN,'(',1,24,23, False, None)
 08 = {LexToken} LexToken(ID,'OUT',1,25,24, False, None)
 09 = {LexToken} LexToken(SPACE,' ',1,28,27, False, None)
 10 = {LexToken} LexToken(ID,'Event',1,29,28, False, None)
 11 = {LexToken} LexToken(AND,'&',1,34,33, False, None)
 12 = {LexToken} LexToken(SPACE,' ',1,35,34, False, None)
 13 = {LexToken} LexToken(ID,'other',1,36,35, False, None)
 14 = {LexToken} LexToken(RPAREN,')',1,41,40, False, None)
 15 = {LexToken} LexToken(LINEFEED,'\n',1,42,41, False, None)
 16 = {LexToken} LexToken(LBRACE,'{',2,1,42, False, None)
 17 = {LexToken} LexToken(LINEFEED,'\n',2,2,43, False, None)
 18 = {LexToken} LexToken(RBRACE,'}',3,1,44, False, None)

The problem is that NsiqCppStyle returns the overloaded operator function name ("operator=") as two tokens, i.e., a FUNCTION ("operator" at offset 5) and an EQUALS ("=" at offset 6). This is incorrect - the overloaded operator function name should be returned as one FUNCTION token with a value of "operator=":

tokenlist = {list: 18}
 00 = {LexToken} LexToken(ID,'Event',1,1,0, False, None)
 01 = {LexToken} LexToken(AND,'&',1,6,5, False, None)
 02 = {LexToken} LexToken(SPACE,' ',1,7,6, False, None)
 03 = {LexToken} LexToken(ID,'Event',1,8,7, False, None)
 04 = {LexToken} LexToken(DOUBLECOLON,'::',1,13,12, False, None)
 05 = {LexToken} LexToken(FUNCTION,'operator=',1,15,14, False, None)
 06 = {LexToken} LexToken(LPAREN,'(',1,24,23, False, None)
 07 = {LexToken} LexToken(ID,'OUT',1,25,24, False, None)
 08 = {LexToken} LexToken(SPACE,' ',1,28,27, False, None)
 09 = {LexToken} LexToken(ID,'Event',1,29,28, False, None)
 10 = {LexToken} LexToken(SPACE,' ',1,34,33, False, None)
 11 = {LexToken} LexToken(AND,'&',1,35,34, False, None)
 12 = {LexToken} LexToken(ID,'other',1,36,35, False, None)
 13 = {LexToken} LexToken(RPAREN,')',1,41,40, False, None)
 14 = {LexToken} LexToken(LINEFEED,'\n',1,42,41, False, None)
 15 = {LexToken} LexToken(LBRACE,'{',2,1,42, False, None)
 16 = {LexToken} LexToken(LINEFEED,'\n',2,2,43, False, None)
 17 = {LexToken} LexToken(RBRACE,'}',3,1,44, False, None)

This is the correct way to parse the overloaded operator function name - tokenlist[5] shows the FUNCTION token with a value of "operator=".

I'd prefer these to be valid Path to remove a bunch of corner cases due to using str (eg: some filesystems don't allow unicode names, but others do), but let's postpone that to later

Originally posted by @kunaltyagi in #35 (comment)

In a future PR, let's see if we can reduce the duplication by:

    def RunPreprocessRule(self, lexer: Lexer, contextStack: ??) -> None:
        """ Run rules which runs in the preprecessor blocks """
        _RunLexerRules(self.preprocessRules, lexer, contextStack)

    def _RunLexerRules(self, rules: List[Callable], lexer,  *args, **kwargs) -> None
        for rule in rules:
            data = lexer.Backup()
            rule(lexer, *args, **kwargs)
            lexer.Restore(data)

Originally posted by @kunaltyagi in #35 (comment)

This could be done for just some of the callbacks

There is an error on the indentation: look at https://github.com/kunaltyagi/nsiqcppstyle/blob/master/nsiqcppstyle_exe.py
line: 380 to 384.

        elif line.startswith("+"):
            arg = line[1: ].strip()
            if arg != "": filter.AddInclude(arg)
            elif line.startswith("-"):
                arg = line[1: ].strip()
            if arg != "": filter.AddExclude(arg)
            elif line.startswith("%"):
                arg = line[1: ].strip()
            if arg != "": filter.AddVarMap(arg, "\"" + arg + "\" of filefilter.txt") 

The verification for "-" and "%" should be at the same level than "+"

Thanks in advance!

The RULE_3_2_F_use_representitive_classname_for_cpp_filename which check filename against class name declared in it. When the class is final, the parser interpret "final" as the classname :-(

The following test reproduce the problem :

def test9(self):
    self.Analyze("test/CamRecorderFactory.h",

"""
class CamRecorderFactory final
{
};
""")
assert not CheckErrorContent(name)

Requested Functionality

Add two new rule callback functions to notify a user rule when NsiqCppStyle is about to start/end its session.

Technical Background

In NsiqCppStyle, a session is defined as the time range beginning just before the first source file is processed, and ending right after the last file is processed. Within this time range, NsiqCppStyle may have processed multiple projects (processing of a single file or an entire directory tree). There is always only one session for a given run of NsiqCppStyle.

Currently, NsiqCppStyle has two sets of callbacks to notify a user rule when processing events start and/or end:

for targetPath in targetPaths:
    if targetPath is a file:
        FileStartRule()
        <process the file>
        FileEndRule()
    else:
        # targetPath is a directory ("project")
        for (each file in the directory tree):
            FileStartRule()
            <process the file>
            FileEndRule()
    ProjectRules()

In a recent user rule I wrote, I needed to "setup" and "teardown" global data at the start and end of the session: at session start I defined required data structures, while at session end I wrote out the accumulated information to an external JSON file.

The request is to define new rule callback functions to notify a user rule when the session starts and ends (e.g., SessionStartRule and SessionEndRule). When implemented, rule writers will have three types of time range notifications to work with:

SessionStartRule()
for targetPath in targetPaths:
    if targetPath is a file:
        FileStartRule()
        <process the file>
        FileEndRule()
    else:
        # targetPath is a directory ("project")
        for (each file in the directory tree):
            FileStartRule()
            <process the file>
            FileEndRule()
    ProjectRules()
SessionEndRule()

(I have already implemented the above, and hope to create a pull request with the changes in the near future.)

Hi all,

we use Doxygen with the following syntax :

/*!
 * \fn          
 * \param   
 * \brief      
 */

But it is considered as violation by RULE_5_3_A ...
Though it is a proper way to use Doxygen :
https://www.doxygen.nl/manual/docblocks.html

Is it a big deal to fix?

Thanks a lot.

Links to nsiqcppstyle.appspot.com yield 404. If no new site exists to replace the now-defunct, please consider creating the wiki for this project instead.

I would also greatly appreciate if anyone could link me / send me the information about the rules. I would like to take this tool into use as a part of automated student submission checking.

Problem Description

Currently any rule file used in an NsiqCppStyle session (e.g., RULE_3_1_A_do_not_start_filename_with_underbar) must reside in the nsiqcppstyle\rules package folder. You can see this in nsiqcppstyle_runmanager.py:

def LoadRules(self, checkingRuleNames):
    ...
    for ruleName in checkingRuleNames:
        ruleModule = __import__("rules." + ruleName)
        self.loadedRule.append(ruleModule)
    ...

I have written many proprietary rules that I do not want to return to the NsiqCppStyle repository. To include them in my NsiqCppStyle checking runs, I must copy my rules to the rules folder. Should I upgrade NsiqCppStyle to a newer version I may need to copy all of my proprietary rules again. Should I push changes through a GitHub pull request, I have to be careful not to add my proprietary rules to the commit. In general, I want to keep my proprietary rules totally separate from the NsiqCppStyle package.

Proposed Solution

The proposal is to support a new command-line option for specifying one or more additional paths that should be searched for rules. These paths can be anywhere in the file system. For all rule names in the filefilter.txt file, NsiqCppStyle should search first the rules folder, followed by any additional folders. If found, the engine should import the rule file. The following Stack Overflow post provides Python code that can be used to import the rule file from anywhere. Here is a test script I wrote:

d:\junk\import-from-anywhere\main.py

import importlib.util
import sys
spec = importlib.util.spec_from_file_location("test.py", "d:\\junk\\test.py")
foo = importlib.util.module_from_spec(spec)
sys.modules["test.py"] = foo
spec.loader.exec_module(foo)
print("foobar=%d" % (foo.foobar))

d:\junk\test.py

foobar = 5

Running main.py shows the import in action:

D:\Junk\NsiqCppStyle\Import-From-Anywhere>python main.py
foobar=5

I experience many false positives in all functions that return a struct or a pointer to struct. Here is a very simple example.

filefilter.txt:

~ RULE_6_1_A_do_not_omit_function_parameter_names

nsiq_test.c:

struct test_struct
{
    int a;
    int b;
};

void func1(int *p, int x);

struct test_struct func2(void)
{
    struct test_struct ts = {1, 2};
    return ts;

    func1(NULL, 0);
}

This gives the following false positive:

/home/pnin/src/skunk/nsiq_test.c(14, 5):  function (func1) has non named parameter. use named parameter.  [RULE_6_1_A_do_not_omit_function_parameter_names]

If I change the return type of func2 to int instead of struct test_struct the error goes away. I suspect that func2 is treated as a struct instead of a function. If I dump contextStack during each iteration in nsiqcppstyle_checker.py, I get this when func2 returns an int:

FUNCTION_BLOCK, func2, LexToken(LBRACE,'{',10,1,91, False, None), LexToken(RBRACE,'}',15,1,165, False, None) >>

And this when func2 returns a struct:

STRUCT_BLOCK, func2, LexToken(LBRACE,'{',10,1,106, False, None), LexToken(RBRACE,'}',15,1,180, False, None) >>

Just a question, is this the new "official" home for nsiqcppstyle or is it just your personal copy ?

Background

NsiqCppStyle currently supports ten (10) types of token callback functions. The rule writer can register one or more of these callback functions with the engine. Currently, the following are the supported callback functions:

When the NsiqCppStyle engine processes a token, one or more callback functions may be invoked. If a user rule has registered callback functions, the rule's callback functions will be invoked. This enables the user rule to perform its work.

Here is the loop that processes the tokens and invokes the callback functions:

def RunRules(ruleManager, lexer):
    ...
    while(True):
        try:
            t = lexer.GetNextTokenSkipWhiteSpaceAndComment()
            ...
                ruleManager.RunLineRule(
                    lexer, lexer.GetCurTokenLine(), currentLine)

            if t.pp == True:
                ruleManager.RunPreprocessRule(lexer, t.contextStack)
            else:
                if t.type == 'TYPE':
                    ruleManager.RunTypeNameRule(lexer, t.value.upper(), t.fullName,
                                                t.decl, t.contextStack, t.context)
                elif t.type == 'FUNCTION':
                    ruleManager.RunFunctionNameRule(lexer, t.fullName, t.decl,
                                                    t.contextStack, t.context)
                elif t.contextStack is not None and t.contextStack.SigPeek() is not None:
                    sigContext = t.contextStack.SigPeek()
                    if sigContext.type == "FUNCTION_BLOCK":
                        ruleManager.RunFunctionScopeRule(lexer, t.contextStack)
                    elif sigContext.type in ["CLASS_BLOCK", "STRUCT_BLOCK", "ENUM_BLOCK", "NAMESPACE_BLOCK", "UNION_BLOCK"]:
                        ruleManager.RunTypeScopeRule(lexer, t.contextStack)
                ruleManager.RunRule(lexer, t.contextStack)
        ...
    try:
        ruleManager.RunFileEndRule(lexer, os.path.basename(lexer.filename),
                                   os.path.dirname(lexer.filename))
    ...

Problem Description

There is no rule callback when a COMMENT (C-Style) or CPPCOMMENT (C++-style) comment token is encountered. Thus, a user rule cannot handle a comment token as encountered in the token stream.

The request is that:

• A corresponding AddCommentRule() registration function should defined
• The registered comment rule should be invoked when either a COMMENT or CPPCOMMENT token is encountered.

Miscellaneous

The RULE_5_3_A_provide_doxygen_function_comment_on_function_in_header rule checks that function declarations are preceded with a Doxygen comment. Because the rule was written without a COMMENT callback, it waits until a FunctionName callback is invoked. At that point, it calls lexer.GetPrevTokenInType("COMMENT") to find the preceding COMMENT token, etc.

Sometimes back-searching for the previous COMMENT token is not convenient. A while back I wrote a proprietary rule that processes hundreds of source files, identifying all Google Test unit test functions. Our coding standard for these functions was to document the functions with a sequence of multiple COMMENT (not CPPCOMMENT) lines. Rather than processing the multiple COMMENT lines post-facto, I wanted to process the comment lines as they were encountered. I, therefore, implemented what is requested in this issue, and would like to merge the implementation back into the repo.

In any case, I believe that a COMMENT or CPPCOMMENT is as valid a token as any, and should have its own dedicated callback function.

Lambdas seem to always produce violations of RULE_4_5_A_matching_braces_inside_of_function_should_be_located_same_column

For example, using async boost callbacks:

auto onAccept = [this](boost::system::error_code ec) {
    if (!ec) {
      RxSessionType::create(this)->start();
    } else {
      std::cout << "Experienced error on accept: " << ec << std::endl;
    }
    _acceptConnections();
};

In the above, it would report both the lambda body braces and if/else braces.

The Problem

NsiqCppStyle has anachronistic code related to updating the local NsiqCppStyle code base. Apparently, NsiqCppStyle was meant to check the code repo at http://nsiqcppstyle.appspot.com. Now that NsiqCppStyle is hosted on GitHub, this "feature" is no longer relevant and should be removed.

Technical Details

Vestiges of remotely updating NsiqCppStyle can be found in three source files:

• nsiqcppstyle_exe.py
• agent.py
• minjson.py

nsiqcppstyle_exe.main() is the starting point:

...
import updateagent.agent
...
def main(argv=None):
    ...
        updateNsiqCppStyle = False
        for o, a in opts:
            ...
            elif o == "--update":
                updateNsiqCppStyle = True
            elif o == "--no-update":
                updateNsiqCppStyle = False
            ...

        if updateNsiqCppStyle:
            console.Out.Ci(console.Separator)
            try:
                updateagent.agent.Update(version)
            except Exception as e:
                console.Out.Error(e)
        ...

The above code makes a call to updateagent.agent.Update(currentVersion), which does the heavy work. Update() also imports updateagent.minjson, which is used to read in JSON data. Now that JSON is fully supported in Python 3, minjson.py is no longer relevant.

In short, the entire updateagent folder should be removed, and related changes made to the remaining code base.

Hello,

I was going though the code and noticed a weird indentation here :
https://github.com/kunaltyagi/nsiqcppstyle/blob/master/nsiqcppstyle_exe.py#L266
This else is currently attached to the previous for instead of the if as the comment suggests :
https://github.com/kunaltyagi/nsiqcppstyle/blob/master/nsiqcppstyle_exe.py#L218

This change was introduced by this commit : f5767c102

Problem Description

NsiqCppStyle processes C++ files that have an extension found in the list (see nsiqcppstyle_exe.py for the extension name logic):

["cpp", "h", "c", "hxx", "cxx", "hpp", "cc", "hh", "m", "mm"]

When I attempted to process a file (under Windows) with the external name of "ACCESSDB.CPP", NsiqCppStyle did not process it.

Technical Reason

nsiqcppstyle_exe.py defines accepted extensions:

extLangMap = {
    "Html": set(["htm", "html"]),
    "Java": set(["java"]),
    "Javascript/ActionScript": set(["js", "as"]),
    "JSP/PHP": set(["jsp", "php", "JSP", "PHP"]),
    "C/C++": set(["cpp", "h", "c", "hxx", "cxx", "hpp", "cc", "hh", "m", "mm"])
}
...
cExtendstionSet = extLangMap.get("C/C++")
...
# if the target is file, analyze it without condition
if os.path.isfile(targetPath):
    `= targetPath[targetPath.rfind('.') + 1:]
    if fileExtension in cExtendstionSet:
        ProcessFile(ruleManager, targetPath, analyzedFiles)

The search for fileExtension in cExtendstionSet is case-sensitive. This is correct for Linux, but for Windows the engine should perform a case-insensitive search.

Suggested fix

If running under Windows, the search should be case-insensitive.