Comments (6)
Now that we have one unit test done with #70, can we please close this issue?
Right now there's too many threads for me to follow left and right and I'm trying to cut that number down if possible.
As far as I can tell, we have answered the original questions and everything else can be tracked in their respective issues.
from phpcs-security-audit.
It is interesting indeed.
However right now my priorities would have to go into documentation instead.
If you find unit tests that fails, that mean we need to update the code to make it right? At this point, as I said in #56, I can't focus myself on that, so I would need people to help with the potential generated work this might create.
Before you start we need to recognize that since this is a security tool, its philosophy is about limiting the number of false negative. Diminishing the false positive will be good, but the tests should have a focus on making sure we're not missing something we should have caught.
We also need to agree on the difference between unit tests and the tests.php file. Right now I do have the requirement of seeing that every sniff raise at least one warning/error in the tests.php file. I also want people to be able to run the tool normally (as running a scan) on tests.php and be able to see the results. I don't know how much what you have in mind would clash with that, but I see a difference between running a test suite and offering a file to see the results of an example scan that hit all rules. We could simply keep both solution, and at best, generate the file from the tests cases if it's easy/possible.
Thanks!
from phpcs-security-audit.
Before you start we need to recognize that since this is a security tool, its philosophy is about limiting the number of false negative. Diminishing the false positive will be good, but the tests should have a focus on making sure we're not missing something we should have caught.
IMO the tests should cover both: prevent false positives and false negatives and ensure that what needs to be reported is reported.
We also need to agree on the difference between unit tests and the tests.php file. Right now I do have the requirement of seeing that every sniff raise at least one warning/error in the tests.php file. I also want people to be able to run the tool normally (as running a scan) on tests.php and be able to see the results. I don't know how much what you have in mind would clash with that, but I see a difference between running a test suite and offering a file to see the results of an example scan that hit all rules. We could simply keep both solution, and at best, generate the file from the tests cases if it's easy/possible.
Well, as a start I would split out the test cases from tests.php
into their own files.
Keeping both will make maintenance more involved as that wouldn't be easily enforced (while having the individual unit tests files can be).
All the same, if I understand you correctly, your concern is more for it to be easy for people to see the results of an example scan. Correct ?
With the split out test cases, that would still be possible with only a minimal adjustment to the command: phpcs ./Security/Tests/ --standard=Security --extensions=inc
would run the scan over all the test case files.
Basically if we'd base the unit tests off the PHPCS native test suite, each sniff will get a SniffNameUnitTest.inc
file with test cases for that specific sniff and a SniffNameUnitTest.php
file with basically just two simple data providers stating on which lines to expect errors and on which lines to expect warnings (and how many on each line).
The PHPCS native TestCase will then run the sniff over the test case file and check that all lines where errors/warnings are expected have those and that all other lines do not have any errors/warnings.
If you like, have a look here for an example of what that looks like: https://github.com/PHPCSStandards/PHPCSExtra/tree/develop/Universal/Tests/Arrays
from phpcs-security-audit.
@jmarcil What about if I start with just adding the unit test setup and then go through every sniff individually (separate PR for each sniff) to set up the unit tests and make any quick fixes for false positives/false negatives I see ?
I can open issues for anything I come across which are false positives/negatives which require more extensive fixes/refactoring of the sniffs.
from phpcs-security-audit.
@jmarcil FYI: I've created an initial setup with tests for one sniff so far.
You can have a look at it here: https://github.com/jrfnl/phpcs-security-audit/tree/feature/initial-unit-test-setup and see a passing Travis build which includes running the unit tests here: https://travis-ci.org/jrfnl/phpcs-security-audit/builds/661271491
from phpcs-security-audit.
If you find unit tests that fails, that mean we need to update the code to make it right? At this point, as I said in #56, I can't focus myself on that, so I would need people to help with the potential generated work this might create.
Oh and just to be clear: help is exactly what I'm offering.
from phpcs-security-audit.
Related Issues (20)
- Fix compliance of project with PHPCS HOT 10
- Strings as assert expressions are deprecated. HOT 4
- Figure out repo organization and ownership for the future
- Add CI/build testing HOT 6
- PR #50 breaks drupal7 usage HOT 1
- phpcs built from Dockerfile gives an error HOT 9
- $utils::is_token_false_positive is fiddly and unstable HOT 1
- Solving EasyRFI via new EasyRFINotice severity HOT 7
- Create new release to fix deprecation warnings HOT 8
- Unable to view Security coding standard after Composer install HOT 3
- ERROR: Referenced sniff "Security.BadFunctions.Asserts" does not exist HOT 2
- Windows user, Unable to find phpcs command or bin file HOT 1
- Update security rulesets
- Potential vulnerabilities are being hidden with concatenation
- Installation instructions not working HOT 1
- Add support for native function imports
- Callback functions warnings
- file_put_contents warning about dynamic parameter
- Question for ErrMiscIncludeMismatchNoExt
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from phpcs-security-audit.