santandersecurityresearch / drheader Goto Github PK
View Code? Open in Web Editor NEWdrHEADer helps with the audit of security headers received in response to a single request or a list of requests.
License: MIT License
drHEADer helps with the audit of security headers received in response to a single request or a list of requests.
License: MIT License
Looking at the findings itself, it is not possible to figure out which values are expected for the scanned target. Please improve the output of the scanner especially for expected and avoid values.
To make drheader easier to install and use we should add uploading to pypi to our release action. This is
We can safeguard the credentials by storing them as github repository secrets , these can then be injected into build scripts and will be excluded from log output.
To make the most of the sphinx documentation in Dr Header we will need to make a few adjustments to make it work with github pages.
We could also at a github action to do this for us and run it as part of the release process
info here on how to do it : https://daler.github.io/sphinxdoc-test/includeme.html
To give users the best experience using Dr Header the documentation will need an update to tweak it more approachable.
We should show a demo with screenshots of it being using against santander.co.uk
We should extract the sphinx documentation and publish it via github pages , this will be seperate task , once its done we should have links into it from the main documentation.
Add setuptools to requirements.txt and increase version
Now that Github Actions is out of beta there have a bigger library of actions that we can use.
Our actions need an update to use these where possible to reduce the risks of 3rd party access to the builds.
The release task need to switch from softprops/action-gh-release@v1 to https://github.com/actions/create-release
In https://github.com/Santandersecurityresearch/DrHeader/blob/master/.github/workflows/release.yml there is a build configuration. It should be run when the a new release tag gets created but it doesn't seem to be happening. This needs investigating and fixing.
Yesterday we released some changes in master to trigger a new action to automatically release our wheel in pypi. Those changes need to be reflected in develop branch too
When forking the project on GitHub it is marked as a Template and is annoying for the final user.
Fork your repository and try to create a new project under your user. The DrHeader repo is shown as a recommended Template.
When runing the saftey scan as part of a build job the following bad dependencies where detected
Run python -m safety check
safety report
checked 79 packages, using default DB
---
-> pycrypto, installed 2.6.1, affected <=2.6.1, id 35015
-> cryptography, installed 2.1.4, affected >=1.9.0,<2.3, id 36351
##[error]Process completed with exit code 255.
Its interesting that the other tools didn't pick this up.
We need to update these dependencies and check that code still works.
Getting the version from the command line
drheader --version
or
drheader -v
display:
version: 1.3.0
get version from the cli
Review test implementation and fix it if needed
To keep ahead of security issues with dependencies we can use dependabot.
https://help.github.com/en/articles/configuring-automated-security-fixes
There are several options to enable and we might have to introduce an insecure dependency to check it.
As the last release (1.6.0) is already multiple month old, could you release a new version (1.7.0)?
I am waiting for the possibility to include custom severity settings via rules.yml
Requirements.txt needs jsonschema 3.1.1, but setup.py needs jsonschema 3.0.2. Both files should be aligned and request the same version
Add flake8, bandit and safety as pre-requirements to perform a merge
Would be great to have DrHEADer have a flag --sarif to output results in the SARIF format.
Current rule for Set-Cookie header is:
Set-Cookie:
Required: Optional
Enforce: False
Value:
Must-Contain:
- HttpOnly
- Secure
It is an optional header (not required). However, when scanning a target that does not respond with a Set-Cookie header, DrHeader returns the following error:
----
rule | Set-Cookie
severity | high
message | Header not included in response
----
This is not expected behaviour, as policy mandates that Set-Cookie is not a required header. Hence, when there's no Set-Cookie set in response, no error should be returned.
drheader scan single <target>
The release doesn't have a tag so its not getting a version number.
if you have switched the github actions plugin used it might not be doing all the stages the previous one did.
i noticed this issue because i saw this in the release section
the tag icon doesn't have a value .
this plugin can make tags for you
https://github.com/marketplace/actions/github-create-tag-release
drHEADer version: 1.7.0
DrHEADer supports to evaluate HSTS (Strict-Transport-Security). For this header, the value "max-age" is needed. As soon as the max-age is not exactly equal to the value from the yaml file, DrHEADer triggers a finding.
Please implement an evaluation, if the set of the evaluated target for max-age is higher or equal to the set value of the DrHEADer yaml file. This would reduce the number of findings of DrHEADer and makes the evaluation of HSTS more reliable.
drHEADers CLI exit status is currently always zero, regardless of any issues found during a scan.
Adding an option to fail with the correct error statuses would allow pipelines to gracefully fail builds if unwanted headers were identified.
drHEADer version: 1.7.0
drHEADer should output a valid JSON file which is empty in case a target can't be evaluated (e.g. if not reachable).
At the moment no JSON output is produced in such a case.
Summarizing, the option "--json" gives inconsistent output in various scenarios.
Review the tests to make sure all the scenarios are covered and add extra tests if needed
Currently, the only accepted value for cache-control header as per default policy is 'no-cache, no-store, must-revalidate'
We've received a suggestion to also allow as a valid value: 'no-cache, no-store, must-revalidate, max-age=0'
Would it be possible to modify the default policy to allow either of these two values?
DrHeader currently has one test class which contains only a few tests and this doesn't seem to give much coverage of its functionality.
Tests follow the form of preparing a headers definition , then feeding this through an instance of DrHeader and the asserting on the results.
This is currently done in individual test methods which is time consuming.
We can recreate them as python generator based test which could iterate through a large data structure containing a test name , parameters and expected results.
This would allow for new tests to be added easily and for coverage to be rapidly improved.
Pytest is particularly good at this and various methods to achieve it are explained here
http://doc.pytest.org/en/latest/example/parametrize.html
Rules for cookies are currently only able to be applied to all cookies. Add the option to define rules that apply only to specific cookies
When building a distribution of Dr Header there is an error building the YAML dependencies.
Running PyYAML-5.1.2/setup.py -q bdist_egg --dist-dir /var/folders/jx/c1xk5bf5693_6kdwh7ypcgd80000gn/T/easy_install-comdawnw/PyYAML-5.1.2/egg-dist-tmp-lba4j7gs
In file included from ext/_yaml.c:596:
ext/_yaml.h:2:10: fatal error: 'yaml.h' file not found
#include <yaml.h>## What I Did
This looks like its just missing the libyaml-dev package which will need apt-get installing in the build scripts. We aren't building eggs anymore so the setup command has changed , however that won't stop this issue.
At the moment, drheader only checks if the the values allowed/forbidden exist in the csp header, but it doesnt check to which directive it belongs.
It will be a good improvement to add more granularity and validate the rules per directive
Drheader currently needs a list of headers to run a bulk scan. It would be a good idea to integrate this with the most common MiTM tools, as ZAP/Burp, so drheader can read the headers in the traffic captured and run the scan against that data
Headers are case insensitive, according to RFC 2616. Drheader is comparing the header names without ignoring the case, and therefore some headers are not properly validated
It was a very simple test. I just created a report file with a header called "server" in lower case. Since this is a header that should not exist, I was expecting Drheader to report it but it didn't as the header it was looking for was "Server"
Changes in github actions to:
These changes will remove the need to manually modify any files with new version for ongoing release or having to generate manually the description file for pypi upload.
Include documentation about "rules.yml" file drHEADer uses as its policy to audit security headers. It will document file format, fields meaning and how to make changes to it.
During the update of the documentation for Dr Headers rules specification it has been noticed that the format of the rules.yml format isn't particularly usable.
Some of the parameters duplicate functionality because they were mostly considered from the development perspective and not so much with usability for all our new users in mind.
This would be a great task for someone wanting to get involved in DrHeader development so we would like to throw this open to the community.
Please add in comments to this issue any specific instances of the configuration needing improvement.
These will be reviewed in a later task which will produce a YAML schema and the corresponding tickets to do the work.
class Drheader:
"""
Something about the core should probably go here
"""
error_types = {1: 'Header not included in response', 2: 'Header should not be returned',
3: 'Value does not match security policy',
4: 'Must-Contain directive missed', 5: 'Must-Avoid directive included'}
Python 3.7 has now reached end of life (https://endoflife.date/python). Additionally, the use of Python 3.7 is incompatible with newer versions of dependencies that have dropped support for <3.8
Drheader can detect headers that leak sensitive information, as server, or X-Client-IP. We want to add another header, x-ibm-client-secret, to the list of headers not allowed
The new header to be added to rules.yml should be:
X-Ibm-Client-Secret:
Required: False
Enforce: False
Value:
DrHeader exit with exit code 0 when detecting issues, it should return a non zero code, then if execute via command-line tools we could use the exit code to understand if there were any errors.
$ drheader scan single https://santander.com
....
$ echo $?
0
When providing invalid arguments, or invalid URLs is already returning a non zero exit code.
$ drheader scan single https://owasp.com
Error: Failed to get headers.
$ echo $?
1
I expect to get a result when I avoid tls certification via cli
I am trying to avoid ssl verification in order to avoid the following error:
drheader scan single https://my-url.org --debug
DEBUG:root:Validating: https://my-url.org
DEBUG:root:Querying headers...
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): my-url.org :888
Error: HTTPSConnectionPool(host='my-url.org', port=888): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)')))
When I add the param verify=false which should disable tls verification, I am getting the following error:
drheader scan single https://my-url.org --verify=false
Usage: drheader scan single [OPTIONS] TARGET_URL
Try 'drheader scan single --help' for help.
Error: no such option: --verify
setup.py has some missing metadata fields for AUTHOR and MAINTAINER.
This should be updated after a quick discussion about what we want to put it in it which
will probably cover how we want to support this.
That might also trigger so documentation updates and possibly support ticket template changes.
To help normalise pull requests and allow people do discover while doing
we can add a template to the repository that will define the template for new pull requests
this can also include basic requirements.
A starter set could be :
This functionality to do this in github is documented here
https://help.github.com/en/github/building-a-strong-community/creating-a-pull-request-template-for-your-repository
create template in .github/.github/pull_request_template.md
It will be nice to have an option on the cli to download the yml file with the custom URL from a remote sever, rather than having to provide this form a local file.
According to RFC 2616 https://www.w3.org/Protocols/rfc2616/rfc2616.html and their updates (7230-7237), headers are not case sensitive.
The tool is doing an exact match between rules and values received, so it should do a case insensitive validation
It would be nice to have DrHeader able to support some sort of suppression mechanism, to avoid flagging known/accepted issues.
For instance, if I'm happy with unsafe-inline for style (e.g. I'm using angular), I can pass some suppression.yaml file with something like:
rule: Content-Security-Policy
- location: style-src
It will be great if DrHeader can produce kind of report in JUnit format, that we can then parse in Jenkins to be use as a traceable source for historic issues.
In addition, this report could be used to break pipelines when issues are present, as per JUnit Jenkins plugin functionality.
We need to create a release to test the release process. This can be done as follows.
bumpversion will check all the version strings in the code base match up to the specified new version.
Create a new branch
check it out
run bumpversion
verify changes.
push to branch
create PR
This Project should be updated according to the best practice reommendations of the OWASP Secure Headers Project https://owasp.org/www-project-secure-headers/
Multiple Headers suggested in the OWASP Secure Headers Project are not scanned with DrHeader (e.g. Cross-Origin-Opener-Policy).
Furthermore, we should merge development into master (Last release was Nov 2, 2020) to apply the deprecated X-XSS header #137.
As per current OWASP recommendations, I suggest we change the value enforced by default policy for 'X-XSS-Protection' header to '0'.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.