NEEDS TO BE FINISHED: detail as to what and why.
- Automate AD-Hoc scanning
- Address issues with AWS scanning with Qualys. Automatically update asset groups with
- Automate creation of asset groups to ensure standard naming and configuration across all Autodesk business units
- Can you provide exact details of what the script does in regards to Mongo
provide details about what pycrypto does in regards to encryption of credentials
##INSTALLATION:##
Dependencies
Python 2.7.8
Python dependencies:
qualysapi: (API for Qualys vulnerability scanner)
boto: (Amazon API)
netaddr: (Used for check rules in the security groups)
jinja2: (Report generation)
lxml: (xml parser)
pymongo: (mongo db interaction)
pycrypto: (cryptographic functions)
###Windows Installation:###
Install Python environment
- Install python 32 bit for Windows adding executables to the path (python version 2 is used because version 3 does not support exec file) https://www.python.org/ftp/python/2.7.8/python-2.7.8.msi
- Install pip (>python get-pip.py), where get-pip.py was obtained from https://bootstrap.pypa.io/get-pip.py
(optional) Install Eclipse
Once Eclipse has been installed, download extensions for python support: *Install PyDev for eclipse (Help -> Install new software) http://pydev.org/updates *Configure the python interpreter (Window -> Preferences -> PyDev -> Interpreter -> Python)
Install dependencies
Go to python scripts folder (C:\Python27\scripts) and run:
pip install qualysapi
pip install boto
pip install netaddr
pip install jinja2
pip install lxml
pip install pymongo
Install compiler (pycrypto requires to be compiled) Install MS Visual C++ 2008 express edition download
Install pycrypto
pip install pycrypto
Delete config file
Delete config.ini
(this is a config file that is automatically created the first time you run the script and enter username/password and is found in the same directory that the script executes from)
Install Mongo DB
unzip mongo distribution
# Run mongo:
mongod.exe --dbpath C:\Mongo\data
###Linux Installation: (verified on A360 hardened CentOS 6.5)###
Install dependencies
sudo easy_install pip
sudo pip install qualysapi
sudo pip install --upgrade qualysapi
sudo pip install boto
sudo pip install netaddr
sudo pip install jinja2
sudo pip install pymongo
sudo yum groupinstall 'Development Tools'
sudo yum install -y python-devel bzip2-devel libxml2-devel libxslt-devel libffi-devel libevent-devel
sudo pip install lxml
Install Pycrypto
mount /tmp -o remount,exec,rw
sudo pip install pycrypto
mount /tmp -o remount,noexec,rw
Note: Because dependencies where installed with sudo, python commands that requires the dependencies will have to run with sudo.
Delete config file
Delete .qcrc (is found in the same directory that the script executes from).
'.qcrc' is a config file that is automatically created the first time the script is runned and the user enters a valid username/password for accessing qualys
Install Mongo DB
sudo pip install pymongo
##DEFINE CONFIGURATION:##
###Configure Mongo###
The scripts interact with an external MongoDB to ensure AWS assets are tracked by instance ID.
Update the Mongo Config (Edit mongo_config.py)
# Update Mongo (yes/no)
available='no'
# MongoDB IP address
ip='127.0.0.1'
# MongoDB Port
port=27017
# Name of the client/database in MongoDB
client_name='MyAWSInfo'
# Collection name where instance info is stored
collection_name='ec2_info'
# Collection name where timestamp of last update for an instance is stored
collection_timestamp='ec2_time'
###Build Configuration Files###
Update Asset Groups configuration file (Edit asset_groups.py)
These asset groups are used to store all internal/external IP's as well as IP's for ad-hoc scans
# Asset group name that holds all external IP addresses in EC2
external='A360-Test-External'
# Asset group name for storing all internal IP addresses in EC2
internal='A360-Test-Internal'
# Asset group name that holds all ip addresses for ad-hoc scans
adhoc='A360 - AdHoc Scanning'
# Asset group id for adhoc scan group
adhoc_id='1432205'
# CSV of VPC accounts hosting multiple products
vpc='A360-PRODUCTION-VPC East-EC2,A360-STAGE-VPC West-EC2'
*ID is used because Qualys API requires the id for launching the report based on an asset group. It does not support asset group names ** These are only required for environments with VPC's that host multiple applications or environments that are defined by subnets. It is possible to run a single scan of all subnets using a single asset group and then report based on defined asset groups
Update Authentication records configuration file (Edit authentication.py)
(The authentication config defines the different authentication records that can be defined in the ad-hoc scans defined with โi option in cmd line. There is a default authentication record that will be used if none other is specified. Records are defined as Windows (win) or Linux (nix) )
default='nix_A360USER'
nix_A360USER='280393'
nix_ADSKSAAS='278847'
nix_ADSKSAASAWS='285088'
nix_BUZZSAW='278848'
win_A360USER='279774'
Update Mail Relay configuration file (Edit mail_config.py)
(this is required to send notifications of scans and reports. For EC2 scans it enumerates all instances, by region and identifies small/micro and non-running instances which will be filtered from scanning per AWS terms of use)
#IP address of the mail server
server='10.151.16.251'
#Message characteristic of the amazon instances's report.
reports_send_from='[email protected]'
reports_subject='Qualys Amazon EC2 Reports (DO NOT REPLY)'
reports_send_to='[email protected],[email protected]'
reports_text='Those reports were generated automatically, please DO NOT REPLY'
#Message characteristic of the scan reports
scan_send_from='[email protected]'
scan_send_to='[email protected]'
scan_text='These report were generated automatically, please DO NOT REPLY'
Update Report Template configuration file (Modify report_templates.py)
(Defines the VM and PC templates that will be used for report generation. ID's of the templates can be found in Qualys)
#VM Report TemplateName: A360 Brief Summary Sev 1-5 (excluding non-running kernels)
vm_template='1671283'
#PC Report Template Name: A360 Benchmark Reference Failed Checks Only
pc_template='1654795'
Update Scan Profile configuration file (Modify scan_profile.py)
(Defines the scan option profiles used for VM and PC scans and the policy ID's used for PC reporting)
#VM Scan Option Profile: Step 4 Authenticated Scan
internal_scan='462100'
#VM Scan Option Profile: External Scan Ahsans Options
external_scan='736834'
#PC Scan Option Profile: A360 Policy Compliance Scan
pc_scan='739787'
#Policies
#A360 - CentOS 6.5, v1.0.0 (CIS based policy)
policy_centos65='88347'
policy_centos65name='CentOS 6.5'
#A360 - Windows Server 2008R2 v1.0.0 (CIS based policy)
policy_win2008R2='85700'
policy_win2008R2name='Windows 2008R2'
Update Scan Appliance configuration file 1.scanner.py (Defines the relationship between scan appliance name and appliance ID and is referenced using the โs switch from the command line)
a360_dc_ashburn='20933'
a360_dc_sanfrancisco='24028'
a360_dc_santaclara='24028'
aws_use_1='70647'
aws_usw_1='77247'
##Scripts Usage:##
- Create asset groups from CSV file
Asset Group CSV file requirements (The script assume there is a header in the first line, which is ignored in the script execution)
Organization | Environment | Product | Account Type | Existing AG |
---|---|---|---|---|
The Autodesk unit that owns the assets (ACS/ A360/ EIS etc) | The environment that the assets reside in (Production/ Stage/ Development) | The name of the product to which the assets belong | Where the assets reside (VPC/ EC2) blank is datacenter | If there is an existing asset group that contains Ips and you want to migrate to the new standard, define the group name and IP's will be added to the new group |
1.1 Script excecution
python qualys.py -a csv -f <csv file>
-f specifies the name of the .csv file containing the information about the asset groups
1.2 Example
python qualys.py -a csv -f myassetgroups.csv
- Encrypt AWS account file
Encrypt AWS credentials CSV file requirements (The script assume there is a header in the first line, which is ignored in the script execution)
CSV fields:
AccountName | AccountNumber | Key | Secret | Asset Group Name |
---|---|---|---|---|
Name of the AWS account | The account number of the AWS account | The IAM key for the AWS account | The IAM secret for the AWS account | Defines the name of the asset group that will be updated |
2.1 Script excecution
python qualys.py -a cipher -f <csv file>
-f specifies the name of the .csv file containing the information about the AWS accounts
After running the commmand, a new file is created with the name .enc. This then deletes the clear text file (filesystem wipe)
2.2 Example
python qualys.py -a cipher -f AWS-Credentials.csv
3 AWS Instance reporting
Generates a report showing instance type, whether the security groups allow Qualys access for scanning and whether instances are in a VPC. It reads each amazon account and make different validation including: instance type, access rules from internal scanners, instances in vpc. This report excludes asset groups which are defined like VPC (requires to have asset group config in CSV file for AWS account and have the asset group like vpc in asset_groups.py)
3.1 Script excecution
python qualys.py -a report -f <encrypted csv file with AWS accounts>
If file is not provided, it will try to use the default one: A360.TREND.DEEP.SVC.csv.enc
3.2 Example
python qualys.py -a report -f A360.TREND.DEEP.SVC.csv.enc
4 Update asset group information based on AWS accounts
It reads each amazon account and update external, internal and asset group info (if avalaible in encrypted csv file with AWS accounts)
4.1 Script excecution
python qualys.py -a update -f <encrypted csv file with AWS accounts>
If file is not provided, it will try to use the default one: A360.TREND.DEEP.SVC.csv.enc
4.2 Example
python qualys.py -a update -f A360.TREND.DEEP.SVC.csv.enc
- ADHOC scan by AWS account number
Launch an external an internal scan in east and west region for the external and internal asset group This scan does not generate report
5.1 Script excecution
python qualys -a scanby -n <number> -s <scanner name> -i <auth name> -m <mail addresses>
<number> is the amazon acocunt number to scan
<scanner name> is the scanner to use (as defined in scanner.py)
<auth name> is the authentication name to use (as defined in authentication.py)
<mail addresses> is a comma separated list of email address where the VM and PC report will be send
if 'default' string is provided it will send the email to the default email recipients defined in the file mail_config.py in the attribute 'scan_send_to'
-i is optional. If not auth name is provided, nix_A360USER is used
-m is optional. If not provided it will not send the report by email. The report file will be stored in the report_output folder
5.2 Example
python qualys.py -a scanby -n 892067615439 -s aws_use_1
python qualys.py -a scanby -n 892067615439 -s aws_use_1 -i WA360USER
python qualys.py -a scanby -n 892067615439 -s aws_use_1 -i WA360USER -m default
python qualys.py -a scanby -n 892067615439 -s aws_use_1 -i -m [email protected],[email protected]
6 ADHOC External Scan by IP addresses or asset group
Launch an external scan to a list of IP addresses
6.1 Script excecution
python qualys -a scanby -t <IP addresses or asset group> -c <aws account> -f <encrypted filename with amazon accounts> -m <mail addresses>
<IP addresses or asset group> A comma separated list of IP address
<aws account> AWS account number. Is required to validate if the IP address to scan belongs to the AWS account <mail addresses> is a comma separated list of email address where the VM and PC report will be send
if 'default' string is provided it will send the email to the default email recipients defined in the file mail_config.py in the attribute 'scan_send_to'
-c is optional only if the scan is by asset group
-m is optional. If not provided it will not send the report by email. The report file will be stored in the report_output folder
6.2 Example
python qualys.py -a scanby -t 54.197.148.95 -c 892067615439
python qualys.py -a scanby -t A360-Test-External
python qualys.py -a scanby -t 54.197.148.95 -c 892067615439 -m default
python qualys.py -a scanby -t A360-Test-External -m [email protected],[email protected]