plus3it / cfn-sonarqube Goto Github PK

Previous logic for ZIP-based install not functioning — use opportunity to switch to RPM:

• Get [repo-def](/etc/yum.repos.d/sonar.repo http://downloads.sourceforge.net/project/sonar-pkg/rpm/sonar.repo)
• Install sonar RPM from new repo-def
• Update systemd unit file to:
  • Reference RPM's /opt/sonar home
  • Run as RPM user (root - no longer run as sonarqube)
• Direct pull of LDAP plugin via URL
• Update which sonar.properties location to target

Problem Description:

AWS has released new instance types that might better align to some deployment-scopes

Expected Behavior:

Support t3 and m5 instance-types where possible

Actual Behavior:

Does not currently support t3 and m5 instance-types at all

(Optional) Fix recommendation:

Update template logic to allow for t3 and m5 instance-types

Solution currently leverages S3 for hosting of service's daily backups. No lifecycle tiering or expiration is enabled. Probably be useful to add a lifecycle policy similar to:

{
    "Rules": [
        {
            "Status": "Enabled",
            "NoncurrentVersionExpiration": {
                "NoncurrentDays": 180
            },
            "NoncurrentVersionTransitions": [
                {
                    "NoncurrentDays": 3,
                    "StorageClass": "GLACIER"
                }
            ],
            "Filter": {
                "Prefix": "Backups/"
            },
            "Expiration": {
                "Days": 45
            },
            "AbortIncompleteMultipartUpload": {
                "DaysAfterInitiation": 7
            },
            "Transitions": [
                {
                    "Days": 5,
                    "StorageClass": "GLACIER"
                }
            ],
            "ID": "BackupTiering"
        }
    ]
}

Problem Description:

It may be desirable to offer the ability to customize database tuning-options. Need the DB to use a custom — rather than the currently used RDS-default — parameter group.

Expected Behavior:

Ability to tune DB behavior via DB parameter-group settings

Actual Behavior:

Current use of RDS-default DB parameter-group precludes tuning customizations

(Detailed) Steps to reproduce:

Deploy RDS DB from existing templates

(Optional) Fix recommendation:

Add a AWS::RDS::DBParameterGroup resource-type into the current RDS templating.

Problem Description:

Amazon Certificate Manager (ACM) is not available for use in all regions/partitions. In these regions/partitions, it will be necessary to use Identity and Access Management (IAM) to host SSL certificates used for ELB-based SSL-termination. To maximize portability, ELB templates should allow use of either ACM- or IAM-hosted SSL certificates.

Expected Behavior:

ELBs support SSL-termination whether ACM is available for use in a given region/partition.

Actual Behavior:

ELBs do not currently support SSL-termination when ACM is unavailable for use in a given region/partition.

Affected Components

The following templates need remediation:

• make_sonarqube_ELBv1.tmplt.json
• make_sonarqube_ELBv2.tmplt.json

Fix recommendation:

Add a Condition{} and Parameters{} components and associated logic within the Resources{} sections to support selection of ACM- or IAM-hosted SSL certificates when launching an ELB template.

Problem Description:

Instances are deployed with default JVM settings. Need to tune based on instance size (and set instance-size based on expected workload-maximums

Expected Behavior:

Web, ElasticSearch and ComputeEngine should be tuned on proportional quiesced freemem basis. Currently desired ratio is 25%/37.5%/37.5%

Actual Behavior:

JVMs runs at 1GiB defaults

(Optional) Fix recommendation:

Need to modify the sqb_app_setup.sh file to set:

sonar.web.javaOpts=-Xmx$(25% of FREEMEM)
sonar.search.javaOpts=-Xmx$(37.5% of FREEMEM)
sonar.ce.javaOpts=-Xmx$(37.5% of FREEMEM)

See Sonar tuning guide for reference.

Problem Description:

ElasticSearch won't start as root. Sonarqube needs to start as a non-root user so that ElasticSearch will also run under that non-root user.

Expected Behavior:

Sonarqube should start as the user sonar:

• The sonar RPM creates a sonar user as necessary
• The bulk of the files within the RPM are installed user:group sonar
• The /opt/sonar/bin/linux-x86-64/sonar.sh script has a RUN_AS_USER parameter that should be set to sonar.

Actual Behavior:

Sonarqube (and sub-processes that are able to) start as the user root because the RUN_AS_USER parameter in the /opt/sonar/bin/linux-x86-64/sonar.sh script is undefined

(Detailed) Steps to reproduce:

Install sonarqube from RPM. Set up sonar.properties. Start sonarqube service. ElasticSearch elements refuse to start - logging an error about prohibition against being started as the root user.

(Optional) Fix recommendation:

"Massage" the /opt/sonar/bin/linux-x86-64/sonar.sh file to ensure that RUN_AS_USER parameter is set to sonar.

May also be fixable via use of User= option in the systemd service-definition

Self-explanatory

Problem Description:

Sonarqube's ElasticSearch startup complains:

WARN  es[][o.e.b.Natives] cannot check if running as root because JNA is not available 
WARN  es[][o.e.b.Natives] cannot install system call filter because JNA is not available 
WARN  es[][o.e.b.Natives] cannot register console handler because JNA is not available 
WARN  es[][o.e.b.Natives] cannot getrlimit RLIMIT_NPROC because JNA is not available
WARN  es[][o.e.b.Natives] cannot getrlimit RLIMIT_AS because JNA is not available
WARN  es[][o.e.b.Natives] cannot getrlimit RLIMIT_FSIZE because JNA is not available

Expected Behavior:

Sonarqube's ElasticSearch does not emit this particular diagnostic message

Actual Behavior:

Sonarqube's ElasticSearch defaults to using /tmp for its jna.tmpdir. STIG-hardened systems place the noexec control on this directory which prevents proper functioning of JNA.

(Detailed) Steps to reproduce:

Harden host system, then start Sonarqube.

(Optional) Fix recommendation:

JNA needs to be configured to use an alternate temp-directory for its activities. Should be able to use the ES_JAVA_OPTS="-Djava.io.tmpdir=/SOME/OTHER/TEMP/DIR" to override the default (and, presumably, allow JNA to work). Set the preceding by adding:

sonar.search.javaAdditionalOpts=-Djava.io.tmpdir=/SOME/OTHER/TEMP/DIR

With a valid /SOME/OTHER/TEMP/DIR (possibly /var/tmp/elasticsearch) to the sonar.properties file.

Problem Description:

Custom RDS ParameterGroups can prevent doing in-place upgrades of PGSQL RDS DBs. Appears upgrades work when using default ParameterGroups (at least when executed outside the CFn context). Need to add a toggle for flipping back and forth between custom and generic when doing stack-updates

Expected Behavior:

Upgrading from 9.x to 10.x succeeds even if custom ParameterGroups are attached.

Actual Behavior:

Upgrading from 9.x to 10.x may fail if custom ParameterGroups are attached.

Fix recommendation:

Add a toggle to RDS template to flip between custom and default ParameterGroup for use during CFn-managed DB version upgrades.

Currently, CWA logging is not enabled in the EC2 template(s). When rebasing against the watchmaker 1.5.6 templates, generic CWA logging should be enabled.

Note: tackle #33 first

Probably want to attach an analytics policy to the backup S3 bucket. Something similar to the following:

{
    "AnalyticsConfiguration": {
        "Filter": {
            "Prefix": "Backups/"
        },
        "StorageClassAnalysis": {},
        "Id": "BackupTiering"
    }
}

Problem Description:

Amazon Certificate Manager (ACM) is not available for use in all regions/partitions. In these regions/partitions, it will be necessary to use Identity and Access Management (IAM) to host SSL certificates used for ELB-based SSL-termination. To maximize portability, ELB templates should allow use of either ACM- or IAM-hosted SSL certificates.

Expected Behavior:

ELBs support SSL-termination whether ACM is available for use in a given region/partition.

Actual Behavior:

ELBs do not currently support SSL-termination when ACM is unavailable for use in a given region/partition.

Affected Components

The following templates need remediation:

• make_sonarqube_ELBv1.tmplt.json
• make_sonarqube_ELBv2.tmplt.json

Fix recommendation:

Add a Condition{} and Parameters{} components and associated logic within the Resources{} sections to support selection of ACM- or IAM-hosted SSL certificates when launching an ELB template.

Probably want to attach an inventory policy to the backup S3 bucket. Something similar to the following:

{
    "InventoryConfiguration": {
        "Schedule": {
            "Frequency": "Daily"
        },
        "IsEnabled": true,
        "Destination": {
            "S3BucketDestination": {
                "Prefix": "StorageReports",
                "Bucket": "arn:aws:s3:::<DESTINATION_BUCKET_NAME>",
                "Format": "CSV"
            }
        },
        "OptionalFields": [
            "Size",
            "LastModifiedDate",
            "StorageClass"
        ],
        "IncludedObjectVersions": "Current",
        "Id": "SonarqubeLayout"
    }
}

Make sure they aren't too aggressive — particularly the EC2s'/ASGs' and RDSes'

self-explanatory

Problem Description:

Templates may not be sufficiently portable if ARNs hardcode the :aws: partition-element into them (won't work in specialty-regions like aws-cn). See AWS::Partition pseudo-param documentation.

Expected Behavior:

All templates should work in all AWS partitions

Actual Behavior:

The make_sonarqube_ELBv1.tmplt.json template will fail if not launched into the default/commercial AWS region

Fix recommendation:

Update enumerated template-files to update all "arn:aws:... string-literals to something more like:

            {
              "Fn::Join": [
                ":",
                [
                  "arn",
                  { "Ref": "AWS::Partition"},
                  …,
                  …
                ]
              ]
            }

Since initial authoring, AWS has updated available PGSQL versions. Per today's (2018-12-10) notifications, AWS is recommending updating running versions to at least 9.6.9.

AWS's currently-supported versions are (application support may vary: test if moving to a higher major):

10.4
10.3
10.1
9.6.10
9.6.9
9.6.8
9.6.6
9.6.5
9.6.3
9.6.2
9.6.1
9.5.14
9.5.13
9.5.12
9.5.10
9.5.9
9.5.7
9.5.6
9.5.4
9.5.2

Sonarqube is no longer supporting RPMs with the current version.

https://community.sonarsource.com/t/is-a-native-packages-repository-still-maintained/1228

Fix recommendation:

Revert back to the .zip install method
or
generate RPM in house.

Problem Description:

Templates last based prior to usage of CloudWatch Agent. Update to include optional CloudWatch logic

Expected Behavior:

Template installs CloudWatch agent in regions that support it.

Actual Behavior:

No hooks for CloudWatch Agent present

(Optional) Fix recommendation:

Re-baseline EC2 templates against latest watchmaker templates

Back up ${SONAR_ROOT}, but especially the ${SONAR_ROOT}/extensions and ${SONAR_ROOT}/conf subirectories

Sonarqube plugins are stored in ${INSTALLROOT}/extensions/plugins: probably want to populate this directory via a simple sync from a staged pull-down location. Notionally, plugin updates could also be accomplished by updating this location's contents and then triggering a Sonarqube redeployment.

Since the initiation of this project, the source watchmaker templates for EC2 have been continually upgraded. The Sonarqube project's templates have not generally been re-baselined to capture the newer functionality found in the watchmaker templates.

Each EC2-deploying template should be re-baselined against the watchmaker 1.5.6 baseline template

Self-explanatory