plus3it / cfn-gitlab Goto Github PK

Describe the bug

Deployment-automation currently does not attempt to preserve the gitlab-secrets.json across instantiations. When users store secrets for CI-automation or other integrations, this will cause CI jobs to fail and the projects' Integrations tab/page to become unloadable (HTTP 500 errors).

Severity

• Completely Broken (No work-around evident)
• Severely Broken (Work-around possible but difficult)
• Moderately Broken (Trivial work-around)
• Nuisance (Functions but untrapped errors can slip through)

To Reproduce

Steps to reproduce the behavior:

1. Launch new GitLab domain
2. Allow users to create CI-jobs or other integrations that use stored secrets
3. Migrate service to new node (version, OS or other related task that redeploys the service to new instance)
4. Attempt to run CI-jobs or other integrations that use stored secrets — failures should be present

Expected behavior

Node-replacing upgrades to not adversely impact service-functionality.

Additional context

Problem matches what's described in the vendor documentation.

Fix Suggestions

Add a launch-time task to check if a gitlab-secrets.json file has been stored to a safe location:

• If yes, download to new instance's /etc/gitlab directory
• If no, treat as new installation and save new gitlab-secrets.json file to a safe location

Is your feature request related to a problem? Please describe.
Current EC2 template design assumes AD/LDAP-integration. Removing that assumption from template will both decrease the number of parameters to pass and allow for deployment to non-AD/LDAP environments.

Describe the solution you'd like
Remove all authentication references from EC2 templates — instead placing them in the site-local gitlab.rb.tmplt file

Notes
Currently only present in the yet-to-be updated make_gitlab_parent-GlusterFS.tmplt.json file.

Since initial authoring, AWS has updated available PGSQL versions. Per today's (2018-12-10) notifications, AWS is recommending updating running versions to at least 9.6.9.

AWS's currently-supported versions are (application support may vary: test if moving to a higher major):

10.4
10.3
10.1
9.6.10
9.6.9
9.6.8
9.6.6
9.6.5
9.6.3
9.6.2
9.6.1
9.5.14
9.5.13
9.5.12
9.5.10
9.5.9
9.5.7
9.5.6
9.5.4
9.5.2

Problem Description:

It may be desirable to offer the ability to customize database tuning-options. Need the DB to use a custom — rather than the currently used RDS-default — parameter group.

Expected Behavior:

Ability to tune DB behavior via DB parameter-group settings

Actual Behavior:

Current use of RDS-default DB parameter-group precludes tuning customizations

(Detailed) Steps to reproduce:

Deploy RDS DB from existing templates

(Optional) Fix recommendation:

Add a AWS::RDS::DBParameterGroup resource-type into the current RDS templating.

Redundant Code-Chunks Make Solution Less Portable
As written steps for Ruby-setup break portability (and have proven superfluous at any rate).

Proposed Solution
Nuke out code-chunks related to Ruby-setup.

Additional
Get rid of redundant backup cron-job setup

Problem Description:

GitLab includes native CI capabilities via runners. Overall toolchain capabilities would be enhanced by making GitLab's runner-functionality available

Expected Behavior:

Service users with a .gitlab-ci.yml in their project-root can leverage GitLab's native CI extensions for their hosted projects.

Actual Behavior:

This project does not currently facilitate the deployment or registration of runners; thus, users of service-instantiations created from these tools don't have access to GitLab's native CI extensions for projects hosted GitLab domains created from this project's automation.

Fix recommendation:

Implement the runner installation and registration steps as part of additional EC2 deployment-automation.

See also:

• ustun gist

Problem Description:

Currently installed backup logic in /etc/cron.d/GitLab_backups is sub-optimal from a performance perspecctive

Expected Behavior:

Maximize S3 performance to be more equivalent to those outlined in AWS documentation

Actual Behavior:

Backup/restore slower than could be. Mostly not a problem, now, but will become a problem as backed-up dataset grows in size (particularly number of elements backed up)

Fix recommendation:

Change current backup method from an s3 sync of the GitLab content to a tar cf - <GITLAB_CONTENT> | s3 cp - s3://<BUCKET>/<KEY>/<TAR_FILE> method

Problem Description:

Upon reprovisioning, all service-users that use SSH for push/pull operations receive spurious main-in-the-middle-attack errors.

Expected Behavior:

After a rebuild event, SSH-based git actions do not experience main-in-the-middle-attack warnings.

Actual Behavior:

After a rebuild event, SSH-based git actions experience (objectively spurious) main-in-the-middle-attack warnings.

Fix recommendation:

Ensure that, upon provisioning:

• A replacement-host checks for availability of prior host's SSH hostkeys, installing them if present
• If no prior hostkeys present, generate new ones and upload to a location that further replacement-hosts can pull from

Problem Description:

CloudWatchLog does not currently monitor the GitLab application's log files

Expected Behavior:

CloudWatchLog should monitor the GitLab application's log files:

• /var/log/gitlab/unicorn/unicorn_stdout.log
• /var/log/gitlab/unicorn/unicorn_stderr.log
• /var/log/gitlab/nginx/error.log
• /var/log/gitlab/nginx/access.log
• /var/log/gitlab/nginx/gitlab_error.log
• /var/log/gitlab/nginx/gitlab_access.log
• /var/log/gitlab/gitlab-rails/grpc.log
• /var/log/gitlab/gitlab-rails/production.log
• /var/log/gitlab/gitlab-rails/application.log
• /var/log/gitlab/gitlab-rails/sidekiq.log

Actual Behavior:

CloudWatchLog does not currently monitor the GitLab application's log files

(Optional) Fix recommendation:

Update template's CWA log-config section to add the above logfiles.

Problem Feature-Request Addresses

Absent the recommended envs, signalling may not work in all regions

Proposed Solution

Ensure maximum portability by adding exports of AWS_CA_BUNDLE and REQUEST_CA_BUNDLE to post-reboot signalling-script.

Describe the bug
Current systemd state-test will misevaluate due to equality check of string-val to number.

Severity

• Completely Broken (No work-around evident)
• Severely Broken (Breaks region-portability)
• Moderately Broken (Trivial work-around)
• Nuisance (Functions but untrapped errors can slip through)

Expected behavior
The systemd state-test should be structured to ensure equality-test compares numbers to numbers.

Fix Suggestions

Update code-segment:

[[ $( systemctl is-active multi-user.target )$? -ne 0 ]]

To:

[[ $( systemctl --quiet is-active multi-user.target )$? -ne 0 ]]

Rationale

GitLab can be enabled to track and report on service-metrics (see: GitLab Performance Monitoring). To support this capability, GitLab needs to be able to write to and extract date from the InfluxDB time-series DBMS.

Solution Description

Add CFn stack(s) — and Jenkins job-descriptions to drive them — for deploying InfluxDB nodes — or (if it isn't overkill/too cost-intensive) InfluxDB clusters (see discussion at High availability for InfluxDB)

Problem Description:

The download links for projects is showing up as HTTP instead of HTTPS.

If you copy the link and try to clone a repo it'll fail until you change http:// to https://.

Make sure they aren't too aggressive — particularly the EC2s'/ASGs' and RDSes'

Problem Description:

Per GitLab Documentation Update, with the release of 11.2, a bug was introduced that causes gitlab-ctl reconfigure to block/hang when executed from systemd. Until/unless this issue is fixed, it will be necessary work around this limitation.

Expected Behavior:

Automated-deployments of GitLab versions 11.2+ function as they had with all prior versions.

Actual Behavior:

Automated redeployments fail due to a hang/deadlock that only happens when deployment is initiated from systemd.

(Detailed) Steps to reproduce:

• Upgrade a pre-11.2 system to 11.2 (or higher), then attempt to do an automated-redeploy.
• Attempt to stand up a new 11.2 (or higher)

Fix recommendation:

None. Fix is contingent on vendor-initiated code-fix.

Workaround

Move gitlab-ctl reconfigure step out of initial build-process into something that runs after CFn has marked the build Green and after systemd has fully booted the system.

Previous automation-flow created the ELB after creation of the EC2. ASG will require inverting that relationship. Similarly, moving standalone deployments to static "infrastructure" with changeable EC2 will similarly necessitate. Therefore, standalone EC2 template needs to be updated to attach itself to designated/available ELB.

Describe the bug

When templates deploy replacement instances, the host SSH keys are not persisted across instantiations. This causes clients using SSH for push/pull to pop a MITM attack-alert

Severity

• Completely Broken (No work-around evident)
• Severely Broken (Work-around possible but difficult)
• Moderately Broken (Trivial work-around)
• Nuisance (Functions but untrapped errors can slip through)

To Reproduce

Use either the Standalone templates to deploy a new stack-set for migration or use the Autoscale templates to do automated rebuilds. Whenever the new instance(s) are made "live", SSH clients pop host-key errors

Expected behavior

Reprovision events are transparent to service-consumers

Fix Suggestions

Add logic to check config-bucket for service host-key files: if present download them in place of the ones generated at instance-launch; if absent, copy-up the host-key files to the config-bucket.

Problem Description:

Updated automation leaves FIPS enabled. When FIPS mode is enabled, anything relying on the hosting-OS's GPG utilities will result in hangs. For example, attempting to view or upload GPG keys to a user profile will cause the GitLab workhorses to timeout and crash.

Expected Behavior:

Automation should ensure FIPS-mode is disabled

Actual Behavior:

Automation leaves FIPS-mode in whatever its starting state is (on spel AMIs, "enabled")

(Optional) Fix recommendation:

Add a secondary watchmaker call to ensure that FIPS-mode is disabled:

salt-call --local ash.fips_disable

Problem Description:

Templates last based prior to usage of CloudWatch Agent. Update to include optional CloudWatch logic

Expected Behavior:

Template installs CloudWatch agent in regions that support it.

Actual Behavior:

No hooks for CloudWatch Agent present

(Optional) Fix recommendation:

Re-baseline EC2 templates against latest watchmaker templates

Problem Description:

AWS has released new instance types that might better align to some deployment-scopes

Expected Behavior:

Support t3 and m5 instance-types where possible

Actual Behavior:

Does not currently support t3 and m5 instance-types at all

(Optional) Fix recommendation:

Update template logic to allow for t3 and m5 instance-types