clojars / infrastructure Goto Github PK

There are some people with misconfigured applications/Nexus repo's who request files way too much. It could be good to use fail2ban to stop them from making requests too often. A first approximation for detecting this would be to check for IP's downloading the same file more than 5 times in half an hour. There are a few things to consider here:

• The larger the time window, the more memory this will consume
• If we're going to move to a CDN, how will we handle misconfigured clients like this? Maybe we can analyse the logs and block IPs out of band?
• We don't want to block normal users, it should probably be run in report only mode for a while to make sure it's calibrated correctly.
• It would be good to build up profiles of what normal maven usage and abusive behaviour look like, as kicking off a build with a clean m2 will generate a lot of (valid) requests.

http://codelog.climens.net/2011/02/13/using-fail2ban-with-nginx-in-debian/
http://blog.teabough.com/fail2ban-api-mailjet/

We would like to change the license to MIT. This would make it possible for people to derive new works from this config for internal purposes without having to redistribute them. Under normal usage, sharing the changes would not be required, but it would if you distributed it (e.g. as part of a package that runs on client sites)

To allow for the relicensing, reply to this issue with "I Agree". If you don't agree then feel free to discuss this more here.

• @danielcompton
• @tobias
• @kahunamoore
• @spencercrissman
• @hypirion

Thanks!

Related to #3, it could be good to have zero downtime deploys where the second Java process gets brought up online before shutting down the first one. Possibly could use Nginx to switch between them. However there were issues in the past with running multiple processes against a single sqlite db, so this might not be a good idea in the short term.

[DEPRECATION WARNING]: The use of 'include' for tasks has been deprecated. Use
'import_tasks' for static inclusions or 'include_tasks' for dynamic inclusions. This feature
 will be removed in a future release. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: include is kept for backwards compatibility but usage is discouraged.
 The module documentation details page may explain more about this rationale.. This feature
will be removed in a future release. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.

We use include in a few places. [import_tasks](https://docs.ansible.com/ansible/devel/modules/import_tasks_module.html) seems like the correct thing to use for us but

• It is still marked as a preview without a backwards compatible interface (?!)
• It's not clear if/how we can use sudo when running those tasks

Any expert Ansible help on the 'correct' thing to do here would be welcome.

All Cron jobs are installed, but I haven't checked to make sure they are always running successfully. This is probably something I need to do as it will require private config I think.

I'm pretty sure I've setup SSL correctly (by copying from the previous server), but it would be good to get another set of eyes over it to make sure it's correct and is up to date with best practices, particularly over the IPv6 stuff.

For security and ease of debugging, it would be good to send logs to somewhere else so we can view them and so they are secure against tampering.

• Adding users still requires manually adding authorised keys
• Bootstrapping a server requires logging in, adding an audible user, and adding them as a sudoer
• nginx seems to need a restart after the initial bootstrap
• rsync has a pid problem after initial bootstrap

There are gaps in the dates of the downloads-{date}.edn files. Need to investigate what's happening.

It would be good to add a health check to make sure the app is running well when deploying a new version. Automated roll back would be nice too.

We only put x-frame-options on requests served by the app, so anything that Nginx serves directly (like redirects and resources) doesn't have this header attached. I'm not sure what the security implications of this are, but worth considering further.

Not quite sure what server config is needed here, @tobias what's the process for deleting JARs?

There are a bunch of good links here, especially debops. I'm not sure whether we use ansible galaxy or just copy the parts of the scripts we like locally.

• http://debops.org
• http://practicalops.com/tag/ansible.html
• http://lattejed.com/first-five-and-a-half-minutes-on-a-server-with-ansible
• http://ryaneschinger.com/blog/securing-a-server-with-ansible/
• https://github.com/vitalk/ansible-secure-ssh
• http://blog.mailgun.com/security-guide-basic-infrastructure-security/

Tasks:

• Fail2ban
• Harden SSH
• Iptables setup

This is a companion issue for:

clojars/clojars-web#559

To fix the server-config aspects.

clojars.org doesn't seem to be gzipping assets. It would be good to figure out why, as it seems like it should be based on the config.

TASK [system : ensure admin user accounts exist] ********************************************
 [WARNING]: when statements should not include jinja2 templating delimiters such as {{ }} or
{% %}. Found: {{ item.active }}

Figure out what the correct way to do this is instead.

It is set to UTC out of the box, but it would be good to make sure it stays like that with an ansible task. This looks promising but needs to have backup turned off.

To build Clojars the app, we need Clojars the site. This is a nasty circular dependency we need to break.

If we're trying to bootstrap a new clojars server, it stands to reason that there may not be a running one in place at the time. There are a few options here for building the app, and two main scenarios to think about: rebuilding the server from scratch when there is no Clojars server, and everyday deployments.

Use a mirror for building Clojars on the clojars server

We can set a mirror in the Leiningen config, and use this for both building from scratch, and everyday builds.

Pros:

• Minimal changes to current build system required
• JAR will still be built on server, simple to understand
• Keeps all config in Ansible

Cons:

• Relies on mirror support to work well in Leiningen
• We would need to modify the clojars-web project.clj to use these mirrors; currently mirrors in profiles.clj won't work for :plugin-repositories and we need the supersport plugin.
• Relies on the mirror server/block store to be available at the time we are rebuilding
• Needs to download all of the JARs at build time for bootstrapping, this can take a while. On an established server this isn't likely to be as big an issue.
• Takes a while to build and test the JAR
• Puts some amount of load on the server

Build Clojars locally ahead of time and copy it to the server

I think @tobias is in favour of this option.

Pros:

• All deps likely to be downloaded so it will be fast in all cases
• Likely to be building with a machine with more resources than a VPS
• Pretty simple

Cons:

• Still needs a mirror available in case the main server is down
• Kind of gets away from the ansible ideal (I think?) as it relies on local state to have a working build toolchain and it is managed out of band from the rest of the Ansible config. May not be as easily reproducible between different people's machines, e.g. profiles.clj differences.
• Build scripts would probably need to reach out from their own folders to the other project

Build Clojars on the server ahead of time and write an artifact to S3 or similar

How this would work:

1. When a new version is ready, an admin would run an ansible play to build the new version on the clojars server (or optionally on a new VM in EC2, but that's probably not necessary).
2. Once the uberjar is built it is copied into S3 (and another redundant object store?) for deployment.
3. When it comes to deploy that version (either straight away, or far off in the future) it doesn't rely on any of our infrastructure being up and running.

This is my personal preference as it seems to be the most fault tolerant, however after listing the cons, I'm not so sure it's the right solution.

Pros:

• Deploys from scratch are much faster as they just need to download a JAR
• Possibly most fault tolerant design
• Doesn't rely on a working Clojure environment on the deployers machine
• Keeps everything in Ansible

Cons:

• It adds a dependency on the object store
• It adds additional complexity around orchestration
• It adds additional complexity keeping more application secrets
• Requires another account

Thoughts welcome!

We had a report that Clojars responds to unknown host headers. We should drop requests that don't match headers that we expect: https://stackoverflow.com/a/28591245/826486

It is currently mounted by a manual modification to /etc/fstab.

• DNS
• Mail SPF

???

We need to make sure we call Yeller with the right config when testing and going in production. It might also be good to disable Yeller entirely when non admins are running the app/ansible scripts.

This needs to be templates in Ansible. Probably only something an admin can do.

I'm not quite sure what do about DB migrations, possibly they should be a resource in the uberjar? That way we always know we've got the right schema for each version.

The version of goaccess bundled with ubuntu is ancient. It would be really handy to have the version described in "Official GoAccess' Debian/Ubuntu Repository" from here installed.

Each admin can provide authorised keys to be put against their user account. These need to be installed. They can take the place of the commented out github public keys script in /roles/system/tasks/main.yml.

I was able to setup rsync to run, but I haven't got this script to run correctly yet.

$ rsync -av --delete  localhost::clojars my-wonderful-copy-of-clojars
rsync: failed to connect to localhost: Connection refused (61)
rsync error: error in socket IO (code 10) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/clientserver.c(105) [receiver=2.6.9]

$ rsync -av --delete  -e 'usr/bin/ssh -p2222' localhost::clojars my-wonderful-copy-of-clojars
rsync: Failed to exec usr/bin/ssh: No such file or directory (2)
rsync error: error in IPC code (code 14) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/pipe.c(86) [receiver=2.6.9]
rsync: connection unexpectedly closed (0 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/io.c(453) [receiver=2.6.9]

I'm not sure if this is an artifact of running through Vagrant on port 2222 or something else is misconfigured.

https://github.com/amarao/ansible-reboot-if-needed-for-upgrade
https://github.com/jnv/ansible-role-unattended-upgrades

https://www.ssllabs.com/ssltest/analyze.html?d=clojars.org&s=173.230.139.200

Essential

• This server supports weak Diffie-Hellman (DH) key exchange parameters. Grade capped to B. (I think this is unavoidable to support Java 6).
• Certificate has a weak signature and expires after 2015. Upgrade to SHA2 to avoid browser warnings.
• This server accepts RC4 cipher, but only with older protocol versions. Grade capped to B. (Need to revise cipher suite for todays best practices). https://cipherli.st

Nice to have

• Forward Secrecy: Weak key exchange
• Enable HSTS preload https://hstspreload.appspot.com ?
• Enable HPKP?
• Check HTTP forwarding on clojars.org. c.f. http://redirectdetective.com and https://m.reddit.com/r/sysadmin/comments/3pxl30/proper_httphttps_forwarding_in_nginx/

With all of these things, we need to keep in mind the ever present spectre of Java 6 and corporate SSL middleware boxes.

It expires on 12 August, needs to be renewed before then, and should use SHA-2.

See clojars/clojars-web#417

Tweak bootstrap script used to generate/touch private files. Should it and the example file sit at the root instead of inside private/?

It would be good to have a message of the day with common scripts so that if you do need to logon to a server you can see the common ones straight away.

https://support.google.com/mail/answer/81126?p=IPv6AuthError&visit_id=1-636100323356104273-1048709963&rd=1#authentication

They were commented out because of a failing test in clojars.test.unit.tools.repair-metadata when run on the server. From memory the file ordering wasn't consistent.

This needs to be stored in private.yml, with a default set in private.yml.example.

I've setup the email as best I can from the previous server, but it doesn't seem like it's working yet. I'm not sure if that's just because I'm running on localhost though and it would be fine on a real server? We also need to check the SSL is setup correctly for Postfix.

• Make postfix send emails
• Make sure SSL is setup correctly

I'm not quite sure how the Nginx logs get rotated.

http://www.codelitt.com/blog/my-first-10-minutes-on-a-server-primer-for-securing-ubuntu/
https://www.inversoft.com/guides/2016-guide-to-user-data-security