Secure the public Jenkins server

[smlambert]

This issue is to encapsulate the requirements and work needed to replace our existing Jenkins server with a new one:

Existing Jenkins server is suffering from some instability. Recent difficult updates have shown that we need a better disaster recovery plan (this is also related to: #1295) It is also a ubuntu-16.04 machine. For all of these reasons, and likely reasons not yet listed, we should begin the work to replace the existing Jenkins server with a new one.

Requirements:

• Similar requirements to existing server
• Ansible playbook for creation/spinup
• Staging server to try out upgrades and major features or changes
• Clearly documented process for deploying updates
• Back up / disaster recovery process (in place and tested) / training for multiple people in multiple timezones to be able to assist
• Policy & documentation for how many jobs to keep in history, how many days to keep jobs
• How many of them are required: 1

Please explain what this machine is needed for:

• nightly/weekly/release builds and testing
• Grinders for debugging and triage
• building tools and dependencies
• building Docker images
• building installers

Considerations:

• Given that this server is for our production builds, should we limit freeform builds, personal duplicates of existing pipelines, etc.?
• Should we have a separate sandbox for 'experimental' jobs and work?
• We should have a script that checks for jobs that have not been run/used in X months to remove old/stale jobs (may not need if all jobs have policy for how many days to keep jobs)
• Consider pushing artifacts to artifactory or other storage so that they are available for longer than

[karianna]

I'd personally like to see this bootstrapped via ansible and that we also build a staging server to test out any major pipeline changes, jenkins upgrades, and so forth.

[smlambert]

re: #2108 (comment) - added to the requirements section

[sxa]

@karianna Since you've been pushing for this and suggesting requirements, are you able to take ownership of this item and progress it?

[karianna]

@karianna Since you've been pushing for this and suggesting requirements, are you able to take ownership of this item and progress it?

I can own it, but I'll be pulling in folks who actually know how to ansible :-). Typical Engineering manager eh ;-)

[karianna]

https://github.com/jenkins-x/jx is something to explore

[sxa]

I also think that for security we should disallow jobs from running on the master node unless there's a clear and explicit need for it.

[karianna]

In addition to Shelley's requirements, I'm also going to add VPN security - this Jenkins should no longer be accessible on the public internet.

[sxa]

In addition to Shelley's requirements, I'm also going to add VPN security - this Jenkins should no longer be accessible on the public internet.

To be absoltely clear here - that statement is about non-HTTPS ports right?
(I agree and was thinking the same, however we also need incoming ports enabled for several of our machines to connect)