HSTS, GitLab, and LetsEncrypt

I set up a test install of GitLab on one of our group's servers, and set up and successfully deployed LetsEncrypt with it. This is awesome! TLS certificates in seconds!

For various reasons, we only have one domain and IP for this overall server, that has multiple VMs on it. I then wanted to set up a monitoring tool, such as NetData.To run this on the same domain I had to set it up on a different port. (This is running in a separate VM to GitLab, but on the same HyperVisor, and hence same domain).

This was all well and good, until you try to visit both in the same browser. GitLab (when running over TLS) enables HSTS by default. This is the behaviour you want for GitLab, i.e. as soon as a client has visited https://mygitlabdomain.com, there should be no way of downgrading a connection to http://mygitlabdomain.com in future visits.

However, for our NetData install (running at e.g. http://mygitlabdomain.com:9999), this was a disaster: as soon as I tried to visit NetData again (from an install that I knew was working fine, and had previously visited), I was a) being redirected to https://mygitlabdomain.com:9999, and b) now only getting ERR_SSL_PROTOCOL_ERROR errors, with no further information. This only occurred after I enabled TLS on GitLab, but didn't notice the link between the two at the time.

Someone sensible suggested that HSTS might be the cause of this behaviour. You can have multiple TLS configurations on the same domain, different ports, but if HSTS is configured for the default ports (i.e. 80 automatically redirecting to 443) then it ties that configuration and requirement for TLS to all ports for that domain.

The immediate, temporary, client-side fix (for testing purposes) is therefore to remove the HSTS config for this domain in Chrome, by going to chrome://net-internals/#hsts and adding the domain in question to the 'delete domain' form. This only temporarily enables the non-port 443 server for use without TLS again.

The longer term fix in (for us at least) seems to be to run the HSTS demanding install (GitLab) on ports other than 80/443, so that the browser doesn't expect TLS for all servers at that domain (on a range of different ports). The better fix of course is to add TLS to all servers on all ports at the same domain!

tl;dr: Be careful with TLS and HSTS when running multiple web-servers on the same domain.

Login attempts to my servers

I've got a couple of years worth of logs from this server and others, showing the good the bad and the ugly. Most interesting is it keeps every IP address that has ever tried to connect to each server in the last 24 hours, and emails me a list of these, among other things.

I downloaded my email inbox from GMail and wrote a quick bit of Python to scrape through a .mbox file for IP addresses that had attempted to login. In decreasing order of number of unique IP address which attempted to connect to one of my servers over SSH, the countries are as follows: US (12,048), China (4,614), GB (2,761), India (1,816), Bahrain (1,507), Brazil (1,123), Canada (1,093).

Note that this is just the country of origin of the IP, it doesn't mean the server was actually controlled by someone in that country -- botnets, people!

Here are those IP addresses plotted on a graph, where the x-axis is the first octet of the address, and the y-axis is the second octet of the address, e.g. x.y._._

(Click link for a higher resolution & DPI version).

IP Adresses small (/16)

The vertical banding is what I'd expect, but I'm fascinated by the horizontal banding at ~52 on the y-axis.

I'll publish the code at some point, but at the moment the associated data files have all the full IP addresses, so I don't want to make them public yet.

"Up-Goer Five your Thesis"

My research, described using only the ten-hundred most common English words:

Computers were made to work, and not be safe against people who want to break into them. Bad people started to find they could make money and have fun by breaking into computers, so good people had to start thinking about how to stop bad people breaking into them.

By this point, a lot of very important computers were out there, with nothing to stop people breaking into them. So we are trying to fix that. Turns out, it's easier to make stuff that stops people breaking into them when you think about it from the start, rather than trying to do it after the computer's been made.

At the moment, if a bad person breaks into one small bit of your computer, a lot of the time, they can do anything they want in the rest of your computer. This is bad. What I'm trying to do is make parts of computers (and the way these parts talk to each other) that don't completely break when just one bit gets broken into.

Made with The Up-Goer Five Text Editor.