HSTS, GitLab, and LetsEncrypt
For various reasons, we only have one domain and IP for this overall server, that has multiple VMs on it. I then wanted to set up a monitoring tool, such as NetData.To run this on the same domain I had to set it up on a different port. (This is running in a separate VM to GitLab, but on the same HyperVisor, and hence same domain).
This was all well and good, until you try to visit both in the same browser. GitLab (when running over TLS) enables HSTS by default. This is the behaviour you want for GitLab, i.e. as soon as a client has visited
httpS://mygitlabdomain.com, there should be no way of downgrading a connection to
http://mygitlabdomain.com in future visits.
However, for our NetData install (running at e.g.
http://mygitlabdomain.com:9999), this was a disaster: as soon as I tried to visit NetData again (from an install that I knew was working fine, and had previously visited), I was a) being redirected to
httpS://mygitlabdomain.com:9999, and b) now only getting
ERR_SSL_PROTOCOL_ERROR errors, with no further information. This only occurred after I enabled TLS on GitLab, but didn't notice the link between the two at the time.
Someone sensible suggested that HSTS might be the cause of this behaviour. You can have multiple TLS configurations on the same domain, different ports, but if HSTS is configured for the default ports (i.e. 80 automatically redirecting to 443) then it ties that configuration and requirement for TLS to all ports for that domain.
The immediate, temporary, client-side fix (for testing purposes) is therefore to remove the HSTS config for this domain in Chrome, by going to
chrome://net-internals/#hsts and adding the domain in question to the 'delete domain' form. This only temporarily enables the non-port 443 server for use without TLS again.
The longer term fix in (for us at least) seems to be to run the HSTS demanding install (GitLab) on ports other than 80/443, so that the browser doesn't expect TLS for all servers at that domain (on a range of different ports). The better fix of course is to add TLS to all servers on all ports at the same domain!