If we're behing a proxy such as Caddy, the remote address will be that
of the proxy, so we have to use the X-Forwarded-For header that the
proxy injects instead.