If we're behing a proxy such as Caddy, the remote address will be that of the proxy, so we have to use the X-Forwarded-For header that the proxy injects instead.