Using Nginx As a Reverse Proxy With SSL

Published 2022-07-08. Last modified 2023-03-02.
Time to read: 5 minutes.

This page is part of the posts collection, categorized under Internet, SSL, Security, Ubuntu, nginx.

Recently I migrated ScalaCourses.com from AWS EC2/S3/CloudFront to a server in my apartment, which has fiber optic internet service. The server’s motherboard is an ASUS Sabertooth x79 with an Intel i7 4820, 32 GB DDR3 RAM, and a 4TB SATA SSD. This motherboard is unable to boot from NVMe drives, so SATA is necessary. At the time of this writing, the server runs Ubuntu 22.04.

The backup server is currently being set up. I am repurposing an old Hackintosh as a fallback to the Ubuntu server. That motherboard is another ASUS Sabertooth x79, with 32 GB DDR3 RAM and two 2 TB SATA drives.

Why Am I Doing This?

Once again, I control my hardware, my software, my network, and all ancillary services. After several years of using PaaS vendor servers, I am now reverting to running ScalaCourses.com on my own hardware and software, using my own network.

No longer will I subject myself to the unlimited financial liability that current PaaS vendors expose their customers to.

Definitions

A proxy is a person or process serving as an authorized agent or substitute for another. In computer science, a more specific term is forward proxy.

A proxy server is a server process that acts as an intermediary between a client requesting a resource, and the process that provides the resource.

A reverse proxy is a process that sits in front of other processes, and forwards client requests to them. The term forwarding process is similar to reverse proxy.

The definitions for proxy, forward proxy and reverse proxy all sound identical. The key difference between a reverse proxy and a forward proxy is that a forward proxy enables computers isolated on a private network to connect to the public internet, while a reverse proxy enables computers on the internet to access a private subnet.

Dealing With Details

The old Pound v2.8-2 reverse proxy that was the front end for the old Play Framework app that runs ScalaCourses.com is no longer viable, and the new version 3 of Pound is incomplete. Depending on configuration, reverse proxies can provide extra security from external attacks on a website, decrypt https requests to http, and act as stream editors for the content.

This site still uses AWS CloudFront, for the moment. I am researching alternatives, with the goal of closing my AWS account... unless they introduce a way to cap financial liability before I fully migrate off AWS.

Apache httpd vs. Nginx

Two popular reverse proxies are Apache mod_proxy_http2 and nginx. Anything Pound can do, both nginx and Apache httpd/2 can do. While they both work well, Apache httpd has an older code base, in fact, many of my websites ran on Apache httpd at the turn of the century.

Nginx is relatively newer than Apache httpd. It is performant, well supported, well documented, and widely available. I decided to use nginx as the reverse proxy because I had found that nginx worked well on previous projects.

Installing nginx and the Letsencrypt software is easy on Debian distros such as Ubuntu:

Shell

$ yes | sudo apt install nginx certbot python3-certbot-nginx

Verifying the Nginx Build

When acting as a proxy server, nginx requires the http_sub_module to translate HTTP content. This allows the website content to be stream edited, so various links and paths that are not translated properly can be fixed up.

The nginx package provided by Ubuntu includes the ‑‑with‑http_sub_module option. You can verify that your instance of nginx was built with the ‑‑with‑http_sub_module option as follows:

Shell

$ nginx -V
nginx version: nginx/1.18.0 (Ubuntu)
built with OpenSSL 3.0.2 15 Mar 2022
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -ffile-prefix-map=/build/nginx-9P0wNJ/nginx-1.18.0=.
-flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat
-Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions
-flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx
--conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log
--error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid
--modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy
--http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-compat
--with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module
--with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module
--with-threads --add-dynamic-module=/build/nginx-9P0wNJ/nginx-1.18.0/debian/modules/http-geoip2
--with-http_addition_module --with-http_gunzip_module --with-http_gzip_static_module
--with-http_sub_module

Nginx SSL Configuration

Two files are needed for Letsencrypt to be able to create SSL certificates for nginx:

The contents of /etc/letsencrypt/options-ssl-nginx.conf need to be stored into /etc/letsencrypt/options-ssl-nginx.conf. I later discovered this file was also available locally at /usr/lib/python3/dist-packages/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf.
The contents of /etc/letsencrypt/ssl-dhparams.pem need to be stored into /etc/letsencrypt/ssl-dhparams.pem. I later discovered that this file was also available locally at /usr/lib/python3/dist-packages/certbot/ssl-dhparams.pem.

Making a Wildcard SSL Certificate

I knew an SSL wildcard certificate would be needed, so I made one using Letsencrypt. Please read Creating and Renewing Letsencrypt Wildcard SSL Certificates for details.

Update – 8 months after writing this article, I wrote about a better way to generate wildcard SSL certificates: Wildcard SSL Certificates for Let's Encrypt with Lego.

Defining the Nginx Website Reverse Proxy

Please see my article entitled Cross-Origin Resource Sharing (CORS) for a discussion of how to configure servers whose content needs to be proxied.

I saved the following configuration in a new file called /etc/nginx/sites-available/scalacourses.com. Note that a single server block answers on ports 80 and 443, using IPv4 and IPv6, for SSL and non-SSL requests, for the apex domain (scalacourses.com) and the www.scalacourses.com subdomain. No redirects are used.

Shell

server {
  listen 80;
  listen [::]:80;
  listen 443 ssl;
  listen [::]:443 ssl;

  server_name scalacourses.com www.scalacourses.com;

  ssl_certificate         /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/fullchain.pem;
  ssl_certificate_key     /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/privkey.pem;
  ssl_trusted_certificate /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/chain.pem;
  include /etc/letsencrypt/options-ssl-nginx.conf;
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

  root /var/www/html;
  index index.html;  # This gets served if the proxied website is down

  location / {
    proxy_pass http://localhost:9000/;
    proxy_set_header Accept-Encoding "";  # sub_filter requires this

    sub_filter 'https://localhost:9000/authenticate/' 'https://www.scalacourses.com/authenticate/';
    sub_filter_once off;
  }
}

I disabled the default site:

Shell

$ sudo rm /etc/nginx/sites-enabled/default

I enabled the new scalacourses.com site:

Shell

$ sudo ln /etc/nginx/sites-{available,enabled}/scalacourses.com

The nginx configuration was tested for syntax:

Shell

$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

The nginx configuration was reloaded:

Shell

$ sudo systemctl reload nginx

Flush DNS Cache

Ensure that DNS requests receive up-to-date values by flushing the DNS cache. The following works on Ubuntu, but not when running on WSL/WSL2.

Shell

$ sudo resolvectl flush-caches

The following works on Windows 10:

Command and PowerShell consoles

C:\Users\Mike Slinn> ipconfig /flushdns

Verifying nginx Works

I verified that nginx was listening on ports 80 and 443. I highlighted the nginx process number – you might need to scroll the output below to the right to see it.

Shell

$ sudo netstat -tulpn | grep ':\(443\|80\)'
tcp    0    0 0.0.0.0:80       0.0.0.0:*       LISTEN   -
tcp6   0    0 :::80            :::*            LISTEN   -
tcp    0    0 0.0.0.0:443      0.0.0.0:*       LISTEN   87487/nginx: master

The executable for process 87487 can be found by:

Shell

$ ls -l /proc/87487/exe
lrwxrwxrwx 1 mslinn mslinn 0 Jul  7 09:13 /proc/5166/exe -> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java*

If more detail is desired, get it from the process list. Enclosing a character of the process id within square brackets is an old trick for only showing the desired process.

Shell

$ ps aux | grep [8]7487
mslinn      87487  2.4  1.3 6596476 439456 ?      Sl   09:13   0:20 java -Xms1024m -Xmx1024m -Dhttp.port=9000 /path/to/jar

If there is any problem getting things to work, it is often helpful to monitor the logs as you click on the web pages:

Shell

$ sudo tail -f /var/log/nginx/*.log

Finishing Up

Make nginx start each time the system starts as follows.

Shell

$ sudo systemctl enable nginx
Synchronizing state of nginx.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable nginx 

$ sudo update-rc.d nginx defaults

Performance Test

KeyCDN offfers free website performance statistics. The result column labeled TTFB means “time to first byte”, which is the length of time required for the website content to begin being received by a user's web browser.

I am in Montreal, Canada. Response time for TTFB reported by KeyCDN varies, from a minimum of 68ms in New York, to a maximum of 815ms in Bangalore. This range of response times is typical for websites that have a centralized process. The speed of light is finite, after all.