nginx on macos behind cloudflare

Somehow, Traefik did not worked out and after few hour crashes in interesting way.

So, decided to give a change to nginx.

Usual stuff first:

brew install nginx

installer logs:

docroot: /opt/homebrew/var/www

default port set in /opt/homebrew/etc/nginx/nginx.conf to 8080 so that nginx can run without sudo.

nginx will load all files in /opt/homebrew/etc/nginx/servers/.

start as service:

  brew services start nginx

start manually:

  /opt/homebrew/opt/nginx/bin/nginx -g daemon\ off\;

this one may be interesting, indeed, lower ports like 80 and 443 require privileges to bind them

From other hand, on home router we are doing port forwarding like: <external_ip>:443 -> <internal_ip>:443

Why not forward to :8080 instead - this one will be first change

Second, because we are behind cloudflare, should we really bother with letsencrypt, why not just use cloudflare cert

Home router

From home router side I have configured nat forwarding

<external_ip>:443 -> <internal_ip>:8080

Cloudflare

From Cloudflare

In SSL/TLS overview page set SSL/TLS encryption mode to full (strict), e.g. traffic will be encrypted everywhere

client --https--> cloudflare --https-->external_ip

In SSL/TLS Origin Server page we can create an certificate signed by Cloudflare, it is trusted by Cloudflare, will make everything secure without the need to rely on Letsencrypt

It will create for us cloudflare.pem and cloudflare.key certificates, also point us to documentation of how to configure origin server.

scp cloudflare.* mini:/opt/homebrew/etc/nginx/

Before proceeding lets give it a try it should work already

/opt/homebrew/etc/nginx/servers/nginx.conf

server {
  listen 8080;
  ssl on;
  ssl_certificate cloudflare.pem;
  ssl_certificate_key cloudflare.key;

  server_name nginx.mac-blog.org.ua;                                                                                                                                #    ssl_certificate      cert.pem;

  location / {
    root html;
    index index.html index.htm;
  }
}
nginx -g daemon\ off\;

it did failed with an error:

nginx: [emerg] no "ssl_certificate" is defined for the "listen ... ssl" directive in /opt/homebrew/etc/nginx/nginx.conf:35

so i have just removed server { ... } from it, we won't need it anyway

and finally nginx.mac-blog.org.ua start working

which technically means we almost done and can just add more and more services

and make a note - no need to bother with certs at all

mTLS

one more step to make thins even better

on SSL/TLS Origin Sergver page scroll down to Authenticated Origin Pulls and enable it

then

wget https://developers.cloudflare.com/ssl/static/authenticated_origin_pull_ca.pem
server {
  listen 8080 ssl;
  server_name nginx.mac-blog.org.ua;

  # https
  ssl_certificate /opt/homebrew/etc/nginx/cloudflare.pem;
  ssl_certificate_key /opt/homebrew/etc/nginx/cloudflare.key;

  # mtls
  ssl_verify_client on;
  ssl_client_certificate /opt/homebrew/etc/nginx/authenticated_origin_pull_ca.pem;

  location / {
    root html;
    index index.html index.htm;
  }
}

this change will enforce ssl verify of the client so only cloudflare can access our nginx

profit - no need to deal with ip adresses lists anymore

things left - configure nginx itself and services

Services

Here few small starter configurations

/opt/homebrew/etc/nginx/servers/grafana.conf

  • docs
  • expected that auth is configured
map $http_upgrade $connection_upgrade {
  default upgrade;
  '' close;
}

server {
  server_name grafana.mac-blog.org.ua;

  listen 8080 ssl;
  ssl_certificate /opt/homebrew/etc/nginx/cloudflare.pem;
  ssl_certificate_key /opt/homebrew/etc/nginx/cloudflare.key;
  ssl_verify_client on;
  ssl_client_certificate /opt/homebrew/etc/nginx/authenticated_origin_pull_ca.pem;

  location / {
    proxy_pass http://localhost:3000;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
  }

  location /api/live/ {
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header Host $host;
    proxy_pass http://localhost:3000;
  }
}

/opt/homebrew/etc/nginx/servers/prometheus.conf

prometheus has no auth, previously with traefik we have configured fully featured oidc, this time - do not even want to waste time on this, good old basic auth will be used instead

following command will hash "mypass" as password for "alice"

htpasswd -nb alice mypass

output will be something like:

alice:$apr1$nInGCQmP$4oknZsVad6f5sQADootPu1

so we are going to save it:

htpasswd -nb alice mypass > /opt/homebrew/etc/nginx/prometheus.htpasswd

and our conf file

server {
  server_name prometheus.mac-blog.org.ua;

  listen 8080 ssl;
  ssl_certificate /opt/homebrew/etc/nginx/cloudflare.pem;
  ssl_certificate_key /opt/homebrew/etc/nginx/cloudflare.key;
  ssl_verify_client on;
  ssl_client_certificate /opt/homebrew/etc/nginx/authenticated_origin_pull_ca.pem;

  location / {
    proxy_pass http://localhost:9090;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;

    # basic auth
    auth_basic "auth";
    auth_basic_user_file /opt/homebrew/etc/nginx/prometheus.htpasswd;
  }
}

DRY

Seems like in majority of services we will repeat the same snippet again again and again

so lets put it in dedicated file

/opt/homebrew/etc/nginx/cloudflare.conf

listen 8080 ssl;
# https
ssl_certificate /opt/homebrew/etc/nginx/cloudflare.pem;
ssl_certificate_key /opt/homebrew/etc/nginx/cloudflare.key;
# mtls
ssl_verify_client on;
ssl_client_certificate /opt/homebrew/etc/nginx/authenticated_origin_pull_ca.pem;

and now our service files becomes something like

server {
  server_name prometheus.mac-blog.org.ua;

  include cloudflare.conf;

  location / {
    proxy_pass http://localhost:9090;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;

    # basic auth
    auth_basic "auth";
    auth_basic_user_file /opt/homebrew/etc/nginx/prometheus.htpasswd;
  }
}

And technically the same is true for location section, so

/opt/homebrew/etc/nginx/proxy_headers.conf

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

makes our site config something like

server {
  server_name prometheus.mac-blog.org.ua;
  include cloudflare.conf;
  location / {
    proxy_pass http://localhost:9090;
    include proxy_headers.conf;
  }
}

you already should see where it goes

Monitoring

nginx has no built in prometheus metrics, but there is stub_status

we supposed to add it to server like so:

  location /nginx_status {
    stub_status;
    allow 127.0.0.1;
    deny all;
  }

which give us very very limited data

so technically there is not much we will be monitoring from this standpoint

but we may be interested in playing with log files instead, which are part of overall nginx config

nginx.conf

there is an pretty cool online tool to play with nginx config

https://www.digitalocean.com/community/tools/nginx

just for backup here is what i have ended up with for v1

worker_processes 1;

events {
  multi_accept on;
  worker_connections 1024;
}

http {
  charset utf-8;
  client_max_body_size 10M;
  default_type application/octet-stream;
  gzip on;
  gzip_vary on;
  include mime.types;
  keepalive_timeout 65;
  sendfile on;
  server_tokens off;
  tcp_nodelay on;
  tcp_nopush on;

  # for ip in $(curl -s https://www.cloudflare.com/ips-v4/)
  # do
  #   echo "set_real_ip_from $ip;"
  # done
  set_real_ip_from 173.245.48.0/20;
  set_real_ip_from 103.21.244.0/22;
  set_real_ip_from 103.22.200.0/22;
  set_real_ip_from 103.31.4.0/22;
  set_real_ip_from 141.101.64.0/18;
  set_real_ip_from 108.162.192.0/18;
  set_real_ip_from 190.93.240.0/20;
  set_real_ip_from 188.114.96.0/20;
  set_real_ip_from 197.234.240.0/22;
  set_real_ip_from 198.41.128.0/17;
  set_real_ip_from 162.158.0.0/15;
  set_real_ip_from 104.16.0.0/13;
  set_real_ip_from 104.24.0.0/14;
  set_real_ip_from 172.64.0.0/13;
  set_real_ip_from 131.0.72.0/22;
  real_ip_header X-Forwarded-For;
  real_ip_recursive on;

  include servers/*;
}

but this one is probably neverending story and will be tuned over time

going back to the roots

while plaing with configs, there are some php examples, decided why not, it is cool to have some ad-hoc scripts without wasting resources

brew install php
brew services start php

so we have php-fpm with this

and now we can do something like

/opt/homebrew/etc/nginx/servers/phpnifo.conf

server {
  server_name phpinfo.mac-blog.org.ua;

  include cloudflare.conf;
  include common.conf;

  root /Users/mini/phpinfo;
  index index.php;

  location / {
    try_files $uri $uri/ =404;
  }

  location ~ \.php$ {
    include fastcgi_params;
    fastcgi_pass 127.0.0.1:9000;
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_param DOCUMENT_ROOT $document_root;
  }
}

of course there are things to tune php itself, monitoring etc but this one is out of scope, for now just need some ready to copy paste sample

reloading

whenever changes are made just use

nginx -s reload

fallback

whenever i try to open whatever.mac-blog.org.ua nginx just points me to first configured server instead of showing 404

to fix that we need

/opt/homebrew/etc/nginx/servers/default.conf

server {
  server_name _; # aka catch all

  include cloudflare.conf;
  include common.conf;

  location / {
    return 404; # do some fancy page here
  }
}

goodbye ngrok

ngrok is not needed anymore, aka

/opt/homebrew/etc/nginx/servers/5000.conf

server {
  server_name 5000.mac-blog.org.ua;

  include cloudflare.conf;
  include common.conf;

  location / {
    proxy_pass http://192.168.105.109:5000;
    include proxy_headers.conf;
  }
}