Selfhosted

If you're willing to use Caddy instead, it's infinitely easier. Websockets is just enabled by default, no shenanigans. Here is an example Caddyfile to use in a Docker deployment (but you can change those http urls to point to localhost for a non Docker deployment):

(caddy-common) {
    encode gzip
    header {
        -Server
        Strict-Transport-Security "max-age=31536000; include-subdomains;"
        X-XSS-Protection "1; mode=block"
        X-Frame-Options "DENY"
        X-Content-Type-Options nosniff
        Referrer-Policy  no-referrer-when-downgrade
        X-Robots-Tag "none"
    }
}

your.lemmy.url {
    import caddy-common
    reverse_proxy   http://lemmy-ui:1234

    @lemmy {
        path    /api/*
        path    /pictrs/*
        path    /feeds/*
        path    /nodeinfo/*
        path    /.well-known/*
    }

    @lemmy-hdr {
        header Accept application/*
    }

    handle @lemmy {
        reverse_proxy   http://lemmy:8536
    }

    handle @lemmy-hdr {
        reverse_proxy   http://lemmy:8536
    }

    @lemmy-post {
        method POST
    }

    handle @lemmy-post {
        reverse_proxy   http://lemmy:8536
    }
}

Caddy has some great plugins that allow you to automate https certificate renewal too, easy to add to any config.

I know you asked about nginx and I'm just telling you "haha just switch," but I had similar headaches with my own deployment when I tried using nginx, and I eventually just gave up and used Caddy. Saved me at least a few hours of headache.

The default Lemmy nginx config should handle websockets properly. Are you putting it behind a second nginx layer?

The nginx config provided in the Docker installation part contains everything needed for nginx. If you are installing lemmy directly on the machine you may need to use different upstreams.

The websocket part is basically the

            # proxy common stuff
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";```
part in the nginx config on that page.

Got it working on my instance by using the following config for the nginx outside of the docker container (you'll need to change the server_name from the server sections and the path to the ssl keys) :

upstream lemmy {
    # depending on your setup, you may want to update this
    server 127.0.0.1:1380;
}

server {
    listen 80;
    listen [::]:80;
    server_name lemmy.pierre-couy.fr;
    location / { return 301 https://$host$request_uri; }
}
server {
    listen      443 ssl;
    listen [::]:443 ssl;
    server_name lemmy.pierre-couy.fr;

    # TLS
    ssl_protocols TLSv1.2;
    ssl_ciphers HIGH:!MEDIUM:!LOW:!aNULL:!NULL:!SHA;
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;
    ssl_certificate     /etc/ssl/certs/pcouy.pem;
    ssl_certificate_key /etc/ssl/private/pcouy.key;

    # HSTS
    add_header Strict-Transport-Security "max-age=31536000";

    location / {
        proxy_pass http://lemmy;
        proxy_set_header Host $host;
        # include proxy_params;
    }

    location ~ ^/(api|pictrs|feeds|nodeinfo|.well-known) {
        proxy_pass "http://lemmy";
        # proxy common stuff
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        # Send actual client IP upstream
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

You can try add these to your nginx conf inside location / {:

proxy_set_header Connection "keep-alive, Upgrade";
proxy_set_header Upgrade websocket;

like this:

location / {
         .... 
         proxy_set_header Connection "keep-alive, Upgrade";
         proxy_set_header Upgrade websocket;
}

I launched my instance last weekend and this was my issue as too.