Debugging a Random 502 Error in Production: When PM2 Was Watching the Wrong Process

Recently I ran into an interesting production issue on one of my Node.js applications.

The API would randomly start returning 502 Bad Gateway errors. The strange part was that the issue wasn't temporary. Once it happened, the backend stayed down until I pushed a new deployment.

At first glance, it looked like a networking issue, reverse proxy issue, or even a server resource issue. The actual root cause turned out to be much simpler—and much more subtle.

Architecture

The stack was fairly straightforward:

Frontend (React/Next)
    ↓
Caddy Reverse Proxy
    ↓
Node.js + Express API
    ↓
PM2 Process Manager

Domain setup:

dev.myapp.com      → Frontend (Port 3000)
devapi.myapp.com   → Backend (Port 5000)

Caddy was responsible for reverse proxying requests to the appropriate application.

The Symptoms

Everything would work perfectly after deployment.

Then, at some unpredictable point:

• API requests started returning 502 errors
• Frontend remained online
• Backend became inaccessible
• The issue persisted indefinitely
• Deploying new code immediately fixed the problem

The most confusing part was that a deployment containing no functional changes would still restore service.

That suggested the deployment process itself was somehow recovering the application.

Initial Investigation

My first assumption was that Caddy was misconfigured.

I verified:

• Reverse proxy configuration
• SSL certificates
• Domain mappings
• Listening ports

Everything looked correct.

Next, I checked whether the backend process was actually running.

lsof -i :5000

The application appeared to be listening on port 5000.

That made the issue even more confusing.

Looking at PM2

While investigating, I noticed something unusual.

Running:

pm2 list

as root showed no applications.

After switching users:

su - dev
pm2 list

I finally saw the running processes.

dev-api
dev-fe

This led me deeper into how the application was being started.

The Real Problem

The backend was not being started directly through PM2.

Instead, the process chain looked something like:

PM2
 └── npm
      └── nodemon
           └── node

PM2 was effectively monitoring the parent process rather than the actual Node.js server process.

This creates a subtle but important problem.

If the Node process crashes, hangs, or becomes unhealthy:

• PM2 may still see the parent process as alive
• Automatic recovery may not occur
• The application remains unavailable
• The reverse proxy returns 502 responses

From PM2's perspective, everything can appear healthy even though the API is no longer functioning correctly.

This perfectly matched the behavior I was seeing.

Why Deployment "Fixed" It

Every deployment performed a PM2 restart.

That restart rebuilt the process chain and launched a fresh backend process.

The application would work again until the next failure occurred.

The deployment wasn't fixing the bug—it was simply restarting the application.

The Fix

I switched the application to a proper PM2 production setup using an ecosystem configuration.

Instead of:

PM2
 └── npm run dev
      └── nodemon
           └── node

the architecture became:

PM2
 └── node server.js

Additional improvements included:

• PM2 ecosystem configuration
• Proper restart policies
• Automatic recovery settings
• Updated deployment commands
• Direct process monitoring

The deployment pipeline was also updated to use PM2's restart mechanisms correctly.

Lessons Learned

A few key takeaways from this incident:

1. Development commands don't belong in production

Tools like nodemon are fantastic for local development but should not sit between PM2 and the actual application process in production.

2. A deployment fixing an issue is a clue

If redeploying consistently resolves a problem, investigate what the deployment changes operationally—not just what code it deploys.

3. Verify what PM2 is actually monitoring

Seeing a process marked as "online" doesn't necessarily mean PM2 is supervising the process you think it is.

4. Start with the architecture

The issue wasn't inside Express, Caddy, or the application logic.

It was in the relationship between the process manager and the application process.

Current Status

The application now runs under a proper PM2 production configuration and is being monitored to confirm long-term stability.

While it's too early to declare the issue permanently solved, the root cause analysis strongly points to process management as the source of the recurring outages.

For now, the random 502 errors have stopped appearing, and the backend is recovering correctly when restarted through PM2.

Sometimes the hardest production bugs aren't caused by application code at all—they're caused by the layers around it.