Web Stack Debugging #1 ~ A Postmortem
This is the incident report for the server 271 running on ubuntu 14.04 to provide a fresh Nginx installation for users.
Issue Summary
On spinning up the sandbox, running curl on port 80 of all ip addresses curl 0:80 returns an error curl: (7) Failed to connect to 0 port 80: Connection refused
Timeline (all times GMT +0)
- 6:19 PM: Server 271 spins up
- 6:19 PM: Port 80 access is denied
- 6:26 PM: Sylvain Kalache alerted teams
- 6:30 PM: On-call Devops engineer is assigned to task
- 6:35 PM: Successful temporary configuration change
- 6:37 PM: Nginx service restarts
- 6:38 PM: Fully online and port functioning
- 6:52 PM: Proper configuration steps taken
Root Cause
At 6:19 PM, server 271 was initialized from pre-configured scripts.
The webserver’s Nginx configuration had errors in the sites-available and sites-enabled configuration default files. This caused an error in the port forwarding as the sites-enabled/default file was configured to run on port 8080 (instead of port 80).
Resolution
At 6:26 PM, the monitoring systems alerted Sylvian who investigated and quickly escalated the issue to the Devops team. By 6:30 PM, the incident response engineer was deployed and by 6:33 PM detected that the server was running on port 8080 as opposed to port 80.
At 6:35 PM, a temporary change was made by copying the config file with the correct port (sites-available) to the sites-enabled file. and at 6:37 PM the Nginx server was restarted and responded with 200 OK when curled at port 80. By 6:52 PM, the correct measure was taken to address the issue.
Corrective and Preventive Measures
The issue can be addressed from the startup configuration files. Or if the server is already running, relink the sites-enabled/default file to the sites-available/default file in the /etc/nginx/ folder
Sincerely,
Alx-africa cohort 10’s Aina Jesulayomi
