DevOps SysAdmin Tips for researching a slow or unresponsive app or system

Some general tips for researching a slow or unresponsive app or system

Starting checklist

• SSH to EC2/server 
  • Check disk space

    • df -h

  • Check processes

    • htop

    • ps aux

  • Check app logs

    • cd app/

    • ls -lhtr logs/

    • vim [log]

  • Check system logs

    • ls -lhtr /var/log/

    • tail -f -n50 /var/log/messages

• SQL Processes
• DB Gui: Tools -> Server Monitor -> SQL

• Check AWS console
• EC2 list
  • Monitoring
• RDS list
  • Monitoring
  • Performance Insights
• Cloud Watch
• Alarms
• Monitoring

• App specific
  • grep (Find in Files, Search)
  • grep cron for script name
  • grep code for database table names, fields
  • grep [everything]
  • Check logs
    • Often located relative to script, in logs/ or log/
    • Can be also be in /var/log/[app dir]/

---

Logs

App logs

Locations will vary
Some apps may be relative to their PHP (or other language) files, often log/ or logs/
Some cron or processing logs may be in /var/log/[process dir]/

Note, log locations should be consistent to facilitate finding them.

Web Server logs

On some errors, such as fatal or configuration errors, PHP (or other language) info is outputted to the web server logs, such as Nginx

Nginx logs

nginx is the process name
PHP runs as a separate process, php_fpm
ls -lhtr /var/log/nginx

tail -n50 /var/log/nginx/[site]_error.log
tail -f /var/log/nginx/[site]_error.log
ls -lhtr /var/log/php-fpm/

System logs

ls -lhtr /var/log/

You may need to login as sudo user ec2-user to view system logs

/var/log/messages

Common log for OS messages

sudo vim /var/log/messages

/var/log/secure

Common log for login, security messages

sudo vim /var/log/secure

---

Web Server Config

Nginx is a common web server 

You can grep for dns or web directories to know

• server_name = what dns is being used
• root = which directory a site is being hosted from
• error_log = where the log files are

grep [site] /etc/nginx/sites.d/*
vim /etc/nginx/sites.d/[site].conf

    server_name [site].com;
    set $env "dev";
    set $app_dir "[repo name]";

    root   /data/sites/$env/$app_dir/public/;

    access_log /var/log/nginx/[site].com_access_log;
    error_log  /var/log/nginx/[site].com_error_log error;

---

Cron

# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name  command to be executed

# EXAMPLE CRON
*/10 * * * * [app user] /usr/bin/php -q /[process dir]/[script] [args if any] &>> \
/var/log/[process dir]/[script].log

• ensure specify the app user
• &>> [log] is shorthand for >> [log] 2&>>1
• > = overwrite
• >> = append
• 2&>1 = redirect errors to standard out eg capture errors in log
• compare to other crons to keep a consistent pattern

System crons

ls -lhtr /etc/cron.d/

Preferred
Crons in /etc/cron.d/ allow for custom runtimes. A cron file per concept/process/logical grouping should be created, and can contain multiple scripts/commands.

Note, there are other potential system crons such as /etc/cron.daily which will run crons .. daily, but without the flexibility of specify when during the day they run; cron.daily often runs at 2am or 3am depending on the OS. Only use /etc/cron.d/

User crons

crontab -l

view cron of current user

crontab -e

edit cron of current user

Not recommended
Crons can also be attached to a user. User crons are not easily accessible and are stored in one file per user, making a large number of crons harder to manage, and overly easy to delete (crontab -r = poof gone).  

To manually backup a users cron 

cd ~
crontab -l  > cron_[Ymd].txt

---

-End of Document-

Thanks for reading