Today, I had a server crash when the /var partition filled up while reindexing. I attempted to clear out extra log files and such but df was still showing greater than 100% utilization. Checking /var/db/mysql showed I had one table that was taking up too much space. du -d 1 -h is a very useful command when tracking down the largest folder usage on partition.
Jul 21 07:39:06 plab2catsg01 kernel: pid 36500 (mysqld), uid 88 inumber 329992 o
n /var: filesystem full
Jul 21 07:39:22 plab2catsg01 kernel: pid 36500 (mysqld), uid 88 inumber 353413 o
n /var: filesystem full
Jul 21 07:39:23 plab2catsg01 kernel: pid 36500 (mysqld), uid 88 inumber 353288 o
n /var: filesystem full
I attempted to stop MySQL by executing /usr/local/etc/rc.d/mysql-server stop and unfortunately, it kept looping while trying to stop the process. I was eventually forced to reboot the server.
On reboot, I thought all was well but then I found the Eventum application I use wasn't showing support emails. I checked the MySQL log files:
plab2catsg01# tail -f plab2catsg01.ten-net.net.err
...
090721 9:15:54 [ERROR] /usr/local/libexec/mysqld: Table './eventum/eventum_mail
_queue' is marked as crashed and last (automatic?) repair failed
090721 9:15:54 [ERROR] /usr/local/libexec/mysqld: Table './eventum/eventum_mail
_queue' is marked as crashed and last (automatic?) repair failed
090721 9:15:55 [ERROR] /usr/local/libexec/mysqld: Table './eventum/eventum_mail
_queue' is marked as crashed and last (automatic?) repair failed
090721 9:15:55 [ERROR] /usr/local/libexec/mysqld: Table './eventum/eventum_mail
_queue' is marked as crashed and last (automatic?) repair failed
...
Continue reading 'How to Repair a Crashed MySQL Table'
Recent Comments