博客MySQL服务惊魂记：Linux磁盘空间不足(Core Dump文件引爆)

昨天在更新完文章《升级博客文章同步微博的插件：PHP正则提取Markdown的图片地址》后，博客突然无法访问了。

MySQL的错误日志

1 查看Nginx的错误日志error.log

惊愕之余，换了多个浏览器打开博客首页无果(一直显示加载状态)，最终报内部服务器500的错误。立即查看Nginx服务的错误日志：

[root@typecodes nginx]# vim error.log 
........
[error] 19620#0: *1 FastCGI sent in stderr: "PHP message: SQLSTATE[HY000] [2002] No such file or directory" while reading response header from php-fpm.sock:", host: "typecodes.com"
........

Nginx的错误日志

2 查看MySQL的错误日志error.log

从上面的Nginx错误日志中可以看出，可能是MySQL的错误。于是查看MySQL的错误日志error.log，但是当时没注意到下面这条重要错误信息提示：No space left on device。

#### 查看MySQL的错误日志error.log
[root@typecodes mysql]# ls
error.log  mysql.pid
[root@typecodes mysql]# vim error.log 
[02:47:34] [20150816_02:47:34] 103 2015-08-15T18:47:11.843790Z 0 [ERROR] InnoDB: Error number 28 means 'No space left on device'.
[02:47:34] [20150816_02:47:34]     es.html
[02:47:34] [20150816_02:47:34] 105 2015-08-15T18:47:11.843805Z 0 [Warning] InnoDB: Retry attempts for writing partial data failed.
[02:47:34] [20150816_02:47:34] 106 2015-08-15T18:47:11.843812Z 0 [ERROR] InnoDB: Write to file ./ibtmp1 failed at offset 11534336. 1048576 bytes should have been written, only 1044480 were written.
[02:47:34] [20150816_02:47:34]     Check that your OS and file system support files of  this size. Check also that the disk is not full or a disk quota  exceeded.
[02:47:34] [20150816_02:47:34] 107 2015-08-15T18:47:11.843821Z 0 [ERROR] InnoDB: Could not set the file size of './ibtmp1'. Probably out of disk space
[02:47:34] [20150816_02:47:34] 108 2015-08-15T18:47:11.843828Z 0 [ERROR] InnoDB: Unable to create the shared innodb_temporary
[02:47:34] [20150816_02:47:34] 109 2015-08-15T18:47:11.947045Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
[02:47:34] [20150816_02:47:34] 110 2015-08-15T18:47:11.947081Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
[02:47:34] [20150816_02:47:34] 111 2015-08-15T18:47:11.947089Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
[02:47:34] [20150816_02:47:34] 112 2015-08-15T18:47:11.947098Z 0 [ERROR] Failed to initialize plugins.
[02:47:34] [20150816_02:47:34] 113 2015-08-15T18:47:11.947143Z 0 [ERROR] Aborting
[02:47:34] [20150816_02:47:34] 114 
[02:47:34] [20150816_02:47:34] 115 2015-08-15T18:47:11.947154Z 0 [Note] Binlog end
[02:47:34] [20150816_02:47:34] 116 2015-08-15T18:47:11.947598Z 0 [Note] /usr/local/mysql/bin/mysqld: Shutdown complete

由于本能反应（BZ的CentOS7.1服务器20G系统盘+20G数据盘），服务器不可能会出现磁盘空间不足的问题。结果再一次忽略了No space left on device提示。

[root@typecodes mysql]# vim error.log 
........
mysql/data
etting it to a valid, non-empty path.
rc-log) starting as process 22504 ...
c builtins
2015-08-15T18:49:59.921545Z 0 [Note] InnoDB: Uses event mutexes
) is used for memory barrier
2015-08-15T18:49:59.921559Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.3
2015-08-15T18:49:59.922232Z 0 [Note] InnoDB: Number of pools: 1
2015-08-15T18:49:59.922362Z 0 [Note] InnoDB: Using CPU crc32 instructions
ze = 128M, instances = 1, chunk size = 128M
pool
e distribution
user: 'TypeCodes_test' host: 'localhost' (Got an error reading communication packets)
user: 'TypeCodes_test' host: 'localhost' (Got an error reading communication packets)
pecodes.log' (errno: 28 - No space left on device)
00 secs
s)
s)
.......
2015-08-16T14:36:25.977210Z 831 [ERROR] Retry in 60 secs. Message reprinted in

后面尝试使用一个MySQL测试用户登录数据库，还是报错：

[root@typecodes ~]# mysql -u TypeCodes_test -p
[02:43:32] [20150816_02:43:32] Enter password: 
[02:43:32] [20150816_02:43:32] ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysql/mysql.sock' (2)

找不到方法，只能先重启一下MySQL服务，结果一直报The server quit without updating PID file的错误。又一次查看日志，最终确认从问题No space left on device入手。

3 查看Linux磁盘空间信息

根据上面MySQL的No space left on device提示，立马使用df -hl命令查看当前CentOS7.1服务器的磁盘空间信息。结果很明显，挂载在/dev/xvdb2数据盘下面的分区/mydata只剩的12M的空间了！而mysql得数据文件正好存放在mydata分区中，悲剧！

CentOS7.1服务器上查看磁盘空间

接着进入到/mydata分区，使用命令du -sh查看当前目录下各个子目录所占用的空间大小。结果太出乎意料了，竟然是存放Core Dump文件的corefile目录占用了16G的空间！

Core Dump文件占用空间过大

这才想起在《CentOS开启coredump转储并生成core文件的配置》文中，设置了Core Dump文件大小为unlimited。日积月累，终于把MySQL搞挂掉了！

4 解决

先删掉corefile目录下面的大部分Core Dump文件，然后使用命令ulimit -c 10240设置coredump file-size为10M。最后，使用命令service mysqld start成功启动MySQL服务！看了下时间，从发现问题到解决用了40多分钟，还好只是个个人博客~

TypeCodes