[Lookup] Error writing file '/tmp', (Errcode: 28 - No space left on device)

[mibby]

Logblock dev 223.

[16:26:13] [Craft Scheduler Thread - 201/ERROR]: [Lookup] SELECT date, replaced, type, data, playername, x, y, z, signtext FROM `lb-world` INNER JOIN `lb-players` USING (playerid) LEFT JOIN `lb-world-sign` USING (id) WHERE type != replaced ORDER BY date DESC, id DESC : 
java.sql.SQLException: Error writing file '/tmp/MYhXevGe' (Errcode: 28 - No space left on device)
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073) ~[spigot.jar:git-Spigot-1351]
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3593) ~[spigot.jar:git-Spigot-1351]
    at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:1566) ~[spigot.jar:git-Spigot-1351]
    at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1422) ~[spigot.jar:git-Spigot-1351]
    at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2902) ~[spigot.jar:git-Spigot-1351]
    at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:476) ~[spigot.jar:git-Spigot-1351]
    at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2608) ~[spigot.jar:git-Spigot-1351]
    at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1784) ~[spigot.jar:git-Spigot-1351]
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2198) ~[spigot.jar:git-Spigot-1351]
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2620) ~[spigot.jar:git-Spigot-1351]
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2570) ~[spigot.jar:git-Spigot-1351]
    at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1474) ~[spigot.jar:git-Spigot-1351]
    at de.diddiz.LogBlock.CommandsHandler$CommandLookup.run(CommandsHandler.java:418) [LogBlock.jar:?]
    at org.bukkit.craftbukkit.v1_7_R2.scheduler.CraftTask.run(CraftTask.java:58) [spigot.jar:git-Spigot-1351]
    at org.bukkit.craftbukkit.v1_7_R2.scheduler.CraftAsyncTask.run(CraftAsyncTask.java:53) [spigot.jar:git-Spigot-1351]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_20-ea]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_20-ea]
    at java.lang.Thread.run(Unknown Source) [?:1.8.0_20-ea]

Checking disk space with 'df -h', I have 176GB available on my main drive, 16GB available for 'tmpfs', 1.8GB out of 2GB available for '/temp', and 1.9GB out of 2GB available for '/var/temp', on my dedicated node. How is there 'No space left on device' ? My logblock database for mysql is 2.4GB large.

[md-5]

tmpfs is stored entirely in ram. If you run low on system ram, tmpfs will run out of space.

[mibby]

But the error is not related to tmpfs?

[frymaster]

The exact error is it can't write to /tmp because there's no space wherever /tmp is mounted. In some Linux distributions, /tmp is mounted as a tmpfs (ramdisk), so even if your disk has plenty of space you can get a "no space" error if you try to write too much there.

From the stack trace it looks as though this happened during a lookup (or lookup-and-rollback) - if the lookup was either returning a lot of data, or possibly even just sifting through a lot of data, large temp files could be created.

Possible solutions are make MySQL use a different temp directory, or to not use tmpfs for /tmp