What is Q4M?

Q4M (Queue for MySQL) is a message queue licensed under GPL that works as a pluggable storage engine of MySQL, designed to be robust, fast, flexible. It is already in production quality, and is used by several web services (see Users of Q4M).
To start using Q4M, download either a binary or source distribution from the install page, and follow the installation instructions. A small tutorial is also avialable. You may use SQL to access Q4M queues, or there is a wrapper module available for perl (Queue::Q4M).
For more information, please read the developer's weblog (Kazuho at Work) or subscribe to the mailing list.

What's New

0.9.14 release (Jul 30 2014)
fix race condition between DELETE and queue_wait() that may cause crash in case compaction is triggered
0.9.13 release (Feb 6 2014)
fix deadlock when using multi-table queue_wait on MySQL 5.1
0.9.12 release (Jan 30 2014)
add CMakeFiles.txt (to support MySQL 5.5 / 5.6)
0.9.11 release (Jan 27 2014)
fix memory leak in conditional subscription
0.9.10 release (May 21 2013)
fix crash when using conditional subscription under high load
0.9.9 release (Mar 27 2013)
fix possible crash / data-loss when trying to access data created by older versions of Q4M (the bug exists in 0.9.7 and 0.9.8)
0.9.8 release (Mar 22 2013)
preliminary support for MySQL 5.5 and 5.6
fix error that may cause the size of the queue file to increase forever
0.9.7 release (Nov 21 2012)
introduce "queue_stats()" function for per-table metrics

Design Goals of Q4M

Robust
Q4M logs all message transfers to disk. A power failure or OS crash would never loose a message.
Fast
Although Q4M syncs every operation to disk, it runs fast by using group commits, and checksums for data recovery. Even with HDD write cache disabled, it is able to transfer thousands of messages per second on an ordinal linux server.
Flexible
Since Q4M works as a storage engine of MySQL, ordinally SQL can be used for administration. It is also possible to JOIN Q4M queues against other MySQL tables. Or you may use triggers for sending data into queue.

Users of Q4M

Q4M is used by some of the most popular web services in Japan. The users include:

Limitations and Known Issues

Limitations, known issues under version 0.5.
  • row size is limited to 512MB
  • table size is limited to 2**63 bytes
  • no support for UPDATE and REPLACE
  • no support for indexes
  • removal of multiple rows from a single DELETE statement is not atomic
  • may cause server crash under extremely low memory condition
  • may lose data if OS crashes while Q4M is performing table compaction (since the directory is not fsynced)