aiodiskqueue

Persistent queue for Python AsyncIO.

release python tests codecov docs pre-commit Code style: black

Description

This library provides a persistent FIFO queue for Python AsyncIO:

  • Queue content persist a process restart

  • Feature parity with Python’s asyncio.Queue

  • Similar API to Python’s asyncio.Queue

  • Sane logging

  • Type hints

  • Fully tested

  • Supports different storage engines and can be extended with custom storage engines

Usage

Here is a basic example on how to use the queue:

import asyncio
from aiodiskqueue import Queue

async def main():
    q = await Queue.create("example_queue.sqlite")
    await q.put("some item")
    item = await q.get()
    print(item)

asyncio.run(main())

Please see the examples folder for more usage examples.

Installation

You can install this library directly from PyPI with the following command:

pip install aiodiskqueue

Logging

The name of the logger for all logging by this library is: aiodiskqueue.

Storage Engines

aiodiskqueue support different storage engines. The default engine is DbmEngine.

We measured the throughput for a typical load scenario (5 producers, 1 consumer) with each storage engine:

Measurements

  • DbmEngine: Consistent throughput at low and high volumes and about 3 x faster then Sqlite

  • PickledList: Very fast at low volumes, but does not scale well

  • SqliteEngine: Consistent throughput at low and high volumes. Relatively slow.

The scripts for running the measurements and generating this chart can be found in the measurements folder.