Camilo Castro
All workCase 04 / 05

Kiln Academic project

Distributed image processing

Upload a batch, define global and per-image transformations, and a pool of worker nodes processes it in parallel through RabbitMQ — with PostgreSQL replication and shared storage across five virtual machines.

  • FastAPI publishes one task per image to RabbitMQ; worker nodes with four threads each consume, process and acknowledge them.
  • A PostgreSQL primary with a streaming-replication standby and NFS for shared files, deployed on five VirtualBox VMs — and on Docker Compose.
  • Kiln, the web client, is dependency-free JavaScript: batch builder, 11 transformation types, live progress and ZIP download.
Client
Course project — Universidad Pontificia Bolivariana
Role
Design and implementation
Year
2026 Academic · open source
Stack
  • Python
  • FastAPI
  • RabbitMQ
  • PostgreSQL
  • NFS
  • Docker Compose
  • VirtualBox
  • Vanilla JS
Source code 1Source code 2
Kiln's batch builder: eight uploaded photos on the left and a stack of global transformations — resize, contrast, sharpen, watermark, convert — on the right.
Batch detail after processing: 8 of 8 images completed, progress at 100% and a ZIP download button.

The brief

A distributed-systems assignment taken past the minimum: a client uploads N images, each with a JSON list of transformations, and the system spreads the work across nodes, tracks every task and hands back a ZIP when the batch is done.

Real machines instead of one process made it interesting: failures can happen anywhere, and state has to stay consistent while they do.

What I built

  • REST API to create batches (files or ZIP), query progress and download results
  • Worker nodes with configurable thread pools consuming RabbitMQ with ack/nack
  • 11 transformations: resize, grayscale, rotate, crop, flip, blur, sharpen, brightness, contrast, watermark, format conversion
  • Batch, task and node state in PostgreSQL, streamed to a read-only replica
  • Kiln web client: JWT login, global and per-image steps, live JSON preview, polling and ZIP download
  • Automatic cleanup of inputs after processing and of outputs after download

Architecture

Kilnweb client · JWT
REST
FastAPIvm-api · uvicorn
AMQP
RabbitMQone task per image
tasks
Worker 14 threads
Worker 24 threads
SQL · NFSSQL · NFS
PostgreSQLprimary · vm-db
WAL
Replicaread-only standby
NFSshared storage

Under the hood

  1. The upload never waits for the queue

    The API stores the files, writes the batch and publishes tasks in a background task, so the client gets its batch id immediately.

  2. Degrade, don't crash

    If RabbitMQ is down, the API still starts and answers new batches with 503 instead of failing at boot.

  3. State as a catalog

    Batch, task, result and node states are catalog tables; nodes register themselves, and a batch closes when its last task completes.

  4. Replication for reads

    WAL streaming replication keeps a hot standby of the database on its own virtual machine.

Gallery

Per-image mode: one photo selected with its own rotate and grayscale steps and a JSON preview.
Per-image steps stack on top of the global ones.
Batch history table with id, image count, date, status and a progress bar.
Batch history, refreshed against the API.
Kiln's docs page: the 11 available transformations with their parameters, the recommended order and the API flow.
Docs rendered live from GET /info: transformations, order and flow.
UML deployment diagram: API server with RabbitMQ and storage, two worker nodes, a primary database and its read-only replica.
The deployment diagram from the project documentation.

Where it stands

Deployed on five VirtualBox VMs over a bridged network and reproducible with Docker Compose. The screens on this page come from a local run processing an eight-image batch.