Documentation

Build with Flint without guessing the contract.

API envelopes, worker controls, handler payloads, deployment notes, and the scheduling behavior that powers the dashboard.

REST

Versioned /api/v1 routes

SSE

Live job status stream

Queue

Heap and timing wheel

Overview

Flint is a background job scheduler built with FastAPI (backend) and Next.js (frontend). It supports three job handler types, two interchangeable scheduling algorithms, real-time status streaming via SSE, and a full management UI.

The REST API follows a consistent envelope format. Every response body has the shape:

json

{
  "message": "Human-readable status string",
  "data": { ... } | null,
  "errors": [],
  "meta": { "page": 1, "limit": 20, "total": 142 } | null
}

Note

All API routes are prefixed with /api/v1/. The frontend proxies these through Next.js API routes so the FastAPI server is never directly exposed to the browser.

Getting started

Prerequisites

You will need Docker and Docker Compose installed on your machine.

bash

docker --version      # >= 24.0
docker compose version # >= 2.20

Installation

Clone the repository and bring up the full stack with one command:

bash

git clone https://github.com/your-org/flint.git
cd flint
docker compose up --build

This starts four services: api (FastAPI on port 8000), web (Next.js on port 3000), db (PostgreSQL), and worker (background job runner).

Configuration

Copy .env.local.example to .env.local in the frontend directory and set your values:

bash

BACKEND_URL=http://api:8000
FLINT_API_KEY=your-secret-key
NEXT_PUBLIC_API_URL=http://localhost:8000

Warning

BACKEND_URL and FLINT_API_KEY are server-side only. Never prefix them with NEXT_PUBLIC_.

API reference

Jobs

GET/api/v1/jobs

List all jobs with optional filters and pagination.

POST/api/v1/jobs

Create a new job.

GET/api/v1/jobs/{id}

Retrieve a single job by ID including logs and dependencies.

PATCH/api/v1/jobs/{id}/cancel

Request cancellation of a pending or processing job.

DELETE/api/v1/jobs/{id}

Soft-delete a job (moves to Bin).

Query parameters for GET /jobs:

Field	Type	Req	Description
status	string	—	Filter by status: pending \| processing \| completed \| failed \| cancelled
type	string	—	Filter by job type.
priority	integer	—	Filter by priority level (1, 2, or 3).
page	integer	—	Page number (default: 1).
limit	integer	—	Results per page (default: 20, max: 100).

Request body for POST /jobs:

Field	Type	Req	Description
type	JobType	yes	send_email \| webhook_delivery \| log_processing
payload	object	yes	Handler-specific payload. See Job handlers.
priority	1 \| 2 \| 3	yes	1 = High, 2 = Medium, 3 = Low.
scheduled_at	ISO 8601	—	When to run. Omit for immediate execution.
interval	string	—	Recurrence interval, e.g. 30s, 5m, 1h, 1d.
dependency_ids	string[]	—	Job IDs that must complete before this job runs.
max_retries	integer	—	Retry limit before DLQ. Default: 3.

json

// POST /api/v1/jobs — example
{
  "type": "webhook_delivery",
  "priority": 1,
  "payload": {
    "url": "https://api.yourapp.com/hooks/payment",
    "method": "POST",
    "body": { "event": "payment.success", "amount": 4900 }
  },
  "interval": "5m",
  "max_retries": 5
}

Dead letter queue

GET/api/v1/dlq

List jobs in the dead letter queue.

POST/api/v1/dlq/{id}/retry

Re-queue a DLQ job from the beginning.

DELETE/api/v1/dlq/{id}

Permanently remove a job from the DLQ.

Note

A job moves to the DLQ once retry_count reaches max_retries. The threshold for alerting is configurable in Settings.

Bin

GET/api/v1/bin

List soft-deleted jobs.

PATCH/api/v1/bin/{id}/restore

Restore a job back to the main queue.

DELETE/api/v1/bin/{id}

Hard-delete a job. Irreversible.

Workers

GET/api/v1/workers

List all registered workers and their current status.

POST/api/v1/workers/{id}/stop

Gracefully stop a worker after its current job.

POST/api/v1/workers/{id}/restart

Restart a stopped worker.

Logs

GET/api/v1/logs

Query the job event log with optional filters.

Field	Type	Req	Description
job_id	string	—	Filter logs for a specific job.
event	string	—	Filter by event type, e.g. job_failed, job_completed.
page	integer	—	Page number.
limit	integer	—	Results per page.

Settings

GET/api/v1/settings

Retrieve current system settings.

PATCH/api/v1/settings

Update one or more settings fields.

Field	Type	Req	Description
scheduler_strategy	string	—	heap \| timing_wheel — switches the active algorithm.
dlq_threshold	integer	—	Alert when DLQ count reaches this number.
alert_emails	string	—	JSON array of email addresses as a string.

Benchmark

POST/api/v1/benchmark/run

Run an algorithm benchmark and return timing results.

Field	Type	Req	Description
n	integer	yes	Number of jobs to insert and pop (e.g. 1000, 10000, 100000).
algorithm	string	yes	heap \| timing_wheel \| both

SSE stream

GET/api/v1/sse/stream

Server-Sent Events stream for real-time job status updates.

The browser opens this connection directly to the FastAPI server via the NEXT_PUBLIC_API_URL env var. Each event is a JSON-encoded object:

json

// Incoming SSE event
data: {"job_id": "a3f9d2c1-...", "status": "completed", "worker_id": "worker-01"}

Tip

The frontend maintains a single global SSE connection inside AppShell via the useSSE hook. Status updates are written to Zustand and reflected immediately in the jobs table and job detail view without a refetch.

Job handlers

send_email

Sends a transactional email via the configured mail provider.

Field	Type	Req	Description
to	string	yes	Recipient email address.
subject	string	yes	Email subject line.
body	string	—	Plain-text body.
html_body	string	—	HTML body. Overrides body if both are provided.

json

{
  "type": "send_email",
  "priority": 2,
  "payload": {
    "to": "alice@example.com",
    "subject": "Your order has shipped",
    "html_body": "<p>Hi Alice, your order <b>#4821</b> is on its way.</p>"
  }
}

webhook_delivery

Posts a signed HTTP request to an arbitrary endpoint.

Field	Type	Req	Description
url	string	yes	Target URL.
method	string	—	HTTP method. Default: POST.
headers	object	—	Key-value map of request headers.
body	any	—	Request body. Serialised to JSON.

json

{
  "type": "webhook_delivery",
  "priority": 1,
  "payload": {
    "url": "https://hooks.yourapp.com/payment",
    "method": "POST",
    "headers": { "X-Signature": "sha256=abc123" },
    "body": { "event": "payment.captured", "amount": 9900 }
  }
}

log_processing

Ingests and processes structured log lines from a named source.

Field	Type	Req	Description
source	string	—	Identifier for the log source, e.g. nginx-prod-01.
log_lines	string[]	yes	Array of raw log lines to process.

json

{
  "type": "log_processing",
  "priority": 3,
  "payload": {
    "source": "nginx-prod-01",
    "log_lines": [
      "2024-01-15T12:00:01Z ERROR connection timeout to db",
      "2024-01-15T12:00:02Z WARN  retry attempt 1 of 3"
    ]
  }
}

Algorithms

Min-heap

The default strategy. Jobs are stored in a binary min-heap ordered by effective_priority. The scheduler always pops the root — guaranteeing the highest-priority job runs next. Insert and pop are both O(log n).

effective_priority is derived from the base priority level combined with scheduled time, so a lower-priority job can bubble up if it has been waiting too long.

Timing wheel

A circular bucket structure where each slot represents a time tick. Jobs are inserted into the slot matching their scheduled time. The scheduler hand advances one slot per tick and dispatches all jobs in the current slot. Both insert and pop are O(1).

Best for high-volume recurring workloads where all jobs have roughly equal priority and precise timing matters more than relative ordering.

Switching at runtime

Send a PATCH /api/v1/settings request with scheduler_strategy set to either heap or timing_wheel. The change is applied to the next job pop — no restart required.

bash

curl -X PATCH http://localhost:8000/api/v1/settings \
  -H "X-API-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"scheduler_strategy": "timing_wheel"}'

Tip

You can also switch algorithms from the Settings page in the dashboard or from the Algorithm Switcher on the landing page.

Deployment

Docker Compose

The repository ships with a docker-compose.yml that wires up all four services. For production, set NODE_ENV=production and ensure your domain is pointed at the Nginx reverse proxy.

bash

docker compose -f docker-compose.prod.yml up -d

Environment variables

Field	Type	Req	Description
BACKEND_URL	string	yes	Internal URL of the FastAPI service. Server-side only.
FLINT_API_KEY	string	yes	Secret API key injected into every proxied request.
NEXT_PUBLIC_API_URL	string	yes	Public URL of the FastAPI service. Used for SSE from the browser.