Architecture

Blazing Flow is built on a hierarchical architecture inspired by manufacturing, where Foremen manage workers executing tasks. This design provides scalability, reliability, and operational flexibility.

System Components

Foreman

The Foreman manages workers on a single machine:

• Worker Lifecycle: Spawns, monitors, and stops worker processes
• Mix Optimization: Dynamically adjusts async/blocking worker ratios
• Resource Management: Monitors CPU, memory, and queue depths
• Health Checks: Detects and restarts failed workers

Key Features:

• Hysteresis-driven optimization to prevent thrashing
• Urgency scoring based on queue depth and age
• Graceful worker replacement

Workers

Workers execute tasks from workflows and steps:

Async Workers

• Run on event loop (uvloop)
• Best for I/O-bound tasks (API calls, database queries)
• High concurrency with low overhead
• Share connections efficiently

Blocking Workers

• Thread-based execution
• Best for CPU-intensive tasks
• Isolated from event loop
• Prevents blocking async workers

Data Flow

Workflow Execution Flow

Example Flow

When you call process_order("12345"):

1. Task queued with order_id="12345"
2. Available worker dequeues task
3. Worker executes workflow, calling each step in sequence
4. Each step may spawn its own tasks
5. Caller receives result or task ID for async tracking

State Management

DAO Layer

Blazing uses a Data Access Object (DAO) pattern for all state:

• ForemanDAO: Foreman state and metrics
• ConnectorDAO: Connection pool definitions
• StepDAO: Step metadata
• UnitDAO: Individual task state
• OperationDAO: Execution history

Persistence Model

All operations are durably recorded:

This enables:

• Resume after failures
• Audit trails
• Performance analysis
• Debugging

Worker Mix Optimization

Blazing automatically optimizes the balance between async and blocking workers based on your workload. The system monitors queue depths and task patterns to adjust worker allocation in real-time.

You don't need to configure this - it happens automatically. You can monitor the optimization decisions and worker mix through the btop dashboard available on blazing.work.

Connection Management

Connector Types

Blazing supports multiple connector types:

SQL Connectors

HTTP Connectors

SSH Connectors

Connection Pooling

Connectors maintain connection pools to avoid overhead:

• Async Pools: Shared across async workers
• Blocking Pools: Thread-local for blocking workers
• Lifecycle: Connections persist across operations
• Health Checks: Automatic reconnection on failure

Monitoring & Observability

Built-in Tools

btop - Real-time Dashboard

Shows:

• Queue depths
• Worker states
• Throughput metrics
• Error rates

Foreman Charts

Visualizes:

• Worker mix over time
• Queue trends
• Optimization decisions

Breadcrumb Logging

Every operation logs structured breadcrumbs:

Access breadcrumbs for debugging:

Scalability

Horizontal Scaling

Add more Foreman instances:

All Foremen process tasks cooperatively.

Vertical Scaling

Increase workers per Foreman:

Next Steps