Guide

Learn how to build resilient, scalable async pipelines with Blazing Flow.

What is Blazing Flow?

Blazing Flow is a distributed task queue and pipeline orchestration engine that makes it easy to build complex, resilient data workflows. It's built on Redis and provides a simple Python API for defining asynchronous pipelines.

Key Features

• 🧭 Declarative Pipelines - Define workflows using simple Python decorators
• ⚡ Automatic Recovery - Jobs automatically resume after interruptions
• 🔄 Dynamic Optimization - Worker mix automatically adjusts to workload
• 🔌 Connection Pooling - Long-lived database and SSH connections
• 📊 Deep Observability - Real-time metrics and execution history
• ⚙️ Flexible Workers - Mix async and blocking workers for optimal performance

When to Use Blazing Flow

Blazing Flow excels at:

Quick Example

Here's a simple data processing pipeline:

Core Concepts

Workflows

Workflows are the entry points to your pipelines. They orchestrate the flow of data through multiple steps.

Steps

Steps are individual processing units. They can optionally receive services to access shared business logic.

Services

Services encapsulate your business logic and can be shared across steps.

Architecture

Blazing Flow uses a hierarchical architecture:

Components

• Foreman: Manages workers on each machine, optimizes worker mix
• Workers: Execute tasks from workflows and steps
  • Async Workers: Event-loop based, best for I/O operations
  • Blocking Workers: Thread-based, best for CPU-intensive tasks

Worker Types

Async Workers

Best for I/O-bound operations:

• API calls
• Database queries
• File I/O
• Network operations

Blocking Workers

Best for CPU-intensive operations:

• Data processing
• Image/video processing
• Cryptographic operations
• Heavy computations

Dynamic Worker Optimization

The Foreman automatically adjusts the worker mix based on queue metrics:

• Monitors queue depth and age
• Calculates urgency scores
• Increases/decreases async or blocking workers
• Uses hysteresis to prevent thrashing

You can configure optimization:

State Management

All operations are durably recorded in Redis:

• Units: Represent workflow executions
• Operations: Represent step calls
• Results: Final outputs and intermediate data

This enables:

• Automatic recovery after failures
• Audit trails for compliance
• Performance analysis
• Debugging with breadcrumbs

Synchronous API

Note: The sync API (SyncBlazing) is designed for learning and prototyping. For production workloads, we strongly recommend using the async Blazing class which provides better performance, proper resource management, and full support for concurrent operations.

Blazing Flow provides a synchronous API for users who are learning or want a simpler getting-started experience.

Option 1: SyncBlazing Class (For Learning/Prototyping)

The simplest sync experience - no async/await anywhere:

Option 2: Sync Helper Methods

If you have existing async code but need sync execution in some places:

When to Use Each API

| Use Case | API | Notes | |----------|-----|-------| | Production workloads | Blazing (async) | ⭐ Recommended - best performance | | FastAPI / async frameworks | Blazing (async) | Native async integration | | Learning Blazing Flow | SyncBlazing | Simpler to understand | | Prototyping | SyncBlazing | Quick iteration | | Jupyter notebooks | SyncBlazing | No asyncio hassle | | Django / Flask (learning) | SyncBlazing | Then migrate to async | | Django / Flask (production) | Blazing with sync helpers | Async inside sync views |

Error Handling

Blazing Flow provides multiple levels of error handling:

Monitoring

Built-in monitoring tools:

btop - Real-time Dashboard

Shows:

• Queue depths
• Worker states
• Throughput metrics
• Error rates

Foreman Charts

Visualizes:

• Worker mix over time
• Queue trends
• Optimization decisions

Next Steps