Bacalhau

Bacalhau is an open‑source framework that orchestrates distributed compute jobs by running them close to the data’s location. It avoids moving large datasets across networks, which reduces latency, lowers ingress/egress costs, and enables processing of petabyte‑scale data within existing security boundaries. The system supports both Docker containers and WebAssembly modules, and it can read from storage backends such as S3, HTTP/HTTPS, IPFS, and local disks, allowing jobs to be defined declaratively in YAML or imperatively via the command line.

The platform consists of a single self‑contained binary that serves as client, orchestrator, and compute node, simplifying deployment and scaling. A dedicated orchestrator schedules jobs while compute nodes execute them, and results can be published to local volumes, S3, or other storage targets. Bacalhau’s permission model provides granular, code‑based access control, facilitating secure, cross‑organizational computation without exposing raw data.

Typical users include data engineers, researchers, and organizations that need to run analytics, machine‑learning pipelines, or batch processing on distributed datasets while respecting data sovereignty and cost constraints. The stable release offers modular execution engines, multiple job types (batch, ops, daemon, service), and integration with common storage services.