Conference Presentation, Keynote, Tutorial
Marius Eriksen: Distributed, Incremental Dataflow Processing in the Cloud with Reflow
- Problem Context: The speaker, representing a bioinformatics team (Braille/Grail), addresses the challenge of detecting early-stage cancer, a task requiring massive computational infrastructure and data workflow systems capable of handling terabytes to petabytes of data over days or hours.
- Existing Landscape Limitations:
- Most current bioinformatics workflow systems are thin front-ends (e.g., YAML, Python DSLs) that wrap backend orchestration tools like Kubernetes or AWS Batch.
- Existing systems rely on manual, static dependency graphs, treating workflows as mechanical constructions rather than programs.
- Current tools lack a coherent data model, rendering input/output data invisible to the workflow engine and preventing true incremental computation.
- Core Design Philosophy:
- Workflows should be treated as ordinary programs rather than dependency graphs; the graph is an incidental implementation detail.
- Reflow enforces a strict data model where all tasks are referentially transparent, enabling stable data names and powerful caching.
- The system integrates language and runtime vertically to minimize complexity, allowing a simple implementation to support complex distributed execution.
- Reflow Technical Specifications:
- Language: A functional language with static types (mostly inferred), a module system (similar to Go), and syntax blending elements of JavaScript and Go.
- Exec Expression: A core mechanism for invoking external binaries; these expressions are first-class values returning tuples of files or directories, seamlessly integrated into the lexical scope.
- Incremental Evaluation: Achieved through lazy evaluation, data-flow semantics (parallelizing independent tasks), and referential transparency (memoization of all reductions).
- Parallelism: Implicit; the system automatically parallelizes any task with no data dependencies without requiring explicit concurrency directives from the user.
- Resource Management: Implements its own cluster manager that dynamically provisions cloud resources (e.g., AWS Spot Instances) based on workload requirements, abstracting infrastructure details from the user.
- Demonstrated Capabilities:
- Hello World: Provisions a cloud instance, runs an
echocommand, and retrieves the result, demonstrating automatic resource provisioning and file hashing for caching. - Dog Montage: Processes hundreds of images in parallel across multiple instances, then aggregates them; subsequent runs with modified parameters only recompute the changed steps (e.g., montage generation) while reusing cached image resizing results.
- Machine Learning Pipeline: Handles a workflow of feature extraction, model training, and evaluation; incremental updates to the training set or model code trigger re-evaluation only for the affected downstream components.
- Hello World: Provisions a cloud instance, runs an
- Data as API Concept:
- Bundles: A mechanism to "freeze" modules and their entire dependency graph (including specific Docker image digests) into a single binary archive, ensuring reproducible, stable computation over time.
- Data Spaces: A namespace mapping symbolic names to bundled modules, creating typed, versioned, and documented data APIs where updates require subtype compatibility to prevent breaking changes.
- System Architecture:
- Alloc (Resource Allocation): An abstraction grouping a Docker daemon, a repository (file store named by content hash), and running containers; allocations live and die together.
- Evaluation Flow: Uses two evaluators (AST and Flow Graph); the Flow Graph contains
execnodes (run to completion),internalizenodes,externalizenodes, andcontinuationnodes that allow the AST evaluator to dynamically expand the graph. - Incremental Strategy: Evaluates top-down to find cache hits, then bottom-up to compute missing values, ensuring only necessary work is performed.
- Current Limitations & Future Roadmap:
- Resource Oversubscription: Memory errors (OOM) are currently handled via reruns with dynamic adjustments, though the team aims to move toward dynamic profiling rather than static resource hints.
- Failure Handling: The language currently lacks explicit error handling mechanisms; unhandled application failures kill the task, though runtime failures (e.g., OOM) are managed by the system.
- Type System Evolution: Planned additions include polymorphic variants to enforce stricter schema validation for file formats (e.g., specific TSV columns or VCF headers).
- Graph Size Optimization: The evaluator has evolved from naive graph reversal to a reference-counted, incremental dirty-checking system, optimizing cost to be proportional to parallelism rather than total graph size.
- Adoption & Status:
- Reflow is open source on GitHub and is the primary compute infrastructure for Grail.
- It is also utilized by the Chan Zuckerberg Institute and several small biotech startups.
- The system requires only user cloud credentials to function, dynamically handling all infrastructure provisioning.