Conference Presentation, Keynote, Tutorial
Marius Eriksen: Distributed, Incremental Dataflow Processing in the Cloud with Reflow
- Plans to develop an early-stage cancer detection assay and invest heavily in computational infrastructure capable of handling jobs ranging from minutes to days and processing terabytes to petabytes of data.
- Reflow will be a purely incremental, implicit parallel functional language with static types (mostly inferred) that treats workflows as ordinary programs, enabling automatic dependency tracking and recomputation only when specific inputs or code change.
- The runtime will co-design with the language to provide a vertically integrated system that encapsulates cluster computing, dynamically provisions EC2 spot market instances with bid price optimization, and maintains reusable resources within five to ten-minute windows.
- Data workflow systems require dynamism to make processing decisions based on actual input, utilizing a strong data model where files are opaque and associative tables (stored in DynamoDB) track value dependencies across external repositories.
- Module architecture supports referential transparency and lazy evaluation, allowing values to act as stable identifiers with exports defined by capital letter identifiers, while bundles freeze dependency graphs into single archives for versioned data APIs.
- Execution involves evaluating an AST into a flow graph to separate concurrency from semantics, with incremental evaluation performing top-down traversal for cache hits followed by bottom-up computation, ensuring cost is proportional to parallelism rather than graph size.
- External binaries can be invoked via
execexpressions that return value tuples and interpolate from a flexible environment, though materialized paths are treated as effectively random and external files are assumed to be immutable for reproducible results. - Future development focuses on enriching polymorphic variant types for specific file formats, improving static type safety to prevent late-stage errors, and moving resource requirements to dynamic hints based on profiling data.
- Current limitations include an absence of native failure handling mechanisms where execution simply stops, prompting plans to implement "keep going" modes for concurrent work.
- The language aims for simplicity and portability with a minimal dependency footprint to operate across different cloud providers, while data spaces enforce policies ensuring updated modules remain subtypes of existing ones to prevent consumer breakage.
- Reflow will be open-sourced on GitHub and is intended for use at Grail, the Chan Zuckerberg Institute, and select biotech startups, with experimental support planned for database integrations like BigQuery.
- Data APIs will transform ad hoc file tracking into a structured model, allowing workflows to be tested, reused across tasks, and documented via
reflow runandreflow doccommands that display parameters, documentation, and inferred types.