Flyte Experiments

About This Repo

This repo contains personal experiments around Flyte. Generally, it explores the question of: what’s it like to build a platform on top of Flyte? As such, we explore topics like:

• Extension via both “custom task types” or backend extensions;
• Constructing and submitting workflows, task instances, etc. dynamically, i.e. without using flytectl.

Despite its task-oriented structure and approach, this repo is not a koans-style tutorial for Flyte. The focus is solidly not on explaining Flyte concepts and mechanisms in and of themselves; any explanation is exclusively in service of explaining/exploring the task/approach at hand.

Getting Started

• Clone this repo
• Create a new virtualenv (I recommend using pyenv and pyenv-virtualenv);
• poetry install

This repo revolves around use of a Flyte sandbox.

First, install flytectl:

brew install flyteorg/homebrew-tap/flytectl

Then, start the sandbox via:

flytectl sandbox start --source .

[0/2] Tasks

Creating, submitting, and executing workflow entirely via code

Goal

Come up with general recipe for end-to-end flow that entirely circumvents flytectl for packaging, publication, etc.; how much of this can we accomplish in Python code?

Stretch the imperative workflow API to its limits, in other words.

[0/2] Extensions

Backend extensions

Goal

Write a backend plugin for Flyte; take advantage of it in a pipeline to do something not natively supported in “vanilla Flyte” – maybe gang scheduling (potentially relevant for distributed training use cases e.g. via Horovod)?

Task plugin

Goal

Write a Flytekit task plugin