prefect

https://github.com/PrefectHQ/prefect

https://docs.prefect.io/

Prefect

We've rebuilt data engineering for the data science era.

Prefect is a new workflow management system, designed for modern infrastructure and powered by the open-source Prefect Core workflow engine. Users organize Tasks into Flows

, and Prefect takes care of the rest.

Demo code

Prefect's Pythonic API should feel familiar for newcomers. Mark functions as tasks and call them on each other to build up a flow.

from prefect import task, Flow, Parameter


@task(log_stdout=True)
def say_hello(name):
    print("Hello, {}!".format(name))


with Flow(
 
"My First Flow") as flow:
    name = Parameter('name')
    say_hello(name)


flow.run(name='world') # "Hello, world!"
flow.run(name='Marvin') # "Hello, Marvin!"

安裝

https://docs.prefect.io/core/getting_started/installation.html

https://docs.docker.com/compose/install/

釋出

https://docs.prefect.io/orchestration/concepts/flows.html#registration

https://docs.prefect.io/api/latest/cli/create.html

prefect create project "my project"

prefect register --project "my project" --path test.py