Cloud Pipelines

Cloud Pipelines project helps users build and run Machine Learning pipelines.

The Cloud Pipelines ecosystem consists of multiple projects that all work together:

  • Tangle. A modern backend-based web app for building and running pipelines.
  • Pipeline Editor (legacy). The original browser-only web app for building and running pipelines.
  • SDK. The Python SDK for creating and debugging pipeline components and pipelines.
  • ComponentSpec/component.yaml schema. The formal definition of the pipeline component format.

Tangle

Tangle is a web app that allows the users to build and run Machine Learning pipelines using drag and drop without having to set up development environment.

Unlike the older Pipeline Editor app that was a client-side browser-only backendless app, the Tangle app relies on a native Cloud Pipelines backend which allows the app to be more feature-rich.

image

Demo

Try the live demo of the Tangle app. No registration is required to experiment with building pipelines. To install your own app instance and execute your pipelines, follow the backend installation instructions.

The app is under active development. Please check it out and report any bugs you find using GitHub Issues.

App features

  • Start building pipelines right away
    • Intuitive visual drag and drop interface
    • No registration required to build. You own your data.
  • Execute pipelines on your local machine or in Cloud
    • Easily install the app on local machine or deploy to cloud
    • Submit pipelines for execution with a single click.
    • Easily monitor all pipeline task executions, view the artifacts, read the logs.
  • Fast iteration
    • Clone any pipeline run and get a new editable pipeline
    • Create pipeline -> Submit run -> Monitor run -> Clone run -> Edit pipeline -> Submit run ...
  • Automatic execution caching and reuse
    • Save time and compute. Don't re-do what's done
    • Successful and even running executions are re-used from cache
  • Reproducibility
    • All your runs are kept forever (on your machine) - graph, logs, metadata
    • Re-run an old pipeline run with just two clicks (Clone pipeline, Submit run)
    • Containers and strict component versioning ensure reproducibility
  • Pipeline Components
    • Time-proven ComponentSpec/component.yaml format
    • A library of preloaded components
    • Fast-growing public component ecosystem
    • Add your own components (public or private)
    • Easy to create your own components manually or using the Cloud Pipelines SDK
    • Components can be written in any language (Python, Shell, R, Java, C#, etc).
    • Compatible with Google Cloud Vertex AI Pipelines and Kubeflow Pipelines
    • Lots of pre-built components on GitHub: Ark-kun/pipeline_components.

App demo (Pipeline building only. To execute pipelines, install the app locally or in Cloud.).

Installation instructions. Run the app and execute your pipelines locally or in Cloud.

Report bugs and request features

Source code: Backend, Frontend, []

Pipeline Editor

Pipeline Editor is a web app that allows the users to build and run Machine Learning pipelines using drag and drop without having to set up development environment.

Video

See the Pipeline Editor in action

Cloud Pipelines - Build machine learning pipelines without writing code

App

Try the Pipeline Editor now. No registration required. image

App features

  • Build pipelines using drag and drop
  • Execute pipelines in the cloud
  • Start building right away
    • No registration required
    • You own your data
  • Pipeline Components
    • Time-proven ComponentSpec/component.yaml format
    • A library of preloaded components
    • Fast-growing public component ecosystem
    • Add your own components (public or private)
    • Easy to create your own components manually or using the Cloud Pipelines SDK
    • Components can be written in any language (Python, Shell, R, Java, C#, etc).
    • Compatible with Google Cloud Vertex AI Pipelines and Kubeflow Pipelines
    • Lots of pre-built components on GitHub: Ark-kun/pipeline_components.
  • Pipelines
    • Create, save, import and export
    • Submit for execution with a single click

Report bugs and request features

Additional information

Contact

Privacy policy