Since Fugue is a framework for distributed compute, it is often paired with a solution that manages Spark, Dask, or Ray clusters. This section will cover using Fugue on top of cloud providers such as Databricks or Coiled. Fugue has a
fugue-cloudprovider package that allows users to easily spin up ephemeral compute for their compute workflows.
Have questions? Chat with us on Github or Slack:
Databricks is the most common provider for Spark clusters. Using the
databricks-connect library, we can easily spin up an ephemeral Spark cluster. We can connect to the SparkSession on the Databricks cluster from a local machine.