Getting Started with Prefect Cloud¶
The following sections will get you set up and using Prefect Cloud, using these steps:
- Sign in or register a Prefect Cloud account.
- Create workspaces for your account.
- Create an API key to authorize a local execution environment.
- Configure Orion settings to use Prefect Cloud.
- Configure storage.
- Run a flow and view flow results in Prefect Cloud.
Sign in or register¶
To sign in with an existing account or register an account, go to https://app.prefect.cloud/.
You can create an account with:
- Google account
- GitHub account
- Email and password
Create a workspace¶
A workspace is an isolated environment within Prefect Cloud for your flows and deployments. You can use workspaces to organize or compartmentalize your workflows. For example, you can create separate workspaces to isolate development, staging, and production environments; or to provide separation between different teams.
When you register a new account, you'll be prompted to create a workspace.
Select Create Workspace. You'll be prompted to provide a name and description for your workspace.
Select Save to create the workspace.
If you change your mind, you can select Workspace Settings to modify the workspace details or to delete it.
Workspace Settings also shows you the
prefect cloud workspace set Prefect CLI command you can use to sync a local execution environment with the workspace.
$ prefect cloud workspace set --workspace "prefect/workinonit"
Deleting a workspace
Deleting a workspace removes any flows, deployments, and storage created on that workspace.
Create an API key¶
API keys enable you to authenticate an a local environment to work with Prefect Cloud. See Configure execution environment for details on how API keys are configured in your execution environment.
To create an API key, select the account icon at the bottom-left corner of the UI and select Settings. This displays your account profile.
Select the API Keys tab. This displays a list of previously generated keys and lets you create new API keys or delete keys.
Select the + button to create a new API key. You're prompted to provide a name for the key and, optionally, an expiration date. Select Create API Key to generate the key.
Note that API keys cannot be revealed again in the UI after you generate them, so copy the key to a secure location.
Configure execution environment¶
Now configure a local execution environment to use Prefect Cloud as the API server for flow runs.
First, Install Prefect in the environment in which you want to execute flow runs.
Next, use the Prefect CLI
prefect cloud login command to log into Prefect Cloud from your environment, using the API key generated previously.
$ prefect cloud login --key xxx_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
If this is your first time logging in with this API key, you will be prompted to make a new profile as well as select a workspace (you can specify a workspace with the
$ prefect cloud login --key xxx_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX ┏━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Select a Workspace: ┃ ┡━━━━━━━━━━━━━━━━━━━━━━┩ │ > prefect/workinonit │ └──────────────────────┘ Creating a profile for this Prefect Cloud login. Please specify a profile name: my-cloud-profile Logged in to Prefect Cloud using profile 'my-cloud-profile'. Workspace is currently set to 'prefect/workinonit'. The workspace can be changed using `prefect cloud workspace set`.
The command sets
PREFECT_API_URL for the new profile.
Now you're ready to run flows locally and have the results displayed in the Prefect Cloud UI.
You can log out of Prefect Cloud by switching to a different profile.
If you need to change which workspace you're syncing with, use the
prefect cloud workspace set Prefect CLI command while logged in, passing the the account handle and workspace name.
$ prefect cloud workspace set --workspace "prefect/workinonit"
If no workspace is provided, you will be prompted to select one.
Manually configuring Cloud settings¶
Note that you can also manually configure the
PREFECT_API_KEY settings to interact with Prefect Cloud by using an account ID, workspace ID, and API key.
$ prefect config set PREFECT_API_URL="https://api.prefect.cloud/api/accounts/[ACCOUNT-ID]/workspaces/[WORKSPACE-ID]" $ prefect config set PREFECT_API_KEY="[API-KEY]"
When you're in a Prefect Cloud workspace, you can copy the
PREFECT_API_URL value directly from the page URL.
In this example, we configured
PREFECT_API_KEY in the default profile. You can use
prefect profile CLI commands to create settings profiles for different configurations. For example, you could have a "cloud" profile configured to use the Prefect Cloud API URL and API key, and another "local" profile for local development using a local Prefect API server started with
prefect orion start. See Settings for details.
When using Prefect Cloud, we recommend configuring remote storage for persisting flow and task data. See Storage for details.
By default, Prefect uses local file system storage to persist flow code and flow and task results. For local development and testing this may be adequate. Be aware, however, that local storage is not guaranteed to persist data reliably between flow or task runs, particularly when using container-based environments such as Docker or Kubernetes, or running tasks with distributed computing tools like Dask and Ray.
Run a flow with Prefect Cloud¶
Okay, you're all set to run a local flow with Prefect Cloud. Notice that everything works just like running local flows with the Prefect API server, but because you configured
PREFECT_API_KEY, your flow runs show up in Prefect Cloud!
In your local environment, where you configured the previous steps, create a file named
basic_flow.py with the following contents:
from prefect import flow, get_run_logger @flow(name="Testing") def basic_flow(): logger = get_run_logger() logger.warning("The fun is about to begin") if __name__ == "__main__": basic_flow()
$ python basic_flow.py 21:40:12.904 | INFO | prefect.engine - Created flow run 'perfect-cat' for flow 'Testing' 21:40:12.904 | INFO | Flow run 'perfect-cat' - Using task runner 'ConcurrentTaskRunner' 21:40:13.176 | WARNING | Flow run 'perfect-cat' - The fun is about to begin 21:40:13.555 | INFO | Flow run 'perfect-cat' - Finished in state Completed()
Go to the dashboard for your workspace in Prefect Cloud. You'll see the flow run results right there in Prefect Cloud!
Prefect Cloud now automatically tracks any flow runs in a local execution environment logged into Prefect Cloud.
Run a flow from a deployment¶
Deployments are flows packaged in a way that let you run them directly from the Prefect Cloud UI, either ad-hoc runs or via a schedule.
To run a flow from a deployment with Prefect Cloud, you'll need to:
- Create a flow script
- Create the deployment using the Prefect CLI
- Configure a work queue that can allocate your deployment's flow runs to agents
- Start an agent in your execution environment
- Run your deployment to create a flow run
Create a deployment¶
Let's go back to your flow code in
basic_flow.py. In a terminal, run the
prefect deployment build Prefect CLI command to build a manifest JSON file and deployment YAML file that you'll use to create the deployment on Prefect Cloud.
$ prefect deployment build ./basic_flow.py:basic_flow -n test-deployment -t test
What did we do here? Let's break down the command:
prefect deployment buildis the Prefect CLI command that enables you to prepare the settings for a deployment.
./basic_flow.py:basic_flowspecifies the location of the flow script file and the name of the entrypoint flow function, separated by a colon.
-n test-deploymentis an option to specify a name for the deployment.
-t testspecifies a tag for the deployment. Tags enable filtering deployment flow runs in the UI and on work queues.
The command outputs two files:
basic_flow-manifest.json contains workflow-specific information such as the code location, the name of the entrypoint flow, and flow parameters.
deployment.yaml contains details about the deployment for this flow.
Create the deployment¶
In the terminal, use the
prefect deployment apply command to apply the settings contained in the manifest and
deployment.yaml to create the deployment on Prefect Cloud.
Run the following Prefect CLI command.
$ prefect deployment apply `deployment.yaml` Successfully loaded 'test-deployment' Deployment '66b3fdea-cd3a-4734-b3f2-65f6702ff260' successfully created.
Now your deployment has been created and is ready to orchestrate future
Testing flow runs.
To demonstrate that your deployment exists, go back to Prefect Cloud and select the Deployments page. You'll see the 'Testing/Test Deployment' deployment was created.
Create a work queue and agent¶
Next, create a work queue that can distribute your new deployment to agents for execution, and a local agent to pick up the flow run for your 'Testing/Test Deployment' deployment.
In Prefect Cloud, you can create a work queue by selecting the Work Queues page, then creating a new work queue.
However, we can also use a Prefect CLI convenience command: starting your agent with a set of tags will create a work queue for you that serves deployments with those tags. This is handy for quickly standing up a test environment, for example.
In your terminal, run the
prefect agent start command, passing a
-t test option that creates a work queue for
$ prefect agent start -t test Starting agent connected to https://api.prefect.cloud/api/accounts/... ___ ___ ___ ___ ___ ___ _____ _ ___ ___ _ _ _____ | _ \ _ \ __| __| __/ __|_ _| /_\ / __| __| \| |_ _| | _/ / _|| _|| _| (__ | | / _ \ (_ | _|| .` | | | |_| |_|_\___|_| |___\___| |_| /_/ \_\___|___|_|\_| |_| Agent started! Looking for work from queue 'Agent queue test'...
PREFECT_API_URL setting for agents
PREFECT_API_URL must be set for the environment in which your agent is running.
In this case, we're running the agent in the same environment we logged into Prefect Cloud earlier. However, if you want the agent to communicate with Prefect Cloud from a remote execution environment such as a VM or Docker container, you must configure
PREFECT_API_URL in that environment.
Run a flow¶
Now create a flow run from your deployment. You'll start the flow run from the Prefect Cloud UI, see it run by the agent in your local execution environment, and then see the result back in Prefect Cloud.
Go back to Prefect Cloud and select the Deployments page, then select Test Deployment in the 'Testing/Test Deployment' deployment name. You'll see a page showing details about the deployment.
To start an ad-hoc flow run, select the Run button from Prefect Cloud.
In the local terminal session where you started the agent, you can see that the agent picks up the flow run and executes it.
$ prefect agent start 'test-queue' Starting agent connected to https://api.prefect.cloud/api/accounts/... ___ ___ ___ ___ ___ ___ _____ _ ___ ___ _ _ _____ | _ \ _ \ __| __| __/ __|_ _| /_\ / __| __| \| |_ _| | _/ / _|| _|| _| (__ | | / _ \ (_ | _|| .` | | | |_| |_|_\___|_| |___\___| |_| /_/ \_\___|___|_|\_| |_| Agent started! Looking for work from queue 'test-queue'... 22:38:06.072 | INFO | prefect.agent - Submitting flow run '7a9e94d4-7a97-4188-a539-b8594874eb86' 22:38:06.169 | INFO | prefect.flow_runner.subprocess - Opening subprocess for flow run '7a9e94d4-7a97-4188-a539-b8594874eb86'... 22:38:06.180 | INFO | prefect.agent - Completed submission of flow run '7a9e94d4-7a97-4188-a539-b8594874eb86' 22:38:09.918 | INFO | Flow run 'crystal-hog' - Using task runner 'ConcurrentTaskRunner' 22:38:10.225 | WARNING | Flow run 'crystal-hog' - The fun is about to begin 22:38:10.868 | INFO | Flow run 'crystal-hog' - Finished in state Completed() 22:38:11.797 | INFO | prefect.flow_runner.subprocess - Subprocess for flow run '7a9e94d4-7a97-4188-a539-b8594874eb86' exited cleanly.
In Prefect Cloud, select the Flow Runs page and notice that your flow run appears on the dashboard. (Your flow run name will be different, but the rest of the details should be similar to what you see here.)
To learn more, see the Deployments tutorial for a hands-on example.