​
Prefect Cloud and proxies

Proxies handle network requests between a server and a client.

To communicate with Prefect Cloud, the Prefect client library makes HTTPS requests. These requests are made with the httpx Python library. httpx respects accepted proxy environment variables, so the Prefect client can communicate through proxies.

To enable communication through proxies, set the HTTPS_PROXY and SSL_CERT_FILE environment variables in your execution environment.

See the Using Prefect Cloud with proxies GitHub Discussion for more information.

You should whitelist the URLs for the UI, API, Authentication, and the current OCSP server for outbound-communication in a secure environment:

  • app.prefect.cloud
  • api.prefect.cloud
  • auth.workos.com
  • api.github.com
  • github.com
  • ocsp.pki.goog/s/gts1d4/OxYEb8XcYmo

​
Prefect Cloud access through the API

If the Prefect Cloud API key, environment variable settings, or account login for your execution environment are not configured correctly, you may experience errors or unexpected flow run results when using Prefect CLI commands, running flows, or observing flow run results in Prefect Cloud.

Use the prefect config view CLI command to make sure your execution environment is correctly configured to access Prefect Cloud:

$ prefect config view
PREFECT_PROFILE='cloud'
PREFECT_API_KEY='pnu_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' (from profile)
PREFECT_API_URL='https://api.prefect.cloud/api/accounts/...' (from profile)

Make sure PREFECT_API_URL is configured to use https://api.prefect.cloud/api/....

Make sure PREFECT_API_KEY is configured to use a valid API key.

You can use the prefect cloud workspace ls CLI command to view or set the active workspace.

prefect cloud workspace ls

┏━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃   Available Workspaces: ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━┩
│   g-gadflow/g-workspace │
│    * prefect/workinonit │
━━━━━━━━━━━━━━━━━━━━━━━━━━━
    * active workspace

You can also check that the account and workspace IDs specified in the URL for PREFECT_API_URL match those shown in the URL for your Prefect Cloud workspace.

​
Prefect Cloud login errors

If you’re having difficulty logging in to Prefect Cloud, the following troubleshooting steps may resolve the issue, or will provide more information when sharing your case to the support channel.

  • Are you logging into Prefect Cloud 3? Prefect Cloud 1, Prefect Cloud 2, and Prefect Cloud 3 use separate accounts. Make sure to use the right Prefect Cloud 3 URL: <https://app.prefect.cloud/>
  • Do you already have a Prefect Cloud account? If you’re having difficulty accepting an invitation, try creating an account first using the email associated with the invitation, then accept the invitation.
  • Are you using a single sign-on (SSO) provider, social authentication (Google, Microsoft, or GitHub) or just using an emailed link?

Other tips to help with login difficulties:

  • Hard refresh your browser with Cmd+Shift+R.
  • Try in a different browser. We actively test against the following browsers:
    • Chrome
    • Edge
    • Firefox
    • Safari
  • Clear recent browser history/cookies

None of this worked?

Email us at <help@prefect.io> and provide answers to the questions above in your email to make it faster to troubleshoot and unblock you. Make sure you add the email address you used to attempt to log in, your Prefect Cloud account name, and, if applicable, the organization it belongs to.

​
General troubleshooting

The first troubleshooting step is to confirm that you are running the latest version of Prefect. If you are not, upgrade (see below) to the latest version, since the issue may have already been fixed. Beyond that, there are several categories of errors:

  • The issue may be in your flow code, in which case you should carefully read the logs.
  • The issue could be with how you are authenticated, and whether or not you are connected to Cloud.
  • The issue might have to do with how your code is executed.

​
Upgrade

Prefect is constantly evolving by adding new features and fixing bugs. A patch may have already been identified and released. Search existing issues for similar reports and check out the Release Notes.

Upgrade to the newest version with the following command:

pip install --upgrade prefect

Different components may use different versions of Prefect:

  • Cloud is generally the newest version. Cloud is continuously deployed by the Prefect team. When using a self-hosted server, you can control this version.
  • Workers change versions infrequently, and are usually the latest version at the time of creation. Workers provision infrastructure for flow runs, so upgrading them may help with infrastructure problems.
  • Flows could use a different version than the worker that created them, especially when running in different environments. Suppose your worker and flow both use the latest official Docker image, but your worker was created a month ago. Your worker is often an older version than your flow.

Integration Versions

Integrations are versioned and released independently of the core Prefect library. They should be upgraded simultaneously with the core library, using the same method.

​
Logs

In many cases, there is an informative stack trace in Prefect’s logs. Read it carefully, locate the source of the error, and try to identify the cause.

There are two types of logs:

  • Flow and task logs are always scoped to a flow. They are sent to Prefect and are viewable in the UI.
  • Worker logs are not scoped to a flow and may have more information on what happened before the flow started. These logs are generally only available where the worker is running.

If your flow and task logs are empty, there may have been an infrastructure issue that prevented your flow from starting. Check your worker logs for more details.

If there is no clear indication of what went wrong, try updating the logging level from the default INFO level to the DEBUG level. Settings such as the logging level are propagated from the worker environment to the flow run environment. Set them with environment variables or the prefect config set CLI:

# Using the CLI
prefect config set PREFECT_LOGGING_LEVEL=DEBUG

# Using environment variables
export PREFECT_LOGGING_LEVEL=DEBUG

The DEBUG logging level produces a high volume of logs, so consider setting it back to INFO once any issues are resolved.

​
Cloud

When using Prefect Cloud, there are the additional concerns of authentication and authorization. The Prefect API authenticates users and service accounts, collectively known as actors, with API keys. Missing, incorrect, or expired API keys result in a 401 response with detail Invalid authentication credentials. Use the following command to check your authentication, replacing $PREFECT_API_KEY with your API key:

curl -s -H "Authorization: Bearer $PREFECT_API_KEY" "https://api.prefect.cloud/api/me/"

Users vs Service Accounts

Service accounts (sometimes referred to as bots),
represent non-human actors that interact with Prefect such as workers and CI/CD systems. Each human that interacts with Prefect should be represented as a user. User API keys start with pnu_ and service account API keys start with pnb_.

Actors can be members of workspaces. An actor attempting an action in a workspace they are not a member of results in a 404 response. Use the following command to check your actor’s workspace memberships:

curl -s -H "Authorization: Bearer $PREFECT_API_KEY" "https://api.prefect.cloud/api/me/workspaces"

Formatting JSON

Python comes with a helpful tool for formatting JSON. Append the following to the end of the command above to make the output more readable: | python -m json.tool

Make sure your actor is a member of the workspace you are working in. Within a workspace, an actor has a role which grants them certain permissions. Insufficient permissions result in an error. For example, starting a worker with the Viewer role results in errors.

​
Execution

The user can execute flows locally, or remotely by a worker. Local execution generally means that you, the user, run your flow directly with a command like python flow.py. Remote execution generally means that a worker runs your flow through a deployment (optionally on different infrastructure).

With remote execution, the creation of your flow run happens separately from its execution. Flow runs are assigned to a work pool and a work queue. For flow runs to execute, a worker must be subscribed to the work pool and work queue, or the flow runs will go from Scheduled to Late. Ensure that your work pool and work queue have a subscribed worker.

Local and remote execution can also differ in their treatment of relative imports. If switching from local to remote execution results in local import errors, try replicating the behavior by executing the flow locally with the -m flag (For example, python -m flow instead of python flow.py). Read more about -m in this Stack Overflow post.

​
API tests return an unexpected 307 Redirected

Summary: requests require a trailing / in the request URL.

If you write a test that does not include a trailing / when making a request to a specific endpoint:

async def test_example(client):
    response = await client.post("/my_route")
    assert response.status_code == 201

You’ll see a failure like:

E       assert 307 == 201
E        +  where 307 = <Response [307 Temporary Redirect]>.status_code

To resolve this, include the trailing /:

async def test_example(client):
    response = await client.post("/my_route/")
    assert response.status_code == 201

Note: requests to nested URLs may exhibit the opposite behavior and require no trailing slash:

async def test_nested_example(client):
    response = await client.post("/my_route/filter/")
    assert response.status_code == 307

    response = await client.post("/my_route/filter")
    assert response.status_code == 200

Reference: “HTTPX disabled redirect following by default” in 0.22.0.

​
pytest.PytestUnraisableExceptionWarning or ResourceWarning

As you’re working with one of the FlowRunner implementations, you may get an error like this:

E               pytest.PytestUnraisableExceptionWarning: Exception ignored in: <ssl.SSLSocket fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0>
E
E               Traceback (most recent call last):
E                 File ".../pytest_asyncio/plugin.py", line 306, in setup
E                   res = await func(**_add_kwargs(func, kwargs, event_loop, request))
E               ResourceWarning: unclosed <ssl.SSLSocket fd=10, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 60605), raddr=('127.0.0.1', 6443)>

.../_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning

This error states that your test suite (or the prefect library code) opened a connection to something (like a Docker daemon or a Kubernetes cluster) and didn’t close it.

It may help to re-run the specific test with PYTHONTRACEMALLOC=25 pytest ... so that Python can display more of the stack trace where the connection was opened.