# AWS Tasks


This module contains a collection of tasks for interacting with AWS resources.

All AWS related tasks can be authenticated using the AWS_CREDENTIALS Prefect Secret that should be a dictionary with two keys: "ACCESS_KEY" and "SECRET_ACCESS_KEY". See Third Party Authentication for more information.

# S3Download

class

prefect.tasks.aws.s3.S3Download

(bucket=None, boto_kwargs=None, **kwargs)[source]

Task for downloading data from an S3 bucket and returning it as a string. Note that all initialization arguments can optionally be provided or overwritten at runtime.

For authentication, there are two options: you can set the AWS_CREDENTIALS Prefect Secret containing your AWS access keys which will be passed directly to the boto3 client, or you can configure your flow's runtime environment for boto3.

Args:

  • bucket (str, optional): the name of the S3 Bucket to download from
  • boto_kwargs (dict, optional): additional keyword arguments to forward to the boto client.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.aws.s3.S3Download.run

(key, credentials=None, bucket=None, compression=None, as_bytes=False)[source]

Task run method.

Args:

  • key (str): the name of the Key within this bucket to retrieve
  • credentials (dict, optional): your AWS credentials passed from an upstream Secret task; this Secret must be a JSON string with two keys: ACCESS_KEY and SECRET_ACCESS_KEY which will be passed directly to boto3. If not provided here or in context, boto3 will fall back on standard AWS rules for authentication.
  • bucket (str, optional): the name of the S3 Bucket to download from
  • compression (str, optional): specifies a file format for decompression, decompressing data upon download. Currently supports 'gzip'.
  • as_bytes (bool, optional): If true, result will be returned as bytes instead of str. Defaults to False.
Returns:
  • str: the contents of this Key / Bucket, as a string or bytes



# S3Upload

class

prefect.tasks.aws.s3.S3Upload

(bucket=None, boto_kwargs=None, **kwargs)[source]

Task for uploading string data (e.g., a JSON string) to an S3 bucket. Note that all initialization arguments can optionally be provided or overwritten at runtime.

For authentication, there are two options: you can set a Prefect Secret containing your AWS access keys which will be passed directly to the boto3 client, or you can configure your flow's runtime environment for boto3.

Args:

  • bucket (str, optional): the name of the S3 Bucket to upload to
  • boto_kwargs (dict, optional): additional keyword arguments to forward to the boto client.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.aws.s3.S3Upload.run

(data, key=None, credentials=None, bucket=None, compression=None)[source]

Task run method.

Args:

  • data (str): the data payload to upload
  • key (str, optional): the Key to upload the data under; if not provided, a random uuid will be created
  • credentials (dict, optional): your AWS credentials passed from an upstream Secret task; this Secret must be a JSON string with two keys: ACCESS_KEY and SECRET_ACCESS_KEY which will be passed directly to boto3. If not provided here or in context, boto3 will fall back on standard AWS rules for authentication.
  • bucket (str, optional): the name of the S3 Bucket to upload to
  • compression (str, optional): specifies a file format for compression, compressing data before upload. Currently supports 'gzip'.
Returns:
  • str: the name of the Key the data payload was uploaded to



# S3List

class

prefect.tasks.aws.s3.S3List

(bucket=None, **kwargs)[source]

Task for listing files from an S3 bucket. Note that all initialization arguments can optionally be provided or overwritten at runtime.

For authentication, there are two options: you can set the AWS_CREDENTIALS Prefect Secret containing your AWS access keys which will be passed directly to the boto3 client, or you can configure your flow's runtime environment for boto3.

Args:

  • bucket (str, optional): the name of the S3 Bucket to list the files of.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.aws.s3.S3List.run

(prefix, delimiter="", page_size=None, max_items=None, credentials=None, bucket=None, last_modified_begin=None, last_modified_end=None)[source]

Task run method.

Args:

  • prefix (str): the name of the prefix within this bucket to retrieve objects from
  • delimiter (str): indicates the key hierarchy
  • page_size (int): controls the number of items returned per page of each result
  • max_items (int): limits the maximum number of total items returned during pagination
  • credentials (dict, optional): your AWS credentials passed from an upstream Secret task; this Secret must be a JSON string with two keys: ACCESS_KEY and SECRET_ACCESS_KEY which will be passed directly to boto3. If not provided here or in context, boto3 will fall back on standard AWS rules for authentication.
  • bucket (str, optional): the name of the S3 Bucket to list the files of
  • last_modified_begin (str, optional): keep items with LastModified greater than or equal to given value. timestamp should be in RFC 3339, ISO 8601 or any pendulum supported format.
  • last_modified_end (str, optional): keep items with LastModified less than or equal to given value. timestamp should be in RFC 3339, ISO 8601 or any pendulum supported format.
Returns:
  • list[str]: A list of keys that match the given prefix.



# LambdaCreate

class

prefect.tasks.aws.lambda_function.LambdaCreate

(function_name, runtime, role, handler, zip_file=None, bucket="", bucket_key="", object_version=None, description="", function_timeout=3, memorysize=128, publish=True, subnet_ids=None, security_group_ids=None, dead_letter_config=None, environment_variables=None, kms_key_arn="", function_tags=None, tracing_config="PassThrough", layers=None, boto_kwargs=None, **kwargs)[source]

Task for creating a Lambda function.

Args:

  • function_name (str): name of the Lambda function to create
  • runtime (str): the identifier of the function's runtime
  • role (str): the Amazon Resource Name of the function's execution role
  • handler (str): the name of the method within your code that Lambda calls to execute your function
  • zip_file (str): path to zip file containing code for Lambda function, either zip_file or (bucket and bucket_key) must be passed
  • bucket (str): an S3 bucket in the same AWS region as your function
  • bucket_key (str): the Amazon S3 key of the deployment package
  • object_version (str, optional): for versioned S3 objects, the version of the deployment package to use
  • description (str, optional): description of Lambda function
  • function_timeout (int, optional): Lambda function timeout in seconds, default is 3 seconds
  • memorysize (int, optional): amount of memory that Lambda function has access to in MB, must be a multiple of 64 MB, default is 128
  • publish (bool, optional): set to True to publish the first version of the function during creation, defaults to True
  • subnet_ids (List[str], optional): list of subnet ids for vpc configuration
  • security_group_ids (List[str], optional): list of security group ideas for vpc configuration
  • dead_letter_config (dict, optional): a dead letter queue configuration that specifies the queue or topic where Lambda sends asynchronous events when they fail processing
  • environment_variables (dict, optional): key-value pairs of environment variables to pass to the Lambda function
  • kms_key_arn (str, optional): the ARN of the AWS key management service used to encrypt your function's environment variables, if not provided, AWS Lambda uses a default service key
  • function_tags (dict, optional): a list of tags to apply to the function, string to string map
  • tracing_config (str, optional): set to Active to samle and trace a subset of incoming requests with Amazon X-Ray
  • layers (List[str], optional): a list of function layers to add to the function's execution environment, specify each layer by its ARN
  • boto_kwargs (dict, optional): additional keyword arguments to forward to the boto client.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.aws.lambda_function.LambdaCreate.run

(credentials=None)[source]

Task run method. Creates Lambda function.

Args:

  • credentials (dict, optional): your AWS credentials passed from an upstream Secret task; this Secret must be a JSON string with two keys: ACCESS_KEY and SECRET_ACCESS_KEY which will be passed directly to boto3. If not provided here or in context, boto3 will fall back on standard AWS rules for authentication.
Returns:
  • json: response from AWS CreateFunction endpoint



# LambdaDelete

class

prefect.tasks.aws.lambda_function.LambdaDelete

(function_name, qualifier="", boto_kwargs=None, **kwargs)[source]

Task for deleting a Lambda function.

Args:

  • function_name (str): name of the Lambda function to delete
  • qualifier (str, optional): specify a version to delete, if not provided, the function will be deleted entirely
  • boto_kwargs (dict, optional): additional keyword arguments to forward to the boto client.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.aws.lambda_function.LambdaDelete.run

(credentials=None)[source]

Task run method. Deletes Lambda function.

Args:

  • credentials (dict, optional): your AWS credentials passed from an upstream Secret task; this Secret must be a JSON string with two keys: ACCESS_KEY and SECRET_ACCESS_KEY which will be passed directly to boto3. If not provided here or in context, boto3 will fall back on standard AWS rules for authentication.
Returns:
  • dict: response from AWS DeleteFunction endpoint



# LambdaInvoke

class

prefect.tasks.aws.lambda_function.LambdaInvoke

(function_name, invocation_type="RequestResponse", log_type="None", client_context=None, payload="null", qualifier="$LATEST", boto_kwargs=None, **kwargs)[source]

Task to invoke a Lambda function.

Args:

  • function_name (str): the name of the Lambda funciton to invoke
  • invocation_type (str, optional): the invocation type of Lambda function, default is RequestResponse other options include Event and DryRun
  • log_type (str, optional): set to 'Tail' to include the execution log in the response
  • client_context (dict, optional): data to pass to the function in the context object, dict object will be transformed into base64 encoded json automatically
  • payload (bytes or seekable file-like object): the JSON provided to Lambda function as input
  • qualifier (str, optional): specify a version or alias to invoke a published version of the function, defaults to $LATEST
  • boto_kwargs (dict, optional): additional keyword arguments to forward to the boto client.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.aws.lambda_function.LambdaInvoke.run

(function_name=None, payload=None, credentials=None)[source]

Task run method. Invokes Lambda function.

Args:

  • function_name (str): the name of the Lambda funciton to invoke
  • payload (bytes or seekable file-like object): the JSON provided to Lambda function as input
  • credentials (dict, optional): your AWS credentials passed from an upstream Secret task; this Secret must be a JSON string with two keys: ACCESS_KEY and SECRET_ACCESS_KEY which will be passed directly to boto3. If not provided here or in context, boto3 will fall back on standard AWS rules for authentication.
Returns:
  • dict: response from AWS Invoke endpoint



# LambdaList

class

prefect.tasks.aws.lambda_function.LambdaList

(master_region="ALL", function_version="ALL", marker=None, max_items=50, boto_kwargs=None, **kwargs)[source]

Task to list Lambda functions.

Args:

  • master_region (str, optional): for Lambda@Edge functions, the AWS region of the master function
  • function_version (str, optional): the version of a function, default is 'ALL'
  • marker (str, optional): specify the pagination token that's returned by a previous request to retreive the next page of results
  • max_items (int, optional): specify a value between 1 and 50 to limit the number of functions in the response
  • boto_kwargs (dict, optional): additional keyword arguments to forward to the boto client.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.aws.lambda_function.LambdaList.run

(credentials=None)[source]

Task fun method. Lists all Lambda functions.

Args:

  • credentials (dict, optional): your AWS credentials passed from an upstream Secret task; this Secret must be a JSON string with two keys: ACCESS_KEY and SECRET_ACCESS_KEY which will be passed directly to boto3. If not provided here or in context, boto3 will fall back on standard AWS rules for authentication.
Returns:
  • dict: a list of Lambda functions from AWS ListFunctions endpoint



# StepActivate

class

prefect.tasks.aws.step_function.StepActivate

(state_machine_arn=None, execution_name=None, execution_input="{}", boto_kwargs=None, **kwargs)[source]

Task for activating an AWS Step Function.

Args:

  • state_machine_arn (str): the Amazon Resource Name (ARN) of the state machine to execute
  • execution_name (str): the name of the execution, this name must be unique for your AWS account, region, and state machine for 90 days
  • execution_input (str, optional, DEPRECATED): string that contains the JSON input data for the execution, defaults to "{}". Setting execution_input through the constructor is deprecated. Set it when running this task instead.
  • boto_kwargs (dict, optional): additional keyword arguments to forward to the boto client.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.aws.step_function.StepActivate.run

(credentials=None, state_machine_arn=None, execution_name=None, execution_input=None, boto_kwargs=None)[source]

Task run method. Activates AWS Step function.

Args:

  • credentials (dict, optional): your AWS credentials passed from an upstream Secret task; this Secret must be a JSON string with two keys: ACCESS_KEY and SECRET_ACCESS_KEY which will be passed directly to boto3. If not provided here or in context, boto3 will fall back on standard AWS rules for authentication.
  • state_machine_arn (str): the Amazon Resource Name (ARN) of the state machine to execute
  • execution_name (str): the name of the execution, this name must be unique for your AWS account, region, and state machine for 90 days
  • execution_input (str, optional): string that contains the JSON input data for the execution, defaults to "{}"
  • boto_kwargs (dict, optional): additional keyword arguments to forward to the boto client.
Returns:
  • dict: response from AWS StartExecution endpoint



# AWSSecretsManager

class

prefect.tasks.aws.secrets_manager.AWSSecretsManager

(secret=None, boto_kwargs=None, **kwargs)[source]

Task for retrieving secrets from an AWS Secrets Manager and returning it as a dictionary. Note that all initialization arguments can optionally be provided or overwritten at runtime.

For authentication, there are two options: you can set the AWS_CREDENTIALS Prefect Secret containing your AWS access keys which will be passed directly to the boto3 client, or you can configure your flow's runtime environment for boto3.

Args:

  • secret (str, optional): The name of the secret to retrieve.
  • boto_kwargs (dict, optional): Additional keyword arguments to forward to the boto client.
  • **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor.

methods:                                                                                                                                                       

prefect.tasks.aws.secrets_manager.AWSSecretsManager.run

(secret=None, credentials=None)[source]

Task run method.

Args:

  • secret (str): The name of the secret to retrieve.
  • credentials (dict, optional): Your AWS credentials passed from an upstream Secret task; this Secret must be a JSON string with two keys: ACCESS_KEY and SECRET_ACCESS_KEY which will be passed directly to boto3. If not provided here or in context, boto3 will fall back on standard AWS rules for authentication.
Returns:
  • Union[dict, str]: The contents of this secret. Either as a dictionary or as a string.



# AWSParametersManager

class

prefect.tasks.aws.parameter_store_manager.AWSParametersManager

(parameter_name=None, boto_kwargs=None, **kwargs)[source]

Task for retrieving values from AWS SSM Parameters Store and returning the parameter value. Note that all initialization arguments can optionally be provided or overwritten at runtime. For authentication, there are two options: you can set the AWS_CREDENTIALS Prefect Secret containing your AWS access keys, which will be passed directly to the boto3 client, or you can configure your flow's runtime environment for boto3.

Args:

  • parameter_name (str, optional): The name of the parameter to retrieve via SSM.
  • boto_kwargs (dict, optional): Additional keyword arguments to forward to the boto client.
  • **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor.

methods:                                                                                                                                                       

prefect.tasks.aws.parameter_store_manager.AWSParametersManager.run

(parameter_name=None, credentials=None, boto_kwargs=None)[source]

Task run method.

Args:

  • parameter_name (str): The name of the parameter to retrieve from SSM.
  • credentials (dict, optional): Your AWS credentials passed from an upstream Secret task; this Secret must be a JSON string with two keys: ACCESS_KEY and SECRET_ACCESS_KEY which will be passed directly to boto3. If not provided here or in context, boto3 will fall back on standard AWS rules for authentication.
  • boto_kwargs (dict, optional): Additional keyword arguments to forward to the boto client.
Returns:
  • str: The parameter value, as a string.



# BatchSubmit

class

prefect.tasks.aws.batch.BatchSubmit

(job_name=None, job_definition=None, job_queue=None, boto_kwargs=None, **kwargs)[source]

Task for submitting a job to AWS batch.

For authentication, there are two options: you can set the AWS_CREDENTIALS Prefect Secret containing your AWS access keys which will be passed directly to the boto3 client, or you can configure your flow's runtime environment for boto3.

Args:

  • job_name (str, optional): The AWS batch job name.
  • job_definition (str, optional): The AWS batch job definition.
  • job_queue (str, optional): Name of the AWS batch job queue.
  • boto_kwargs (dict, optional): additional kekyword arguments to forward to the boto client.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.aws.batch.BatchSubmit.run

(job_name=None, job_definition=None, job_queue=None, batch_kwargs=None, credentials=None)[source]

Submit a job to the AWS Batch job service.

Args:

  • job_name (str, optional): The AWS batch job name.
  • job_definition (str, optional): The AWS batch job definition.
  • job_queue (str, optional): Name of the AWS batch job queue.
  • batch_kwargs (dict, optional): Additional keyword arguments to pass to the boto3 submit_job function. See the submit_job # noqa documentation for more details.
  • credentials (dict, optional): your AWS credentials passed from an upstream Secret task; this Secret must be a JSON string with two keys: ACCESS_KEY and SECRET_ACCESS_KEY which will be passed directly to boto3. If not provided here or in context, boto3 will fall back on standard AWS rules for authentication.



# AWSClientWait

class

prefect.tasks.aws.client_waiter.AWSClientWait

(client=None, waiter_name=None, waiter_definition=None, boto_kwargs=None, **kwargs)[source]

Task for waiting on a long-running AWS job. Uses the underlying boto3 waiter functionality.

For authentication, there are two options: you can set the AWS_CREDENTIALS Prefect Secret containing your AWS access keys which will be passed directly to the boto3 client, or you can configure your flow's runtime environment for boto3.

Args:

  • client (str, optional): The AWS client on which to wait (e.g., 'batch', 'ec2', etc)
  • waiter_name (str, optional): The name of the waiter to instantiate. Can be a boto-supported waiter or one of prefect's custom waiters. Currently, prefect offers three additional waiters for AWS Batch: "JobExists" waits for a job to be instantiated, "JobRunning" waits for a job to start running, and "JobComplete" waits for a job to finish. You can find the definitions for all prefect-defined waiters here. # noqa You may also use a custom waiter name, if you supply an accompanying waiter definition dict.
  • waiter_definition (dict, optional): A valid custom waiter model, as a dict. Note that if you supply a custom definition, it is assumed that the provided 'waiter_name' is contained within the waiter definition dict.
  • boto_kwargs (dict, optional): additional kekyword arguments to forward to the boto client.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.aws.client_waiter.AWSClientWait.run

(client=None, waiter_name=None, waiter_definition=None, waiter_kwargs=None, credentials=None)[source]

Task for waiting on a long-running AWS job. Uses the underlying boto3 waiter functionality.

Args:

  • client (str): The AWS client on which to wait (e.g., 'batch', 'ec2', etc)
  • waiter_name (str, optional): The name of the waiter to instantiate. Can be a boto-supported waiter or one of prefect's custom waiters. Currently, prefect offers three additional waiters for AWS Batch: "JobExists" waits for a job to be instantiated, "JobRunning" waits for a job to start running, and "JobComplete" waits for a job to finish. You can find the definitions for all prefect-defined waiters here. # noqa You may also use a custom waiter name, if you supply an accompanying waiter definition dict.
  • waiter_definition (dict, optional): A valid custom waiter model, as a dict. Note that if you supply a custom definition, it is assumed that the provided 'waiter_name' is contained within the waiter definition dict.
  • waiter_kwargs (dict, optional): Arguments to pass to the waiter.wait(...) method. Will depend upon the specific waiter being called.
  • credentials (dict, optional): your AWS credentials passed from an upstream Secret task; this Secret must be a JSON string with two keys: ACCESS_KEY and SECRET_ACCESS_KEY which will be passed directly to boto3. If not provided here or in context, boto3 will fall back on standard AWS rules for authentication.



This documentation was auto-generated from commit ad73429
on May 24, 2022 at 22:55 UTC