Prefect Docs

# Kafka Tasks

This module contains a collection of tasks to produce and consume Kafka events

# KafkaBatchConsume

class

prefect.tasks.kafka.kafka.KafkaBatchConsume

(bootstrap_servers=None, group_id=None, topics=None, request_timeout=1.0, auto_offset_reset="earliest", message_consume_limit=None, kafka_configs=None, **kwargs)[source]

Task for consuming a batch of messages from Kafka topics.

Args:

  • bootstrap_servers (str, optional): comma separated host and port pairs that are the addresses of kafka brokers.
  • group_id (str, required): name of the consumer group the consumer will belong to. Must be specified either at init or runtime.
  • topics (List[str], required): list of topic names to consume messages from. Must be specified either at init or runtime.
  • request_timeout (float, optional): Maximum time to block waiting for message, event or callback
  • auto_offset_reset (str, optional): configurable offset reset policy
  • message_consume_limit (int, optional): max number of messages to consume before closing the consumer
  • kafka_configs (dict, optional): a dict of kafka client configuration properties used to construct the consumer.
  • **kwargs (Any, optional): additional keyword arguments to pass to the standard Task init method

methods:                                                                                                                                                       

prefect.tasks.kafka.kafka.KafkaBatchConsume.run

(bootstrap_servers=None, group_id=None, topics=None, request_timeout=1.0, auto_offset_reset="earliest", message_consume_limit=None, kafka_configs=None, **kwargs)[source]

Run method for this Task. Invoked by calling this Task after initialization within a Flow context, or by using Task.bind. Args:

  • bootstrap_servers (str, required): comma separated host and port pairs that are the addresses of kafka brokers
  • group_id (str, required): name of the consumer group the consumer will belong to
  • topics (List[str], required): list of topic names to consume messages from
  • request_timeout (float, optional): Maximum time to block waiting for message, event or callback
  • auto_offset_reset (str, optional): configurable offset reset policy
  • message_consume_limit (int, optional): max number of messages to consume before closing the consumer
  • kafka_configs (dict, optional): a dict of kafka client configuration properties used to construct the consumer.
  • **kwargs (Any, optional): additional keyword arguments to pass to the standard Task init method Returns: - List of consumed messages


# KafkaBatchProduce

class

prefect.tasks.kafka.kafka.KafkaBatchProduce

(bootstrap_servers=None, topic=None, messages=None, flush_threshold=None, callback=None, kafka_configs=None, **kwargs)[source]

Task for producing a batch of messages to a Kafka topic.

Args:

  • bootstrap_servers (str, required): comma separated host and port pairs that are the addresses of kafka brokers
  • topic (str, required): name of topic to produce messages to. Must be specified either at init or runtime.
  • messages (List[dict], required): list of messages to produce into a topic where a single message is a dictionary with a key and a value.
  • flush_threshold (int, optional): threshold of messages produced before flushing
  • callback (Callable, optional): callback assigned to a produce call
  • kafka_configs (dict, optional): a dict of kafka client configuration properties used to construct the producer.
  • **kwargs (Any, optional): additional keyword arguments to pass to the standard Task init method

methods:                                                                                                                                                       

prefect.tasks.kafka.kafka.KafkaBatchProduce.run

(bootstrap_servers=None, topic=None, messages=None, flush_threshold=None, callback=None, kafka_configs=None)[source]

Run method for this Task. Invoked by calling this Task after initialization within a Flow context, or by using Task.bind. Args:

  • bootstrap_servers (str, required): comma separated host and port pairs that are the addresses of kafka brokers
  • topic (str, required): name of topics to produce messages to
  • messages (List[dict], required): list of messages to produce into topics where a single message is a dictionary with a key and a value.
  • flush_threshold (int, optional): threshold of messages produced before flushing
  • callback (Callable, optional): callback assigned to a produce call
  • kafka_configs (dict, optional): a dict of kafka client configuration properties used to construct the producer.


This documentation was auto-generated from commit ffa9a6c
on February 1, 2023 at 18:44 UTC

← Kubernetes Tasks →