Connect to Aiven for Apache Kafka® with command line tools#

These examples show how to send messages to and receive messages from an Aiven for Apache Kafka® service using command line tools.

Pre-requisites#

kafka-console-producer.sh and kafka-console-consumer.sh are part of the Apache Kafka® toolbox included with the open source Apache Kafka® code. They may be found in the bin directory of the unpacked archive.

For kafka-avro-console-producer follow the installation instructions in its GitHub repository, or get the Confluent platform using the Tar archive option at Quick Start for Confluent Platform. The kafka-avro-console-producer tool is in the bin directory of the unpacked Confluent archive.

Note

You do not need to install the Confluent platform to use the kafka-avro-console-producer tool with an Aiven for Apache Kafka service that is using Karapace.

To setup the configuration file needed by all three tools, follow the guide to set up properties to use the Apache Kafka toolbox.

Variables#

Variable

Description

HOST

Host name for the connection

PORT

Port number to use for the Kafka service

CONFIGURATION_PATH

Path to your configuration file for Apache Kafka toolbox

SCHEMA_REGISTRY_HOST

Host name for your schema registry

SCHEMA_REGISTRY_PORT

Port number for your schema registry

SCHEMA_REGISTRY_USER

User name for your schema registry

SCHEMA_REGISTRY_PWD

Password for your schema registry

TARGET_TOPIC

The name of the Kafka topic to be written to/read from

In the command lines below, values in { and } are to be replaced - so {PORT} would be replaced by the appropriate port number, for instance 12345.

Produce messages#

With kafka-console-producer you can send multiple messages into your topic.

kafka-console-producer.sh --broker-list {HOST}:{PORT} \
  --topic {TARGET_TOPIC} \
  --producer.config {CONFIGURATION_PATH}

Once the connection is successfully established you can send messages one after another by typing them in the terminal. For example:

message1
message2

Produce messages with schema#

With kafka-avro-console-producer you can include the schema by connecting to your schema registry

Note

  1. The schema.registry.url value must be a full URL, typically starting with https://

  2. Aiven’s Karapace is an acceptable schema registry for this purpose. See Use Karapace with Aiven for Apache Kafka® for how to enable it for your Aiven for Kafka service. The SCHEMA_REGISTRY_ values for the command line can be found on the service Overview page, on the Schema registry tab.

kafka-avro-console-producer --broker-list {HOST}:{PORT} \
  --producer.config {CONFIGURATION_PATH} \
  --topic {TARGET_TOPIC} \
  --property value.schema='{"type":"record","name":"Test","fields":[{"name":"id","type":"string"}]}' \
  --property schema.registry.url={SCHEMA_REGISTRY_HOST}:{SCHEMA_REGISTRY_PORT} \
  --property basic.auth.credentials.source=USER_INFO \
  --property basic.auth.user.info={SCHEMA_REGISTRY_USER}:{SCHEMA_REGISTRY_PASSWORD}

After the connection is established you can send messages according to selected schema. For example:

{"id": "1"}

Note

For more information on how to use kafka-avro-console-producer see the Confluent developer documentation.

Consume messages#

With kafka-console-consumer you can read messages from your topic. For example, run the command below to start reading from the beginning of the topic.

kafka-console-consumer.sh --bootstrap-server {HOST}:{PORT} \
  --topic {TARGET_TOPIC}  \
  --consumer.config {CONFIGURATION_PATH} \
  --from-beginning