Log Capturer Configuration

This step is to allow GART’s log capturer to fetch the logs of specified database tables to avoid causing irrelevant logs to be tracked.

Edit Kafka configuration ($KAFKA_HOME/config/server.properties) as follows:

delete.topic.enable=true

Set up a configuration of Debezium. Please replace the fields in the configuration file ($KAFKA_HOME/config/connect-debezium-{mysql,postgresql}.properties) that have sharp brackets (<>) with the actual contents (e.g., database.user, database.password).

The following are the parts of each configuration file that need to be changed.

For Postgresql

 1database.history.kafka.bootstrap.servers=<kafka bootstrap servers, e.g., localhost:9092>
 2schema.history.internal.kafka.bootstrap.servers=<kafka bootstrap servers, e.g., localhost:9092>
 3
 4name=<name of the connector, e.g., test-connector>
 5connector.class=io.debezium.connector.postgresql.PostgresConnector
 6database.hostname=<postgresql host, e.g., 127.0.0.1>
 7database.port=<postgresql port, e.g., 5432>
 8database.user=<postgresql user>
 9database.password=<postgresql password>
10database.dbname=<which database is needed to capture, e.g., ldbc>
11table.include.list=<list the tables in the order of vertices, then edges>
12snapshot.mode=<if enable bulkload, set as "always", otherwise set as "never">
13
14slot.name=<name of replication slots, please avoid repetition>
15
16plugin.name=pgoutput
17publication.autocreate.mode=filtered

For MySQL

 1database.history.kafka.bootstrap.servers=<kafka bootstrap servers, e.g., localhost:9092>
 2schema.history.internal.kafka.bootstrap.servers=<kafka bootstrap servers, e.g., localhost:9092>
 3
 4name=<name of the connector, e.g., test-connector>
 5connector.class=io.debezium.connector.mysql.MySqlConnector
 6database.hostname=<mysql host, e.g., 127.0.0.1>
 7database.port=<mysql port, e.g., 3306>
 8database.user=<mysql user>
 9database.password=<mysql password>
10database.include.list=<which databse is needed to capture, e.g., ldbc>
11table.include.list=<list the tables in the order of vertices, then edges>
12snapshot.mode=<if enable buldload, set as "initial", otherwise set as "never">

Automatic Configuration

To simplify this part of the configuration, we have provided scripts for the same.

# In the docker image
gart-env$ cd /workspace/gart/

# Show the help information
gart-env$ ./scripts/update_kafka_config_file.py --help