> ## Documentation Index
> Fetch the complete documentation index at: https://private-7c7dfe99-mintlify-8a08bda2.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Controlling the syncing of a Postgres ClickPipe

> Doc for controlling the sync a Postgres ClickPipe

export const Image = ({img, alt, size}) => {
  return <Frame>
      <img src={img} alt={alt} />
    </Frame>;
};

This document describes how to control the sync of a Postgres ClickPipe when the ClickPipe is in **CDC (Running) mode**.

<h2 id="overview">
  Overview
</h2>

Database ClickPipes have an architecture that consists of two parallel processes - pulling from the source database and pushing to the target database. The pulling process is controlled by a sync configuration that defines how often the data should be pulled and how much data should be pulled at a time. By "at a time", we mean one batch - since the ClickPipe pulls and pushes data in batches.

There are two main ways to control the sync of a Postgres ClickPipe. The ClickPipe will start pushing when one of the below settings kicks in.

<h3 id="interval">
  Sync interval
</h3>

The sync interval of the pipe is the amount of time (in seconds) for which the ClickPipe will pull records from the source database. The time to push what we have to ClickHouse isn't included in this interval.

The default is **1 minute**.
Sync interval can be set to any positive integer value, but it is recommended to keep it above 10 seconds.

<h3 id="batch-size">
  Pull batch size
</h3>

The pull batch size is the number of records that the ClickPipe will pull from the source database in one batch. Records mean inserts, updates and deletes done on the tables that are part of the pipe.

The default is **100,000** records.
A safe maximum is 10 million.

<h3 id="transactions">
  An exception: Long-running transactions on source
</h3>

When a transaction is run on the source database, the ClickPipe waits until it receives the COMMIT of the transaction before it moves forward. This with **overrides** both the sync interval and the pull batch size.

<h3 id="configuring">
  Configuring sync settings
</h3>

You can set the sync interval and pull batch size when you create a ClickPipe or edit an existing one.
When creating a ClickPipe it will be seen in the second step of the creation wizard, as shown below:

<Image img="https://mintcdn.com/private-7c7dfe99-mintlify-8a08bda2/KeiVE4MGTrCd9SX4/images/integrations/data-ingestion/clickpipes/postgres/create_sync_settings.png?fit=max&auto=format&n=KeiVE4MGTrCd9SX4&q=85&s=4d5a4cb7cb962ea6dc9ffd7fd6097efb" alt="Create sync settings" size="md" width="902" height="654" data-path="images/integrations/data-ingestion/clickpipes/postgres/create_sync_settings.png" />

When editing an existing ClickPipe, you can head over to the **Settings** tab of the pipe, pause the pipe and then click on **Configure** here:

<Image img="https://mintcdn.com/private-7c7dfe99-mintlify-8a08bda2/KeiVE4MGTrCd9SX4/images/integrations/data-ingestion/clickpipes/postgres/edit_sync_button.png?fit=max&auto=format&n=KeiVE4MGTrCd9SX4&q=85&s=553b3d87ab4d16adeaa83405b81dac91" alt="Edit sync button" size="md" width="933" height="417" data-path="images/integrations/data-ingestion/clickpipes/postgres/edit_sync_button.png" />

This will open a flyout with the sync settings, where you can change the sync interval and pull batch size:

<Image img="https://mintcdn.com/private-7c7dfe99-mintlify-8a08bda2/KeiVE4MGTrCd9SX4/images/integrations/data-ingestion/clickpipes/postgres/sync_settings_edit.png?fit=max&auto=format&n=KeiVE4MGTrCd9SX4&q=85&s=1118aa8d50477192270bdba9459ac7a9" alt="Edit sync settings" size="md" width="598" height="588" data-path="images/integrations/data-ingestion/clickpipes/postgres/sync_settings_edit.png" />

<h3 id="tweaking">
  Tweaking the sync settings to help with replication slot growth
</h3>

Let's talk about how to use these settings to handle a large replication slot of a CDC pipe.
The pushing time to ClickHouse doesn't scale linearly with the pulling time from the source database. This can be leveraged to reduce the size of a large replication slot.
By increasing both the sync interval and pull batch size, the ClickPipe will pull a whole lot of data from the source database in one go, and then push it to ClickHouse.

<h3 id="monitoring">
  Monitoring sync control behaviour
</h3>

You can see how long each batch takes in the **CDC Syncs** table in the **Metrics** tab of the ClickPipe. Note that the duration here includes push time and also if there are no rows incoming, the ClickPipe waits and the wait time is also included in the duration.

<Image img="https://mintcdn.com/private-7c7dfe99-mintlify-8a08bda2/KeiVE4MGTrCd9SX4/images/integrations/data-ingestion/clickpipes/postgres/cdc_syncs.png?fit=max&auto=format&n=KeiVE4MGTrCd9SX4&q=85&s=ff3c9cd9ee086da594319467da4d9854" alt="CDC Syncs table" size="md" width="2428" height="626" data-path="images/integrations/data-ingestion/clickpipes/postgres/cdc_syncs.png" />
