IP allowlist

Which RudderStack IPs should I allowlist?

You will need to allowlist the following RudderStack IPs to enable network access:

• 3.216.35.97
• 23.20.96.9
• 18.214.35.254
• 54.147.40.62
• 34.198.90.241
• 100.20.239.77
• 52.38.160.231
• 34.211.241.254
• 44.236.60.231

The EU cluster IPs are listed below:

• 3.66.99.198
• 3.64.201.167

Data syncs and retries

When does RudderStack load data into the warehouse destination?

RudderStack lets you choose how frequently it should sync the data into the data warehouse. The default time is 30 minutes. However, you can choose to extend this value up to 24 hours. You can also configure the time of the day when RudderStack loads the data.

Is there a way to force load my data into the warehouse?

Yes, there is - you can configure the below values in your config.yaml file:

Copy
warehouseSyncFreqIgnore = true #if set true this will ignore syncFrequency and syncStartingAt values which are configured in UI. By default this is false
uploadFreqInS=1800 #This field lets you control syncPeriod if above field set to true

Where can I view the status of my data sync?

You can view the warehouse upload status in the Live Events section of the destination in the RudderStack dashboard.

What happens if the cluster or the destination service is down? Is there a possibility of data loss?

If a warehouse destination is down or unavailable, RudderStack will continue to retry sending the events (on an exponential backoff basis, for up to 3 hours).

RudderStack stores the syncs as staging files and retries sending them at a later time when the cluster is up again. This allows for a successful delivery without any missing data.

Does RudderStack keep retrying sending events indefinitely? How long can a service be down without losing any data?

After retrying for up to 3 hours, RudderStack marks the syncs as aborted. Once the service is up and running again, you can go to the Syncs tab in the RudderStack dashboard and retry sending the data.

Is there a scenario where I can lose my data without the ability to retry syncs?

RudderStack stores the data in the form of staging files in the object storage associated with the warehouse destination. Usually, the data is stored in the bucket for 30 days. However, it completely depends on the retention policy you have set for the bucket.

Can I tie a value in the rudder_discards table back to the actual event in another table?

Yes, you can.

The rudder_discards table contains columns like row_id (corresponding to the event's messageId), table_name, column_name, and column_value.

You can use the table_name and map the row_id to the id column in your event’s table.

Staging

What are the files written in the location <namespace>/rudder-warehouse-staging-logs/?

RudderStack collects all the unprocessed data flowing to the warehouse destinations as staging files. It stores these files in the object storage at the location <namespace>/rudder-warehouse-staging-logs/.

Once the staging files are processed, RudderStack separates them by the event name and sends them to the specified destination.

How can I delete the staging files in my buckets?

RudderStack loads all the events configured with your warehouse into staging buckets. You can set a retention policy to delete these files from the staging bucket after a certain time. If you do not set any retention policy, the files will keep on accumulating.

Make sure that the retention policy duration is longer than the warehouse sync frequency duration so that RudderStack can re-run event uploads for a longer time, if required. It is recommended to set the retention policy duration to 30 days/1 month.

Can I grant RudderStack limited access to the Azure Storage resources?

Yes, you can enable the Use shared access signature (SAS) Tokens setting while configuring your Blob Storage in the dashboard settings, as shown:

Optimizing syncs

How do I avoid lags in data syncs while heavy loads are running on my data warehouse?

You can use either of the following approaches to avoid data sync lags in your warehouse:

• Approach 1: In the warehouse destination connection settings, set an exclusion window which lets you set a start and end time when RudderStack will not sync data to your warehouse. You can run heavy loads during this time interval.

• Approach 2: In the warehouse destination connection settings, your can increase the sync frequency interval so that there are lesser number of warehouse syncs throughout the day. You can also run the data syncs during the non-peak hours by setting the Sync Starting At time interval.

• Approach 3: If you are performing only writes/updates on your data warehouse, you can set up a read replica of your database and connect it to your load generation tool. Connect the read/write replica to RudderStack, as shown:

How can I speed up my warehouse uploads?

You can speed up your warehouse uploads by configuring the following parameters in the config.yaml file in your RudderStack setup:

What happens if I exceed the table column limit in my warehouse destination?

If you exceed the permissible table column limit in your warehouse, RudderStack will not sync data to it.

How can I handle the column count threshold in my warehouse?

To avoid exceeding the column count threshold in your warehouse, you can use any one of the following approaches:

• Go to your warehouse and drop the unwanted table columns. Refer to the following links for managing your warehouse column count:
  • Amazon Redshift
  • Google BigQuery
  • Snowflake
  • Databricks
  • PostgreSQL
  • MySQL

For other warehouses, refer to their respective documentation.

• If you're syncing semi-structured event data that is not defined by a fixed schema, you can leverage RudderStack's JSON Column Support feature to send the data to your warehouse without worrying about the column count limit.

Namespace, prefix, and warehouse schema

Can I modify the warehouse schema once it is created? Will RudderStack overwrite or discard the changes when updating the schema?

You can modify the warehouse schema once it is created by adding or altering the columns, or changing the column data type. RudderStack honors the updated warehouse schema and sends the incoming data to the warehouse accordingly.

For example, if you update a column's data type in the schema, RudderStack will consider the latest, updated column data type and typecast all the incoming data accordingly, before sending it to that column.

How is the Namespace field populated for a warehouse destination?

RudderStack sets the Namespace field depending on the following conditions:

• If the Namespace and Database fields are present in your warehouse configuration:

  • For ClickHouse destination, the Database field is set as the namespace.
  • For all the other destinations, the Namespace field is populated as it is.

• If RSERVER_WAREHOUSE_(DEST_TYPE)_CUSTOM_DATA_SET_PREFIX parameter is present, the namespace is populated as (CUSTOM_DATA_SET_PREFIX)_(SOURCE_NAME). For example:

Copy
Parameter value:
RSERVER_WAREHOUSE_SNOWFLAKE_CUSTOM_DATA_SET_PREFIX="<some_value>"

Namespace:
<some_value>_<source_name>

• If none of the above fields are specified in the warehouse configuration settings, the source name is populated as the namespace for the first sync and picked up from the cache for all the subsequent syncs. In case the source name is changed, the namespace still remains the same as it is picked from cache.

Can I change the namespace (schema name) of my data warehouse in RudderStack?

Yes, you can. RudderStack lets you change the namespace (schema name) where it loads all the data in your warehouse.

You can specify your desired schema name in the Namespace field while configuring your warehouse destination in RudderStack, as shown:

How can I add the table prefixes if there are multiple sources connected to a single destination?

You can use RudderStack's Transformations feature to filter the events by the sourceID and then update the event name to add a prefix.

A sample transformation logic is shown below:

Copy
export function transformEvent(event, metadata) {
  const meta = metadata(event);
  if (meta.sourceId === your_source_id) {
    event.event = '<prefix>' + event.event;
  }
  return event;
}

How does RudderStack configure the schema name before loading the data in the warehouse?

RudderStack configures the schema name based on the values mentioned in the below table. The table columns are defined as follows:

• Sync: Indicates whether RudderStack performs the first data sync after the source is set up or the subsequent syncs.
• Namespace: The Namespace field set by the user while configuring the warehouse destination in the RudderStack dashboard.
• Warehouse.<destType>.customDataset Prefix: The RSERVER_WAREHOUSE_(DEST_TYPE)_CUSTOM_DATA_SET_PREFIX parameter in the config.yaml file, in case the user has a RudderStack deployment locally/in their own environment.
• Source name: Name of the source connected to the warehouse destination.

How does RudderStack configure the table names?

RudderStack sets the table names by picking up the event names and modifying them to follow the snake case convention (for example, source name -> source_name).

For the following sample snippet, RudderStack sets the table name as product_purchased.

Copy
{
  "userId": "user123",
  "event": "Product Purchased",
  "properties": {
    "name": "Rubik's Cube",
    "revenue": 4.99
  },
  "context": {
    "ip": "14.5.67.21"
  },
  "timestamp": "2020-02-02T00:23:09.544Z"
}

The table name truncation logic is as follows:

• For PostgreSQL, the table name is truncated after 63 characters.
• For Data Lake destinations (S3 Data Lake, GCS Data Lake, Azure Data Lake), there is no set limit.
• For other destinations, the table name is truncated after 127 characters.

How does RudderStack handle schema data type mismatch when sending events to a warehouse destination?

RudderStack automatically handles the data type conversion in some common scenarios depending on the warehouse column data type.

Can I change the name of the table in which my events are synced?

Yes, you can add a transformation to change the event name and, consequently, the name of the table in which the events are synced.

As seen in the above FAQ, RudderStack sets the table names to the event names modified in the snake case.

A sample transformation used to change the event name is as shown:

Copy
export function transformEvent(event, metadata) {    
  if (event.event.toLowerCase() === '<old_event_name>') {
      event.event = '<new_event_name>';
  }
  return event;
}

How does RudderStack configure the column names? / How does RudderStack normalize arrays and complex event properties in the warehouse schema?

RudderStack flattens the event properties and converts them into snake case (for example, source name -> source_name). Further, each of these event properties acts as a warehouse column.

1. The below example shows how the standard properties are normalized:

Copy
{
  product: {
    name: "iPhone",
    version: 11
  }
}

Normalized properties (Column names)

Copy
product_name : "iPhone"
product_version : 11

2. The below example shows how the array properties are normalized:

Copy
{
  products: [
    {
      name: "iPhone",
      version: 11
    },
    {
      name: "Android",
      version: 16
    }
  ]
}

Normalized property (Column name)

Copy
products: "[{ \"name\": \"iPhone\", \"version\": 11  },  { \"name\": \"Android\", \"version\": 16  }]"

3. The below example shows how the complex properties are normalized:

Copy
{
  products: {
    iPhone: {
      version: 11
    },
    Android: {
      version: 16
    }
  }
}

Normalized properties (Column names)

Copy
products_i_phone_version : 11
products_android_version : 16

How does RudderStack determine the column data type? Can I change an existing data type for a column?

RudderStack determines the data type of a column based on its value in the first event (during the first upload sync).

For example, suppose column_x is received with the value as 1. RudderStack then sets the data type of this column as int in the event table.

To set your preferred data type for a particular column, it is highly recommended to follow these steps:

1. Create a column in the warehouse with a dummy name and the required data type.
2. Cast the data from the original column and load it into the dummy column.
3. Drop the original column.
4. Rename the dummy column to the original column name.

Why am I not able to see the properties added at the top level of an event in warehouse destination?

RudderStack drops any non-standard properties (properties apart from the standard properties) declared at the top level of an event. However, you can add such properties in the context or properties section of the event payload.

Warehouse-specific settings

While configuring the Snowflake destination, what should I enter in the Account field?

While configuring Snowflake as a destination in RudderStack, you need to enter your Snowflake connection credentials which includes the Account field, as shown below:

The Account field corresponds to your Snowflake account ID and is a part of the Snowflake URL.

The following examples illustrate the slight differences in the Snowflake account ID for various cloud providers:

What are the SAS token permissions required for the warehouse destinations?

If you're using Azure Blob Storage as an intermediate object storage for your data warehouse destinations, you will need to assign the following permissions for your SAS token:

Identity resolution

How can I achieve identity resolution with multiple sources connected to one warehouse destination?

The default namespace will be the source name with some modifications. This will cause issues while creating mappings for identity resolution across different sources, leading to multiple RUDDER_ID created for the same user.

To avoid this, you can set the same namespace for each source, as shown:

Setting a namespace ensures that the mappings are consistent and only one RUDDER_ID is created for each user - irrespective of the number of sources connected to your warehouse destination.