Continuously loading S3 data into ClickHouse

Mark Needham Principal Product Marketing Manager, ClickHouse

In this video, we'll learn how to continuously load data into ClickHouse from AWS S3, using the S3Queue table engine. We start by exploring the config required on the serve to enable this functionality, before examining the S3 bucket that we're going to load into ClickHouse. We walk through the two modes of ingestion that we can use and adjust the flush frequency so that data is available in ClickHouse quicker. Finally, we use shadowtraffic.io to generate some more data and stream it into our S3 bucket, before checking that it's made its way into ClickHouse.