Migrate to ClickHouse

Migrate from Elasticsearch or OpenSearch to ClickHouse

Elastic was built for search and stretched into logs, security and analytics. ClickHouse does the aggregations it struggles with at a fraction of the cost, and now covers search and observability too.

How the migration works
Sub second
Queries over billions of rows, from columnar storage and vectorised execution.
50%+
Cost savings observed in the migrations we have delivered from SQL Server, OpenSearch and Snowflake.
100s
Of concurrent users served with consistent latency, so analytics can go into your product.
Anywhere
Open source. Run it on any cloud, on premise or as ClickHouse Cloud.
Why teams move

Why Elasticsearch users come to us

Elastic and OpenSearch are the platforms we compete against most often. Elastic was built around full text search and has been extended over the years into observability, security analytics and general purpose analytics. It does a lot, but people hit a ceiling with it: aggregations over large volumes are slow, the flexibility is not there for the questions they want to ask, and the cluster is expensive to keep running.

The cost is structural. Elasticsearch keeps a lot of data in memory and on fast disk, indexes everything on the way in, and needs a cluster of sizeable nodes to stay responsive. Retention gets cut and data gets sampled to control the bill. ClickHouse compresses the same data many times smaller, reads it from cheap storage and answers aggregation queries far faster.

We migrated a contact centre analytics platform from Postgres and OpenSearch to ClickHouse Cloud and halved the cost while adding real time intraday analytics. For observability, ClickStack now sits on top of ClickHouse to cover the search, dashboards and alerting that Kibana and OpenSearch Dashboards provided.

Sound familiar?
  • [01]Cluster cost grows with data volume, and retention is cut or data sampled to contain it.
  • [02]Aggregations and analytical queries over large indices are slow or time out.
  • [03]Index management, shard sizing, reindexing and JVM tuning consume engineering time.
  • [04]Questions that are easy in SQL are hard or impossible to express in the query DSL.
  • [05]Licensing changes and the Elastic versus OpenSearch split have made the roadmap uncertain.
What changes

Elasticsearch to ClickHouse: faster, cheaper, simpler

ClickHouse is the fastest open source analytical database on the market. These are the differences that matter most for teams coming from this platform.

[01]

A fraction of the infrastructure

ClickHouse compresses log and event data typically ten times or more and serves it from object storage, so the same retention needs a far smaller footprint. Our OpenSearch migration halved the running cost.

[02]

Aggregations that actually finish

ClickHouse was built for the GROUP BYs, time series and joins that Elastic struggles with. Queries over billions of events return in under a second, from any angle, without pre aggregating.

[03]

SQL and full text search

Everything is standard SQL, so analysts and tools can use it directly. ClickHouse now includes text indexes and search functions for the log search use cases that kept people on Elastic.

[04]

Observability with ClickStack

For logs, metrics and traces, ClickStack provides the UI, alerting and OpenTelemetry pipeline on top of ClickHouse, replacing Kibana or OpenSearch Dashboards with a much cheaper backend.

How we migrate

Accelerated migration with CDC

Elastic has no transaction log to replicate from, so the migration is pipeline driven. Whatever is currently writing to Elasticsearch, whether Logstash, Vector, Fluent Bit, an OpenTelemetry collector or Kafka, is pointed at ClickHouse in parallel, with ClickPipes handling the Kafka and object storage sources in ClickHouse Cloud. Historical indices are exported and bulk loaded for the retention window you need. Dashboards and alerts are rebuilt in ClickStack, Grafana or your existing BI tool before the old cluster is retired.

  1. 01

    Assess

    We review the schemas, the query workload and the cost of the current platform, then agree the scope, the target architecture and the business case before anything is built.

  2. 02

    Design the ClickHouse schema

    Tables are redesigned for columnar storage rather than copied across one to one. Sorting keys, partitioning, codecs and materialised views are chosen against your real queries.

  3. 03

    Load and replicate

    Historical data is bulk loaded, then changes are streamed continuously so ClickHouse stays in line with the source for as long as the two need to run side by side.

  4. 04

    Run in parallel

    Dashboards, applications and reports are pointed at ClickHouse and validated against the source. Performance and cost are measured rather than assumed.

  5. 05

    Cut over

    Workloads move across in stages, the old platform is scaled down and the remaining spend is retired. Nothing is switched off until the numbers reconcile.

  6. 06

    Support

    After go live we can monitor the environment, tune it as the workload grows and provide an escalation path under SLA through our proactive support service.

Tooling we use
ClickPipes

The managed ingestion service built into ClickHouse Cloud. Its CDC connectors take an initial snapshot and then stream every insert, update and delete, with no infrastructure for you to run.

Vector, OpenTelemetry and ClickStack

Log and metric shippers write natively to ClickHouse. ClickStack, the ClickHouse observability stack, provides the OTel collector, UI and alerting for observability workloads.

We are tool agnostic. The right choice depends on whether you are going to ClickHouse Cloud or self hosted, how much control you want over the pipeline and what else is in the stack. Zero downtime is the norm, not the exception.

Beyond the migration

One database, many use cases

Moving off Elasticsearch is usually the start. Because ClickHouse is so versatile, the same platform tends to pick up more workloads once it is in place.

Real time analytics

Dashboards and embedded analytics over data that is seconds old, served to many users at once.

Data warehousing

A warehouse that is fast enough to query directly, without the overnight batch or the cube layer.

Observability

Logs, metrics and traces at a fraction of the cost of Datadog, Splunk or Elastic, with ClickStack on top.

Time series

Market data, IoT, telemetry and events at hundreds of thousands of rows per second with long retention.

Search and security analytics

Full text search, SIEM and threat hunting over high volume event data using plain SQL.

Agentic AI

A data platform fast and cheap enough for agents that turn one question into dozens of queries.

Related work

Migrations we have delivered

Telco

Migrating A Contact Centre Analytics Platform From OpenSearch To ClickHouse Cloud

Migrating a workforce performance analytics platform from Postgres and OpenSearch to ClickHouse Cloud, achieving 50% cost reduction and real-time intraday analytics.

Cyber Security & Networking

Building A High Throughput Network Telemetry Platform For Security And Operations Monitoring

Designing and building an open source ClickHouse cluster to capture network telemetry from thousands of routers at 500k rows per second, powering SOC and NOC analytics integrated with Splunk.

Further reading

Ready to move off Elasticsearch?

Start with a short call. We will look at your schemas, queries and current spend, tell you honestly whether ClickHouse is the right fit, and put a number on the performance and cost you can expect.

Support after go live