Highly Scalable Observability Data Pipelines with Vector
Vector is an open source platform for creating highly flexible and scalable observability data pipelines. It runs on your infrastructure and connects a wide variety of data sources—from Syslog to HTTP to Prometheus to Kubernetes logs—to a wide variety of sinks—from Kafka to S3 to Clickhouse to SaaS platforms like Datadog. Those pipelines can serve a wide array of use cases, such as ensuring data locality, enabling easy migration between SaaS vendors, data enrichment, and cost reduction. Built in Rust and designed for maximum throughput, minimum resource usage, and operational ease of use, Vector may just be the tool you’re looking for to power observability in your large-scale systems.
In this workshop, you’ll learn how to use Vector to create processing pipelines for your logs and metrics via Vector’s built-in event transformation DSL (Vector Remap Language). We’ll use it to modify logs and metrics as they flow through your Vector instances, and showcase Vector aggregators to process data from multiple observability agents (including the Datadog Agent and Fluent-Bit). The general approach will be hands-on and geared toward real use cases.