Artem Krylysov

Datadog collects billions of events from millions of hosts every minute and that number keeps growing and fast. Our data volumes grew 30x between 2017 and 2022. On top of that, the kind of queries we receive from our users has changed significantly. Why? Because our customers have grown in sophistication: they run more complex stacks, want to monitor more data, and run more complex analyses. That, in turn, puts pressure on our timeseries data store.

Data stores have a number of tricks in their bag to offer good performance. One of the most critical ones is the judicious use of indices, a key data structure that can make queries fast and efficient, or unbearably slow. Over the years, our homegrown indices that were put in place in 2016 became a performance bottleneck for queries and a source of increased maintenance. We knew that we had to learn from these challenges and come up with something better.

This blog post provides an overview of the Datadog timeseries database and the challenges of timeseries indexing at scale. We’ll compare the performance and reliability of two generations of indexing services.

Full-Text Search is one of those tools people use every day without realizing it. If you ever googled "golang coverage report" or tried to find "indoor wireless camera" on an e-commerce website, you used some kind of full-text search.

Full-Text Search (FTS) is a technique for searching text in a collection of documents. A document can refer to a web page, a newspaper article, an email message, or any structured text.

Today we are going to build our own FTS engine. By the end of this post, we'll be able to search across millions of documents in less than a millisecond. We'll start with simple search queries like "give me all documents that contain the word cat" and we'll extend the engine to support more sophisticated boolean queries.

String interning is a technique of storing only one copy of each unique string in memory. It can significantly reduce memory usage for applications that store many duplicated strings.

A few months ago I released the first version of an embedded on-disk key-value store written in Go. The store is about 10 times faster than LevelDB for random lookups. I'll explain why it's faster, but first let's talk about the reason why I decided to create my own key-value store.

Running Go on AWS Lambda is not something totally new - developers figured out how to launch Go binaries from Python a while ago, but it wasn't convenient and had some performance implications.

A few days ago Amazon announced an official Go support for AWS Lambda.

Cgo is a mechanism that allows Go packages call C code. The Go compiler enables cgo for every .go source file that imports a special pseudo package "C". The text in the comment before the import "C" line is treated as a C code. You can include headers, define functions, types and variables - everything a normal C code can do:

package main

/*
#include <stdio.h>

void foo(int x) {
    printf("x: %d\n", x);
}
 */
import "C"

func main() {
    C.foo(C.int(123)) // x: 123
}

Go has a powerful built-in profiler that supports CPU, memory, goroutine and block (contention) profiling.

package main

import (
    "net/http"
    _ "net/http/pprof"
)

func hiHandler(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte("hi"))
}

func main() {
    http.HandleFunc("/", hiHandler)
    http.ListenAndServe(":8080", nil)
}

Here are the slides from my talk "Scraping the Web with AWS Lambda and PhantomJS" given at Greater Philadelphia AWS User Group meetup on May 25, 2016.

You can find the source code of PhantomJS/Node.js web scraper for AWS Lambda at https://github.com/akrylysov/lambda-phantom-scraper.

A couple of weeks ago after spending some time with Python profiler, I discovered that Python’s json module is not as fast as I expected. I decided to benchmark alternative JSON libraries.