RavenDB 5.3 New Features: Incremental time series & implementing lambda base...

RavenDB 5.3 New FeaturesIncremental time series & implementing lambda based accounting

Everyone is on the cloud these days, and one of the things that I keep seeing pushed is the notion of usage based billing. Basically, the idea that you are paying for what you use.

Let’s assume that we are building a software as a service where users can submit an image and you’ll do some computation on that. The actual details aren’t relevant. What matters is that your pricing model is based around how much time processing each image takes and how much memory is used. You are running this on many machines and need to figure out how to do billing at the end of the month. It turns out that this can be quite a challenge. With incremental time series, a lot of the details around that just go away.

Here is how you can implement this:

async Task Process(ProcessImageCmd cmd) { var sp = Stopwatch.StartNew(); var size = cmd.Image.SizeInBytes * GetMemMultiplierFor(cmd.Image.Format); await ProcessImage(cmd.Image.GetStream()); // do actual work sp.Stop(); using var session = store.OpenAsyncSession(); await session.StoreAsync(new ProcessingRecord(cmd.AccountId, cmd.Image.Name,sp.ElapsedMilliseconds)); session.IncrementalTimeSeriesFor(cmd.AccountId, "INC:Processing") // record the details for billing .Increment(new[]{ sp.ElapsedMilliseconds, size }); await session.SaveChangesAsync(); }

You count the required memory as well as the actual runtime and record that in an incremental time series. We are also storing the details  in a separate document for that particular run in the same transaction (if the user cares about that level of detail). The interesting bit about how this can be used is that the data is now immediately available for the user to see how much they are going to be billed.

Typically, a lot of time is spent in figuring out how to record those details efficiently and then how to query and aggregate those. We tested time series in RavenDB to billions of data points, and the internal format lends itself very well to aggregated queries.

Now you can take the code above, run it on 100s of machines, and it will all end up giving you the proper result in the end.