The two clocks of billing

When a fact was true and when the system learned about it are two different questions. Many hard problems in billing result from a disagreement between them.

All billing systems that support real customers manage two timelines: effective time (when a fact became true) and wall-clock time (when the system learned about it). Operations like backdating a subscription aren’t an edge case, instead they’re a common occurrence that needs to be solved.

This post walks through three features in Orb that only make sense when you consider both clocks – event backfills, plan version migrations, and accounting period locks. The underlying rule supporting all of these features is to be flexible on the effective clock and tough on the wall clock.

The two clocks

The first clock is the effective time, when a fact is true in the real world. A customer’s price might be $1 per API call starting on January 1st. Events may need to be backfilled to cover February’s time period or a contract amendment may need to apply starting from the newly renegotiated renewal date. All of these are effective times.

The second is wall-clock time. This is sometimes also called record time or transaction time and represents when the system came to understand a given fact. The contract amendment might be relevant starting in March, but it was signed in April. The events that were backfilled covered February’s time period but they landed yesterday.

This is referred to as bitemporal history by Martin Fowler and his article is one I’ve referenced many times to understand the domain. For billing in particular, these two clocks frequently disagree and are the cause for many hard problems in the billing domain including irreproducible invoices and shifting revenue. These problems, I argue, all stem from reconciling these two clocks.

For example, Widgets Inc. is billed for API calls at $0.05 per call. Their April invoice went out on May 1st for $1,200 matching 24,000 API calls tracked in Orb. On August 14th, an employee at Widgets Inc. noticed there was a multi-day outage during the month of April in their reporting pipeline that resulted in 6,000 API calls not being reported to Orb. In total, the April invoice was for 30,000 API calls and a total of $1,500.

This is shown below:

The question now remains: April’s invoice went out on May 1st for $1,200 – was it wrong?

When taking the perspective from the effective clock, the invoice was wrong (it should’ve been $1,500). When taking the perspective from the wall clock, the invoice was correct since it accurately represented the amount that Orb was aware of on May 1st. So which answer do we trust? Frustratingly, both answers are correct! We need both clocks in order to build a system that can express these situations before we can decide on how to act on them.

Below, we’ll cover three places in Orb where features need to be able to track both clocks in order to act on these types of scenarios.

Event backfills

Orb bills end customers from raw usage events rather than from a pre-aggregated count. In this architecture, Orb customers send us events that have an event name, a timestamp, a customer identifier, and a dictionary of arbitrary values. Every invoice in Orb that bills for usage is computed from these events.

When the event stream turns out to be incorrect or incomplete, a backfill is used to replace the events for a specific time frame with the corrected set of events. It is designed with audit-ability in mind as Orb never overwrites or deletes any of the ingested events, so any events that are superseded by a backfill are archived. These events are still queryable, but no longer for billing.

You can see the two clocks in action directly for the backfill object in our API docs; here are their relevant fields:

This is the whole thesis of this post found directly in our API docs. timeframe_start and timeframe_end answer “which period of time is this about?” whereas close_time and reverted_at answer “when did we believe it to be true?” They have to move independently; Widgets Inc.’s outage happened in April regardless of when someone noticed.

Widgets Inc.’s backfill chooses a time frame of April and closes it in August. When it closes, there are two answers to the question “what did April contain?” and both of them are queryable.

If you pay attention, you’ll notice that any individual event didn’t change. The events Orb held on May 1st continue to carry their April timestamps and so do the 6,000 events that were backfilled in August; April is when the API calls happened. What changed is the set of events Orb believes April contains. You can still ask the question “what did the metric see when April’s invoice was sent” and get a precise answer because the events the backfill superseded were archived instead of deleted.

A backfill can also be reverted in Orb with reverted_at tracking when this happens. When reverting, an undo is the only sensible way to revert event data because aggregations over events may have used the backfill data to calculate invoice numbers before it is ultimately reverted. If this revert were powered by a delete instead, the billing system could not properly answer questions from an audit around why the invoice numbers changed.

As a direct result of having two clocks, there is one caveat worth surfacing. Orb recommends against backfilling event data over any invoicing period where an invoice has been sent to the customer without coming up with an explicit communication plan. Orb can backfill data over time periods of previously issued invoices, but this may require sending customers updated invoices and explaining what has happened. A backfill can re-write what events were sent during April but it can’t un-issue an invoice that has already been sent (and potentially also paid). The flexibility on effective time decreases when the wall-clock time has already executed some action in the real world, like sending an invoice.

This raises a further question though: what happens when the usage data was correct from the beginning but the price we applied was wrong?

Plan version migrations

In Orb, one of the ways to make large scale pricing changes is with Orb’s plan version migration functionality. With this feature, pricing changes can be accomplished through the creation of a new numbered plan version and running a migration that moves selected existing subscriptions on to the new version at a scheduled time. The docs above cover the product mechanics and there are two posts on the product motivation and how it was built.

The important part for our discussion is a plan migration’s scheduled time. This scheduled time allows for subscription-specific timing options where you choose one of the following for your migration: immediately, on a specific date, start of invoice, and start of next invoice. These timing options are all effective times that apply to each individual subscription.

For example, if you have one subscription that starts on the 7th of the month and another that starts on the 13th of the month, you could roll out your pricing change when the next invoice for each comes up, the 7th and the 13th of next month, respectively.

Every option in this list is answering “starting from when is this new pricing change true?” Nothing in this selection is constrained by when the pricing change occurs. This wall-clock time is just when someone initiates their migration.

This leads to three total scenarios of combinations of effective time and wall clock time:

1. Effective time is after the wall clock time → a scheduled pricing change (the easy scenario)
2. Effective time is now or the same as the wall clock time → change prices right now (also straightforward)
3. Effective time is before the wall clock time → a backdated pricing change (the difficult scenario since invoices may have already been issued)

The easy scenario: a scheduled change

Acme Corporation is on Version 2 of the Starter Plan which charges $0.05 / LLM token used. On February 10th, Version 3 was published which now charges $0.06 / LLM token used and a pricing change is scheduled for the start of the next invoice, March 1st. Let’s say for our example that Acme Corporation uses 10,000 tokens each month.

This pricing change is easy for one particular reason: the record of the change occurs before it becomes effective. All invoices ever issued were generated with the full knowledge of this change because this is only affecting future invoices. No issued invoice ever contradicts what is true on the effective time timeline and there is nothing to reconcile.

Most people, when thinking about a pricing change, tend to imagine this example and this is why billing systems using a single clock feel perfectly fine until those systems start running into the complexity of the real world.

The difficult scenario: a backdated change

Imagine the same scenario as above but with a slight twist – the pricing change was communicated late. On June 5th, a contract amendment arrived with the new pricing, $0.06 / LLM token used, effective March 1st. A backdated pricing change is run on June 6th to reflect the new pricing and amend incorrectly calculated invoices. (This is supported first-class in Orb where backdated pricing changes will recalculate and re-issue any affected invoices.)

The effective time clock now shows that three months were billed incorrectly:

The table lays out where things differ and shows that the changes are tedious but not necessarily hard to implement. Three invoices will need to be recomputed to collect the missing money. Where things get tricky is that these invoices are issued documents with respect to the wall-clock time. They have already been sent, maybe paid, maybe synced to an ERP system, and maybe even partially credited. A backdated pricing change has to reconcile the new effective time ground truth against every downstream artifact with the previous understanding of the ground truth.

This requires machinery to recalculate issued invoices, crediting previously paid invoices, voiding credit memos that no longer apply, and more. This functionality exists because the past has already occurred on the other clock, the wall clock, not because pricing changes are inherently hard.

This isn’t the end of the line for us though! The money for the above pricing changes has to land somewhere in the books and these books have their own opinions about time.

Accounting period locks

To help you run your business operationally, Orb provides reporting on revenue, among other things, that divides your earned revenue into monthly accounting periods. These accounting periods can be closed or locked to prevent future revenue from being added to them. This is an important feature of tracking revenue as the revenue you have reported for a given month may not be allowed to change. Our docs on Orb’s revenue recognition methodology have more information about this.

When an accounting period is closed, the numbers reported within it are frozen and closing a period guarantees this. These periods also cannot interleave meaning everything before a given closed period must also be closed. Any activity that belongs to a closed period (e.g. a backdated invoice) is not rejected but instead is recognized as a catch-up adjustment on the first day of the next open period.

If you read the above with the effective time and wall-clock time timelines in mind, you can understand how Orb supports the functionality. The period over which information applies is the effective time and the instant at which it is incorporated into the system’s corpus of information is the wall-clock time.

If we were to say that “June is closed” that doesn’t mean that no new information about the month of June may arrive because we can’t stop the real world from generating late arriving information about the month of June. Instead, “June is closed” means that information about the month of June that arrived after June has been closed cannot re-write June’s numbers. The corrected information keeps its effective date, June, but its financial impact posts at the earliest moment that the wall clock permits. This is something that Accountants have been doing forever called a prior-period adjustment. The real area where billing systems can fail is when they can’t tell these two clocks apart resulting in a lock that is either too strict (no corrections to prior periods) or incorrectly implemented (the numbers change and the lock does nothing).

The difficult scenario, continued: the changes meet our locks

Back to the example of Acme’s backdated migration. Imagine your Finance team closes the books about ten days after the month ends. So on June 5th, the day the contract amendment occurs, March and April are closed and May is still open. Where do the three $100 deltas go?

Check out the before and after of the revenue report too:

The June 6th row is the one that Orb shows in its reports and, at first glance, it looks a bit wrong. May shows $800 of revenue for a month that should only have $600. The math doesn’t quite add up, does it? Except that it does – the total matches what we’d expect and we haven’t made any changes to the reported history of March and April. This is the entire promise of what closing the books is! Both the second and third rows in the table above are both true, but our report can’t be faithful to both the effective time and wall-clock time clocks, it must choose one.

The desired period locking behavior is what helps us choose which clock to follow faithfully (in this instance, wall-clock time) and the catch-up adjustment we see above is the reconciliation of the two timelines.

^{Aside: the usual single-clock schema is a mutable updated_at column on a price. In this case, updated_at is a wall-clock timestamp for the latest belief only. The moment the row is overwritten, the ability to answer “what did we believe when we generated that invoice in the past” is no longer answerable. Without it, you can’t regenerate invoices, explain what happened to the customer, or defend revenue reporting during an audit.}

Flexible on one clock, strict on the other

If I take a step back and look at the above three scenarios, one design rule for building a billing system comes to the surface for me: be flexible on the effective time and be strict on the wall-clock time.

Be flexible on the effective time clock. The world is quite messy and billing systems are downstream of that mess. Contracts get signed late, usage arrives late, pricing changes occur retroactively because someone made a deal months ago that no one knew about. Any billing system that rejects backdated operations is ignoring the real world. This pushes that mess into one-off spreadsheets, manually granted credits, or custom scripts written by an Engineer to reconcile the state based on what was supposed to happen. These resources usually don’t contain clocks themselves and must be reconstructed later when someone finds an issue or an audit occurs. This means you must agree to rewriting the effective past.

Be strict on the wall-clock. Every flexible operation above is recorded and never destructive. Superseded usage events are archived as opposed to deleted. Migrations reconcile historical invoices that they invalidated. Accounting period locks guarantee that closed periods stay closed, with corrections surfacing in future open periods. The record of what the system believed and when it believed it all survive the corrections, which make these corrections safe to allow in a billing system.

These aren’t competing clocks that need to be balanced. A strict wall-clock is what allows you flexibility in the effective time clock. You can allow rewriting of the effective time in the past because the wall-clock record on every single rewrite is immutable. Flexibility in the effective time clock without a strict wall-clock leads to a billing system that can’t be trusted. A strict wall-clock yields safety, but without a flexible effective time clock it leads to a billing system that isn’t useful.

This is what we mean when we say that Orb is flexible and safe. They are claims about two different clocks. A strict wall-clock allows for safety and helps make a flexible effective time clock possible.

Recapping

• Effective time is when a fact is true. Wall-clock time is when the system learned it. In billing, they frequently disagree. Backdating is a common occurrence in billing systems.
• Event backfills track both clocks as fields: a time frame stating which period of the world they correct and a close time stating when Orb started believing it. Superseded events are archived rather than deleted, making any event correction explainable and reversible.
• Plan version migrations (sometimes shortened to “migrations”) are a feature to change pricing in bulk that allows for selecting a specific effective time at which to apply new pricing. Scheduled changes are easy because the record of when the pricing change occurs happens before the effect of when it happens (i.e. it’s in the future). Backdated migrations are the difficult ones because invoices may already have been issued and need to be reconciled with the new pricing.
• Accounting period locks leverage the wall-clock time to decide which accounting periods can accept changes. A lock doesn’t stop new information about the past from arriving but it does determine where the impact lands on the effective time clock. When these are in disagreement, Orb records catch-up contributions.
• The design rule underneath: be flexible on the effective time clock and strict on the wall-clock. A history of immutable records makes rewriting the effective past safe.

The general version of all of this is Martin Fowler’s bitemporal history which posits that we should stop overwriting and instead store facts as immutable versions that carry both sets of timestamps. Record history is append-only – you never change what you knew on August 1st, you instead layer September’s information on top.

Orb tracks these as first-class concepts in many of our product features. This requires more work, but answering the question of “why does this invoice say $100” requires a foundation that has been built with this functionality in mind.

Based on my experience thus far, my rule of thumb is this: any value in a billing system that can be amended or corrected retroactively secretly requires tracking two time dimensions. Most people get one of the two dimensions right, but often forget the second. You can model these two time dimensions from the beginning or you can rediscover their existence the next time you have a billing incident.