Skip to content

From Raw Events to Training Examples

A model does not train on raw reality. It trains on rows that engineers construct from events, timestamps, labels, and features. This page is about that construction step, and the single idea, prediction time, that keeps it honest.

Rapid Recall

Products log raw events (account, behavior, security, payment, review) that live at different times, in different systems, and some are delayed or wrong. The ML data engineering job is to turn them into training examples that match the production decision. Prediction time is the moment the model would have decided in production; every feature must be computed from information available at or before it, and every label must come from after it. The observation window is the past period used for features; the label window is the future period used for ground truth. A clean row is an entity, a prediction time, features known before it, and a label confirmed after it. That structure is point-in-time correctness.

§1 What the product actually logs

Start with what the product actually logs. In checkout fraud, raw events might include:

  • Account events: account created, email changed, password reset, two-factor enabled.
  • Behavior events: product viewed, item added to cart, checkout started, payment submitted.
  • Security events: failed logins, device changes, IP changes, suspicious velocity signals.
  • Payment events: authorization success/failure, payment method, billing country, chargeback notices.
  • Review events: human reviewer decision, customer support report, bank dispute outcome.

These events live at different times and in different systems. Some are generated by your application servers. Some come from payment providers. Some are delayed. Some are wrong. The ML data engineering job is to turn them into training examples that match the production decision.

§2 The most important idea: prediction time

Prediction time is the moment the model would have made its decision in production. In this scenario, prediction time is payment submission. Every feature must be computed using information available at or before that moment. Every label must come from after that moment.

This sounds obvious, but it prevents many subtle bugs. If a training row for transaction T includes a field that was updated after the transaction, the model has seen the future. Offline metrics improve, but the model will not have that information in production.

§3 Observation window and label window

The observation window is the historical period used to compute features. Example: "failed logins in the last 10 minutes" or "total spend in the last 7 days." The label window is the future period used to determine ground truth. Example: "chargeback within 30 days."

A clean training row can be described like this:

Entity: transaction_id = tx_123

Prediction time: 2026-05-01 10:03:14

Features: facts known before 10:03:14

Label: whether fraud was confirmed by 2026-05-31

This structure is the foundation of point-in-time correctness. If you can explain it clearly, many ML system design questions become easier. The discipline of never using a feature updated after prediction time is developed further in Features, Freshness, and Leakage.

Interview Questions

Q1: What is prediction time and why does it matter? Prediction time is the moment the model would have made its decision in production, for example payment submission in fraud detection. Every feature must be computed from information available at or before that moment, and every label must come from after it. Violating this lets the model see the future, which inflates offline metrics and then fails in production where that information does not exist.

Q2: Explain the observation window versus the label window. The observation window is the past period used to compute features, such as failed logins in the last 10 minutes or total spend in the last 7 days. The label window is the future period used to determine ground truth, such as a chargeback within 30 days. Features look backward from prediction time; labels look forward from it.

Q3: How would you describe a single clean training row? As an entity (transaction_id), a prediction time, a set of features computed only from facts known before that time, and a label determined from a fixed future window after it. That shape is point-in-time correctness, and being able to state it clearly makes most ML data design questions easier.