Feature Store Point-in-Time Join Correctness: Event Time, Availability Time, and Training Leakage Gates
Joining features only by entity and nearest timestamp can still leak data that arrived after the label into the training set. This article freezes event time, availability time, TTL, and snapshot watermarks while ensuring that offline training and online serving consume the same feature-definition generation.
Key Points
- Topic Focus: AI、DevOps
- Suitable for: indie hackers, full-stack engineers, AI application developers
- Core value: quickly understand background, practical paths, common pitfalls, and engineering advice
A fraud model's label time is Friday at 10:00. A customer balance event occurred at 09:58 but did not enter the warehouse until Friday at 10:06. If the training query only selects event_timestamp <= label_timestamp, that balance will be treated as "known at the time," introducing future information that would never have been available online. Conversely, if the online store retains only the latest value, replaying an old order will read future state. A point-in-time join is not merely a window SQL query; it is a contract defining when something happened, when it became knowable, and which version of a definition was available.
Training Dataset Snapshots freeze the sample set, while Training Data Schema Evolution governs field semantics. This article further constrains how label rows meet historical features. Feast's point-in-time join uses entity keys and event time to select features from before the label time, and it can use a created timestamp to handle revisions with the same event time. However, the platform must still explicitly define business availability time and its late-arrival policy.
Version Three Kinds of Time Together
Publish featureViewRelease.v1, binding the entity key schema, feature field ID, source snapshot, transform digest, eventTimeField, recordedTimeField, created/revision rules, TTL, time zone, null semantics, offline query implementation, online materialization strategy, and owner. eventTime indicates when a business fact took effect, while recordedTime indicates when the warehouse accepted a revision; neither proves that the prediction service could already read the value. Every audience and online generation must also retain a servableAt receipt recording when each value was first materialized successfully. Training eligibility requires both eventTime and the corresponding audience's servableAt to be no later than the sample cutoff.
schema: featureViewRelease.v1
view: account-risk@12
entityKeys: [tenant_id, account_id]
eventTime: balance_changed_at
recordedTime: warehouse_committed_at
ttl: 30d
sourceSnapshot: iceberg:snapshot-8841
transformDigest: sha256:6a1...
online: { audience: fraud-api, generation: 57, receipts: materialization-ledger@57 }
With asynchronous materialization, the warehouse commit and servableAt can differ by several minutes. A receipt binds the entity, feature revision, audience, online generation, first readable time, and write-confirmation digest. It may be issued only after the production read path actually returns the same canonical value from that generation. If a historical system lacks this evidence, the relevant samples can only be marked AVAILABILITY_UNPROVEN; an aggregate materialization watermark must not be used to fabricate a visibility time for each value.
Label samples must also carry predictionAt, the entity schema, and the permitted source watermark. Multiple revisions with the same event time must be resolved using a stable source revision, never the database's return order. TTL is calculated from the business time specified by the contract. Missing, expired, and not-yet-available values must be encoded separately rather than all being filled with zero. Dynamic authorization and deletion still follow Training Data Deletion Propagation; the reproducibility of an old snapshot does not mean it may still be used for training.
One Release Connects Offline and Online
The state flow is DRAFT -> MATERIALIZED -> LEAKAGE_TESTED -> PARITY_TESTED -> APPROVED -> ACTIVE -> RETIRED. The offline builder outputs the training matrix root, each column's maximum event, recorded, and servable times, the source watermark, and the leakage audit. The online builder outputs the same release digest, entity coverage, materialization watermark, and receipt root. The only acceptance point is the registry, which activates a complete release for a model audience using CAS. The model package must reference the feature release at the same time; neither may independently track latest.
Training queries may read only from a fixed source snapshot. A late revision creates a new dataset generation rather than modifying an old matrix in place; any retraining must explicitly record the differences. Online updates are deduplicated by (entity, featureView, sourceRevision), and the system must confirm before release that they have not crossed a deletion or authorization generation. Database CDC delivers changes, and Cache Revalidation can reuse a generation fence, but only the feature registry defines which value is valid for a particular prediction.
Attack Time Boundaries with Leakage Fixtures
Tests construct an old event that enters the warehouse one second after the label, a value that becomes readable online ten minutes after entering the warehouse, two created revisions with the same event time, clock rollback, DST, a local date mistakenly interpreted as UTC, exact equality at the TTL boundary, a backfill overwriting a newer online value, an offline null defaulted to zero online, an expired source snapshot, and a late deletion outbox event. An independent oracle checks event time, recorded time, and audience-specific servableAt together. If the receipt is removed or the wrong online generation is substituted, the release must be rejected rather than falling back to event time or warehouse commit time.
Release gates can require: 100% agreement with the oracle across 100,000 generated time cases; zero accepted cases where any feature has servableAt(audience,generation) > predictionAt; zero accepted cases with missing or wrong-generation materialization receipts; zero differences between canonical offline and online values; at least 99.5% coverage for mandatory entity slices; zero old features served after the deletion generation; and P99 lag for new materializations below the declared SLO. When samples are insufficient or the source cannot provide availability time, mark the result LEAKAGE_UNPROVEN; it must not pass the training gate.
For rollout, first shadow-generate the new matrix for existing training jobs and compare the selected source revisions. Then allow only one low-risk model to read the new online generation before expanding to 5% and 25%. If an anomaly occurs, switch the model and feature alias together to the previous complete release; keep the old matrix and audit read-only. A rollback must not rematerialize data that has already been deleted or allow an old online value to cross the current authorization floor. RAG Knowledge Freshness focuses on the timeliness of document evidence; here, the acceptance target is the structured facts that were genuinely available at prediction time.
Related
Related Content
Article
What Is Vibe Coding? Will It Change How Programmers Work?
Technical trends, engineering practices, and an indie developer perspective on Vibe Coding, AI, and independent development.
Article
From Copilot to Codex: AI Coding Tools Enter the Agent Era
Technical trends, engineering practices, and indie-developer perspectives around AI, AI Agents, and Vibe Coding.
Article
Complete Technology Stack Analysis for LLM Application Development
Covers tech trends, engineering practices, and indie developer perspectives around large models, RAG, and AI.
Article
Security Risks of AI Code Generation: Don't Blindly Trust Vibe Coding
Around Vibe Coding and AI: technology trends, engineering practices, and an indie developer's perspective.