Season 1 · Episode 9

Ship a Real Data Product

VMenu's research, imports, audits, and deployment.

This lesson drops soon — check back here for the video and voiceover.

Behind every “simple” data product is an unglamorous engine: collection, normalization, deduplication, import, audit. This lesson follows VMenu — a real restaurant data product — and teaches the discipline that keeps tens of thousands of records trustworthy: contracts before collection, idempotent imports, honest handling of timeouts, and verification from the user’s side.

The problem this lesson solves

D

a

t

a

p

r

o

j

e

c

t

s

f

a

i

l

s

i

l

e

n

t

l

y

.

A

r

e

-

r

u

n

d

u

p

l

i

c

a

t

e

s

h

a

l

f

t

h

e

c

a

t

a

l

o

g

.

A

p

a

r

t

i

a

l

u

p

d

a

t

e

w

i

p

e

s

f

i

e

l

d

s

t

h

e

e

n

d

p

o

i

n

t

r

e

p

l

a

c

e

s

w

h

o

l

e

s

a

l

e

.

A

t

i

m

e

o

u

t

g

e

t

s

r

e

t

r

i

e

d

b

l

i

n

d

l

y

a

n

d

i

m

p

o

r

t

s

t

h

e

s

a

m

e

b

a

t

c

h

t

w

i

c

e

.

T

h

e

d

a

s

h

b

o

a

r

d

l

o

o

k

s

f

i

n

e

;

t

h

e

d

a

t

a

i

s

r

o

t

t

i

n

g

.

T

h

e

p

r

o

b

l

e

m

t

h

i

s

l

e

s

s

o

n

s

o

l

v

e

s

i

s

b

u

i

l

d

i

n

g

a

p

i

p

e

l

i

n

e

w

h

e

r

e

e

v

e

r

y

s

t

a

g

e

i

s

s

e

p

a

r

a

t

e

,

e

v

e

r

y

r

e

-

r

u

n

i

s

s

a

f

e

,

a

n

d

e

v

e

r

y

c

o

u

n

t

r

e

c

o

n

c

i

l

e

s

.

The lesson, step by step

Define the data contract before collecting anything

Restaurant identity, address, menu section, item, price, source, update status, timestamps. The contract is agreed before the first scrape — it’s what “correct” means for every record that follows.

Separate the stages

Collection, normalization, deduplication, import, and audit are distinct stages with distinct jobs. When stages blur, bugs hide: a dedup problem looks like an import problem, and you fix the wrong one.

Make imports idempotent

Rerunning the same batch must reconcile, not duplicate or erase fields. Idempotency is the difference between “safe to retry” and “pray it worked the first time.”

Respect the danger of partial updates

When an endpoint replaces a full record, a partial update isn’t partial — it’s a deletion of everything you didn’t send. Always pass the full record through, or use an endpoint that doesn’t replace.

Build quality reports

Menu-less venues, suspiciously thin menus, missing dayparts, duplicate candidates, closed businesses. The reports are the product’s immune system — they find the rot the pipeline can’t feel.

Treat timeouts as unknown outcomes

A timed-out request didn’t necessarily fail; the server may have finished. Reconcile exact counts before retrying. “It timed out, run it again” is how duplicates are born.

Deploy like it matters

Correct authorship on commits, environment variables in place before the deploy that needs them, production checks after. The lesson’s hard-won rule: env changes don’t apply to already-built deployments.

Verify from the user’s side

Search a venue, inspect menu sections, test mobile behavior, confirm analytics. The pipeline’s opinion of itself doesn’t count — the user’s experience does.

Worked example

An import batch of several hundred venues times out halfway. The wrong move: re-run the batch. The lesson’s move: query what actually landed, reconcile exact counts against the source, identify the missing slice, and import only that — idempotently, so even an overlap reconciles instead of duplicating. Final counts match the source to the record. That reconciliation is the whole lesson in one incident.

Your takeaway

• Contract first: define the record shape before collecting a single row.
• Separate stages; make re-runs idempotent; never trust a blind retry.
• Partial updates on replace-style endpoints are deletions in disguise.
• Quality reports and count reconciliation are non-negotiable.
• Verify from the user’s side — search it, open it on mobile, check analytics.

The BRAVE lens

Every episode runs through the BRAVE method: brief the outcome, reveal relevant context, authorize the next action, verify the evidence, and evolve the system.