Kacam.net

Product design

这个实验能不能上线

一份「赢了」的实验报告摆在你面前,里面埋了四个坑。

Copy the prompt, then pick a model. It plays the interviewer and asks you through it scene by scene instead of handing you the answer.

On a phone with the app installed, the system opens the app directly. Links cannot open external apps inside WeChat — open it in a browser first.

The copied text ends with the answer key the model grades you against. Don't read it yourself first — working it out on your own is the point.

这一份的形式和别的不一样:它直接给你一份实验报告 —— p = 0.04,转化率相对提升 10%,结论建议全量上线 —— 然后问你批不批。

报告里埋了四个足以让结论完全反过来的问题,而它们都要你自己问出来:你不问样本量,它不会说;你不问护栏,它也不会说。

最后会问一句现实里一定会遇到的:老板说「橙色明显更好看,直接上吧」,你怎么回。 不接受「顶回去」,也不接受「那就上」。

★★★★★ · about 40 min · Tests: 样本量与 MDE、多重比较、新奇效应、被忽略的护栏

What this one tests

Experiment method Evidence and numbers Metric design

Have you done this one?

Sign in to mark this one as practised — next time you come back you'll see when you did it.

Notes from people who did this one

Nobody has written yet. After you run it, say what came out of the conversation — which scene stumped you, which conclusion you changed afterwards.

Sign in to write a note.

How to use this

There is no model answer here — but there are definite wrong answers. Only three things make a case hold up, and every prompt's answer key carries a "common wrong answers" section that is more useful than the checklist itself.

The three things that make it hold up

  1. You can say whose problem this is and what it is — **with evidence** (not "I think users would want it")
  2. You can say what success looks like — **and what that number is today** (no baseline, no acceptance criteria)
  3. You can name the cost and what you gave up — **the hardest one to fake**, which is why every prompt hunts for it

How to start

  1. Pick a question, open it, copy the prompt.
  2. Paste it into a brand-new conversation — not one where you already discussed it.
  3. Say nothing. Wait for its first question.

Four commands you can call out at any time

  • 跳过 move to the next scene now
  • 提示 give a direction, not the answer
  • 直说吧 spell this point out in full, then keep pressing on the trade-offs
  • 复盘 jump to the wrap-up and print the review card

Three traps

  • Don't rush to a solution. More than half the lost points here come from designing before asking — especially describing features before defining what success means.
  • Expect to be pushed for numbers. "It'll perform better", "users will like it" get cut off: which metric, what is it today, and when do you look?
  • When it starts praising you, point it back to rule five — feeling validated crowds out noticing what you missed.

Start with the north star metric; it underpins everything else. Then pick by what's on your plate: a backlog you can't order → prioritisation; nobody using it or retention sliding → retention and onboarding; need to price something → pricing. Save the experiment review for later, and keep redoing the one about turning a one-line request into a spec — it's the closest to daily work.