QCVC readers don’t have low accuracy in the first 2 questions as we had thought

In https://github.com/andymatuschak/qcc/issues/160, Michael observed that the first few questions have unusually low accuracy. He noticed on 2019-11-28 that the first two questions’ accuracies no longer seem terribly low.

Michael mentioned making changes to the text before those questions to prompt people more. Looking back through the Git history, the only such change I see is on 2019-03-30, focused on trying to improve the accuracy of question 3 (b7d5b082f0232040d2fd4adf79b81eedf1946b14).

So… what happened to the accuracies of the questions before and after that change? Query

• card 1: 90% ± 1.4% -> 94% ± 0.79% (N=1622, 3523)
• card 2: 93% ± 1.2% -> 96% ± 0.67% (N=1594, 3483)
• card 3: 67% ± 2.3% -> 65% ± 1.6% (N=1573, 3459)

I’d say: “not much.” Given that this change was made so early (just 12 days after public launch), this could be a cohort effect as easily as a causal change of the edits. It’s interesting that the edits focused on question 3, but that’s the one which least clearly changed

WITH
  cards AS (
  SELECT
    *
  FROM
    `logs.latestEssaysCards`
  WHERE
    essayName = "qcvc"
    AND essayOrder < 3),
  markings AS (
  SELECT
    reviewMarking,
    essayOrder AS cardNumber,
  IF
    (reviews.timestamp < TIMESTAMP("2019-03-30 12:44:11-07:00"),
      "pre",
      "post") AS condition
  FROM
    `logs.reviews` AS reviews
  JOIN
    cards
  USING
    (cardID)
  WHERE
    beforeInterval IS NULL
    AND isRetry IS NOT TRUE),
  means AS (
  SELECT
    cardNumber + 1 AS cardNumber,
    condition,
    COUNTIF(reviewMarking = "remembered") / COUNT(*) AS accuracy,
    COUNT(*) AS N
  FROM
    markings
  GROUP BY
    condition,
    cardNumber),
  cis AS (
  SELECT
    *,
    1.96 * SQRT((accuracy * (1 - accuracy)) / N) AS CI95
  FROM
    means)
SELECT
  *,
  accuracy - CI95 AS lower,
  accuracy + CI95 AS upper
FROM
  cis
ORDER BY
  cardNumber,
  condition