I got curious: how accurate are our estimates? I examined study session durations from the last few months, and it seems we’ve been underestimated session durations—even though we got the median question review duration about right!
To keep analysis simpler, I used only sessions with 50 questions (the cap). There were 1056 since September. I threw out sessions longer than 50 minutes (1 minute per question).
Even so, the median session is 9.4 minutes, which is about 11 seconds per card on average. Query 1
I don’t really understand that: 11 seconds is a really long time to be interacting with a flashcard. My guess is that the distribution of card review durations has lots of outliers.
So I looked at the individual card reviews which comprise this data. 11 seconds is at the 73rd percentile when lumping all sessions’ reviews together. The median card takes 6 seconds to review. So this distribution has a much heavier tail than the overall session duration distribution. Query 2
But is this distribution mostly defined by the 90th percentile sessions? Let’s look at individual question review data from sessions at 40th-60th percentile duration. Query 3
Actually, this data looks pretty much the same! Median of 7 seconds (slightly higher!); 11 seconds is at the 71st percentile.
My interpretation: while most questions take only ~6 seconds (seems reasonable), any given session has a handful of questions which require more thought. Sure, many of the >30 second samples are just task-switching, but 30s is 95th percentile, so we’d reach largely the same conclusion even if we ignored them.
To approximate this, I’ve changed the emails to use 11s as the average card duration (was 5s). Commit
WITH samples AS (SELECT cardCount, TIMESTAMP_DIFF(c.timestamp, s.timestamp, SECOND) AS seconds FROM `logs.studySessionCompletions` AS c JOIN `logs.studySessions` AS s USING (sessionID, userID) WHERE s.timestamp >= TIMESTAMP("2019-08-01") AND cardCount = 50)
SELECT * FROM UNNEST((SELECT APPROX_QUANTILES(seconds, 100) FROM samples WHERE seconds/cardCount < 60)) WITH OFFSET AS percentile
WITH samples AS (SELECT cardCount, sessionID, TIMESTAMP_DIFF(c.timestamp, s.timestamp, SECOND) AS seconds FROM `logs.studySessionCompletions` AS c JOIN `logs.studySessions` AS s USING (sessionID, userID) WHERE s.timestamp >= TIMESTAMP("2019-08-01") AND cardCount = 50),
reviews AS (SELECT *, TIMESTAMP_DIFF(timestamp, LAG(timestamp) OVER (PARTITION BY userID, sessionID ORDER BY timestamp ASC), SECOND) AS duration FROM (SELECT * FROM samples WHERE seconds/cardCount<60) JOIN `logs.reviews` USING (sessionID))
SELECT * FROM UNNEST((SELECT APPROX_QUANTILES(duration, 100) FROM reviews)) WITH OFFSET
WITH samples AS (SELECT cardCount, sessionID, TIMESTAMP_DIFF(c.timestamp, s.timestamp, SECOND) AS seconds FROM `logs.studySessionCompletions` AS c JOIN `logs.studySessions` AS s USING (sessionID, userID) WHERE s.timestamp >= TIMESTAMP("2019-08-01") AND cardCount = 50),
reviews AS (SELECT *, TIMESTAMP_DIFF(timestamp, LAG(timestamp) OVER (PARTITION BY userID, sessionID ORDER BY timestamp ASC), SECOND) AS duration FROM (SELECT * FROM samples WHERE seconds >= 488 AND seconds < 660) JOIN `logs.reviews` USING (sessionID))
SELECT * FROM UNNEST((SELECT APPROX_QUANTILES(duration, 100) FROM reviews)) WITH OFFSET