User research on application prompts

On 2020-04-09 emailed:

• Alex Ellis apellis@gmail.com
• huub.vandenbroek@gmail.com
• yetkinpulcu@gmail.com
• Stig Johan Berggren stigjb@gmail.com
• Christian Kroer chrkroer@gmail.com
• mihi@mool.hu
• kr.srinath2011@gmail.com
• cmpalmer1@gmail.com
• Stephen Malina stephenmalina@gmail.com
• dmitry.urbanovich@ardecs.com

Google Cloud Platform

WITH
apReviewers AS (SELECT userID, ANY_VALUE(email) AS email, COUNT(*) AS apCount, ANY_VALUE(ru.timestamp) as registrationTimestamp FROM `logs.reviews` JOIN `logs.registeredUsers` AS ru USING (userID) WHERE promptIndex IS NOT NULL AND sessionID IS NOT NULL GROUP BY userID ORDER BY apCount DESC),
apCondition AS (SELECT userID, JSON_EXTRACT_SCALAR(extra, "$['condition']") AS condition FROM `logs.userEvents` WHERE JSON_EXTRACT_SCALAR(extra, "$['experimentName']") = "labelApplicationPrompts")
SELECT * FROM apReviewers JOIN apCondition USING (userID) ORDER BY apCount DESC

My questions regarding application prompts

• How do they affect how a review session feels?
  • Current belief: at current frequencies, they don’t meaningfully affect how review sessions feel.
• What story do readers tell about the effect of these questions on their understandings?
• How does accuracy vary with repetition count? Is P(correct | correct last time) lower due to the variation?
• Should application prompts be visually distinguished from recall prompts?

Qualitative insights on application prompts

They induce a different “mode” of thinking, which readers find enjoyable.

• “making me pause and putting me in a “calculate” mode to mentally, or by hand, compute the answer. The questions aren’t too hard and it is a different kind of challenge as compared to remembering the answer to a fixed question.” — Srinath K R Re: Hello from Quantum Country!
• “It is more fun than recalling what was written in the parer exactly. Nevertheless it deepens the knowledge, because rater than just passively remembering we have to use the memorized knowledge actively.” — Mihaly Orosz Re: Hello from Quantum Country!
• “I was a little surprised at first when I saw this question, since it went against the precedent of the questions being pretty trivial to answer after just having read the text. I did find it a little hard… I would say it was completely a “good” kind of hard, since I reinforced that understanding and was still able to get it right.” Re: Question about a question

Transitions back and forth between recall and application prompts aren’t jarring; people know what to do when answering a question.

• “I don’t remember exactly how I felt at the point I encountered the problem, but I’d say that it wasn’t a strong feeling.” — Amir Ebrahimi Re: Question about a question
• “Yes, the application prompts feel differently. At least, they require me to read the question with attention to details which I then have to manipulate in my head. When I face a usual prompt, it mostly feels like I don’t actually read the question (which is okay, I think).” — Dmitry Urbanovich Re: Hello from Quantum Country!
• “I have not noticed a visual difference between the application prompts. I do notice that some questions are more of a theoretical or practical nature.” — Huub van den Broek Re: Hello from Quantum Country!

They may be too easy to be meaningfully helpful.

• “…could have promise, but so far I think that they are probably too easy to really develop my thinking much.” — Christian Kroer Re: Resubscribe please