How to Test Your README with Docker and Real Users

Testing documentation with real users requires paying participants 25 euros an hour to follow a README while speaking aloud, uncovering human comprehension errors that automated checks miss.

ActivityBot README Testing Project Details

Terence Eden conducted usability testing on the README for his ActivityBot project by paying volunteers to record their sessions, SitePoint reported. Each participant spent an hour working through the installation steps while narrating their thought process on video calls.

ELSOLITARIO.ORG reported that the experiment took place as part of requirements for NLnet foundation funding, which mandates real-user installation testing. Eden recruited volunteers through Mastodon to tear apart the documentation in front of his camera. The format required each participant to share their screen and talk through every decision before executing it.

Automated Docker Verification Versus Human Cognitive Testing

Automated mechanical verification in a clean ubuntu:24.04 container catches missing package dependencies, deprecated CLI flags, and broken file references. Human cognitive verification identifies comprehension gaps, ambiguous instructions, and unstated prerequisites that scripts cannot detect.

ELSOLITARIO.ORG noted that human testing targets the curse of knowledge, which prevents project authors from recognizing when their instructions skip implicit steps. While teammates share an author vocabulary and miss hidden assumptions, outside volunteers reveal the exact sentence where a first-time user gets stuck. Jakob Nielsen research from 1993 indicates that five users are enough to discover most usability issues in documentation.

Frequently Asked Questions

What is the difference between README testing with real users and review by teammates?

Teammates share the author vocabulary and understand implicit setup requirements, making them blind to missing instructions. Outside volunteers do not possess this background knowledge, exposing exact comprehension gaps and confusing phrasing.

How much does running usability tests on documentation with paid volunteers cost?

Terence Eden paid participants 25 euros per hour for their time. His ActivityBot testing involved roughly six hours of conversation for a total expenditure of about 150 euros.

Can simulating users with an LLM replace real README testing?

Language models cannot replicate the authentic cognitive friction, hesitation, and behavioral responses of a human reader encountering undocumented assumptions for the first time. Real user sessions record actual confusion points and emotional reactions during video calls.

How many technical documentation validation sessions are needed to find the main problems?

Research by Jakob Nielsen from 1993 demonstrates that five user testing sessions are sufficient to uncover the majority of usability problems.

Which author biases are hardest to detect without documentation testing?

Assumptions regarding sudo privileges, command-line argument syntax, and mandatory restart steps are extremely difficult for authors to spot independently. Authors treat these actions as self-evident even when they are omitted from the written instructions.