Volume XLVI, No. 2 | February 8, 2024

Battle of the Books: Evaluating the Reading Comprehension of HTML and PDF Textbook Formats

Introduction 

Executive function refers to the cognitive skills needed for planning, monitoring, and controlling cognitive processes. As the brain matures and executive functioning improves, it facilitates prioritizing, critical thinking, decision making, goal directed behavior, self-control, and emotional regulation. It is believed that executive functions play a role in reading comprehension (Butterfuss & Kendeou, 2018). Three main components of executive functioning have been investigated in relation to reading comprehension. These three components include working memory, inhibitory control, and attention shifting or cognitive flexibility (Ober et al., 2019).

While reading, working memory is used to hold, monitor, and revise relevant information to construct a coherent understanding of the text. Information in the current text is organized and encoded based on its relation to the reader’s prior knowledge and conceptual organization, which is then used to draw inferences and construct the meaning of the text (Ober et al., 2019).

Inhibitory control is related to the ability to resist or suppress the intrusion of information from working memory that is irrelevant or distracting. This allows the focus to remain on the relevant information while actively suppressing the irrelevant information that may decrease the likelihood of making incorrect or irrelevant interpretations of what is being read (Butterfuss & Kendeou, 2018).

The third component, attention shifting or cognitive flexibility, refers to the ability to switch attention between multiple tasks, operations, and mental sets. This applies to reading comprehension as the reader needs to be able to shift attention between the phonological, syntactic, and semantic features of printed text in order to simultaneously consider multiple aspects of the text that contribute to understanding the overall meaning (Butterfuss & Kendeou, 2018).

With the increasing use of digital textbooks and Open Educational Resources (OER), many studies have investigated the association between reading comprehension and type of media. While students may prefer digital texts and feel confident in their comprehension, Singer and Alexander (2016) found that students performed better in recalling main ideas and other relevant information when using printed texts. When reading on screen, students are more likely to skim the surface of the text looking for specific information rather than using print-based reading strategies that encourage them to make connections and draw inferences, which are necessary for comprehension. In addition, it has been suggested that the interactive features and animations often found in e-books serve as amusing distractions rather than contributing to overall reading comprehension (Herold, 2014).

Several explanations for deficits in reading comprehension when reading on a screen have been proposed. One explanation focused on whether reading on screen is impacted by the brain’s difficulty in creating a spatial representation or cognitive map of the text, which impacts reading speed and impairs the ability to go back to search for a specific piece of information, the ability to recall the content, and the overall comprehension of the material read (Hou et al., 2017). Another theory proposes that the increased sensorimotor and kinesthetic sensations that are used in reading printed text enrich our engagement with the material and influence our comprehension (Mangen et al., 2019). The shallowing hypothesis proposes that because digital media involves quick shifts and interactions that provide immediate rewards, readers using digital devices may have difficulty with sustained attention and therefore, with reading comprehension (Delgado et al., 2018).

As the use of digital media becomes more essential to educational materials, the evaluation of available formats may be beneficial. HTML format may offer ease in scanning and scrolling (Nielsen & Kaley, 2020), but they are not necessarily more compliant with accessibility guidelines than PDF format (Goulden, 2020) that provide the reader with a page format that is more similar to printed text. This study investigated whether differences in reading comprehension would be evident when giving students the textbook in either HTML format or PDF, which could impact student success.

Method 

Four total General Psychology (PSY 101) classes were chosen for comparison over two semesters at Harford Community College. The semesters were separated into two phases. Phase One took place during fall 2022 and Phase Two took place during spring 2023. Two classes taught by one instructor were given the link for the HTML version of an OER textbook, while the two other classes taught by a second instructor were only given access to the PDF version. Four chapters of the book were chosen by the instructors, and identical quizzes were created for each chapter and posted on Blackboard.

Students were told that the purpose of the quizzes was to evaluate the readability of the textbook. They were asked to read each chapter and answer the quiz questions prior to any discussion of the chapter during class meetings, as well as prior to having access to PowerPoint presentations that would be presented during lectures. The quizzes were posted one week before each chapter was to be covered in class and announcements were made to remind students to complete each reading and each quiz by the specified date. Each quiz was worth 10 points and all quizzes were included in the calculation as part of the students’ grades for the class.

Results

Four quizzes with ten questions each were assigned to PSY 101 students in the PDF group (N=58) and the HTML group (N=49) across two semesters. Overall, the PDF group performed slightly better (M=9.24) than the HTML group (M=9.14).

Overall, in Phase One, the average quiz score for the PDF group (n=31) was 9.17/10 and the average quiz score for the HTML group (n=26) was 8.95/10. Therefore, the results of Phase One do not show a statistically significant difference between classes (p = .25). 

In Phase Two, the overall average quiz score for the PDF group (n=27) was very similar to that of the HTML group (n=23) with the PDF group scoring an average of 9.31/10 and the HTML group scoring an average of 9.36/10. The results of Phase Two do not show a statistically significant difference between classes (p = .75).

Discussion

Though there was not a statistically significant difference between the PDF group and the HTML group, there was a clear trend of higher performance across the PDF group overall. The results from this study can enable instructors to make informed decisions about the text options and other reading materials, such as research studies, provided to students.

It is important to recognize a few limitations with this study. To begin with, the sample size was quite small, which we tried to address by continuing the study for a second phase. Secondly, it is difficult to justify internal validity when we cannot be positive about the integrity of reading. We cannot be certain that students based their answers strictly from reading the text provided to them. Finally, we cannot ensure that the quiz format assessed reading comprehension because students may have skimmed the text in search of their answers. In future studies, it would be ideal to have a larger sample size and enforce more restrictions to ensure reading integrity. 

Nicollette Krenzien and Theresa Lilienthal, Assistant Professors, Psychology, Harford Community College, nkrenzien@harford.edu

References

Butterfuss, R. & Kendeou, P. (2018). The role of executive functions in reading comprehension. Educational Psychology Review, 30(3), 801-826. https://link.springer.com/article/10.1007/s10648-017-9422-6 

Delgado, P., Vargas, C., Ackerman, R., & Salmerón, L. (2018). Don’t throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension. Educational Research Review, 25, 23-38. https://doi.org/10.1016/j.edurev.2018.09.003 

Goulden, J. (2020, July 22). PDF versus HTML: Dispelling the myths. Crawford Technologies. https://crawfordtech.com/document-accessibility/pdf-versus-html-dispelling-the-myths/ 

Herold, B. (2014). Digital reading poses learning challenges for students. Education Digest, 80(1), 44-48. https://www.edweek.org/teaching-learning/digital-reading-poses-learning-challenges-for-students/2014/05 

Hou, J., Rashid, J., & Lee, K. M. (2017). Cognitive map or medium materiality? Reading on paper and screen. Computers in Human Behavior, 67, 84-94. https://doi.org/10.1016/j.chb.2016.10.014 

Mangen, A., Olivier, G., & Velay, J-L. (2019). Comparing comprehension of a long text read in print book and on Kindle: Where in the text and when in the story? Frontiers in Psychology, 10(38). 1-12. https://doi.org/10.3389/fpsyg.2019.00038 

Nielsen, J. and Kaley, A. (2020, June 28). Avoid PDF for on-screen reading. Nielsen Norman Group. https://www.nngroup.com/articles/avoid-pdf-for-on-screen-reading/ 

Ober, T. M., Brooks, P. J., Plass, J. L., & Homer, B. D. (2019). Distinguishing direct and indirect effects of executive functions on reading comprehension in adolescents. Reading Psychology, 40(6), 551-581. https://doi.org./10.1080/02702711.2019.1635239 

Singer, L. M., & Alexander, P. A. (2016). Reading across mediums: Effects of reading digital and print texts on comprehension and calibration. The Journal of Experimental Education, 85(1), 155-172. https://doi.org/10.1080/00220973.2016.1143794