× Copyright Disclaimer Privacy Policy Author Guidelines Current Issue Archive Publishing Ethics Join As Reviewer Advertise Submit Article Abstracting And Indexing Editorial Board Contact Editor-in-Chief Scope of the Journal About the Journal


JAPER is indexed in SCOPUS

Comparative study of traditional face-to-face teaching, audience response system, and a flipped classroom plus audience response

Naoto Nakagawa1*, Toshiyuki Yamashita1

1 School of Pharmaceutical Sciences, Ohu University, Koriyama, Fukushima, Japan.

Correspondence: Naoto Nakagawa, School of Pharmaceutical Sciences, Ohu University, Koriyama, Fukushima, Japan. [email protected]


ABSTRACT

The audience response system (AUD) and flipped classroom (FLP) strategies have been applied globally in higher education. However, to the best of our knowledge, there have been no studies comparing AUD and AUD with FLP(FLP+AUD). This study examined whether FLP+AUD might be the most effective teaching method for pharmaceutical students compared with the traditional face-to-face (TRA) and AUD lecture styles. We recruited second-year (P2) university students in Japan from 2017 to 2019 and conducted TRA, AUD, and FLP+AUD studies in 2017, 2018, and 2019, respectively. We then compared the exam scores for the summative evaluations for the following numbers of participants: 49 in 2017, 78 in 2018, and 90 in 2019. The academic backgrounds of students were similar at the end of their freshman year, except for their GPA. The median pharmacology exam scores following the TRA, AUD and FLP+AUD lectures were 73.3%, 90.0%, and 93.3%, respectively. An analysis of covariance using GPA revealed that AUD and FLP+AUD significantly improved exam outcomes compared to TRA (p < 0.001). However, FLP+AUD did not significantly improve scores compared to AUD alone (p = 0.487). Post hoc power analysis showed that the power of our study was 0.916, indicating that the Type II error was less than 0.1. In conclusion, active learning methods, including AUD and FLP+AUD, were significantly more effective than TRA methods, and both AUD and FLP+AUD had the same learning outcomes for short-term knowledge retention.

Keywords:  Active learning, Audience response system, Flipped classroom, Pharmacology


Introduction 

Japan’s Ministry of Education, Culture, Sports, Science, and Technology (MEXT) defined active learning academically with the following statement in 2012:

Students that have little purpose and motivation need to adopt an attitude of independent study. Universities must re-examine their teaching methods and provide opportunities for students to engage activities and take part in interactive classes. “Expected approaches to the university” include developing interactive learning, a smaller number of students per class, and the active use of teaching assistances and student assistants. Such changes illustrate promoting guidance and utilizing communication technology and information [1].

Considering this statement, Japan universities have begun to adjust their classes and experiment with new approaches. The flipped classroom (FLP) approach promotes active learning by using information and communication technology (e.g., videos, podcasts, readings, and websites) to transfer additional knowledge outside the scheduled class time. Studies have shown positive learning performance using the FLP approach among pharmacy students [2-9]. Others reported negative findings [10, 11]. Researchers have also studied FLPs in other academic disciplines [12-16]. A recent meta-analysis comparing FLP to traditional face-to-face (TRA) lectures [17] found that the FLP method improved student learning. Overall, the FLP model seems to be a positive approach for teaching pharmaceutical students.

Universities have also tried the audience response system (AUD), an active learning method sometimes used as an FLP component focusing on interactivity between teachers and students, such as in-class quizzes. This study sought to determine whether combining FLP with AUD (FLP+AUD) might produce better learning outcomes than either method alone by comparing TRA and AUD for teaching pharmaceutical students at a Japanese university.

Materials and Methods

Targeted students

Pharmacist education in Japan is a 6-year program; second-year students are called P2 students. This study targeted P2 students in pharmacology courses offered at Ohu University from 2017 to 2019. The pharmacology program comprised 13 subjects for second to fourth-year students. The P2 students took courses in the following five subjects: drug receptors and pharmacodynamics, biologically active substances, introduction to autonomic pharmacology, immunopharmacology, and the endocrine system.

The authors of this study taught the endocrine system courses, which covered the hypothalamus, pituitary abnormalities, adrenal cortical and adrenal medulla hormones, and adrenal cortical dysfunction. The P2 students attended lectures on these topics twice a week for a period of two weeks (1.5 h/class: four classes and 6 h in total) in the fall quarter (September–November), using TRA in 2017, AUD in 2018, and FLP+AUD in 2019. The authors collected the P2 students’ external exam scores and prerequisite GPA in the freshman year (P1) to obtain academic background information for all three years. The students’ prerequisite subjects were mathematics, chemistry, biology, physics, basic science exercises, basic pharmaceutical training, introductory chemical thermodynamics, functional morphology, biochemistry, basic analytical chemistry, physical chemistry, and organic chemistry.

Learning outcomes

The course of pharmacology emphasizes knowledge. Thus, the expected outcome was that the P2 students would fully understand the endocrine system.

Audience response system

The AUD lectures were conducted using Clica (http://clica.jp/LP/), an active learning Internet-based tool that collects real-time data on students’ reactions, comprehension, and opinions during lectures. Posts containing their responses could be displayed on their smartphones and a screen in the class for them to view and use in collaboration with other students and teachers. Thus, they could learn others’ perspectives on the information presented in the lectures and compare them with their interpretations. During the lectures, the teachers explained each topic and then asked multiple-choice questions using Clica. The P2 students were given approximately 1 m to respond, and their responses were then displayed on the screen. The lecturers provided additional explanations and handouts as necessary.

Flipped classroom plus AUD

The 6-hour classes were flipped. The teachers uploaded the four prerequisite videos to YouTube and emailed each video’s URL to the students three days before the lecture. Sequentially, the run times of the four videos were 10:16, 10:50, 13:16, and 10:18 (m:s). The teachers retrieved data regarding viewing times and the number of video views from YouTube’s analytics. At the start of each lecture, they administered multiple-choice questions to students through Clica. The teachers and the students viewed the students’ responses on the screen and the teachers provided small additional lectures using handouts as necessary based on their responses.

Primary outcome

The study’s primary outcome was a comparison of the endocrine system exam scores following each of the three teaching methods: TRA, AUD, and FLP+AUD. The exam for evaluating the P2 students’ understanding had 30 MCQs. Identical questions were administered to each of the 3 class cohorts, but the questions and multiple-choice options were re-shuffled annually.

Anonymous questionnaire survey

To gauge the students’ preferences for the AUD and FLP+AUD lectures, the authors created anonymous questionnaire surveys and distributed them during the last endocrine system lecture in the pharmacology course for students in the 2018 and 2019 cohorts. Students in the TRA lecture did not complete the survey as the questionnaire did not apply to them.

Ethical considerations

The university ethics committee provided ethical approval for the study (No. 220). At the end of the last lecture, the lecturers explained the purpose of the study to the students. They obtained written informed consent from students who expressed their intention to participate. All procedures were conducted following the relevant ethical guidelines and regulations.

Statistics

The G*Power software, version 3.1, calculated the sample size required: 159. The required size was obtained by assuming an ANOVA with an effect size of 0.25, a power of 80%, an alpha error probability of 0.05, and three groups.

Categorical data were analyzed using Fisher’s exact tests and the chi-square test. An analysis of covariance (ANCOVA) was carried out when the background information between the 3 cohorts was significantly different. Statistical significance was set at < 0.05, and statistical analyses were done using EZR (“Easy R”). EZR is a graphical user interface for the programming language of R [18].

The data of the current study are available from the corresponding author upon reasonable request.

Results and Discussion

The background information of the P2 students in the 2017, 2018, and 2019 cohorts shows the total number of students was 217, which was larger than the requisite sample size of 159. There were no significant differences in the gender distribution between the years or the external exam scores. Additionally, the prerequisite GPA for the first-years (P1) was not significantly different, except in 2018 (Table 1).

 

 

Table 1. Background information for P2 students in the three cohorts

                                                        Year

 

Method

2017

2018

2019

p value

Traditional face-to-face (TRA)

Audience response system (AUD)

Flipped classroom plus AUD (FLP+AUD)

Students (Male/Female)

49 (18/31)

78 (33/45)

90 (38/52)

0.795*

External Comprehensive Examination***: Median (P1: Fall)

0.90

0.87

0.89

0.315**

Chemistry: Median

0.95

0.82

0.90

0.108**

Math / Physics: Median

0.78

0.91

0.86

0.335**

Biology: Median

0.99

0.89

0.87

0.118**

Prerequisite GPA: Median (P1)

2.7

2.1

2.7

< 0.001**

Female (Median)

2.8

2.2

2.9

0.015**

Male (Median)

2.4

1.8

2.8

< 0.001**

*A chi-square test was utilized to evaluate significance, defined as p < 0.05.

** A Kruskal–Wallis test was utilized to evaluate significance, defined as p < 0.05.

***These numbers were the ratios of student scores’ average to the total average score.

 

 

Comparing the three years’ exam scores reveals the medians of the percentage of correct answers in 2017, 2018, and 2019 were 73.3%, 90.0%, and 93.3%, respectively (Figure 1). As the GPA in 2018 was significantly different from 2017 and 2019, we conducted an ANCOVA with GPA as the covariance (Figure 2). We found a significant difference among the scores associated with the three cohorts (p < 0.001). However, there was no significant difference between the AUD and FLP+AUD groups (p = 0.487). Post hoc power analysis showed that our study’s power was 0.916, indicating that the Type II error was less than 0.1.

 

 

Figure 1. Percentage of correct answers.

 

Figure 2. Analysis of covariance with GPA.

 

 

TRA is expressed as traditional face-to-face, FLP+AUD as flipped classroom plus AUD, and AUD as audience response system.

The students’ response rates for both the AUD and FLP+AUD lectures decreased significantly over the four classes (C1–4: in the one-way ANOVA results for the AUD lectures: C1 = 84.7%, C3 = 76.5%, and C4 = 68.1%; < 0.001 and in the one-way ANOVA results for the FLP+AUD lectures: C1 = 87.8%, C2 = 80.0%, C3 75.0%, and C4 = 66.5%; p < 0.001). The response rates for the C2 AUD lectures were not obtained due to unexpected technical network issues that occurred in the class.

The pre-lecture video views gradually decreased; however, the viewing time lengths varied by topic. Some students reviewed the videos after the lectures (Table 2).

 

 

Table 2. Viewing times and video views, pre-and post-lecture, for FLP plus AUD lectures

 

Video Time

Pre-lecture

 

Post-lecture to Exam

Audience Retention (%)

 

Viewing time

Video Views

Viewing time

Video Views

1st

10 m, 16 s

9 m, 12 s

133

 

4 m, 06 s

41

44.4

2nd

10 m, 50 s

7 m, 06 s

106

 

2 m, 54 s

28

43.1

3rd

13 m, 16 s

10 m, 18 s

84

 

2 m, 54 s

21

56.4

4th

10 m, 18 s

5 m, 30 s

50

 

2 m, 54 s

25

66.9

1st: Hypothalamus, Pituitary abnormality

2nd: Adrenal Cortical Hormone, Adrenal Medulla Hormone

3rd: Adrenal Cortical Dysfunction (1)

4th: Adrenal Cortical Dysfunction (2)

 

 

The results from anonymous questionnaire surveys of the AUD, FLP, and AUD lectures show that students’ self-described readiness, enjoyment, fulfillment, and willingness to talk to others were significantly higher for the FLP+AUD lectures than for the AUD-only lecture (p < 0.001, p = 0.019, p = 0.006, and p = 0.005, respectively) (Figure 3).

 

 

a)

b)

c)

d)

e)

f)

g)

h)

i)

j)

Figure 3. Anonymous Questionnaire Survey.

(n= 37 and 50 for AUD and FLP + AUD, respectively).

 

 

This study compared the FLP+AUD, TRA, and AUD lecture styles to determine the most effective teaching method for pharmaceutical students at a Japanese university. Final exam scores showed that AUD and FLP+AUD, both “active learning methods,” were more effective than TRA. We believe that our study is the first to compare the three teaching methods. Post hoc power analysis showed that our study’s power was 0.916, indicating that the Type II error was less than 0.1. We found no significant difference in learning outcomes between the AUD and the FLP+AUD groups. Therefore, AUD and FLP + AUD had the same learning outcomes.

Numerous studies have shown that AUD is an effective instructive method in classrooms [19-26]. Rubio et al. elucidated that long-term retention of learned topics was sustained for up to 3 months [26]. However, Doucet et al. reported after 1 year, post-exam scores decreased significantly [20]. Hussain and Wilby’s systematic review reported that AUD improved students’ immediate recall following an educational activity, but the effects were not sustained [23]. Thus, the AUD method appears best suited for acquiring a short-term understanding. This study used endocrine system exam scores as a summative evaluation of the three teaching methods. The results demonstrated that AUD and FLP+AUD were equivalent teaching methods for short-term knowledge retention.

The students’ response rates for AUD in classrooms decreased significantly over time, both in the AUD and FLP+AUD lectures. Most of the students were familiar with AUD from previous classes. Therefore, some ingenuity in the AUD style might be necessary to maintain their attention and improve their academic performance. An example such as Jeopardy! style review game may be effective in overcoming this concern [27].

Both AUD and FLP+AUD effectively increased students’ learning motivation (Figure 3). Readiness was significantly higher in the FLP+AUD group than in the AUD group. This finding is reasonable because the combined FLP+AUD method required students to watch videos before attending classes. Some students also watched the videos after the lectures and before taking their exams (Table 2), thus suggesting that FLP videos are an effective tool for reviewing materials outside of class. If faculty members encourage students to watch post-lecture videos, FLP+AUD could be an even better teaching method. The students reported finding the FLP+AUD videos enjoyable and fulfilling. The videos used in the classes in this study were short (the longest was 13 m:16 s). Similar research has confirmed that positive results are associated with reasonable runtimes to retain students’ interests [5, 7].

Fryer revealed that incentives for scholarly inputs, including good behavior, attendance, and wearing uniforms were more effective than those for scholarly outputs, such as better grades [28]. For example, Giuliano et al. reported that increasing the weight of quizzes from 7.5% to 15% of the overall grade to emphasize the importance of pre-class readiness positively affected outcomes [2]. Our research did not offer incentives to students for scholarly input or output. Fryer’s research suggests that offering incentives to pharmaceutical students for scholarly inputs could improve outcomes, and future studies should examine this effect in the pharmacology course.

This study had some limitations. First, we did not track the costs of preparing FLP videos to assess whether the time and labor required represented a significant burden on teachers. There were no monetary costs associated with creating the video files in PowerPoint (Redmond, WA, USA: Microsoft Corporation, USA) or using YouTube or Clica. However, we spent a significant amount of time preparing the videos (e.g., recording and exporting video files from a PowerPoint file). Two lecturers conducted this study; however, we cannot predict whether all faculty members could cost-effectively employ the teaching method without a comprehensive evaluation of the required resource consumption. One potential option for expanding the use of this method could be having teaching assistants produce videos [29]. One prior study reported that implementing an FLP required 127% more faculty time than the previous year’s TRA lectures, but that time could decrease once a course has been established and utilized for multiple quarters or even years [30]. Second, this study gained insights into the short-term effects of FLP+AUD on knowledge retention, but the long-term effects were not examined. We recommend future research, particularly longitudinal studies that examine the long-term effects, to refine future academic strategies.

Conclusion

This study examined whether combining FLP with AUD might produce better learning outcomes than either method alone and compared with the TRA and AUD styles. We found that the active learning methods, namely AUD and FLP+AUD, were the most effective teaching methods among the three based on the endocrine system exam scores for P2 students in a pharmacology course. Both methods had the same learning outcomes for short-term knowledge retention.

Acknowledgments: We are indebted to Prof. Hayasaka, who provided technical help and encouragement.

Conflict of interest: None

Financial support: None

Ethics statement: This study was approved by the university ethics committee (No. 220). At the end of the last lecture, the lecturers explained the purpose of the study to the students and obtained written informed consent from the students who were willing to participate.

References

1.       The Ministry of Education, Culture, Sports, Science, and Technology. Toward a qualitative change in university education to build a new future. Accessed February 12, 2020. Available from: https://www.mext.go.jp/b_menu/shingi/chukyo/chukyo0/toushin/1325047.htm

2.       Giuliano CA, Moser LR. Evaluation of a Flipped Drug Literature Evaluation Course. Am J Pharm Educ. 2016;80(4):66. Available from: https://pubmed.ncbi.nlm.nih.gov/27293233/

3.       Koo CL, Demps EL, Farris C, Bowman JD, Panahi L, Boyle P. Impact of Flipped Classroom Design on Student Performance and Perceptions in a Pharmacotherapy Course. Am J Pharm Educ. 2016;80(2):33. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4827584/

4.       González A. Turning a traditional teaching setting into a feedback-rich environment. Int J Educ Technol High Educ. 2018;15:32. Available from: https://educationaltechnologyjournal.springeropen.com/articles/10.1186/s41239-018-0114-1

5.       Kugler AJ, Gogineni HP, Garavalia LS. Learning Outcomes and Student Preferences with Flipped vs Lecture/Case Teaching Model in a Block Curriculum. Am J Pharm Educ. 2019;83(8):7044. Available from: https://pubmed.ncbi.nlm.nih.gov/31831896/

6.       Lichvar AB, Hedges A, Benedict NJ, Donihi AC. Combination of a Flipped Classroom Format and a Virtual Patient Case to Enhance Active Learning in a Required Therapeutics Course. Am J Pharm Educ. 2016;80(10):175. Available from: https://pubmed.ncbi.nlm.nih.gov/28179724/

7.       Wong TH, Ip EJ, Lopes I, Rajagopalan V. Pharmacy students' performance and perceptions in a flipped teaching pilot on cardiac arrhythmias. Am J Pharm Educ. 2014;78(10):185. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4315207/

8.       Sergeevna SM, Efimovna LE. Improving training of pharmaceutical specialists for consultation in pharmacy organizations using interactive forms of education. Pharmacophore. 2020;11(2):7-14. Available from: https://pharmacophorejournal.com/article/improving-training-of-pharmaceutical-specialists-for-consultation-in-pharmacy-organizations-using-interactive-forms-of-education

9.       Faller EM, Hernandez MT, Hernandez AM, Gabriel JR. Emerging roles of pharmacists in global health: an exploratory study on their knowledge, perception, and competency. Arch Pharm Pract. 2020;11(1):40-6. Available from: https://archivepp.com/storage/models/article/Bhl2fYgnQmZOI4BeNtQzeLH1wImn7jkfLJnuwzpeHcqaJbIz9hUdV4pr8ItR/emerging-roles-of-pharmacist-in-global-health-an-exploratory-study-on-their-knowledge-perception-a.pdf

10.    Bossaer JB, Panus P, Stewart DW, Hagemeier NE, George J. Student Performance in a Pharmacotherapy Oncology Module Before and After Flipping the Classroom. Am J Pharm Educ. 2016;80(2):31. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4827582/

11.    Khanova J, McLaughlin JE, Rhoney DH, Roth MT, Harris S. Student Perceptions of a Flipped Pharmacotherapy Course. Am J Pharm Educ. 2015;79(9):140. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4727361/

12.    Flores Ò, del-Arco I, Silva P. The flipped classroom model at the university: analysis based on professors’ and students’ assessment in the educational field. Int J Educ Technol High Educ. 2016;13:21d. Available from: https://educationaltechnologyjournal.springeropen.com/articles/10.1186/s41239-016-0022-1

13.    Kurihara Y. Flipped Classroom: Effects on Education for the Case of Economics. J Educ e-Learn Res. 2016;3(2):65-71. Available from: https://files.eric.ed.gov/fulltext/EJ1148450.pdf

14.    Nouri J. The flipped classroom: for active, effective, and increased learning – especially for low achievers. Int J Educ Technol High Educ. 2016;13:33. Available from: https://educationaltechnologyjournal.springeropen.com/articles/10.1186/s41239-016-0032-z

15.    Pattanaphanchai J. An Investigation of Students' Learning Achievement and Perception using Flipped Classroom in an Introductory Programming Course: A Case Study of Thailand Higher Education. J Univ Teach. Learn. 2019;16(5):Article 4. Available from: https://ro.uow.edu.au/cgi/viewcontent.cgi?article=1878&context=jutlp

16.    Wozny N, Balser C, Ives D. Evaluating the Flipped Classroom: A Randomized Controlled Trial. J Econ Educ. 2018;49(2):115-29. doi:10.1080/00220485.2018.1438860

17.    Hew KF, Lo CK. Flipped classroom improves student learning in health professions education: a meta-analysis. BMC Med Educ. [Internet]. 2018 Mar [cited 2020 Nov 21];18(1):38. Available from: https://bmcmededuc.biomedcentral.com/articles/10.1186/s12909-018-1144-z.

18.    Kanda Y. Investigation of the freely available easy-to-use software 'EZR' for medical statistics. Bone Marrow Transplant. 2013;48(3):452-8. Available from: https://pubmed.ncbi.nlm.nih.gov/23208313/

19.    Cain J, Black EP, Rohr J. An audience response system strategy to improve student motivation, attention, and feedback. Am J Pharm Educ. 2009;73(2):21. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2690899/

20.    Doucet M, Vrins A, Harvey D. Effect of using an audience response system on learning environment, motivation and long-term retention, during case-discussions in a large group of undergraduate veterinary clinical pharmacology students. Med Teach. 2009;31(12):e570-9. Available from: https://pubmed.ncbi.nlm.nih.gov/19995158/

21.    FitzPatrick KA, Finn KE, Campisi J. Effect of personal response systems on student perception and academic performance in courses in a health sciences curriculum. Adv Physiol Educ. 2011;35(3):280-9. Available from: https://pubmed.ncbi.nlm.nih.gov/21908838/

22.    Gauci SA, Dantas AM, Williams DA, Kemm RE. Promoting student-centered active learning in lectures with a personal response system. Adv Physiol Educ. 2009;33(1):60-71. Available from: https://pubmed.ncbi.nlm.nih.gov/19261762/

23.    Hussain FN, Wilby KJ. A systematic review of audience response systems in pharmacy education. Curr Pharm Teach Learn. 2019;11(11):1196-204. Available from: https://pubmed.ncbi.nlm.nih.gov/31783969/

24.    Liu FC, Gettig JP, Fjortoft N. Impact of a student response system on short- and long-term learning in a drug literature evaluation course. Am J Pharm Educ. 2010;74(1):6. Available from: https://www.ajpe.org/content/74/1/6

25.    Mostyn A, Meade O, Lymn JS. Using Audience Response Technology to provide formative feedback on pharmacology performance for non-medical prescribing students--a preliminary evaluation. BMC Med Educ. 2012;12:113. Available from: https://pubmed.ncbi.nlm.nih.gov/23148762/

26.    Rubio EI, Bassignani MJ, White MA, Brant WE. Effect of an audience response system on resident learning and retention of lecture material. AJR Am J Roentgenol. 2008;190(6): W319-22. doi:10.2214/AJR.07.3038

27.    Cusick J. A Jeopardy-Style Review Game Using Team Clickers. MedEdPORTAL. 2016;12:10485. doi:10.15766/mep_2374-8265.10485

28.    Fryer RG. Financial Incentives and Student Achievement: Evidence from Randomized Trials. Q J Econ. 2011;126(4):1755-98. Available from: https://scholar.harvard.edu/fryer/publications/financial-incentives-and-student-achievement-evidence-randomized-trials

29.    Spangler J. Costs related to a flipped classroom. Acad Med. 2014:89(11)1429(11). Available from: https://journals.lww.com/academicmedicine/Fulltext/2014/11000/Costs_Related_to_a_Flipped_Classroom.2.aspx

30.    Liaw SS, Huang HM. How Factors of Personal Attitudes and Learning Environments Affect Gender Difference toward Mobile Distance Learning Acceptance. Int Rev Res Open Dis. 2015;16(4):104-132. Available from: https://www.researchgate.net/publication/283486483_How_Factors_of_Personal_Attitudes_and_Learning_Environments_Affect_Gender_Difference_toward_Mobile_Distance_Learning_Acceptance

 


Contact SPER Publications


SPER Publications and Solutions Pvt. Ltd.

HD - 236,
Near The Shri Ram Millenium School,
Sector 135,
Noida-Greater Noida Expressway,
Noida-201301 [Delhi-NCR] India