Introduction 

Recently, active learning methods have been applied to higher education levels worldwide [1-3]. The Japanese government for education has also recommended that universities adopt active learning methods [4]. In particular, the utilization of information and communication technology (ICT) has been recommended. A flipped classroom (FC) is an active learning method for ICT utilization. The utilization of an FC enhanced the learning performance of pharmacy students [1, 5-8]. Formative assessment (FA), used in higher education levels in Japan [9, 10], enhances learning outcomes. However, it remains unclear which teaching method is the most effective in enhancing learning outcomes. Additionally, the impact of different teaching methods on the long-term retention of knowledge has also been an interesting issue in higher education. It has been reported that the knowledge retention following active learning, including the FC or an audience response system, was three months [11]. Therefore, the current study aim to examine the effect of teaching methods such as FA and FC lectures on the short- and long-term retention of Japanese P4 students (fourth-year students) in a drug information course. Pharmacist education in Japan is a six-year program, and P4 students study this course before starting their pharmacy practice in P5.

Materials and Methods

P4 students who chose the drug information course at the Ohu University from 2018 to 2020 were the target of this study. This course consisted of one lecture per week for seven weeks (90 minutes/class) during spring quarter. The FA lectures were in 2018, and FC lectures were carried out in 2019 and 2020. To obtain academic background information concerning P4 students in 2018, 2019, and 2020, prerequisite grade point averages (GPAs) were collected.

The drug information course is knowledge-based, and so it was expected that P4 students fully understood the drug information concept. The course consisted of seven lectures: “Drug Information and Drug Information Activities,” “Drug information regarding the development process of drugs and after marketing of drugs,” “Drug information sources,” “Patient information and its sources,” “Evidence-based medicine,” “Statistics for evaluating statistical information,” and “Tailor-made pharmacotherapy.”

FA lectures were conducted as traditional in-person lectures. A lecturer explained topics with handouts and PowerPoint presentations. Then, a paper-based reflection quiz was administered at the end of the lecture for about ten minutes as the formative assessment. Correct answers were provided immediately at the end of the lecture period.

FC was conducted in 2019. Prerequisite videos were uploaded on YouTube, and at least three days before the lecture, the URL of each video was emailed to students. The length of the first, second, third, fourth, fifth, sixth, and seventh videos was 10 min and 52 s, 17 min, and 33 s, 24 min, and 21 s, 26 min and 2 s, 20 min, and 19 s, 14 min, and 40 s, and 24 min and 14 s, respectively. Data on the number of views were retrieved from YouTube Analytics. In class, start of each lecture, multiple-choice questions were given to students through Clica^TM (http://clica.jp/LP/) from the beginning of each lecture. The lecturer and students saw their responses on the screen. Using handouts, the lecturer added extra explanations on the topics depending on the students’ incorrect responses. 

FC with incentives was implemented in 2020. Prerequisite videos that were identical between 2019 and 2020 were uploaded to C-learning^TM (NETMAN. Co. Ltd, Japan), which has been applied as an ICT system at Ohu University School of Pharmaceutical Sciences. Additionally, a pre-quiz regarding the topic was uploaded to C-learning^TM at the same time. The watching and answering tracks of each student were monitored in C-learning^TM. As an incentive, the lecturer awarded one point per topic, which was added to the final exam score if a student watched prerequisite videos as well as answered the pre-quizzes before attending class. This strategy was based on Fryer’s report that incentives provided for educational inputs such as attendance, good behavior, or wearing uniforms are more effective than those for educational outputs like better grades [12]. In class, at the start of each lecture, multiple-choice questions were given to students through C-learning^TM. The lecturer and students viewed their responses on the screen and, depending on the students’ incorrect responses, the lecturer added extra explanations using handouts of the topics. Therefore, the most significant difference between FC and FC with incentives was the incentive for educational inputs. When comparing the learning outcomes between methods, scores lacking incentive points for the FC with incentives were utilized.

The primary outcome of the study was to examine the short-term retention and long-term retention using an internal summative assessment of the drug information course and external exams regarding drug information section between three ways of teaching, i.e., FA lectures, FC lectures, and FC with incentive lectures. Exam schedules of drug information courses for P4 students at Ohu University are shown in Figure 1. The internal summative assessment was used to evaluate the 1^st short-term retention. The 1^st external exam was used to evaluate long-term retention due to a minimum four-month interval from the internal summative assessment. After three review lectures were conducted, the 2^nd and the 3^rd external exams were used to evaluate the 2^nd short-term retention. Correct answered percentages were used to evaluate the internal summative assessment. Questions comprising the internal summative assessment were identical among the three cohorts; however, the questions and multiple-choice answers were randomly reshuffled. Adjusted ratios by national average were utilized for the evaluation of external exams.

The appropriate ethics committee approved this study (No. 260). At the end of the final lecture, the lecturer wrote a letter to the students to inform them about the aim of the study. The students who were interested in participating in the study gave a written informed consent. Following the relevant regulations and guidelines, all methods were performed.

Using G*Power software, the sample size for the study was calculated. To carry out the calculation, a One-Way Analysis Of Variance (ANOVA) with an effect size of 0.25, an alpha error probability of 0.05, power of 80%, and the number of groups being set to three was used. The total sample size was set at 159.

ANOVA was conducted for parametric distribution data. Kruskal-Wallis test was performed for nonparametric distribution data. Using the Fisher’s exact test and chi-square test, the categorical data were analyzed. If the background information between the three cohorts was significantly different, ANCOVA would be performed. A p-value (p) < 0.05 was considered statistically significant. Levene’s test was performed to investigate the degree of variance. with A graphical user interface, EZR (“Easy R”) (Saitama Medical Center, Jichi Medical University, Saitama, Japan), employed for the programming language R (The R Foundation for Statistical Computing) was used to perform all statistical analyses [13].

Results and Discussion

Table 1 shows the history of P4 students in 2018, 2019, and 2020. Notably, each year, gender differences were not significantly different and the total number of included students exceeded 159 (χ ² = 0.23, p = 0.89). However, the prerequisite GPA for each year was significantly different (p < 0.001). The number of students who video-viewed 100% for the FC and the FC with incentives was 47% and 71%, respectively (χ ² = 7.29, p < 0.01).

¹ To determine significance, Chi-square or Fisher exact test was used, defined as p < 0.05.

² To determine significance, Kruskal-Wallis test was used, defined as p < 0.05.

³ Students with viewing rates of 100% were counted.

For the first evaluation of short-term retention, the correct answered percentages of the internal summative assessments are shown in Figure 2a. ANCOVA could not be performed because there was a significant interaction between the group variables and covariance. A significant difference was noted between the three teaching methods (p = 0.004). Furthermore, a significant difference was found between FC and FC with incentives (p = 0.004). Conversely, no significant difference was detected between the FA and FC with incentives (p = 0.13). Levene’s test showed a significant difference between the three teaching methods (p < 0.001), which means that the variance of the FC with incentives was the smallest among the three methods.

To evaluate the long-term retention, adjusted ratios of the 1^st external exam scores for the drug information section were compared between the three teaching methods. Figure 3 shows that since the GPAs between 2018, 2019, and 2020 were significantly different, and there was no significant interaction between group variables and covariance, ANCOVA with GPA as covariance was carried out. No significant difference was detected between the three teaching methods (p = 0.49).

For the second evaluation of short-term retention, adjusted ratios of the 2^nd and 3^rd external exam scores for the drug information section were compared among the three teaching methods. ANCOVA with GPA as the covariance was carried out, as shown in Figure 2b. A significant difference was observed between the three teaching methods (p < 0.001), and between FC and FC with incentives (p < 0.001). Meanwhile, no significant difference was observed between the FA and FC with incentives (p = 0.34).