ABBREVIATIONS
Objective
structured clinical examination (OSCE)
Computer-based
testing (CBT)
Ministry of Education, Culture, Sports, Science and Technology
(MEXT)
Ministry of Health, Labor and Welfare (MHLW)
Common Assessment and
Testing Organization for medical colleges（CATO）
Supervisor (SV)
Simulated patient (SP)
Item response theory
(IRT)
INTRODUCTION
Educational
courses for training radiologic technologists in Japan
As of April 2023, the Ministry of
Education, Culture, Sports, Science and Technology (MEXT) and the Ministry of
Health, Labor and Welfare (MHLW) have jurisdiction over the training schools
for radiologic technologists that meet the government's designation
regulations. On the one hand, training schools under the jurisdiction of the
MEXT are four-year universities, which generally have graduate schools and
focus on research and development. On the other hand, training schools under
the jurisdiction of the MHLW include three- and four-year vocational schools,
which focus on the acquisition of practical skills. Both training schools offer
educational programs to acquire the knowledge and skills necessary to become a
radiologic technologist, and after graduation, students must pass a national
examination in order to practice as a radiologic technologist.
Objective
Structured Clinical Examination (OSCE) and Computer Based Testing (CBT) in
Japan
In Japan, a shared examination to assess
whether medical students possess the required knowledge and skills before
starting clinical training will be implemented from May 2023 under the guidance
of a publicly established organization (the Common Assessment and Testing
Organization for Medical Colleges: CATO), (Ordinance of the Ministry of Health,
2023). The shared examinations include the OSCE and the CBT. While the CBT is
an examination to assess the level of knowledge of students, the OSCE evaluates
the level of knowledge of students. Moreover, CBT examines students' knowledge
levels, whereas OSCE is a practical examination that evaluates students'
emotions, attitude, and skills, and a supervisor (SV) evaluates the candidate
who performs the practical examination on a simulated patient (SP) using a
standardized method. Presently, the OSCE and CBT in Japan are preceded by
medical and dental education. Notably, OSCE/CBT is also being introduced in the
education of other healthcare professions, and it is being attempted in the
education of pharmacists (Tokunaga et al.,2011),
physical therapists (Kamioka et al.,2011), nurses
(Takashima & Arao, 2021), optometrists (Takasaki
et al.,2008), and dietitians (Kawakami et al.,2008). In the field of radiologic
technology education, some examples of OSCE/CBT practices have been reported
and some results have been recognized (Sekine et al.,2007, Hatakeyama et al.,2009,
Muto, 2012, Eto et al.,2021, Aoyama et al.,2022).
Aim of the study
This
paper presents the results of a literature survey on the current status of
preclinical training evaluation systems for radiologic technology education in
Japan, and the educational effects of the OSCE/CBT, introduced in 2018 as a
preclinical training evaluation system to comprehensively evaluate students'
skills, attitudes, and knowledge at the training institution, in the credit
certification examination with regard to on-campus training.
Current status of preclinical training in Japan
(on-campus training)
(1)
General on-campus practice
During
the on-campus training program at a radiologic technologist training school,
students acquire the following skills and knowledge:
1) Patient treatment skills: This includes the ability to communicate
with patients, understand their conditions, and respect their privacy. Through
practical training and role-playing using SPs and human phantoms, trainees
learn how to interact with patients and provide appropriate explanations.
2) Ability to respond to examinations: This includes the ability to
perform accurate and prompt examinations, understand positioning techniques
necessary during imaging, and comprehend the setting of imaging conditions.
Through practical training using the human phantom and SP, the trainees will
acquire the techniques and skills necessary for examinations.
3) Clinical image reading skills: This includes the ability to judge
whether images are normal or abnormal, how to evaluate image findings, and how
to prepare reports. Clinical image evaluation skills are improved through
training in reading images related to imaging anatomy and images of major
diseases.
Fujii et al. of Fujita Health University studied
the usefulness of role-plays in patient care for students prior to clinical
practice and reported that role-play practice had an immediate effect on the
acquisition of communication skills such as emotional transference (Fujii
et al.,2003), while
Kawamura et al. of Ibaraki
Prefectural University of Health Sciences reported
that the addition of
evaluation of patient care by simulated patients to OSCE enabled multifaceted
evaluation and was expected to improve educational effectiveness (Kawamura et al.,2011).
Thus,
through on-campus training, students comprehensively acquire the skills,
knowledge, and attitudes required for clinical practice and are able to provide
safe, high-quality care to patients.
(2)
Examples of on-campus training in our school
The
on-campus training at the school is conducted in the form of cross-training, whereby
students take turns taking on the roles of the patient and technician and learn
photography skills such as hospitality and positioning.
Four
faculty members with practical experience as radiologic technologists are in
charge of the on-campus training. Notably, 40 students per class are divided
into groups of 10, and practical skills instruction for each modality is
provided in rotation. The educational content of the school consists of
radiography, computer tomography (CT), magnetic
resonance imaging (MRI), ultrasound equipment, X-ray TV, mammography, non-mydriatic fundus camera, bone densitometry, and procedures
that extend the duties of radiological technologists (needle removal and
hemostasis, insertion of catheters through the anus).
In
the on-campus training, students are required to write a "Preparation
Report" to be submitted before the training and a "Practice
Report" to be submitted after the training as assigned studies. The
preliminary report is to be submitted to the supervising instructor by the day
before the practical training and is based on prior studies of the knowledge
required for the practical training, such as imaging methods, modality
characteristics, and image anatomy. At the introduction of the practical
training, the instructor asks oral questions to the students to check their
level of understanding. This makes the students understand what they will learn
in the practical training and allows them to approach each training session
with a sense of urgency.
Current status of OSCE in Japan
(1)
General OSCE Practice
Osaka
Butsuryo University conducts a clinical skills
education program using simple chest X-ray examination as an example. In this
program, practical examinations are conducted for SPs, including explanations
before examinations, safe positioning, setting of imaging conditions,
wheelchair operation and assistance with movement on the bed, assuming imaging
operations for postoperative patients. In addition, a written examination on normal
image anatomy and clinical images of major diseases displayed on an imaging
monitor is attempted (Yamaguchi
et al.,2015).
At
Ibaraki Prefectural University, third-year students prior to clinical
practice are evaluated on patient care,
radiographic and nuclear medicine examination techniques, radiotherapy site
positioning, and dose calculation. Notably, SPs are students (senior
fourth-year students) and faculty members who are instructed to ask appropriate questions once during
the examination,
and SVs use the comments from SPs as a reference for scoring. After the completion of
clinical practice, an evaluation of clinical images of X-ray, CT, MRI, and SPECT is added in the form of an oral examination by the SV (Hatakeyama
et al.,2007).
The level of knowledge of
clinical imaging is included as part of the OSCE in both universities,
and CBT has not yet been introduced for assessment.
(2)
Examples of OSCE implementation in our school
In
our OSCE, the faculty member in charge of practical skills instruction in
on-campus training was the SV, and the SP was a student (a second-year student
in the Department of Radiologic Technology, the same as the examinee). As the
OSCE was conducted as a credit examination for on-campus training, the
evaluation was not conducted by the SPs who were students, and any questions or
complaints from the SPs were considered by the SVs and included in the
evaluation.
As
the venue for the practical examination, four radiography examination rooms
(hereinafter referred to as "stations") were prepared for each
modality, and one SV, one SP, and one PC for progress management were assigned
to each station. Students participated in the OSCE in the roles of both
examinee and SP and worked on the tasks performed at each station.
The
OSCE timer (Dolphin ver. 1.50, Japan), a commercially available software, was
used to manage the OSCE time. This enabled automatic and simultaneous OSCE time
management by connecting multiple PCs for progress management via LAN and
synchronizing the countdown display on the display with voice announcements.
To
ensure that the OSCE assignments were consistent with the content taught in the
on-campus training, assignments were prepared from radiography, CT, MRI,
ultrasound, mammography, and X-ray TV modalities and made available to
examinees two weeks before the OSCE. Examinees were allowed to cross-train on
all the tasks prepared in the stations prior to the OSCE. However, in the
actual OSCE, it was difficult for the candidates (80 students in 2 classes) to
complete all the tasks in all the stations and for the SVs (4 SVs) to evaluate all
the tasks due to the heavy workload and time constraints of the SVs. Therefore,
just before the OSCE, one set of examinees (4 persons) drew lots to determine one
station to be evaluated.
Current status of CBT in Japan
(1)
General CBT practice
In
the common medical and dental examinations in which public CBTs have been
introduced, the CBT consists of a huge pool of questions that differ from one
examinee to another. For this reason, item response theory (IRT) is used to
correct scores according to difficulty. In the education of radiologic
technologists, no official CBT has been introduced, and each school employs its
own method.
Sekine reported on the
construction of a CBT system using e-learning software at Tokyo Metropolitan University.
In this report (Sekine.,2011), a database of national
examinations (about 1,400 questions) was compiled as a set of questions, and a
system was constructed to extract and submit 100 questions focused on practical
training subjects. The system was implemented before and after clinical
practice, and the results of an analysis of changes in the score rate showed a
significant increase in the category of "Radiographic Techniques."
(2)
Examples of CBT in our school
The
school's CBT utilized an infrared remote-controlled class response system,
Clicker Nao (Fine Woods, Japan). This system is
linked to PowerPoint (Microsoft, USA) via an add-in, enabling the display of
examination question slides created in PowerPoint on a computer display and
script control. The examinee uses the numeric keypad on the clicker terminal
(Model-H) to wirelessly transmit answers to a controller connected to the
computer.
The
format of the CBT is a "five-choice" format, in which the examinee
chooses one correct answer from among five options. The questions were
presented consecutively on the computer display at regular intervals, and the
examinee was not allowed to go back and answer the questions once the next
question was displayed. The CBT took approximately 30 minutes for 50 questions,
including practice questions to confirm the operation of the clicker terminal.
Because
all examinees answered the same questions in our CBT, the questions were not
weighted according to their difficulty level but were evaluated (50 points)
using a point system in which one correct answer to a question was worth one
point.
Criteria for passing the preclinical
evaluation (OSCE and CBT)
(1)
General acceptance criteria
In
Japan, the standard of achievement for both the CBT and the OSCE will be
defined as a passing score for the shared examination. This unified standard
will be communicated to examinees and examiners, and IRT359 (a score of
approximately 65%) is expected to be the minimum passing standard.
In
the radiologic technology field, Osaka Butsuryo University
uses a quantitative evaluation (2 levels) and a qualitative evaluation (5
levels) for the achievement level of skills and attitudes in the OSCE
(Yamaguchi et al.,2015). The quantitative evaluation
is based on the completion of all items, and the qualitative evaluation is
based on the average value of all items being 3.0 or higher, with no
"1" (inferior) being scored. The achievement level of knowledge is
evaluated quantitatively for each question (2 levels), and the passing criterion
is a correct answer rate of 60% or higher.
(2)
Criteria for acceptance in our school
The
school has introduced the OSCE/CBT as a preclinical training credit evaluation
and conducts it twice a year as a regular examination in the first and second semesters.
The grades for on-campus training are evaluated not only by the OSCE and CBT
but also by a comprehensive evaluation that includes attendance and the
evaluation of training reports. The evaluation ratio presented in the syllabus
is OSCE (20%), CBT (20%), attendance (20%), and
practice report (40%). Particular emphasis was placed on the evaluation of the
practical training reports, which students spend a lot of time writing and
teachers put a lot of effort into correcting and guiding them. At the end of
the semester, the grade was calculated by multiplying the percentage of each
evaluation and converting it to a perfect score of 100 points. Grades were A
(100–80 points), B (79–70 points), C (69–60 points), and D (59 points or less),
with C or above being passing and D being failing. Students who failed the OSCE
and CBT were reexamined and reevaluated by submitting a report on their
assignments.
MATERIALS AND METHODS
Determining the effectiveness of preclinical
practice assessment in our school
(1)
Survey subjects
The
subjects of the survey were third-year students in the Department of Radiologic
Technology who graduated from the school in March 2022, passed the preclinical
evaluation (twice/year) in the second-year on-campus training conducted by the
school, and experienced 10 weeks of clinical training in the third year.
(2)
Ethical considerations
The
purpose of this study was fully explained to the subjects in writing and
orally, and informed consent was obtained before the questionnaire survey was
conducted. In addition, a check box was provided in the answer column to
indicate whether or not the questionnaire results would be used for research
purposes, and consent was obtained from all respondents. This study was
conducted with the approval of the Ethical Review Committee.
(3)
Survey methodology
The
survey items were 1) students' positioning experience in clinical practice, 2)
modalities and sites where positioning was experienced, and 3) experiences of preclinical
training (on-campus training, OSCE, CBT) that were useful in clinical practice.
A web questionnaire form (Google Form, USA) was adopted. The responses to the
open-ended questions were categorized and tabulated. The questionnaire was
collected from March 15 to 31, 2022.
RESULT
Effectiveness assessment of preclinical
practice evaluation in Japan
A
post-clinical training questionnaire survey was conducted on students who
received preclinical training at Osaka Butsuryo
University11. Eighty-two students responded to this survey, yielding
a valid response rate of 98.8%. To Question 1, "Was the clinical skills
training provided before clinical practice useful?" 67 students (81.7%)
responded "strongly agree" or "agree.” To Question 2, "Which
of the clinical skills education items were particularly useful?” 41 (50.0%),
50 (1.0%), 39 (47.6%), and 26 (31.7%) students responded that they found the
training useful, suggesting that the training had an effective educational
effect on the students participating in the clinical training.
Evaluation of the effectiveness of the
preclinical training assessment in our school
In
a survey conducted after clinical practice at the university, 44 out of 59
students responded, for a response rate of 74.6%, with 28 male students (63.6%)
and 16 female students (36.4%).
(1)
Students' positioning experiences in clinical practice
During the clinical training period,
students experienced an average of 26.7 positioning sessions per 10 weeks. The
breakdown of the number of positioning experiences (number of students) was as
follows: 0 (3 students) for observation only, 1–10 (13 students), 11–50 (22
students), and 51 or more (6 students). Some students indicated that they had
more than 100 positioning experiences. As shown above, the implementation
status of the medical participation training in terms of students' positioning
experience varied significantly from one facility to another.
(2)
Modalities and sites where positioning was experienced
The modality in which students
experienced positioning in clinical practice (number of experiences: multiple responses)
was radiography most frequently (41 students), followed by CT (13 students),
MRI (3 students), and DEXA bone densitometry (2 students). In terms of the part
of the body experienced (number of patients with multiple experiences), in radiography,
the chest was the most common (37 patients), followed by the abdomen (14
patients), lumbar spine (10 patients), knee (9 patients), and hand, shoulder,
cervical spine, foot, hip, and head all had 5 or fewer patients. In CT
positioning experience, nearly half of the patients had positioning experience
in the head (5 patients), followed by the chest (3 patients), abdomen (2
patients), and unknown
(3
patients).
In
MRI positioning experience, patients had positioning experience in the abdomen
(2 patients), followed by the neck (1 patients). None of the students had
experience in positioning for mammography, X-ray TV, ultrasound equipment, nor
non-mydriatic fundus cameras. Furthermore, none of
the students had experience in positioning for nuclear
medicine examinations nor radiation therapy.
(3)
Experience with preclinical education (on-campus training, OSCE, CBT) that was
useful in clinical practice
Figure 1 shows the modalities that students
indicate are useful in their clinical practice (multiple responses). In order
of the positioning experience in clinical practice, the most common responses
were: radiography, CT, and MRI. In the case of MRI, 2.6 times more students
answered that their on-campus training experience was useful than their OSCE
experience.
Figure 2 shows the positioning areas
experienced in clinical practice and the body part of radiography that are
useful in on-campus training (multiple answers) in the radiography class. As a
result, most students answered thorax, abdomen, lumbar spine, and knee in order
of their positioning experience in clinical practice. However, many students
answered that their positioning experience in clinical practice was useful in
the hip, head, and breast, where they had little experience in positioning in clinical
practice. All of the students who answered that breast positioning was helpful
were female.
DISCUSSION
The student's positioning experience
According to a previous survey, the current status of clinical training
in the education of radiologic technologists is that the majority of clinical
training is either observation-type clinical training or simulated clinical
training conducted outside the framework of actual medical practice because
students do not have a sufficient level of clinical practice ability to
participate in medical practice, and the guidance system of clinical training
facilities is insufficiently developed (Tokushige
& Hoshino, 2021). However, our
survey conducted in this study revealed that the majority of students (41 out
of 44 students: 93%) had positioning experience in clinical practice.
Many students at four-year universities experience clinical training at
medical institutions affiliated with universities (university hospitals). University
hospitals have educational, research, and clinical functions. While it is an
attractive facility that provides adequate education, the radiology department
is highly occupied during the daytime, and there is often insufficient time for
teaching and supervising students in addition to normal duties. Therefore, it
is expected that clinical training will be mainly in the form of observation,
given the need for safe and efficient operations.
While most four-year universities have affiliated medical facilities, most
of our clinical training facilities are private hospitals (100 to 200 beds),
where many of our graduates work as radiologic technologists. These private
hospitals have a relatively flexible work schedule, which allows for sufficient
guidance and supervision of students, and thus, a clinical training guidance
system that is enthusiastic about fostering junior students has been
established.
Effective preclinical training (educational effects of on-campus
training, OSCE, and CBT)
Based on the
questionnaire survey, the modalities (body parts) in which students had the
most opportunities to experience positioning under the guidance and supervision
of instructors during clinical practice were X-ray (chest, abdomen, lumbar
spine, and knee) and CT (head and chest). Therefore, we believe that the OSCE
is an effective way to evaluate students' clinical practice skills for these
modalities (sites) in preclinical practice evaluations.
Notably, MRI and X-rays (hip and head) were the
modalities in which students had few opportunities to experience positioning in
clinical practice. The reasons
for the lack of positioning experience in hip joints in X-ray examinations may
be due to the pain involved in the imaging position and consideration of the
patient's sense of embarrassment. In head X-ray examinations, CT has become the
mainstream. We believe that it is effective to improve students' understanding
of clinical practice by giving them experience in imaging techniques for modalities
and sites where positioning experience is not expected in clinical practice
during on-campus training.
Other than MRI, mammography, ultrasonography, and X-ray TV were the most
frequently observed modalities. The reason for this
may be that examination results are easily influenced by the knowledge and
skills of the examiner. Nuclear medicine examinations and radiotherapy
examinations were also mainly observed because these examinations are
relatively invasive to the patient and require the concentration of skilled
technologists to perform the examinations and treatments. Therefore, providing
students with experience through on-campus training and knowledge acquisition
through CBT as much as possible is considered to be an effective preclinical
training method, not only for MRI but also for other training items that are
mainly performed on site.
Limitations of the study
This study was limited to
Japan. Therefore, the results are limited, although they may be useful in other
countries with different educational systems.
Notwithstanding the
abovementioned research limitation, it is significant to note that this study
reveals the characteristics of effective imaging techniques and knowledge for
preclinical practice evaluation.
CONCLUSIONS
A survey of students'
positioning experiences in clinical practice revealed the current implementation of the practice of medical care
participation. Consequently, it is effective to evaluate students in the OSCE
for X-ray examinations of the chest, abdomen, lumbar spine, knees, and so on, as
well as CT examinations of the head and chest, which have high frequency of
positioning experiences in clinical practice. In addition, it is effective to
provide students with experience in MRI, ultrasonography, and mammography,
which are mainly performed on site, and to improve their knowledge level
through CBT as part of preclinical training.
The results of this study suggest that the
educational effectiveness of future clinical training with medical participation
will strongly depend on the students' positioning experience during clinical
training. Therefore, cooperation between training schools and medical
institutions is considered to be extremely important to prevent differences in
the level of practical skills instruction that students receive during clinical
practice. The challenge is that both training schools
and medical institutions need to share a common understanding of the level of
practical training for radiologic technologists, and it is necessary to improve
the content of preclinical training education while collaborating with external
organizations.
Acknowledgments
We would like to
express our deepest gratitude to all the students who cooperated with the
survey in conducting this study.
We would like to
thank Editage (www.editage.com) for English language
editing.
DECLARATIONS
Funding: None
Conflict of interest: No declaration
Ethical Approval: Before the survey,
ethical approval for the study was obtained from Research Ethics Committee. Individual
written consent was also obtained.
REFERENCES
1.
Aoyama
R, Nakawatari T, Eto M, Shiraishi J (2022). Preclinical education program for
participatory clinical practice in radiological technology education:
Questionnaire survey of students after clinical practice. Jpn J Educ
Radiol Technol 2022;10(1):13-22.
2.
Eto M, Aoyama R, Nakawatari T, Kamei O (2021). Educational effectiveness and
challenges of preclinical training assessment based on students' attitude. Jpn J Educ Radiol
Technol 2021;9(1):13-9.
3.
Fujii S, Aada
K, Hori R (2003).
An examination of the usefulness of role plays in patient treatment for
students before the start of clinical practice (Report 2). Jpn J Radiol Technol 2003;59(9):1042-.
(In Japanese)
4.
Hatakeyama R, Kubota N, Monma M (2009). Introduction of a clinical skills
evaluation method in radiological technologist student education and clinical educational
lecturer system.
J Jpn Soc Radiol Technol Educ 2009;1(1):5-12.
5.
Hatakeyama R, Kubota N, Mori K, Abe
S, Mori K, Ishikawa E, et al (2007). Introduction of clinical skills assessment
in radiological technologist education and its educational effects on clinical
practice. Innervision 2007;22(12):77-81.
(In Japanese)
6.
Kamioka Y, Shinozaki M,
Tachibana K, Yamamoto S, Miyata S, Aoyama T, et al (2021). Relationship between
objective structured clinical examination (OSCE)
results and clinical training achievements in clinical clerkships for physical
therapy students. J Jpn Soc
Med Educ 2021;52(2):97-101.
7.
Kawakami
T, Kubota M, Kawakami Y, Koyabu T, Tomioka K, Murakami Y, et al (2008). Objective structured
clinical examination for evaluating the education of registered dietitians. Jpn J Nutr Diet 2008;66(3):133-140.
8.
Kawamura
T, Abe S, Kubota N (2011). Preliminary study about assessment of students by
the stimulated patients in OSCE of radiological sciences. J Jpn
Soc Radiol Technol Educ 2011;3(1):2-7.
9.
Muto
H (2012). A model for summative evaluation in medical
education for radiological technologist.
J Jpn Soc Radiol
Technol Educ 2012;4(1):8-13.
10.
Ordinance of the Ministry
of Health (2023),
"Labour and Welfare No. 153 of
2022" website, Available: https://elaws.e-gov.go.jp/document?lawid=504M60000100153 (August.25,2023).
11.
Sekine N (2021). Development of an evaluation system
for basic knowledge and problem-solving abilities before clinical practice in
radiology education using computer-based testing. Jpn J Educ
Radiol Technol 2021;9(1):27-31.
12.
Sekine N, Ueno J, Nyui Y (2007). Development of the computer-based testing
system in the education of radiological technologist. IEICE Technical
Report (Institute of Electronics, Information and Communication Engineers) 2007;107(391):37-40.
13.
Takasaki
H, Tabuchi A, Takihata Y,
Oka M, Namba T, Yoneda T,
et al (2008). Preliminary report of introducing the objective structured
clinical examination for orthoptists. Jpn J Nutr Diet 2008;66(3):133-140.
14.
Takashima T, Arao H (2021). Literature review of the current state of the objective
structured clinical examination for nursing students of BSN program. J Kumamoto
Health Sci Univ (18),
43-56, 2021.
15.
Tokunaga
J, Takamura N, Ogata K, Setoguchi
N, Sato K (2011). Implementation of advanced objective structured clinical
examination trial in pharmacist education and evaluation of results. Jpn J Pharm Health Care Sci 2011;37(2):79-89.
16.
Tokushige Y, Hoshino S (2021). Survey of the implementation status of
clinical practice in radiological technologist education. J Jpn
Assoc Radiol Technol 2021;68(4):369-76.
17.
Yamaguchi
I, Takeshita M, Yamada J, Noguchi A, Ogata Y, Komizu M (2015). The trial of clinical skills educational
program before clinical training in Butsuryo College
of Osaka: Efficacy and the problem of evaluation. Bull Osaka Coll Phys Ther
2015;3(1):11-7.
