Sponsored by:
Implementation & Benefits of an Assessment Platform in Physician Assistant Education
Introduction to Physician Assistant Education
On-going student evaluation is essential in physician assistant (PA) education. Documentation of achievement of learning objectives, comprehensive assessments including complex diagrams and color images, and student ease of use and acceptance are key factors in the evaluation process for this student population. PA education averages 27 months in duration and is conducted at a master’s level in the United States. There are now 190 accredited PA programs in this country.
The PA Program at Tufts University School of Medicine commenced in January 2013. Part of the implementation of the program included the introduction of computer-based testing utilizing the ExamSoft™ system. All students were required to have portable laptop or tablet devices for use in class and at home. iPad use by students was introduced in the second year of operation. In the first two years of operation 231 computer-based examinations were given to the first two classes and a total of 5,073 questions were entered into the question bank. Less than 10 traditional “paper” examinations were given during this time period. Faculty were surveyed as to satisfaction with regards to ease of use, question entry, and results reporting. All principal and adjunct faculty were highly satisfied. Students were also polled as to their opinion of computer-based testing and over 95% responded “satisfied” or “highly satisfied”.
Implementation from Paper to Computer-Based Exams
Since this was a new educational program, there were no pre-existing computerized question banks. All questions were either imported from Microsoft WORD™ files or entered directly into the software program’s question portal. Direct entry permitted faculty to categorize each question as “Blueprint” items as designated by the National Commission on Certification of Physician Assistants (NCCPA). The categorization process permitted the linking of a question to specific learning objectives or NCCPA Blueprint items. Questions imported from WORD™ files could be categorized at a later time. Subsequently, students would be offered areas of deficiency on which to concentrate their studies after an assessment was conducted. Faculty could also note areas which required emphasis in subsequent classes.
Academic Advantages of Computer-Based Testing
At the completion of an assessment, the multiple choice, True/False, and Fill-in-the-blank question items would be scored instantly by the software. Simple statistical analyses are performed instantly by the software, providing the instructor with grade ranges and means as well as Point Bi-serials in addition to a detailed item analyses. Individual question performance would be recorded and faculty could review the question’s performance over the course of several classes, offering the instructor the option of keeping, revising or retiring the question in the question bank.
Turn-around time for results reporting of assessments containing non-essay questions is almost instantaneous, but faculty have the added advantage of adjusting answers or removing the question item from the assessment entirely. Essay questions are presented to the instructor or grader in a clear format in the font of the grader’s choosing, thus facilitating reading and grading. Students may also be given options such as spell-check and calculator use with the instructor’s approval.
In medical education, the ability to imbed color clinical images into examinations is essential but can be costly if the exam is reproduced on paper. High volume reproduction of color images on paper can cost as much as $.35 per color page. Black and white reproduction of these images is an alternative but not ideal. Black and white reproduction costs can range between $.03 - $.05 per page adding considerably to operational costs. In PA education where in the first didactic year 15 to 30 courses may be conducted, including anatomy and physiology, pharmacology, pathophysiology, etc. frequent assessments are given throughout each course. At our institution each course was mandated to have at least three assessments per semester. At least 90 assessments were given to students in their first year and 75 in their second clinical year. Our first cohort was comprised of 30 students in our 25 month-long program and underwent a total of 165 assessments. If these examinations were produced on paper, the average exam would have been about 12 pages in length. Also, assume 10% of these pages would have been in color. The following table depicts printing costs for paper examinations for one cohort of 30 students over 25 months.
Advantages of Computer-Based Testing
The ability to bank and evaluate questions over time is an enormous cost saver and quality booster to the evaluation process. Tracking question performance and eliminating poor performing questions adds to the evaluation process of any educational program.
Tying questions to learning objectives and accreditation standards, and being able to demonstrate these processes saves valuable faculty and staff time during in an accreditation process.
If assessments are scored effortlessly and instantly by the software package, students can be offered more questions per evaluation. This also saves faculty time and effort, allowing more time for question scrutiny through item analysis.
Exam security can be enhanced using computer-based software as in most cases it offers an option to scramble the order of questions and answer choices. ExamSoft™ permits use of the student’s own laptop as its security features “lock-down” students’ computers during the examination, prohibiting student access to the hard drive and the internet. The software tracks all student keystrokes and creates a log of all entries during the examination. These exams can also be offered in an unsecured mode which allows students to take exams at home in an “open book” format.
Estimating faculty time and expense of manual grading examinations is difficult to assess given the wide range of salaries in academia. Creating item analyses and tracking long-term performance of questions is also time consuming. Other non-computer-based testing approaches still have manual components to grading but the speed and efficiency of computer-based systems are difficult to ignore.
Printing costs may be significant in larger programs, particularly if color printing is involved. Software licenses can run between $40 and $50 per year per student. In our case use of this software resulted in a simple cost saving of over $1,500 in printing costs alone. The added value of developing a categorized and reusable question bank, faculty time saved in scoring exams, and a greater understanding of question performance, and demonstration of areas of student deficiency is difficult to quantitate financially. However, from a program perspective this represents a quantum leap in the evaluation process of students and the program as a whole. The “green” value of computer-based testing over wasted paper is obvious.
The added value of computer-based testing serves the student, instructor, and the educational program. Cost savings aside, the speed and efficiency alone are a plus in physician assistant education. Growing class sizes, need for timely turnaround in results, and also on-going practice on computer-based examinations are all advantages of these new systems. Medical boards, bar examinations, and even some civil service examinations are computer-based; students should be acclimated to this process during their education so they will be comfortable taking their summative examinations.
