Objective Assessment of Orthopaedic Skills

Medical education is moving away from facilitator driven to learner-centered, patient focused education.1 This has changed the emphasis to gaining practical skills as opposed to theoretical knowledge. Surgical disciplines call for psycho motor skills that are traditionally learned by practice on live patients. Simulation provides a good alternative without inherent risk to the patients.2 For example, task trainers have been used to teach/learn basic skills such as cannulation and venipuncture.3 Orthopedic office procedures are commonly used by many general practitioners and internists as part of their daily practice4,5 and are useful for medical students to learn. However it has been noted that these are not adequately taught during medical training,6 and that there is no organized training of such skills in most medical curricula. Realizing the need to teach these skills in an organized manner Vogelgesang et al. developed an instructional program to teach aspiration and injection techniques of the knee and shoulder to medical students and residents. They used didactic lecture and hands-on workshop using anatomical models made of synthetic materials. They found that didactic and workshop trained students outperformed the traditional group and were more confident in doing procedures.7 We have been using task trainers for teaching various techniques such as injections for carpal tunnel, trigger finger with tenosynovitis and knee joint lavage. At present these skills are being taught in a nonsystematic manner and we do not have an objective method of evaluating them. We decided to systematize the delivery and evaluation of these skills using our existing task trainers in order to deliver standardized, uniform and comprehensive training in these skills for students posted in orthopaedics.


INTRODUCTION
Medical education is moving away from facilitator driven to learner-centered, patient focused education. 1his has changed the emphasis to gaining practical skills as opposed to theoretical knowledge.Surgical disciplines call for psycho motor skills that are traditionally learned by practice on live patients.Simulation provides a good alternative without inherent risk to the patients. 2For example, task trainers have been used to teach/learn basic skills such as cannulation and venipuncture. 3thopedic office procedures are commonly used by many general practitioners and internists as part of their daily practice 4,5 and are useful for medical students to learn.However it has been noted that these are not adequately taught during medical training, 6 and that there is no organized training of such skills in most medical curricula.Realizing the need to teach these skills in an organized manner Vogelgesang et al. developed an instructional program to teach aspiration and injection techniques of the knee and shoulder to medical students and residents.They used didactic lecture and hands-on workshop using anatomical models made of synthetic materials.They found that didactic and workshop trained students outperformed the traditional group and were more confident in doing procedures. 7e have been using task trainers for teaching various techniques such as injections for carpal tunnel, trigger finger with tenosynovitis and knee joint lavage.At present these skills are being taught in a nonsystematic manner and we do not have an objective method of evaluating them.We decided to systematize the delivery and evaluation of these skills using our existing task trainers in order to deliver standardized, uniform and comprehensive training in these skills for students posted in orthopaedics.

MATERIALS AND METHODS
The study was conducted during period of March-2016 to July-2016.Ethical clearance was obtained from the college ethics and research committee.A total of 80 MBBS students were recruited in the study on voluntary basis.The students were divided into three groups for instruction -i.e.didactic (lecture), video and hands on training.The students were not randomized using any specific technique but were divided into aforementioned three groups according to their roll numbers i.e. the first one in group one, 2 nd student in group two and so on and so forth.All the participants were lectured on the indications for knee joint injection and aspiration, the risks of the procedure, sterile technique, anatomic landmarks, needle position, and needle insertion through the superolateral approach, informed consent, post procedure management and instructions to the patient.The video group was shown a video of the procedure, in addition to this lecture.The hands-on practical group was shown the procedure on the anatomic model and was allowed to freely practice on the model for one hour every day.The lecture and the video group students did not have access to the model until the day of injection.On the day of assessment, each group of students was given 20 min to become familiar with the model and equipment.After 20 min each student performed the task of knee joint injection through superolateral approach on an anatomical knee model.The knee models used had moderate joint effusion so as to simplify the task.Before the instruction (didactic, video or hands on training) a MCQ test (10 questions with single best answer) was used for base line assessment of student's knowledge of the knee anatomy.Each participant was asked to rate his or her confidence in performing knee joint injections with use of a l0-point Likert scale, anchored with "No confidence at all" (1 point) and "Extremely confident" (10 points).During the task the competency was assessed using newly developed rating scale for knee joint injection/arthrocentesis (Apeendix1) by a single rater.A post procedure MCQ test (10 questions with single best answer) based on knee joint anatomy was conducted.Again, each participant was asked to rate his or her confidence in performing knee joint injections with use of a l0-point Likert scale, anchored with "No skill" (1 point) and "Extremely skillful" (10 points).

Implementation Planning •
Permission was taken from the Dean of the Institute and Head of Department of Orthopaedics to conduct this project.

•
The proposed study project was presented before the ethics and research committee of the institute: Permission to carry out the project and ethical waiver was obtained.

•
The department colleagues were informed and persuaded to help with the study.• Informed consent forms were prepared.• Adequate copies of newly developed rating scale were made.

•
Two MCQ tests based on Knee joint anatomy with each comprising of 10 single best answer questions were prepared.The MCQs given before and after were same set of questions.• Adequate copies of self-confidence assessment forms were prepared.

The Assessment tool
The assessment tool for knee joint injection/arthrocentesis was prepared after discussion with experts.The assessment tool was a task specific rating scale that measures the components of the given procedure that the subject completes.The rating scale had nine items for which 0 to 4 marks were allotted.These items included explains procedure/ benefits/adverse effects to patient, orally obtains informed consent, selects/arranges the adequate materials, properly positions the limb, does sterile skin preparation, identifies and marks the landmarks, uses appropriate needle positioning, does post-procedural dressing, orally provides post-procedural instructions to the patient.The tenth item successful joint entry was indicated by joint fluid entry into the syringe and did not carry any marks.Higher scores (maximum 36) indicated greater knowledge and ability to complete the necessary procedural steps of knee joint injection.Four experts (3 from the institution and one external orthopaedic surgeon) were asked to rate each item as clearly representative, somewhat representative and not representative of the knee joint injection skills assessment.Out of the initial 11 items in the rating scale 2 items were omitted.The remaining 9 items were included based on the criteria that at least 3 out of 4 experts viewed them as clearly representative.A pilot study was done on 20 subjects.After the pilot study and with inputs from three internal and one external expert, the tool was further refined and finalized for the study.The components of the rating scale included are shown in Appendix 1.

Discussion with faculty, students and sensitization
The first session with the faculty members of department of Orthopaedics was a sensitization session wherein the faculty members were introduced to the need of objective assessment of orthopaedic skills, and its possible utility to students and teachers.The proposed study plan was also discussed with the faculty.One of the faculty from the hospital who was not associated with the institution acted as rater and was instructed on the use of assessment tool.The students who participated in the study were informed about the study.Written informed consent was obtained.
In the following sessions the faculty helped in conducting the study.

RESULTS
A total of 80 fourth year MBBS students participated in the study.The study sample consisted of thirty-two men and forty-eight women, with a mean age of twenty-five years (range from twenty-two to sixty years).On descriptive analysis, the participants scored highest in "Explains procedure/ benefits/adverse effect to patient" (mean score=3.4) and "Orally obtains informed consent" (mean score=3.3)out of maximum possible of 4. The lowest mean score was for item "Orally provides postprocedural instructions to the patient" i.e. 1.1 out of maximum possible of 4. On comparing the means by one-way Anova, the mean score was highest for the hands on training group (26.3) followed by video group (20.5) and lecture group (16.2) respectively (Table 1).The difference in the scores between three groups was statistically significant (p<0.0001).
The average score of 1 st MCQ test was 3.2±1.50out of maximum possible of 10.The average score of post procedural MCQ test was 6.6±0.78.The self-confidence in performing knee joint injections increased from 2.3±0.74 to 6.6±1.14.Both the increase in MCQ score and confidence level were statistically significant.The overall internal consistency (cronbach's alpha) of the assessment tool developed was found to be 0.63.The Cronbach's alpha seems to be less which shows less internal consistency.However high convergent validity (inter-item correlation) was found for 8 out of 9 items in the assessment tool (Table 2).

DISCUSSION
Traditionally, senior surgeons do subjective assessment of surgical competency during intraoperative observation.The chief drawback of this type of assessment is the lack of consistency.As aptly put by Moorthy "Without objective, valid, and reliable assessment training programmes cannot ensure the learning of skill, tackle deficiencies in training, and implement remedial measures". 8It has been noted that joint injection skills training is not a part of most medical curricula and the need for a systematic training and evaluation of these skills has been realized. 6he observer based assessment tools include task specific checklists/rating scales and global rating scales.These tools can allow for continuous assessment during the training period.It has been suggested that global rating scale is a better assessment tool than task specific checklist, either alone or in combination with a global rating scale.However, Insel et al. 9 pointed out that task specific checklists are important since they "captures the ability to complete all of the necessary steps of the procedures, reflecting basic knowledge and technical ability." The advantages stated for these tools are that they objectively assess the operative skills and are free from recall bias associated with direct observation by preceptors.Another advantage is that the skills can be tested both in a laboratory setting and during real time procedure.The disadvantage is the need for trained observers or experts for assessment.

Validity and Reliability
The tools used for objective assessment should be valid and reliable.Validity means the extent to which a test/measure/concept is well founded and correlates with the real world.It includes construct, content, predictive, face and concurrent validity.
Construct validity is the ability of a test to actually measure what it is supposed to measure.Convergent validity is considered as subcategory or subtype of construct validity.To establish convergent validity, we need to show that measures that should be related are in reality related.It was evident from the fact that inter-correlations for all items with the total score were very high.The item-total correlation in Table 2 shows that the convergent validity eight out of the nine items of the assessment tool was high.
Content validity means the extent to which the test measures the domain, which is being measured.During the development of the tool, four orthopaedic experts graded the various items of the rating scale according to their relevance.The final tool contained all the essential items as agreed upon after discussion with the experts.That proves the content validity of the tool, however content validity ratio (CVR) was not calculated. 10ce validity means the extent to which a test resembles a real life situation.It is tested by 'users' opinion about the functionality and realism of a test". 11In this study the face validity of the assessment tool was assessed from the overall impressions of the orthopaedic experts.Concurrent validity of a test means the extent of correlation with a previously validated tool or a gold standard test for the same trait.Concurrent validity applies to validation studies in which the two measures are administered at approximately the same time.However, since no other previously validated tool was used for comparison in this study, the concurrent validity of this newly developed tool could not be established.Other variants of validity like Predictive, discriminant, criterion validity etc. were not assessed.Reliability means the extent to which the test to gives consistent results.
Inter-rater reliability means the extent to which test results are consistent between different observers whereas the test-retest reliability measures the extent of consistency in results when undertaken over a period of time.Since only single rater was used and the skills were not assessed before the educational intervention, both inter-rater reliability and testretest reliability of this tool are unknown at present.Internal consistency refers to the correlation between different items in the same test and is measured by Cronbach's alpha.The overall internal consistency (Cronbach's alpha) of the assessment tool developed was 0.63.Besides having validity and reliability, the tool/test should be convenient to apply (i.e.feasible) and cost-effective.The limiting factor of use of this tool is that it requires an expert assessor.The hands on training group scored highest in the knee injection task as assessed by the new tool.This further proves that the psychomotor skills are best learned through hands on training.Change in confidence is a crude marker of competence.Nevertheless, all the instruction groups showed increase in confidence and knowledge of joint anatomy.However, it was not the primary aim of the study.

Outcomes: What this study adds
In this study we attempted to devise an objective tool for assessment of joint injection skills.After a review of the literature, no validated evaluation method of joint injection skills was found.The other positive results that emerged from the study were increase in student's knowledge of anatomy and confidence level as assessed by MCQ test and the confidence questionnaires.

Limitations
• Lack of randomization makes this study lower on hierarchy of evidence.• In this study the skills were not tested on live patients.How well the skills performed on model transfers to practice is not known.Also, we can't predict that for how long the skill will be retained without further practice.• A single rater assessed all the subjects, hence inter-rater reliability was not assessed.

•
No test-retest reliability was tested.• Another interpretation of construct validity is the ability to discriminate between experts and novice.If this tool succeeds at differentiating between experts and novice, it will further confirm its construct validity, that however remains to be seen.

CONCLUSION
Availability of an objective assessment tool for orthopaedic skills will systemize the delivery and evaluation of these skills in our setting.Further the experience gained through this study will be used to develop objective assessment tools to evaluate the skills in other medical disciplines.
In conclusion, this newly developed tool is a valid and practical tool to evaluate knee joint injection technique.As medical students/ professionals gain proficiency in joint injection technique by practice, this tool will capture the gain in skill.

Table 1 : Mean task specific rating scale scores
12