Objective structured assessment of veterinary students' suture skills before and after training in theory and practice

Abstract

This study evaluated the effectiveness of the Objective Structured Assessment of Technical Skills (OSATS) in measuring and improving veterinary students’ proficiency in simple suturing. Forty final-year veterinary students participated in a structured training session that included theoretical instruction, video demonstrations, and hands-on practice sessions. Performance was assessed pre- and post-training using the OSATS, incorporating both a global rating scale and a procedural checklist. Before training, only 50% of the students met the passing criteria, with common deficiencies in needle handling, suture placement, and wound-edge eversion. After the intervention, the pass rate increased to 90%, with significant improvements across most performance indicators (P < 0.05), although deficiencies in movement fluidity persisted. Gender analysis revealed no statistically significant differences in the overall performance. These findings confirm that OSATS is a valid and reliable tool for objectively assessing technical surgical skills in veterinary education. This study supports its integration into veterinary curricula to standardize competency evaluation and enhance training outcomes. Future research should explore its application in diverse surgical procedures and assess long-term skill retention through follow-up assessments. The results offer practical guidance for veterinary educators aiming to develop structured, skill-based learning environments that ensure clinical readiness.

Introduction

Surgery is a medical specialty that involves mechanical manipulation of the anatomical structures of a living being for medical purposes. It is a branch of medicine that prevents, cures, or rehabilitates diseases in patients by sectioning, separating, repairing, or replacing tissues or organs using instruments (Chen & Lindeman 2024). Therefore, it is an essential practical discipline that requires manual skill. Although operative skills represent only a part of the qualities necessary to become a well-trained surgeon, developing these abilities plays a fundamental role. Therefore, evaluating the acquisition of technical competencies is essential for establishing the level of learning, providing feedback, evaluating training programs, and accrediting the aptitude of the training surgeon (Szasz et al., 2015; Fecso et al., 2017).

Traditionally, surgical skills have been evaluated in the operating room through supervision and feedback. However, this method has been criticized for being too subjective and not necessarily representing the surgeon's actual skill level. New techniques, such as minimally invasive surgery, require additional skills with different learning curves that necessitate training methods outside the operating room. This progress has led to an increased interest in objective assessment methods of surgical skills that are valid and reliable (Sidhu et al., 2004; Langebæk et al., 2020; MacArthur et al., 2021).

Over the last decades, various evaluation systems have been developed to measure operative competencies. Traditional methods include questionnaires, procedure logs, time recording during procedures, direct observation, and mortality and morbidity data. These methods are widely used and have some value in assessing knowledge and clinical judgment but lack objectivity, validity, and reliability for assessing psychomotor skills. The new methods include checklists, electronic analysis systems (e.g., advanced Dundee endoscopic psychomotor tester, ADEPT), motion analysis devices (e.g., Imperial College Surgical Assessment Device, ICSAD), virtual reality simulations, analysis of the final product in bench models, global assessment scales, and error recording and grading systems. These methods were designed with appropriate methodological attributes to objectively measure the performance of psychomotor skills (Darzi & Mackay, 2001).

Regardless of the method, it must be reliable and valid, defining validity as "the property of being true, correct, and by reality." Furthermore, validity can be evaluated in different ways: a) apparent validity (which represents the functionality and realism of a test); b) content validity (i.e., the content of a test is appropriate for measuring what it should measure); c) construct validity (in which the test measures the trait it is supposed to measure, in this case, technical surgical skill); d) concurrent validity (comparison of a test with another that measures the same trait); and e) predictive validity (the extent to which a test allows the prediction of performance). Apparent and content validity are usually confirmed by the opinion of a group of experts, whereas construct validity can be assessed by determining its ability to discriminate between novices and experienced candidates (Sugden & Aggarwal, 2010).

Reliability refers to the accuracy of the test, which is the consistency between evaluations and among evaluators. Finally, for a valid and reliable evaluation regimen to be successfully adopted by researchers or program directors, it must also be feasible to implement. Although many reliable and valid methods for assessing surgical performance have been described, feasibility issues such as cost and practicality limit their widespread use (Farrell et al., 2022). The ideal assessment tool in this respect would be cost-effective, simple to perform, ergonomically attractive, low-maintenance, and not require expensive or complex installation (Kiely et al., 2015).

Objective Structured Assessment of Technical Skills (OSATS) was one of the first methods developed for the objective evaluation of technical skills and is the most widely studied. It is a performance-based examination designed to assess the technical abilities of surgeons during training. It was originally developed at the University of Toronto by J. A. Martin et al., (1997) who suggested that the Objective Structured Clinical Examination (OSCE), which measures clinical competencies, could be used as a powerful assessment tool for technical surgical skills. Thus, they developed OSATS, which consists of six stations where trainee students perform established tasks over a 15-minute period using a bench or live animal model. Six tasks were selected from the specific objectives of third-year general surgery residents at the Department of Surgery of the University of Toronto. Performance during task execution is determined through direct observation using a three-part assessment form that includes a list of specific tasks, a global assessment scale, and an approval/fail judgment. The checklist was individually prepared for each procedure and contained between 22-32 specific steps to be evaluated. The scale consists of seven generic components of operative skills rated on a five-point Likert scale, with middle and extreme points set by explicit descriptors to assist in the assessment criteria (Martin et al., 1997). From this original concept, several modifications have been proposed to adapt the method to different educational contexts and specific procedures of various medical and surgical specialties (Niveditha et al., 2022; Huang et al., 2024; Hopmans et al., 2014; Hoyt et al., 2022).

Currently, OSATS is one of the most widely accepted methods and is considered the standard for evaluating surgical skills. Martin et al. (1997) demonstrated that this method has high reliability and construct validity in a laboratory simulation environment, as well as in animal models and cadavers. Subsequently, other studies have concluded that it is possible to assess technical skills using this methodology, which has been validated in specialties such as Urology and Gynecology and Obstetrics for specific surgical procedures.

Despite numerous studies supporting the use of OSATS to assess surgeons in training, there is limited evidence regarding its use in veterinary medicine (Schnabel et al., 2013; Fransson et al., 2012; Kuzminsky et al., 2023; Langebæk et al., 2020; MacArthur et al., 2021; Farrell et al., 2022; Chen et al., 2017; Hincapié-Gutiérrez et al., 2023) and no publication in veterinary suture technique. Schnabel et al. (2013) designed and used global scales to evaluate the effectiveness of two surgical technique courses taught to veterinary medicine students and concluded that such instruments are useful for objective evaluation and as a formal feedback mechanism. Furthermore, they emphasized the importance of incorporating formal assessment instruments for these skills into the curriculum and projected this assessment tool as a basis that can be modified for veterinary schools and/or specific courses.

Based on the existing evidence regarding its use, OSATS in human surgical skills acquisition proves to be a valid and reliable tool as a formal assessment method to verify the acquisition of technical skills in students, providing greater objectivity to their evaluation and improving feedback. We anticipate that OSATS can be similarly applied to veterinary skill assessments. In this study, we evaluated the single suture technique skills before and after a theoretical suturing class followed by practical training of students in the last semester of veterinary medicine.

Material and method

As part of the internship of the activities in Small Animal rotation, 40 students, representing 40% of the course, were invited to the Surgical Skills Training Center at the University of Chile. This session was included in the rotations required for the final semester of veterinary medicine. All these students had previously received knot-tying training in the first semester of the final year in the dry lab of the Surgery and Anesthesia course.

The methodology involved evaluating students with OSATS before commencing practical activities. They watched a suturing instructional video, followed by 10 min of hands-on practice. The video, which lasted 4 min and 30 s, shows the execution of three simple interrupted sutures, and was the same one that had been shown to these students during the first semester of their final year in the dry lab of the Surgery and Anesthesia course. After watching the video, they were asked to perform the same task. High-density synthetic polymer-based table-mounted simulators with horizontal incisions were used. The polymer was affixed to a plastic base with suction cups to maintain its adhesion to the table. Each student was provided with a surgical instrument set comprising a Mayo needle holder, toothed dissecting forceps, straight Mayo scissors, and a non-absorbable monofilament nylon 3/0 suture.

Each student was individually recorded executing the assigned task using a video camera mounted on a tripod positioned beside the table. The field of view was limited to the polymer and the students’ hands. Each student was identified by a number placed on the polymer base.

After this first recording, a theoretical class lasting approximately 30 min was conducted by a veterinary educator/instructor showing the elements used and their handling, followed by a training block of one hour and 30 min, where the professional corrected the instrument handling and knot-tying technique of each student. Each student was provided with a table simulator and the aforementioned instruments. Finally, each student was again individually recorded, repeating the task that was initially assigned.

The videos were analyzed using the Objective Structured Assessment of Technical Skills (OSATS) for simple suturing, comprising a global rating scale (Table 1) and a checklist (Table 2) aimed at providing specific guidance on which areas of the procedure to evaluate. Described in Table 1 shows the global assessment aiming to measure the overall quality of surgical performance and is scored on a Likert scale from 1 to 5 points with explicit midpoint and endpoint descriptors for each item. The checklist evaluates specific components of the surgical procedure and is scored as No, Partial, and Complete, assigning 0, 1, and 2 points, respectively, to each item.

Table 1. Global assessment. This scale assesses general components of surgical performance that are common to various procedures.

Item	5	3	1
1. Instrument handling	Proper use of instruments at all times	Proper handling most of the time	Improper handling of instrument(s)
2. Suture quality	Excellent technique, excellent dexterousness, excellent closure	Moderately good technique, moderate manual dexterity, acceptable closure	Poor technique, poor manual dexterity, problems with closing
3. Knot quality	Excellent technique, excellent knot execution, safe and proper knotting	Moderately good technique, some knots better than others, knotting mostly safe	Poor technique, couldn't tie all three knots, unsure knotting
4. Regards with tissues	Very gentle handling of tissue with minimal or no damage	Careful handling of tissue but occasionally causing damage	Frequently used unnecessary force or caused tissue damage
5. Movement & Fluidity	Clear economy of movements, continuous flow and rhythm, minimal tissue manipulation	Some unnecessary movements, reasonably efficient, delicate progression, resumption of occasional tissue	Lots of unnecessary movements, frequent stops/reset, frequent tissue resumption

Table 2. Performance checklist. This scale makes it possible to highlight the specific steps of simple suturing that are poorly performed.

ITEM	NO	PARTIAL	COMPLETE
1. Loads sutures with the tip of the needle holder, taking the needle between the middle and the back third
2. Penetrates tissue perpendicularly
3. Invert edges by viewing needle input and output
4. Positions the points 0.5 to 1 cm on the edge symmetrically
5. Prone-supine following the curve of the needle
6. Knots with instrument properly
7. Tighten the square knots together
8. Performs three knots (2-1-1)
9. Adequate suture tension on tissues

The final test score was calculated by combining the scores obtained in the global assessment and checklist for a total of 43 points. To determine each student's level of competence, this is whether they Approved or Failed, it was established as a requirement not to have a minimum score in any item of the global scale, and to achieve a performance of 60% of the total test score (> 26 points), which corresponds to the minimum score required to obtain an acceptable and mostly firm suture (secure pattern and knot).

Additionally, percentage analyses of student performance were conducted for each item of the global assessment and checklist, aiming to characterize their performance in the OSATS and identify items with the highest percentage of correct, incorrect, or regular responses. Furthermore, performance was analyzed by gender to ascertain whether there were significant differences between male and female students for any specific item.

Pre- and post-data were analyzed using the Wilcoxon test for paired non-parametric variables to determine whether the differences were significant. Student No. 19 was excluded from this analysis because it had a null difference between their scores, resulting in a final sample size of 39 students (n = 39). The analysis was conducted using P = 0.05.

Results and discussion

The Objective Structured Assessment of Technical Skills (OSATS) was used to evaluate veterinary students' suturing abilities before and after a theoretical-practical training session. Final scores were recorded, and performance thresholds were applied to determine the approval rates (Table 3).

Table 3. Percentage of students who obtained No, Partial, or Complete for each item on the pre-class and post-class checklist.

Item	Total Pre-class			Total Post-clase
	No	Partial	Complete	No	Partial	Complete
1	18 (45%)	10 (25%)	12 (30%)	11 (27.5%)	3 (7.5%)	26 (65%)
2	11 (27.5%)	9 (22.5%)	20 (50%)	3 (7.5%)	2 (5%)	35 (87.5%)
3	23 (57.5%)	14 (35%)	3 (7.5%)	10 (25%)	15 (37.5%)	15 (37.5%)
4	18 (45%)	19 (47.5%)	3 (7.5%)	9 (22.5%)	17 (42.5%)	14 (35%)
5	10 (25%)	23 (57.5%)	7 (17.5%)	3 (7.5%)	13 (32.5%)	24 (60%)
6	5 (12.5%)	5 (12.5%)	30 (75%)	1 (2.5%)	1 (2.5%)	38 (95%)
7	0	2 (5%)	38 (95%)	1 (2.5%)	0	39 (97.5%)
8	2 (5%)	2 (5%)	36 (90%)	1 (2.5%)	1 (2.5%)	38 (95%)
9	7 (17.5%)	22 (55%)	11 (27.5%)	3 (7.5%)	7 (17.5%)	30 (75%)

OSATS Pre-Class Results

In the initial assessment, 50% of students approved the test. Among those who failed, most received minimum scores for at least one item on a global scale. No significant difference in performance was found between male and female students.

The most common errors occurred in item 3, which required eversion of the wound edges (entry and exit of the needle visible), with 57.5% of students failing. Items 1 (proper loading of the suture needle) and 4 (symmetrical placement of sutures 0.5-1 cm from the edge) followed, with a 45% error rate.

Students performed best on item 7 (correct approximation of square knots) and item 8 (completion of three knots), with 95% and 90% success rates, respectively. Men slightly outperformed women on these tasks (P < 0.05). Items 5 (needle rotation following its curve) and 9 (maintaining adequate tension) showed the highest rates of partial success, at 57.5 and 55%, respectively.

On the global scale, the lowest performance was in item 1 (instrument handling), with a minimum score of 40 %, primarily due to improper use of dissecting forceps. Conversely, Item 4 (tissue respect) received the highest percentage of maximum scores (65%), particularly among female students. Item 2 (suture quality) had the highest partial completion rate (85%).

OSATS Post-Class Results

After training, the approval rate rose to 90%, with 17 of the initially failing students approving the test; however, three students failed both assessments. Error rates decreased significantly across several items; for example, errors in Item 1 dropped from 45 to 27.5% and in Item 3 from 57.5 to 25%. Item 5 (movement and fluidity) was the most frequently partially completed task, with 70% of the students still struggling.

Global scale scores showed significant improvement. The minimum scores were limited to items 1 and 3 (5% each), whereas the maximum scores increased notably in items 4 (from 65% to 95%) and 1 (from 55 to 90%). Female students improved markedly in tissue handling (Item 4: 65.4 to 96.2%), whereas male students improved in instrument handling (Item 1: 71.4 to 92.9%).

These findings confirm the effectiveness of OSATS as a reliable and objective tool for evaluating and enhancing surgical skills in veterinary students. It successfully identified specific weaknesses in pre-training performance, including improper suture loading, inadequate edge eversion, and poor stitch symmetry, consistent with earlier research on structured skill assessments (Martin et al., 1997).

The significant improvements in post-training assessments reinforced the value of structured practical training (P < 0.05). As supported by Sanfey and Dunnington (2010), deliberate practice (a structured, effortful practice method that focuses on improving performance) and proper feedback (information that helps someone improve their performance or behavior) are critical for acquiring complex motor skills. However, continued deficiencies in movement fluidity suggest that some competencies require more extensive or repetitive training to achieve mastery and automatization (Anastakis et al., 2003).

A key limitation of this study was the absence of a long-term follow-up to determine skill retention. As highlighted by Hopmans et al. (2014), one-off training sessions may not result in lasting competences. Incorporating longitudinal assessments would improve the evaluation of the persistence of learning outcomes. Future studies should include long-term skill retention assessments at one- to six-month intervals.

Additionally, this study supports the potential of OSATS as a foundation for creating veterinary-specific evaluation tools. Standardizing assessment protocols across institutions could enhance the consistency of surgical education and facilitate skill certification, aiding professional development and curriculum accreditation (van Hove et al., 2010).

Implications for Curriculum Design

Veterinary educators can use these findings to support the integration of OSATS-style assessments into surgical training. Emphasizing hands-on learning with iterative feedback cycles is crucial for achieving competency in the essential techniques. Further, tailoring assessments to specific veterinary procedures could provide more targeted insights into student performance and inform curriculum development at both the undergraduate and continuing education levels. A longitudinal design could enhance this study and generate generalizations to broader student populations.

Declarations

Competing Interests statement

The authors declare that they have no competing interests.

Ethics Statement

All the participating students signed an Informed Consent Letter

Acknowledgments

Partially funded by Clinical Skills Center (CHC), Faculty of Medicine, West Campus, and grant FIDOP Nº 2023-28, University of Chile