|Year : 2016 | Volume
| Issue : 2 | Page : 49-52
The use of OSATS like tools for the comprehensive assessment of technical skills for otolaryngology procedures
Associate Professor, Department of Ophthalmology and Otolaryngology Head and Neck Surgery Umm Al-Qura University Makkah, Saudi Arabia; The Head and Neck and Skull Base Center King Abdullah Medical City (KAMC), Makkah, Saudi Arabia
|Date of Web Publication||6-Jan-2020|
MD, FACS Osama Marglani
The Head and Neck and Skull Base Center King Abdullah Medical City (KAMC) Makkah
Source of Support: None, Conflict of Interest: None
Introduction: It has always been difficult to objectively evaluate the competency of otolaryngology surgical residents. Recently introduced objective structured evaluation of technical skills (OSAT) is increasingly being used for assessment of operative skills. Not much data is available to demonstrate its efficacy in otolaryngology residents. The aim of this study was to review the available literature published on OSATS as a tool for objective evaluation of otolaryngology residents to provide a comprehensive overview of these tools to be utilized by both surgical educators and residents in the field, and to point out the various deficiencies in order to develop and validate tools in certain fields of the otolaryngology procedures for future research.
Material and Methods: In order to confirm the validity of OSAT as reliable evaluation tool, We have reviewed the ten recently published studies in the literature using OSAT as a tool for evaluation of various otolaryngological procedures. We identified three intended uses for the OSATS, namely surgical skill evaluation, progress and divide them into procedure specific and global rating of a particular surgical field and program evaluation.
Results: All these studies demonstrate OSAT can be instrumental in evaluating otolaryngology curriculum, in addition to assessing the level of competence in residents. These studies further demonstrate improving OSAT score in residences as they progress in their surgical residencies, confirming the reliability of this tool we also found a lack of some procedure specific tools that can be helpful and is imperative to develop and validate in the future.
Conclusion: Assessment of surgical skills using OSAT program has the potential to be used as a reliable tool for allowing adequate monitoring of resident’s progress and should be implemented as part of resident evaluation in otolaryngology program. Although many tools have been developed, still further refinements of the available tools and development of more procedure specific tools are necessary.
Keywords: objective assessment, otolaryngology, resident program
|How to cite this article:|
Marglani O. The use of OSATS like tools for the comprehensive assessment of technical skills for otolaryngology procedures. Saudi J Otorhinolaryngol Head Neck Surg 2016;18:49-52
|How to cite this URL:|
Marglani O. The use of OSATS like tools for the comprehensive assessment of technical skills for otolaryngology procedures. Saudi J Otorhinolaryngol Head Neck Surg [serial online] 2016 [cited 2021 May 5];18:49-52. Available from: https://www.sjohns.org/text.asp?2016/18/2/49/275263
| Introduction|| |
Gastroesophageal reflux disease (GERD) is a complex Assessment of surgical competency of residence is an integral part of any surgical training program; training in the otolaryngology is not different. Residents usually learn surgical skills in the operation theater by copying the technique of surgical guide or a role model. It is very difficult to objectively assess the surgical skills. Traditionally the skills of resident trainees have been assessed subjectively with criteria such as a number of cases, operative time, log books of cases which are usually biased as they are assessed by supervising surgeons. Recently the demand for accountability and transparency in surgical field has increased; strict implementation of ACGME guidelines and the hospital accreditation policies has shifted the focus to more objective and uniform evaluation of residents and health care providers. Many tools have been adapted for evaluation over past years like simulation devices, motion analysis and global rating for endoscopic sinus surgery (GRESS) etc. Recently used OSAT is more applied and accepted tool for objective evaluation. The documented literature for OSAT as a tool for otolaryngology procedure is limited to certain procedure. OSATS consists of skill-related variables that are typically graded using a 5-point Likert scale; with 1, 3 and 5 points anchored with explicit descriptors, originally developed at the University of Toronto . This demonstrated high reliability and validity in the simulation laboratory settings in animal models and cadaveric models. Datta et al  in 2004 revealed that OSATS can be used in the clinical practice for evaluation of operative skills. They demonstrated that assessment of technical skills in the inanimate models could be translated to the performance in the operation theater, therefore assessment is done in the actual work place. After publication of these results OSATS have been used in many residency programs mainly general surgery, obstetrics and gynecology. The data evaluating their usefulness in the otolaryngology program is limited to certain specific surgical procedures. In order to establish their efficacy as a reliable tool we conducted a comprehensive literature review of published articles on the OSATs use in otolaryngology. To the best of our knowledge this is the first review of utilization of OSATS in otolaryngology. The aim of our article was to review the available data for OSAT in the literature to provide comprehensive overview of these tools to be utilized by both surgical supervisors and residents in this field, and define the gaps in the procedures to guide for future research.
| Material and Methods|| |
To justify the use of OSAT as reliable tool in the specialty of otolaryngology the sufficient literature is lacking, the published studies are mainly single center studies demonstrating its use for evaluating the residents in a particular phase of a training program and comparing them at different training levels. We searched Medline, CINAHL, Web of Science and Scopus. We selected 12 articles recently published from 2005 to 2015. All articles were focused on evaluation of OSATS as tool for ENT surgical procedures. Out of these 10 articles used Corenbach’s α (alpha) as measurement of internal consistency. Most of the studies relied heavily on previously developed competency tools from General Surgery.
| Results|| |
Of the 12 articles published from 2005 to 2015 a total of 271 otolaryngology residents were evaluated by otolaryngology faculty, as part of the OSATS two evaluation methods were used by evaluator, this included Task specific check list (TCL) and Global rating scale (GRS). All the studies found good internal consistency. These studies are summarized in [Table 1].
| Discussion|| |
The aim of our study was to review the available literature on OSATs as a valid and reliable tool to assess the operative skills of otolaryngology head and neck surgery residents in the operation theater. Assessment is defined as the process of collecting and analyzing information for decision making and providing optimal feedback. In medical training it is used to measure the progress of a resident in the residency program and achievement of surgical competency which is expected at that level (summative evaluation). It is therefore useful in directing the residents in certain areas of surgical skills deficiencies that needs more attention, extra-work and training. Various tools have been designed in the past for evaluation of competency of otolaryngology residents, however, not much detailed data is available on the reliability of these tools currently used as shown in [Table 2].
Direct observation involves a senior faculty observing a learner during a surgical procedure, based on the observation an opinion is formed regarding surgical competency, the method is fast, provide immediate feedback but judgment rely on subjective and unstructured observation. Another measure more frequently used particularly for newly hired medical staff is the operative case log book system, the number of cases performed by residents are entered in the log book it indicates the adequacy of surgical training program and the experience gained by the resident. But all the above mentioned tools are highly biased as they rely on subjective assessment by the examiner, this lead to the search for an ideal assessment tool.
The OSAT technique was developed by Martin et al  and colleagues for general surgery residents, they assessed the residents in the lab on two parameters, a Task specific checklist (TCL) and global rating scale (GRS). The TCL contains specific stepwise task related to each procedure. This aims to reduce the subjectivity of the observer, on the other hand global rating scale (GRS) includes the overall performance measures like familiarity of the technique, ease of using surgical instruments and handling of tissue and instruments, now OSATS is increasingly used for most of the surgical subspecialties to assess the skills at both trainee and specialist level and to provide immediate and detailed constructive feedback, these tools can be used in the animal laboratories, operation rooms  and simulation centers, they can correct the residents focus of training to cover his weaknesses, and provide a stimulus for both the educator and trainee for feedback discussion. Clinical competency is a combination of cognitive function and decision making. This systemic review of the articles gives an overview of the evidence on availability and reliability of the different observational tools used in otolaryngology head and neck surgery. There are two basic tools used in the review studies: first is task specific tool which provides list of tasks related to each procedure, out of 12 articles we reviewed 9 used the task specific checklist either as alone or in combination with the global rating scale. For most of the procedures task specific checklist provides list of all the steps needed to be done by a resident and it is marked on the scale from 0 to 5 thus decreasing the variability among observers. Another tool is the global rating scale which incorporates the overall understanding of the resident related to the procedure including the decision making and post operative care. Overall all the above studies using the OSAT as a tool for assessment have demonstrated a high degree of internal consistency with the Cronbach’s (alpha) value of 0.9.
Although the requirement and need for developing such tools for assessment is clearly defined in the literature not many tools have been developed to cover all the domains and procedures of otolaryngology, head and neck surgery. Out of all available tools OSATS are most reliable tool available, a combination of the global and task specific assessment tool in otolaryngology seems to be a most comprehensive solution for technical skills, and is gaining great attention in the surgical skill assessment in the otolaryngology certifying residency programs. This review focused on reviewing these otolaryngology procedures assessment tools, both in its TCL and GRS forms, a study by Francis et al  done in 2010 objectively assessed mastoidectomy skills of residents in operating room , they evaluated 15 residents and they concluded very good internal consistency of 0.930, similar results were reported in studies objectively assessing temporal bone drilling skills  and residency based training for transoral robotic surgery . Both studies reported OSAT as reliable tool for resident evaluation. Other studies developed their own OSAT tools for evaluation of residents for surgeries like septoplasty (Obeid et al) , endoscopic sinus surgeries (Marglani et al) , tracheostomy (Al Qahtani KH et al)  , thyroidectomy (Stack et al)  and bronchoscopy (Ishman et al) . They evaluated residents both on GRS and TCL they all found good internal consistency in their results of more than 0.9. A study conducted by Roberson et al  for the development of tool for assessment of tonsillectomy procedure in 2005 evaluated 45 residents at various level of training, they used Kappa as coefficient for inter observer variability, they found OSAT as very useful tool for evaluation and recommended to be used as reliable method for surgical evaluation in otolaryngology residency program, all these OSATS tools published are summarized in [Table 3].
Continuous assessment and real time feedback are important in the structured learning of surgical skills and will prove to be increasingly important in the documentation of trainees surgical competency, some essential assessment tools are still lacking in the literature and will needs to be developed and validated by the TCL and GRS especially in the field of facsioplastic subspecialty like rhinoplasty and otology like tympanoplasty and cochlear implantations, and are worth future research, development and validation.
| Conclusion|| |
The objective assessment of surgical skills in otolaryngology head and neck surgery is still in its infancy. There is a need to reduce surgical errors therefore a demand for more objective measures of skills performance has become paramount. Assessment of surgical skills using OSAT program has the potential to be used as a reliable tool for allowing adequate monitoring of residents progress and should be implemented as part of the resident evaluation in otolaryngology program. Although many tools have been developed, still further refinements of the available tools and development of more procedure specific tools are necessary.
Special thanks for Dr Ahmed Sheikh in helping to review the literature edit and review this article.
| References|| |
Martin JA, Regher G, Reznick RK et al, objective structured assessment of technical skills (OSATS) for surgical residents. Br J Surg.
Datta V, Bann S, Beard J, Mandalia M, Darzi A. Comparison of bench testevaluations of surgical skill with live operating performance assessments. J AmColl Surg.
2004 Oct; 199(4): 603-6. PubMed PMID: 15454146.
Francis HW, Masood H, Chaudhry KN, Laeeq K, Carey JP, Della Santina CC, LimbCJ, Niparko JK, Bhatti NI. Objective assessment of mastoidectomy skills in the operating room. Otol Neurotol.
2010 Jul ;31(5):759-65. doi:10.1097/MAO.0b013e3181e3d385
Zirkle M, Taplin MA, Anthony R, Dubrowski A. Objective assessment of temporal bone drilling skills. Ann Otol Rhinol Laryngol.
2007 Nov; 116(11): 793-8.
Curry M, Malpani A, Li R, Tantillo T, Jog A, Blanco R, Ha PK, Califano J, Kumar R, Richmon J. Objective assessment in residency-based training for transoral robotic surgery. Laryngoscope.
2012 Oct; 122(10): 2184-92.
A Obeid AA, Al-Qahtani KH, Ashraf M, Alghamdi FR, Marglani O, Alherabi A. Development and testing for an operative competency assessment tool for nasal septoplasty surgery. Am J Rhinol Allergy.
2014 Jul-Aug; 28(4): e163-7.
Marglani O, Alherabi A, Al-Andejani T, Javer A, Al-Zalabani A, Chalmers A. Development of a tool for Global Rating of Endoscopic Surgical Skills (GRESS) for assessment of otolaryngology residents. B-ENT.
2012; 8(3): 191-5.
Al-Qahtani Kh, Alkhalidi Am, Islam T. Tool For Assessing Surgical Tracheostomy Skills In Otolaryngology Residents. B-ENT.
2015; 11(4): 275-80.
Stack Bc Jr, Siegel E, Bodenner D, Carr Mm. A Study of Resident Proficiency with Thyroid Surgery: Creation of a Thyroid-Specific tool. Otolaryngol Head Neck Surg.
2010 Jun;142(6):856-62. Doi: 10.1016/J.Otohns.2010.02.028.
Ishman SL, Benke JR, Johnson KE, Zur KB, Jacobs IN, Thorne MC, Brown DJ, Lin SY, Bhatti N, Deutsch ES. Blinded evaluation of interrater reliability of an operative competency assessment tool for direct laryngoscopy and rigid bronchoscopy. Arch Otolaryngol Head Neck Surg.
2012 Oct; 138(10): 916-22. P
Roberson DW, Kentala E, Forbes P. Development and validation of an objective instrument to measure surgical performance at tonsillectomy. Laryngoscope.
Ahmed A, Ishman SL, Laeeq K, Bhatti NI. Assessment of improvement of trainee surgical skills in the operating room for tonsillectomy. Laryngoscope.
2013 Jul; 123(7): 1639-44. doi: 10.1002/lary. 24023. Epub 2013 Mar 11.
Lin SY, Laeeq K, Ishii M, Kim J, Lane AP, Reh D, Bhatti NI. Development and pilot-testing of a feasible, reliable, and valid operative competency assessment tool for endoscopic sinus surgery. Am J Rhinol Allergy.
2009 May- Jun; 23(3):354-9.
Reznick RK. Teaching and testing technical skills. Am J Surg.
1993 Mar; 165(3):358-61. PubMed PMID: 844754
Diaz Voss Varela DA, Malik MU, Thompson CB, Cummings CW, Bhatti NI, Tufano RP Comprehensive assessment of thyroidectomy skills development: a pilot project. Laryngoscope.
2012 Jan;122(1): 103-9. doi: 10.1002/ lary.22381.
[Table 1], [Table 2], [Table 3]