INTRODUCTION
Up to 70% of all anterior cruciate ligament (ACL) injuries occur via a non-contact mechanism.1 The most frequent movement pattern associated with non-contact ACL injuries in young female athletes includes a deceleration event paired with a change of direction on a planted foot.1 It is also known that female athletes are two to ten times more likely to rupture their ACL when compared to their male counterparts.2–7 Neuromuscular control deficits at the hip and trunk have been theorized to result in altered lower extremity (LE) neuromuscular control and subsequently higher knee abduction loads in female athletes.8–11 Specifically, lateral trunk motion with the body shifted over the stance limb has been found to be related to high knee abduction moments and medial knee collapse.12,13 Additionally, females demonstrate altered hip recruitment strategies with decreased gluteal activation, greater hip moments, and increased quadriceps activation that result in increased knee abduction moments and higher ACL injury risk.14–19
Numerous qualitative tools to assess movement quality utilizing two-dimensional (2-D) analysis have been found to be reliable and valid.20–24 However, the majority of these assessment tools only evaluate landing mechanics and do not address cutting movement. Recently, several assessment tools utilizing 2-D video have been introduced to assess cutting technique. Weir et al. assessed the reliability and validity of a 2-D video-based screening tool to predict peak knee moments during an unplanned 45-degree side-step cut in a group of junior (age = 15.1 ± 1.2 years) and senior (age = 22.1 ± 2.3 years) elite female field hockey players.25 The screening tool involved 2-D kinematic measurement of frontal and sagittal plane variables using video analysis software and reported poor to excellent intra-rater and inter-rater reliability.26 The Cutting Movement Assessment Score (CMAS), a qualitative scoring system to evaluate a 90-degree cutting maneuver, was found to be a reliable and valid tool to assess risky movement patterns during a 90-degree cutting task in college-aged athletes.27
The Cutting Alignment Scoring Tool (CAST), demonstrates good inter-rater and intra-rater reliability for the assessment of LE and trunk alignment in young athletes (age=14.7+1.2 years).26 The CAST involves dichotomous scoring of four frontal plane movement variables observed from 2-D video during a planned 45-degree side-step cut. However, the CAST may not sufficiently address sagittal plane foot and ankle assessments. Decreased ankle plantarflexion angles and decreased knee flexion angles have been associated with higher knee joint loads during cutting maneuvers.28–31 In an attempt to assess both frontal and sagittal plane trunk and LE alignment during a 45-degree side-step cut, the authors of this study developed the Expanded Cutting Alignment Scoring Tool (E-CAST). The primary purpose of this study was to examine the reliability of the E-CAST among physical therapists. This study consisted of three aims: 1) to assess the inter-rater reliability of the E-CAST, 2) to assess the intra-rater reliability of the E-CAST, and 3) to examine rater agreement of each component of the E-CAST. The hypotheses were: 1) there would be good–to-excellent inter-rater reliability, 2) there would be good-to-excellent intra-rater reliability, and 3) there would be good-to-nearly perfect agreement in the E-CAST variables including; cut width, trunk lean, knee flexion and plantarflexion, and moderate agreement in knee valgus variables of the E-CAST.
METHODS
Study Design
A repeated measures study design was used. To achieve the study aims, inter-rater and intra-rater reliability were calculated based on the first and second reliability tests. The study protocol was developed based on the Declaration of Helsinki and ethical standards in sport and exercise science research.32 Institutional Review Board approval was obtained prior to commencement of the study.
Participants
A total of 25 adolescent female athletes (age 13.8 ± 1.4 years, mass 52.4 ± 9.3 kg, height 161.7 ± 6.0 cm) were recruited from local middle school, high school and club sport teams (Table 1).
Inclusion criteria were: 1) age between 12 and 17 years, and 2) actively participating in sports requiring cutting and pivoting in the prior 12 months. The following exclusion criteria were used: 1) LE injury within the prior six months, 2) past history of LE surgery, 3) a positive response on the Physical Activity Readiness Questionnaire (PAR-Q+), and 4) history of scoliosis. The PAR-Q+ was used to determine the participants’ readiness and safety for physical activity. A positive response of the PAR-Q+ indicates the need to seek further advice from a physician prior to engaging in physical activity.33 All participants provided written informed assent, and their parent or legal guardian provided signed consent. Data collection was performed in a movement science lab at a local sports medicine center.
Data Collection
Prior to performing the 45-degree side step cut task, a five-minute warm up on an exercise bike (Matrix Fitness, Cottage Grove, WI) was performed. Participants practiced the side step cut three times in each direction or until they felt comfortable with the procedure. They were instructed to sprint at 80% of their maximum speed in a forward direction toward the “opponent cone” and to pivot and perform the side step cut (Figure 1).
This procedure was modeled after a testing protocol described by McLean et al.34 Specifically, participants decelerated, planted on the right foot, and performed a side step cut, running in the left direction between cones placed along a 45-degree line of progression. The procedure was repeated planting on the left foot and running to the right direction (Figure 1). Then, participants completed three trials planting on the right LE and three trials planting on the left LE, with a trial considered “good” if the subject’s foot landed within the stance/pivot area necessary for successful completion of the task. The testing order was standardized for all participants following the protocol by Butler et al.26 Video data were captured at 60 frames per second with 1080p quality using three Sony RX10 IV cameras adjusted to 36 inches tall. Two cameras were positioned 136 inches from either side of the stance/pivot area, and one camera was positioned 146 inches in front of the stance/pivot area. Participants performed a total of six cutting maneuvers with one trial randomly selected for analysis. All videos were slowed by 50% for visual analysis and participants’ faces were blurred using Corel VideoStudio.
Qualitative Measurement Scale
A clinically established checklist, E-CAST, was developed to examine the quality of trunk and LE movement during the cutting maneuver based on 2-D video. The checklist was devised based on the previously reported scoring system (CAST). It involves a dichotomous rating system, with scoring defined as “1” when a movement fault was present and “0” when optimal movement patterns were observed. The E-CAST evaluates the original frontal plane variables of the CAST including; trunk lean to the opposite direction of the cut, increased cut width, knee valgus at initial load acceptance (Static Evaluation), and knee valgus throughout the cutting task (Dynamic Evaluation) as well as two new variables that are assessed in the sagittal plane including; plantarflexion angle and knee flexion angle. The E-CAST checklist is shown in Table 2.
Raters
Two raters consisting of two pediatric sports medicine doctors of physical therapy were chosen because of their clinical roles in treating young athletes. The two raters were chosen from the same institution and had seven and five years of clinical experience, respectively. The raters independently viewed a total of 25 videos. All raters provided their consent to participate in the current study.
Procedures
One of the six trials, which consisted of three right LE and three left LE cutting maneuvers from each of the 25 subjects, was chosen at random. A review of current research in this area led to the sample size selection. The videos were provided to each rater along with a reference sheet containing images demonstrating optimal and sub-optimal movement strategies and elaborated definitions used in the E-CAST (Appendix A). The raters were instructed to view the videos independently. They were allowed to review the videos as many times as necessary and could pause the video as needed. All videos were evaluated using each rater’s personal smart phone device. The raters were given one week to complete the first reliability session. After the first reliability session, a two-week wash-out period was given. Next, the second reliability session was performed, using the same method outlined for the first reliability session. The sequence of videos was randomized in the second reliability session.
Statistical Analysis
Reliability was determined by calculating intraclass correlation coefficients (ICC) for the E-CAST total scores, with a 2-way mixed-effects model and 95% confidence intervals (95% CIs) for inter-rater and intra-rater reliability. For the first aim, the individual and cumulative inter-rater reliabilities were calculated within the first and second reliability sessions. The individual and cumulative intra-rater reliabilities were calculated between the first and second reliability sessions. ICC values less than 0.50, between 0.50 and 0.75, between 0.75 and 0.90, and greater than 0.90 were defined as poor, moderate, good and excellent reliability, respectively.35 To attain study aim 2, a kappa coefficient was calculated for each of the checklist variables using the formula; k= Pr(a) – Pr(e)/1 – Pr(e), where Pr(a)= relative observed agreement between raters and Pr(e)= hypothetic probability of chance agreement. The kappa coefficient was interpreted based on the scale of Landis and Koch36 with 0.01-0.20 being slight, 0.21-0.40 fair, 0.41-0.60 moderate, 0.61-0.80 good, and 0.81-1.00 almost perfect. All statistical analyses were conducted using SPSS Statistics 22 (IBM Corp. Released 2013. IBM SPSS Statistics for Windows, Version 22.0. Armonk, NY: IBM Corp).
RESULTS
Inter-rater reliability for the first reliability session was moderate (ICC: 0.73, 95% CI 0.43-1.20) and inter-rater reliability for the second reliability test was moderate (ICC: 0.70, 95% CI 0.39-1.01). The cumulative inter-rater reliability, a combination of first and second inter-rater reliability, was moderate (ICC: 0.71, 95% CI 0.50-0.91). Intra-rater reliability for Rater 1 was moderate (ICC: 0.72, 95% CI 0.43-1.02) and intra-rater reliability for Rater 2 was good (ICC: 0.84 95% CI 0.60-1.07, Table 3). The cumulative intra-rater reliability of the two raters was good (ICC: 0.78, 95% CI 0.59-0.96, Table 3). Intra-rater kappa coefficients for each variable are presented in Table 3 and ranged from moderate to excellent for all variables (k = 0.50-0.84). Inter-rater kappa coefficients are presented in Table 4 and ranged from slight to excellent for all variables (k = 0.20-0.90).
DISCUSSION
The primary purpose of this study was to assess the intra-rater and inter-rater reliability of the E-CAST. The E-CAST demonstrated moderate inter-rater reliability (cumulative ICC: 0.71, 95% CI 0.50-0.91) and good intra-rater reliability (cumulative ICC: 0.78, 95% CI 0.59-0.96). The findings did not support the first hypothesis that the E-CAST would demonstrate good-excellent inter-rater reliability because only moderate inter-rater reliability was found. However, the second hypotheses was supported with the E-CAST demonstrating good-excellent intra-rater reliability. The third hypothesis was not supported as moderate agreement was found for trunk lean, cut width and plantarflexion and only fair agreement was found for knee flexion. Additionally, valgus variables demonstrated good-to-almost perfect agreement.
The findings of this study are generally in agreement with the previous work which reported good cumulative inter-rater reliability (ICC: 0.81, 95% CI 0.64-0.91) and good cumulative intra-rater reliability (ICC: 0.75, 95% CI 0.59-0.85) of the CAST. When comparing the cumulative inter-rater reliability of the E-CAST to the cumulative inter-rater reliability of just the two physical therapist raters for the CAST, the E-CAST demonstrated higher inter-rater reliability (ICC: 0.71, 95% CI 0.50-0.91 vs ICC: 0.46, 95% CI 0.28-0.61). This may have been a result of the raters’ greater experience with assessing cutting movement errors as the same two physical therapists were used to determine the reliability of both the CAST and the E-CAST. When comparing the cumulative intra-rater reliability of the E-CAST to the cumulative intra-rater reliability of just the two physical therapist raters for the CAST, the E-CAST (ICC: 0.78, 95% CI 0.59-0.96) demonstrated slightly higher intra-rater reliability than the CAST (ICC: 0.77, 95% CI 0.43-0.91). A study conducted by Dos’ Santos et al reported a similar finding. According to his study, moderate inter-rater reliability (ICC =0.69) and excellent intra-rater reliability (ICC =0.95) were found when utilizing a qualitative scoring system to evaluate a 90-degree cutting maneuver in collegiate athletes.27 There are, however, several differences in study design. Dos’ Santos et al27 reported intra-rater reliability of the CMAS for only one rater. The current study reported the average of two raters which may have contributed to the lower intra-rater reliability found in the E-CAST compared to the CMAS. Additionally, Dos’ Santos et al. only used a one-week wash out period compared to a two-week washout period used in this study.27
In a another study by Weir et al., an unplanned 45-degree side step cut was assessed in 15 junior (age 15.1 + 1.2 years) and 15 elite senior (age 22.1+ 2.3 years) female field hockey players utilizing 2-D video based measurements of whole body kinematics.26 Weir et al. found excellent intra-rater reliability (ICC: 0.99-1.00) for angular measurements including dynamic knee valgus angle, trunk lateral flexion angle, knee flexion angle, thigh abduction angle and trunk flexion angle, and poor to good reliability (ICC: 0.38-0.54) for displacement based measurements including foot placement and dynamic medial knee shift.26 Weir et al. found good to excellent inter-rater reliability (ICC: 0.66-0.97) for all angular measurements and poor to excellent inter-rater reliability (ICC: 0.22-0.87) for displacement based measurements.26 Again, several differences in study design need to be discussed between the work of Weir et al. and this study. First, Weir et al. used an un-planned cutting task versus a planned cutting task, which was used in our study. An un-planned cutting task has been shown to result in greater knee joint loads when compared to planned cutting maneuvers.37 The work of Weir et al. utilized 2-D measurement of full body kinematics compared to the qualitative assessment used in this study.26 It is unknown if 2-D measurement is a more reliable and valid method for evaluating trunk and LE alignment during a cutting task, and further work in this area is needed. However, a primary concern of incorporating 2-D measurements is that it may increase the complexity of the tool, and therefore, time and effort for evaluators, potentially reducing ease-of-use in clinic and in the field.
The second aim of this study was to evaluate rater agreement of each component of the E-CAST. Almost perfect kappa coefficients were found for static valgus and good kappa coefficients were found for dynamic valgus. Moderate kappa coefficients were observed for trunk lean, cut width, and plantarflexion, while only fair kappa coefficients were noted for knee flexion. The hypothesis that there would be good to almost perfect agreement for trunk lean, cut width, knee flexion and plantarflexion variables was not supported. Only moderate agreement was found for trunk lean, cut width and plantarflexion and only fair agreement was found for knee flexion. The hypothesis that valgus variables would demonstrate moderate agreement was also not supported. The results were not consistent with previous work which found only fair kappa coefficients for static and dynamic valgus variables and almost perfect kappa coefficients for cut width.26 One potential explanation for the difference in results is that the CAST assessed reliability using six raters consisting of two physical therapists, two sports medicine physicians and two athletic trainers, while the E-CAST only assessed the reliability between two physical therapists. Furthermore, in this study almost perfect agreement was found for static valgus and good agreement was found for dynamic valgus. In our perspective, this may be a result of having an additional sagittal plane view which may improve consistency with time point identification. The two sagittal plane variables (knee flexion angle and plantarflexion angle) demonstrated the lowest and third lowest inter-rater reliability (k = 0.32 and k = 0.53, respectively) which contrasted with the authors 'expectations. Initially, the authors speculated that sagittal plane variables would demonstrate higher inter-rater reliability than frontal plane variables and the addition of two sagittal plane variables in the E-CAST would increase its intra-rater and inter-rater reliability in comparison to the CAST. One explanation for this outcome may be a result of the increased range of motion that occurs in the sagittal plane. This wider range of motion in the sagittal plane compared to the more restricted range of motion that occurs in the frontal plane may increase each rater’s variability in movement fault identification.
It is currently unknown if qualitative screening tools evaluating cutting technique in young athletes are predictive of ACL injuries. Given that the two sagittal plane assessments demonstrated the lowest and third lowest inter-rater reliability, future work should explore the predictability of frontal plane variables alone in identifying athletes who are at high risk for ACL injury. If frontal plane variables are found to be predictive of ACL injury, sagittal plane assessments may not be needed. This would simplify the screening tool, which would likely improve its adoptability by coaches. The development of screening tools that are able to accurately identify high-risk cutting movements with one camera view may provide coaches and practitioners with an efficient and effective strategy to screen athletes for ACL injury and enhance injury prevention interventions.
Limitations
The current study had a number of limitations. First, the E-CAST only evaluated reliability of physical therapists. Coaching staff spend the most time working with athletes; thus, determining the reliability of the E-CAST amongst coaches would greatly increase its clinical utility. Coaches are likely best positioned to perform team-based injury prevention screenings. Providing coaches with a reliable and valid screening tool to identify athletes who are at high risk for ACL injury would help them in determining who would benefit most from injury prevention interventions. Future research should aim to determine the reliability of the E-CAST with coaches. Additionally, this study only assessed the reliability between two raters, future studies should evaluate the tool’s reliability between multiple raters. Next, the operational definitions for each variable were written with varying criteria. For example, an approximate degree reference was provided for trunk lean and knee flexion, a body reference was provided for cut width and plantarflexion, and a qualitative description was provided for the knee valgus variables. This variability may have contributed to rater confusion when using the tool. Future work should consider utilizing consistent reference terminology when defining movement fault criteria. It should also be acknowledged that this study used a planned cutting task. Different outcomes may be expected with the use of an unplanned cutting task which has been shown to result in greater knee joint loads when compared to planned cutting maneuvers.37 While the use of an unplanned cut may be more generalizable, it is more difficult to capture in the lab and in both the in-clinic and on-field settings. With the use of an unplanned cut, two sagittal plane cameras would be needed in order to capture LE biomechanics on the stance limb. An additional camera view would increase the amount of equipment needed and the complexity of the screening tool for on field or clinic use. Additionally, in the lab setting, using a planned cut allowed for reduced errors in data collection, as this allowed the athlete to line up with the force plate without having to target a spot on the floor. Lastly, it is unknown if the E-CAST is a valid tool for predicting ACL injury risk during a cutting maneuver. It is important to investigate whether or not 3-D kinematic variables are correlated with visually identified movements. Future studies should aim to determine its predictive validity and its criterion validity with 3-D motion capture.
CONCLUSION
This study demonstrated that the E-CAST, a qualitative evaluation tool using frontal and sagittal plane videos to identify at-risk movements for ACL tear in side-step cutting, demonstrated moderate inter-rater and good intra-rater reliability among physical therapists. However, the addition of sagittal plane variables in the E-CAST resulted in a decrease in inter-rater reliability of the tool. These findings suggest that the E-CAST can be used as a reliable tool to evaluate trunk and LE alignment in the frontal and sagittal plane during a cutting task by physical therapists. However, the use of a frontal plane assessment alone may be more reliable. Future work is recommended to determine the predictive validity of the CAST and the E-CAST in identifying individuals at risk for ACL injury.
