Reliability of a Novel Qualitative Assessment Tool for Evaluating a Run Plant Task

INTRODUCTION

Sports injuries are a major concern in athletics and despite preventive measures, the incidence of these injuries remains high. Many sports injuries affect the lower extremity (LE), making injury prevention a critical focus in sports medicine. Knee injuries, which constitute 41% of sports-related injuries, further highlight the need for effective prevention strategies.¹ Among knee injuries, anterior cruciate ligament (ACL) tears, meniscus tears, posterior cruciate ligament tears, and damage to articular cartilage are prevalent and can significantly impact an athlete’s career and quality of life.¹ A 2019 injury surveillance study reported roughly eight LE injuries per 10,000 athletic exposures, with greater injury rates in girls compared to boys.² ACL injuries are of particular concern with data showing a 2.3% annual increase in the rate of ACL injury in children and adolescents over a 20-year period.³

Rapid deceleration during sport is a common mechanism of ACL injury, often occurring immediately following foot strike with an extended knee position, excessive hip flexion and posterior trunk position relative to the center of mass.^4–6 Furthermore, during change of direction tasks, trunk and LE alignment have been shown to influence knee joint loading. Specifically, increased hip internal rotation and a more internal foot progression angle have been associated with increased knee abduction moments during sidestep cutting tasks.^7,8 Poor trunk alignment, including a lateral trunk lean over the plant foot or failure to rotate the trunk in the direction of the cut, have also been associated with higher peak knee abduction moments.^9,10 Additionally, greater knee valgus angles during cutting tasks have been associated with increased knee abduction moments.^11,12 Collectively, these movement pattern deviations are linked to higher knee abduction moments, a known ACL injury risk factor.^13–15 Injury prevention strategies that aim to address movement errors may help decrease the risk of ACL injury associated with rapid deceleration. However, to maximize efficiency and effectiveness of preventive interventions, screening tools that identify individuals at risk are necessary.

Two-dimensional (2D) qualitative and quantitative scoring tools have been proposed as efficient and cost-effective solutions for analyzing trunk and LE movement.^15–19 However, these tools have primarily focused on landing and change of direction tasks.^7–11 To the author’s knowledge, only one study has evaluated the reliability and validity of a 2D assessment of trunk and LE movement during a deceleration task.¹² The scoring system utilized a three-dimensional (3D) motion capture environment with high-speed cameras and a force plate to capture the resultant ground reaction force vector.²⁰ The ground reaction force vector was then used to assess five 2D scoring criteria (frontal plane knee angle, pelvis angle, trunk angle, knee flexion angle, and hip-trunk flexion angle).¹² While the authors reported excellent reliability and validity compared to 3D motion capture, the necessary use of force plates reduces the tool’s clinical feasibility.¹² Therefore, the purpose of this study was to assess the intra- and inter-rater reliability of a qualitative assessment of trunk and LE alignment during a run plant task. The hypothesis was that the qualitative assessment tool would demonstrate good to excellent intra- and inter-rater reliability.

MATERIALS AND METHODS

Study Design and Participants

A repeated measures study design was used with two Raters and 25 participants. A convenience sample of healthy female athletes (12-17 years of age) were recruited from local school and club sport teams. This study was part of a larger study aim which required participants to have participated in a sport requiring cutting or pivoting in the 12 months prior to testing. Participants were excluded if they reported a LE injury in the prior six months or a history of LE surgery. All participants provided informed written assent, and a parent or guardian provided written consent prior to testing. The study was approved by the host Institutional Review Board prior to study commencement.

Data Collection

Participants performed a run plant task while video data were recorded at 60 frames per second with 1080p quality (Sony Cyber-shot DSC-Rx10). Two cameras simultaneously recorded the participant. Cameras were positioned 136 inches (345.4 cm) in front of and 136 inches (345.4 cm) from either side of the participant at a height of 36 inches (91.4 cm). Participants were instructed to run forward and plant their left foot into the box outlined on the floor and then backpedal to the starting line (Figure 1). Trials in which their left foot was completely planted in the box were counted as acceptable trials. The left LE was chosen due to camera placement in the motion laboratory that only allowed for recording of the left side of each participant. A total of three trials were collected, and a representative trial was selected based on the fastest entry velocity.

Figure 1.Depiction of the run plant task.

Qualitative Assessment

A qualitative checklist assessing trunk and LE alignment was developed by the study team based on the work of Butler et al. and Ulman et al.^10,13,14 Movement fault criteria are shown in Figure 2 and Table 1. Each criterion was assessed during the final plant of the run plant task. The final plant was defined as the last foot contact prior to changing directions. The criteria of trunk lean, foot position, and dynamic valgus were observed in the frontal plane, and the criteria of sagittal alignment and preparatory deceleration steps were observed in the sagittal plane. The criterion of trunk lean was based on the work of Ulman et al., and assessed during the loading phase.¹³ The loading phase was defined as the period from initial foot contact of the final plant to the cessation of knee flexion. Specifically, in the frontal plane, if the midpoint between the shoulders moved outside of the plant leg foot in the ipsilateral direction, the trunk lean criterion was marked as present. The criterion of foot position was defined by the work of Dos Santos et al., and assessed during the loading phase.^9,15 Specifically, if the foot was positioned to point inward (internally rotated), or positioned to point outward (externally rotated), the foot position criterion was marked as present. The criterion of dynamic valgus was defined by the work of Butler et al., and assessed during the loading phase.^10,14 The criterion was marked as present if the plant limb demonstrated valgus (thigh adduction, thigh internal rotation, genu valgum, or knee abduction). The criterion of sagittal alignment was defined by the work of Ulman et al., and assessed at the end of the loading phase of the final plant.¹³ Specifically, if the trunk segment did not remain roughly parallel with the lower leg (anterior edge of tibia), the criterion of sagittal alignment was marked as present. Finally, preparatory deceleration steps were defined by the work of Mulligan et al. and Havens et al., and assessed during the entire run plant task.^16,17 During the task, if there was no noticeable change in step length or less than three preparatory deceleration steps, the criterion was marked as present. Preparatory deceleration steps were defined as smaller steps as the athlete approached the target.

Figure 2.Trunk and lower extremity alignment during a run plant task

Raters identified movement faults by viewing the recorded video data and scoring the observed quality of movement using the movement fault criteria. Specifically, if a movement fault was present, the participant was awarded a score of 1. If the movement fault was not observed, the participant was awarded a score of 0. Individual items were then summed to generate a total score, with higher scores indicating poorer movement quality. Additionally, to assess the reliability of the scoring criteria without the confounding factor of the Rater identifying the timepoint of interest, static images were also scored according to the movement fault criteria. The images were created by an independent member of the study team not involved in the scoring process. The timepoint in which the images were selected for the sagittal view was based on the sagittal alignment criteria of greatest trunk flexion. For the frontal image, the timepoint in which the plant limb knee was most medial was utilized.

Procedures

Two Raters consisting of one physical therapist (Rater 1) with 14 years of clinical experience and one biomechanist (Rater 2) with extensive motion analysis experience were chosen to score both the videos and the static images. The Raters attended a one-hour training session prior to independently viewing and scoring the 25 videos and the associated static images. Each Rater was provided with the videos and images along with a scoring sheet containing the movement fault definitions, time point definitions, and reference images showing correct and incorrect performance. Each Rater was permitted to view the videos and images as many times as they wanted and could adjust the video playback speed and pause as needed. Videos were scored first, followed by static images. One week was provided to complete the scoring, followed by a two-week wash out period. After the wash out period, videos and images were randomized, and a second one- week scoring session was completed.

Statistical Analysis

Descriptive statistics were computed for all scoring items and total scores for each round. To distinguish systematic scoring differences (bias) from measurement consistency (reliability), two complementary sets of analyses were performed. Intra-rater differences in single items and total scores were identified using McNemar tests and Wilcoxon signed-rank tests, respectively. Similarly, inter-rater differences in scoring items and total scores for each round were identified using Cochran’s Q tests and repeated measures ANOVAs, respectively. These hypothesis-testing procedures were used to determine whether Raters or scoring sessions differed systematically in their scoring (i.e., presence of bias). Post hoc analyses were subsequently performed for positive findings to identify paired differences. Specifically, pairwise McNemar tests and Wilcoxon signed-rank tests were used. Statistical significance was determined when p<0.05 (R programming, version 4.3.0, R Development Core Team).

To quantify measurement consistency independent of systematic bias, interclass correlation coefficients (ICCs) were computed to assess reliability for each scoring item and the total score using a two-way mixed-effect model as well as 95% confidence intervals (95% CIs). Intra-rater reliabilities were calculated between the first and second scoring rounds and inter-rater reliabilities were calculated within each round. ICC values were defined as poor (less than 0.50), moderate (between 0.50 and 0.75), good (between 0.75 and 0.90), and excellent (greater than 0.90).¹⁸ Given that several scoring items were binary and exhibited skewed distributions, kappa statistics and percent agreement were also reported to describe categorical agreement.

RESULTS

Videos of 25 female subjects (15.65 ± 1.10 years, 166.28 ± 6.86 cm, 60.48 ± 6.88 kg) performing a run plant task were assessed by two Raters, at two sessions separated by a two-week wash out period. There were no significant differences in scoring items or total score between rounds for each Rater, reflecting consistency in scoring. (Table 2). Of note, differences in dynamic valgus and preparatory deceleration steps could not be compared between rounds as both Raters consistently marked these items as present for all 25 subjects in both rounds. Additionally, there was no variance in the scoring for trunk lean and lateral foot position, as all 25 videos were scored as not present for both rounds. Intra-rater ICCs ranged from 0.43 and 0.69 for Rater 1, and 0.54 and 0.88 for Rater 2 across scoring items (Table 3). Reliability was moderate for sagittal alignment and good to moderate for medial foot position across Raters. However, poor to moderate intra-rater reliability was found for the total score.

When examining inter-rater reliability, there were no significant differences in sagittal alignment or medial foot position scores in both rounds (Table 2). For the total score, Raters significantly differed in round 1 (mean difference: 0.24, p = 0.048), but not in round 2 (mean difference: 0.24, p = 0.066). Differences could not be evaluated due to the lack of variance in the identification of trunk lean and lateral foot position. Kappa values for dynamic valgus and preparatory deceleration steps are not reflective of the agreement between Raters and rounds due to the skew in positive responses. Percent agreement was 96% for dynamic valgus and preparatory deceleration steps in both rounds. Inter-rater ICCs were found to be moderate for sagittal alignment and medial foot position across both rounds. Moderate inter-rater reliability was found for the total score in round 1, however, there was poor reliability between Raters in round 2. Overall, item and total scores failed to reach good reliability between Raters (Table 3).

Comparing intra-rater reliability when the scoring criteria were applied to the static images, there were no significant differences in scoring items or total score between rounds for each Rater (Table 2). Intra-rater ICCs ranged from 0.54 and 0.83 for Rater 1, and 0.46 and 0.88 for Rater 2 across scoring items (Table 4). Reliability was moderate to good for sagittal alignment and medial foot position. Total score showed moderate reliability across Raters. Between Raters, there was no significant difference in round 1 for any scoring item or total score, however sagittal alignment was different between Raters in round 2. Inter-rater ICCs were found to be moderate for medial foot position between Raters. However, poor inter-rater reliability was found for sagittal alignment and the total score. Overall, intra-rater reliability improved for sagittal alignment and total score for Rater 1 with the use of the images over the videos. Inter-rater reliability decreased for sagittal alignment in both rounds and round 1 total score when scoring the images compared to videos.

DISCUSSION

The purpose of this study was to assess the intra- and inter-rater reliability of a qualitative assessment of trunk and LE alignment during a run plant task. The hypothesis, that the tool would demonstrate good to excellent intra-and inter-rater reliability, was only partially supported. Medial foot position was the only item to achieve good intra-rater reliability. Similarly, medial foot position, and sagittal alignment were the only items that demonstrated good inter-rater reliability. No criteria achieved excellent reliability and intra- and inter-rater reliability for the total score was poor to moderate.

In both rounds, the items of dynamic valgus and preparatory deceleration steps were consistently rated as present. While sagittal alignment and medial foot position demonstrated moderate to good intra-rater reliability, the positive skew and lack of variability in dynamic valgus and preparatory deceleration steps likely contributed to the overall lower intra-rater reliability for the total score. This suggests that the operational definitions for these items may not have been clear enough to discriminate between the presence or absence of these movement faults during the run plant task. It is possible that the definitions may have been too restrictive, resulting in consistent ratings of “present” or “not present.” As such, the criteria may not have adequately captured the range of movement quality during the run plant task. Future work should aim to refine the definitions for these items to better support their inclusion in the assessment tool.

Dynamic knee valgus is a known risk factor for non-contact ACL injury.¹⁹ While prevalent in the literature and associated with high knee joint loads, lack of true definitions of “high” or “increased” make the extent of dynamic knee valgus difficult to assess visually. Additionally, due to the multiplanar contributions to dynamic knee valgus, 2D analysis becomes challenging. Erdman et al. established a 2D scoring criterion for knee valgus that demonstrated good validity with 3D motion capture during a single-leg squat, single-leg drop landing, and a double-leg vertical jump.²⁰ Given that frontal plane knee motion and knee abduction angles differ widely based on the task,²¹ it is recommended that similar criterion are established and validated for the run plant task.

With regard to the number of preparatory deceleration steps, the current study utilized a three-step cut off (less than three steps indicated the presence of the movement fault). The approach for the run plant task was 6-8 meters in length which may not have been long enough for athletes of all heights to allow three or more preparatory deceleration steps. Future work should consider a longer approach distance or use a lower number of preparatory deceleration steps as a criterion. Additionally, the cut off of three preparatory deceleration steps may not have been specific enough to discriminate between the presence or absence of an effective deceleration strategy. Effective braking strategies require multiple short and quick steps to adequately slow down segments prior to a change of direction.²² Several works have examined the biomechanical profiles of deceleration, focusing on the penultimate step that precedes the final step.^15,17,23,24 These studies have found that athletes who are able to generate higher braking forces during the penultimate step, are better able to dissipate force and slow down segments, thus reducing braking force during the final plant.^15,17,23,24 However, these studies have focused largely on cutting tasks and have not explored assessment of braking strategies qualitatively. Future work, examining the optimal number of preparatory deceleration steps to adequately slow down segments during a run plant task would be beneficial to enhance the definition for this criterion used in this study.

No variance was observed in the scoring of lateral foot position or trunk lean, with these items marked as not present in all 25 videos for both rounds. This again suggests that the operational definitions for these items may have been too strict, leading to consistent scoring of not present. It is also possible that these movement faults may not be prevalent enough during a run plant task to warrant inclusion in the assessment tool. When comparing medial versus lateral foot position, previous research has shown that medial foot position is associated with additional suboptimal movement patterns. The findings of Riesenberg et al. showed that athletes who exhibited an internal foot progression angle (medial foot position) were more likely to demonstrate an upright trunk position and increased trunk rotation toward the cut direction. Given these associations and the absence of lateral foot position in the current study, medial foot position may be more important to include when assessing movement quality during a run plant task than lateral foot position.²⁵ As such, the inclusion of medial foot position is recommended in future qualitative assessment tools.

Frontal plane trunk lean is considered an important variable for assessment of dynamic tasks such as hopping and lateral changes of direction due to its known impact on knee joint loading. Dempsey et al. looked at the effect the cutting technique on knee joint loading in fifteen healthy males. They found that a trunk lean in the opposite direction of the cut produced greater knee valgus moments at the knee.²⁶ Although frontal plane trunk lean is widely considered an important movement deviation that may increase knee joint loading during jumping and lateral change of direction tasks, the current study suggests that it may not be valuable when assessing a run plant task due to its low prevalence.^27–29
The poor to moderate inter-rater reliability observed for the total score was likely also influenced by the lack of variance and skew in positive responses for the items of dynamic valgus and preparatory deceleration steps. However, inter-rater reliability was moderate to good for sagittal alignment and medial foot position across both rounds, suggesting that these items may be useful. In a systematic review, Belkhelladi et al. reported decreased knee flexion angle and increased trunk extension (upright trunk) as contributing factors to increased load on the ACL.³⁰ Similarly, medial foot position has been associated with several suboptimal movement patterns known to increase knee joint loads during change of direction tasks.²⁵ Given the acceptable reliability for these items and their relationship to ACL loading, they should be considered for inclusion in future qualitative assessments of a run plant task.

When assessing the run plant task by applying the scoring criteria to the images, intra-rater reliability improved for sagittal alignment and the total score for Rater 1. This suggests that static images may offer more clinical value when examining an athlete’s movement quality across timepoints by the same clinician. However, inter-rater reliability decreased for sagittal alignment in both rounds and for the round one total score. Videos, while more complex, might provide more contextual information that supports between Rater agreement. While not explicitly explored in the current study, these findings also suggest that time point identification may have been more variable when applying the scoring criteria to the videos for Rater 1 than for Rater 2, particularly for the item of sagittal alignment. The definition of sagittal alignment requires the scorer to identify the end of the loading phase of the final plant, which was operationally defined as the cessation of knee flexion. During knee flexion, the tibia shifts either anteriorly or posteriorly in addition to the knee moving into flexion.³¹ These coupled movements can make it challenging to visually identify the time point of maximum knee flexion utilizing 2D video assessment, even when playback speed is slowed. It is possible that the physical therapist had a more difficult time distinguishing the exact point of cessation of knee flexion as indicated by the improvements in intra-rater reliability when the images were used for assessment.

Limitations

This study has several limitations that need to be considered. First, the two Raters were chosen from different sports medicine disciplines. While this increases generalizability of the tool, the lack of homogeneity may have impacted the study findings. Given the role physical therapists play in movement assessment for return to sport decision-making after ACL reconstruction, this study should be repeated with a group of physical therapist Raters. Next, the Raters received a single one-hour training on how to utilize and interpret the scoring tool for both the videos and the static images. This may have been insufficient given the complexity of identifying movement faults from both 2D video and still images. The limited training may have contributed to the lower reliability for some items. Future studies should examine the impact of expanded Rater training on the tool’s reliability. Finally, this study involved healthy adolescent female athletes, reducing generalizability to males, older athletes, and those who have had an ACL reconstruction. Given that clinicians would be most likely to utilize this tool as part of return to play decision-making after ACL reconstruction, future work should evaluate this tool in a post ACL reconstruction cohort.

CONCLUSION

The qualitative assessment of trunk and LE alignment during a run plant task demonstrated poor to moderate intra-rater reliability for Rater 1 and moderate to good intra-rater reliability for Rater 2. Inter-rater reliability ranged from poor to moderate across both rounds. Given the poor to moderate reliability of the total score, the tool is not currently recommended for stand-alone clinical decision-making. While the variables of medial foot position and sagittal alignment demonstrated moderate to good reliability, other variables may benefit from enhanced operational definitions that allow for improved discrimination between the presence and absence of movement faults during a run plant task. As such, future studies should examine the reliability of the tool with refined definitions for dynamic knee valgus, trunk lean, and preparatory deceleration steps. Additionally, examination of the tool in a post ACL reconstruction cohort or in athletes with other lower extremity injuries is necessary prior to use in return to play decision-making.

Conflicts of Interest

The authors declare no financial disclosures or conflicts of interest