INTRODUCTION
Anterior cruciate ligament (ACL) injury is most commonly treated with rehabilitation, and about 50% of patients undergo surgical reconstruction of the ruptured ligament.1 To sustain a second ACL injury after ACL reconstruction can be devastating; and this further increases the risk for long-term consequences, such as osteoarthritis and lower levels of physical activity and perceived quality of life.2–5 A common goal for patients treated with rehabilitation and ACL reconstruction is to return to sport (RTS).6,7 As participating in sports is associated with a risk of sustaining a second ACL injury,8 a primary concern is to evaluate the patients readiness for RTS, i.e. returning to sports with minimal risk for suffering a second ACL injury.
Responses to psychological patient-reported outcomes (PROs) can differ between patients who suffer a second ACL injury and those who do not, as patients who suffer a second ACL injury have reported greater scores, that is, better responses, on validated PROs.9,10 Collected evidence, including the results from systematic reviews on psychological outcomes after ACL reconstruction and clinical practice guidelines,11–13 as well as patients’ experiences,14 suggest that psychological PROs should be used in test batteries to help identify whether patients are well prepared for RTS after ACL reconstruction.
An issue related to the use of PROs in RTS test batteries is, which cut-off values best identify whether patients “pass” (patients can be cleared to RTS) or “fail” (patients should be advised against RTS)? Different cut-offs for different PROs and outcomes have been proposed.15,16 Notably, some cut-offs commonly used in the decision-making if the patient can RTS have not been anchored against a second ACL injury,15 but are instead based on responses from PROs that were collected several months prior to patients suffering a second ACL injury.16 It is important to acknowledge that the reasons for a second ACL injury are multifactorial and include genetic, physical, psychological, trauma, and surgical factors.17–20 Recovery of symmetrical quadriceps strength has been proposed as a possible protective factor that can reduce the risk of a second ACL injury in high-level athletes with RTS.21 However, the results are inconclusive.22,23
There is a need to better understand which cut-offs for PROs best predict a second ACL injury in patients treated with ACL reconstruction, especially regarding protective factors for a second ACL injury, such as the recovery of symmetrical quadriceps strength. This study aimed to identify cut-off values for three commonly used psychological PROs that could differentiate patients who suffer a second ACL injury from those who do not within two years of RTS after ACL reconstruction with respect to recovery of symmetrical quadriceps strength.
METHODS
Following the guidelines from the Enhancing the Quality and Transparency of Health Research (EQUATOR) network, the Standards for Reporting Diagnostic Accuracy (STARD) statement was used as a checklist to report this study.24
Data for the present study were prospectively collected from a rehabilitation outcome registry, the Project ACL. The registry was established in 2014, which aims to improve the care of patients with ACL injuries. Data in the Project ACL consists of results from muscle function tests and PROs collected prospectively before the surgery (in case of ACL reconstruction), at 10 weeks, 4, 8, 12, 18, and 24 months, and every five years with ACL injury/reconstruction as a baseline.25 Prior to participation in the Project ACL, written consent was collected. Ethical approval was obtained from the Swedish Ethical Review Authority (registration number:2020-02501).
Inclusion and exclusion criteria
All consecutive patients registered in the Project ACL with one ACL injury treated with reconstruction with or without sustaining a subsequent second ACL injury within two years from RTS [Tegner Activity Scale (Tegner)26 ≥6] were eligible for inclusion. The patients included in this study were followed up for two years after RTS. Patients were excluded if one of the following criteria were fulfilled: age <16 or >50 years; not participating in knee-strenuous sports prior to index ACL injury, i.e. reporting a Tegner Activity Scale26 of <6; sustaining a second ACL injury before surgical reconstruction; or not participating in any of the follow-ups of project ACLs. Second ACL injuries are reported by patients themselves, responsible physical therapists, or the test leader at follow-up in Project ACL.
Patient Reported Outcomes
The PROs used in this study were the ACL-Return to Surgery after Injury scale (ACL-RSI),27 Knee Injury and Osteoarthritis Outcome Score subscale Quality of Life (KOOS QoL),28 and Function in sport and recreation (KOOS Sports), the 18-item version of the Knee Self-Efficacy Scale (K-SES18),29 and Tegner Activity Scale.26
The ACL-RSI aims to measure patients’ emotions, confidence, and risk appraisal of RTS after an ACL injury. Herein, the validated 12-item short version was used.27 Each item was graded from 0 to 10, where 10 is the highest response, representing the best possible psychological response to RTS (highest confidence and emotion, and lowest risk appraisal). The final score is calculated by summing the total score of all items (highest score 120), and then normalizing the score to a 0-100 scale as proposed in the original paper.27
The K-SES18 aims to evaluate knee-related self-efficacy,29 that is, the belief in one’s ability to perform a physical task, such as running or jumping. The K-SES18 comprises 18 items divided into two subscales: present (14 items) and future (four items) knee self-efficacies. Each item was graded from 0 to 10, with 10 being the most positive response, representing the greatest belief in carrying out a given physical task. The results from each item were added and divided by the number of items to generate a mean value for the subscales.
The KOOS QoL and KOOS Sports were used in the assessment of patients with ACL injury.28 They comprise four (QoL) and five (Sports) items, respectively. The KOOS QoL assess how often patients are reminded of their knee problems, whether patients can trust their knee function, have to make life changes due to knee function, and whether patients experience problems related to the knee. The KOOS Sports comprises five items assessing a patient’s perceived difficulty in performing different tasks, such as running and jumping during the last week. Each item is rated from 0 to 4 on a 5-point Likert scale, and a normalized score from 0 to 100 is calculated for each subscale, where 0 indicates the most severe symptoms, and 100 indicates no symptoms.
The Tegner Activity Scale (Tegner)26 aims to measure strenuous knee activity. Patients grade their activity based on work and sports activities on a scale from 1 to 10, where one represents disability because of knee problems and 10 represents, for example, national or international level soccer. From level 6 on the Tegner, only sports activities are registered, and therefore when patients rate Tegner ≥6, it is assumable patients are active in a knee strenuous sport.
Strength testing
Strength tests for unilateral concentric knee extension and flexion were performed according to a standardized protocol,30 with an isokinetic dynamometer [Biodex System 4 (Biodex Medical Systems, Shirley, New York, USA)]. The testing procedure started with a standardized warm-up of 10 minutes on a stationary bike and sub-maximum trials on each test. The injured leg was tested first, followed by the uninjured leg. Isokinetic testing was performed at an angular velocity of 90°/s with the patients in a seated position. Three maximum repetitions with approximately 40 s of rest between each repetition were performed, and the greatest peak torque was recorded in the Project ACL database. For this study, the results from the unilateral knee extension test were extracted for analysis.
Study execution
In this study, demographic data, results from PROs, and strength tests for the quadriceps were extracted for analysis from the Project ACL in November 2021.
Time frame for returning to knee-strenuous sports typically varies between 6-13 months31 after primary ACL reconstruction. Results of the PROs and strength tests from the follow-up closest in time to RTS (Tegner Activity Scale, level ≥6) were selected for the included patients as the “index test.”
Primary outcomes of this study were the cut-off values with sensitivity and specificity for predicting a second ACL injury for each of the included PROs. Therefore, patients who did not go on to suffer a second ACL injury were treated as a reference, and patients who suffered a second ACL injury were treated as having the outcome of interest. Clinical reference standard was not applicable, as the analysis aimed to predict the outcome of interest. In order to account for the recovery of symmetrical quadriceps strength as a protective factor,21,32 sensitivity analyses were performed with regard to patients who had and had not recovered ≥90% of their quadriceps strength in the injured limb compared to the uninjured limb.33 Symmetrical quadriceps strength was presented with the limb symmetry index (LSI), where result from the injured limb was divided with result from the uninjured limb and multiplied by 100.
Statistics
Receiver operating characteristics (ROC) were calculated for each PRO at available follow-ups within two years of RTS (Tegner ≥6). The ROC is a graphical method of displaying the discriminatory accuracy of a marker (in this case, responses to PROs) for separating two populations, or distinguishing between patients affected by an outcome, that is, suffering a second ACL injury within two years after RTS (return to Tegner ≥6), and individuals not affected by the outcome of interest, that is, patients who did not suffer a second ACL injury within two years after RTS. A patient was assessed as “positive” if the tested PRO value was greater than a given threshold value; otherwise, the patient was assessed as negative.34 The accuracy of any given threshold value was measured by the probability of a true positive (sensitivity) and true negative (specificity), and presented with two summary measures for ROC: the area under the curve (AUC) and Youden index. The area under the curve (AUC) is a measure of the ability of a classifier to distinguish between outcomes and is used as a summary of the ROC curve. The higher the AUC, the better the performance of the model in distinguishing between positive and negative outcomes. When AUC is comprised between 0.5 and 1, there is a chance that the classifier can be able to distinguish the positive class values from the negative class values, depending on the AUC value, according to the following rule of thumb: 0.5 = no discrimination, with the same value of a coin flip; 0.5-0.7 = poor discrimination, not much better than a coin flip; 0.7-0.8 = acceptable discrimination; 0.8-0.9 = excellent discrimination; and >0.9 = outstanding discrimination.35 Therefore, when interpreting the AUC; values between 0.7 to 0.8 were considered to reflect acceptable accuracy.36 The Youden Index is a frequently used summary measure of the ROC and it measures the effectiveness of a diagnostic marker and enables the selection of an optimal threshold value (cut-off point) for the marker.34 The Youden index value ranges from 0 to 1,37 where 0 means that a diagnostic test gives the same proportion of positive results for groups with and without the disease, i.e. the test is useless, while a value of 1 indicates that there are no false positives or false negatives, i.e. the test is perfect.
Statistical analyses were performed using Statistical Analysis System (SAS) software version 9 (SAS Institute Inc., Cary, North Carolina, USA). Mean values with standard deviations (SD) or medians with minimum and maximum are presented for the demographic data. Significance level was set at 0.05.
RESULTS
In total, 641 patients (355 men, 61%) were included in the present study. Figure 1 shows the inclusion and exclusion process. Table 1 presents the demographic characteristics of the patients.
Figure 2 presents the frequency of a second ACL injury stratified by months from return to knee-demanding activity (Tegner ≥6).
Table 2 presents the Youden index with specificity and sensitivity, as well as the model AUC for all assessed PROs.
The cut-offs that best differentiated patients who suffered a second ACL injury after ACL reconstruction from patients who had not were: ≥71.7 for the ACL-RSI, <56 for the KOOS QoL, ≥96.0 for the KOOS Sports, ≥9.4 for the K-SES18 present and ≥7.0 for the K-SES18 future. The AUC and Youden index for each cut-off were below acceptable values, indicating the inability of the cut-offs to properly differentiate between patients who suffered a second ACL injury after ACL reconstruction and those who did not. Table 3 presents the number of patients who suffered a second ACL injury in relation to the achievement of the cut-off values for the different PROs.
Effect of recovery of symmetrical quadriceps strength
Table 4 presents the cut-off analysis stratified according to patients who had and had not achieved an LSI of ≥90% in the quadriceps strength test at the time of RTS.
Interpretation of results
Stratifying patients based on achieved symmetrical quadriceps strength (≥90% LSI) or not at the time of RTS did not lead to cut-off values with better psychometric properties, i.e., the Youden index or AUC compared with the cut-offs determined for the entire cohort. In the current results, the best Youden index was 0.24, and the best AUC was 0.613, which is well below acceptable levels.36,37 Therefore, the calculated cut-offs appear unable to separate the two outcomes, with or without a second ACL injury.
Table 5 presents the number of patients who suffered a second ACL injury in relation to the achievement of the cut-offs for the different PROs in patients who achieved symmetrical quadriceps strength at the time of RTS.
DISCUSSION
The main finding from this registry study was that the determined cut-off values for different PROs were not able to differentiate patients who suffered a second ACL injury up to 24 months after RTS following ACL reconstruction from patients who did not. Additionally, achieving symmetrical quadriceps strength, i.e., ≥90% LSI did not improve the cut-offs’ psychometric properties. Regardless of recovering symmetrical quadriceps strength, patients with either a higher or lower knee perceived quality of life, knee-related self-efficacy, confidence, emotions, and risk appraisal appear as likely to suffer a second ACL injury. Therefore, it can be assumed that passing the calculated cut-off for PROs used in the present study is not sufficient for safe RTS. Caution must thus be taken if patients are cleared to RTS based on the cut-off values used in the present study, regardless, if they achieve ≥90% LSI in quadriceps strength or not.
The used cut-off values for PROs in this study could not generate an acceptable AUC value (0.7-0.8),36 indicating that the cut-off values are no better than ‘flipping a coin’ when used to assess the risk of suffering a second ACL injury. The poor discriminative ability of the cut-offs can be partly explained by the specificity and sensitivity values. The sensitivity ranged between 0.19-0.59 for all the cut-offs except for the K-SES18 future (0.78). Using the ACL-RSI as an example, a sensitivity of 0.50 means that half of the patients who suffered a second ACL injury had a score above 71.7. Further, when a patient scores above 71.7 on the ACL-RSI, there is approximately a 50% chance to correctly guess if the patient will go on to suffer a second ACL injury (AUC = 0.553); therefore, the psychometric value for the cut-off is very low (Youden index = 0.13). Since the cut-offs that best differentiated patients who suffered a second ACL injury from patients who did not have low AUC values,36 there is a possibility that the PROs used in this study are not suitable for identifying patients who are at increased risk of a second ACL injury at the time of RTS.
One issue related to the inability of PROs to identify patients who will continue to suffer a second ACL injury can be related to the development of the PROs used in this study. The ACL-RSI was developed to assess confidence, emotion, and risk appraisal of RTS after ACL injury,27 which are three different psychological domains (constructs). However, the ACL-RSI is summarized into a single score, ranging between to 0-100; this may be questionable because the scale comprises three domains,38 which can lead to limitations as it induces indirectness towards the outcome when interpreting the PRO as a single score.
When assessing the KOOS, using a stringent psychometric method, i.e. the Rasch analysis, criteria for one-dimensionality are respected only in two out of the five subscales, i.e. KOOS Sports and KOOS Quality of life.39 Furthermore, the KOOS´s development quality has been rated as “inadequate,” and the KOOS´s psychometric properties with regard to patients with an ACL injury have been reported to be “poor,” “insufficient,” and “inconsistent”, but the authors opinions are based on that studies evaluating the psychometric properties are lacking.38,40 The inconsistency reported in the psychometric properties of the KOOS with regard to patients with an ACL injury can be due to issues during scale development, since no patients who suffered an ACL injury participated in the development and were asked whether the items were relevant.
As for the K-SES18, a recent publication reported that it has acceptable reliability and validity for measuring self-efficacy in patients with ACL injury.41 However, no Rasch analysis has yet been performed on the K-SES18, and good psychometric values can be obtained by correlating the K-SES18 with other scales of debatable psychometric properties, such as the KOOS, leading to possible misinterpretation of results. Finally, the inability of PROs to identify patients who would suffer a second ACL injury is likely related to the fact that patients with a second ACL injury were not taken into account when developing PROs.
Limitations
One limitation of this study was the relatively small number of patients in the second ACL injury group (n=64), which could have influenced the statistical power. A post-hoc power calculation was performed, which showed that 33 patients in each group were needed to detect an AUC of 0.7 with a 90% power and an alpha of 0.05. Due to the heterogeneity in the population of patients who suffered an ACL injury, 33 patients per group might still not be sufficient for the statistical power; therefore, the results should be appreciated with caution. Another limitation of the present study could be its conceptual character. A second ACL injury is multifactorial, and both high and low values on PROs can be risk factors for a second ACL injury.20 Assuming that both too high and too low knee related self-efficacy could induce a greater risk for a second ACL injury, using only one cut-off value (reflecting a greater risk for a second ACL injury) for one single scale could be conceptually wrong. If a PRO is sensitive in predicting a second ACL injury, future studies might need to use two different cut-off values: taking K-SES as an example, one cut-off reflecting an extremely high knee-related self-efficacy, and one cut-off reflecting an extremely low knee-related self-efficacy. Notably, some patients returned to the pre-injury Tegner level as early as four months after the ACL reconstruction. It is unclear how this might have affected the results. A further possible limitation of this study concerns age as the ACL-RSI has been reported to be more sensitive for change in younger patients (≤ 20 years) who go on to suffer a second ACL injury.16 Patients in this cohort were on average 24.8 years, and it is not known whether the calculated cut-offs are influenced by age, and if age-specific cut-offs may be better to predict second ACL injuries.
Conclusion
The determined cut-off values for three commonly used psychological PROs could not differentiate patients who suffered a second ACL injury from those who did not within two years from RTS, regardless of whether the patients achieved symmetrical quadriceps strength or not, in a cohort of patients who had undergone with ACL reconstruction. Therefore, clinicians are recommended to not only rely on cut-off values or a single PRO of those analyzed in this study when making decisions on which patients are at risk of experiencing a second ACL injury when returning to sports after ACL reconstruction.
Acknowledgments
We would like to thank Editage (www.editage.com) for English language editing, and Bengt Bengtsson at Statistiska Konsultgruppen for the help with statistical analysis.
Conflict of interest
Authors declare no conflict of interest.