INTRODUCTION

Healthcare providers routinely assess postural stability as part of the diagnosis and management of sport-related concussion.^1,2 Numerous research tools and protocols have been developed to quantify alterations in postural stability and parse the nature of disruptions in the postural control system;³ however, it remains unclear whether objective measurement systems provide a more comprehensive clinical picture than rater-assessed tests. A critical first step in determining this is to understand the relationship between different measurement systems in a large, healthy sample. The current study assesses three measures of postural stability that were collected on the same day in a large sample of uninjured NCAA Division I collegiate athletes.

The Balance Error Scoring System (BESS) is one of the most used concussion diagnosis assessments due to its ease of administration, portability, and low cost. The BESS uses a series of closed-eye stances on firm and foam testing surfaces to characterize postural stability under conditions of varying difficulty.⁴ Despite its popularity in athletics for concussion diagnosis and return-to-play judgements, the BESS has notable limitations. The BESS can detect changes in gross postural stability post-injury^5–7 but may have limited utility beyond three days post-injury^8,9 due to its reliance on subjective ratings and suboptimal interrater reliability.^8,10,11 Large changes in BESS scores may be necessary to overcome rater variability and provide meaningful detection of postural stability alterations.¹⁰ Additional concerns are its limited sensitivity in detecting postural stability deficits post-injury when compared to an individualized pre-injury baseline measurement^12–14 or to matched uninjured controls^12,15–17 along with substantial floor and ceiling effects in uninjured samples.^18–20

In research settings, force plates are preferred for assessing postural stability due to their precision, robust quantification, and ability to assess a variety of stances and balance protocols. Force plate technologies include, but are not limited to, the BioSway (Biodex Medical Systems, Inc., Shirley, NY), the NeuroCom Balance Master (Cephalon, Norresundby, Denmark), and Bertec (Bertec, Columbus, OH), and have been used to evaluate changes in postural stability following concussion.^21–24 Despite providing greater precision and a variety of derived stability measures, force plates are more expensive and less portable than the BESS and therefore may have limited accessibility and feasibility for clinical or sideline assessments. Moreover, a recent study of a large NCAA Division I athlete sample revealed sizable heterogeneity of individual balance performance measured using a force plate system.²⁵ This raises questions of whether the precision of such postural stability measures is necessary for monitoring an athlete’s diagnosis of, and recovery from concussion.

The past decade has seen the development of numerous mobile-based instrumented measurements of postural stability.²⁶ Mobile applications (i.e., apps), such as C3Logix (NeuroLogix Technologies, Inc., Cleveland, OH), use inertial sensors such as accelerometers and gyroscopes to capture accelerations of the total body center of mass across multiple planes of motion.²⁷ These apps, when integrated with existing balance protocols, such as the BESS, hold promise for the diagnosis and management of athlete concussion because they combine the ease of administration of the traditional BESS with the option to augment rater-assessed errors with objective measures using a portable device. Although these apps are touted as being more affordable and portable than force plates, findings regarding reliability and validity are mixed.^{4,19,26,28–31} Inertial sensors have been shown to augment subjective balance measures, such as error counts using the BESS.¹⁹ Instrumenting the BESS, or its modified version, with an inertial sensor improved psychometric value³² and sensitivity to detect differences between concussed and non-concussed individuals^14,15,17,33 compared to error counts alone. However, no large studies of balance performance using mobile inertial sensor measurement systems in collegiate athletes exists.

The purpose of this study was to examine the relationships between metrics of balance performance using different measurement systems in uninjured, healthy collegiate athletes. It was hypothesized that there would be significant positive, within-person correlations between ellipsoid volumes, as measured with inertial sensors, and error counts on the BESS when these measurements were collected concurrently. It was further hypothesized significant positive, within-person correlations between ellipsoid volumes, as measured by inertial sensors, and Sway Index scores, as measured using a force plate system when the measurements were collected on the same day, but not concurrently.

METHODS

Participants

Collegiate athletes 17-23 years of age were recruited from a northeastern US university during standard pre-participation physicals that took place during entry into the athletic program (2013 – 2016). All uninjured members of the university’s 22 NCAA Division I athletic teams who received medical clearance for athletic participation were invited to participate. Participation was voluntary and written informed consent was obtained from 703 athletes. Of those, balance data were obtained from 534 (19.4 +/- 1.2 years at time of consent; 47% female). This study was approved by the university’s institutional review board.

Procedures and Variables

All postural stability data were collected in the Department of Sports Medicine as part of standard physical exam and health screening assessments. Participants completed one or two balance assessments depending on the year of recruitment and time constraints of the athletes and clinical staff. Of the full sample that was tested using the C3Logix app (n=534), 177 participants from 21 teams were also tested using the Biodex Biosway force plate system on the same day (18.9 +/- 1.0 years at time of consent; 45% female); Biosway data collected on a different day was not used for the present analyses. In all instances where two balance assessments were performed on the same day, BioSway testing was performed before C3Logix. Prior to beginning balance testing, participants were asked to identify their dominant foot (e.g., “push off or power foot”, “the foot you jump off to dunk”, or “3-pt stance foot”). Balance assessments were performed in socks, took less than five minutes per athlete and were administered by medical staff, certified athletic trainers, or trained graduate-level research assistants. All participants received instructions from the test administrator via a common script.

Demographics. Age, sex, height, and weight were obtained from medical charts completed by the Department of Sports Medicine as part of standard pre-participation physical exam. Medical chart review was a separate component of the larger study that examined factors that predicted concussion risk and recovery rates and some charts were unavailable at the time of review. Data from these clinical records were available to the research staff for approximately 75% of participants (216 men, 192 women). Table 1 reports estimates of athletic team demographics based on available data.

C3Logix. C3Logix (NeuroLogix Technologies, Inc., Cleveland, OH) is an iPad-based comprehensive concussion management system that combines standard BESS error counting with inertial sensor (accelerometer and gyroscope) data.^19,34 Participants were assisted with placing a custom-built belt that securely held the iPad at approximately sacral height, with the screen of the device facing away from the back of the body according to C3Logix instructions. Participants were told to close their eyes and place their hands on their hips. The administrator initiated the test by pressing the button on the C3Logix app screen. Following a countdown, an auditory cue announced the start and end of each of six 20-s conditions that parallel those used by BESS: double leg, single leg, and tandem leg stances, first on a firm surface and then on a standard, six-inch thick Airex foam pad. During each condition, the C3Logix app records all postural movement and calculates ellipsoid volumes¹⁹ while the administrator manually counts the number of errors using standard BESS procedures (hands off of the hips; eyes opened; a step, stumble, or fall; hip flexed or abducted beyond 30°; forefoot or heel lifted off of the testing surface; or out of proper test position for > 5 s) and records it in the app.⁴ Larger ellipsoid volumes and more errors indicate a more unsteady posture.

BioSway. BioSway (Biodex Medical Systems, Inc., Shirley, NY) is a portable force plate system.^35,36 Participants were asked to stand on the device with their hands at their sides, look straight ahead, and remain as motionless as possible during measurement. As necessary, heels were repositioned to ensure feet were equidistant and comfortably placed. The administrator pressed the ‘Collect Data’ button to initiate the countdown and start the protocol. Participants completed four 20-s double leg stance conditions per the standard Modified Clinical Test of Sensory Interaction in Balance (mCTSIB) protocol:³⁷ eyes open or eyes closed, on a firm surface (BioSway platform) or a foam surface (standard, six-inch thick Airex foam pad placed on the BioSway platform). The device recorded average movement from center, which is termed Stability Index. The standard deviation of Stability Index was used to compute Sway Index.³⁵ Higher Sway Index is indicative of greater posture instability. The present data are derived from a previously published larger dataset²⁵ and include only those individuals who were tested with C3Logix and BioSway on the same day and only during the double leg, eyes closed, firm surface (double leg/firm surface) and double leg, eyes closed, foam surface (double leg/foam surface) conditions as they parallel those measured by C3Logix.

Statistical Analysis

Univariate analyses were performed using SAS 9.4 (SAS Institute, Cary, NC) to determine the balance characteristics of the whole sample and assess distributional assumptions for analyses. From C3Logix, data from each of the six conditions (double leg/firm surface, tandem leg/firm surface, single leg/firm surface, double leg/foam surface, tandem leg/foam surface, single leg/foam surface) were assessed separately. Raw ellipsoidal volumes of motion data were downloaded from the C3Logix app, a constant (+1) was added for ease of graphical interpretation, and data were log transformed to eliminate violations of homogeneity of variance. Ellipsoidal volume data from four individuals were excluded because they exceeded realistic expectations of a normal baseline test (i.e., >3 standard deviations from next highest score; final n = 530). Spearman’s rank-order correlational analyses were used to evaluate the relationship of concurrent error counts and ellipsoid volumes for each condition. From BioSway, Sway Index data from two conditions (double leg/firm surface and double leg/foam surface) were downloaded because they were the only conditions for which C3Logix collected parallel data. Only data from individuals who completed the BioSway and C3Logix protocols on the same day were included in the present analyses (n = 177). Sway Index scores were log transformed and their relationship to C3Logix ellipsoid volumes during parallel conditions was tested using Spearman’s rank-order correlations. These data were then normalized using z-score transformations for graphical interpretation.

RESULTS

Table 2 presents normative data from the C3Logix app, including average number of errors, percent of participants showing no errors, and raw ellipsoid volumes for each condition. Errors were uncommon in the double leg/firm surface condition (1%). Errors were more common in all other conditions, and 20% of the sample demonstrated maximal error counts (10+) in the single leg/foam surface condition. Substantial individual variability in ellipsoid volumes was noted across all conditions except the double leg/firm surface condition. The wide distribution of ellipsoid volumes calculated by C3Logix is also evident in Figure 1, which shows ellipsoid volumes by error counts for each of the six conditions.

Figure 1.C3Logix Balance Assessments.

Log-transformed ellipsoid volumes of motion from a sample of 530 uninjured athletes were plotted by the number of BESS errors recorded during each of the 20-s conditions. (A) Double Leg/Firm Surface condition. (B) Double Leg/Foam Surface condition. (C) Tandem Leg/Firm Surface condition. (D) Tandem Leg/Foam Surface condition. (E) Single Leg/Firm Surface condition. (F) Single Leg/Foam Surface condition. Circles represent individual data points.

C3Logix: Comparison of error counts vs. ellipsoid volumes of motion

Spearman correlation analyses of error counts and log-transformed ellipsoid volumes, collected concurrently with C3Logix, revealed significant correlations in the tandem leg/firm surface, r_S (526) = .68, p < 0.0001, and single leg/firm surface r_S (526) = .58, p < 0.0001, conditions, but not in the double leg/firm surface condition, r_S (526) = -.02, p= 0.7139. Errors and log ellipsoid volume were also significantly correlated in all three stances completed on the foam pad: double leg, r_S (526) = .22, p < 0.0001, tandem leg, r_S (526) = .62, p < 0.0001, and single leg, r_S (526) =.22, p < 0.0001. Visual inspection of Figure 1 confirms these relationships with trend lines, but also illustrates the magnitude of sample variance, especially on the harder stances. To further characterize the nature of the relationships, the two single leg and tandem leg correlational data were also graphed by sex (Figure 2) and by team, using five teams with large samples (women’s lacrosse n = 35, men’s lacrosse n = 47, women’s soccer n = 41, men’s soccer n = 33, football n = 80; Figure 3).

Figure 2.C3Logix balance assessment correlations separated by sex.

Log-transformed ellipsoid volumes of motion from men (open circles) and women (closed circles) were plotted by the number of BESS errors recorded. (A) Tandem Leg/Firm Surface condition. (B) Tandem Leg/Foam Surface condition. (C) Single Leg/Firm Surface condition. (D) Single Leg/Foam Surface condition. Circles represent individual data points.

Figure 3.C3Logix balance assessment correlations from five teams separated by sport.

Log-transformed ellipsoid volumes of motion from men’s and women’s soccer, men’s and women’s lacrosse, and football were plotted by the number of BESS errors recorded. (A) Tandem Leg/Firm Surface condition. (B) Tandem Leg/Foam Surface condition. (C) Single Leg/Firm Surface condition. (D) Single Leg/Foam Surface condition. Markers represent individual data points.

C3Logix vs BioSway: Comparison of ellipsoid volumes of motion to Sway Index scores

Data from the subset of the sample who completed the C3Logix and BioSway protocols on the same day were then analyzed to determine whether the two technologies provide similar rankings of postural stability. Spearman correlation analyses of log-transformed ellipsoid volumes (C3Logix) and Sway Index scores (BioSway) revealed significant correlation in the double leg/firm surface condition, r_S (175) = .27, p= 0.0003, and the double leg/foam surface condition, r_S (175) = .22, p= 0.0032. Although statistically significant, the relationship of the two same-day balance measurements showed substantial variability. Therefore, to visualize the nature of these correlations, Figure 4 presents data from the foam surface condition after normalization with z-score transformations. In Figure 4, each participant’s Sway Index (black circles), which were measured first, were ranked and plotted in ascending order on the x-axis. Ellipsoid volume z-scores (gray crosses) were plotted to align vertically with the Sway Index z-scores (i.e., ellipsoid volumes were plotted in order of Sway Index z-scores). Several outliers are evident among the Sway Index z-scores, but removal of these values did not substantively change the correlation results.

Figure 4.Association of same-day objective balance assessments from C3Logix and BioSway collected during the Double Leg/Foam Surface condition.

Individual BioSway Sway Index scores were z-score transformed and plotted in ascending order (from best balance [lowest z-score] to worst balance [highest z-score]) across the x-axis. C3Logix ellipsoid volumes were transformed into z-scores and plotted in order of the individual’s Sway Index score to align each participant’s two data points vertically. A negative z-score implies better than average balance. Several outliers on the BioSway protocol are evident.

DISCUSSION

The present study assessed postural stability in a large sample of uninjured collegiate athletes using different measurement systems. Statistically significant correlations were observed but many of the correlation coefficients were modest (r_S < .30) and varied by condition and across modalities. Graphical illustration of individual data offer insights into how and why this may be the case.

C3Logix: Comparison of error counts vs. ellipsoid volumes of motion

This is the first large dataset available in uninjured collegiate athletes from which the nature of an inertial sensor-based balance measurement and its relation to rater-assessed BESS error counts can be considered. These data may be valuable as a normative dataset for C3Logix balance assessments in collegiate athletes. However, Figure 1 reveals that the use of normative averages for ellipsoid volume as proxy measures of baseline performance may be problematic due to the large sample variance, which was evident across nearly all error count levels in the tandem and single leg conditions. This is paralleled by large sample variance in error counts which ranged from none to maximal for the three more difficult stances. Furthermore, in many conditions, the relationship between errors and average ellipsoid volumes appears non-linear across the range of error counts. For example, in the tandem leg/foam surface condition (Figure 1), average ellipsoid volumes do not substantively vary across four to nine errors. Thus, in a sample of age-restricted, young, fit, and uninjured adults, there is substantial baseline heterogeneity on both measures of postural stability. Although there were no a priori hypotheses about sex differences or sport differences, graphs were created to visually assess patterns by sex (Figure 2) or specific teams (Figure 3); none were evident. It may therefore be speculated that the non-linear relationship between measures may reflect subtle differences in balance strategies or adjustments used among individuals.¹⁹

Rank order of individual ellipsoid volumes and errors from C3Logix were significantly positively correlated, particularly during the moderately difficult, firm surface conditions; this is in line with Simon et al.³¹ These correlations were hypothesized because both measurement modalities exhibit construct validity for postural stability and because an acute postural adjustment, such as a step, stumble, or fall, would count as an error and simultaneously increase ellipsoid volume. However, even the strongest correlation, observed in the tandem leg/firm surface condition (r_S = .68), was lower than would be expected when assessing the same construct concurrently, especially considering the age and health of this sample. This further supports that individuals utilize different strategies to maintain postural stability¹⁹ and that objective measures may provide more insight into these patterns than subjective observations.

In the double leg conditions, which are considered easier, a “floor” effect in error counts and ellipsoid volumes is clear. This effect has been shown to negatively impact reliability of the BESS, particularly during the double leg conditions,²⁰ but does not have the same impact on C3Logix reliability.³¹ While floor effects are less than ideal from a statistical perspective due to their restriction of sample variance, they may be clinically useful. Based on the results of this study, a clinician should reasonably expect that during the double leg stances, an athlete or young adult would show no errors and have very small ellipsoid volumes if they are uninjured; any deviation from this “floor” may indicate clinically relevant alterations in postural stability. This aligns with a recent study that reported greater ellipsoid volumes in athletes after concussion compared to pre-injury, but only in the double leg/firm surface condition.¹⁴

The C3Logix app provides potential advantages for concussion diagnosis and recovery beyond clinician-rated BESS error counts, including precise quantification less prone to interrater variability and long-term data storage, without the need to add additional balance protocols to concussion assessment batteries. This may be especially notable if ellipsoid volume is sensitive enough to detect clinically relevant disturbances in postural stability using only the double leg, firm surface condition. Securing an iPad around the waist adds very little burden to the clinician and patient within a controlled environment; however, the utility of an iPad during sideline testing should be assessed. Special considerations including testing surface (i.e., turf or court), shoe type (i.e., cleats, barefoot, or sneakers), personal padding or equipment, and the clinician-patient burden of securing the iPad quickly and effectively during a high stress and time sensitive situation remain to be investigated.

C3Logix vs BioSway: Comparison of ellipsoid volumes and Sway Index

This study also assessed postural stability measured with two objective balance measurement systems (C3Logix and BioSway) at different time points on the same day. Although statistically significant, the magnitude of the correlations (r_s’s < .30) was surprisingly low. Previous studies have supported the validity of inertial sensors for postural stability assessments, but agreement may be dependent on the parameters measured and sensor location on the body.²⁷ The weak correlations in the current study appear to be partially due to differences in calculated parameters between the devices as opposed to sensor location (i.e., sacral placement of iPad vs. underfoot center of pressure from force plate). For example, the C3Logix app, placed at the sacrum, computes ellipsoid volume using the acceleration of the center of mass along anterior-posterior, medial-lateral, and trunk rotational planes of motion.¹⁹ Thus, ellipsoid volume considers the rate of change in position and speed over time. Contrary to the weak correlations between C3Logix and the BioSway force plate in the current study, the VSR Sport force plate sway velocity (NeuroCom, Clackamas, OR) demonstrated strong correlations with C3Logix ellipsoid volume.³⁰ Similar to the accelerations captured by ellipsoid volume, sway velocity measures changes in position over time. The BioSway, however, computes Sway Index by taking the standard deviation of the person’s average position relative to center,³⁵ but does not consider rate of change over time. It is reasonable to suggest that this may partly explain the strong correlations shown by Miyashita et al.³⁰ and the weak correlations shown in the current study. Further, the differences in results may be due to concurrent assessment in a small, single sport sample versus non-concurrent assessment analyzed in the current, large, multi-sport sample.

Limitations and Future Directions

Despite the strengths of this study, including the use of a large and multi-sport sample assessed on three balance modalities, there are several limitations. The current study was a component of a larger study designed to prospectively track concussion; therefore, the balance protocols were not designed to provide concurrent assessment of C3Logix and BioSway, nor were tester reliability indices assessed. Data are from Division I athletes from a single university and while including athletes from 22 sports, results may not be generalizable to all collegiate athletes. Graphs were created to allow visual inspection of whether correlations may have systematically varied by sex or sport, but none were observed. Corresponding statistical analyses of sex and sport differences were not performed due to a lack of a priori hypotheses and, in the case of sport, insufficient power. Future studies that are designed and powered to assess such correlational patterns are thus still needed. Subsequent psychometric testing should be performed to comprehensively characterize the relationship between balance measurement approaches. In addition, future research should weigh parameter sensitivity for detecting postural stability deficits against clinician-patient burden to optimize clinical protocols. Such research should further determine the measurement modality that is the most appropriate for real-time sideline concussion assessment.

CONCLUSION

Substantial inter-individual variability in postural stability was observed in this sample of young, uninjured athletes, regardless of the assessment method used. The lower-than-expected correlation coefficients between simultaneous and same-day assessments of balance raise questions about the reliability of normative or individual baseline balance measurements for use in “change from baseline” approaches to injury diagnosis. Postural stability is a complex construct and different measurement systems may be providing unique information about the strategies or postural adjustments that individuals utilize to maintain balance. Further research is needed in order to determine and recommend the most optimal postural stability measurement system and metric for use in concussion sideline assessments, diagnoses, and recovery.

Source of Funding

This project was funded in part by the NJ Commission on Brain Injury Research (NJCBIR13IRG028) and NIH/NIAAA (K02AA025123) to J.F.B., and NIH Institutional Research and Academic Career Development Award (IRACDA) K12GM093854 to J.S. Financial support was provided to C.E. through the School of Health Professions at Rutgers Biomedical and Health Sciences.

Conflict of Interest

None of the authors declare a conflict of interest.