| title | subtitle | author | date | output | ||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Association between leadership commitment and telework satisfaction at NASA (2020): weighted sex-adjusted stratified analysis |
DOCUMENT: SAR-2022-007-GJ-v02 |
**From:** Felipe Figueiredo **To:** techsavvy32 (fiverr.com) |
2022-02-05 |
|
\newpage
Association between leadership commitment and telework satisfaction at NASA (2020): weighted sex-adjusted stratified analysis
Document version
| Version | Alterations |
|---|---|
| 01 | Initial version |
| 02 | Unweighted analysis included as an alternative result; common dataset between this and the associated analysis |
- CI: confidence interval
- FEVS: Federal Employee Viewpoint Survey
- OPM: U.S. Office of Personnel Management
- OR: odds ratio
The Federal Employee Viewpoint Survey (FEVS) addressed leadership commitment, professional development, and telework satisfaction while accounting for gender (OPM, 2020). This analysis addresses a subset of the FEVS survey reflecting NASA employees.
Quantify the association between leadership commitment and telework employee satisfaction at NASA from the 2020 Federal Employee Viewpoint Survey.
Raw data was collected as a census of the eligible population (OPM, 2020), and statistical weighting was applied at the data collection to adjust for non-responses in in the census attempt.
These survey weights (variable postwt) allow for the estimation of the association under study in the source population.
The raw data was filtered to reflect only NASA survey respondents (where agency code equals NN).
There were a total of 10588 observations in the raw data, before cleaning procedures.
The raw data is expected to reflect a total employee population at NASA at 16809 employees but after cleaning procedures the observations in the analytical data represents a total of 14908 NASA employees. Survey questions measured responses in a 5-point Likert scale between 1 (strongly disagree) and 5 (strongly agree). Some questions offered the option to choose "X" (Don't know) as the answer. These unknown answers were considered non-answers and treated as missing values.
All variables in the analytical set were labeled according to the raw data provided and values were labeled according to the data dictionary for the preparation of production-quality results tables and figures.
This analysis will focus on two questions from the FEVS survey, where the main interest is employee satisfaction
(q29 -- How satisfied are you with the Telework program in your agency?)
as the dependent variable and leadership commitment
(q58 -- Managers promote communication among different work units (for example, about projects, goals, needed resources))
as the independent variable.
As per the data cleaning process, the dependent variable was renamed to dv and the independent variable was renamed to iv in the analytical dataset.
Additionally, to calculate the OR the responses were categorized as binary responses, where agreement was aggregated from the "agree" and "strongly agree" responses, in variables dv2 and iv2.
After the cleaning process 7 variables were included in the analysis with 9405 observations.
Primary outcome
Odds of participants that are satisfied with telework at NASA from the 2020 Federal Employee Viewpoint Survey.
The association will be stratified by the sex of survey respondents.
The epidemiological profile of the study participants will be described. Demographic (sex, age and BMI) will be described as mean (SD)
or as counts and proportions (%), as appropriate.
All comparisons between groups will be performed as univariate analyses.
Differences in distribution of categorical variables will be assessed with the
chi-square test with the adjustment of design effect for weighted survey data.
The OR will be used as a measure of effect of the independent variable on the dependent variable.
The stratification by sex will be used to assess if the effect changes across male and female strata.
As a rule of thumb, a minimum change of 20% in the OR will be considered before concluding that there is an interaction between sex and the independent variable.
The homogeneity of the OR across strata will be assessed with the Cochran-Mantel-Haenszel test.
All evaluations will be performed as complete case analyses.
All analyses will be performed using the significance level of 5%.
This analysis was performed using statistical software R version 4.1.2.
The sample evaluated in this study is comprised of 9405 observations that, after the survey weights adjustment, are representative of 14908 NASA employees, out of a total of 16809. Two thirds of the study population are males (66.4%, Table 1).
Both survey questions addressed in this study showed most NASA employees demonstrated high levels of satisfaction when the survey was conducted. The proportion of employees that agree or strongly agree with the main outcome of this study (q29 -- How satisfied are you with the Telework program in your agency?) was 94.6%. The proportion of employees that agree or strongly agree with leadership commitment (q58 -- Managers promote communication among different work units (for example, about projects, goals, needed resources)) was 80.7%.
Table: Table 1 Characteristics of the study population. Each of the survey questions had 5 alternatives for answer: 1: strongly disagree, 2: disagree, 3: neither agree nor disagree, 4: agree, 5: strongly agree.
| Characteristic | N = 14,908 |
|---|---|
| Sex | |
| Male | 9,902 (66) |
| Female | 5,006 (34) |
| How satisfied are you with the Telework program in your agency? | |
| 1 | 63 (0.4) |
| 2 | 196 (1.3) |
| 3 | 543 (3.6) |
| 4 | 4,159 (28) |
| 5 | 9,947 (67) |
| Managers promote communication among different work units. | |
| 1 | 301 (2.0) |
| 2 | 780 (5.2) |
| 3 | 1,789 (12) |
| 4 | 6,654 (45) |
| 5 | 5,384 (36) |
In order to compare how the responses to the leadership commitment and telework satisfaction
relate to each other, a cross-tabulation of all answers from each study participant is shown in Table 2. There is a detectable statistical difference in the distributions of the two responses (see section Observations), where the chi-square adjusted for survey design was significant.
This cross tabulation shows that the largest proportion of survey respondents answered 4 (agree) or 5 (strongly agree) to both questions. This means that there is a detectable association between leadership commitment and telework satisfaction.
Considering this is a high-powered study, this could be due to statistical power alone, meaning that while a difference in proportions is detectable this analysis does not inform how large this difference is.
The effect size of this association will be reported in the next section.
Table: Table 2 Cross tabulation of raw survey responses.
| Characteristic | Overall, N = 14,908 | 1, N = 63 | 2, N = 196 | 3, N = 543 | 4, N = 4,159 | 5, N = 9,947 | p-value |
|---|---|---|---|---|---|---|---|
| Managers promote communication among different work units. | <0.001 | ||||||
| 1 | 301 (2.0) | 13 (21) | 17 (8.5) | 33 (6.1) | 131 (3.2) | 106 (1.1) | |
| 2 | 780 (5.2) | 11 (18) | 28 (14) | 61 (11) | 349 (8.4) | 330 (3.3) | |
| 3 | 1,789 (12) | 12 (19) | 29 (15) | 138 (25) | 693 (17) | 917 (9.2) | |
| 4 | 6,654 (45) | 23 (36) | 92 (47) | 217 (40) | 2,218 (53) | 4,105 (41) | |
| 5 | 5,384 (36) | 3 (5.5) | 31 (16) | 93 (17) | 768 (18) | 4,488 (45) |
In order to estimate the size of the effect of the association a binary categorization was performed between the two survey responses, where agreement aggregates all answers 4 (agree) and 5 (strongly agree). Table 3 shows the contingency table that cross-tabulates these variables. There is a detectable statistical difference in the distributions of the two responses (see section Observations), where the chi-square adjusted for survey design was significant.
The overall (unadjusted) effect of the association was significant (OR: 3.42, 95% CI: [2.95, 3.95], p<0.001). This means that when survey respondents perceive leadership commitment, they are 3 times as likely to report telework satisfaction
as participants that do not perceive such commitment from leadership. Alternatively this translates to a 2-fold increase in the group of interest when compared to the reference group.
Table: Table 3 Cross tabulation of dichotomized responses, overall and by sex of survey respondents.
| Characteristic | Disagreement, N = 803 | Agreement, N = 14,106 | p-value | Disagreement, N = 586 | Agreement, N = 9,316 | p-value | Disagreement, N = 216 | Agreement, N = 4,790 | p-value |
|---|---|---|---|---|---|---|---|---|---|
| Managers promote communication among different work units. | <0.001 | <0.001 | <0.001 | ||||||
| Disagreement | 343 (43) | 2,527 (18) | 249 (43) | 1,674 (18) | 94 (43) | 853 (18) | |||
| Agreement | 460 (57) | 11,579 (82) | 337 (57) | 7,642 (82) | 123 (57) | 3,937 (82) |
In order to control for a possible confounder between the association and the sex of the study participant, the analysis was stratified by sex. The Mantel-Haenszel adjustment for the OR was similar to the overall unadjusted effect estimate (adjusted OR: 3.42, 95% CI: [2.95, 3.96], p<0.001). For comparison, the relative difference between the overall effect of association and the adjusted effect is on the order of 0.1%, so we may rule out confounding between sex the association in this study. There is no need to report the adjusted OR in this context.
In order to assess whether or not there is an interaction between sex and the association under study we can compare the difference between the effects observed in each sex strata. The effect sizes for males (OR: 3.37, 95% CI: [2.84, 4.00], p<0.001) were similar to the effect of females (OR: 3.53, 95% CI: [2.66, 4.66], p<0.001). For comparison, the relative difference between the effect of association on males and females is on the order of 4.4%, so we may rule out interaction between sex and the association in this study. There is no need report stratum-specific effects of association in this context.
In summary, since there is no evidence of either confounding or interaction with sex. This means that, when they perceive high levels of leadership commitment, both men and women appear to show similar levels of telework satisfaction
when compared to the reference group. In this context it is appropriate to simply report the unadjusted estimate of effect as the result of the analysis (OR: 3.42, 95% CI: [2.95, 3.95], p<0.001).
After accounting for the complex design of the survey that acquired the raw data used in this report, the analysis effectively simulates a census of the NASA employees perception of the workplace culture. This large sample provides high levels of statistical power to detect even small differences in proportions. This is the reason that most p-values can be reasonably expected to be significant in this sample.
On the other hand when considering the measure of effect of the associations evaluated here, this large power becomes an asset since it provides a high precision in the estimation of effects, meaning all confidence intervals are narrow and we have more confidence in the estimates provided. Since these are the main results to report, in this analysis the high power is welcome.
Although an alternative unweighted version of the analysis is provided as an option in the Appendix (8.2.2), it is not recommended that survey weights be ignored. Survey designs can account for multiple sampling of certain individuals (which can occur in randomized sampling designs) and in some cases may also need to over-sample minority populations in order to increase precision in the estimates obtained from these groups. Survey weights adjust for these design choices and a decision to discard these weights will not incorporate the complexity of the source survey design and will likely introduce bias in the analysis results. In this analysis the FEVS dataset includes multiple observations of the same individuals -- ignoring sample weights will count all duplicates as multiples, effectively giving them an extra weight and biasing the estimates. This bias can be hard to quantify and so it is recommended that the main analysis (4.2.2) is used as the basis for the conclusions. More details of the sampling design of the FEVS survey can be found at the technical report (OPM, 2020).
There is a positive association between leadership commitment and telework satisfaction
where employees that perceive high levels of commitment they derive 3 times as much satisfaction at the workplace in a telework environment
when compared with employees that do not perceive high leadership commitment.
These
proportions do not appear to vary by sex, and males and females appear to perceive leadership commitment and have satisfaction at different levels.
- SAP-2022-007-GJ-v02 -- Analytical Plan for Association between leadership commitment and telework satisfaction at NASA (2020): weighted sex-adjusted stratified analysis
- OPM (2020). 2020 Federal Employee Viewpoint Survey -- Technical report (https://www.opm.gov/fevs/reports/technical-reports/).
N/A
In order to assess the whether or not males and females answered the questions at different proportions of agreement (while considering the full 5-point scale answers) chi-square tests were conducted, with the design-adjustment for survey weights.
The proportion of participants that perceive high levels of leadership commitment
varies by sex (p=0.035). Additionally, the proportion of participants that perceive telework satisfaction
varies by sex (p<0.001).
Sex is associated with both leadership commitment and employee satisfaction. This means that when considering each survey question on its own, men and women tend to give different
answers to the 5-point scale questions.
In this section the analysis in section 4.2.2 is replicated with the raw data sampled from the survey, ignoring the design-adjusted survey weights.
In order to estimate the size of the effect of the association a binary categorization was performed between the two survey responses, where agreement aggregates all answers 4 (agree) and 5 (strongly agree). Table A1 shows the contingency table that cross-tabulates these variables. There is a detectable statistical difference in the distributions of the two responses (see section Observations), where the chi-square adjusted for survey design was significant.
The overall (unadjusted) effect of the association was significant (OR: 3.47, 95% CI: [2.88, 4.18], p<0.001). This means that when survey respondents perceive leadership commitment, they are 3 times as likely to report telework satisfaction
as participants that do not perceive such commitment from leadership. Alternatively this translates to a 2-fold increase in the group of interest when compared to the reference group.
Table: Table A1 Cross tabulation of dichotomized responses, overall and by sex of survey respondents, disregarding the survey weights.
| Characteristic | Disagreement | Agreement | Total | p-value | Disagreement | Agreement | Total | p-value | Disagreement | Agreement | Total | p-value |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Managers promote communication among different work units., n | <0.001 | <0.001 | <0.001 | |||||||||
| Disagreement | 209 | 1,578 | 1,787 | 148 | 1,007 | 1,155 | 61 | 571 | 632 | |||
| Agreement | 280 | 7,338 | 7,618 | 199 | 4,699 | 4,898 | 81 | 2,639 | 2,720 | |||
| Total, n | 489 | 8,916 | 9,405 | 347 | 5,706 | 6,053 | 142 | 3,210 | 3,352 |
In order to control for a possible confounder between the association and the sex of the study participant, the analysis was stratified by sex. The Mantel-Haenszel adjustment for the OR was similar to the overall unadjusted effect estimate (adjusted OR: 3.47, 95% CI: NA, p<0.001). For comparison, the relative difference between the overall effect of association and the adjusted effect is on the order of 0.1%, so we may rule out confounding between sex the association in this study. There is no need to report the adjusted OR in this context.
In order to assess whether or not there is an interaction between sex and the association under study we can compare the difference between the effects observed in each sex strata. The effect sizes for males (OR: 3.47, 95% CI: [2.77, 4.34], p<0.001) were similar to the effect of females (OR: 3.48, 95% CI: [2.46, 4.91], p<0.001). For comparison, the relative difference between the effect of association on males and females is on the order of 0.3%, so we may rule out interaction between sex and the association in this study. There is no need report stratum-specific effects of association in this context.
In summary, since there is no evidence of either confounding or interaction with sex This means that, when they perceive high levels of leadership commitment, both men and women appear to show similar levels of telework satisfaction
when compared to the reference group. In this context it is appropriate to simply report the unadjusted estimate of effect as the result of the analysis (OR: 3.47, 95% CI: [2.88, 4.18], p<0.001).
Both this document and the corresponding analytical plan (SAP-2022-007-GJ-v02) can be downloaded in the following address:
https://philsf-biostat.github.io/SAR-2022-007-GJ/
This analysis is part of a larger project and is supported by another analysis, linked below. The other analysis employs the same methods here to answer a related research question, using different data from the FEVS survey.
Association between leadership commitment and professional development at NASA (2020): weighted sex-adjusted stratified analysis
https://philsf-biostat.github.io/SAR-2022-008-GJ/
Due to confidentiality the data-set used in this analysis cannot be shared online in the public version of this report.
| id | dsex | dv | iv | dv2 | iv2 | postwt |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 | ||||||
| ... | ||||||
| 9405 |
Table: Table A1 Analytical dataset structure