Review Protocol for Programs Providing Employment Supports

Preparing to load PDF file. please wait...

0 of 0
100%
Review Protocol for Programs Providing Employment Supports

Transcript Of Review Protocol for Programs Providing Employment Supports

Clearinghouse for Labor Evaluation and Research

People with Disabilities Review Protocol Draft: July 22, 2014

REVIEW PROTOCOL FOR PROGRAMS PROVIDING EMPLOYMENT SUPPORTS FOR PEOPLE WITH DISABILITIES
Highlights
• The objectives of this systematic review were to determine the quality of existing causal evidence regarding the effectiveness of interventions meant to improve employment outcomes for people with disabilities; to describe lessons learned from the implementation of such programs; and to provide descriptive information about specific subpopulations of interest.
• The review focused on programs that seek to improve employment outcomes for youth and working-age adults with disabilities. The review also considered people with psychiatric disabilities, especially those with traumatic brain injury (TBI) or posttraumatic stress disorder (PTSD).
• The review examined research with causal, implementation and process, and descriptive analysis. Reviewers assessed the quality of causal evidence presented in impact studies and evaluated the implementation or process analyses associated with them. All other research was summarized but not deeply reviewed.
Introduction
The topic of this review protocol is employment interventions for people with disabilities. Compared to those without disabilities, people with disabilities are more likely to be unemployed, to receive lower wages when employed, to have lower levels of education and work experience, and to require income and other governmental support.1 Federal government expenditures to support people with disabilities in fiscal year 2008 reached $357 billion.2
The effects of programs on the labor market outcomes of targeted populations are of primary importance to this review. The review addressed this specific research question:
• Which programs are effective in improving the labor market outcomes of people with disabilities?
Because health status is an important factor in determining the extent to which a person with a disability can work and because employment can have a positive association with health, the review also addressed this secondary research question:
• Which programs are effective in helping people with disabilities improve their health status and therefore potentially improve their labor market outcomes?
People with disabilities often respond differently to employment interventions based in part on the type of disability. Interventions especially effective for some subpopulations may not be as
1 Houtenville, A. J., & Ruiz, T. (2012). Annual disability statistics compendium: 2012. Durham, NH: University of New Hampshire, Institute on Disability.
2 Livermore, G., Stapleton, D. C., & O’Toole, M. (2011.) Health care costs are a key driver of growth in federal and state assistance to working-age people with disabilities. Health Affairs, 30(9), 1664–1672.

1

Clearinghouse for Labor Evaluation and Research

People with Disabilities Review Protocol Draft: July 22, 2014

helpful to others. Consequently, in addition to including research that considers all people with disabilities, the review also looks at studies that focus on people with psychiatric disabilities, with a special emphasis on people diagnosed with TBI or PTSD.

In addition to determining the quality of causal evidence supporting the effectiveness of employment programs for people with disabilities, the Clearinghouse for Labor Evaluation and Research (CLEAR) conducted in-depth reviews of the implementation or process studies associated with these impact studies. CLEAR also reviewed the research on interim outcomes associated with demonstration projects when there was no associated impact study or when the impact study was forthcoming.

In the rest of this evidence review protocol, we set forth the criteria by which research was determined to be eligible for review, the specific causal evidence guidelines used to evaluate the quality of the causal evidence, and an outline of review procedures. Appendix A describes the methods used to identify the relevant research.

Eligibility Criteria
CLEAR conducted a broad literature search to identify all the research papers and reports that examined one of the research questions of interest. This search included impact studies examining the effectiveness of a given program; implementation studies associated with these impact studies; and a broad range of descriptive studies. The identified studies were then screened against three eligibility criteria; studies meeting these criteria were entered into the citation database and received a first-level review (see the CLEAR Policies and Procedures for further information about the two levels of review). Additional screening criteria were applied to determine which studies received a second-level review.

Three criteria were used to determine inclusion of a study in the citations database and firstlevel review:

1. Does it examine the population of interest? For the general population of people with disabilities, the research eligible for review under this protocol must include either working-age adults aged 18 to 64 or transition age youth aged 14 to 25 who either (1) have a medical documented impairment that prohibits substantial gainful activity (SGA) and is expected to last for a continuous period of at least 12 months or to result in death or (2) have a functional limitation as indicated by responses to questions from the American Community Survey.3 These age ranges and definitions of disability are consistent with those used by SSA’s disability programs and employment initiatives as well as those used by other federal agencies.4 For the subpopulation of interest, we use the same age range and impairment requirement we use for all people with disabilities.

3 In 2014, SGA is defined as $1,800 a month for blind beneficiaries and $1,070 a month for non-blind beneficiaries. SGA is the measure that the Social Security Administration (SSA) uses to determine whether a person’s impairment has resulted in a disability that prohibits work enough to warrant a benefit award.
4 The age range for transition age youth is from the SSA’s Youth Transition Demonstration. The disability definition is that used by SSA to determine eligibility for disability income support programs. The age range for the working-age population ends at age 64 because starting at age 65, Social Security disability beneficiaries are transitioned onto Old Age and Survivors Insurance benefits.

2

Clearinghouse for Labor Evaluation and Research

People with Disabilities Review Protocol Draft: July 22, 2014

2. Does it examine a program designed to improve labor market outcomes? To be eligible for review, the research must examine a program designed to improve the labor market outcomes of people with disabilities. The program may target all people with disabilities or specific subpopulations. Programs can include incentives and services such as income support incentives, tax incentives, benefit and employment counseling, health insurance, medication management, and other customized supports.
3. Was it conducted in a relevant time and place? To be the most relevant to current practitioners, policymakers, and other stakeholders, the research must have taken place in the United States, including the 50 states, the District of Columbia, territories, and tribal entities on or after January 1, 1985.

Research that meets these criteria is included in the citation database accessible at http://clear.dol.gov. In addition to the citation of the original research, the website provides a link to help interested users locate the research. Finally, CLEAR reviewers draft highlights of every eligible study. These concisely summarize the research objective, description of the program, research methods, and key findings.

Selected studies also underwent a more comprehensive second-level review. For impact studies, this second-level review included assessing the quality of the causal evidence presented in the study, which is summarized in a causal evidence rating. For implementation and other descriptive studies using either qualitative or quantitative methods, the second-level review included assessing the technical qualities of the research approach.

1. Does it contain an impact analysis? Research that used quantitative methods to assess the effectiveness of a program (and other eligibility criteria) received a second-level review as long as it contained an outcome of interest.5 Because one of the goals of the review was to determine which programs are effective at improving the labor market outcomes of people with disabilities, impact studies had to contain at least one outcome in the employment and earnings domain, such as the employment rate or probability of being employed, retention in a job, wages or earnings, and access to employer-provided benefits. Outcomes in the employment and earnings domain are further separated into those examining short-term outcomes, such as retention in a job measured 6 months after the intervention, and long-term outcomes, such as earnings measured 30 months after the intervention.
In addition, because health status is closely tied to labor market outcomes for this population, analyses of secondary outcomes related to health status were also included in the review, as long as the analysis also examined at least one labor market outcome. The health status domain includes outcomes such as self-reported health status, number of physician visits, number of hospitalizations, number of emergency department admissions, total health care expenditures, and out-of-pocket health care expenditures. Like outcomes in the employment and earnings domain, health status outcomes are further divided into short-term and long-term outcomes.

5 Causal studies in this topic area were reviewed according to CLEAR Causal Evidence Guidelines, Version 2.0. The full set of guidelines can be found at http://clear.dol.gov.
3

Clearinghouse for Labor Evaluation and Research

People with Disabilities Review Protocol Draft: July 22, 2014

2. If not an impact study, is it an implementation or other descriptive study associated with an impact study that received a second-level review? To provide information on implementation experiences and other related information relevant to the interpretation of an impact study, CLEAR also conducted second-level reviews of the implementation studies associated with impact studies.6
Causal Evidence Guidelines
This review included both experimental and nonexperimental causal research. CLEAR assessed the quality of evidence for randomized controlled trials (RCTs) using an adaptation of the Institute for Education Science’s What Works Clearinghouse (WWC) standards.7 RCTs can receive a High causal evidence rating if there are no obvious confounds to the RCT design and if the level of attrition in the RCT is low, as assessed using the WWC’s conservative attrition boundary. If CLEAR determines that the RCT does not provide high causal evidence, the research is reviewed using the nonexperimental causal evidence guidelines developed by CLEAR.

Nonexperimental Causal Evidence Guidelines Specific to the Topic Area

In collaboration with a technical work group of experts, Mathematica Policy Research developed a set of evidence guidelines to be used in reviewing nonexperimental studies with causal designs. These causal designs include instrumental variables, difference-in-differences, fixed and random effects, and other types of regression analyses. Research designs that met the causal evidence guidelines received a Moderate causal evidence rating; this rating indicates that there is evidence that the study establishes a causal relationship between the intervention being examined and the outcomes of interest, but there may be other factors that were not included in the analysis that also could affect outcomes of interest. Designs that did not meet the guidelines received a Low causal evidence rating, which indicates that we cannot be confident that the estimated effects are attributable to the intervention being examined.

Causal evidence guidelines for nonexperimental studies were tailored to the topic area of interest. In particular, the topic area protocol set forth the specific types of control variables that had to be included in nonexperimental regression analyses (other than those using fixed effects) for a study to receive a Moderate causal evidence rating. The topic area protocol also described whether changes in group composition should be a concern for the review. Some control variables were only required in certain circumstances. For example, studies that focus on youth with disabilities were not required to control for previous employment because most of that population does not have a substantial work history. The principal investigator (PI) used discretion to determine which variables had to be controlled for, given the intervention and target population of interest.

Control variables. The control variables for the people with disabilities protocol were:

6 Implementation studies in this topic area were reviewed according to CLEAR Guidelines for Reviewing Implementation Studies.
7 See http://ies.ed.gov/ncee/wwc/InsidetheWWC.aspx for details.
4

Clearinghouse for Labor Evaluation and Research

People with Disabilities Review Protocol Draft: July 22, 2014

• Age
• Race/ethnicity
• Gender
• Current beneficiary status (when appropriate)
• Length of benefit receipt (when appropriate)
• Primary impairment (when appropriate)
• Location (at least one measure, which could include, for example, urban/rural status, state, or demonstration site) (when appropriate)
• Pre-intervention earnings or employment status (at least one measure, which could include, for example, pre-intervention earnings or wages, pre-intervention employment status, or pre-intervention work history) (when appropriate)
• Pre-intervention measures of the outcomes being examined (such as out-of-pocket health expenditures or self-reported health status) (required when looking at health status outcomes)

Regression methods that incorporated a matching design, in which statistical methods are used to create a comparison group that is as similar as possible to the group receiving the intervention, had to match on these control variables; if not, they had to include them as controls in the regression.

Changes in group composition. Changes in group composition are relevant for nonexperimental research designs that use aggregate data, for which such a change is potentially a concern. For instance, a difference-in-differences analysis comparing the average change in earnings of program participants to nonparticipants could be biased if the earnings for participants who did not complete the program were not included in the post-intervention outcome measure.
Outcome Domains of Interest to the Review
The following five outcome domains are of interest for all CLEAR topic areas:

• Short term employment status (that is, employment status in the first 12 months after the intervention start date)
• Long term employment status (that is, employment status after the first 12 months in the intervention)
• Short term earnings (that is, earnings in the first 12 months after the intervention start date)
• Long term earnings (that is, earnings after the first 12 months in the intervention)
• Gains in educational attainment, degree completion, or certificate/certification completion

In addition to the five core outcome domains, for this topic area, we report on two other outcome domains:

5

Clearinghouse for Labor Evaluation and Research

People with Disabilities Review Protocol Draft: July 22, 2014

• Health. This domain includes any measures of health status, such as general health status, unmet health needs status, functional limiataion sttaus, and ability to perform activities of daily living.
• Disability benefit applications and receipt. Some interventions in this topic area focus on influencing disability benefit applications, disability benefit receipt, or utilization of employment supports and services. When applicable, we report on these outcomes.
Review Procedures
For first-level reviews of all types of research, a trained reviewer used an abbreviated rubric to systematically capture information about the research question of interest, design, setting, data, methods, and key findings. A quality assurance reviewer confirmeds the information contained in the rubric is accurate.

For second-level reviews of all types of research, a trained reviewer read each report that meets topic area criteria in detail; applied the full set of relevant review guidelines; and documented all aspects of the review in a comprehensive rubric. In addition to the fields contained in the abbreviated rubric, the comprehensive rubric contains an assessment of the technical aspects of the research and considerations for interpreting the findings. If the research does not have a causal design, and thus does not have a causal evidence rating, the comprehensive rubric underwent a quality assurance review by a senior CLEAR staff member to confirm that the information contained in the review rubric is accurate and verifiable.

However, second-level reviews of causal research were given additional scrutiny to ensure the accuracy of the assigned causal evidence rating. If the first reviewer assessed the quality of causal evidence as High or Moderate, a second reviewer also reviewed the study to confirm that such a rating is warranted. Any discrepancies between the two reviewers’ ratings were resolved by the PI or the content expert. If the first reviewer assigns a rating of Low, the PI examined the comprehensive rubric and confirmed that the rating is appropriate. When a report containing causal research does not contain sufficient information to determine its causal evidence rating, CLEAR may contact the study authors to gather this information; whether this step is taken depends on the age of the study and the quantity of information that would need to be gathered (so as not to overly burden study authors). Authors receive a minimum of four weeks to respond, and reasonable requests for extensions are granted. If the information is provided by the authors, it is incorporated into the review and factors into the causal evidence rating. If the authors do not provide the relevant information, the design is given the highest rating that can be determined with the information available in the report.

6

Clearinghouse for Labor Evaluation and Research

People with Disabilities Review Protocol Draft: July 22, 2014

APPENDIX A

CLEAR conducted a comprehensive literature search to identify research meeting the
eligibility criteria described in the review protocol. The search was conducted using the Scopus
database, which indexes 20,874 peer-reviewed journals (including 2,800 open access journals), 367 trade publications, and 5.5 million conference papers and books.8 Unpublished research was
identified by searching the Social Science Research Network, which contains abstracts on over 544,800 scholarly working papers and forthcoming papers.9 The parameters for both searches
were:

• Location in the United States
• Published in the English language
• Published from 1985 to the present
• Studies of working-age adults and transition age youth with disabilities, including those with psychiatric disabilities or with traumatic brain injuries or post-traumatic stress disorder
• Exclusion of editorials, letters, newspaper articles, or commentary
• Studies that were quantitative and qualitative, including randomized controlled trials, causal studies, comparison studies, descriptive studies, implementation studies, and impact studies

The search used combinations of the following search terms (asterisks indicate truncation; “w/” indicates proximity searching):

8 http://www.info.sciverse.com/scopus/scopus-in-detail/facts 9 http://www.ssrn.com/
7

Clearinghouse for Labor Evaluation and Research

People with Disabilities Review Protocol Draft: July 22, 2014

Table A.1. Concepts and Key Search Terms Used in Literature Search

Concept Social Security Age General population Subpopulations
Intervention
Study type for general population
Study type for sub-populations

Keywords
“disability insurance” “social security” “supplemental security income”
college “high school dropouts” “high school drop-outs” post-secondary” “transition w/3 youth” “work* w/3 adult*” “working age adult*”
beneficiar* disab* impair*
“intracranial injur*” “mental illness*” PTSD “post traumatic stress disorder*” “post-traumatic stress disorder*” “posttraumatic stress disorder*” “psychiatric disabilit*” TBI “traumatic brain injur*”
“accelerated benefit*” “benefit w/3 counseling” demonstration “employment w/3 counseling” “employment w/3 service*” “employment w/3 support*” “employment w/3 training” “employment w/3 transition” “income support” intervention* “job w/3 training” “labor market outcome*” “supported employment” “tax w/3 incentive*” "vocational rehabilitation" “work w/3 incentive*” “work w/3 training” “workplace w/3 support*”
"comparison group*" "control group*" QED “quasi experimental design” “quasi-experimental design” RCT*” “randomized control trial*” “randomized controlled trial*” "treatment group*"
“descriptive studies” “descriptive study”

In addition to the database searches, relevant research was identified via Custom Google Search Engine searches of the websites of 20 federal agencies and other organizations. The websites of these agencies and organization were searched because they have planned,

8

Clearinghouse for Labor Evaluation and Research

People with Disabilities Review Protocol Draft: July 22, 2014

implemented, or evaluated programs that promote employment among people with disabilities. These 20 agencies and organizations were:

• Abt Associates • AIR (American Institutes for Research) • ASR Analytics • Centers of Medicare and Medicaid Services • Department of Health and Human Services • Department of Labor • Department of Health and Human Services • DIR • ICF International • Lewin • MDRC • Navigant • NORC • Rand • RTI • Substance Abuse and Mental Health Services Administration • Social Security Administration • Social Solutions • Substance Abuse and Mental Health Services Administration • Urban Institute • Westat

9

Clearinghouse for Labor Evaluation and Research

People with Disabilities Review Protocol Draft: July 22, 2014

APPENDIX B

Studies with a high causal evidence rating
Bohman, T., Wallisch, L., Christensen, K., Stoner, D., Pittman, A., Reed, B., & Ostermeyer, B. (2011). Working Well—the Texas Demonstration to Maintain Independence and Employment: 18-month outcomes. Journal of Vocational Rehabilitation, 34, 97–106.

Chambless, C., Julnes, G., McCormick, S., & Brown-Reither, A. (2009). Utah SSDI '1 for 2:' Benefit Offset Pilot Demonstration Final Report. Salt Lake City, UT: University of Utah Center for Public Policy & Administration.

Davis, L., Leon, A., Toscano, R., Drebing, C., Ward, L., Parker, P., Kashner, T. & Drake, R. A (2012). Randomized controlled trial of supported employment among veterans with posttraumatic stress disorder. Psychiatric Services, 63(5): 464-470.

Decker, P., & Thornton, C. (1995). The long-term effects of transitional employment services. Social Security Bulletin, 58(4), 71-81.

Delin, B., Hartman, E., Sell, C., & Brown-Reither, A. (2010). Testing a SSDI Benefit Offset: Evaluations of the Wisconsin SSDI Employment Pilot. Menomonie, WI: University of Wisconsin-Stout.

Fraker, T., Baird, P., Mamun, A., Manno, M., Martinez, J., Reed, D., & Thompkins, A. (2012). The Social Security Administration’s Youth Transition Demonstration Projects: Interim report on the Career Transitions Program. Princeton, NJ: Mathematica Policy Research.

Fraker T., Baird P., Black A., Mamun A., Manno M., Martinez J., Rangarajan A., & Reed D. (2011). The Social Security Administration’s Youth Transition Demonstration Projects: Interim Report on Colorado Youth Wins. Princeton, NJ: Mathematica Policy Research.

Fraker, T., Black, A., Mamun, A., Manno, M., Martinez, J., O’Day, B., O’Toole, M., Rangarajan, A., & Reed, D. (2011). The Social Security Administration’s Youth Transition Demonstration Projects: Interim report on Transition WORKS. Princeton, NJ: Mathematica Policy Research.

Fraker, T., Honeycutt, T., Mamun, A., Manno, M., Martinez, J., O’Day, B., Reed, D., & Thompkins, A. (2012). The Social Security Administration’s Youth Transition Demonstration projects: Interim report on Broadened Horizons, Brighter Futures. Princeton, NJ: Mathematica Policy Research.

Fraker, T., Mamun, A., Manno, M., Fraker, T., Mamun, A., Manno, M., Martinez, J., Reed, D., Thompkins, A., & Wittenburg, D. (2012). The Social Security Administration’s Youth Transition Demonstration Projects: Interim report on West Virginia Youth Works. Princeton, NJ: Mathematica Policy Research.

Fraker, T., Black, A., Broadus, J., Mamun, A., Manno, M., Martinez, J., McRoberts, R., Rangarajan, A., & Reed, D. (2011). The Social Security Administration’s Youth Transition Demonstration Projects: Interim report on the City University of New York’s project. Princeton, NJ: Mathematica Policy Research.

10
ReviewDisabilitiesPeopleReportOutcomes