Canadian Forces Mental Health Survey (CFMHS)

Detailed information for 2013

The purpose of this survey is to collect information about the mental health status and the need for mental health services in the Canadian Forces.

Data release - August 11, 2014

• Questionnaire(s) and reporting guide(s)
• Description
• Data sources and methodology
• Data accuracy

Description

The purpose of the Canadian Forces Mental Health Survey is to collect information about the mental health status and the need for mental health services in the Canadian Forces. The information will be used to continue work on preventing mental disorders, overcoming barriers to care, and delivering effective and efficient mental health care to Canadian Forces members.

The specific objectives of the survey are:
1. To assess the mental health status and functioning of Canadian Forces members (all regular forces and reserve members with a previous deployment to Afghanistan) on both illness and positive mental health continuums through selected mental disorders, mental health problems, and well-being;
2. To assess timely, adequate, and appropriate access to and utilization of formal and informal mental health services and supports as well as perceived needs;
3. To evaluate changes in patterns of mental health, service use, and functioning from the CCHS on Mental Health and Well-being Canadian Forces Supplement (2002).
4. To evaluate the mental health impact of the Canadian Forces work environment and deployment in support of the mission in Afghanistan.

Reference period: Data are valid for up to 12 months prior to each respondent's individual interview date.

Subjects

• Health
• Mental health and well-being

Data sources and methodology

Target population

The target population consists of all full time regular members of the Canadian Forces and reservists who have been deployed in support of the mission in Afghanistan.

Instrument design

The Canadian Forces Mental Health Survey (CFMHS) questionnaire was developed by Statistics Canada in collaboration with the Department of National Defence (DND), who conducted a content prioritization consultation with their stakeholders. The content for this survey is partly based on a selection of mental disorders from the previous Canadian Forces survey on Mental Health, done in 2002 (as a Supplement of the Canadian Community Health Survey, Cycle 1.2 on Mental Health and Well-Being). The other content areas come from existing sources such as the Canadian Community Health Survey - Mental Health (CCHS - Mental Health) done in 2012 and other special studies.

Content was chosen using the following set of criteria:
- Issues identified as data gaps from the stakeholder consultations
- Significant number of Canadian Force members affected by the targeted issue
- Significant impact on the Canadian Force's work environment and health care services.
- Data has potential for health improvement with policy intervention
- Issues identified as priority for the support/development of programs and policy, surveillance requirements and/or research
- Comparability with previous cycles of CCHS Mental Health survey

The CFMHS questions are designed for computer-assisted interviewing (CAI), meaning that, as the questions were developed, the associated logical flow into and out of the questions was programmed. This includes specifying the type of answer required, the minimum and maximum values, on-line edits associated with the question and what to do in case of item non-response.

In collaboration with Statistics Canada's Questionnaire Design Resource Centre, the questionnaire was subjected to qualitative testing, which took place in September and October 2011 and consisted of one-on-one interviews with Canadian Forces members. The objective was to evaluate respondent reactions to and understanding of the survey, as well as their willingness to respond to the questions. The qualitative testing was also used to obtain estimates of the timing for the various sections.

The content for this study is partly based on a selection of mental disorders from the World Mental Health Survey (WMH). The other content areas come from existing sources such as the Canadian Community Health Survey (Cycle 5.2) and other special studies. An Expert Group of mental health professionals guided the content development and strategic direction of the study. As well, the survey program is supported by a standing Advisory Committee, comprised of provincial and territorial ministries of health, Health Canada and Canadian Institute on Health Information (CIHI) representatives.

Qualitative testing for the CFMHS took place in September/October 2011 to evaluate respondent reactions with regards to the sensitivity of the subject matter, their ability to understand and willingness to respond to the questions.

This qualitative testing consisted of one-on-one interviews and was organized by the Questionnaire Design Resource Centre at Statistics Canada. The purpose of this testing was to: obtain information on the respondents willingness and ability to provide the information and determine what would encourage response; determine how the CF membership would react to Statistics Canada, an external agency, collecting the survey information.

Sampling

This is a sample survey with a cross-sectional design.

It was decided that 6,550 responding full time members and 1,550 primary reservists would be required at survey completion to enable the analysis sought at satisfactory precision levels. To ensure the survey ended with the targeted responding populations, the sample sizes were enlarged before data collection to take into account out-of-scope persons and anticipated non-response. For this reason a sample size of 9,200 full time members and 2,250 reservists were initially selected.

The sample frame used for this component was an administrative list of all regular members and all reserve force members having been deployed to Afghanistan at least once, as of September 2012. This list was provided to Statistics Canada by the Department of National Defence (DND) and was taken from the Canadian Canadian Forces Human Resource Management System. This sample frame contained some information on each person such as gender, region, rank, military base, environment, first language, etc.

For this survey, each target population (regular force members and reservists) was stratified by rank, and the regular member population was also stratified by whether or not they had been deployed to Afghanistan. To avoid very small cells the rank characteristic was collapsed into three categories. Once the stratification criteria were established, the total sample had to be allocated between the strata. For this survey it was decided that a good balance between the reliability of the estimates for each design stratum and for the entire target population was required.

The sample was selected from the sample frame using a systematic sampling approach within each design stratum. Within each design stratum, the units were sorted by CF environment (land, air, sea), CF support base, gender and first language and the final sample was obtained using a systematic sampling scheme. Such an approach guaranteed a proportional representation of units for each of these characteristics although these criteria were not explicitly included in the stratification. Once the sample was selected, a tracing of all selected persons in the sample was conducted in order to determine the actual contact information of each member.

Data sources

Data collection for this reference period: 2013-04-15 to 2013-08-31

Responding to this survey is voluntary.

Data are collected directly from survey respondents.

The length of the collection period allows for spreading the workload in the field and more time in which to contact respondents who might be departing or returning from field deployments and or training courses. The vast majority of CFMHS interviews are conducted face-to-face during working hours in private on-base rooms, reserved by DND for survey interviewing.

The computer-assisted person interviewing method (CAPI) is used and the questionnaire was programmed in BLAISE.

CAPI offers a number of data quality advantages over other collection methods. First, question text, including reference periods and pronouns, are customized automatically based on factors such as the age and sex of the respondent, the date of the interview and answers to previous questions.

Second, edits to check for inconsistent answers or out-of-range responses are applied automatically and on-screen prompts are shown when an invalid entry is recorded. Immediate feedback is given to the respondent and the interviewer is able to correct any inconsistencies.

Third, questions that are not applicable to the respondent are skipped automatically.

View the Questionnaire(s) and reporting guide(s) .

Error detection

Most editing of the data was performed at the time of the interview by the computer-assisted interviewing (CAI) application. It was not possible for interviewers to enter out-of-range values and flow errors were controlled through programmed skip patterns. For example, the CAPI application ensured that questions that did not apply to the respondent were not asked.

In response to some types of inconsistent or unusual reporting, warning messages were invoked but no corrective action was taken at the time of the interview. Where appropriate, edits were instead developed to be performed after data collection at Head Office. Inconsistencies were usually corrected by setting one or both of the variables in question to "not stated".

Imputation

The household income variable was imputed on the survey. Missing values due to either respondent refusal or respondent's lack of knowledge of household income were completed using statistical techniques. The main variable of interest was INC_3: 'Total household income - best estimate' but all variables that were derived based on income were also affected. The income variables along with an imputation flag (INCFIMP4) indicating which values were imputed are provided on the data file.

Estimation

The CFMHS used a list of regular members and a list of reservists of the Canadian Forces to select its sample. These two lists represent the eligible regular members and reservist populations at the time of the survey. According to the sample design, the population of regular members was divided into six strata based on deployment status and rank while the population of reservists was divided into three strata based on rank. A sample was selected using a systematic sampling approach in each of the nine strata. This is called Weight 1.

Removal of out-of-scope units

Among all persons sampled, a certain proportion of them are identified during tracing or collection as being out-of-scope. Persons that were not members of the Canadian Forces at time of collection are considered out-of-scope for the survey, as well as members posted outside of the ten provinces at the time of the tracing. Records for these persons were simply removed from the sample, leaving only in-scope persons. This is called Weight 2.

Person non-response

During collection, a certain proportion of selected persons inevitably resulted in non-response. This usually occurred when a person refused to participate in the survey, provided unusable data, or could not be reached for an interview. Weights of the non-responding persons were redistributed to responding persons within response homogeneity groups (RHGs). In order to create the response groups, a scoring method based on logistic regression models was used to determine the propensity to respond, and these response probabilities were used to divide the sample into groups with similar response properties. The information available for non-respondents was limited; as a result, the regression model mostly used characteristics available from the frame, such as gender, CF element and geographic information. Note that RHGs were formed independently within each stratum of each population (regulars and reservists). Consequently, Weight 3 corresponds to the final CF Mental Health Survey weight that can be found on the master file with the variable name WTPM.

Quality evaluation

The 2013 CFMHS underwent data certification after production of the master file by comparing the data to those released in 2002 as part of the CCHS Mental Health - Canadian Forces supplement.

Disclosure control

Statistics Canada is prohibited by law from releasing any information it collects that could identify any person, business, or organization, unless consent has been given by the respondent or as permitted by the Statistics Act. Various confidentiality rules are applied to all data that are released or published to prevent the publication or disclosure of any information deemed confidential. If necessary, data are suppressed to prevent direct or residual disclosure of identifiable data.

In order to prevent any data disclosure, confidentiality analysis is done using the Statistics Canada Generalized Disclosure Control System (G-Confid). G-Confid is used for primary suppression (direct disclosure) as well as for secondary suppression (residual disclosure). Direct disclosure occurs when the value in a tabulation cell is composed of or dominated by few enterprises while residual disclosure occurs when confidential information can be derived indirectly by piecing together information from different sources or data series.

Revisions and seasonal adjustment

This methodology does not apply to this survey.

Data accuracy

Response Rates

In total and after removing the out-of-scope units, 8,393 regular members and 1,867 reservists were selected to participate in the CF survey. Out of these selected persons, a response was obtained for 6,700 regular members and 1,500 reservists which results in an overall response rate of 79.8% for regulars and 78.7% for reservists.

Survey Errors

The estimates derived from this survey are based on a sample of individuals. Somewhat different figures might have been obtained if a complete census had been taken using the same questionnaire, interviewers, supervisors, processing methods, etc. as those actually used. The difference between the estimates obtained from the sample and the results from a complete count under similar conditions is called the sampling error of the estimate.

Errors which are not related to sampling may occur at almost every phase of a survey operation. Interviewers may misunderstand instructions, respondents may make errors in answering questions, the answers may be incorrectly entered on the computer and errors may be introduced in the processing and tabulation of the data. These are all examples of non-sampling errors.

Over a large number of observations, randomly occurring errors will have little effect on estimates derived from the survey. However, errors occurring systematically will contribute to biases in the survey estimates. Considerable time and effort was made to reduce non-sampling errors in the CFMHS. Quality assurance measures were implemented at each step of data collection and processing to monitor the quality of the data. These measures included the use of highly skilled interviewers, extensive training with respect to the survey procedures and questionnaire, and the observation of interviewers to detect problems. Testing of the CAI application and field tests were also essential procedures to ensure that data collection errors were minimized.

A major source of non-sampling errors in surveys is the effect of non-response on the survey results. The extent of non-response varies from partial non-response (failure to answer just one or some questions) to total non-response. Partial non-response to the CFMHS was minimal; once the questionnaire was started, it tended to be completed with very little non-response. Total non-response occurred either because a respondent refused to participate in the survey, or because the interviewer was unable to contact the selected respondent. Total non-response was handled by adjusting the weight of persons who responded to the survey to compensate for those who did not respond.