Calibration Studies Testing

Overall Sample | Scale Setting Sub-Sample | Additional Chronic Pain Sample | Sleep and Wake Disturbance Sample | PROMIS Standards for Measure Developmentpdf icon | PROMIS Standards slide deck pptx icon

From July 2006 to March 2007, data were collected from the U.S. general population and multiple disease populations. The sampling plan was developed for collecting responses to the candidate items from the targeted PROMIS domains and was designed to accommodate multiple objectives:

  1. create item calibrations for each domain;
  2. estimate profile scores for various disease populations;
  3. create linking metrics to legacy questionnaires (e.g., SF–36);
  4. confirm the factor structure of the domains; and
  5. conduct item and bank analyses. Because of the large total number of items (> 1000), it was not possible for participants to respond to the entire pool. Based on an estimate of 4 questions per minute, the number of items administered to any respondent was limited to about 150 (37 minutes).

PROMIS Wave 1 Sample

Two data collection designs ("full bank" and "block administration") were used. Some individuals were administered full banks of 56 items for a subset of the PROMIS Calibration Studies domains while others were administered 14 blocks of 7 items selected from the 14 item banks (3 physical functioning banks, anxiety, depression, anger, substance abuse, fatigue impact, fatigue experience, social–role performance, social–role satisfaction, pain interference, pain quality, pain behavior). In addition to the PROMIS items and appropriate "legacy" items (items from widely used fixed length measures such as the SF–36) completed by those administered full banks of items, participants were administered approximately 21 auxiliary items consisting of global health rating items and sociodemographic variables including age, income, number of hospitalizations, disability days, whether they take prescribed medicines, body mass index, gender, race/ethnicity, whether the person was married or living with someone, educational attainment, and whether the person was working full–time, a full time student, or not working full time or going to school full–time. There were also a series of questions about the presence and degree of limitations related to 25 chronic medical conditions: hypertension, angina, CAD, heart failure, heart attack, stroke or TIA, liver disease, kidney disease, arthritis or rheumatism, osteoarthritis, migraines, asthma, COPD, diabetes, cancer, depression, anxiety, alcohol or drug problems, sleep disorder, HIV/AIDS, spinal cord injury, multiple sclerosis, Parkinson's disease, epilepsy, and amyotrophic lateral sclerosis.

The full–bank administration allows for evaluation of dimensionality and calibration within item banks (domains). The block administration permits an evaluation of associations between domains. The ability to calibrate items on a common metric (item linking) is possible because of the administration of blocks of items from the full bank to general population and clinical samples. Each item was administered to at least 900 respondents from the general population and 500 respondents with a chronic medical condition.

Data were collected primarily by YouGovPolimetrix (www.polimetrix.com, also see www.pollingpoint.com), a polling firm based in Palo Alto, CA. YouGovPolimetrix operates PollingPoint.com, a centralized portal that allows interested individuals to provide their views about public policy and other current issues. The respondents for a typical YouGovPolimetrix Internet survey are selected from the PollingPoint panel, a panel of over one million respondents who have provided YouGovPolimetrix with their names, physical addresses, email addresses, and other information, and who regularly participate in online surveys. YouGovPolimetrix uses a sample matching procedure to select a representative sample when a listing of the population is available but it is difficult to draw a random sample from the population frame. The sample matching methodology starts with a listing of all respondents in the desired or target population. Next, a random sample of the desired size is selected from the population listing (the "target sample"). Third, for each element of the target sample, the closest match is selected from the PollingPoint panel. This method has been shown to give accurate results in a wide variety of applications, even for groups significantly underrepresented on the Internet. Its validity depends upon the panel of available respondents being sufficiently large and diverse, not upon Internet usage or other types of behavior. For PROMIS, we specified targets in terms of gender (50% female), age (20% in each of 5 age groups: 18–29, 30–44, 45–59, 60–74, 75+), race/ethnicity (10% black and Hispanic), and education (10% less than high school graduate).

Panelists have been recruited by a variety of methods, such as e-random digit dialing, invitations via web newsletters, and Internet poll-based recruitment where panelists have opted to participate in a survey advertised on the World Wide Web. The topics of these surveys vary greatly – from politics to polls on popular entertainment. All of the PollingPoint panelists have provided their e-mail so that they may receive survey invitations to participate in future surveys. Panel members receive minimal compensation.

This methodology was developed by Stanford Political Science Professor Douglas Rivers and was recently evaluated using the 2005 California Special Election. The matched samples performed as well or better than conventional random-digit-dial (RDD) telephone surveys, correctly predicting the outcomes for all of the propositions and having an average absolute error of 3 percent (consistent with normal sampling error).

In general, YouGovPolimetrix has reported a base response rate of 67% for general population surveys (similar to the general population component of the PROMIS Calibration Studies data collection) and rates at or exceeding 80% with more specialized, clinical samples such as the ones in PROMIS. They also utilize sample matching methodologies for selection of representative samples from non–randomly selected pools of respondents, which is ideally suited for Web access panels.

The PROMIS Calibration Studies sample included 21,133 respondents, with 1,532 recruited from primary research sites associated with PROMIS network sites and the vast majority (19,601) from YouGovPolimetrix’s panel sample. YouGovPolimetrix sample data was collected using their website on a secure server. The PROMIS network site data was collected using the PROMIS Assessment System.

The PROMIS Statistical Center (PSC) received de–identified datasets from YouGovPolimetrix. Each of the 7,005 individuals (6,676 from YouGovPolimetrix, 236 from UNC, and 93 from Stanford) assigned to administration of full item banks was administered a pair of the 14 item banks. The 7–item blocks for each of the 14 banks were administered to 14,128 individuals (6,245 general population, 7,883 clinical samples). The clinical samples included persons with heart disease (n = 1,156), cancer (n = 1,754), rheumatoid arthritis (n = 557), osteoarthritis (n = 918), psychiatric illness (n = 1,193), chronic obstructive pulmonary disease (n = 1,214), spinal cord injury (n = 531), and other conditions (n = 560).

The clinical population supplied by YouGovPolimetrix was identified through a pre–survey of 250,000 YouGovPolimetrix panel members. These respondents were asked to complete the PROMIS clinical form. Persons who self–reported currently being diagnosed with a given condition were eligible for inclusion in the Calibration Studies clinical sample associated with that condition. It should be noted that the general population sample included people with and without diseases (general population respondents were also administered the clinical form but their responses did not impact their participation in the general population sample).