PRODUCING DATA

Producing new data is expensive, but clear answers often require new data in order to answer a specific question. Alternatively, "available data" is data that was collected for some other purpose, but that may be used to help answer a present question.

Statistical designs for producing new data rely on either random sampling or controlled experimentation.

The purpose of sampling is to study a part in order to gain information about the whole. For example, accountants may sample a firm's inventory to verify the accuracy of its entire records.

An experiment randomly allocates the treatment on the experimental units or subjects in order to observe their response.

Design of experiments

The design of an experiment begins with a description of the response (dependent) variable or variables, the factors (explanatory or independent variables) and what specific treatments will be administered.

Treatment => observation

Before-and-after

observation 1 => treatment => observation 2

Experiments should compare treatments rather than attempt to assess a single treatment in isolation.

e.g. compare the response of the treated group to a control group not given the treatment.

The Control of the effects of other factors and variables (placebo effect, selection of subjects, etc.) is the first aim of the statistical design of experiments.

Bias - the design of a study is biased if it systematically favours certain outcomes.

Randomisation, Replication and Blocking

Randomisation produces groups of experimental units that are more likely to be similar in all respects before the treatments are applied than using non-random methods.

At the end of the study if the differences in the outcome variable between the two groups is too large to attribute to chance, then the difference is called statistically significant. The decision about how large a difference is required to be "significant" depends on statistical inference using the laws of probability. This will be discussed in later sections.

Another principle is that experiments with more subjects are more likely to detect differences than those with fewer subjects. Repeating an experiment on many subjects is called replication.

If it is known, before the experiment is carried out, that other variables of no interest influence the outcome (e.g. age or sex of a patient), then randomisation can be carried out within subsets of experimental units defined by these variables. This is called a BLOCK DESIGN.

For example, the response to treatment for a given type of cancer is expected to depend on the sex of the patient. Ideally, equal numbers of males or females are required in the control or treatment groups and this can be achieved by randomising 50% of males to the treatment group as shown below.

Example - Mice in the lab

In a laboratory experiment with mice, the 5 mice to receive the treatment were selected by the investigator by plunging his hand into the cage containing 10 mice and catching them one at a time.

What is wrong with this method of sampling?

Can you suggest a better method?

This method may appear to be random but perhaps the mice chosen are slower, friendlier or even sicker than the mice who were not caught. This would result in treated group being slower or sicker than the control group. Randomisation would give each mouse an equal chance to be chosen in the treatment group.

The steps in randomisation usually consist of

• assign labels to the experimental units
• use a method to select labels at random e.g. using random number tables or computer generated random numbers.

Comparison Groups

Ideally you take a collection of subjects or objects which are initially alike, divide them into groups, treat the groups differently and measure the outcomes (or responses)

If the investigator randomises the subjects to the respective groups and hence the intervention, the study is called an experiment.

If subjects are not randomised, it is an observational study.

Randomisation is obtained by use of a coin or a table of random numbers.

(References: Moore and McCabe, chapters 3.1-3.3, Freedman, Pisani and Purves, chapters 1 and 2)

Examples of Observational Studies

Two "measurements" were made on each subject

political party supported

views on national debt

These were "measured" at the same time so this is an example of a cross-sectional study.

(ii) Cigarette smoking and lung cancer

It would be unethical to conduct an experiment because you couldn't make people smoke cigarettes (or not smoke cigarettes) and follow them for 20 years or more to see who developed lung cancer.

Also taking a sample from the general population and dividing the sample into those with and without lung cancer and smokers and non-smokers (i.e. cross-sectional study) would result in a sample with very few cases of lung cancer (At any time there are not many people in the population who are alive and are known to have lung cancer since their life expectancy after the diagnosis of cancer is two years).

(Reference: Bonnet, Dickman et. al. 1992. Cancer Survival in South Australia. Univ. Newcastle)

Therefore, a different study design is used in which a group of people with lung cancer (e.g. from a hospital) and a group of people without lung cancer (e.g. their neighbours) are selected by the investigator and asked about their cigarette smoking in the past. This is an example of a retrospective case-control study.

Confounding

A potential problem with observational studies is that there may be factors which differ between the comparison groups and which may affect the outcomes.

If such a factor affects the outcomes it is called a confounding effect.

You need to look out for such factors when you analyse data and make sure you present the results appropriately.

Example - Gender and discrimination at UCB

Is there gender discrimination in the admission of graduate students at University of California at Berkley ?(Freedman, Pisani & Purves, ch. 2)

Table 1

	Students Applied	% admitted
Men	8,442	44%
Women	4,321	35%

Admissions were decided by "major" (equivalent to Faculty in Australia).

Table 2

	Men		Women
Major	Applied	% admitted	Applied	% admitted
A	825	62	108	82
B	560	63	25	68
C	325	37	593	35
D	417	33	375	35
E	191	28	393	24
F	373	6	341	7
Total	2691	45	1835	30

This table doesn't cover all "majors" so the numbers are not the same as in Table 1

• For each "major" women did better than, or about the same as, men - but not overall, why?
• For majors with high admission rates (e.g. A and B) most applicants were men, so many men were admitted
• For majors with lower admission rates (e.g. C and E) more applicants were women, so fewer women were admitted.
• In this example, "major" is a confounding factor because it differs between the comparison groups (men and women) and it affects the outcome (admission)
• The best way to present these data is to show clearly the effect of the confounding factor (e.g. Table 2).
• Crude rates or % (e.g. Table 1 or bottom row of Table 2) should not be used because they give a false impression.
• How do we know whether there is a confounding factor?
• Look at possible confounders when designing the study and collect data to measure these variables.
• Allow for these possible confounders in the calculations when comparing groups.
• If a known confounder exists, then it can be controlled for either in the experimental design by blocking (where randomisation occurs within the blocks) or in the analyses (this is not always possible).

Questionnaire Design

• there are several kinds of information that can be collected using questionnaires. For example, a question to measure someones attitude or behaviour relating to a certain topic such as cigarette smoking, may be more sensitive than a question about an attribute, such as the persons gender or nationality
• be clear about the purpose of your questionnaire, the information you will need and how it will contribute to the research question
• consider the pros and cons of using a questionnaire for your research and be sure that it is the most suitable method
• decide what type of information your questionnaire will be seeking and whether open (where respondents can create their own answers) or closed (respondents have a choice of a limited number of responses) questions, or perhaps a combination of the two, are most appropriate
• list the broad question areas and then write specific questions, keeping in mind the respondent population and make sure that questions are simple enough to be easily understood
• the questions should follow a logical order. The questionnaire must look and read simply and clearly
• pilot-testing is essential, both informally among friends and formally among a sample of respondents
• define the sample population and consider how the questionnaire will be completed and returned for processing
• establish coding categories for each question so that the data entry is made easier
• decisions concerning the analysis to be conducted must be made while the questionnaire is being constructed.