Components

Experiments v. Observations

Designed Experiment: A study where the researcher controls the conditions under which observations are taken and imposes treatments on individuals in order to observe the responses.

Observational Study: A study where the independent variable is not controlled by the researchers.

• Take two plots of wheat

  • Plot A is a standard species

  • Plot B is genetically modified

Vocab

• Experimental Unit (EU): The smallest unit to which a treatment is independently assigned/applied.

• Observational Unit (OU): The smallest unit on which observations are made.

• Treatment (Trt): Experimental condition applied to experimental unit.

• Factor: A controlled independent variable; a variable whose levels are set by the experimenter.

• Response: The thing we measure to determine the effect of the treatments

In Practice

Researchers want to understand the difference between two different species of clover

• They plant both breeds in equal sizes plots

• Each plot is watered/fertilized/cared for the exact same

• After 15 days the researchers harvest half of each plot, then the other half after 30 days

• Each harvest is dried and weighed in bulk

Pros

• The “signal” or information in experimental data can be very clear

  • We can get rid of messy components of processes

  • We can control for things we don’t want

• Experiments can be built to study interactions between factors and treatments

Cons

• Experiments are an ethical minefield

• It’s expensive to get a lot of data

• If the design is bad, the results are bad

Completely Randomized Design

Randomized Complete Block Design

• Blocking: Taking a group of individuals known prior to the experiment, who share some attribute that is expected to affect the response.

Matched Pairs

Two individuals are determined to be similar in the ways that are important to a study, one is given treatment A and the other is given treatment B.

• Twin studies

• Genetically identical crops / mice

• Topographically identical locations?

  • Ukraine and Kansas are (shockingly) topographically and meteorologically similar

Repeated Measures

Each individual is given both treatments with randomized order and assignment of order to avoid bias.

How much is this bag?

How old am I?

Independent Processes

• If observing variable A doesn’t change our observation of variable B

  • The variables are independent

• We observe that fish in a stream near Manhattan are all purple

  • Will this effect the fish in Monterey Bay in California?

Why do we care?

• If our data was generated from a series of dependent processes

  • We have to do something to adjust for that

  • The guesses on bag prices might still have an accurate variance

  • But we have to consider the shift in guesses

• If our data is from an independent process we can work with it as it is

Independent and Identically Distributed

• Each individual in the experiment much be worked with identically

  • Beyond any application of treatment

• Imagine if you performed a study on two groups of rats

  • One was provided a supplement meant to improve cognition, the other is a control group

• Both are run through a maze and timed on speed of completion

  • But the control group was malnourished prior to being put in the maze

Replication

• If I can’t repeat your experiment using your exact same steps and get similar results

  • That’s not a good experiment

  • Your results are likely invalid

  • You’ve potentially committed academic fraud

• Replication in observational studies

Fallacy of Accuracy

• All experiments, studies, and models are crude representations of reality

  • That’s the point

• If a study is making strong claims like:

  • Our experiment perfectly captures the biological processes

  • These results display the exact expected outcomes in nature

Double check their work

Wolves

Idaho, Montana, and Wisconsin histotically have an issue with wolves killing cattle and sheep on ranches. The states keep track of every wolf that’s killed by farmers or hunters as well as every cow/sheep that’s killed by wolves.

• Over time they found that as more wolves were killed more cattle and sheep were killed.

  • Does this make sense?

  • What could be causing this result?

Confounding and Collinearity

Confounding: Two variables in a study/experiment with relatively indistinguishable effects.

Collinearity: Variables are so heavily confounded that we cannot separate the information in one from another at all, they’re 1:1 paired.

• We have to adjust for these problems in our design and analysis

• Experiments can prevent both of these problems if we design well

  • If we design poorly we can end up with data that looks like an observational study

Bias

Systematic bias

• One outcome is systematically preferred over others

  • An improperly callibrated scale

  • A poorly designed survey

  • Inconsistent treatment of subjects

• Unlike data fraud, this isn’t always intentional

• Unlike random error, this is always preventable

Incentives

If you’re conducting a trial to determine the impact of a social welfare program, should you reward your participants?

• Poor/homeless individuals are easy to exploit

• Paying people for their time and effort is how our economic system functions

Placebos

You acquire \(3000\) patients for an experimental HIV vaccine trial and provide \(100\) of them a placebo.

• Is it wrong to administer a placebo?

• You’ve created a clear effect of treatment and eliminated a lot of confounding variables. Isn’t that good?

Intellectual Property

You want to determine the pest resilience of a strain of GMO corn. Your experiment is industry sponsored and is taking place at an industry owned farm, directly adjacent to a family owned farm. Your crop spills over to the family owned farm and changes the genetic makeup of their crop.

• Have you harmed or helped them?

• Do you owe them compensation?

• Are they stealing your intellectual property by selling their crop now?