Tuesday, July 22, 2014

Creating an activity about Bayes' Theorem from a blog post

When I teach a topic I haven't taught for a while, I usually refer to some old texts, my course notes, and the internet for new ways of presenting the topic. In my statistics class we were to cover Bayes' Theorem, a topic I have always enjoyed presenting, but never felt that I got quite 'right'. Students seemed disconnected from the idea, and weren't able to answer basic questions about the idea during the first lecture.

To address this I wanted to create an activity where students were to apply Bayes' Theorem in a relatively simple way. Searching the internet I found the article (an essay really) An Intuitive Explanation of Bayes' Theorem by Eliezer S. Yudkowsky, and thought it did a good job explaining the basic idea, and even includes different presentations of the same example. These different presentations are used to discuss innumeracy in health professionals, but provided me a variety of ways of presenting this example.

After some self-editing and debate, I settled on using the simplest presentation of this example; Statistics I Activity - Bayes' Theorem. While the questions aren't directly about Bayes' Theorem, it gets students more familiar with conditional probabilities and how to compute them.

Creating an activity from a blog post or article has some advantages I didn't realize until I presented the activity to students:

  • Students are able to confirm their answers by reading the article, a noble goal by itself. 
  • I don't mention where I pulled the example from, so they have to search through the article to find the 'answers'. If they read through parts of the article by accident, even better.
  • The content becomes richer by pulling from outside resources. I dislike the idea that a course is just about what I, as the instructor, want from students. The ideas and concepts we are talking about are greater than just me, the textbook, and the student. Using someone else's perspective on the topic makes the course 'bigger'.
If you have any questions, comments, or suggestions, feel free to share them below.

Monday, July 21, 2014

What would you like to see on Applied Abstractions?

As with most data-driven people, I've been keeping an eye on the (limited) statistics for the site, and I've noticed some trends I would like to follow-up on. Please fill out the following form to help drive where this blog goes.

Friday, July 11, 2014

Part III - Forming an interesting in-class Statistics activity

I've discussed what I wanted for this experiment activity, what my plan was, and now I'll talk about how it went and what to change in future activities. Overall I thought it went pretty well, but there were a few major changes that had to be made on the fly. The biggest one being the math tests themselves; I completely underestimated my student's basic math ability and we came up with useless data. Someone mentioned that they thought it was part of the activity, and was a great way to show how 'messy' statistics really is. I'm glad they thought that, because I really didn't anticipate it.

The initial discussion of how to construct this experiment was useful and demonstrated a number of ideas we discussed in class. Controlling for certain variables turned into a big part of the discussion, namely how to control for people with natural math ability. We decided to do a paired sample, pairing those people of the same math ability by their score on the first test. I asked if this was really the best measure, and we had a good conversation of how to measure someone's math ability, and how for some people, that's their job.

To control for some of these variables, and to construct a basic demographic survey, I had students develop a few survey questions that may help explain some variation in math ability. This discussion included what to ask, how to ask it, and what kinds of variables (categorical, numerical, etc.) we were measuring. I suggested a question about how long it has been since you took a math class, and some students wanted to do a categorical variable of 0-6 months, 7-12 months, etc. I responded with the question "Is it easier to turn numerical data into categorical data or categorical data into numerical data?" and we talked about converting from one data type to another, and how we were going to use the data.

We also talked about what would happen if a person took two similar math tests back to back. One student mentioned that people would become fatigued,  and rightly so. To limit this I asked what we could do to limit the fatigue, and we discussed the pros and cons of long and short tests. I also mentioned the idea of activating previous knowledge and that after seeing the first test, students would remember how to complete the questions for the second test. We 'settled' on giving both tests to both control and experimental group, and 18 basic math questions... since thats what was in the packet. I know this isn't in the true spirit of exploratory activities, and some people might deride me for exerting this amount of control over the process. I want students to explore this material and engage with it, but doing everything on the fly doesn't seem conductive to these aims. Without some kind of structure students get bored, annoyed, disengaged with the learning process. I can deal with the first two things (barely) but the third I can't.

Students then took the survey we constructed together (number of months since last math class, sex, age, handedness, work status) and the first test. I took them all, handed them back out randomly, and we graded them. I then had students come up to the computer to enter in the information in Excel. This was a good step since it showed that data entry is an important step, one people take for granted. It is time consuming work, and must be done accurately. This was at the 1-hour mark and once they entered the data we had a 5-minute break.

Once the data was entered there were some survey responses that didn't make sense. Instead of age, one person put 'old'. For the number of months since last math class, someone put the categorical variable response 0-6 months even though we settled on a numeric one. I then discussed data cleansing and that our simple decisions on how to handle these discrepancies has real impact on our data. For the 0-6 month responses, we inputted 3 and included an asterix. For the 'old' response we took the oldest age in the data set (26) and replaced 'old' with that number, including an asterix.

After grading and inputting the data, all but two people received perfect scores on the first test. We discussed how we couldn't use this data since we are looking for improvement in math ability after attempting this puzzle, and we can't show improvement if everyone scored perfectly the first time. I quickly made another test that was more difficult (basic algebra, roots, percentages), had students complete it, graded it, and the scores were much more varied.

This turned out great, even though my veins went to ice when I looked at the initial scores. Students saw that our question couldn't be answered with the data set we so carefully constructed and we had to start again. This demonstrates the 'messiness' of statistics I try to get across to them and how you really have to rely on sound statistical principles, and your understanding of the context to get good data.

Creating the sample was now fairly simple, we paired people based on their initial scores and randomly assigned one to the control group and one to the experimental group. In the pairing we noticed that we had an odd number of people. I asked if there was an observational unit (person) that their initial test and survey information seemed to be outside everyone else's. We decided to remove the 'old' entry from above, since it did not seem comparable with the others. Once we did that we created each of the groups.

The experimental group then had 10 minutes to work on the puzzle. I did not say they had to complete it, just that they had to attempt it. The control group worked on the part of the activity that was to be turned in, descriptive statistics and a box-and-whisker plot of the four data sets (pre-test/post-test, control/experiment). Once the time was up I made another quick test, administered it, we graded it, and I collected the scores.

Getting all the data together there was a distinct difference in the control and experimental group. Averaging the before and after puzzle scores for both groups, it was clear that the experimental group's averages were about ten percent higher than the control group's. I then found the differences in scores, averaged those, and found the percent increase to be about six percent. These two averages came out to two different numbers, but I asked which one would I pick to use to sell my puzzle? We then had a conversation about these sorts of averages, how to compute other similar numbers, and how marketers do similar things in their promotional materials.

Overall I thought it was great, and would probably keep most of it, including the too-easy of tests. I would be a little more prepared and have some back-up tests to use, but it really demonstrated that your initial plan sometimes doesn't work. I would like to include scatterplots and linear regression models next time, but it is not included in our Statistics I course.

I could have done a better job of demonstrating how to control for a variable, and used some basic descriptive techniques to do so. For example, breaking the control and experimental groups into sex, handedness, or work status could show if there were any significant differences in these groups.

Thank you for reading these posts about me struggling through the planning, development, and execution of this activity. If you have any thoughts or questions feel free to post them below.

Thursday, July 10, 2014

Forming an interesting in-class Statistics activity - Part II

After the last post I spent some time thinking about my options for an experiment to do in-class that fulfilled all of my requirements. After some thought I decided on the following plan.

  • Claim to the class that I have a logic puzzle that I believe will help people with their math skills. I want to be able to put something on my website saying that this puzzle helped people with their math by some incredible percentage.
  • Walk students through how I was to do this, including samples, factors to control for, and how to create the tests. I guided the conversation so that we would also collect information on how long ago they took their previous math class, to ensure that the control and experimental groups had a variety of math ability levels (measured by an initial test), to anonymize the results, and discuss how we would use the class setting in the most appropriate manner.
  • I created a packet with all the necessary tests, the puzzle, and data recording forms. When printed I assigned random numbers to each packet. I would hand out these packets at this time, asking students to remember their number.
  • Students took the first test, I collect it, hand it back out randomly, and we graded the results.
  • Collecting the data in Excel on the projector, we would then make our sample based on the survey questions we agreed on and the scores. At this time we would compare different sampling methods and how to make them work.
  • Have the students in the experimental group work the logic puzzle, and the control group start on computing descriptive statistics of the pre-puzzle scores. I would later ask all students to compute descriptive statistics for both tests, both samples, as what they were to turn in.
  • After 10 minutes, have all students take a similar test, collect them, hand them back out randomly, grade them, and collect their scores.
  • Having collected this information we would then do some basic statistical analysis on each group, comparing means, medians, standard deviations, and possibly five-number summaries. Comparing the means and medians of each group, and comparing the difference in the two scores and then their means, I would create two different reasonable measures for what I could include on my website. I would like to do some linear regressions, but that material isn't covered in this class but in Statistics II. 
  • Wrap-up with a discussion on how to make this experiment better. Guide the conversation to the placebo effect (I did mention to everyone that I thought these puzzles would help.), blinding, sources of bias, and anything else students mentioned. 
Here is the packet I put together, Statistics I In-Class Activity: Experiment. Tomorrow's post will be about how the activity worked, what didn't, and how I'd like to change it in the future.

If you have any suggestions, or you use this or a similar activity in your Statistics class, feel free to share by commenting below. Thanks!

Tuesday, July 8, 2014

Forming an interesting in-class Statistics activity.

This Wednesday I've scheduled an in-class activity that relates some of the ideas we've discussed in my Statistics I class. So far they include surveys, descriptive statistics (mean, median, quartiles, and standard deviation), sampling, experiment design, and observational studies. Writing this on Tuesday morning, I have nothing written down but I have a couple ideas of what I want this activity to include:

  • Perform an experiment and/or an observational study. I covered what these are in a lecture, but I really want them to attempt one (or both) on their own so they get a concrete idea of what they are. Doing both would be best since they would then be able to compare the methods of both.
  • Take appropriate samples based on the research question. Ideally this would be done with students in the class so they can see how to take appropriate samples.
  • Compute and use descriptive statistics to compare and contrast different samples. Quantitative reasoning and analysis are a core focus in statistics. 
  • Talk about statistics. Whether it be in groups or in a presentation, forming ideas and communicating them to others is another core skill in statistics.
I also know what I don't want:
  • A rote activity that requires no input from students. I want them to struggle with questions about sampling, which statistic to compute, what to do next. Through this struggle I want them to appreciate principles like variability, controlling for certain variables, and how to construct arguments for one action or another.
For an experiment I have a few ideas:
  • Performing an intervention dealing with basic math skills. The idea would be to assess if a certain intervention (a logic puzzle, or a game) has any effect on student performance of a basic math test. We would discuss how to create a control, how to create the samples, how to administer the tests, etc. While the direct applicability of this example may be a stretch, it would be a constructed activity that students could perform.
  • Have the students develop their own question and proceed from there. This has some issues, primarily because of the unending and uncertain nature of such a question. Without some guidance the possibilities are a bit too large, which generally leaves students not choosing anything at all.
  • Provide a context (marketing manager, nurse's aide, etc.), and have students develop an interesting question that can be answered in-class. While being more specific we may not be able to answer such questions in-class. For example, if we wanted to do A/B testing of a website, we would have to construct this example fairly quickly, and perform the experiment on people who weren't part of the development of the website.
For an observational study:
  • Given a data set have students develop an appropriate research question, and try to answer it using the data. This could be generalized further by having multiple data sets and have students pick one, or assign different groups different data sets.
  • Taking the survey data collected at the beginning of the course, ask an appropriate research question that can be answered either by the previously collected data, or by another survey in class. The survey data is anonymous so it would be difficult to answer some questions.
You can probably guess which way I'm leaning, but I'll post the completed activities tomorrow. Feel free to post in the comments if you have any ideas or suggestions.