Quality Matters Reviewer Status

This last week I completed the Quality Matters Peer Reviewer Course, and successfully applied to become a Quality Matters Peer Reviewer. It has taken quite a bit of my time these last two weeks, and I'm a bit behind on LAK13, and need to catch up.

I enjoyed the Peer Reviewer Course, and gained a broader understanding of what QM reviews should focus on. The primary area I had difficulty with was providing balanced feedback without sounding repetitive or insincere. I consistently provided constructive feedback that used evidence from the course and the QM Rubric, but was a bit terse and may have turned off the course instructor. I'm generally good at providing positive feedback to students and faculty, but didn't include many positive statements or comment. I suppose I was focusing on the rubric and the course, and not the fact that there was a person behind the course.

I do wish the course relied on individual files less. Almost every link in the course was a separate Word or pdf file. By the end of the course I had two dozen files to wade through. Granted, I have these files for future reference, but having the option to download them, or view them as web pages would be preferable.

As a personal preference, the course used an anthropology course as a sample course to review, and I wish they had chosen a different discipline. Out of all social sciences, I've always had the most difficulty understanding anthropology. A friend of mine just earned their masters in anthropology, and called anthropology the study of human behavior that doesn't fit into any other social science. This is obviously useful, but a discipline that has no clear definition or guiding topic area rubs against my training in axiomatic thinking.

LAK13 Assignment #1 - Analytics: Logic and Structure Postscript

In Learning Analytics and Knowledge 2013, the first assignment, Analytics: Logic and Structure (link is live only if you are registered for the course), has the following description:

For this assignment, develop an analytics model to gain insight into a complex topic using both qualitative and quantitative methods. Select a particular topic or subject area that interests you (current events, historical activities, a learning challenge) and detail how you will "interrogate" this subject using various analytics tools or techniques. Your project can be in the form of a presentation, a blog post, a video, a simulation, or other digital artifact. The important aspect of this assignment is to walk through the processes and considerations that pre-date tool selection.

There are additional details, but this is the core charge of the assignment. I commented on the assignment in the forums:

Our first assignment seems to allow for a wide variety of projects. This is understandable, given the variety of backgrounds of students, varying understandings of statistics, and the convergence of a number of fields that contribute to Learning Analytics.

I then went on to ask some clarifying questions, and made some suggestions as to the structure of the project, specifically getting a data set to work from and then developing an analytics model. Looking back I realize my suggestions were running counter to the intent of the assignment. Mr. Siemens is looking to replicate the situation that people in analytics are dealt with; hodgepodges of data silos, inconsistent objectives from the different institutions (or within a single institution), regulatory barriers, and a legion of other issues. By having us develop a relevant question we would be put in a position to deal with these barriers, and share our experiences in getting around them.

I appreciate his commitment to using authentic contexts, but feel a bit more direction would have been helpful. The description doesn't specify that the context is necessarily learning analytics, and if it were, the issues above would prevent real data from being gathered. In a later post I mention:

It is starting to look like getting a usable dataset is going to be the primary issue for our projects for this course. The sort of data sets that we are looking to use usually contain sensitive information, and in the case of our US colleagues (myself included), using them in such an open setting would run afoul of FERPA. I have a dataset I am working with, but do not feel my institution is in a place to intelligibly create a data policy, let alone a data openess policy.

All this points to a disconnect between my goals for this course, and Mr. Siemens' goals for this assignment. This course being part of my personal and professional development, I want some clear tools and techniques to analyze student-generated data with when I leave this course. I have serious reservations about using my real data, and thus want an available data set that will help me develop those tools and techniques. In this assignment Mr. Siemens is more concerned about the way we frame our analytics questions, and our plans for how to answer them.

To satisfy both of our goals, I've decided to look at student and teacher performance. There is a wide range of open datasets available, and there are powerful questions about learning analytics that can be approached. LA focuses on related, but different datasets, however the tools and techniques to analyze this data set should transfer to analyzing student generated data in an LMS.

LAK13 Assignment #1 - Analytics: Logic and Structure

Introduction

There is an effort within US public K-12 schools to analyze student performance, and use that analysis to improve schools. The most public and political aspect of this effort has been the use of this data to measure teacher effectiveness. A few school districts have released student and teacher information (Los Angeles, New York City, etc.), and have made them widely available. This data can be used to answer basic questions about student and teacher performance.

Questions

My main question; is this a good idea? Are student scores a good reflection of teacher effectiveness? There are a few other related questions I'd also like to explore:

• Are the analysis methods between school districts transferable? 
• Is the value-added model a 'good' one?
• Should parents use the teacher ratings/rankings to make decisions about where their child goes to school?
• Do these ratings/rankings say anything useful about college and work readiness, unemployment, crime, etc.?

Potential Issues

This issue is a politically charged one, and I'm concerned about getting an unbiased dataset, and that the ratings/rankings contain hidden assumptions that are not based on fact. Having this concern does not mean that I won't use certain datasets, but I will work under a trust and verify policy. 

Pulling in datasets from multiple school districts, agencies, and bureaus may cause issues of data comparability. I'm unsure of how to deal with these issues, and would appreciate any suggestions.

Data Sources

There is a wide range of raw data, and measures based on this data, available to the public;

• The L.A. Times released the Los Angeles Teacher Ratings in May 2011. Using data from the Los Angeles Unified School District they calculated the value-added to student scores on a range of measures, by teacher and school. 
• In February 2012, the New York City released a smaller set of teacher data, which The Wall Street Journal used to create teacher ratings and posted them to Grading the Teacher. The NYC Open Data site also has a wide range of education data.
• The Colorado Department of Education's SchoolView Data Center contains a variety of data on student and teacher performance. Colorado School Grades also shares this data on their site, and released a dataset on Kaggle for wider distribution and analysis.  
• For demographic information The United States Census Bureau's Data Access Tools site contains a wide variety and sources.
• The U.S. Department of Labor, Bureau of Labor Statistics, and the U.S. Department of Justice, Bureau of Justice Statistics both allow access to useful data sets.

The only student and teacher performance data that is in an accessible state is the Colorado School Grades' data from Kaggle. The data from NYC Open Data has always been fairly accessible, but with the wide variety of data types, I'm a bit unsure of their usability. At this time I am unsure if I can get access to the LAUSD data in a usable format.

Next Steps

I would like to continue researching available datasets, and from there identify the ones that seem most usable. Once those have been identified, use R to perform exploratory data analysis, and identify any useful trends. Using the ratings/rankings of different school districts on student and teacher data that has a different measure may also help identify potential flaws in each measure.

195 Posts about MOOCs

Jay Cross at the Internet Time Blog has compiled 195 posts about MOOCs. Plenty by George Siemens, who is currently teaching CN-1370-LAK2013 Learning Analytics and Knowledge 2013  (which is pretty great so far), a few by Stephen Downes, but most seem to be a bit older. With MOOCs changing, seemingly, everyday, current articles are necessary.

Building Connections - A Life Lesson Reminder

Two events today reminded me about the necessity of meeting new people, and building connections. I've been focused over the last few months on courseware development, a LMS upgrade, Google Apps for Education integration, numerous MOOCs I've been taking, and the odds and ends of helping to run online courses. Growing my network has not been a very high priority.

The first reminder was finishing Sonia Sotomayor's memoirs My Beloved World. In it she details a rough upbringing, and charts her trajectory through Princeton, Yale Law, being an ADA, private practice, volunteer work, and ends with her entering a federal judgeship. She is very frank about growing up in the Bronx, and discusses difficult and emotional issues she lived through. She is equally frank about how her network of people helped her to get where she is today. After entering Princeton it seems that opportunities opened up to her because she naturally enjoyed meeting people. She discusses how when meeting someone, she tries to learn something from them. While this is an admirable quality in and of itself, it has the added benefit of growing her network, and using it strategically when the time calls for it.

The second reminder was a presentation by Patrice Torcivia from Empire state College. A faculty member at my home institution is working with her on a grant project, studying online 'study abroad' methods. As the Instructional Designer for the courseware she will be using for the course, I was curious about some of the tools and workflow of the project. An offhanded comment between Patrice and the faculty member caught me off guard, it was something like "We met a year ago, and now we're part of this project." It reminded me that my work doesn't have to be the constant barrage of spreadsheets, schedules, trainings  development, and teaching. There is a social dynamic to this work that I'm neglecting, and should be a part of.

My mom always said that some life lessons need to be relearned on occasion. This is just one of those occasions.

MOOC Update: Slate wadding into the morass.

Will Oreums' article Online Class on How To Teach Online Classes Goes Laughably Awry on Slate is a bit too snarky for its own good. It does sum up the recent press about the course fairly well, but he does get a few things wrong;

In other words, Morrison concludes, "The honeymoon with MOOCs is over." A Twitter hashtag for the course, #foemooc, stands as a testament to the wreckage.

Most of the tweets on #foemooc are a testament to the amount or reporting about the issue, not the MOOC itself. Right after the course was canceled, there were around 30-40 tweets from students about the course. As of the writing of this, most tweets are either news organizations riding the story or Instructional Designers proclaiming the necessity of 'intelligent' (in their view) design for courses of all sizes.

The failure of a Coursera course about Coursera courses is clearly an embarrassment for company and concept alike. 

The course wasn't about Coursera courses specifically, but about online education. Sure, MOOCs fall within this, but they were not the focus of this course. This turn of phrase may be witty, but is inaccurate.

2/2 MOOC Update: FOEPA Canceled

So this is an interesting development, just received this email;

Dear Robert Weston,

We want all students to have the highest quality learning experience. For this reason, we are temporarily suspending the "Fundamentals of Online Education: Planning and Application" course in order to make improvements. We apologize for any inconvenience that this may cause. We will inform you when the course will be reoffered.

Yours, 
Fatimah

This being the first week, there were a few issues that seemed solvable. Group assignments disappearing in a shared Google Spreadsheet, discussion forums not allowing you to refine your search by sub topics, and student complaints about 'chaos' were surmountable.

The one area that remained consistent, but questionable, was the content. The 'brain based learning' paper I mentioned in my last post was suspect.