Analysis of the Harvard Charter School Study

Beth Zirbes and her daughter Sonia

Beth Zirbes, Fairbanks High School Mathematics and Statistics Teacher

Ms. Zirbes has a bachelor’s degree in Mathematics from Gustavus Adolphus College in St. Peter, MN and Master’s degrees in both Mathematics and Statistics from UAF. During her maternity leave this spring she applied some of her skills to compare the performance of neighborhood and charter schools in response to the much-touted Harvard Charter School Study.


When I first saw the results of the Harvard study concerning charter schools I was simultaneously unsurprised and skeptical. I was unsurprised as I have seen many very bright young students in my AP classes come from charter schools. I was skeptical as I suspected much of this performance could be attributed to the type of student who attends Alaska’s charter schools. As a comparative analysis of Alaska’s charter schools and neighborhood schools had not been done, I set out to do one myself.

To determine whether charter schools outperform neighborhood schools I looked at the performance of all schools on the 2018-2019 PEAKS ELA (English Language Arts) assessment from the Alaska Department of Early Childhood Education and Development’s (DEED) report card to the public, as this year was within the same time frame as the data from the Harvard study. The performance of each school is given under the “2018-2019 Performance Evaluation for Alaska’s Schools (PEAKS)” tab. I used the data on this page for every school in the state which allowed me to analyze test scores and demographic characteristics such as the proportion of the school who are economically disadvantaged, English language learners (ELL), and special education (SPED). Demographic characteristics are only given for the set of test-takers and thus all summaries and analyses are for students in grades 3-9 during the 2018-2019 school year. I also removed all correspondence schools from my dataset as these students were not included in the Harvard study, do not take NAEPs tests, and have very low participation rates on state tests. For my analysis on performance, I also restricted my dataset to include schools only in districts where charters are an option to ensure that the student populations were as similar as possible.

At first glance, it appears that charter schools are more successful than neighborhood schools. At charter schools 52.5% (1,866 out of 3,554) of students were proficient on the ELA assessment versus 40.1% (18,655 out of 46,574) of students at neighborhood schools. However, these differences could be explained by the differences in demographics of the student bodies at these schools. To rule this out as a potential issue, statisticians control for these variables in their mathematical models. We can then ask, do the charter schools outperform neighborhood schools that have similar characteristics? Or do charter schools do any better than we would expect, given their student populations? In short, the answer is no, they do not. When I fit a model which controlled for socioeconomic status alone, the type of school (charter versus neighborhood) was not significant. (For anyone who knows statistics, the p-value associated with type of school was 0.57. It wasn’t even close.) In summary, there is no evidence that charter schools outperform neighborhood schools in terms of ELA proficiency once we consider their socioeconomic make-up.

During my data exploration, I discovered that charter schools, on average, have very different student bodies than neighborhood schools. Charter schools have far fewer economically disadvantaged students, far fewer ELL students, and are comparable to neighborhood schools in terms of SPED populations. Here is a summary of how these populations differ for all Alaskan students in the relevant grades in all of Alaska’s brick and mortar schools for the 2018-2019 school year:

  •  Neighborhood schools were 52.2% economically disadvantaged (30,780 out of 58,929 students) compared to 31.3% in charter schools (1,219 out of 3895 students).
  • Neighborhood schools were 15.5% ELL (9,150 out of 58,929 students) compared to 9.3% in charter schools (363 out of 3895 students).
  • Neighborhood schools were 16.3% SPED (9,162 out of 58,929 students) compared to 13.7% in charter schools (532 out of 3895 students).

However, these summaries are highly influenced by a few outliers and obscure some large discrepancies, especially in terms of the economically disadvantaged and ELL students.

  • Of the charter schools, 46.4% (13 out of 28) have economically disadvantaged rates below 20%, compared to just 3.5% of neighborhood schools (15 out of 426).
  • Only 10.7% of charter schools (3 out of 28) have ELL percentages above 10%, compared to 36.9% of neighborhood schools (157 out of 426).

Much of the local reporting on the results of the Harvard study focuses on how Alaska’s charter schools rank at the top of the rankings presented in the paper but at the bottom with regards to NAEP rankings, which include all schools (but not correspondence schools). It is important to note that the Harvard charter school rankings and the rankings presented by NAEP are not comparable. The Harvard charter school study ranked charter schools of each state while controlling for various factors, such as socioeconomic status along with many other characteristics. Comparing the rankings presented in the Harvard paper to those reported by NEAP is then meaningless, as the NAEP rankings do not control for any of these factors. If one wanted to compare the performance of charter schools to neighborhood schools state by state, one would need to expand the dataset of the Harvard study to include all neighborhood schools in those states and then apply the methodology presented in the Harvard paper to determine the rankings.

Even if we did a comparison of charter schools and neighborhood schools and found that charter schools did better, we still cannot conclude charter schools are causing the performance difference we observe. A comparative study like this is an example of an observational study and because it is impossible to control for all confounding factors, such as parental involvement, we can’t conclude success is caused by the school type. To definitively conclude that charter schools were causing the observed difference in success compared to neighborhood schools we would have to randomly assign some students to go to a charter school and some students to go to neighborhood schools. After some time, we would then compare the results. Obviously, this is impractical as many charter schools do not have busing, require volunteer hours, can remove students for poor attendance, and some do not even have lunch services.

Before the state uses the results of the Harvard study to change the approval process for charter schools we need to understand if charters are better and, if so, why. So far, I have not seen convincing evidence that charter schools outperform neighborhood schools when we control for various student characteristics. I have an idea for further study which I believe should be completed before any changes to policy are made. We can examine the performance of students who got admitted to charter schools via the lottery to those who applied but did not get in and attended their neighborhood schools instead. The group who was admitted is likely similar to those who applied but were not. This would be as close to a randomized experiment as one could hope to have. From this one could determine whether various factors were causing differences in performance, such as class size and teaching methodology. Additionally, we could use results of such a study to determine which factors correlate with success and apply these strategies in all our schools. As more than 90% of the students at our in-person schools are in neighborhood schools, such reforms will be more wide-reaching than simply adding a few more charter schools.