Must Read Drop-Out Study

Senators Ted Kennedy (D-MA), Richard Burr (R-NC), and Jeff Bingaman (D-NM) yesterday introduced the Graduation Promise Act (GPA), which is being promoted by a coalition including the Center for American Progress, Jobs For the Future, Alliance for Excellent Education and The National Council of La Raza.

But before anybody moves on graduation rate policy everyone ought to all read this AMAZING! literature review on reducing drop-outs (with at a glance summary here), just published online by the Center for Public Education, and authored by education consultant Craig Jerald, who, if you don't know his work, is a man to watch in ed policy.

Though it was released completely without fanfare, the review shows that there is a lot more knowledge (including rigorous experimental studies) on preventing drop-outs than most people think. Jerald must have a vault of research studies in his basement because I've never seen a bibliography so diverse or comprehensive. From it he draws valuable and myth-busting insights like:

1) What schools do matters, even for the most at-risk students- in a randomized study using Check and Connect in Philadelphia, schools cut 4 year drop-out rates for high-risk students (low-income African-American males with disabilities and single-parent families) by 1/3rd and 5 year rates by 1/2.

2) Decreasing drop-out rates does not automatically increase graduation rates; in one Check & Connect study 4 year drop-out rates fell from 58 to 39 percent, while 4 year grad rates were up only 1 point (30% vs. 29%). To be effective programs have to address both monitoring/counseling and instruction.

3) Easy = boring - several studies found that making classes more challenging classes helped kids graduate and similarly unchallenging classes led to higher drop-out rates.

4) For programs to work they have to be intensive - low intensity, counseling, tutoring, or self-esteem workshops all had no effect on drop-out rates; successful programs essentially involved a caring, committed adult monitoring student attendance and grades almost daily, coaching students in academic and life skills, and intervening immediately if they start to fall off; as you can expect this kind of intensive approach can be expensive.

This is just the tip of the useful-info-iceberg and I highly encourage you to invest the time to read the full lit review and not just the at-a-glance summary. Coming soon, what I think this all means for grad rate improvement policy...

The Angel in the Details - Performance Pay that Works

Though everyone's sure to be blogging about this one, I couldn't help but jump on the bandwagon to mention something so worthy of enthusiasm. The Center for Teacher Quality has just come out with a report on teacher pay, which among other things advocates a salary range from $30K - $130K for teachers, incentives to work in high-poverty schools, performance pay for individual teachers and small groups of teachers who work collaboratively, and creating a career ladder with novice, professional, and expert designations for teachers----with corresponding levels of pay, responsibility, and authority. The study was led by an diverse group of 18 effective teachers, giving its conclusions that much more legitimacy.

What's great about this study is not just that it advocates things like performance pay and challenge pay--which are easy to advocate, but harder to implement--but that it puts enough nuance into its formulation of these ideas to offer some usable solutions. For example, one of the most frequent criticisms of performance pay is that it sets teachers in competition with each other and undermines the collaboration and teamwork necessary to good teaching. Though this argument can be overblown, some common, but ill-conceived formulations of performance pay, such as Florida's recent attempts, do produce irrational results. Florida offered performance bonuses to the top 25% of teachers in every school. Certainly this does de-incentivize the sharing of instructional know-how that would help fellow teachers advance to top performance, because if you help others rise they might surpass you and jeopardize your bonus. Likewise the Florida plan simultaneously rewards tons of bad teachers--even if every teacher is terrible in a school someone is definitionally going to be in the top 25%--and neglects very good teachers who teach in schools full of very good and some truly outstanding teachers.

And regardless of whether these problems turn out to be big or small, the perception among teachers that performance pay is against, not for, them is as much of problem as are the purported consequences of such programs on teacher collaboration and the inaccurracy of the program in recognizing success. The point of performance bonuses is to motivate teachers to perform better, recognize teachers who do, and retain quality teachers in the school system. In other words we want happy, hard-working teachers. If even great teachers distrust performance pay because of the way its designed, the program will be fail to motivate or retain. And if the average teacher, who is neither amazing nor incompetent, is pissed off about it you are not helping motivation or retention among the vast majority of teachers.

A typical approach some have proposed to these problems is school-wide bonuses for student achievement. But this idea just sets up a typical prisoner's dilemma system of incentives. If I work hard and no one else does I get no bonus. If everyone else works hard and I don't I still get a reward. So what incentive (other than the intrinsic ones, which existed prior to the peformanc pay program) would motivate me to work any harder? Collective accountability rarely produces incentives for individuals. Also, schoolwide bonuses often include all staff, administrators, secretaries, janitors etc. While I have any objection to paying the lunch lady well, but no amount you put in her pocket is going to raise student achievement.

But the CTQ recommendation heads objections off at the pass. By allowing all teachers to be eligible to receive bonuses based on absolute not relative performance it undercuts the harmful competition argument. And by offering bonuses for collaborative teams of teachers based on the performance of the students they teach together, it actually incentivizes the sharing of best practices and mutual support that opponents say peformance pay undermines. And pay for small teams minimizes if not eliminates the collective action problems associated with school-wide programs. As astute students of collective action know small groups create a regularized flow of communication about participation in group activities and relationship-based accountability for group success. Lastly, small-groups provide opportunities for leadership among a much broader range of school staff, and most importantly among staff who are still anchored in the classroom unlike most school administrators.

A combination of universal eligibility-bonuses for individuals and small, collaborative groups is simple elegant solution to one element of the problem of building teacher buy-in for performance pay. Read the full report for a treasure trove of such ideas. Kudos.

Can Schools Save Drop-Outs?

Some Discouraging Stats from Philly:

Likelihood of dropping out given certain life problems:

-Abused or neglected: 71.3%
-In Foster Care: 75.2%
-Juvenile Justice placement: 90.1%
-Gave birth during high school: 68.3%

-Number of high school age children with these problems in Philadelphia: 13,393

Source: Unfulfilled Promise: The Dimensions and Characteristics of Philadelphia's Dropout Crisis, 2000-2005

Some Encouraging Stats From Philly:

Effects of the "Check and Connect" identification and intervention program on a high-risk population in Philadelphia:
-No intervention (control group) Drop-out Rate (5 year): 94%
-Treatment Group Drop-out Rate (5 year): 42%
-Improvement due to program: Decrease 5 year drop-out rate by 1/2

-Effects of "Career Academies" alternative schools program on a high-risk population in Philadelphia:
-No intervention (control group) Drop-out Rate (4 year): 32%
-Treatment Group Drop-out Rate (4 year): 21%
-Improvement due to program: Decrease 4 year drop-out rate by 1/3

Source: "Keeping kids in school: Lessons from research about preventing dropouts" by Craig Jerald. Research summary available through the Center for Public Education. (White paper forthcoming).

Some food for thought From Philly:

-Cost per student of the "Check and Connect" program: $1600 per year
-Number of students in need of such programs in Philadelphia School District: ~85,000 to 90,000
-Cost for all Philadelphia's drop-outs of the "Check and Connect" program: ~$135 - $145 million
-Percent of students nationwide who fail to graduate: ~25% - 33%
-Number of students nationwide who fail to graduate: ~12 - 16 million
-Cost of check and connect nationwide: ~$19 - $25 billion
-Current federal Title I expenditures: 12.7 billion
-Increase necessary to save these students: Doubling federal expenditures

Take home message:

It can be done, but it's going to cost us.

More on this later.

A Question of Scale: Class-size Reduction and America's Misplaced Priorities

Everyday Americans and politicians alike think class-size reduction is a key to any plan to improve education in America. I used to agree. Then I ran the numbers. I've since come to the conclusion that class-size reduction is a $40 billion mistake. Allow me to explain.

Though much research has shown (and common sense confirms) that teacher quality is the key variable when it comes to student improvement, teacher quality is hard to measure. And without accurate measurement, it is impossible to compare the impact of the various known components of teacher quality to the impact of other seemingly helpful interventions such as reducing class-size, instituting after-school programs, hiring additional school counselors etc. Lacking a scale for comparison we can't evaluate financial trade-offs and politicians are likely to go for popular, feel good programs of which class-size reduction is the American favorite. But new research has given us exactly the tools we need to make precise comparisons and the financial cost-benefit analysis this research makes possible is simply damning for class-size reduction.

The best source of this information is a new study called "How and Why Do Teacher Credentials Matter for Achievement" (Clotfelter, Ladd, and Vigdor 2007). Using value-added methodology*, this study took 10 years of data for every student and teacher in the state of North Carolina and used it to analyze the impact of various factors -- including teacher experience and credentials, student socio-economic background, and class-size -- on student achievement. Though the study was not new in conception, its massive data set suggests that its results may be the most reliable to date.

Of particular interest are the following results (for math achievement**):

Input	Increase in student achievement
The first 3 to 5 years of teacher experience:	7.2% - 9.1% of a standard deviation
(vs. a brand new teacher)
Teacher having a regular teaching license:	3.3% - 5.9% SD
(vs. having an emergency license)
Teacher same race as student:	2.0% - 2.9% SD
-All additional years of teacher experience:	2.0% - 2.8% SD
(beyond 5 up to 27)
Teacher is National Board certified:	2.0% - 2.8% SD
Teacher scored 1 SD above average on	1.1% - 1.5% SD
a teaching licensure exam:***
Reducing class-size by 5 students per teacher:	1.0% - 2.5% SD
Teacher attended a competitive college:	0.7% - 1.0% SD
(vs. attending an uncompetitive college)
Reducing class-size by 1 student per teacher:	0.2% - 0.5% SD
Teacher has an advanced degree:	–0.3% - 0.2% SD
(usually MA)

                                                                             

The numbers above represent the amount of improvement that students experienced when given the input listed. It's very important to note that the results above are measured in percent of a standard deviation, not in percent of a test score. If you aren't statistically inclined enough to interpret what standard deviations mean, don't worry; the percentages above still tell us the relative impact of these factors, which is the crucial factor when comparing trade-offs.

Comparing those relative factors you'll notice several things. First of all, teacher experience and licensure rank near the top of the list while class size reduction and teachers having masters’ degrees rank at the bottom. But more important than the ranking is how much smaller of an impact class-size reduction has than some of these other factors. For example, having a teacher who is not a novice (7.2% - 9.1% SD) exerts an influence 3½ to 7 times greater than the impact of reducing class size by 5 students (1.0% - 2.5% SD).

But the really mind-blowing results come when you start comparing class-size reduction to giving students a teacher with a reasonably good (though not unlikely) combination of teacher credentials (estimated by adding the relevant values in the table above). Clotfelter and Ladd do their own estimates of this sort and come up with a combined effect size of 15% - 20% SD for math (and 8% -12% SD for reading) of a well-credentialed teacher.

The first time I read this portion of the study I said to myself, "Yes, class size is less important,” but that finding is not particularly novel to anyone who follows this sort of research. However when I decided to actually compare how much less important class size is my jaw dropped. The effect size of teacher credentials is 8 to 10 times that of a major class size reduction in math and 6 to 8 times as big in reading.

To really understand the importance of these differences in scale you need to look at a few statistics in order to put a dollar amount next to a policy choice. America currently has a nationwide student to teacher ratio of about 16 to 1 (roughly 50 million public school students divided by roughly 3.1 million teachers). The average teacher salary in America currently floats around $48,000, which means we are spending roughly $150 billion dollars a year on teacher salaries. Cutting class size in half would require doubling the number of teachers and therefore doubling the amount we spend on teacher salaries, putting the figure at around $300 billion. (This does not even account for the increased costs of benefits, additional classrooms, and other factors needed to enact such a decrease in class size.)

The next important question is, "What would we get for such a monumental expenditure?" Based on the results in North Carolina we can estimate. If class size reduction has a linear effect-size (granted an assumption worth exploring) cutting it in half (i.e. reducing it by 8 students per teacher nationwide) would achieve an effect size in math eight times that of reducing it by one student per teacher, in other words 1.6% to 4% SD. To give you a sense of the meaning of that effect size, it is in the same ballpark as giving students a teacher who is of the same race (about 2% - 3% SD). The former intervention would cost nationwide about $150 billion dollars (or better than half of the $250 billion that all the states combined spend on education) and the other would cost roughly nothing.

The analysis so far has not compared the impact of class-size reduction to the impact of teacher effectiveness differences that have nothing to do with credentials. Put simply, quite a lot of research has shown that there is a huge difference in the performance of teachers that is not attributable to differences in credentials. (See for example the seminal "Teacher Effects on Longitudinal Student Achievement" Jordan, Mendro, and Weerasinghe 1997 or more recently Kane, Rockoff and Staiger's work published in simplified version at "Photo Finish" in Education Next and available at http://www.hoover.org/publications/ednext/4612527.html).

It's not possible to directly compare the effect sizes from the North Carolina data set to those in these other studies, but what these other studies have shown is that in much the same way that teacher credentials are an order of magnitude more important than class size, teacher talent is an order of magnitude more important than teacher credentials. This subject is actually worthy of another post entirely. The simple upshot is that if a direct comparison shows that the cost-benefit ratios of reducing class size and improving teacher credentials are in different categories, the cost-benefit ratios of reducing class size compared to increasing the concentration of teaching talent are on different planets.

So how about instead of setting a goal of halving class size, requiring us to spend $150 billion annually, we instead set a goal of doubling teacher pay, so that average is around $100 K and simultaneously impose a ruthlessly competitive tenure process where districts only offer tenure to teachers with dramatically high value-added?

For those who think that all this discussion of halving class size (i.e. cutting it by 8 students per teacher) is a crazy hypothetical, it's worth looking at the trend over the past 30 to 40 years. Since 1970 class size has gone down in America by about 6 pupils per student, a roughly 27% decrease. There is no indication that teacher quality has gone up by 27% or even 2.7% in that same time period.

Judging by the education news media, I must be the only person running these numbers. Every other news article, position statement by a politician, teacher blog, or opinion on the street, assumes that reducing class size is a high priority and should be done whenever possible. But it is quite simply the most misplaced priority out there and no one seems to be drawing any attention to this fact. The mild experiments in different versions of performance pay, about which there have been such intense fights, represent just peanuts compared to the money that has been sunk, and that prevailing sentiment proposes to keep sinking, into class-size reduction.

Of course increasing the teacher talent pool would require recruiting a large number of new teachers. So here's a proposal of what we could do with the money we would have saved from not decreasing class-size. How about completely free college for anyone who meets a high academic threshold (3.5 GPA, 700 or higher Math SAT) and commits to majoring in math or science and then teaching math or science in a low-income school for 4 years after college? (I mention these hypothetical credentials based on the 7% of a SD increase you get from having a teacher with licensure test scores not 1, but 2 SD’s above the mean). This approach to teacher recruitment would also help 100,000 students a year with college access and in particular could offer high-achieving low-income students a powerful opportunity to attend college while serving the communities from which they come. (I imagine this might have the side effect of improving the student-teacher racial matching mentioned above, which was measured as having an effect size similar to that of halving class size.)

If we assume a good state college costs $20,000 a year (tuition, room, board, books, everything) for four years and say we had 100,000 takers for our national program each year, it would only cost us $8 billion for each cohort (or roughly half of current Title I costs). If on the other hand we spent the same $8 billion reducing class size we could hire about 160,000 new teachers and achieve a nationwide decrease in class size of about 0.8 pupils per teacher.**** Based on Clotfelter, Ladd, and Vigdor’s numbers the latter intervention would likely produce a nationwide effect size of less than 0.2% to 0.5% of a SD; (0.2% – 0.5% of SD would occur if we achieved a full 1 pupil reduction in class size). Not 2% - 5% SD, but 0.2% - 0.5% SD. For 8 billion dollars.

As I mentioned above, states and school districts have actually enacted this horrendously inefficient second intervention. If we simply undid the 6 pupil per teacher, nationwide class-size decrease of the past thirty years, which would require laying off roughly 830,000 teachers nationwide (about 27%), and put us back to our previous roughly 22 to 1 ratio (not terrible), we would have an additional $40 billion or so a year to spend (830,000 teachers times $48,000 per year in salaries). If instead of teacher recruitment through financial aid we wanted to use this money to increase teacher salary, it would allow us to raise salaries for the remaining two and a quarter million teachers by almost $18,000 a year across the board. That's a raise that people considering teaching would notice.

If districts wanted to be more strategic than proposed above, we could raise salaries for 75% of those teachers (about 1.7 million of them) by $10,000 across the board ($17 billion total) and raise salaries for the top quartile (roughly 560,000 teachers) by approximately $40,000 per teacher ($23 billion total). Frankly, I would be for this even if top quartile was determined not by value-added, but by peer-review and principal evaluations since I believe we'd still be getting enough overlap between actually good teachers and people being highly compensated. We could then have those $100K a year teachers that so many of us dream of (or dream of being) and not just a few of them.

Maybe we could do all this without laying off any teachers if we just don’t replace the almost 1 million who are nearing retirement. But to do that we’d have to convince all of America to give up our most beloved, and misguided, policy intervention. We'd have to convince Americans that the bottom line in improving student achievement is teacher quality not teacher quantity. We'd have to convince them that there is a real trade-off being made and that every time you spend money reducing class-size you are not spending it producing, recruiting and retaining effective teachers. We’d have to convince them that bigger classes are actually better for education.

--Dewey

*Value-added methodology is a way of evaluating student-progress as opposed to just absolute test scores. To radically oversimplify, if a teacher takes a student who is scoring about a 60 on an exam at the beginning of a year and teaches that student to the level where she scores an 80 on the same (or a very similar) exam by the end of the year that teacher has added value of 20 points. If another teacher took a student who was scoring 85 and took her to scoring 90 that teacher added 5 points. Even though the second teacher's student scored significantly higher, we would say the first teacher did a better job helping her student grow, in fact a dramatically better job. This is a very gross oversimplification of value-added, but it conveys the basic idea. Most value-added models take into account various other factors besides just the student's initial test score, in order to not hold teachers accountable for factors outside of their control that differentiate their students from the students of other teachers.

**The fact that these scores are for math, not reading is actually quite important. In general the impact that school-based factors have on reading tests scores is significantly lower than the impact that school-based factors have on math scores and conversely the impact of non-school factors such as parental education levels is much bigger for reading than for math. Given that parents typically interact with their children more through language than through math, it makes intuitive sense that parental (and peer or community) impact on language skills would be larger than parental impact on math skills. The relevant effect sizes of class-size reduction on reading achievement are 1.0% - 2.0% SD for a reduction of 5 students per teacher.

***Scoring one standard deviation higher than average would mean you scored in roughly the top 16% of test takers. Scoring two standard deviations higher than average would mean you scored in the top 2% of test takers.

****Figures come from the following rough calculation: Current ratio: 50 million current public school students divided by 3.1 million current public school teachers = 16.1 students per teacher. Ratio after spending $8 billion on a class-size reduction initiative: 50 million current students divided by 3.26 million teachers = 15.3 students per teacher. Difference: 16.1 – 15.3 = 0.8 student per teacher reduction. This assumes that the new Math and Science corps replaces retiring teachers as opposed to being added to the current number of teachers. 

Post-script:

The one thing that this analysis doesn't address is the potential of non-linear effect sizes for class-size reduction. Clotfelter et al. actually found striking non-linearity in the effect size for increased teacher test scores with a teacher scoring 1 SD better than average producing barely 1% SD increase in student achievement, but a teacher scoring 2 SD better than average on the licensure test producing a whopping 7% SD increases. Frankly, non-linear effect sizes for class-size seem more likely than not. At the extremes it is obvious that teaching 40 elementary age children with even mild discipline issues starts to become ridiculous, and teaching a class of 4 children is essentially a form of tutoring. At some point in the future I'm going to come back to this issue. I'll just say for the moment that the kind of changes in class-size discussed in this paper i.e. changes of 6 pupils per teacher, are not creating these extreme cases in which we would predict severe non-linearity. More on this in time to come.