Re-thinking the Rankings

The annual rankings of the country’s best high schools by Newsweek and U.S. News are much anticipated and debated by Bethesda-area parents. But an analysis of the rankings raises questions about their usefulness

By Sid Groeneman

Recent posting on “DC Urban Moms and Dads” blog:

I’m looking at a neighborhood in Bethesda in a [high school] district that ranks in Newsweek and not U.S. News, and trying to do my best to compare what I’m getting there in terms of schools versus other Bethesda neighborhoods.

When U.S. News & World Report’s 2008 list of “America’s Best High Schools” was released in December, Bethesda-Chevy Chase High School Principal Karen Lockard received inquiries from concerned parents. B-CC had been ranked as the nation’s 64th best high school on the 2008 Newsweek list, but it was missing from U.S. News’ top 100. One parent e-mailed: “Should I be worried?” On the school’s Listserv a parent posted the following message about the rankings: “I wonder why B-CC HS is not mentioned at all. I thought B-CC was one of the better schools in Maryland.”

Each year, Newsweek and U.S. News publish lists of what the magazines say are America’s best public high schools. And each year the rankings are the subject of much conversation—and some consternation—among parents, students, teachers and administrators. In the 2009 Newsweek rankings, released in June, four county schools (Richard Montgomery, B-CC, Thomas S. Wootton and Winston Churchill) were ranked among the nation’s top 100, with two others (Walt Whitman and Walter Johnson) narrowly missing. Three schools (Whitman, Wootton and Churchill) had turned up on the 2008 U.S. News top 100 list.

There’s no disputing the popularity of rankings. We love learning what’s considered “best”—whether it be the top-10 college football teams, the best places to live, the highest rated wines or the best films of the year. Rankings capture our interest as a reflection of our natural competitiveness as well as our curiosity. We want our team, town, company or favorite musician to be included among the most highly regarded. Bethesda-area residents—no slouches when it comes to competition—probably pay even more attention than others to who’s on top. Combine that with being preoccupied with the schools we attended—and those our children attend or will some day—and it shouldn’t be surprising the annual high school rankings generate intense local interest.

Most years, Montgomery County Public Schools (MCPS) has more schools in the top 100 of both lists than any other school system in the country. And MCPS is quick to get the word out about the rankings, often issuing a press release the same day the rankings are published. The MCPS release issued after this year’s Newsweek rankings were published quoted school board President Shirley Brandman as saying the rankings validate that the school system is “seeing the benefits of providing the academic support that allows our students to aim high and achieve at the highest levels.”

Many MCPS high school principals say that Superintendent Jerry Weast frequently mentions the rankings as a point of pride at principals’ meetings and other forums. Whitman Principal Alan Goodwin says Weast makes sure he extends individual congratulations to principals whose schools are in the top 100. Such attention from the pinnacle of the county’s education hierarchy can’t help but establish performance standards and set expectations for schools to at least maintain their strong rankings—whether or not the criteria underlying the rankings make good sense as policy.

But do the rankings really mean much? Do they really reflect which schools are the best? As someone who regularly works with statistics (and years ago helped compile survey data for U.S. News’ popular college and graduate school rankings), I feel obliged to question the validity of the methods used to rank high schools.

Newsweek ranks schools based on the Challenge Index, which was developed by Washington Post education reporter (and Bethesda resident) Jay Mathews. A school’s Challenge Index score is the number of Advanced Placement (AP), International Baccalaureate (IB) and Cambridge tests taken by all students in a school year divided by the number of graduating seniors. (AP courses are well-known; IB and Cambridge also consist of rigorous courses for which students can receive college credit. Like AP, their standardized exams are graded by outside examiners.)

By this measure, all 23 Montgomery County public high schools old enough to be included rank among the top 4 percent in the nation on the 2009 Challenge Index.

Mathews is a man on a mission. Inspired by Jaime Escalante’s success with math students at Garfield High School in East Los Angeles (a story told in Mathews’ 1988 book, Escalante: The Best Teacher in America, and made famous in the movie Stand and Deliver), Mathews’ goal is to improve students’ academic preparation, especially in lower-and middle-income neighborhood schools. His solution is to expose more students to challenging course work, and he unapologetically describes his purpose as “advocacy as well as evaluation.”

He says the Challenge Index’s key attributes make it the singular best measure of a school’s quality: It can be easily understood; it points directly to implementing positive change through rigorous course offerings; and it can be applied meaningfully to all schools—unlike quality evaluations based on traditional measures such as test scores, which, he says, are inherently biased toward schools in wealthier, upper-middle-class neighborhoods.

Mathews’ work has won many supporters locally. Walter Johnson Principal Christopher Garran says that Mathews has done a “tremendous job in helping schools open doors to AP for all students” by promoting the need to expose more students to rigorous course work. Principals Joan Benz of Churchill and Michael Doran of Wootton praise Mathews’ accessibility and his willingness to improve the Challenge Index methodology, which he has incrementally revised over the years.

Critics, including Andrew J. Rotherham of Education Sector (a Washington, D.C., think tank that promotes change in education policy), have attacked the Challenge Index for not effectively capturing what it purports to measure (school quality), in part because it doesn’t gauge student achievement, only the number of rigorous course exams taken. Mathews’ response is that exposure to rigorous material alone has proven salutary effects, and that most schools in average-income and lower-income areas would never have a chance to be ranked and recognized if test scores were the main standard. The index has also been criticized by Rotherham, and by Bob Morse, U.S. News’ director of data research, for ignoring graduation rates and achievement gaps among socioeconomic and ethnic segments.

Mathews admits feeling ambivalent about Newsweek headlining the Challenge Index rankings with “America’s Top High Schools.” He recognizes that some readers can be misled by a superficial glance at the list without taking the time to decipher what it’s based on. But he defends the practice on the grounds that “…dramatizing the index with this title is the only sure way to grab readers’ attention, without which its positive impact would be lost.”

U.S. News’ approach to ranking high schools nationally is different from Mathews’ method and far more computationally complex. To be ranked on U.S. News’ “Gold Medal List,” a school must do significantly better on standardized state English and math tests than statistically expected given its economic makeup; be in the top half of its state (approximately) in the performance of its minority students; and, among the schools remaining after these first two hurdles, achieve one of the 100 highest scores on the “College Readiness Index.” The “College Readiness” formula combines two components: the percentage of 12th-graders who had taken an AP or IB exam during or before their senior year; and the percentage who passed at least one exam—equivalent to an AP test score of 3 and an IB score of 4. (The pass rate component is weighted more heavily in the College Readiness formula.) Morse of U.S. News explains: “The first two steps are designed to ensure that the schools serve all their students well; the third assesses the degree to which schools prepare students for college-level work.” (An 18-page explanation of their methodology is available at http://tinyurl.com/d748to.)

Most of the principals I spoke with had a good grasp of the simpler Newsweek Challenge Index but a vaguer notion of the more intricate way U.S. News ranks schools. That’s partly because the Challenge Index has been around since the late 1990s, whereas U.S. News has published national high school rankings only during the past two years. Mathews also has a prominent local presence as a regular writer for The Washington Post. But some of the difference in comprehensibility probably can be traced to U.S. News’ more complex procedures. That’s evident in most principals’ inability to describe the steps followed or the criteria applied.

But greater complexity doesn’t necessarily mean that U.S. News’ method is an inferior way to rank schools. Morse, who manages the U.S. News project, cites three ways its methodology is superior: “We measure success [tests passed]—not just quantity of tests taken. We factor in how well schools do in serving economically disadvantaged students and minorities. And we recognize schools within their respective state.”

The U.S. News rankings have been attacked by education experts for intermixing highly selective “elite” schools, such as Thomas Jefferson High School for Science and Technology in Alexandria, Va.—currently their top-ranked school in the U.S.—with schools having open admissions. Morse says that U.S. News “…sees no good reason to exclude selective schools, as they are tuition free and theoretically open to all. To hide the fact that they’re among the best would deny reality.”

This “apples and oranges” criticism also applies to a lesser degree to Newsweek’s list, which includes charter and magnet program schools where at least some students must apply for admission in an academically competitive process. Unlike U.S. News, Mathews eliminates some academically elite schools, setting the cutoff at the level of the highest average SAT/ACT scores of any “normal enrollment” school in the country. This year, schools had to have an average SAT score below 1950 or an average ACT score below 29 to qualify as “non-elite.” He explains: “It would be deceptive for us to put them [schools above this threshold] on this list [because] the Challenge Index has been designed to honor schools that have done the best job in persuading average students to take college-level courses and tests. It does not work for schools that have no, or almost no, average students.”

As of 2009, schools with AP exam pass rates lower than 10 percent (schools that would have made the list in previous years) have also been eliminated from the main Newsweek ranking. Most of these schools, which are typically located in low-income neighborhoods, have recently introduced their students to academically challenging courses as a form of “shock therapy.” Such schools are now recognized by Newsweek in a separate, “Catching-Up” list. There was no other choice, since retaining them in the main Challenge Index ranking would have destroyed its credibility, Mathews says.

Mathews defends his decision to keep neighborhood schools with area-wide selective magnet programs (e.g., Richard Montgomery and Montgomery Blair) in the Challenge Index ranking, because “schools like RM have large populations of average students who are positively affected by the school’s pro-AP and pro-IB orientation.” Another consideration Mathews cites is fairness: He cannot justify removing such schools from the list when other close-by schools—such as Whitman and Churchill, which have higher average SAT scores in part because they are more economically segregated—are on it. By my count, 49 of the top 100 Newsweek schools and 61 of the top 100 in the U.S. News ranking have academically competitive admissions for at least some significant portion of their students. It seems reasonable to question which apples and oranges are fair to lump together in the same rankings basket.

Just as the Challenge Index is too limited in its concept of best schools, the U.S. News procedures try to cover too much. The crux of the problem is combining three essentially different criteria (college readiness, overcoming economic disadvantage and minimizing ethnic group disparities) into one ranking. Schools that excel in one aren’t necessarily those that excel in the other(s). Some schools that do the greatest job of preparing their minority students might not have total-school achievement scores that are among the best. Other schools characterized by superlative overall college readiness might score only slightly above average relative to their economic profile. In trying to incorporate “economic disadvantage” and the reduction of ethnic group (minority) achievement gaps together with schoolwide high achievement, the U.S. News ranking risks confounding different educational objectives. Depth and breadth of performance, and exceeding expectations, should be reported separately, rather than conjoined. Separate rankings would be easier to understand, more informative and less disputable—although perhaps less likely to help sell magazines.

Other criticisms can be lodged against both sets of rankings. For one, the criteria are heavily tilted toward college preparation. Several principals, including Whitman’s Goodwin, noted how nonacademic programs that help students succeed and are a huge part of some students’ lives (arts, music, sports, civic activities, etc.) are not examined.

Another complaint concerns disparate school size. Some schools in the upper echelon have enrollments of only a few hundred students or an even smaller number. Darryl Williams, principal at Blair, the largest high school in the county with nearly 2,700 students, says that comparing large and small schools can be misleading, as smaller schools often are in a position to specialize and can give extra personal attention to students.

For all their flaws, the rankings do serve a purpose by focusing attention on improving access to rigorous courses that are essential in preparing for college. Benz, the Churchill principal, says the rankings provide a “self-reflection opportunity.” Goodwin and Wootton’s Doran both mentioned how their high rankings set expectations and keep the pressure on for their schools to continue to out-perform.

Referring to the many qualitative features that go into making a great school, Doran says, “The rankings are measuring the brain of the school—not the heart of the school.”

Sid Groeneman is the head of Groeneman Research & Consulting (www.groene man.com), a Bethesda company specializing in opinion, policy and marketing surveys. He received a Ph.D. in political science from the University of Minnesota. His wife, Beth Groeneman, is the IB Diploma Program Coordinator at B-CC High School.