Why the DMN‘s DISD Coverage Kills Me Dead

This happened more than a week ago, and I’ve just been too caught up in stuff going on here [insert sound of 40s being poured] to post it. So, a quick recap is necessary. A think tank, the Brookings Institute, produced a study of big-city school districts that showed DISD was 2nd overall nationally in terms of academic gains in the past seven years. First, read this Associated Press account, which lays it out in a straightforward fashion:

DALLAS – Achievement test scores at big-city school districts in Texas still lag far behind their suburban and rural counterparts but they’re making great strides and narrowing the gap, according to a report by an education think tank released Wednesday. A study of 37 of the nation’s largest urban school systems by The Brown Center on Education Policy at the Brookings Institution in Washington, D.C., found that city schools are improving more than other school districts in their respective states. … It was designed to determine how big-city school districts fared when compared to their suburban and rural peers. The study was able to standardize scores between states, even those using different tests. … Dallas showed the biggest improvement among the large Texas cities, and was 2nd overall nationally. … In 2000, Dallas was outscored by 100 percent of the state’s school districts. By 2007, just 90 percent of suburban and rural districts did better than Dallas – a significant improvement given its demographics, the study’s author said.

Now, this is good news, right? Not great, because the district, as noted, still has a long way to go. But it suggests things are getting better. Is that your takeaway? Well, I don’t think it would be if you just read how the DMN covered it.

After the study came out, Kent Fischer at the Morning News posted this item, dissecting it. His three main points are below (I’ve numbered them, but the bold emphasis is his):

1. First, and most importantly, the report is an indicator of student achievement, but stops short of confirming it. “To make scores comparable, we computed a z-score for each city, an indicator of the distance–expressed in standard deviation units–between the city district’s test score and its state’s average score. State averages are fixed at 0.00. Positive scores are above average and negative scores are below average. This kind of relative measure means all test scores could be falling in a particular state and city schools would look good by going down less.”

Translation: A district’s ranking is not determined by whether or not achievement actually improved. It’s possible scores went down, but if the state average went down more, the district will look much better.

2. The report excludes 86 percent of district kids. “We combined data on fourth-grade reading and eighth-grade math into a single composite score for each school district. These are two crucial grades, respectively, for reading and math achievement. Although unlikely, perhaps academic achievement in other grades or subjects has behaved differently.”

Translation: DISD’s ranking is based on the scores of about 14 percent of its total student body — meaning 144,000 kids are not included in the study. Notably left out are high schools, which fail to graduate about half their students. Also left out: scores in Science, Social Studies and writing.

3. No clues given as to why. “Trends in performance are presented for 2000-2007. Big city schools have made significant improvement. Smaller urban districts have made similar gains. Unfortunately, the data cannot pinpoint reasons for the improvement, and any discussion of causes is automatically speculative.”

Translation: Although they rank districts, the researchers don’t know why, academically, one ranks ahead of another.

Now, I’m not very bright, but I know a thing or two about a thing or two. One of the things I know just enough about to be dangerous is statistical analysis. And the idea that the total number of students surveyed has anything to do with the quality of the report’s findings is absurd. The only question is whether the sample size is statistically valid. But why have me explain this in my inelegant fashion? I asked the study’s author, Tom Loveless, to respond to Fischer’s post. He said:

Yes, he’s making some poor arguments. I will address three.

1. Scores didn’t go down in Texas from 2000-2007. They went up. So the argument is moot, and the standard imposed by our study is more rigorous, not less, than simply reporting a gain. It looks at performance relative to a rising state average.

2. A composite score was constructed from 4th grade reading and 8th grade math scores.  Such a composite is representative of a district’s academic performance. It doesn’t leave anyone out. By the writer’s logic, we shouldn’t believe the Dow Jones Industrial Average as a stock market indicator because it is only composed of 30 stocks and “leaves out” several thousand stocks.

3. We can’t tell why the scores are going up or down. That’s simply a limitation of the data because we don’t have good measures of the policies that states, districts, and schools were implementing from 2000-2007. It is an honest statement of the limitations of the research.  The writer has taken exactly the wrong lesson from the study, and he gives exactly the wrong advice. Readers of research should distrust studies that make claims they cannot document with evidence, not studies that admit to their limitations.

To be fair, the story the DMN ran the next day was more balanced, I think, than Fischer’s post, in which he said he was simply putting the study in perspective (as though people who read his paper’s coverage are in danger of thinking DISD is TOO kickass these days). But one thing in the story gnawed at me. It was this quote from UCLA’s David Silver, a research expert who could comment on the study’s methodology. Silver’s points were presented in the News this way:

David Silver, a senior researcher at UCLA, said the study is “not air-tight.” But he said it’s likely the best Loveless could do without being able to follow individual students over time. But he said the study could do more to take demographic changes into account.

So I sent him a message asking him two question: one, could he take an objective look at Loveless’ responses to Fisher’s point and tell me who is right; and two, could he expand on the “not air-tight” quote? I told him I was really trying to understand whether the study had validity. Here are his responses to the first two points:

1. I agree with Loveless here (assuming that his statement about scores rising in Texas is correct). His decision to compare District scores to the statewide average is reasonable and transparent in his discussion of the methodology.

2. Again, the choice of a composite score is a reasonable decision. It does leave out students, but there is no reason to assume that the sample is misleading when averaged over a large district or state. Would it have been preferable to incorporate all grade levels into the score? Probably, but he was undoubtedly limited by the availability of data.

To the overall “air-tightness” of the report, which sort of gets to No. 3, he says two things:

Essentially, Loveless found that the average student in large urban districts in 2007 was closer to the statewide average than he/she was in 2000. I’m fairly comfortable with his methodology to make that specific statement. The two main caveats are: 1) that the rankings of districts/states should be used only as a rough guide, since the characteristics of the tests may be very different across states; and 2) it is very likely that the demographics of many districts (e.g., Long Beach, San Francisco, Los Angeles, Las Vegas, New York) changed considerably relative to surrounding areas over these “real estate bubble” years, as large numbers of low- and moderate-income families moved to more affordable districts outside of the big cities.

He then sent along this follow-up:

One other point. The “air-tight” quote came from this statement that I made:

“I think Loveless makes an interesting case. As I said, the case is not air-tight, but given the limitations of the data (namely, that he couldn’t follow individual students over time), he probably did the best he could. Hopefully, we’ll have definitive confirmation of these findings in the near future, now that California (and Texas?), among others, have finally begun tracking students with longitudinal Statewide Student Identifiers.”

So a better phrase might have been that “his methodology for assessing change over time and ranking districts is not perfect, but given the limitations of the data . . .”

In the end, no harm, right? Maybe Fischer was just viewing this news in the harshest, most skeptical light possible, and that’s his job, right? Well, sure, maybe. I just wish a fuller picture of the district could be told, because I know there are also fascinating success stories out there, and when a small measure of success is noted, I wish he wouldn’t see it as his job to post a poorly reasoned takedown of said success. Then I wouldn’t get so tired and bent out of shape when I read a story that, while technically accurate, misses the point. It makes me believe even more strongly that the paper and the TV stations in town view DISD as a wounded antelope whose entrails are to be feasted on because, hey, lions gotta hunt.

Newsletter

Get a weekly recap in your inbox every Sunday of our best stories from the week plus a primer for the days ahead.

Find It

Search our directories for...

Restaurants

Restaurants

Bars

Bars

Events

Events

Attractions

Attractions

View All

View All

Comments

16 responses to “Why the DMN‘s DISD Coverage Kills Me Dead”

  1. GuiltyBystander says:

    A-gen-da.

    DMN’s bosses and editors have their kids in private schools or attend Highland Park or live in Plano. They consider DISD inferior and comical. If their reporting was balanced, at least the criticisms leveled could be taken at face value. But the DMN is far from fair and balanced when it comes to DISD.

    And Eric: But don’t call the DMN a lion. Maybe a jackal, but not a lion.

  2. Daniel says:

    You mean you actually pour a 40 into a glass?

    Very elegant — er, “bespoke” — of you.

  3. Former DISD Teacher says:

    While not exactly on topic, I take any opportunity to make a point that no one seems to get:

    The most significant reason only half of DISD’s high school students graduate is that so many are undocumented. What motivation do they have to graduate from high school knowing that without a social security number they can never get a legitimate job?

  4. Gadfly says:

    I didn’t understand the forties reference, so I did some minor research and came up with something very interesting.

    http://en.wikipedia.org/wiki/Edward_Forty_Hands

  5. Matt says:

    But isn’t the real lesson that if you want your kid to get a good education, 90% of the suburban districts will do it better than DISD? Sure, recognize that DISD is “making great strides”. But if you were purchasing any other service, would you choose the guy that bragged that he improved from “the worst choice in town” to the “as good as any in the bottom 10%”?

  6. julie says:

    Conservatism’s goal re: education is to CLOSE public schools. Period. That was the point of the evil “No Child Left Behind” law–it set very high goals with NO financial support. Schools that didn’t meet goals were closed.

    Conservatives hate anything that might benefit the society at large. That includes free public schools, college loans, county hospitals, free highways, etc etc etc.

    Conservatives would make all schools private–and then maybe give you a “voucher” to some such bs to help pay for schools. Anyone who cannot see that you’d get a 30cent voucher for each dollar of tuition cost is naive.

    Fortunately, America seems to have finally figured this out..hence, the results of the recent election.

    I’m not saying our schools are great–they’re not. But as the report above shows, they CAN be improved.

    Former DISD teacher: so how about we just BAR undocumented students from school? I mean, instead of trying to educate these children who are here and part of our society so that they one day just might rise above their current situation, let’s just turn them away & keep them uneducated. Sorry, America is better than that.

  7. Former DISD Teacher says:

    Julie, I did not make myself clear. I want EVERYBODY educated! I just found I had a hard time answering my students when they asked, “But Miss, what difference does it make if I graduate? I can’t get a job anyway.”

  8. Lynn says:

    Just proves the old adage: statistics are like a bikini–they reveal a great deal while covering some important points!

  9. Former DISD Teacher says:

    Also Julie, we’re not talking about “rising above” a situation. How do you propose these kids get a social security number? There is no job you can get in this country without one. It’s not about rising above. . . But let’s not get into immigration. That’s just not what the post is about.

  10. julie says:

    I’m in favor of providing a path to citizenship to undocumented workers/families.

    OK, I expect a lot of flying tomatoes to come my way now.

    I’m also in favor of stronger, less porous borders, if that helps any of you whose blood pressures are rising.

    A combination of a realistic, attainable path to citizenship combined with good, free education will do our society the most good.

    Because you know I’m all about the common good.

  11. R says:

    The 40 reference in the post refers to honoring a fallen comrade. It seems to have originated with rap music people who, at the funeral of a dead friend would pour out a 40 oz. bottle of Malt Liquor on the ground in honor of their friend.

    Jeez I feel white. I shall now attempt to explain Facebook to my mother.

  12. BF says:

    Hey, the DMN does whatever it can to generate readership, particularly when ad revenues are becoming harder to come by. It seems to become more tabloid by the day. More TO coverage and Jerry speculation, less about sports (ask Evan Grant about it). Gloomy real estate articles above the fold to scare the masses into picking up the paper. So a DISD bashing does the same thing.

    But for the record, Matt’s comment is spot on. Yes, it’s improvement, but let’s not get carried away – DISD is still near the bottom of the steaming heap.

  13. (Shakes head, dumbfounded) how did Kent avoid the layoffs – must be TDMN thinks he’s right “on point”.

  14. ernest t bass says:

    “How do you propose these kids get a social security number? There is no job you can get in this country without one.”

    They pass out Social Security cards like candy on Harry Hines Blvd.

    And employers say “now git to work, I’ve done my due diligence.” Then the business owners become wealthier and rail against every form of taxation.

    What a country!

  15. Louisa Meyer says:

    But Kent Fischer and DMN do support sampling and in its smallest form: just one thief, pervert, incompetent or over age student = front page story of school district in crisis followed by “in depth” series, columnists commentary and blognauseam.

  16. LakeWWWooder says:

    When Newsweek named White, Woodrow and Hillcrest to its top high schools in the USA list (TAG and SEM were at the top) TDMN wouldn’t print it and the columnists started to criticize the formula.

    When 103 schools were ranked exemplary or recognized we were treated to a long story explaining how that happened because some schools got waivers.

    When Lakewood Elementary was named a Blue Ribbon school we were told that such honors are really not that special.

    When the parents of Woodrow Wilson High started a campaign to get parents to look fairly at choosing the school over a private school “Choose Woodrow”, the entire community was raked over the coals and the white parents who send their kids to a 75% minority school were basically called racist.

    I could go on, I’ve got 35 years of examples.