Tuesday, December 28, 2010

Standardized Testing Gone Wild

Ace Ventura: Pet Detective edition.

Mississippi had a problem born of the age of soaring student testing and digital technology. High school students taking the state’s end-of-year exams were using cellphones to text one another the answers.

So the state called in a company that turns technology against the cheats: it analyzes answer sheets by computer and flags those with so many of the same questions wrong or right that the chances of random agreement are astronomical. Copying is the almost certain explanation.

Since the company, Caveon Test Security, began working for Mississippi in 2006, cheating has declined about 70 percent, said James Mason, director of the State Department of Education’s Office of Student Assessment. “People know that if you cheat there is an extremely high chance you’re going to get caught,” Mr. Mason said.

Ain't that American ingenuity? To go hand-in-hand with the colossal, standardized testing-industrial complex, we now have an ancillary cottage industry springing up and making millions: test security.

As tests are increasingly important in education — used to determine graduation, graduate school admission and, the latest, merit pay and tenure for teachers — business has been good for Caveon, a company that uses “data forensics” to catch cheats, billing itself as the only independent test security outfit in the country.
Don't you love it? I'm sure a "CSI: Test Forensics" series is coming this fall.

However, things are not quite as they seem. As usual, when pointy heads and academics get involved and start asking questions, all hell breaks loose.
Caveon’s methods are not without critics. Walter M. Haney, a professor of education research and measurement at Boston College, said that because the company’s methods for analyzing data had not been published in scholarly literature, they were suspect.

“You just don’t know the accuracy of the methods and the extent they may yield false positives or false negatives,” said Dr. Haney, who in the 1990s pushed the Educational Testing Service, the developer of the SAT, to submit its own formulas for identifying cheats to an external review board.

David Foster, the chief executive of Caveon, said the company had not published its methods because it was too busy serving clients. But the company’s chief statistician is available to explain Caveon’s algorithms to any client who is curious.

Er, you're "too busy"? That's like me saying I've come across some great scientific discovery, only to refuse to publish or submit my data for peer review because I'm "too busy" in the classroom. "But if you're curious, I'll fill you in via email."

This company also issues cease and desist orders to websites and bloggers who dare to post information about standardized tests, such as the LSAT.

For the Law School Admission Council, which administers the LSAT four times a year to a total of more than 140,000 people, Caveon patrols the Internet looking for leaked questions on sites it calls “brain dumps,” where students who have just taken an exam discuss it openly.

“There’s all kinds of stuff on the blogs after the test trying to guess which stuff will show up in the future; there’s a whole cottage industry,” said Wendy Margolis, a spokeswoman for the council.

Caveon, which declined to reveal what it charges clients, sends letters to the people who operate those Web sites requiring them to take down the material under the Digital Millennium Copyright Act.

While referring to standardized test questions as "intellectual property" is a bit of a stretch, the billion-dollar test prep industry uses these security companies to bully and threaten (with slap suits, no doubt) anyone who dares to question their monopolies.

But even stranger are the characters running these companies.

Standardized testing is controversial with some parents and educators, but not to Dr. Fremer, Caveon’s longtime president, who recently gave up managerial duties.

More rumpled academic than business type, Dr. Fremer has an air of great confidence and interest in his own ideas. “At this stage of my life, I’m an icon,” he said without an iota of self-consciousness.

Dr. Fremer has little patience with critics who say standardized tests do not accurately measure academic prowess. “Fundamentally,” he said, “testing is a way of ascertaining what you know and don’t know and developing ranks, and the critics go right to the ranks. Well, it does rank, but on the basis of knowledge of the subject, and if you think that’s not important, there’s something improper about the way you think.”
LOL. Anyone remember "Ace Ventura: Pet Detective" from the 90's? Alrighty then!

Given that this "icon" makes his living in standardized testing, I'm not surprised to see that "critical thinking" is a foreign concept to him. Only "improper thinking" could ever entertain the idea that standardized tests measure more than just your test-taking ability.

But I digress...interestingly, this company was brought in to the Atlanta scandal and was part of what the state rejected before launching its own criminal investigation.

Hired to analyze English and math tests from Atlanta students after a state audit identified dozens of schools where cheating might have occurred, Caveon found far fewer problems. It identified a dozen elementary and middle schools at which cheating had probably taken place, but it essentially exonerated 33 others on the state’s list of suspect schools.

The Georgia governor criticized that conclusion and appointed his own investigators in August. In an interview with The Atlanta Journal-Constitution, he accused Caveon of seeking to “confine and constrain the damage” and suggested it was trying to protect its business prospects with other school districts.

Incredible. The more of this investigation that comes to light, the more convoluted the mess becomes. It's almost a question of who's zoomin' who?

But it also illustrates a point I've been making for a decade in the classroom: threaten the standardized testing-industrial complex, and you will be dealt with accordingly. From slap suits to criminal investigations to possible imprisonment, you don't mess with standardized tests.

No comments: