What index did you use to evaluate student performance?
We used NAEP scores — that's the National Assessment of Educational Progress. This is a standardized test of student achievement in math, language and social studies in grades 4, 8, and 12 overseen by the National Assessment Governing Board. We derived the ranking above by summing 2005 4th grade Math and Verbal scores.
Why choose NAEP scores instead of No Child Left Behind scores?
Under No Child Left Behind (NCLB), each state creates its own test — and they vary greatly from state to state. The NAEP test is completely independent of No Child Left Behind, and the same test is given in every state. Therefore, it offers a more accurate comparison of student achievement than the NCLB tests can.
How did you choose the budget profile?
The dollars-per-student index is based on state-by-state subtotals for federal funds to NCLB Act, with Impact Aid.
The subtotal excludes educational spending that goes to the state under other federal programs, such as the special education grants under the Individuals With Disabilities Act, Head Start, Pell grants to college students, and others (see the state profile PDFs linked above for a full list of excluded spending).
"Impact Aid" is federal money that goes to schools districts that educate children whose parents don’t pay local property taxes because they live on federally owned land – Indian reservations, military bases, low-rent housing and other federal properties.
What does this data tell us?
At this point, very little. Myriad factors affect the quality of education and the way money is spent from school to school within a district, and from district to district within a state. For instance, the very inspiration for the No Child Left Behind Act was the "achievement gap" between subgroups of students: rich and poor, minority and white, English Language Learners (ELL) and native speakers. Without data about a state's internal demographics, conclusions drawn about how wisely and fairly money is spent are bound to be wrong.
OK, then why is it useful?
Although this data doesn't tell a complete story on its own, it provides part of the evidence needed to back up (or call into question) government assertions about funding needs, the adequacy of a state's education system, how fairly federal education dollars are distributed, and any number of other questions about aid distribution and outcomes.
For instance, a look at the rankings shows that there is no clear relationship between statewide performance and the amount of federal spending on each child. A reporter might compare how each state spends its federal dollars with how, to whom and by whom they are allocated and used, and determine why they appear to have so little direct relationship to scores.
How did you collect all this data?
It was tricky, because NEA.org and EdTrust.org store their data in in Adobe Acrobat (PDF) format. This is has become the de facto data format for making information publicly-available and easy-to-read; but, ironically, it's a particularly difficult format to decode into mash-up-able data.
The solution we found was to use Yahoo Search. Yahoo decrypts and caches all web-accessible PDFs in HTML format (e.g. here and here.) Yahoo's fantastic search API made it simple to download the cached HTML into our custom pattern-matching code. Because HTML is an open, unencrypted plaintext data-format, we could then parse the information into a database and mash it up.
What's next?
More information: With additional data, this mash-up would have the potential to tell us a lot. A fully transparent presentation would accommodate every factor that determines the population being tested and the money that pays for that population’s education. Additional data could include: information about per capita income (which relates, at least in NCLB’s “achievement gap” terms, to performance), average district-by-district spending on education in each state, immigrant population, the presence of alternative (non-tax) sources of funding for schools, and so forth.
More code: It would be great if someone built an application to reduce the amount of custom-development and data-cleaning necessary to mash-up PDF-encoded data. This could be a stand-alone web application, a command-line utility or maybe even a plug-in for Yahoo Pipes.
Who are you people, anyway?
Leah Nelson is a reporter who lives in New York with her fiance, Jason, and puppy, Kilby. Her writing has appeared in In These Times, City Hall, the Columbia Journalism Review, and elsewhere.
Tim Jones is an itinerant freelance web technologist. Today, he's working with Ashoka: Innovators For The Public and The Electronic Frontier Foundation. His online home is Toneland.net.
Much thanks to The Sunlight Foundation's Mashup Congress Contest, the catalyst for this project.