Blogger Perdido Street School reports that 45 percent of the teacher ratings for the public schools of Buffalo, New York, were inaccurate. The ratings were outsourced to a company in Utah that acknowledged its errors. Its miscalculations resulted in 1,089 teachers receiving lower ratings than they should have.
The blogger quotes the story in the Buffalo News:
Lower-than-correct scores were given to educators who teach more than one grade level or subject and are required to meet multiple sets of student learning objectives. The company had rewritten its scoring calculations over the summer to enable it to produce scores more rapidly, Rosenthal said. But in doing so, it inadvertently created the calculation error for this group of teachers.
The district’s data chief, Genelle Morris, said a teacher brought the error to the district’s attention. According to the teacher’s manual calculations, she had met her performance targets, but that was not reflected in the online calculations produced by Truenorthlogic. The district checked her calculations and ran them internally through the district’s Information Technology Department and found it could not replicate Truenorthlogic’s scores.
The company soon uncovered the source of the error and the corrected results were posted online for teachers to view late Thursday morning.
Perdido Street adds this observation about the Governor who insisted on creating the evaluation system (APPR) and who insists it is “scientific” and “objective”:
It will be fun to see Cuomo twist himself into a pretzel to defend APPR even as the Common Core/Endless Testing edifice comes down around him.
Make no mistake – he will do just that.
APPR is his baby – his donors wanted it, he pushed for it, when pushback came, he fought to impose it on the state through the budget.
He will not want to admit it is as flawed and error-riddled as the Common Core implementation (which he can blame on NYSED), the Common Core tests (which he can blame on Pearson) or the Common Core curriculum (which he can blame on NYSED.)
When it comes to APPR, he has no one to blame but himself.
Diane Ravitch's blog 
“The district’s data chief, Genelle Morris, said a teacher brought the error to the district’s attention. According to the teacher’s manual calculations, she had met her performance targets, but that was not reflected in the online calculations produced by Truenorthlogic.”
Yet another indication that teachers are smarter than the data monkeys “evaluating’ them.
That Utah company should lose their contract, by the way.
LikeLike
Terrific. Utah teachers start getting test-based evaluations this year, and I’m sure that this company will be the one doing them. Wonderful. We already have to register for PD through truenorthlogic, and the registration process truly sucks.
LikeLike
Not to worry – Utah probably outsourced them to a firm in Buffalo . . .
LikeLike
TAGO!
LikeLike
Let’s just round it up to a 50% error factor. Then ask yourself how happy some of the bozos who are mandating this BS would be if their retirement packages were “outsourced” and there was a 45-50% error…not in their favor that was discovered a year from their retirement date.
LikeLike
It’s not just errors in calculation that are worrisome.
Hardly. At least they can be fixed (if some smart teacher points out the errors)
But the standard errors of VAM scores are often large relative to the score differences used to (supposedly) distinguish between “good” and “poor” teachers. (see this article, for example)
This stuff is all basically junk science and the people pushing it are crackpots — data monkeys who are too stupid and/or too dishonest to get jobs doing honest statistics.
LikeLike
The APPR is a farce . My score is incorrect too but in my favor. This is a very strange world.
LikeLike
We need a new term for error on top of invalidity.
LikeLike
“45% of teachers did not get the correct dubious rating”
LikeLike
The “unconfidence interval”
LikeLike
Yes, a dubious diffidence interval or rancid hiccup.
How about REIF: rounding error of an improper flush?
Or MFS: misspelling of a forged signature?
LikeLike
“Flies-in-burg Uncertainty Principle”
“The uncertainty in VAM scores is always greater than the score difference needed to distinguish “good” from “poor” teachers”
or in math terms
deltaVAM > deltaSHAM
And while we are on the topic of
“VAMmechanics”
“Simultaneously good and bad!”
What the VAM has said
Teachers are like Schrödinger’s Cat
Both alive and dead
And, as it relates to the simultaneous “irrationality” and rationality of single sentences
“Irrational States”
Simultaneous sense and bunk
VAMMAD now has said
Akademos, Schrödinger’s Cat
Both alive and dead
LikeLike
make that “gold and scat”
“Irrational States”
Simultaneous gold and scat
VAMMAD now has said
Akademos, Schrödinger’s Cat
Both alive and dead
LikeLike
Btw
Rule of thumb (I haven’t always followed, to my chagrin) regarding trolls and cranks:
Do not engage. I repeat, do not engage.
Refute content without specific reference, and try not to directly reply or acknowledge. Ignore everything other than content that must be refuted, or ignore that too. Leave it to others, or let the cranks or trolls discredit themselves. They always do, ultimately, if that is what they are.
LikeLike
Thanks, Adademos, as you have probably noticed, I have all too often engaged with people who join the discussion in an attempt to get it sidetracked. They say things that are plain wrong, or off-based, or whatever. They don’t share our purposes, our concerns, our hopes. I will remember your good advice.
LikeLike
Thanks. I’ll try to remember it, too.
LikeLike
Akademos,
Is ECONOMAD nonspecific enough?
LikeLike
Yeah. You can reference, just don’t let it lead to engagement, with of course Klingons.
LikeLike
Andrew Cuomatose has an plan to get past the testing Fubah he created in NYS.
He has told wardens in upstate maximum security prisons to let two murderers escape each month so he can join photo-op manhunts for them complete with bloodhounds that will keep attention off him (and Teflon Tisch) while he is protecting the Empire State from evil.
LikeLike
Of course, this screwup from an out-of-state-vendor just adds insult to injury since the SLO process as no purpose other than intimidating teachers and micromanaging them, and expecting them to predict scores of their students on tests.
Among the teachers who are victims of the “miscalculations” are “educators who teach more than one grade level or subject and are required to meet multiple sets of student learning objectives.”
I will wager that the computer entry system is unfit to produce accurate teacher-student rosters and the multiple results from different SLOs for the same teacher, at different grade levels, and perhaps different schools.
I know that the Ohio system for data entry on SLOs gets really absurd. The system does not believe that a single art teacher can teach about 800 grade four students. The Ohio program is not set up to allow “every other day” or “every other week” programs in music or art. It is also programmed to treat an instructional year as 180 days without taking into account the loss of instructional time for testing systems that are deeply flawed to begin with.
Teachers in Buffalo should take no comfort from this history of the companyIt seems to have screwed up in the midst of a buyout…begin quote
“Helping teachers is at the core of our mission. Founded in 2000 as iAssessment, Truenorthlogic has grown to support nearly 1,000 school districts across the country.
From humble beginnings, a small band of developers, led by Truenorthlogic founder Jeanette Haren, created a statewide assessment system in California that provided insight into teacher technical expertise and the impact of new technology initiatives across the state.
“We wanted to create a company that was founded on the idea of helping teachers grow in their career and their ability to teach in this ever-changing, fast-paced technological environment,” said Founder and Chief Product Officer, Jeanette Haren.
By 2004, the company chose a new name, Truenorthlogic, and expanded its focus to include professional development and mentoring solutions, as well as workflow automation systems that streamlined many of the paper-based processes districts relied upon for observations and compliance reporting.” End of quote
New quote “We felt that the name Truenorthlogic better represented our mission to help educators find direction in building their careers and best serving their schools and students, thus came the ‘true north.’ ‘Logic’ represents how we provide that direction and support through technology as a software company,” explains Jeanette.
Under Jeanette’s leadership, Truenorthlogic created a one-of-a-kind comprehensive approach to managing educator excellence at every stage of their career.
Truenorthlogic’s Connect Platform fully supports the cycle of continuous educator improvement and its connection to student achievement.
Built on years of experience and feedback from leading education agencies across the nation, Jeanette’s innovative vision for a unified K-12 professional growth solution is more flexible today than ever before, capable of meeting the evolving challenges of K-12 school systems.
In 2014, Truenorthlogic was acquired by Weld North Holdings LLC, an investment company led by former Kaplan, Inc. CEO, Jonathan Grayer. Truenorthlogic has achieved many accomplishments with top-tier clients, first as a start-up, then as a fast-growing niche provider, and now as a leader in the K-12 educator effectiveness, professional learning, and talent management market.”
The new owner (in Connecticut) is probably interested in the data gathered by Truenorthlogic and if not the data, the “trusted” algorithms for their services. The new owner, Weld North Holdings, is a digtal learning company with a portfolio that includes, four companies in k-12 education (and several others):
Edgenuity, is a leading provider of core and elective instruction for students in grades 6-12, that creates courseware and online instruction programs.
Generation Ready is the nation’s leading professional development partner, working with school leaders and teachers to elevate student achievement through professional learning services and targeted solutions.
Imagine Learning harnesses the power of technology to instruct young students with engaging language and literacy software, delivering better ways of learning to schools and districts across the country.
And Truenorthlogic, described as “the market-leading provider of SaaS-based K-12 talent management solutions designed to support an educator’s entire professional growth cycle.”
I looked up SaaS. SaaS is “software deployed as a hosted service and accessed over the Internet.” A typical SaaS application is offered either directly by the vendor or by an intermediary party called an aggregator, which bundles SaaS offerings from different vendors and offers them as part of a unified application platform. These offerings are usually marketed by subscriptions to use the application. Fees vary by type of application and usage.
Someone should try to get a copy of this contract and see it it is possible to calculate the per-teacher cost for processing the data… and at minimum, get a refund
LikeLike
Laura, thank you for your insights and expertise. I looked up the leaders of Turenorthlogic, and they are basically computer science majors along with a CFO that has an economics degree from Harvard. They are edupreneurs that are jumping on the accountability bandwagon selling their magic formulae to whatever state that will write them a check. As you point out, there are many flaws, shortcomings and limitations to what they purport to do. The fact that they figure they can pitch teacher training as well when they have absolutely no background in education is absurd. What is worse is that states are squandering money on algorithms to nail teachers to the wall at the same time they are cutting teachers and budgets. We have a bunch of mindless fools in state governments flushing tax dollars down the toilet in pursuit of the “perfect algorithm” when they should be spending tax dollars educating our future voters.
LikeLike
Utah’s FULL of organizations such as these. Didn’t I tell you that Utah’s been laboring under “reform” for 30 years now? Well, this kind of garbage is what comes from a state that severely underfunds schools, denigrates teachers, and looks the other way ethically?
LikeLike
Utah is still operating on the “gold rush” get-rich-quick mentality.
It explains many things, among the most prominent the “artificial heart” and “Cold Fusion” shams.
If you have a crackpot idea for a magic elixir, Utah is a great place to get it funded at public expense. The more outlandish the idea, the more likley to get funded.
LikeLike
The artificial heart worked, actually. Variations on the Jarvik 7 heart from back then are still being used. I read an article about it recently.
LikeLike
Someone wasn’t career or college ready!
LikeLike
Unfortunately, teachers who receive growth scores that are calculated by NYS have no way of checking whether their scores are accurate. I have yet to find one person who has any information regarding the formula that NYS uses. Errors abound in the tests themselves, yet teachers are supposed to assume their scores are accurate? And they will be fired without so much as a double-check? Unacceptable. Who is accountable if accountability cannot be determined?
LikeLike
That is one of the main points of the Ledermans’ suit. The law states that the evaluation system has to be transparent and understandable. The VAM used in New York is neither.
LikeLike
The Buffalo Teacher’s Federation “negotiated” a formula with the Board of Ed – 60 points on artifacts and observations, 20 on SLO (student learning objective) and 20 on LMA (prediction on how lowest 50% of students will do). My husband got a score of 96 – Highly Effective.
Getting this approved by the state was a back and forth battle (Buffalo has been a thorn in Cuomo’s side since day one of his “takeover”).
Each school district came up with its own formula. Syracuse screwed itself when a large percentage of their rating was based on assessment results. That’s why entire schools failed to have even one teacher rated as highly effective.
FYI: This is not the first time the BPS has had difficulty with the scores. Two years ago over 20% of the teachers received incomplete scores when the district ran out of the funding to pay for the scoring process.
LikeLike
45% error? More like 100% if there is no validity to VAM.
LikeLike
You mean to tell me there’s a correct way to calculate VAM?
LikeLike
This is the same city that just lost more than 1/2 its schools to receivership AND to a superintendent hand-picked by NY’s new education commissioner, MaryEllen Elia. This is also the same district that has the misfortune of dealing with Carl Paladino on its BOE. He is a local developer (among other things) with huge stakes in the charter school boon in Buffalo. This is the same city’s schools that are supposedly doing so poorly? Based on…what? Manipulated data? Smell something fishy? It’s not Lake Erie.
LikeLike
Some of the schools in receivership should have been taken off the list when they met the agreed upon benchmarks, such as McKinley High School which has shown the required improvement (yet the state refuses to change their status).
We were playing a guessing game the other day – name the schools which were NOT in receivership. I’d like to see a list of Who’s Who and What’s What.
Ironically, Buffalo has one of the top rated schools in the COUNTRY – City Honors. Nobody ever seems to mention them when talking about the quality of schools in the Queen City.
LikeLike
Buffalo Teachers picketed against receivership schools in front of McKinley High School last Wednesday.
LikeLike
No comments from VAM supporter and continual litigant VA-SPG?
LikeLike
Yeah, I was wondering when he’d show up. What do you have to say, VirginiaSGP????
LikeLike
I think he’s in his basement trying to fix his emperor’s clothes torn apart while being caught in the hatch of his submarine.
LikeLike
What will make VAMMADs circuits smoke most is that STEM “experts” made the error and an “imperfect” classroom teacher found the error doing a “manual calculation”.
LikeLike
Ironic number crunching machine is failing itself as evaluator. Teacher: A Machine: F
LikeLike
45%?
Oops . . . . I suppose Buffalo thinks it merely impact people’s reputations, good will, public perception, and, oh yes, livelihoods.
Buffalo is city that celebrates incompetence and seethes at the truth and any semblance of acumen.
LikeLike
“Falsesouthillogic”
Truenorthlogic was in err
Pointed south instead of north
Little truth or logic there
Needle swinging back and forth
LikeLike
This score was a nice little welcome back to school gift found in teacher’s mail boxes their first or second day on the job.
LikeLike
Last yeàr the SED took a closer look at the scores and ordered about 15 Buffalo teachers to be recatagorized as Ineffective.
LikeLike