From a reader:
“FYI. Tennessee computers across entire state crashed on second day of writing test. Attached is letter from admin of White Station High School in Memphis. Thought you’d be interested:
White Station High School
February 4, 2014
Dear Parents,
We started our state writing assessments yesterday. Everything flowed smoothly. Today the online portal crashed statewide. As a result, today’s testing had to be stopped and no juniors scheduled for testing today were able to complete testing. We have been told there can be no testing done tomorrow either. We hope to resume testing on Thursday. Students who were originally supposed to test today will test Monday, February 10, 2014 and students who were supposed to test tomorrow will test Tuesday, February 11, 2014. We will keep you posted related to any other possible changes.
Thanks so much for your patience with us. We planned this one down to the smallest detail and then technology failed us. It is frustrating but we will regroup and make it work. Thanks again.
Sincerely,
Carrye Holland
White Station High School”
Diane Ravitch's blog 
So, students started today and couldn’t complete, but others didn’t start? One group has seen some questions (and the format), but another hasn’t? Clearly, any results from this section are invalid.
What company was handling this? A similar thing happened with spring statewide Minnesota testing. The state decided not to continue its contract and instead just gave a $30+ million contract to Pearson. A number of folks are interested in experience with different companies. So what company was it in Tennessee?
I don’t know the answer for the state end of things but I do know that even some wealthy districts have had trouble getting the system to work stably. When schools have to spend their entire annual training and technology budgets making sure that laptops don’t crash when hundreds try to access the same resources at the same time it is not pretty. And they are testing this out on second graders this year just so they will be able to handle the test environment next year. So swallow this- 100 second graders trying to take a timed test by laptop complete with go-forward go-back buttons confirmation screens and so on and barely the IT power in most places to handle it, much less if something goes wrong.
TN does not have the infrastructure to handle the demand for this much technology and data flow. Once again, people with no experience in running schools are unable to realize the real consequences of their decisions.
Measurement, Inc. is hosting this Tennessee Writing Assessment via their MIST software.
Unfortunately so. This software does not allow students to even use proper grammatical structure with book titles. There is no italics or underlining button, and my students have been taught to underline or italicize book titles. Someone who taught it last year told the students to place them in quotation marks. I could have died. If the state is going to have a program that is supposed to grade the students in such an area (Conventions and Language), then they need to make sure it addresses such conventions and language correctly.
This reminds me of the great innovations that Bloomberg was supposed to bring to testing on the high school level when he took grading out of the schools and put the tests on the computers. It slowed down the grading so much that some students didn’t know if they would graduate at their graduation. It also prevented excellent conversations that often happened as a by-product of grading at our own schools. What ended up happening was inefficiency, depersonalization and of course added expenses which enriched the coffers of the testing companies.
This seems to be the way of things..
Someone who has never been in a classroom… finally learns to use an i-pad and then …decides that everyone should use technology and do away with the humans..
“depersonalization” …”Humans replaced by a machine”
From the NYT front page:
“ADELPHI, Md. — Business leaders have pledged more than $750 million as part of a White House initiative to strengthen access to technology for 99 percent of students within five years, President Obama announced on Tuesday…Several other companies have agreed to join the president’s initiative over the next few years, including Sprint, which has pledged to provide Internet access to 50,000 underprivileged students, and Apple, which has promised to give iPads, MacBooks and other devices worth a total of $100 million to disadvantaged schools.”
So that the computers don’t crash during testing?
Beware of strangers bearing gifts.
Controlling computers and making schools need computers is a big deal for “corporate ed reformers” to hold onto and increase their initiatives. In reality, each student in a school does not need to have a computer.
What are the tax benefits to these corporations? The telecoms received huge tax breaks after pledging to make cable available to remote and rural areas. We still have places in TN that don’t have cable.
Obama’s reliance on private entities to solve major societal problems leads to corruption and dysfunction.
is your objection to the concept of testing, or just the foul up? Seems like a leap to go from “this test wasn’t done correctly” to “don’t test students.” Where’s the argument? I can only infer that the argument is that because this test wasn’t done correctly, then it’s impossible to test all students correctly?
You have it backwards. Understandable being a relative newcomer. Most of us have argued against standardized testing for a long list of reasons. The main argument is that these tests have been used (since NCLB was adopted) to threaten, coerce, and punish. On-line testing hasn’t changed the punitive nature of a disproven reform. Would you mind describing your agenda here?
Couldn’t have been said better!
No, I do not have it backwards. This article argues against online testing. Is the objection to the **concept** of online testing, or to this particular test? From the article, the way I read the article, the conclusion implied (but not stated) is that testing is pointless.
Absolutely, it should sometimes be punitive, if by that you mean some teachers should be fired. Other teachers should be lauded, hopefully most.
This specific test wasn’t done well, and maybe was poorly done, I have no idea about those specifics.
Again, the article implies that no online testing should be done. What is your agenda, to stop online testing?
I would say “sure, go ahead and put some fill-in the blank/multiple choice tests online” and use the data. Track students, teachers and schools.
Most if it will be no-brainer results that are already known, that poor students have extra hurdles, etc, of course. That doesn’t mean that the tests shouldn’t be done.
Again, what is your agenda? Improve the tests, cancel the tests or what?
thufirh – what is your educational background? It makes it easier to have a discussion if we know your starting point.
There are two discussions going on simultaneously. Here is my own analysis of the situation.
#1) I did not feel this was a good test for various reasons (see my critique).
#2) I did not feel that the online format for this test was user friendly. In other words, the formatting could affect the outcome of the test, resulting in a lower score for some students not adept at using the computer (vs if they took the test using a paper version). The results would then be invalid. And that is assuming there are no glitches with the computer system.
The bottom line is that the assessment must test the given material and not test how well the child is at taking the test in a given format. So, are we testing reading or keyboarding?
NYS has had various sorts of assessments over the years. I personally do not have a problem with these sorts of tests. In the past, the teachers examined the results for each child (it was the following year’s teachers) and used them for instructional purposes. The emphasis was on student growth from year to year. Teacher’s did not teach to the test.
In recent years the assessments became high stakes testing. Now they were used to rank schools. Pressure was applied, and teachers spent part of the year teaching to the test.
Now, testing has become the raison d’être for education. Students are given a post test at the beginning of the year (a pretest designed for the student to fail). Then the following months are spent teaching to the test, via lessons and sample test questions. In NYS, the push is on for more and more sample tests preparing for the upcoming annual assessment. In the elementary grades, reading, writing, and math take precedence over all other subjects. Science and social studies are a side bar – perhaps visited via reading passages. These tests not only rate the school, now they also rate the teacher. If the school gets a low score it might be closed. If a teacher gets a low score they might be fired. In some states, if a student gets a low score they might be retained. To make things worse, the grading system for the last round of assessments given in NYS were designed to fail over 2/3rds of the students. To add insult to injury, the tests do not always give a true indication of the child’s ability. Some people are better test takers than others.
So it is several points – the validity of the test and how the results are being used. In my mind, neither currently passes the smell test.
(And I haven’t mentioned the psychological damage of the current system – for students and staff.)
The impetus behind the drive to test all students is the data industry itself, thufirh. They’ve gotten regulatory control of public resources through their “accountability” legislation, which gives them coercive authority over the lives of individual children.
“Data” and testing provide no benefit whatsoever to the children or communities, and systems to impose forced data collection are a sickening financial waste.
It’s rotten and wrong from the roots up, all the way to the tips of every branch. There’s no contradiction whatsoever.
Ah, see I would simply disagree with some of your points. Of course, your right that there’s a commercial interest here. Probably these testing companies collude and make every attempt to overstate the utility of a few multiple choice tests.
Absolutely, these tests are of no direct value to students.
I’m sure there are different, actual, conspiracies surrounding these tests.
None of which is an argument against testing per se.
I’m just amazed that you’re arguing that no (online, multiple choice) test would conceivably be of any utility. Is that really your position?
Or, are you perhaps arguing that these tests aren’t very good in and of themselves? Because you seem to start from a reasonable critique, but then…
thufirh,
This Common Core College and Career Readiness Assessment Prorgam (C.C.C.C.R.A.P.) instantiates a backward, nineteenth-century theory of education: Learning is mastery of the bullet list (the standards), and schooling is the doling out of punishment and rewards (extrinsic motivation), via summative assessments, to make sure that the learning is done. The theory was neatly summed up in the New England Primer, the first textbook ever published on these shores:
The idle fool
Is whipp’d in school.
The whole stinking Son of NCLB, NCLB II: The Nightmare Is Nationalized system of standards-and-summative-testing-based education distorts pedagogy and curricula horrifically and runs counter to any real educator’s prime directive, which is to nurture intrinsic motivation to learn, to create self-motivated, life-long learners.
I absolutely agree with your points, only not with what I take to be your conclusion.
No test is perfect. Are you letting the perfect be the enemy of the good here? That is my take on this, that valid criticicms of this specific test, its implementation, and, overall, this category of testing, is probably valid.
That being said, this type of testing could be of great use.
Again, thufir, please do some reading before thinking you’re challenging anyone on this blog. That issue has been covered in depth. Diane’s blog has been active going on two years now, all of which is archived – you could learn a lot spending time in those archives before displaying your ignorance here.
Nice switch of topic.
I am going to guess that school districts don’t negotiate performance penalties into these lucrative testing contracts, so the companies don’t really have an interest in getting things to work right.
Bingo.
Is anyone surprised? As anyone ever had a pencil crash?
Only with Staples pencils.
LOL…that is true..I started using the golf-course pencils!!
Ah, pencil pirates! I had forgotten the daily hallway trawling to collect the pencils (from my classroom) left behind. In my last district, they hung on to their pencils (high poverty), but, strangely enough, they still didn’t have them in my classes. Golf pencils were recommended as an aid to memory. Few wanted to write for the entire period with one. I ended up collecting IDs when they borrowed pencils. Otherwise, I couldn’t keep pencils in stock. One of those hidden costs of teaching.
LOL, well said, never2old!
It is Pearson. And Tennessee’s infrastructure is so old that it cannot handle the internet volume. Many schools are not equipped. Causing municipalities to borrow money to buy computers. We are taking tests on laptops that will not hold a charge and some schools only have 35 computers for everyone, causing students to rotate through making testing take weeks and weeks. This is all horribly mismanaged and a huge all waste of tax dollars.
And just this evening Nashville’s Mayor got the council to approve BORROWING $6mill for PARCC compliant computers for MNPS schools. He would not fund the teacher training on the technology so the district will have to find the other $7.1 mill that they need elsewhere. Unfunded RTTT state mandated testing making tax payers fork out more money….http://www.tennessean.com/article/20140204/NEWS04/302030106/Council-approves-plan-borrow-6M-equipment-Common-Core-testing
Why? Because it reflects a deeply confused extrinsic punishment and reward model of education and because it will sell a lot of computers and software.
When technology works it’s great, but it doesn’t always work.
There are always tie ups in Buffalo when all the teachers try to input the quarter grades. The system can only handle so many users, then it either slows down or crashes.
Systems crash. Even good ones. The Buffalo and Erie County Public Library System crashed (many years ago). There was no backup. The lists of books which had been signed out as well as over due items were lost. A calamity.
On a smaller scale, the school libraries in the Buffalo Public Schools are fully automated. When the system crashes, you cannot sign out books to the children. You also cannot check in books. Sometimes data is lost. If the system is down for several days, this causes problems, especially since librarians are only half time in the high traffic elementary schools. Of course, this is inconvenient, but small potatoes compared to a statewide shutdown.
Remember the problem Obama Care had with online capacity?
Look for similar mishaps across the country if assessments are digitized.
Crashes can occur on individual computers, school computers, district computers, and even (or especially) on state wide computer systems.
Our problems are just beginning. It will be interesting to see how Tennessee resolves this issue.
“When technology works..it is great”
I agree …except that it is still in its infancy and is not ready for these experiments in testing..
I can not take an online test as well as a paper test…I need to write on the paper..on the question..highlight….I need to be in the problem…not staring at a screen ..I get through it but I know I do not do as well as if I had a paper version that I could write all over while I am thinking ..Sometimes my thoughts come out so fast I have to scribble them down…My style..
Speaking of using all of this DATA…Has any research been done on the paper test vs the computerized test?
There are different learning styles and different ways to address each individual’s needs but this has been dumped for
“One Size Fits All”
Human Interaction is so very important,…Individualization is so important…
Technology can not be stopped but it must be used “not to replace humans” but to enhance all that we do..
Not data, but an anecdote. I took the pencil and paper GRE as a senior in college and then 10 years later with the computer based version when applying to a PhD program. Exactly the same score.
I have not seen anything in the protocols released to date for Smarter that would bar the use of scratch paper as an aid when taking a computer based assessment.
The two biggest challenges I expect with the new Smarter assessments are helping students learn how to use the accessibility resources that are included in the assessment and navigating the long text passages that will require scrolling. Those may be points of frustration.
Stiles
You took the GRE 10 years later!!
You should have scored a Perfect Score 10 years later
10 years??????
No comparison..
I should apologize..I read my comment and slapped myself as it was wrong. I see what you are saying….But to each his own.. …I, personally can not take a math test on a computer screen…..I find the other tests to be just as user unfriendly…..I like to get into the problem…part of my brain sensors…I was born a doodler..
Stiles
Taking the GRE as an adult applying to grad school CANNOT be compared to asking 20.000,000 eight to fourteen year olds to do the same. You clearly have no experience working in impoverished schools with underprivelged children. You don’t have to believe me, just wait and watch as this computerized testing nightmare unfolds.
NY Teacher, I have worked in public education for 23 years. I started in a rural school district that had and has a large number of economically disadvantaged students. When I was an elementary school principal, we were Schoolwide Title I. My student teaching was in New York City. My child attends school in a medium sized urban district where approximately half of the students are economically disadvantaged. On the other hand, I have also worked in a predominantly middle class suburban district where I served as the technology director and later as a central office administrator.
If students are accustomed to using technology in their learning on a regular basis, a computer based test will not be a “nightmare” simply because it is accessed via technology. In some respects, computer based tests can be fairer environments for a range of learners because they can be adaptive tests that incorporate universal design principles (not all are).
If students attend a school where technology is not a part of the learning environment (and likely the infrastructure and support are lacking), then a computer based test could be a nightmare. To me, it is less a matter of whether the students are economically disadvantaged than a question of how the school has incorporated technology over the last 15+ years.
Stiles
Let me try again.
Taking the GRE as an adult applying to grad school CANNOT be compared to asking 20.000,000 eight to fourteen year olds to do the same, including the learning disabled, (including countless unlabeled dyslexic students) and ELL. You don’t have to believe me, just wait and watch as this computerized testing disaster unfolds. How quickly you managers/administrators forget.
NY Teacher, I mentioned the GRE up thread in response to the question on the comparability of paper and computer based test. I agree the experience of an adult taking the GRE is different than a school aged child taking an assessment. Probably a better comparison would be NWEA’s studies on the paper Achievement Level Test (ALT) and the computer adaptive Measures of Academic Progress (MAP).
If your school does not currently use computer based assessments, I understand your concern. In my region every district, regardless of whether it is urban, rural, or suburban uses a computer based assessment (either STAR or MAP) as part of the universal screening and there is no disaster.
Whether we screen too much with standardized measures is a question are reviewing as a district, but the suitability of computer based assessments for large scale assessment is well accepted.
Stiles – you don’t give me enough information to judge your response. Where are you from? How large are the schools? How long/many days does it take to complete the test? How many computers are in each school? How long have the schools been using computer assessments? Has a comparison been made between hard copy and computer copy? What kind of computer instruction is provided during the school day? What is your background in proctoring g exams? Etc. Your comments don’t mesh with my own experiences, but, for arguments sake, I’ll respond as if your points were accurate.
I have no objection to computer generated testing, in theory. I have taken online tests in college classes (where my answers were inadvertently changed as I scrolled and my computer crashed before the test was completed and had to be rebooted – costing me valuable time). It’s hard to believe whole districts did not have any issues.
My objection, in terms of computer testing, was the format of this particular test. It was not user friendly, a must for children. In NYS, the ELA and Math Assessments simultaneously occur over a three to four day period for grades 3 to 8. Many of the schools in Buffalo are PreK to 8 schools, which means the majority of the students will be taking the test at the same time. The exam is taken over a one and a half to two hour period each day, longer for students with IEPs. Individual classrooms MIGHT have a few computers. There is usually a computer lab with 30 computers, but not all will be operative. The library might also have a bank of 10 to 20 computers. Even if the technology worked – how is the entire school, or even an entire grade of about 100+ students, going to simultaneously take the exam? Even if the times were staggered, there is still not enough hardware to accommodate the entire testing body.
And we are assuming there will be no glitches as almost half of a district containing 36,000 students are utilizing the same computer system (which couldn’t handle the teachers inputting their SLOs – an evaluation point – onto the computer site without crashing).
Imagine the million students in NYC.
Now, what was your point again?
Ellen, I am in an upper midwest state that is part of the Smarter Balanced testing consortium, so I’m not familiar with PARCC. The district I work in has an enrollment of a bit more than 4,000. The adjacent urban district my child attends has an enrollment of 25,000. In our district, we have been using computer based assessments for more than a decade. We provide keyboarding instruction at the elementary level, but otherwise no separate technology instruction below high school. At our high school, we have courses in the arts and technical education where technology plays a central role in the course. Our teachers use technology in their classes when they feel it supports the teaching and learning. I have proctored MAP tests when regular proctors have been absent. Our student to computer ratio varies some from school to school and we are not a 1:1 district, but I feel we have an adequate mix of fixed and mobile labs, plus laptops for a district that is not yet 1:1.
My point is that computer based testing is not inherently worse than paper based testing. In fact, it is often superior because there is the opportunity to incorporate media, make the test adaptive, and include accommodations and universal access features. And unless you are using a Scantron with your paper tests, teachers receive the results in time to utilize them if they choose.
I agree that there could be problems with the rollout of the new computer based tests. For a few reasons. You mention the importance of format and I agree. The integrated universal accessibility features can be very useful, but only if students know how to use them. There is a trend toward longer reading passages and this usually requires scrolling. If students are not accustomed to reading long passages on a screen and scrolling up and down, that could be a frustration. The quantity and reliability of school technology varies from school to school, especially in states that do not have separate funding for instructional technology. Also, when a school or district decides to implement a computer based test, it is a local decision and readiness is usually considered carefully. With the consortia tests, there are layers from the consortium to the state to the district to the school. Each is an opportunity for the ball to get dropped in the planning process.
I know that for Smarter Balanced, the assessment window will be twelve weeks across the consortium (possibly shorter depending on when a state finishes the school year). It won’t be the case that all students statewide will be taking the test on the same day, unlike what is sometimes the case with paper tests. Also, states are considering (this may or may not happen) assigning weeks within the window for grades so as to ensure distribution of testing. Smarter Balanced assessments are untimed too. In our state, we have readiness guides and capacity spreadsheets to help schools and districts in their planning for next year. Our technology director and I are currently meeting with each of our schools to complete these technology check lists.
In a nutshell, I feel computer based testing is an established approach, but if all parties (consortia, states, districts, schools, and staff) do not plan ahead, there certainly could be problems next year.
Ezra Stiles, if you like computer testing, that is swell. Why should everyone else be compelled to use it? As a parent and grandparent, I much prefer teacher-made tests, immediate feedback, and human judgment. One size does not fit all. Does it?
Stiles, after ten years, it seems your section of the country has worked out the kinks for your current testing protocol using online tests.
Not so in NYS, especially in the Buffalo Public Schools. A twelve week window, vs one week with a one week makeup period for two sets of back to back timed assessments, is more conducive to utilizing limited facilities. (This would not work with our secure New York State Regents Exam which must be taken simultaneously across the state.) Hiring computer teachers to teach keyboarding to our students would also help. Grading is compiled and rated by an outside agency, so we would not get the immediate results you mentioned.
Of course, the formatting is another issue. Since our students don’t have experience with this type of test, and computer access is limited, it will take years to get them up to speed. Plus, we will need to invest in hardware and infrastructure at a time when there is not enough funds to hire the right number of staff. I’m not sure there will be enough time or money to get all our ducks in a row.
And, even then, the content of the actual test is more than questionable.
I suppose we will have to compare notes on this topic once our state catches up. Your situation sounds ideal. I’m not sure it will stay that way when the full impact of common core hits. There are many variables we have not discussed.
The world I live in is far from serene. “Turbulent” might be a better word.
Diane, I do not prefer computer based assessments to teacher created or course based formative and summative assessments (ideally common collaboratively developed assessments). However, if standardized benchmark assessments are required for RtI and there are state mandated accountability assessments, then I want them to be as fair as possible for students and the information to come to teachers as quickly as possible. In my experience computer based tests are better than paper based standardized tests.
We have too much standardized testing, but I disagree that computer based standardized tests are worse than the old tests they replace. Am I haggling over details instead of the big picture? Yes, it’s one if my flaws.
Ezra, as I said before, that is your choice. The question is: Do you think that every child in the USA should be compelled to take a computer-adaptive test, even in schools that differ from your view? I like my eggs cooked a certain way, but I don’t think everyone in the US should eat their eggs the way I like them. The biggest beneficiaries: the testing corporations, especially Pearson. Why should they have the power to write the questions and decide for the entire nation “what matters most”? Who made them the czar of US education? The power to write the nation’s tests is the power to regulate its minds. No, thank you. I prefer the cacaphony of a free marketplace of ideas.
Diane, I do not think any student should be compelled to take any test that is not organic to a class they are taking. In my preferred world, public school districts would be required to adopt a local assessment plan that was developed with teacher participation and parents could opt their child out of an assessment that was part of their coursework.
To me, the question is why mandatory testing? Not, why mandatory online testing?
If I work in a setting where a standardized universal screener is required for RtI and accountability assessments are required, then I want the assessment to be as fair to the range of students and the information to be provided as quickly to teachers as possible. I know almost every poster in this thread would like to see mandatory standardized assessment reduced or eliminated and I agree with that. However, I disagree with the idea that it is less bad to require students to take a paper test where the items are clustered around the grade level proficiency target and the results come back in 3-5 months vs. a computer adaptive test where the items are more individualized to the student’s instructional level and the results come back rapidly.
The discussion of standardized accountability testing is important because that connects to the use of the data for high stakes decisions that can undermine public education. However, if we are looking solely at instructional time and the impact on students, standardized universal screening for RtI is something I would encourage you to consider as a topic.
Ellen, it is very sad to see the direction of public education in New York State. I am an upstate NY native and started teaching in NY. There is no excuse in any state, let alone a wealthy state like NY, for the lack and inequitable distribution of resources. There has always been a lot of funding inequity in the Empire State, but the recent years of leadership failure at the state level seem to have really driven the public education system into crisis in many areas.
We have our own challenges in my state now, primarily funding and privatization, but at least our state agency has judgment and competence.
The courts have told the state they must end the inequities (it started with NYC suing, but Buffalo was included in the results). We are still waiting. If anything, education has less funding now than before – for all districts.
Where in upstate NY are you from? Syracuse?
>I know that for Smarter Balanced, the assessment window will be twelve weeks across the consortium (possibly shorter depending on when a state finishes the school year). It won’t be the case that all students statewide will be taking the test on the same day, <
Stiles,
Do you know how SBAC plans on maintaining test security with a 12 week window?
NYS Teacher, the SBAC assessments are computer adaptive, so the assessment adjusts question difficulty based on based on the student responses. As a result, there is no single form for the assessment, students may not have much overlap on items, and there is little if any additional security provided by testing simultaneously. The performance tasks are not computer based, but I am not familiar with how they will be scored and incorporated with the computer based portion of the assessment.
Even with paper based tests, not all states follow the New York model of grades testing at the same time statewide. Some states allow districts to set local schedules within a state window as long as physical security of the examination documents is maintained.
Ellen,
Up in the North Country near Watertown.
Stiles – I’ve been to Watertown. Hot in the summer, cold and snowy in the winter. A beautiful town when the weather in nice.
Part of the problem in education in NYS is the diversity – Rural, Urban, Suburban, ethnicity, refugees, poverty, race, ESL, Special Ed, socioeconomic factors – more so than almost any other state, especially if you factor in NYC.
Within the state, there are many excellent school districts. It’s too bad that they are forced to adopt a system aimed at the lower third when they had already worked through the kinks and mastered the previous system.
City Honors in Buffalo, NY is one of the best schools in the country where all the students graduate, go to college, and have excellent careers, some reaching world renown. If the goal of CC is to produce college ready students, then everyone at that school should have passed the assessment, yet, they had a large group of low scorers (about 30%). Something isn’t right.
At this point in time, since all things are not equal, a computer generated national exam is not in the best interest of education. Until they can come up with a fair exam in a fair environment which has been field tested in both paper and computer formats, I think we should pass. And the final decision should be made by the local school districts, not forced on schools unilaterally.
I hope other states learn from the mistakes of NY. Watch and DON’T repeat.
Stiles
I was under the impression that the RTTT/CCSS aligned computer tests produced by PARCC and SBAC were not being administered until next school year. Are you referring to a different SBAC assessment?
Are you saying that test security has not been an issue because students take customized versions of the exam. This is a foreign idea to anyone who teaches in NYS and does not seem to be in the punitive spirit of NCLB and RTTT reform. If students take different versions of a math or ELA test then how are the scores used to evaluate teachers? Or, do you teach in one of the 5 states that rejected RTTT/CCSS?
NY Teacher, SBAC pilot tested items last year, will field test this spring, and will introduce the operational assessments for the 2014-2015 school year. While SBAC assessments will not be administered until next year, the consortium has been fairly transparent in their communication and adaptive testing has been an announced feature of SBAC from the beginning.
Yes, test security is less of a concern since students take a customized version of the exam. It is a different approach from a fixed form assessment and relies on Item Response Theory (IRT). I am not a psychometrician and do not feel qualified to go into the technical aspects, but to use an Olympic metaphor it is similar in respects to figure skating where individuals perform routines of different difficulty and are scored on their success in performing different sets of assessment items. That is how comparability between students is approached.
Regardless of the assessment approach, comparing student achievement and comparing teacher performance are entirely different. All of the limitations in terms of measurement specifications and data quality apply, plus the question of whether what can be measured reliably represents the underlying learning. These questions don’t go away.
Economists who develop value added models will view the adaptive tests favorably because the standard error is uniform across the range of student achievement. Fixed form assessments that evaluate students to a proficiency standard have items that are clustered around the difficulty of the proficiency standard. Students who are near the proficiency level typically have a low standard error on the score and students who are either well below or above the proficiency level will have scale scores with higher standard errors.
When value added data is available that adjusts for student characteristics, I feel it can provide useful feedback to educators as one lens, but do not feel it should be used for any high stakes decisions. I certainly do not feel that educator evaluation is a compelling reason to mandate a standardized assessment that is not already integrated into a course a student is taking. I have not seen any evidence to support that.
Up thread, you asked about the implications of the twelve week testing window. When the SBAC technical advisory committee met they endorsed a long window as suitable for evaluating student achievement with a possible retake option. I do not believe they addressed the question of a long window and using the results for educator evaluation. Even if one accepts the value-added concept, a long window raises questions of comparability of “dosage” and data quality that must be acknowledged.
Appreciate your thorough responses. Still finding it somewhat confusing because any state that applied for RTTT and/or sought NCLB waivers was required to evaluate teachers using test scores. That issue is the root of many of our problems here in NYS. Because of the fact teat teacher’s careers and reputations are on the line, having a large testing window would certainly tempt at least some to explore the lack of security. How does SBAC end up scoring and evaluating students. What is the value of the testing in your view. Here in NY the Pearson tests administered last spring were kept very secretive, and test results were only used to tell parents how poorly their children fared. Many teachers were placed on improvement plans as well as a result of test scores.
NY Teacher, you pose good questions.
What is the value of testing? We should not make a high stakes decision for a student based on a single piece of information and the same applies to teachers. Testing can provide useful information, but we need to use multiple measures and understand the limitations of each measure. Time is unforgiving. We don’t have time to engage in activities that don’t promote student learning.
Improvement plans and test scores. No teacher should be dismissed based on a single evaluation, let alone a single set of test scores. If the evaluation process suggests a concern, the teacher should be placed on intensive supervision with specific goals, more frequent feedback, and coaching (peer and evaluator). In my district, intensive supervision is a three semester duration and our preferred outcome is improvement and retention vs. termination.
SBAC has done a decent job of being transparent and involving educators in the member states. One of our teachers has been an item reviewer for content and I have colleagues in neighboring districts who have been bias reviewers or reviewers for the professional development library. There are still questions about where the cut scores will be set for the performance levels and how the computer based portion and the performance tasks will be combined.
Regarding test security, it is the individualized mix of test items that provides test security instead of a simultaneous test event. It is a different approach than what you have experienced in New York, but it is not a lack of security. As someone who has proctored Regents and elementary NY assessments, it is a completely different approach.
My strong preference is that educator effectiveness systems be centered on feedback, coaching, and professional growth instead of high stakes rewards and sanctions. In my experience, we have plenty of good teachers and school districts should worry about how to support them and help them grow professionally. Lack of quality leadership is a more significant problem than a lack of classroom expertise. Yes, as in any field there are some struggling practitioners. Often they can be coached up and if not, then we do need to separate them, but they (and the institution) deserve an opportunity to grow and due process.
Stiles – unfortunately, what is happening in your state is not the situation in NYS which has adopted CCSS, high stakes testing (with limited feedback which is not timely), plus an aggressive teacher evaluation system which targets teachers whose students fail to get results on a test with an artificially high cut score.
Perhaps your district can stem the tide, but our state ed dept is in danger of drowning both students and faculty in their quest for misguided rigorous ideals inappropriate for the given age levels.
Glib, glub, glub!
Ellen and Stiles,
On a lighter note, if we keep this up I’m pretty sure that we will collectively set the DR blog record for the longest/narrowest thread!
Ellen
We feel each other pain. NYSED/BOR/GC have dragged us all completely over the brink of sanity. Staring into the abyss is no fun. If we quit, they win. STAY FIERCE and FEARLESS, always.
Stiles
Still wondering how your district is in compliance with RTTT or NCLB waiver requirements. Once again thanks for the detailed and thoughtful responses. My apologies for the initial negativity but our frustration here in NY sometimes gets the best of us.
NY Teacher, it is getting to be a long thread! I don’t want to convey the impression that everything is fine in my state, but we have so far avoided the severe problems that New York is experiencing. My understanding is that all states that won RTTP grants or were granted ESEA waivers had to submit plans that met the federal government’s parameters, but states had some flexibility in how the met those requirements and how quickly. States that most closely conformed the the Department of Education’s criteria, won RTTP grants. Other states didn’t hew quite as closely. Sometimes they had to resubmit their waiver applications. I’m talking about differences in degree instead of differences in kind, but if you have insomnia you can browse the different state plans at the FedEd’s web site and you will see some diversity.
That makes sense. Here in NY they rung all the RTTT bells and “won” the biggest prize available – $700 million. A fraction of what it will eventually cost to implement; especially if they decide to re-join PARCC. Now being one year ahead of the curve has had some hidden benefits including early exposure to the tail that wags the dog: Pearson assessments. Total disaster. 70% failure rate among all 8 to 14 year olds. Parental outrage has followed; especially in many of the more affluent districts where they know that their schools are succeeding (not failing). Political pressure on the state senate and assembly is happening now and just today the politicians have threatened to take action regarding CC implementation if the NT Board of Regents refuses to act. Cuomo cherry picked a committee to “study” the implementation of CCSS but that’s just a set up. Political pressure on Cuomo could be its undoing. Part of the problem is that NY would have to return and forfeit a fairly large sum of money if they decide to reject it. And that’s the clean money. Cant imagine all the pockets that have been lined during this process. Sounds like your state hasn’t completely lost its head; good luck and sleep well knowing that we are DR blog record holders.
The word “Testing” in this letter to parents is used so many times that all the other words are irrelevant. Remember how often the Bush administration and the media slammed us with the word “Terrorism?” Now it is TESTING. **sigh**
I have to chuckle…:-) So true!
That’s “turrurism” to you begtodiffer!!
The biggest reason why is simply that reformers think it will work really well. It is simply over optimism, and a lack of understanding how little investment has been put into technical infrastructure in schools over the past 15 years.
In particular, all the “disruptive innovation” rhetoric has been particularly damaging because the entire premise is that cheaper, less sophisticated, but ubiquitous tech solutions are most effective — which they are sometimes — but in this case we have so much lost ground to make up that correctly implementing these tech solutions in schools can’t be done with some quick, cheap disruption, but would require a much more systematic, long term development of human and technical infrastructure.
Also, running something episodic like a testing system is inherently hard because you have to go from zero to full speed perfectly when the testing period starts. Under the best circumstances, bugs are inevitable.
I don’t buy for one minute that the “reformers” really think it will work. If they thought that, they’d want it for their own kids’ schools. No “reformer” yet has put their own kid in a school that practices any one piece of the “reform” that they want for other people’s kids. That’s because they know it’s all a load of… something I’m too polite to say on Diane’s blog.
“…bugs are inevitable.” Let me fix that for you: “bugs are *intentional*”.
It is a bit of speculative hair-splitting, but I *do* think that reformers think the awesome new technology is going to be awesome. It seems to be an important piece in maintaining the cognitive dissonance in the head of people like President Obama. “Reform” isn’t going to be about fill-in-the-bubble tests because the NEW tests will be awesome and high tech. And while reformers obviously aren’t sending their kids to KIPP schools, I’m sure they can *imagine* awesome high tech adapted assessments that they wouldn’t mind their children taking once and a while.
It *seems* perfectly reasonable.
” I’m sure they can *imagine* awesome high tech adapted assessments that they wouldn’t mind their children taking once and a while.”
…once all the kinks are worked out on the peons.
Then why don’t they want that awesome new technology in their kids’ schools? For that matter, why don’t they want the testing in their kids’ schools?
That’s a bit misleading because “they” probably send their kids to either private schools or better public schools. There’s no logic to the claim that if “they” don’t send their kids to aschool of your choice that their argument is invalid. The argument “they” make stands on it own, or falls on its own.
In part because the awesome new stuff still only exists in their imagination. Seriously… it is always just over the horizon.
And reformers and other wealthy people do send their children to schools that have plenty of access to technology that actually works… AND small class sizes. It would not be shocking or problematic if kids in prep school are taking cool interactive physics tests on their iPads.
“No “reformer” yet has put their own kid in a school that practices any one piece of the “reform” that they want for other people’s kids. That’s because they know it’s all a load of… something I’m too polite to say on Diane’s blog.”
You can use Robert’s new acronym. I forget how many Cs but it is something like this: CCCCRAP. You get the message.
This C.C.C.C.R.A.P. (The Common Core College and Career Readiness Assessment Program) represents the triumph of technocratic philistinism over humane scholarship and learning.
Robert.
Thank you for your dynamic description!!
Genius comment…as always…
About that RTTT..What goes up…must come down…
Racing….that is what perturbs me the most…
Racing to see who can score the best on a test….
Running over and flattening the futures of the students and the teachers and America….without any regard to Human Culture or emotion..
This maddening dash to the top reminds me of “Running with the Bulls” and being trampled by these “Greedy Portly Political Bulls”
Very well said, Neanderthal!
The dummies always go with the sports metaphors. Years ago, I taught in a Catholic girls’ school. We had a drunken old priest who would get on the PA every morning, already tanked, and say, “Girls, you gotta get out there today and carry that ball!” LOL.
Love your acronym. Even though the words behind it are beginning to fade, it matters not one whit.
And they’ll tell us how that lousy gubmint is constantly screwing up. Are people ever going to wake up to how private companies are intentionally making the government look bad? The Nation had a great article about that yesterday regarding the HAMP program and Obamacare. Everything she says applies to education/testing too: http://www.thenation.com/article/178218/all-presidents-middlemen
When it comes to the top-down imposition of technology in the classroom, I have two slogans/pieces of advice I’d like to suggest:
1. Beware of Geeks Bearing Grifts.
2. Nothing Dulls Faster Than the Cutting Edge.
Guess we need to keep a sharpening wheel by the guillotine, eh!!
Online testing is not about a “new and improved” adaptive and interactive tests, promoting 21st century skills, or preparing students for college and careers.
The accuracy and reliability of the data collected is not even a concern because the test program will automatically generate new test items based on student responses whether the student misread the question, lacked the knowledge/understanding being tested, or just wasn’t trying…
It’s all about collecting, mining, and sharing student data in real time…..corporate profits, not student proficiency.
Simple Thoughts
We are all born with innate qualities that allow us to become the persons that we are today…and the persons we can become tomorrow..
We can NOT ignore the individuality of people…humans….
Our emotions…Our feelings…Our creativity…Our thoughts…Our talents…Our need to be a part of this world and to have a purpose in this life.
This can not be ignored in the world of education…
yet I see the above…… being thrown off the top of whatever they are racing to….
Among many of the hidden costs of reform accountability regime is the vast amounts of administrative and teacher time that is consumed in implementing one test after another. Hours, days, months that could be used for planning staff development programs, coaching teachers, curriculum development are tossed away for the managerial details associated with the distribution, supervision, and collection of test materials. But this is the essential lesson of governmental regimes that promote fear base approaches to policy development. It cost a lot of money to inspect and sanction. In trust-based organizations, money flows to the central purposes of the organization, not to elaborate mechanisms to see if the purposes are carried out. Of course, the problem with the current reform regime in Washington is that the central purposes of their organization is not valued by anyone in the profession —thus, the resort to fear based policies.
Well said, Alan.
The education policy think [sic] tank called the Thomas B. Fordham Institute, one of the pushers of deform on the country, is hosting an event today to discuss its new study called “The Student and the Stopwatch,” which is meant to counter this notion that too much time is given to testing.
http://www.edexcellence.net/events/the-student-and-the-stopwatch-how-much-time-is-spent-on-testing-in-american-schools
You see, the Fordham Institute believes in big data. And here is how you get data, Fordham Institute style: You decide what answer you want, and then you gather data that will yield this result. It’s the same model that states have been following when reporting their test scores to the DOE for years now under NCLB: manipulate your cutoff scores, do trial runs of tests and then throw out test questions, and cook up fanciful raw-score-to-scaled-score conversion charts until you get the outcomes you were looking for to begin with.
In this case, Fordham will, predictably, report just on the actual time that kids spend sitting in front of the state test, and then they will give this as a ridiculously low percentage of the total amount of time that the kid spends in school or in school and doing homework or out of the kid’s entire year. But Fordham will have overlooked the little inconvenient truth that NCLB and RttT have made testing an obsession in schools–have turned our schools into test prep factories in which students, teachers, and administrators, spent enormous amounts of time
taking the tests themselves
taking state-exam-specific practice tests
taking diagnostic, benchmark, and practice tests purported to prepare kids for taking the state tests
doing test prep activities in class
reviewing test problems in class
working in print and online test prep products
writing in state test formats
grading writing using state test-style rubrics
learning how to use those test materials
familiarizing themselves with the rules for those materials
doing test-formatted activities in their regular textbooks, which, like kids’ classes, have been infected by testing mania
reviewing test results
complying with testing mandates
holding data chats
holding pep rallies to boost test morale
analyzing their test data
manipulating test data to put it into their software systems
and so on.
Teachers will now commonly say things like, “Well, it’s February. The school year might as well be over. It’s going to be all FCAT all the time from here on in.”
preparing materials for parents explaining the tests
Of course, if the Thomas B. Fordham Institute were really interested in scholarship instead of simply being a PR firm for Education Deform, it would answer the question, “How much time is spent on the tests?” by gathering data on all these different kinds of time spent on the tests, and if they did that, they would find that
THE TESTS HAVE TAKEN OVER AND HAVE DRAMATICALLY DISTORTED WHAT IT MEANS TO BE IN SCHOOL
Then that would be a problem with the specific test(s) and its implementation, as well as general bad management. I don’t see how you go from those points “don’t do any standardized testing.” In the big picture, that would also involve almost willfully ignoring data about where each student was at thebeginning of the school year versus the nd, and where that group, class and school as whole started from.
Then that would be a problem with the specific test(s) and its implementation, as well as general bad management. I don’t see how you go from those points “don’t do any standardized testing.” In the big picture, that would also involve almost willfully ignoring data about where each student was at thebeginning of the school year versus the nd, and where that group, class and school as whole started from.
“We planned this one down to the smallest detail and then technology failed us.”
Yep, tweren’t our fault, technology is guilty, off with its. . . power cord????
Two bits says, they don’t have anything about non-performance in the contract!
So, my question is what is happening to the real learning experiences of these high school juniors as they prepare for college and career? How much valuable learning time has been wasted while these tests stop, start, restart, stop again? The students know that these tests are meaningless in terms of their own college and career plans – they are more interested in AP an, IB or SAT test, not these state tests. A group of juniors in a school I know of boycotted the test and sat down in the hallway.
Thank you Duane…you put a smile on my face!
SBAC means dog in Russian (pronounced S-baka, written sobaka, but o is silent).
So this is what it is – SBAC-bites!!!
good to know, Preeti!
A friend of mine was upset that she lost and hour and a half of instructional time yesterday, and then she will lose time again when they try to retake the test.
Alan C Jones
>Of course, the problem with the current reform regime in Washington is that the central purposes of their organization is not valued by anyone in the profession —thus, the resort to fear based policies.<
Once again you have hit the nail on the head.
All fear, no trust is a recipe for failure.
We are aware of Diane’s history. She has made no secret of it having publicly discussed the change in her thinking. You will have to update your Diane bashing and take it elsewhere to a less aware audience..
Dianemary
If she’s really poison to education, what’s your suggested antedote?
All bark – no bite; as expected.