Guyette: How the EAA's Buzz Program Exploited Detroit's Most Vulnerable Kids
In June of 2011, Gov. Rick Snyder stepped behind a microphone at Detroit's Renaissance High School to announce the start of a revolutionary new approach to education in Michigan.
The problem of poor academic performance would be addressed in dramatic fashion.
"We do have too many failing schools in our state," he said. "If you look at us statewide, only 16 percent of our kids are college-ready. That's absolutely unacceptable.
"We need to focus on a new way of doing things."
The target would be Michigan's lowest-performing schools. The bottom 5 percent.
The stakes could not have been higher. As the governor explained it, the future of both the city and the state as a whole would be riding on this experiment in education.
"For Detroit to be successful, it depends on having successful schools. For Michigan to be successful, it depends on having a successful Detroit," Snyder declared. "So we're all in this together, and we're going to make this happen as a team."
A little more than a year after that speech, the Education Achievement Authority (EAA) opened its doors to students. Instead of taking on the challenge of lifting up all of the state's low-performing schools, or even a large number of them in different areas, decision-makers — operating in a way that was anything but transparent — decided to have as this experiment's proving ground 15 Detroit schools. Three of those would be independent charters. The other 12 would serve as the focal point for the EAA's attempt to radically reconstruct the way in which students are educated.
In all, about 10,000 students — largely poor, predominantly African American, often lagging years behind in terms of academics — would be the test subjects.
In more ways than one.
The system itself would be unique, with all strings leading back to the governor.
The legal loophole through which the EAA slipped into being is a little-used state law that allows two units of government, acting in cooperation, to create a third public entity. It this case, it was Detroit Public Schools (DPS) — under the control of a Snyder-appointed emergency manager — and the Eastern Michigan University Board of Regents, the majority of whom are gubernatorial appointees, that entered into what's called an inter-local agreement that created the EAA.
It is overseen by an 11-person board, with the governor appointing seven members and EMU and the DPS's emergency manager each selecting two more.
And so this became the test of a completely new system of schooling.
It turned out to be another kind of test as well.
A test of software, developed by one for-profit corporation and marketed by another.
A product named Buzz.
LANDING IN DETROIT
It came to Detroit from Kansas City, along with John Covington, the controversial figure hired by the EAA board to be the new system's first chancellor. Along with Buzz, Covington also brought to Detroit a group of administrators who worked under him in Kansas City. A key member of that team is Mary Esselman, first hired on as the EAA's Chief Officer, Accountability, Equity, and Innovation for the EAA, and later promoted to the position of deputy chancellor.
Covington is gone now, having departed under a cloud of scandal generated by news reports of the credit card spending that occurred under his watch.
But the software remains, significantly upgraded twice since it arrived. Those upgrades were made possible because of the students and teachers at the EAA, who were bitten again and again and again by the many bugs that plagued Buzz for the first two years of its use in Detroit.
Created by a Utah-based company called Agilix Labs, Buzz is education software that provides what its marketing material describes as an individualized learning experience. With the help of $100,000 from the EAA, Buzz was merged with other educational software created by the School Improvement Network [SINET], also based in Utah. Another $250,000 from the EAA would eventually pay for improvements suggested by the teachers, students, and administrators who were using it, according to Esselman.
In December 2013, Gov. Snyder had a number of questions about all of this, including how these products, in conjunction with the EAA model, might be taken statewide.
"The Buzz student-centered learning platform is a joint project between EAA, School Improvement Network and Agilix," according to a written response crafted by Esselman and company representatives.
How do we know that?
The information is found in internal EAA documents obtained through a Freedom of Information Act request filed by Thomas Pedroni and shared with the ACLU of Michigan. Pedroni is an associate professor of curriculum studies at Wayne State University. He's also the director of the Detroit Data and Democracy Project, which he founded in 2011 to "provide timely policy briefs, public testimony, and authoritative perspective on education issues for regional education reporters, community leaders, and community-based organizations."
For those who have followed the EAA saga at all, it's no secret that Pedroni, an expert on urban education, is a critic of the EAA's methods.
The documents he obtained, however, speak for themselves. There are thousands of them — including the email from Snyder, which asks about Agilix, the School Improvement Network, their relationship with the EAA, and how the whole model might be expanded beyond Detroit.
The dual-crafting of the response to Snyder's query, involving a back-and-forth between Esselman and company employees with edits taking place at each stop, reflects the close working relationship between the three entities.
Sometimes it's difficult to tell where the interests of one stops and the other begins.
For example, the response to Snyder points out that Esselman is working with SINET on something called a LumiBook that will describe her vision of the "student-centered" learning process. (A LumiBook, according to the School Improvement Network website, is "the next step in the evolution of the ebook — an online reading platform that lets you become part of the book — developing ideas along with fellow readers and the author that just might become the book's newest chapters.")
In terms of Buzz and the School Improvement Network platform it's been wedded with, the software has "been developed to be agnostic to instructional delivery and resource source, which means it can be used for virtual schools, blended instruction, distance learning, traditional instruction with differentiation, and online assessment," the governor was told.
A product at the forefront of education in the Internet age — at least as it was portrayed to Michigan's self-described "nerd" governor.
But in reality, what internal EAA documents reveal is the extent to which teachers and students were, over the course of two school years, used as whetstones to hone a badly flawed product being pitched as cutting-edge technology.
In fact, a SINET employee in November of 2013 informed Mary Esselman of his "fear" that another school district might want to start using Buzz (re-branded as GAGE for the purpose of marketing the product to others) before a second major upgrade could be finished and ready for use in March of the following year.
Records show that such an upgrade did finally land in April of 2014, and was installed over spring break. Another two months passed before a press release was issued announcing that the upgraded product would be available to selected school districts for the start of the 2014-2015 school year.
Agilix and the School Improvement Network began working with Covington and his team in Kansas City.
"In Kansas City, the leadership team implemented the model with limited technology...," according to the response provided to Snyder. "In Michigan, they have had the opportunity to select staff and leverage a strong teaching and learning platform with strong, short-cycle innovation."
By short-cycle innovation they mean this: improvements were made as Buzz moved from Kansas City (where it is no longer used) to Detroit. And in the two years since its arrival here, it has gone through technological upgrades significant enough to warrant press releases heralding the breakthroughs that were achieved.
"We're building this plane as we fly it," is a phrase numerous sources we've interviewed have attributed to Mary Esselman, who was in the thick of the technological planning.
Part of that build-it-as-they-go model included paying inexperienced Teach for America instructors to provide curriculum content that was loaded into Buzz when it arrived at the EAA. They were recent college grads who didn't study to become teachers and who lacked certification, coming to the EAA with only a few weeks of training in the art of teaching. (About 25 percent of the EAA's teachers were from TFA when its schools opened in 2012.)
"We selected eight teachers, some newer teachers and some more experienced, to provide additional choice options for students on a stipend basis," said Esselman, who provided a written response to questions from the ACLU. "Four were TFA teachers."
That's anything but usual.
"Typically in schools, curriculum is developed via teams of teachers working under the guidance of a curriculum specialist (someone with many years of teaching experience and a master's degree but we are now consistently seeing people with doctoral or education specialist degrees in these positions)," according to Stephen A. Wellinski, an associate professor of teacher education at Eastern Michigan University. "Usually these teams are built around the more experienced teachers. It is problematic when novice or non-teachers are in charge of creating school-based curriculum and course content."
What is the response of the EAA to criticisms that it has too many inexperienced teachers and a stubborn reliance on educational software that was rife with problems?
In essence, they say, the real proof is in the results. And, within months of beginning operation, key administrators were already proclaiming that, according to internal testing, their new approach was achieving remarkable results.
The stunning scores coming out of the EAA were eagerly touted by Agilix and the School Improvement Network.
"During the 2012-2013 school term, 59% of students achieved 1.5 or more year's growth in Reading and 58% of student achieved 1.5 or more year's growth in Math," boasts a "case study" posted on the Agilix website.
Those impressive gains, said the people running the EAA, were achieved by combining a change in philosophy with tailor-made technology. EAA students have longer school days and a longer school year than their DPS counterparts. They also had schools that embraced Student Centered Learning, described as an approach that, using the definition employed by the EAA, met students where they were academically rather than forcing them to march lock-step through the traditional, age-based grade system.
And then there was the technology provided by Agilix and SINET. Instead of a teacher with a marker standing in front of a whiteboard and students with textbooks dutifully taking notes, the computer would provide the bulk of their educational experience. Teachers would serve primarily as an adjunct, helpers who facilitated and provided a watchful eye. Teachers too had software to assist them in their professional development, a SINET product called PD360 — with PD standing for "professional development."
Before long, those two companies partnered, with taxpayers kicking in $100,000 to pay for Buzz and PD 360 to be merged, allowing it to be marketed as a single product for customers interested in a package deal. (Contrary to what EAA documents say, a SINET executive told us that Buzz and PD360 had been merged in Kansas City.)
Moreover, key EAA officials devoted much time and energy to promoting both the EAA model and the technology they were using to achieve truly remarkable success.
A symbiotic relation emerged. The companies needed the EAA's students to do well in order to prove the effectiveness of their products when making sales pitches to other schools and districts; the EAA needed the companies to do well, so that more money could be funneled back into the product improvement that would, conceivably, promote the kind of student advancement that would attract even more kids, and the state funding attached to them, to the EAA classrooms.
None of this is idle speculation or hyperbole from critics.
The emails provided us clearly show company officials time and again asked top EAA administrators — especially Covington and Esselman — to assist in efforts that would help promote their products, from making speaking appearances at conferences to coming up with data on a moment's notice to help provide fodder to an influential blogger. Promoting the work being done by the EAA, and promoting the products being used, were all one and the same.
As for the promises that successful sales would return to the district, there is this telling email from a Agilix official to Mary Esselman:
"Sorry I've been so out of touch the past several months — though don't think you and the EAA aren't top priority — it's nearly all I discuss, promote and focus on. I've just had to make sure SINET is owning the project, getting familiar and able to support and drive it so I can focus on driving sales that drive more money to perfecting your model."
In other words, achieving stellar test results was vital to the success of the EAA model. And the scores really shined, reflecting remarkable success.
At least the ones obtained through something called the "Scantron Performance Series," tests internally administered by the EAA.
There is, however, a shadow that's been cast over all those sparkling Scantron scores.
Numerous teachers interviewed by the ACLU told us that, because of intense emphasis on producing positive test results, students were allowed to re-take tests when they failed to perform well. It was described as standard procedure throughout most, if not all, EAA schools.
Asked by the ACLU if she ever became aware of these types of improper testing procedures, Esselman responded: "Yes. We were made aware at a public meeting and immediately made the necessary steps with our school leaders to address this issue."
Adding more darkness to those shadows is this fact:
In stark contrast to the internal test results are the state's standardized achievement tests, known as MEAP. The most recent MEAP results show that a high majority of EAA students are either stagnating in terms of reaching math and reading proficiency, or falling even further behind.
Asked about those contradictory results, Esselman responded: "The EAA has adopted a student-centered model of teaching and learning that allows teachers to personalize instruction to meet the needs of each child. Our teachers are focused on individual student progress rather than teaching to the state test."
The irony in that is that persistently low MEAP test scores were the reason schools were taken out of the Detroit Public School System and put under the control of the EAA.
Even so, for Cory Linton, an executive vice president at School Improvement Network, the proof of the EAA's success and the success of the software that is a focal point of its system, is in those Scantron scores.
"These kids are getting the best educations of their lives," he said in a phone interview.
When asked about the many bugs in Buzz when it arrived in Detroit, he said those kinds of problems are typical when software gets rolled out. It is part of the process. He also contended that the emails we've examined only reflect postings to a help desk set up specifically to deal with problems quickly and efficiently.
It's not much different than people having problems with an iPhone, he explained. There are bound to be some problems when using any technology, no matter how well it is designed. And that it's not fair to focus only on the problems being emailed to a help center, because it distorts the picture, providing an inaccurate portrayal of what was going on in the big picture, or how well the software works and how effective it is overall.
One thing that he's wrong about, though, is the scope of the documents used to compile this report.
BIG PICTURE, BIG PROBLEMS
Rather than just pleas for help in dealing with software snafus, many of the emails we examined dealt with bigger-picture issues involving Esselman and the two companies.
One particularly damning email actually came from a SINET employee.
In November 2013 — at which point Buzz had been the keystone of education in the EAA for 14 months — the School Improvement Network's Brian Chandler sent a message to the EAA's Mary Esselman saying this about the work Agilix was doing, and the performance of Buzz, which by that point had been upgraded twice since first being introduced in Kansas City:
"We have reiterated to them [Agilix] again and again they have done a terrible job meeting schedules and expectations. ... In the end, they have finally admitted the current version of Buzz is little more than a hack that really can't be extended much further without a significant rewrite."
"It is hard to defend language like that," Linton admitted.
Esselman's reaction at the time, recorded in an email, was to say she found Agilix's self-evaluation of Buzz to be "hugely concerning," and that her first call about the situation would be to the "governor."
It's not known if that call was ever made, or what the governor's reaction might have been.
But that same email had Esselman being told that the big fear, at least in the eyes of this SINET employee, was that sales efforts in Kentucky would bear fruit too soon, and they would be forced to deliver the "hack" model currently being used in Detroit rather than the upgrade expected to be installed in Detroit in about four months' time.
Another four months of some of Michigan's most vulnerable students being forced to use a product that, as the SINET employee admitted to Esselman, was clearly not "satisfactory to your needs at the EAA."
In a long exchange of emails with the ACLU, SINET Vice President Cory Linton wrote:
"SINET never believed it was a hack. Unfortunately, the email does exist — but that was not an official SINET position. To me, the most important question is what is the impact on the students? Are they having a better learning experience or not? I realize this is very debatable and the data can be interpreted many ways — this is why it is so important to go visit the schools and talk to the students directly, so you can make your own judgment about whether the schools are being successful or not. I can tell you unequivocally — as an official spokesperson for the company — that SINET's motive from the beginning was to help these disadvantaged students have a positive learning experience, and be prepared for a successful career or college after leaving these schools."
As for talking with teachers and students, we have done that.
During the course of our investigation, the ACLU interviewed a dozen current and former teachers, one former administrator, and several students. Many of the teachers asked to remain anonymous, saying that, because they lacked the protections afforded by a labor union, they feared retaliation by the EAA if they spoke on the record. Others feared that the EAA would blackball them. Others, however, did agree to go on the record.
One of them is Jordan Smellie, a tech-savvy former music teacher now working in the IT industry.
"Buzz overall I would describe as a travesty. To say it was incomplete when it arrived is giving it too much credit," he said. "The software was in a state that any other firm would have never released. The design was poor, front to back, top to bottom. The user experience was horrendous. It was incredibly slow, if it worked at all."
Nearly everyone interviewed talked about the dearth of content when Buzz first arrived in the fall of 2012, which is contrary to Mary Esselman's unequivocal written assertion that Buzz arrived on time and fully formed.
Two of the teachers we interviewed — both of whom no longer work for the EAA — provided signed affidavits detailing accounts of their EAA experiences. One of them is Delbert Glaze, who formerly taught at Nolan Elementary School.
He recounted for us how EAA administrators had thrown into a Dumpster textbooks left behind by DPS as it vacated the building to make room for the EAA. So he had to use Buzz. But by January of 2013, he noted, "once the Buzz platform had advanced through all of the Web pages for a particular unit, Buzz instructed the students to complete a multiple choice test. When the test was available, students were allowed to re-take the test as often as necessary and thus ultimately figure out the right letter to mark and move toward the next lesson. However, at various points throughout the year, the tests would disappear from the Buzz platform. As a result, students were unable to progress in their curriculum and, in some instances, were forced to completely restart the lesson and thus fall several weeks or months behind."
Instances of test scores and other material disappearing are well-documented in the emails obtained through the Freedom of Information Act documents that serve as the foundation of this report.
Brooke Harris, who formerly taught at Mumford High School, signed a statement that mirrors many of the statements made by Glaze.
Among other things, she noted that during the first year Buzz was being used, "The Buzz platform did not have content for certain elective classes for the entire school year."
Despite that, she stated, administrators either applied heavy pressure or offered incentives for teachers to abandon traditional teaching methods and embrace Buzz and its companion technologies.
"I was told that in the student-centered model, my role as a teacher was primarily to supervise students to make sure they were using Buzz."
And then there are the first-person accounts of life in an EAA school provided to us by the three children of Detroit resident Christina Lee. Sitting on a couch in the living room of their rented home on Detroit's west side, they explained why their mom pulled them out of Nolan, a school for children in kindergarten through eighth grade, after attending for two years.
"Nobody did what they were supposed to do," said Destiny, 15. "Kids were getting on Facebook, Instagram, porn sites."
"Yeah, you know, like the ones where you do video chat."
("Buzz is a digital platform that functions inside of the firewall that is provided through the Detroit Public Schools IT network. The firewall is in place to filter inappropriate content. This firewall was in place at the time of launch and continues to be place today," Mary Esselman responded in writing when asked if children using Buzz were able to access pornography sites.)
As with teachers who left the EAA, she might be classified as "disgruntled." She was caught using a phone to videorecord girls staging a fight in a bathroom. She was, according to her mother, suspended for 180 days.
"I know I did wrong," Destiny said. "But 180 days? I begged them to let me come back to school, but they wouldn't let me."
Deon, 14, said, "I didn't get to learning anything. If there's a teacher to help me, I can catch on. But being on a computer all the time doesn't help me. I was doing basic math, over and over. Simple stuff, like what is 3 + 3. Just sit on the computer all day."
The youngest, Aynnya, now 7, recounted the challenge of trying to get help working on a computer. Part of that stems from the fact that her family has no computer at home.
"I'd tell my teacher, 'I don't get this,' and she'd say, 'You have to figure it out for yourself,' and I'd say, 'But I don't get it,'" she recounted, her eyes catching fire from the anger that experience still kindles.
In an email responding to questions about bugs in Buzz, Cory Linton explained that there are bound to be problems, and that is the nature of the software business.
"Like all software, as you have more usage, you continually refine and make it better. This is software industry standard practice," he wrote in an email responding to questions from the ACLU. "A best-selling business book, The Lean Startup, explains this process in detail. But the basic idea is that you build software, release it to end users, get feedback, make adjustments, and release again. It is an iterative process, which is why you see so many 'Updates' on applications on your smartphone.
"You can only figure out so much in a lab, and then you learn more when real users use the software. This doesn't mean the users are guinea pigs. I spent 9 years at Microsoft, and many years doing this exact process in the Microsoft Word development team. With every version, we would immediately visit clients and see what they wanted changed/improved, and then incorporate that feedback into the next version — which usually took 18-24 months to release. That process has been followed closely with 'Buzz' — with a new version coming out every 18 months or so."
Having obtained the documents much of this report is based on, Pedroni is well aware of what they contain, and doesn't easily swallow claims that what went on at the EAA should be considered the unavoidable complications that come with creating and deploying software.
"Wouldn't the 'end users' have deleted the Buzz app off their smartphone a long time ago and gone with something else?" he asked. "If the app was a 'hack' for 18 months at least, according to its developers, I would have deleted that app from my smartphone after realizing it didn't do what was being promised.
"The last thing I'd do is use it as the almost sole basis of teaching children I loved, forcing them to endure it for hours and hours a day, week after week, month after month, sacrificing them so they could make it 'better' for some for-profit, private venture. This product was supposed to already be known for lifting schools from the bottom 5 percent. Instead, the system Gov. Snyder single-handedly created brought in a product that was already in its second iteration but still a hack. My smartphone doesn't have space for that kind of poor quality and I'd never force my children to be guinea pigs to assist it in becoming more than a hack."
Linton, recognizing both the controversy and the significance of what the EAA represents, offered an idea.
"I would say, 'Let's see what the impact on the students is.' Is anything more important that that?
"What is their attitude about learning, better or worse? Are there more or less discipline incidents? Do their parents notice a difference in their children? And, most importantly, are the students learning more or less? There is much debate over the data ... we should be focusing on accurately measuring the learning so that we can see what the impact on these students is. I would say we should be pushing to have a neutral, third-party come in and look comprehensively at the impact of the EAA's new educational approaches and systems, and look at all aspects of education and the impact on the student. Such a comprehensive analysis would focus on the big picture so that we all can learn what worked in the EAA, what needs to be tweaked, and so forth — looking at such measures as student learning, student and parent attitudes, culture and climate of the school, student attitudes about their future careers and attending college, and so forth."
We agree that the parents of EAA students and the public at large need to know if the claims of astounding gains being made are actually true, and what the effects of this new learning model really are.
We also agree that more transparency is necessary.
And with Covington now gone, the EAA wants to put the focus on the future. “Everything is under review,” spokesman Mario Morrow said. “It is a new day for the EAA.”
As things move forward, however, one question needs to be anhttp://agilix.com/products/buzz/swered: How do the leaders involved in creating the EAA, the board charged with overseeing it, and the educators responsible for running it justify allowing a terribly flawed program to be used for two years on 10,000 children in schools that were already deemed to be failing?