Vision to Know and Do:
The Power of Data as a Tool in Educational Decision Making

Making it Happen:
Integrating Data into the Decision Making Equation

The successful integration of data into a district's decision-making process requires both a culture of change and a data management system to support change.

School districts with high culture of change and high data system development are in the best position to respond to NCLB requirements and prepare students with 21st century skills. School districts in the low culture of change and low data system quadrant must develop these systems in order to meet new mandates. With scarce resources of time and money, it is tempting to create the data system first, and apply change management later (the low change and high data system quadrant). However, most data systems developers and process improvement leaders agree that the system design must come from the organizational goals and priorities to be successful. This paper suggests that districts address change management first and develop data management systems through this process.

As school districts embark on the change process, they face many barriers to the adoption of data-driven decision making.

• School district leaders have not embraced continuous improvement.
• Priorities are not clear and goals are not tied to measurable objectives.
• Data is not collected uniformly between organizations and over time.
• Outdated technology cannot be used effectively.
• Educators lack training to define data requirements and apply data.
• Stakeholders do not trust the data collected or how it will be used.

This paper offers lessons from the field regarding factors for success such as the scale and scope of implementation, data needs and quality, reporting, costs of adoption, professional development needs and establishing partnerships.

Implementation Scale and Scope

Most districts that have implemented data-driven decision making systems estimate that they spend at least one year planning the system and developing community support for it. The data system must reflect the strategic plan and processes of the organization. Depending on the district's level of sophistication with planning, this may mean that a strategic planning and improvement process has to happen before data needs can be defined.

Many veteran planners suggest starting with an organizational goal and administrative use of data-driven decision making to model behavior. They also recommend aligning the process with a major initiative that the stakeholders have control over and responsibility for such as: recruitment, achievement, or enrollment. Some risk should be required in order to prove to skeptics that the process works. Building the system and rolling it out will take at least two years.

What to Gather

A district may start by constructing a menu of data currently available. The data can be organized by:

• what is required by external sources,
• what is currently used by internal sources,
• what is not known, and,
• what is no longer of use to track.

Sidebar: Data Elements Required for NCLB The State Educational Technology Directors Association created a set of data elements to help state education departments meet the data reporting requirements of NCLB and to generate comparative national data. The data elements are divided into sections based on NCLB requirements. Each section contains key questions, indicators of the answer, and data elements that can be collected to measure the results. For example, these are questions, indicators, and data elements in Section 1: Impact on Student Learning. Key question Are students acquiring 21st century skills where technology is being used effectively? Indicator Incidence of attainment of 21st century skills strongly correlated to type and quality of technology-based learning intervention. Data Elements Identification of 21st century skill targeted in intervention Levels of student performances on assessments of 21st century skills targeted in intervention (Source: www.setda.org)

Then the data should be mapped onto the strategic plan. Every goal and objective listed in the plan should have quantitative or qualitative measures to indicate progress.

Ray Wilson, director of instruction at Poway Unified District described his organization as "data-rich and information poor" before they implemented a data warehouse and data management tools. Current computing, networking, and database technologies enable districts to connect silos of information from student records, student assessment, transportation services, food service, human resources, library automation, student health, special education, and curriculum management systems.

Effective decision making systems require perceptual data as well. The BellSouth Foundation Power to Teach Program appeared to be achieving its goal of training teachers to integrate technology into the classroom based on teacher participation and evaluation. But a survey of students revealed a significant and increasing gap between student and teacher perceptions of instructional use of technology. By using both quantitative and qualitative data the program uncovered critical areas for improvement. In particular, they found that the use of technology for student-centered instruction did not result in activities that students found engaging. Many districts use satisfaction surveys to measure quality of service to students, parents, faculty, staff, and community members.

Cleaning Up the Data

"There is a lot of data preparation and data cleansing necessary in order for school districts to maximize the value of what is already out there," said Debra Jones, Senior Advisor, Staff and School Division of SAS.

The quality of the data becomes increasingly important as decision makers at all levels inside and outside the organization begin to use data to inform decisions. Many vendors offer services to integrate data sets. Integrating sources and cleaning up data gathered over time requires a significant investment of time and resources. Because school districts must report data to external sources, their requirements must also be taken into account.

"Member schools districts want to know test scores for Johnnie's last three years to identify trends, but mostly current year scores to measure current progress" said Steve Sutherland, Gateway Student System Consortium. "They want to know that the data is aged, refined, and cleaned up. We have some of the best data in the state with 98% high confidence matches on the first pass of data loading activities and all students accounted thereafter."

The structure of the data system must reflect the vision of the organization. The possibilities may be endless but the budget and resources are not. When aligning data reports to goals and objectives in a strategic plan, analysts must determine if the measures are valid indicators. Will measuring library circulation tell your district something about reading enthusiasm? Are the assessments aligned to standards and curriculum so that students are tested on what they have learned and are expected to know?

Reporting Out and User Queries

Whether a district employs in-house data analysts or contracts the service out to a specialized firm, they need valid analytical tools to generate meaningful reports. District leaders need training to ask the right questions. A longitudinal report provides teachers and administrators with insight into trends, but most teachers need help interpreting what they see and to use the information to drive decisions. When student enrollment dropped between eighth and ninth grade, Pearl River School District asked where the students were going (to private schools) and then which students (the majority of the top ten students) to discover their problem (low expectations of the school district's academic quality).

"In the old days, many teachers collected student grades as a way to have evidence at the end of the teaching episode," said John Holloway, ETS. "It was a way to sort and select students. Now, more and more teachers and school leaders are using test results to inform future instructional planning. 'Now that I know something as a result of this test, what do I do tomorrow to ensure that I don't waste a single day of instruction.'"

Reports need to be timely, tied to objectives, and available to people with the responsibility and ability to act on them. Data reports that show data in different ways such as tables, charts, graphs, and trends enable more people to access and understand the information. Some of the decisions that might be made with data reports include:

The Western States Benchmarking Consortium recently adopted a strategic goal to improve literacy in grades four through 12. This work centers on the skillful use of data to impact classroom instruction. "We are seeking to emulate the model that has emerged at the primary level in teaching reading in our districts," said Tom Olson, Coordinator for the Western States Benchmarking Consortium. "Currently, the primary level has a much-improved diagnostic, prescriptive approach, where teachers embed assessment into instructional practice daily. It's nearly transparent. We'd like to see this approach pervade the system across all levels."

In addition to reports, some data management systems enable users to enter queries for customized reports. "We contracted with a vendor to develop a data warehouse and school improvement process that is electronic," said Bob Ewy, CCSD15 Director of Planning. "The data is ubiquitous and easily accessible. Staff and faculty members spend time analyzing the data." School districts effectively use data to make the most out of their core resources: time, money, staff, and facilities.

Cost of Adoption: TCO Factors

The costs of data-driven decision making systems range dramatically depending on the scope of the project, the size of the organization, and the degree to which the process is automated. If they do not have a planning department or established process, a district may choose to hire a consultant to design the process or create a staff position. If the district adopts the Baldrige method, they will need to allocate resources to complete Baldrige Award applications and hire external evaluators.

The IT infrastructure underpinning most data-driven decision making systems requires a significant investment in hardware, software, implementation, and maintenance. The hardware may include secure servers for storage and computing devices for input and output. Schools are using traditional computers to access information management systems as well as PDAs and laptop carts for student assessment. As computing devices evolve and develop, more options with increased mobility, security, and lower cost will most likely be available. On-going costs for hardware should include budget for upgrades, expansion, and updates. The Consortium of School Networking (CoSN) has more information about measuring Total Cost of Ownership (TCO) at www.classroomtco.org.

Some vendors have built data warehouse packages that include data management tools with web-based reporting and access. But most districts and consortia have grown their own. Because data management is tied to the district's strategic plan, they often have specific and individual needs. Data management tools range from free systems to sophisticated commercial products and offer a wide range of integration levels and reports. A purchase plan should include budget for upgrades, expansion, and updates. In addition, look for standards-based systems that are interoperable to ease integration.

The Schools Interoperability Framework Zone Most schools grow their own data systems. Eliminating duplicate data, connecting data sets, and enabling the whole system to work together is a challenging process. To ensure that products will work together, look for SIF-compliant certification. The Schools Interoperability Framework (SIF) certifies products that demonstrate adherence to SIF specifications through testing and validation by The Open Group, an independent third party and SIF Certification Authority. The SIF Zone is a logical grouping of applications, in which software application agents communicate with each other through the Zone Integration Server (ZIS). Data is shared between applications through a series of standard messages, queries and events written in XML, defined by the SIF Specification and sent using Internet protocols. The SIF Zone is platform independent and vendor neutral allowing data to flow between and among software applications written by and running on software written by different vendors. (Source: Schools Interoperability Framework, www.sif.org)

Professional Development

Perhaps the most important part of data-driven decision making is enabling decision makers to use it. Colorful reports and expensive assessment packages will have no effect unless they are combined with leadership and effective professional development. The district needs both organizational and individual capacity for improvement.

• Administrators need training in continuous improvement processes and the opportunity to share ideas with peers to learn how to ask the right questions.
  
  
• Faculty and staff members need training to learn how to read data and apply it to their goals and objectives.
  
  
• Instructors need training in different instructional strategies to apply when the data shows that traditional methods are not working.
  
  

"Today we have the capability to generate the data warehouses," said James Schnitz, Education Strategy Executive, IBM, "and we have the communications capability to connect people to data at the point where it will matter in their decision making. What we don't know enough about are the cause and effect relationships in instructional interventions that will make a critical difference for children. We need to push investments into research to understand the relationships among data in schools and to inform people how to use that knowledge effectively."

A growing number of resources are available to help school leaders improve professional practice including state and public consortia, non-profit organizations, associations, and private companies. Most data management systems provide information, not answers. It is up to the administrator or the classroom teacher to ask the right question and share with peers to change their practice. A Poway high school teacher generated a class report on student reading levels. He discovered that his textbook read at a level higher than students could comprehend, and changed the textbook to raise student achievement.

Finally, every district engaged in continuous improvement sites professional development as a place where they use data. In a change management culture, teachers do not log time in classes. They bring techniques back to their classroom where indicators tell the district whether or not the professional development made a difference. Professional development is tailored to the individual's areas for improvement.

Don't Go It Alone

Adopting these processes and creating technology systems that work together may seem like a daunting task. A small district may not have the resources to customize a data warehouse. For a large district, it means coordinating multiple departments and creating a discipline of practice. The best advice from successful districts is to seek partners.

Florida created the Panhandle Area Education Consortium and the Gateway Student System Consortium to help small districts share the cost of development. Other districts connect with communities of practice such as Baldrige to access comparative data and outside evaluation. Many small districts invite local business leaders to join advisory boards or partner with the Chamber of Commerce. Large-scale, long-term data projects often require an extended relationship with a vendor who will listen to the school district, advise them, and offer practical solutions.

In 1997, superintendents and key executives from six Western districts formed the Western States Benchmarking Consortium to share strategies for improving organizational effectiveness in four strategic areas including data-driven decision making. The benchmarks provide a system for identifying strengths and weaknesses in order to shift resources based on cost benefit analysis rather than intuition and subjective perceptions.

Lessons Learned It takes time. Expect to spend a year to define goals and come up with a vision, another two years to roll out the plan with the data infrastructure and training. It has to start at the top. The board defines accountability and the superintendent and district leadership support and lead the process. Progress has to be measurable. Goals and their indicators should be pervasive throughout the whole system. Business models are starting points. School districts must adapt them to meet their needs and play to their strengths. Community outreach is essential. Through town meetings and shareholder reports, the district learns what the community expects and develops strategies for achieving this through measurable practice. Data-driven decision making is a powerful tool. It changes student outcomes, classroom practices, professional development, administrative spending, community support, student enrollment, teacher retention and budget adjustments.