How to tell if EdTech is working (and what to do when it isn't)

How to tell if EdTech is working (and what to do when it isn't)

Why usage isn't the same as impact

Schools have spent billions on devices, apps, and platforms over the past decade. Ask most administrators or teachers whether any single tool is actually improving learning, though, and the honest answer is usually a shrug. The numbers that get reported (logins, minutes on screen, licenses activated) measure usage, not efficacy, and the two get conflated constantly. Usage tells you students showed up. Efficacy tells you they gained measurable skills and understanding while they were there. A tool that keeps students busy and quiet is not the same as a tool that teaches, and that difference is what separates an expensive digital worksheet from something that actually moves learning forward. This article walks through the concrete signs, evidence standards, and pilot methods you can use to evaluate any EdTech product, whether it's already deployed in your district or still sitting in a sales deck.

Key takeaways

• Usage metrics like logins and screen time tell you a tool is being used, not that it's teaching anything.

• EdTech that works pushes students to create and problem-solve rather than click through fixed content.

• Effective tools adapt to individual skill gaps and give teachers measurable time back instead of adding work.

• Vendor marketing isn't evidence; look for independent research, ESSA tiers, and transparent methodology.

• A controlled pilot in your own classrooms, with pre-agreed performance targets, is the final test that matters.

What "working" actually means: engagement vs. efficacy

Districts tend to default to one of two metrics when they evaluate technology. The first is engagement: time on task, login frequency, completion rates. The second is efficacy: actual learning growth, skill mastery, and whether students retain what they learned. Engagement is easier to measure, so it dominates most dashboards, but on its own it's misleading. A student can rack up forty hours in a platform that only auto-grades multiple-choice quizzes and walk away with no deeper understanding than when they started. The Office of Educational Technology has made this point directly: the nature of technology use, not the amount, is what determines its value to learning.

A useful mental model here is the digital worksheet test. If a tool only digitizes something a teacher could already do with paper, it isn't adding pedagogical value, no matter how polished the interface looks. And the stakes run higher than wasted budget. Research on EdTech adoption has found that unproven tools can widen gaps between student groups when their claimed benefits don't hold up in practice, which makes evaluation an equity question as much as a financial one. That's why independent, published efficacy research matters so much in this space, and why so few vendors offer it. The five signs that follow are a practical checklist for making the shift from "is it being used" to "is it working."

Sign #1: students are creating, not just consuming

You can learn more from watching what students do inside a tool than from any usage report. Passive use looks like reading flat PDFs, clicking through a fixed quiz sequence, or completing digital worksheets; active use looks like building, writing, coding, and solving problems that don't have one right answer.

1: Passive use indicators

Watch for these patterns. Students interact with the tool only to take standardized multiple-choice tests or finish auto-graded digital worksheets. Content arrives in a fixed, linear sequence with no room for exploration or student choice. And the software's main output is a score or a completion percentage rather than something a student made. Any one of these alone is a yellow flag. All three together means you're paying for a worksheet with a login screen.

2: Active, working indicators

The healthier pattern looks different. Students use the tool to build original multimedia projects, write, code, or design solutions to open-ended, real-world problems. The software prompts them to apply knowledge practically instead of just recalling it, which mirrors the project-based and inquiry-based models researchers consistently point to as effective technology integration. The teacher's role shifts too: facilitating collaborative, student-centered work rather than administering a device. Tools like SchoolAI's Spaces are built around this kind of teacher-designed, open-ended interaction.

3: The critical-thinking check

The deepest version of this sign is whether the software challenges students to problem-solve or just walks them down a predetermined click path. Vendor claims about "deeper thinking" should be backed by independent studies that actually track student reasoning. Research examining whether SchoolAI makes students think is one example of the kind of evidence to look for: studies measuring whether an AI tool deepens reasoning rather than replacing it.

Sign #2: the tool adapts to each student, not the whole class

Call this the conveyor belt test. If every student in a class moves through the exact same linear sequence regardless of what they already know, the tool isn't personalizing anything; it's automating a single pace for everyone. Effective EdTech should accurately assess prior knowledge before deciding what a student sees next, instead of assuming a uniform starting point. A class of thirty students has thirty different starting points, and software that ignores that is just a faster conveyor belt.

The real test comes when students diverge from the expected path. When a student struggles, the software should automatically surface the foundational sub-skills needed to close that specific gap, not repeat the same content at the same difficulty and hope for a different result. When a student demonstrates mastery early, it should pivot to advanced, higher-order challenges instead of making them idle on the same track as their peers. That's what personalized learning means in practice: the path changes based on what each student needs next.

Adaptive pathways matter because the goal was never finishing a module. The goal is closing individual skill gaps over a semester or a year, and that longitudinal view is what separates real efficacy research from a one-time usage snapshot. Look for outcome data at that scale before scaling a tool further. A multi-school study of regular SchoolAI users, for example, examined whether adaptive use correlated with stronger engagement, deeper thinking, and student confidence. That's the kind of evidence a district should expect to see before expanding any tool beyond a pilot.

Sign #3: it gives teachers time and insight, not more work

Here's the blunt test: if a tool takes more time to manage, grade, or troubleshoot than it saves, it is not working, no matter how strong its learning claims are. Effective EdTech should visibly cut the administrative load in lesson planning, grading, and data aggregation, and that reduction should be measurable rather than anecdotal. "Teachers seem happier" is not a metric. Survey data on how teachers actually spend the time they get back after adopting an AI tool is, which is why that research exists for SchoolAI specifically. Time-saved claims are easy to make and hard to verify, so ask vendors for teacher-reported data before you believe the brochure.

The second half of this sign is what the tool gives back. Instead of a vague score or percentage, it should generate clear, actionable dashboards showing exactly where each student is stuck and why, so a teacher can act on the data the same day. In practice that means flagging the specific standards or skills a student hasn't mastered yet, not just an overall grade, which is what a real-time view like Mission Control is designed to surface. And the time saved on routine grading should land somewhere specific: one-on-one and small-group instruction. Automation is supposed to free up relationship-building, not replace it. If teachers get time back but nothing changes in how students experience the classroom, the tool is optimizing the wrong thing.

Sign #4: it's backed by real evidence, not just marketing

Most EdTech brochures claim to "improve outcomes" without citing a single independent study, so evaluators need ways to check claims that don't rely on the vendor's own word. Several frameworks and clearinghouses exist for exactly this purpose:

• ISTE Seal / EdTech Index. Check whether a product has earned the ISTE Seal, which validates usability and educational design against an established external standard rather than a vendor's self-assessment.

• ESSA tiers and clearinghouses. Resources like Evidence for ESSA show whether a product has undergone independent randomized controlled trials (RCTs) demonstrating impact. Know the tiers before you read the label: strong evidence means multiple RCTs, while promising means correlational data only. Both get marketed as "research-backed."

• The Triple E Framework. Ask whether the tool Engages students in the stated learning goal, Enhances their understanding of it, and Extends that learning to real-world application. Those are three separate questions. Plenty of tools pass the first and fail the other two.

• A science-of-learning caveat. No single method is sufficient on its own. RCTs show whether a tool works on average across a large sample; co-design and classroom-based research show why and how it works in a specific context. Strong evaluations combine both rather than leaning on one.

• The transparency check. Favor vendors who publish their methodology, sample sizes, and limitations openly instead of sharing only favorable summary statistics. That openness is the marker of legitimate research practice versus marketing dressed up as research, and it's the standard behind SchoolAI's published research practices.

Sign #5: it holds up in a controlled pilot in your own classrooms

Published research and third-party certifications show a tool can work somewhere. Only a pilot in your own district shows whether it works in your classrooms, with your curriculum and your student population. Structure it like an experiment: introduce the tool to a targeted subset of comparable schools or classrooms while a matched control group continues without it, so any difference in outcomes can be attributed to the tool rather than to unrelated factors. Then tie the vendor contract to specific, pre-agreed performance targets verified during the pilot, instead of renewing based on satisfaction surveys. That one move shifts accountability from the sales team to the classroom data.

Quantitative results aren't the whole picture, though. Collect student testimonials that describe which specific concepts they mastered or how their thinking changed, not surface-level feedback about points and badges. And track implementation fidelity alongside student outcomes: even a proven tool will underperform if it isn't used the way it was designed to be used, so record how closely teachers followed the intended model. A pilot that measures outcomes and implementation together tells you whether the tool failed or the rollout did. Those are different problems with different fixes.

Turning these signs into a simple EdTech audit

The whole audit comes down to a shift in the question. Not "are students logged in," but three harder ones: are students demonstrably learning more, are teachers demonstrably freed up, and is there independent evidence behind the claims? The five signs above work whether you're evaluating a tool already deployed across your district or vetting new software before a purchase decision. Same checklist either way, and it fits on one page.

There's one honest complication. Most instructional AI tools on the market are built for open-ended engagement rather than guardrailed, curriculum-aligned use, which makes them hard to evaluate against these signs because there's no structured mastery data to check in the first place. SchoolAI was designed around that exact problem. Students get safe, teacher-designed, guardrailed AI experiences, and educators get real-time mastery data showing whether students are hitting the outcomes that matter, turning the five signs above from a manual audit into something a dashboard shows automatically. If you're ready to test it against your own criteria, request a demo or sign up today. And if your team is planning a district-wide rollout, SchoolAI's implementation resources are the place to start.