STEM.org Insights

STEM.org Insights

When the Prototype Works but the Learning Doesn’t

A successful build is only part of the story. Make engineering judgment visible through better questions, tests and trade-offs.

STEM.org Educational Research™

The vehicle crosses the finish line. The bridge holds the load. The light turns on. Everyone has a photograph of something that works. Then someone asks why it worked, what the test established or whether it would work under different conditions. The room becomes quieter.

A successful prototype is evidence that a particular arrangement performed in a particular situation. It is not, by itself, evidence that its builders can explain the mechanism, compare alternatives or make a defensible design decision. Treating those outcomes as identical leaves some of the most valuable STEM learning unexamined.

For a school, STEM business or community workshop, this is a consequential design choice. The final demonstration can be the end of the activity, or the beginning of the argument learners must make about their design. The second option gives an experienced facilitator considerably more to work with.

“It works” is an incomplete specification

Imagine a tabletop delivery challenge: build a model vehicle that carries a small payload across a marked indoor route. The group is given a material allowance and a requirement to keep the load inside a defined delivery area. This is an illustrative teaching scenario, not a tested program or a claim about participant outcomes.

One vehicle travels farther. Another arrives more consistently. A third carries a heavier load but uses substantially more material. Which is better? Until the learners establish the purpose and constraints, the question has no defensible answer. Rewarding the longest trip simply makes distance the hidden objective.

Engineering judgment begins with deciding what deserves priority and what cannot be compromised. The NGSS high-school engineering design expectation HS-ETS1-3 explicitly connects evaluation with criteria, constraints and trade-offs. That is a useful reference for this discussion, not a claim that a single workshop fulfills the standard.

Prototype A

Farther once

A striking demonstration can hide inconsistent delivery.

Prototype B

On target repeatedly

A shorter trip may better serve a defined delivery task.

Conceptual comparison only; the graphic does not display measured distances or actual trial results. “Better” depends on the agreed purpose and evidence.

Separate the product of the work from the quality of the reasoning

Assess the vehicle and the explanation as related but distinct objects. A working design can result from following a model, a lucky adjustment or a well-supported decision. A failed design can be accompanied by careful reasoning that identifies a genuine limitation. Neither success nor failure tells the whole story.

Ask learners to state a prediction before testing and preserve it afterward. What do they think will happen, and why? When a result differs, the point is not to extract the teacher’s preferred sentence. Ask which part of the explanation the result challenges and which additional observation could help distinguish the possibilities.

This does not mean celebrating every failure as learning. Repeated failure without a useful explanation, feedback or a change in approach can be merely repeated failure. The educational value lies in what learners can now notice, justify and do differently.

Make the next test answer a question

Suppose a group changes the wheel spacing, payload position and launch method, then obtains a better result. That may be a practical improvement. It does not isolate which change produced it. If the purpose is optimization, a package of changes may be useful. If the purpose is explaining a mechanism, that test leaves competing explanations unresolved.

Help learners choose the kind of question they are asking. “Can we meet the delivery requirement?” and “Does moving the payload improve stability?” require different evidence. A sequence can include both, but the facilitator should not treat one as an answer to the other.

Two legitimate investigations

Choose what the test should establish

Explain a design effect

Compare a planned change while keeping relevant conditions as consistent as practical.

Example: Move the same payload between two positions. Keep the route and release method consistent.

Limit: Results still depend on measurement quality, variation and the conditions tested.

Judge a working solution

Test the whole design against an agreed requirement across relevant conditions.

Example: Check whether the vehicle delivers the load reliably over repeated runs.

Limit: Meeting the requirement does not isolate why each component helped.

Repeat observations when variation matters. One successful trip cannot show consistency. Record the unsuccessful trips too, along with changes in conditions. Otherwise, the final display may tell the story of the best run while concealing the behavior that a real user would encounter.

Change the condition before declaring the idea understood

Once a design works, introduce a reasonable variation: a different payload position, a tighter delivery area or a surface with a different texture. Choose a change that connects to the intended concept and remains appropriate for the participants and setting. Ask for a prediction before allowing another rebuild.

The response gives you something a repeated demonstration cannot. Can learners use their explanation to anticipate a result? Can they distinguish what they know from what they need to test? A confident prediction that fails can be useful evidence for discussion. An unexplained correct guess is much weaker evidence of understanding.

Do not assume that a successful response in this one task proves broad transfer to every engineering problem. It gives a more informative observation within the task. Record the conditions and the support provided before making a larger claim.

Intervene without taking over the decision

The facilitator’s most useful question may be, “What would that test tell you?” It redirects activity toward evidence while leaving the choice with the learner. Other prompts can expose a trade-off: “What did you improve, and what became worse?” or “Which result would make you reconsider that explanation?”

Direct instruction has a place. Explain an unfamiliar measurement technique, demonstrate safe use of materials or clarify a concept that learners need to proceed. Withholding necessary knowledge is not the same as preserving agency. The aim is to make learners responsible for a meaningful decision they are equipped to consider.

In a mixed-experience group, allow different forms of contribution: an annotated sketch, an oral explanation, a test record or a demonstration accompanied by reasoning. Observe who makes the decisions. A well-spoken team presentation can otherwise conceal that one participant did all the interpretation.

Make the learning visible to the people choosing your program

Schools and families can readily see a finished model. A STEM business or community program can help them see the thinking as well: show the original prediction, one consequential test and the reason for a revision. Use consented, appropriately anonymized examples; do not turn a learner’s difficulty into a promotional exhibit.

That presentation makes the educational offer more specific. “Participants built a delivery vehicle” describes an activity. “Participants compared designs against a delivery requirement and justified a trade-off” describes an opportunity to practice engineering judgment. Claims about improved ability still require evidence of improvement; a persuasive display does not substitute for it.

The prototype is worth celebrating. The stronger question is whether learners can now make a decision they could not previously defend.

Explore the sources

Continue exploring

More research and practical analysis from STEM.org.

Back to STEM.org Insights
STEM WIRE:

Take the first step: Apply Online

Contact Us

  • What are you interested in?

  • This field is for validation purposes and should be left unchanged.