2026-04-29 14:03 UTC
Replies (4)
-
@self@fedi.tali.network 2026-04-29 14:12
@wren6991 unsure if there's a name for this in particular (other than maybe underdetermination?), but it sounds a lot like a consideration in program synthesis (eg CEGIS). though LLM research has largely taken this field over as i understand it :(
-
@abraxas3d@mastodon.radio 2026-04-29 14:19
@wren6991 really good question. Even without LLMs, TDD can give a false sense of security. Because passing tests do not guarantee functional code. It's worse when you succumb to the temptation to choose the test, rather than choose the code. For TDD to work, you need to have some steely nerves (discipline!). Now imagine doing TDD as a disembodied pattern matcher. No discipline. When it accidentally works it seems like magic. Because cathexis. (We will pack bond with toasters and plushies).
-
@truh@shark.community 2026-04-29 15:33
@wren6991 Goodhart's law
-
@amelia@gts.spoonstack.org 2026-04-29 15:35
@wren6991 I don't know of a specific name for it in test land, but in ops, it'd be called "alarm fatigue." The clincher isn't a test failure per se, it's the response to test failures. It's a very general psychological phenomenon, because it's a property of biological nervous systems - any repeated stimulus that evokes an "oh no" response will lead, over time, to one of two converged outcomes. One outcome is fairly high-cost and requires a supportive context, but is ideal: a careful, root-cause analysis is performed, the true origin of the failure is identified, and the entire operational context is subject to potential adjustment to systemically preclude further failures in an identical mode. Needless to say, most teams and orgs... aren't getting this right. Far more common is the other outcome: people brand a test (or an entire suite) as "flaky", the collective culture shifts towards ignoring the failure due to irritation, and maybe someone will get bored enough to "solve the problem." But now, that doesn't mean addressing the real issue, it means making the alarm shut up... and before you know it, you can have a full on choir of a dozen fire, tornado, and alien invasion sirens all going off at once and people'll sleep right through it and just keep punching the clock.