Anti‑Patterns Where AI Breaks Systems
Artificial Intelligence promises speed, automation, and insight. Yet in real-world software projects—especially enterprise, GovTech, ERP, and industrial systems—AI often breaks systems instead of improving them.
This usually does not happen because AI models are "bad", but because they are applied with the wrong mental model.
This article documents common anti‑patterns we see when AI is introduced into production systems, why they fail, and how experienced software developers can avoid them.
1. Replacing Deterministic Logic with AI
The Anti‑Pattern
Using AI to replace business rules, validations, or workflows that already have clear logic.
Examples
- Using LLMs to decide loan approval instead of rule-based credit logic
- Replacing tax calculation rules with AI predictions
- Letting AI decide access control or permissions
Why It Breaks
- AI outputs are probabilistic, not guaranteed
- Same input may produce different results
- Errors are hard to reproduce and debug
Better Pattern
Use deterministic code for the core, and AI only for ambiguous inputs.
If the system must be correct every time, AI should not be the final decision-maker.
2. No Human-in-the-Loop for High-Impact Decisions
The Anti‑Pattern
Deploying AI that makes irreversible or high-impact decisions without human review.
Examples
- Auto-rejecting citizens’ applications
- Auto-firing alerts without review
- Automated fraud blocking without appeal paths
Why It Breaks
- AI confidence ≠ correctness
- Edge cases become invisible victims
- Trust in the system erodes quickly
Better Pattern
Introduce review thresholds:
- Low confidence → human review
- High confidence → auto-action with audit trail
3. Treating AI Output as Truth Instead of Suggestion
The Anti‑Pattern
Assuming AI output is authoritative and skipping validation.
Examples
- Copying AI-generated SQL directly into production
- Trusting AI-generated security advice blindly
- Using AI summaries as legal or policy truth
Why It Breaks
- AI hallucinates convincingly
- Errors sound confident
- Bugs propagate silently
Better Pattern
Treat AI as a draft generator, not a source of truth.
AI writes the first version. Engineers own the final one.
4. No Clear System Boundary for AI
The Anti‑Pattern
Letting AI spread across the system without defined scope.
Examples
- AI directly calling databases
- AI modifying system state freely
- AI logic mixed with core services
Why It Breaks
- Hard to reason about behavior
- Impossible to test properly
- Security risks increase dramatically
Better Pattern
Design AI as a separate component:
- Input → AI → suggestion
- Core system decides what to do
Clear boundaries make failures survivable.
5. Prompt Engineering Instead of System Design
The Anti‑Pattern
Trying to solve architectural problems by endlessly refining prompts.
Examples
- Complex multi-page prompts replacing business specs
- Prompt hacks instead of validation logic
- Encoding rules inside text prompts
Why It Breaks
- Prompts are not versioned logic
- Behavior changes with model updates
- No testability or guarantees
Better Pattern
- Use prompts for language tasks
- Use code for rules, constraints, and policies
Prompts are configuration, not architecture.
6. No Fallback Path When AI Fails
The Anti‑Pattern
Assuming AI will always be available and correct.
Examples
- System stops working when AI API is down
- No alternative flow for low confidence outputs
- No manual override
Why It Breaks
- External dependencies fail
- Latency spikes
- Model behavior drifts
Better Pattern
Always design:
- Timeouts
- Fallback logic
- Manual override paths
A system that cannot degrade gracefully will eventually fail catastrophically.
7. Ignoring Explainability and Auditability
The Anti‑Pattern
Deploying AI without being able to explain or audit decisions.
Examples
- “The AI said so” as justification
- No logs of prompts, inputs, or outputs
- No traceability for decisions
Why It Breaks
- Regulatory issues
- Operational mistrust
- Impossible post-incident analysis
Better Pattern
Log and store:
- Inputs
- Outputs
- Confidence scores
- Decision paths
If you cannot explain a decision, you cannot defend it.
8. Using AI to Hide Broken Processes
The Anti‑Pattern
Applying AI on top of poorly designed workflows.
Examples
- Using AI to clean inconsistent data instead of fixing sources
- Using AI chatbots to compensate for bad UX
- Using AI summaries to hide missing integrations
Why It Breaks
- Technical debt increases
- Root causes remain unsolved
- Costs grow silently
Better Pattern
Fix the process first, then apply AI to amplify it.
AI magnifies system quality—good or bad.
9. Measuring Success by Demos, Not Reliability
The Anti‑Pattern
Declaring success because the demo looks impressive.
Examples
- No load testing
- No failure testing
- No long-term evaluation
Why It Breaks
- AI behaves differently in production
- Edge cases dominate at scale
- Small error rates become large incidents
Better Pattern
Measure:
- Error rate
- Recovery time
- Human intervention frequency
Production reliability beats demo brilliance.
Final Thought: AI Does Not Break Systems — Misuse Does
AI is neither magic nor dangerous by itself.
It breaks systems when:
- Developers surrender responsibility
- Architecture is replaced by prompts
- Uncertainty is treated as certainty
The role of experienced software developers is more critical than ever.
Your job is no longer to write every line of code.
Your job is to design systems that remain trustworthy even when AI is wrong.
That is real engineering in the AI era.
Get in Touch with us
Chat with Us on LINE
iiitum1984
Related Posts
- AI 反模式：AI 如何“毁掉”系统
- 为什么我们不仅仅开发软件——而是让系统真正运转起来
- Why We Don’t Just Build Software — We Make Systems Work
- 实用的 Wazuh 管理员 Prompt Pack
- Useful Wazuh Admin Prompt Packs
- 为什么政府中的遗留系统替换往往失败（以及真正可行的方法）
- Why Replacing Legacy Systems Fails in Government (And What Works Instead)
- Vertical AI Use Cases Every Local Government Actually Needs
- 多部门政府数字服务交付的设计（中国版）
- Designing Digital Service Delivery for Multi-Department Governments
- 数字政务服务在上线后失败的七个主要原因
- The Top 7 Reasons Digital Government Services Fail After Launch
- 面向市级与区级政府的数字化系统参考架构
- Reference Architecture for Provincial / Municipal Digital Systems
- 实用型 GovTech 架构：ERP、GIS、政务服务平台与数据中台
- A Practical GovTech Architecture: ERP, GIS, Citizen Portal, and Data Platform
- 为什么应急响应系统必须采用 Offline First 设计（来自 ATAK 的启示）
- Why Emergency Systems Must Work Offline First (Lessons from ATAK)
- 为什么地方政府的软件项目会失败 —— 如何在编写代码之前避免失败
- Why Government Software Projects Fail — And How to Prevent It Before Writing Code
