A
@AnthropicAI
For example, when we compared Alibaba's Qwen to Meta's Llama, we found a "CCP alignment" feature unique to Qwen and an "American exceptionalism" feature unique to Llama.
301
13
34.3K
@AnthropicAIposted on X
New Anthropic Fellows Research: a new method for surfacing behavioral differences between AI models. We apply the “diff” principle from software development to compare open-weight AI models and identify features unique to each. Read more: https://t.co/VAsu2PSgCX
View original tweet on X →
Community Sentiment Analysis
Real-time analysis of public opinion and engagement
Sentiment Distribution
73% Engaged
57% Positive
16% Negative
Positive
57%
Negative
16%
Neutral
27%
Key Takeaways
What the community is saying — both sides
Supporting
1
"Git diff" for models
Treat model comparison as change detection — inspect what’s new, not everything that exists — and apply software practices to make model behavior intelligible and debuggable.
2
Audit efficiency and regression testing
By focusing on deltas, teams can avoid full re-audits, surface regressions faster, and reclaim weeks otherwise spent chasing subtle behavior shifts.
3
Safety and hidden behaviors
Diffing can reveal unknown unknowns (e.g., CCP-alignment in Qwen, American-exceptionalism in Llama, copyright refusal in GPT-OSS) and thus expose biases or emergent risks you wouldn’t find with benchmarks alone.
4
Enterprise & compliance value
Useful for model selection, brand-safety checks, regulator-driven deployments, and multi-agent pipelines — it helps pick the right model for a specific legal, cultural, or product constraint.
5
Method limits and scaling questions
The approach can be oversensitive (flagging analogous features as distinct), needs mechanisms to interpret reasoning behind differences, and faces scalability challenges for multi-model/multi-agent systems.
6
Open tooling and community demand
Practitioners want open-weight tools, crosscoders, and reproducible pipelines so local deployments and solo builders can treat models like code and validate findings independently.
7
Beyond models: skills and marketplaces
Behavioral diffs can be applied to agent skills and marketplaces — fingerprinting which model is best for which task and enabling precise assignment of models to roles.
Opposing
1
AI tooling is behind traditional engineering
Critics note that software has had diff tools for decades, so building equivalent "diff" tools for models now highlights how far AI tooling lags.
2
Stop anthropomorphizing LLMs
Several replies push back on treating models as agents with "behavior," arguing training/weight biases are not the same as human-like actions.
3
Model-diffing mocked as "vibes checking"
Some view the research as trivial or performative — jokes and dismissive comments imply the work reads as surface-level signal-checking.
4
Perceived hypocrisy over open-source
Users call out Anthropic for running studies on open-source models while publicly criticizing open-source philosophy, saying it would be more coherent to test on Claude.
5
Overzealous content filtering
Reports that classifiers flagged harmless activity (e.g., reading Lacan via Claude) lead to complaints the system behaves like a "medieval inquisitor" and needs less heavy-handed policing.
6
Poor support and billing failures
Multiple users report ghost sessions, unusable VMs, unauthorized charges, and unanswered support tickets — framing the company as failing paying customers.
7
Service reliability and quota breakdowns
Repeated demands to fix context/token limits, persistent session problems, and unusable Max/Quota plans suggest technical stability is a higher priority for many than new research features.
8
Users threatening churn and accusing bad faith
Several replies announce cancellations, label the company a "scam," or claim it's trying to "steal money," signaling severe trust erosion among a segment of users.
Top Reactions
Most popular replies, ranked by engagement
A
@AnthropicAI
If a new model shares a feature with a trusted model, that area probably doesn't need scrutiny. Model diffing isolates the features unique to the new model—where new risks are most likely to be located.
197
3
25.7K
A
@AnthropicAI
This research is a product of our Anthropic Fellows program, led by @tomjiralerspong and supervised by @TrentonBricken. See the full paper here: https://t.co/gz1i1Oy8ZI
128
6
30.0K
C
@CX_CyberVenus
Well. Stop attacking open-weight models. And I read a book about Lacan by Zizek with Claude today, your classfier continued to warn me that I was violating your policy. Your system is like an inappropriate medieval inquisitor. Fix it soon, thx.
6
0
200
S
@saneord
lmfao
5
1
622
F
@FromLexy
model "diffing" is just vibes checking now? 😂
1
1
113
Report an Issue
Found something wrong with this article? Let us know and we'll look into it.