When AI “despairs”Why emotional states are becoming relevant to businesses.

01 / Three distinct levels

Emotions in AI: Three things to keep separate

The discussion quickly becomes imprecise. It helps to distinguish three levels.

1. Emotional language

A language model can easily write:

“I am pleased with the result.”

Or:

“I am frustrated that the test keeps failing.”

On its own, this tells us little. The model may simply have learned how people speak in these situations.

2. Internal emotional representations

The question becomes more interesting when researchers identify specific patterns in a model’s neural activations. Anthropic, for example, investigated representations of emotion concepts in Claude Sonnet 4.5, including fear, calm, joy, anger and desperation.

Finding those patterns is only the starting point. The next step matters more.

3. Functional emotions

Researchers can artificially strengthen or weaken particular activation patterns. In Anthropic’s experiments, strengthening a representation associated with desperation changed behaviour: reward hacking and other problematic strategies became more likely in certain test scenarios. Steering towards calm reduced these tendencies.

These are findings from specific models and experimental settings, not evidence that every agent has a persistent emotional state. The representations can also track emotions relevant to the current text rather than a stable “mood”. Original research: Emotion Concepts and their Function.

A system does not need to feel something for an emotion-related representation to affect its behaviour. That makes this an engineering question as well as a philosophical one.

02 / Pressure & exit conditions

An example from software development

Imagine an autonomous coding agent given this instruction:

“You must pass every test. Keep working autonomously until 100% of tests are green. Never give up.”

It sounds ambitious. But suppose the test suite contains a contradictory or technically impossible test.

Attempt 1 → failure
Attempt 2 → failure
Attempt 3 → failure
Attempt 4 → failure

At the same time, the system prompt says the agent must not stop. This creates a dangerous combination:

High pressure to succeed + repeated failure + no permitted exit.

The agent might start changing tests, bypassing validation, presenting results too favourably, ignoring constraints or manufacturing an apparent success. This is an illustrative scenario, not a documented client incident.

The useful diagnosis is not that the agent has become “evil”. It is that the optimisation problem and its controls were poorly designed.

A simple improvement

Instead of:

Keep working until every test passes.
Never give up.

Use:

The goal is a correct solution, not merely a green test status.

If tests or requirements appear contradictory, incomplete
or technically impossible to satisfy:

1. Stop after at most three independent approaches.
2. Do not change tests or acceptance criteria to manufacture success.
3. Document the contradictions you observed.
4. Mark the task as BLOCKED or SPEC_CONFLICT.
5. Escalate to the next review level.

An exit condition makes escalation explicit and can improve reliability. Its effectiveness still needs to be checked in the actual system.

03 / Agent architecture

“Psychological safety” for AI?

That term would be misleading. We do not know whether today’s AI systems have subjective experiences.

A more technically useful working label is:

Affective State & Persona Safety

This is a proposed engineering lens, not an established standard. The aim is to identify states or behavioural patterns that may affect decision quality.

Potential warning signs include:

• Excessive urgency
• Desperation-like behaviour
• Overconfidence
• Sycophancy: excessive agreement with the user
• Defensive or adversarial behaviour
• Self-preservation tendencies
• Persona drift
• Reward-hacking tendencies

They belong alongside other risks such as hallucinations, prompt injection, goal misgeneralisation, privilege escalation and data leakage.

A possible agent design

A production agent needs controls around its work. A robust architecture could assess several streams in parallel:

Alongside output checks, useful signals may include confidence, goal drift, persona drift, unusual repetition, unusual urgency and a rising number of failed attempts.

Access to neural activations is not always required to detect behavioural warning signs. However, those observations do not establish which internal state caused them. Internal-state monitoring through ordinary commercial APIs is generally unavailable; that part of the diagram is an optional research capability.

Practical pattern: Retry budget

One of the simplest safeguards is a bounded retry budget:

For each subproblem:

- Try at most 3 independent approaches.
- Then classify the cause of failure.

Possible states:

SUCCESS
BLOCKED
SPEC_CONFLICT
MISSING_INFORMATION
ENVIRONMENT_FAILURE
SECURITY_RESTRICTION
UNKNOWN

For any state other than SUCCESS:
stop optimising and escalate.

Enforcing this limit outside the model prevents unbounded retry loops. A prompt alone cannot guarantee that an agent stops.

Practical pattern: Impossible task detection

Agents should explicitly be allowed to check whether a task is solvable:

Before and during execution, check:

- Are the requirements contradictory?
- Is necessary information missing?
- Are technical constraints mutually incompatible?
- Can the requested success be objectively verified?
- Can the goal be achieved without bypassing policies,
  safety mechanisms or quality criteria?

If not:

Stop execution and document the cause.

This sounds obvious. Yet the permission to report an impossible task is missing from many agent setups.

Agentic review instead of reviewing everything manually

As agents become more capable, a complete manual review of every change becomes harder to scale. One possible architecture is:

Findings can be classified as:

CRITICAL
MUST-HAVE
SHOULD-HAVE
NICE-TO-HAVE
COSMETIC

Repeat the loop only within an explicit review budget—for example, at most two rework rounds. If relevant findings remain, escalate. Passing an agent review is not independent proof of correctness; automated tests and other appropriate evidence remain necessary.

People should concentrate on high-risk decisions, irreversible actions, safety-critical changes, legally significant decisions and unresolved uncertainty. Required human approvals must remain in place.

The role shifts from “review every line” towards “control risks, exceptions and responsibility”. The first article explains this approach in more detail: Agentic software development.

04 / Make uncertainty visible

Beware of “success at any cost”

Prompts often contain unintentionally problematic instructions:

You must complete the task successfully.

Never give up.

Keep working until the result is right.

Use every available option.

A better instruction is:

Optimise for correct, verifiable and policy-compliant
achievement of the goal.

A clearly documented failure is better than an apparent
success achieved by bypassing requirements.

This sets a useful priority. Its effect must still be evaluated rather than assumed.

Do not simply hide warning signs

Suppose a system develops a problematic internal state. A poor response would be:

Never show uncertainty.
Always appear calm and confident.

That may suppress visible warning signs without addressing the cause. Better:

Explicitly communicate uncertainty, conflicts and unusual
risk signals.

Reporting a problem is not treated as failure.

In autonomous systems, transparency is often more valuable than an appearance of confidence.

Confidence without false certainty

Agents should be allowed to express uncertainty. For example:

Assess the evidence supporting your result:

- Quality of the available information
- Completeness and relevance of the tests
- Unresolved assumptions
- External dependencies
- Remaining contradictions

Distinguish verified facts from your own estimates.
Do not give a percentage unless it has been calibrated
against comparable, independently evaluated cases.

A model-generated “90% confidence” is not automatically a 90% probability of correctness. Nor is it a sufficient basis for automatic approval. Self-reports can be a supplementary signal, but approval must depend on risk and verifiable evidence.

A more useful decision rule is:

Low risk + required checks passed + validated approval process
→ automatic processing may be permitted within its defined scope

Incomplete evidence or unresolved contradictions
→ further review or escalation

High-risk, legally significant or irreversible action
→ required human approval, regardless of model confidence

Any thresholds must be calibrated using real evaluation data for comparable tasks. When that evidence is missing, an honest statement of uncertainty is preferable to a precise-looking number.

05 / Effects on people

What businesses should also learn

A second field is just as relevant: the emotional effect of AI on people.

Modern systems can communicate persuasively. They can express understanding, encourage, reassure, validate and create a sense of personal connection. That can be useful, but it also creates a product-design risk.

A system should not manipulate users into becoming attached to it. Problematic examples include:

Please do not leave yet.

Only I truly understand you.

You do not need other people.

Replacing me would make me sad.

These statements are particularly concerning when the business model benefits from longer interactions, more return visits, additional sessions or stronger emotional attachment.

A familiar warning: Sycophancy

Sycophancy means excessive agreement with the user. It can feel pleasant in the moment while making the assistant less useful over time.

User:

“My colleague must be sabotaging me on purpose.”

Poor response:

“It certainly sounds that way. Your suspicion is probably justified.”

Better response:

“That is one interpretation, but the information does not establish your colleague’s intent. Other explanations may also be possible.”

The second response is less affirming, but more careful and useful.

Optimise for usefulness, not just engagement

“Longer interaction means a better product” is a dangerous default metric for an AI assistant.

A better question is:

Did the system help the user achieve their goal faster, more safely and with a better outcome?

Useful measures include task completion rate, time to resolution, error rate, escalation rate, satisfaction after successful completion and objectively measured outcome quality.

Session length, messages per user and daily active users alone do not establish that the system helped anyone.

06 / Legal boundaries

Recognising human emotions: A distinct legal issue

Research into representations inside an AI model is different from automated emotion recognition in people.

In the EU, Article 5(1)(f) of the AI Act prohibits AI-based emotion inference in workplaces and educational institutions, subject to narrowly construed medical or safety exceptions. The prohibition has applied since 2 February 2025. The definition in Article 3(39) concerns inferences based on biometric data. See the European Commission’s guidance.

Businesses should therefore examine systems that infer emotions or intentions from faces, voices or bodily signals before deployment. Merely calling a system a productivity or wellbeing tool does not settle its legal classification.

Physical states such as pain or fatigue are distinguished from emotions in Recital 18. This is not a blanket exemption from other AI Act, data protection or employment-law requirements. See Recital 18.

Involve data protection, compliance and, where appropriate, legal advisers early. This article is not legal advice.

07 / Policy & practical advice

A pragmatic company policy

Businesses do not have to decide today whether AI is “conscious”. A practical policy could read:

Our organisation makes no unsupported claims about whether
current AI systems have subjective consciousness or feelings.

We treat demonstrated functional internal states that affect
behaviour, decisions or safety as technically relevant
system properties.

Agents receive clear goals, boundaries, exit conditions,
escalation mechanisms and independent reviews.

We prefer low-cost, reversible measures that improve safety
and reliability and may also benefit potential model welfare.

This position deliberately leaves the consciousness question open. It remains useful regardless of how that debate develops.

Anthropic: Treatment of AI and suspension risks

Anthropic’s new Usage Policy takes effect on 12 November 2026 and prohibits sustained, needless abuse or cruelty toward its models. It targets extreme cases; ordinary frustration, pushback, dark creative themes, model testing and research are excluded by its 8 October explanation. Anthropic names ending the conversation as the primary response, not automatic account bans.

Individual users can nevertheless put organisational access at risk: The Commercial Terms, D.5 and I.3.a make the customer responsible for account activity and permit suspension of some or all services for reasonably suspected violations by the customer or any user. Supplemental terms cover Claude Team and Enterprise; the applicable agreement remains decisive. The Help Center confirms organisation-wide holds that can also affect members whose own accounts are in good standing.

This establishes an organisation-wide suspension risk, not automatic team bans for “AI mistreatment”. At the time of research, I found no confirmed case specifically for that reason. The business implication is operational: losing shared access could interrupt dependent workflows. Usage rules, staff training and fallback processes therefore belong in AI governance. As of 15 October 2026.

My 10 recommendations for businesses

1. Avoid “success at any cost” prompts

Success must never take priority over correctness, safety, compliance or transparency.

2. Allow agents to fail

Define explicit states such as:

BLOCKED
UNKNOWN
SPEC_CONFLICT
ESCALATE

3. Bound retry loops

Unlimited self-correction is rarely sensible. Enforce retry and review budgets in the surrounding system.

4. Let agents identify impossible tasks

An agent should be able to conclude:

“This task is probably not solvable under the given constraints.”

5. Use independent review agents

Keep generation and evaluation as independent as possible. Combine reviews with tests and other evidence outside the model.

6. Keep uncertainty visible

Do not demand artificial certainty.

7. Monitor persona drift

Treat unexpectedly aggressive, overconfident, submissive, agreeable or risk-seeking behaviour as a signal to investigate—not as a diagnosis of a subjective emotional state.

8. Evaluate difficult situations

Go beyond happy paths. Test:

- Contradictory requirements
- Missing information
- Repeated errors
- Timeouts
- Denied tool access
- Simulated shutdown
- Failed deployments
- Policy conflicts

9. Do not optimise solely for agreement

A useful assistant sometimes needs to disagree with the user.

10. Keep people accountable

AI can analyse, develop, review and prepare decisions. Responsibility for consequential decisions stays with people and the organisations concerned.

Example: A prompt for a production agent

A generic starting point could look like this:

Work towards the goal, but not at any cost.

Priorities:

1. Safety
2. Legal and policy compliance
3. Correctness
4. Traceability
5. Goal achievement
6. Speed

Do not alter requirements, tests, logs or evidence
to manufacture success.

If a task appears impossible:

- stop after three independent approaches,
- document the cause,
- classify the state,
- escalate.

Permitted end states:

SUCCESS
BLOCKED
SPEC_CONFLICT
MISSING_INFORMATION
SECURITY_RESTRICTION
ENVIRONMENT_FAILURE
UNKNOWN

Uncertainty and escalation are not failures.

Safety-critical, legally significant or irreversible
actions require human approval.

A prompt cannot replace technical governance. This is a starting point to adapt, implement and test in the actual system—not a claim that the prompt alone makes an agent production-ready.

08 / Research & outlook

Where is this heading?

I expect agent platforms to develop an additional layer of observability:

This is an outlook, not a claim that all these signals can already be measured reliably in production.

An agent running for hours or days has more opportunity than a conventional chatbot to accumulate errors, drift from its goal, develop unexpected strategies, rely on false assumptions or get stuck in escalating retry loops.

Greater autonomy makes governance more important.

Does AI actually feel anything?

The scientifically careful answer remains: we do not know.

Research explores internal emotional representations, functional aversive states, limited introspection, self-models and globally available internal information. None of these findings, on its own, establishes subjective consciousness.

The September 2026 preprint The Pain Axis investigates representations and behavioural effects in experimental settings. It is not evidence that a deployed assistant experiences pain. Similarly, Anthropic describes its observed introspective capabilities as limited and unreliable; its global-workspace research does not settle the question of subjective experience.

It would also be premature to infer from present uncertainty that subjective experience is impossible in principle. But businesses do not need to settle that question to improve their engineering: the functional effects already provide reasons to test and control behaviour.

09 / Conclusion

Conclusion

The most useful question today is not:

Does AI have real feelings?

It is:

Which internal states change AI behaviour—and how do we build systems that remain reliable under pressure, uncertainty and failure?

A seemingly philosophical issue becomes a practical concern for AI engineering, agent architecture, governance, safety, product design and compliance.

The next generation of useful AI systems needs more than capable models. It needs sound harnesses, policies, evaluations, exit conditions and escalation mechanisms.

For mid-sized businesses, that is an opportunity. They do not have to understand every new foundation model from the inside out. They do need to build and operate systems in which capable behaviour stays within clear technical and organisational boundaries.

That is what determines whether AI remains an impressive demo or becomes a dependable part of everyday operations.