Artificial Intelligence: Tool, Adviser or New Authority?

Tool, adviser and authority are not the same role

We first need to distinguish three roles. A tool extends our capability: it calculates, searches, translates, sorts or helps create a draft. An adviser goes a step further by offering an assessment, recommendation or option intended to influence judgment. Authority emerges when its output starts determining what will happen even though a human is formally still present.

That transition is not determined by technology alone. The same model may be an ordinary assistant in one organisation and a practical decision-maker in another. If an employee lacks the time, information, authority or psychological safety to reject a recommendation, a human signature on the final decision does not amount to meaningful human oversight.

The proper unit of analysis is therefore always the human–AI–institution system: who sets the objective, who shapes the inputs, who understands the limitations, who can reject the output, and who bears the consequences when the decision is wrong.

Generative systems have a distinctive feature: they can produce fluent, confident and useful-looking answers even when the underlying basis is weak or wrong. NIST uses the term confabulation for confidently generated but erroneous or fabricated content. This is not a moral failure of the system; it is a risk arising from the way such models generate outputs.

Humans, however, readily mistake fluency for competence. A well-structured answer, expert tone and speed can reduce the felt need to verify. That is why “How to Evaluate a Source” remains essential: the question “who says this?” becomes, with AI, “what is this output based on, and how can I verify it independently?”

AI can therefore be an excellent first map, but a map is not evidence. For consequential claims there must remain a path to a primary source, dataset, calculation or other verifiable basis.

A recommendation is not a decision

The most dangerous shift often happens quietly: a system is introduced as “decision support”, then its recommendation becomes the default decision. A person formally clicks approve but no longer performs independent judgment. A recommendation becomes a decision when the cost of departing from it becomes greater than the cost of accepting it without scrutiny.

This can arise from time pressure, productivity targets, hierarchy, or simply because deviation requires extra documentation. In such an environment an organisation may speak of a “human in the loop” while the human is actually only the final administrative station in an automated process.

This article therefore insists on a clear boundary: the system may recommend, but it must be defined in advance who actually decides, with what information, and with what real power to depart from the recommendation.

It seems intuitive that combining human judgment with a strong model should always outperform either alone. Research shows a more complicated picture. A meta-analysis of human–AI collaboration experiments found that AI often improves human performance relative to unaided humans, yet the human–AI combination does not necessarily outperform the better member of the pair on average.

The reason is understandable. A person has to know when to trust the system and when not to. If users cannot recognise cases in which AI is strong and cases in which it is weak, useful automation can introduce a new class of errors.

System quality therefore includes more than model accuracy. It also includes the quality of collaboration: calibrated trust, communication of uncertainty, verifiability, user competence and sufficient time for independent judgment.

Automation bias: when recommendations receive too much weight

Research has long described automation bias: the tendency to give automated advice excessive weight or to reduce one’s own verification because automation is present. Systematic reviews indicate that the phenomenon is not confined to one domain and can be strengthened by workload, task complexity, time pressure and high confidence in the system.

This does not mean people always trust algorithms too much. The opposite problem also exists: unjustified rejection of good automated advice. The goal is therefore not “trust less”, but trust in a calibrated way—in proportion to demonstrated performance, the specific context and the possibility of independent checking.

“Why Conformity Often Overrides the Facts” showed how group pressure can alter judgment. AI creates a related but different risk: a recommendation can feel like silent expert consensus even when it is merely a model output that nobody in the room has actually verified.

The EU AI Act requires high-risk AI systems to be designed so that they can be effectively overseen by natural persons while in use. The crucial word is effectively. Oversight is not meaningful if the human does not understand the system’s basic purpose and limitations, lacks sufficient information, or has no authority to intervene.

Meaningful oversight therefore requires at least four things: the ability to understand the context of an output, time to verify it, a genuine right to reject it, and the ability to stop or bypass the system where necessary. NIST similarly stresses clearly defined roles and responsibilities in human–AI configurations and asks whether humans are actually empowered to challenge system outputs.

If an organisation assigns responsibility to a human but gives that person no power to change the decision, it has created responsibility without authorship. If it gives power but not competence or time, oversight becomes a formality.

It also matters that the overseer is not dependent solely on the same summary produced by the system. If a person sees only the final output without access to key facts, alternatives or warnings, the apparent ability to reject can become hollow. Oversight needs an independent information path, not merely an approve-or-reject button.

Responsibility must not disappear inside the automation chain

“Does Authority Remove Personal Responsibility?” showed that division of labour does not erase personal and institutional responsibility. With AI the problem is even sharper because the chain can be long: model developer, service provider, integrator, deploying organisation, process owner and the person who uses the output.

It is a mistake if every link points to another. The developer says it did not make the decision; the organisation says it merely used a recognised tool; the operator says they followed the recommendation; management says a human made the final decision. Responsibility must follow the actual capacity to influence the objective, design, deployment, use and correction of the system.

OECD, UNESCO, NIST and the Council of Europe therefore each emphasise accountability, traceability and oversight in different ways. Technology can assist a decision, but it cannot become the place where an institution deposits its own moral and legal responsibility.

The greater the consequence, the stronger the right to challenge

It is not the same for AI to recommend music and for AI to participate in decisions about employment, credit, healthcare, insurance, social entitlements or security measures. The greater the potential harm to a person, the less acceptable unreviewable automated authority becomes.

For decisions that significantly affect rights, the Council of Europe stresses information, procedural safeguards and the ability to challenge the decision or the use of the AI system. This is essential: a person needs more than an explanation that “the algorithm assessed the risk”; they need a route by which an error can be demonstrated and corrected.

A right to challenge is a practical boundary between an advisory system and untouchable authority. If a decision cannot be understood, reviewed, contested or corrected, technical efficiency begins turning into institutional opacity.

AI as a source must pass the same test as any other source

In everyday use, AI often acts as search engine, summariser, explainer and conversational partner at once. That makes it easy to forget that the answer is not a primary source. A model may synthesise knowledge correctly, but it may also mix periods, attribute a claim to the wrong author, or confidently fill a gap.

“How to Evaluate a Source” therefore becomes even more important with AI. For a consequential claim we ask: can I reach the original document? Is the source current? Is this fact, interpretation or forecast? Is there contrary evidence? Can the result be reproduced through an independent route?

The healthiest relationship with AI is neither blind trust nor automatic distrust. It is provisional trust proportional to verifiability, with more checking as the stakes rise.

An institution can turn AI into authority without intending to

De facto authority often does not arise because someone explicitly decides that “AI should rule”. It arises through organisational incentives. If a system processes more cases, lowers costs and standardises workflows, every human deviation can gradually start looking like inefficiency.

Over time employees may defend the system’s decision because doing so is safer for their careers; managers may reduce expert staff because automation is expected to compensate; data may be collected in the form most convenient for the model; and the process itself may be redesigned around what can be measured easily. At that point AI is no longer merely a tool inside the institution—the institution begins adapting itself to the tool.

This is an important link back to “Institutional Capture: When the System Starts Serving Itself”. Institutional capture need not come only from an outside interest. A system can also begin serving its own metric, procedure or automation if it loses contact with the purpose for which it was created.

Long-term dependence can also weaken human expertise. If people perform the underlying judgment themselves less and less often, an organisation may eventually lose the capacity to recognise when the system is wrong. Good AI deployment must therefore preserve not only the right to intervene but also the competence to intervene—through training, periodic manual checks and retention of domain expertise.

Twelve questions: does AI support judgment or replace it?

Before AI takes a consequential role in a process, we can run a simple audit. 1. What is the system’s task? 2. Does it provide information, a recommendation or an actual decision? 3. Who is the formal and who is the real decision-maker? 4. Does the user understand the system’s main limitations? 5. Can the basis of an important output be checked? 6. Is uncertainty or the boundary of reliability communicated?

Continue: 7. Does a human have a real right and enough time to reject the recommendation? 8. Is deviation from AI punished or administratively discouraged? 9. Who is accountable when harm occurs? 10. Can an affected person challenge the decision and obtain correction? 11. What happens if the system fails or is unavailable? 12. Do we periodically test whether AI still serves the original purpose?

If the answers are unclear, the problem may not be the model itself. It may be the institution that quietly transformed a recommendation into a command and dispersed responsibility so widely that nobody experiences it as their own.

the articles from “Monopoly, Plutocracy and the Concentration of Economic Power” through “Artificial Intelligence: Tool, Adviser or New Authority?” examined different pathways through which power can reconcentrate: wealth, delegation, emergency powers, institutional capture, false decentralisation, data, attention algorithms, identity-payment chokepoints and finally automated authority. The common pattern is the same: power becomes more dangerous as the capacity to verify, exit, object and correct becomes weaker.

With AI the final boundary is especially personal. No model can take over the conscience described in “Obey an Order or Follow Conscience?” or erase the responsibility described in “Does Authority Remove Personal Responsibility?”. It can broaden our view, alert us to something we missed, and even often judge better than we do. But wherever a decision affects a person, it must remain clear who authored the action and who can answer for it.

This closes Stage 9—Protect distributed power. “Personal Dependency Audit: What Does My Life Actually Depend On?” begins the final action-oriented part of this developmental arc and turns the question directly toward the reader: which systems, people, resources and technologies does my own life actually depend on—and where do I have a fallback path?