NextFin News - Paula Goldman, Salesforce's chief ethical and humane use officer, is making a pointed argument as she releases her first book: the "human in the loop" model that dominated AI ethics for a decade is broken for autonomous agents, and companies that do not replace it with a "human at the helm" design will find their AI deployments failing at the moment they matter most. Her book, Manage the Machine: How to Harness Human-AI Collaboration at Work, landed on shelves September 15, 2026, through Basic Venture, at a time when enterprise buyers are moving from AI pilots to production-scale agent deployments — and discovering that the governance question is the one that decides whether the rollout survives.
The timing is deliberate. In a mid-September interview, Goldman laid out the mechanism: agents now perform more actions daily than any manager can review, so the choice of which work goes to an agent and where a person stays in charge decides the outcome. That is the thesis of the book, and it is also the quiet pitch for why governance — not just capability — is becoming the differentiator in enterprise AI.
The Event and the Argument
Paula Goldman, Salesforce's first-ever Chief Ethical and Humane Use Officer and an EVP of Product, published Manage the Machine on September 15, 2026 — 304 pages, hardcover at $32.00, ebook at $18.99. She is a member of the US National AI Advisory Committee, established by Congress to advise the president, and was named to Fast Company's AI 20. The book carries an endorsement from Salesforce chair and CEO Marc Benioff:
AI is the most transformative technology of our time — but its true impact will be defined by how we shape the partnership between humans and machines. Manage the Machine lays out a clear roadmap for how businesses can harness AI's immense potential while keeping trust, creativity, and human judgment at the center.
The book is a practical guide to integrating AI into work across sales, corporate strategy, HR, marketing, talent acquisition, product innovation, and legal affairs, with case examples from Walmart, SharkNinja, and Nike. Its structure tracks the organizational journey: Part I covers AI and customers (sales, customer service, marketing and digital commerce); Part II covers people operations (people management, talent acquisition, human-AI teams); Part III covers how work gets shared (creativity and product innovation, legal and corporate affairs). The through-line: keep humans at the center of the transformation.
The tension is this. The ethics conversation has largely been about review gates — a human checks the machine's output before it ships. Goldman's argument, repeated across her September interview circuit, is that this model collapses under agentic AI, where the volume of autonomous actions exceeds any manager's review capacity. The replacement she proposes — "human at the helm" — shifts the design question upstream: not "who reviews the output?" but "which work should the agent do at all, where are its boundaries, and what audit trail proves it stayed inside them?"
Why this matters now: Salesforce is not a bystander in this argument. The company reported fiscal second-quarter revenue of $11.35 billion for the quarter ended July 31, 2026, up 11% year over year and above the $11.32 billion Wall Street consensus, with Agentforce ARR reaching $1.5 billion. The governance framework Goldman is articulating is the operating manual for the very products Salesforce sells. When the chief ethics officer and the product organization are describing the same architecture, ethics has become product strategy.
Why "Human in the Loop" Dies When Agents Scale
The central mechanism Goldman is describing is a capacity problem, not a moral one. A "human in the loop" model assumes a human can review each output before it acts. That assumption held when AI produced one draft, one recommendation, one classification at a time. It does not hold when an agent executes dozens or hundreds of actions a day across CRM records, customer communications, and workflow systems. Goldman's framing was explicit: agents now perform more actions daily than any manager can review on their own.
The consequence is a governance failure mode that most enterprises have not yet priced in. If review capacity is the bottleneck, then scaling agents without redesigning the control point does not produce more output — it produces more unreviewed output, and the risk accumulates silently until a consequential decision goes wrong. Goldman's answer is to move the control point from the output to the boundary: define which tasks can be safely delegated, where humans must take the lead, what the agent's boundaries are, and what audit trail captures what happened. In her book's language, this is the difference between asking "is this answer correct?" and asking "should this decision have been delegated at all?"
That is a structural shift in how enterprises will buy AI. It moves the procurement question from model benchmarks to governance design — and it favors vendors who can ship the audit trail, the boundary-setting, and the consequence-mapping as a product, not as a compliance appendix.
The Expertise Pipeline Nobody Is Asking About
The second-order consequence is the one most executives are not asking about. If agents absorb entry-level work — the drafting, the research, the first-pass analysis — then the pipeline that used to train junior employees into senior judgment evaporates. Goldman raised this directly in the discussion: the talent pipeline is at risk when the work that used to build expertise is delegated away.
This is the hidden cost of the productivity gain. An organization that delegates the bottom of the skill ladder today gains efficiency now but may lack the senior judgment it needs in five years, because the juniors never got the reps. Goldman's "human at the helm" framing is, in part, an answer to this: humans stay at the helm not because they are slower, but because the helm is where judgment is built. The design question becomes which tasks develop human capability and which merely consume it — and reserving the capability-building work for humans is an investment in future capacity, not a concession to inefficiency.
The counter-argument is straightforward: every productivity technology has destroyed jobs at the bottom while creating higher-value roles above. The printing press, the spreadsheet, and the search engine all eliminated categories of apprentice work, and the economy reorganized around higher-skill roles. If AI follows that pattern, the "pipeline problem" is a transition cost, not a structural break.
Goldman's position anticipates this. Her argument is not that entry-level work must be preserved for its own sake, but that the training function of that work must be deliberately replaced — through structured human-AI teams, explicit coaching loops, and reserving consequential decisions for human judgment. Whether that deliberate replacement happens is the variable. If companies treat delegation as pure cost reduction, the pipeline hollows out. If they treat it as a redesign of how judgment is built, the helm stays staffed.
Accuracy Jumped to the Top of the Principles List
There is a quieter signal in Goldman's recent commentary that matters for how investors should read the AI race. In a mid-2026 interview, she noted that Salesforce had a set of AI principles, then updated them because accuracy "all of a sudden sort of jump[ed] to the top of the list of things that we want to make sure that we got right." The technology, she said, "keeps changing, keeps progressing, keeps getting more powerful, more general," while the principles — the "true north" — remain steadfast, and the company adjusts to keep covering the ground it is meant to cover.
That is an important admission about the state of the technology. As models become more general and more agentic, the failure modes shift from obvious hallucinations to subtle, consequential errors in workflow — a wrong customer record update, a misrouted escalation, an overconfident recommendation acted on without question. Accuracy is no longer a model-quality metric; it is a system-safety metric. And system safety is harder to benchmark, harder to insure, and harder to recover from when it fails.
This is where the market's current pricing of the AI trade may be incomplete. The market is largely pricing AI as a feature and revenue race — ARR growth, agent deployments, seat expansion. Salesforce's own numbers reflect that framing: Agentforce ARR hit $1.5 billion in the quarter, and fiscal Q2 revenue of $11.35 billion beat the $11.32 billion expected. But the governance layer Goldman is describing is the layer that determines whether that ARR survives contact with a real enterprise procurement cycle, a real audit, or a real consequential-decision failure. The vendor that can prove its agents operate inside defensible boundaries — with audit trails, consequence mapping, and human-helm design — is selling risk reduction, not just productivity. Risk reduction is what enterprise buyers pay premium margins for.
The Regulation Tension — Rules as a Moat
Goldman's role also sits at the center of a regulatory tension she has not been shy about. US technology companies, as Salesforce CEO Marc Benioff put it at Davos, "hate regulation." Yet the enterprise AI market she serves is moving into consequential decision-making — hiring, credit, customer service, legal, corporate affairs — which is exactly where regulators are moving in. Goldman's framing is that rules need to keep up with the technology, and interpretations of the rules need to keep up with it, and that tension is natural as agentic AI evolves.
The strategic reading: a company that builds its governance framework ahead of regulation converts compliance into a moat. The competitors who can afford to build audit trails, consequence mapping, and human-helm architecture are the large incumbents; smaller players and open-weight deployments often cannot. Goldman's "true north" language — principles that remain steadfast while the implementation adjusts — is the language of a company preparing to be the compliant default in a regulated market. Salesforce's investment in and use of Anthropic, a company that has drawn regulatory attention over its ethical red lines, underscores how deeply this sits in the industry's current politics.
The counter-thesis here is that regulation moves slowly, and the AI capability race moves fast — so the first-mover on features captures the market before the first-mover on governance gets its rules written. That was true in the early cloud era and in social media. But enterprise AI is different from consumer social in one decisive way: the buyer is a risk committee, not a teenager with a smartphone. Enterprise procurement cycles reward the vendor that can answer the audit question, and they punish the vendor that cannot — even if the laggard's model is better.
What It Means — and What Would Prove It Wrong
Goldman's argument reduces to a testable claim: the enterprise AI winners will be the companies that solve the governance-design problem, not the ones with the most capable models. That has three concrete implications.
First, for vendors: governance features — boundary setting, audit trails, consequence classification, human-helm handoff design — move from compliance appendix to core product. Salesforce is already positioned here through Agentforce and Agent Fabric, and Goldman's book is the field manual for that product line.
Second, for enterprise buyers: the procurement checklist expands. The question is no longer "what can this agent do?" but "what should it not do, who owns the outcome, and what proof do we have that it stayed inside the lines?"
Third, for the labor market: the "human at the helm" model only works if companies deliberately rebuild the training pipeline that entry-level work used to provide. Otherwise the helm has no trained humans to staff it.
Time-horizon split. In the short term, the AI trade remains a revenue-and-ARR story. Salesforce's stock, which traded around $255 in mid-September 2026 with a trailing P/E near 23 and a 52-week range of $146.32–$269.11, will continue to move on quarterly AI ARR and guidance. Governance is a talking point, not yet a valuation driver. Over the medium term, as deployments move from pilot to production, governance failures will surface — a consequential-decision error, an audit finding, a customer escalation that an agent mishandled. Vendors with defensible governance architecture will win procurement cycles they would otherwise lose. This is where the "human at the helm" argument converts from philosophy to revenue. Over the long term, if agentic AI becomes the default operating layer for enterprise work, governance design becomes a durable competitive advantage — a regime shift, not a cycle. The companies that solved it early will have a moat that compounds; the ones that treated it as compliance will be retrofitting under regulatory pressure.
Scenarios. The base case is that governance becomes a standard enterprise procurement requirement within 18–24 months, and vendors with built-in audit and boundary tooling win disproportionate share of regulated-industry deals in financial services, healthcare, and legal. The upside case is that a high-profile agent failure in a consequential-decision context accelerates the shift, making governance the single largest differentiator in enterprise AI — and Goldman's framework becomes the industry reference architecture. The downside case is that the capability race continues to outrun governance concerns; buyers prioritize features over controls, and the "human at the helm" model remains a differentiator for a premium niche rather than the market default.
Falsifying signal. The thesis fails if, over the next four enterprise procurement cycles, governance features remain a tie-breaker rather than a requirement — specifically, if fewer than half of large enterprise AI RFPs in regulated industries include explicit audit-trail, boundary-setting, or human-handoff requirements. That is the metric to track in procurement surveys and vendor earnings commentary.
What to watch. The specific signals: whether Salesforce and its peers begin breaking out governance-related product revenue or attach rates; whether regulatory guidance on agentic AI in consequential decision-making hardens in 2026–2027; whether a high-profile agent failure triggers a procurement shift; and whether Goldman's "human at the helm" language migrates from her interviews into enterprise RFP templates.
The AI race is often described as a contest of models. Goldman's argument is that it will actually be decided by a design choice most companies have not yet made: not whether humans stay in the loop, but whether they are still at the helm when the machine can act faster than they can watch.
Explore more exclusive insights at nextfin.ai.
