Mark Zuckerberg published a 6,500-word manifesto on Monday detailing his vision for personal AI systems. The essay outlines the potential of “personal superintelligence” within Meta AI, building on concepts he previously floated in the Wall Street Journal and during earnings calls. While this is his most detailed public statement on the subject, the core ideas are not new. The piece attempts to counter pessimism about artificial intelligence by describing a future where abundant superintelligence improves society. Yet, Zuckerberg repeatedly acknowledges the high probability of things going wrong.
Many readers find AI unsettling, viewing it as intrusive or unpleasant. Zuckerberg’s approach to this concern does little to reassure the public. Instead of addressing the root causes of distrust, the essay repeatedly demonstrates how that trust was eroded. Social media remains a fraught topic in technology. Facebook is unpopular with the US public, and Zuckerberg is widely disliked. A recent survey showed that 64 percent of Americans believe social media has harmed democracy, with similar figures across both parties wanting stricter regulation. Just last weekend, a court fined the company $567 million for harm done to children. These issues fuel the current anxiety surrounding AI. The public does not trust tech executives to ensure new technologies benefit society. Rather than rebuilding that trust, this manifesto highlights the failures of the past.
Abstract philosophy versus product reality
A significant portion of the text relies on vague generalities about intelligence, reminiscent of arguments Zuckerberg and Jack Dorsey once used regarding free speech. Here is one example:
As everyone gains more powerful tools, each person will become more capable of shaping the future, not less. People and institutions with competing interests naturally check and balance each other to lead towards positive outcomes. The best and most realistic path to building a positive AI future is by delivering superintelligence to everyone.
This reads like a philosophical conclusion rather than a description of a specific product called “personal intelligence” that Meta plans to sell. If you already distrust this figure, the text may seem like an attempt to convince oneself that nothing bad can happen.
The disconnect becomes clearer when looking at specific examples. Zuckerberg’s view on education is as follows:
Everyone will have a personalized tutor and coach with a PhD in every subject and unlimited patience to help you learn anything you want. Students will have extra help in areas they need it that is currently only available to those whose parents can pay. Adults will have a superintelligent learning assistant that knows exactly how to teach you new job skills, new languages, new hobbies, or anything else you’re interested in.
This product already exists in the form of consumer chatbots like ChatGPT, Claude and Gemini. While these tools can help you learn, their primary use in education is often to avoid learning. A personalised tutor can complete homework and write assigned essays. Because there is no robust watermarking system, teachers cannot verify which essays were generated by AI.
This is a low-stakes example of harm, yet it is happening now. Designing technology for good can produce unintended negative consequences. The fact that one of the most powerful people in the world refuses to acknowledge this makes people nervous.
Zuckerberg also argues that AI will improve the legal system. He poses a thought experiment about superintelligent lawyers. He suggests that if only one person had such a lawyer, they would have an unfair advantage in court, leading to a worse society. However, if everyone had access, justice would be carried out more fairly and efficiently than today, where skills and resources are often imbalanced.
This is a loaded example. Access to legal AI might create a more just society, but it could also add complexity to the existing bureaucratic mess. It might unleash a new wave of vexatious litigants that clog the courts with the legal equivalent of spam. It is hard to feel calm about these possibilities, and Zuckerberg’s lack of worry increases concern.
Freemium models and dynamic pricing
In other sections, descriptions of Meta’s existing business practices take on an oddly abstract character. Zuckerberg explains his commitment to a freemium model this way:
Everyone will have free or affordable access to these tools. We will offer free versions that will be accessible to billions of people. For those who want to pay to use more compute, there will be a dynamic auction mechanism that will guarantee that everyone gets the lowest price possible for the intelligence and compute they’re using while also ensuring the capacity is used for whatever people collectively find most valuable. This will ensure the benefits of superintelligence are distributed widely.
In broad strokes, the argument holds weight. Zuckerberg is right that access is an issue and that a freemium model allows for more access than requiring everyone to pay for compute upfront. Dynamic markets for spot compute already exist. However, every consumer AI product currently insulates users from the spot price of the compute they use. Surge pricing is a terrible user experience, particularly for a tool you rely on for work.
I assume Zuckerberg knows this and is not actually planning to set token prices through dynamic auctions. But I am less sure of that now than I was before reading this piece.
There are better ways to discuss this. For all their faults, Sam Altman and Dario Amodei do a good job communicating with the public. They do exactly what Zuckerberg resists: acknowledging the dangers of AI, emphasising their own precautions, and trying to convince listeners that they can be trusted.
Communication can only do so much. Sometimes safeguards fail and the inherent danger of the enterprise becomes clear. Without trust-building, the industry may not always survive those failures.
