Artificial Intelligence Cannot Live Only on the Internet
Over the past decade, artificial intelligence has made remarkable progress. From image recognition and text generation to coding and scientific assistance, machines increasingly appear capable of “understanding” and “solving” problems. Yet these capabilities reveal clear limitations when applied to more complex, unfamiliar, and continuously changing real-world environments. Today’s mainstream AI still works primarily with Internet data such as text, images, video, and audio, and it is therefore strongest on problems that can be represented through these modalities. Language understanding, image interpretation, video generation, and coding all fit naturally within this information space. But many real-world problems do not. Equipment states, continuous numerical signals, sensor readings, manufacturing processes, and dynamic feedback from physical systems cannot be fully represented by Internet text and audiovisual data alone. As AI moves further into manufacturing, energy, robotics, and scientific experimentation, this gap will become increasingly important. The next major advances in AI may therefore depend not simply on making models larger or adding more Internet data, but on reconsidering several more fundamental questions: how models should compute, how they should represent the real world, and what kinds of data they should learn from.
Architectural Homogeneity and the Low Efficiency of Sequential Reasoning. Modern AI has become highly homogeneous in model architecture. Whether in language models, multimodal systems, or increasingly large foundation models, most leading approaches are built around the Transformer and its variants. One of the Transformer’s great strengths is that much of its training computation can be parallelized. During generation and inference, however, mainstream models still rely heavily on sequential autoregressive decoding: one token must be generated before the next can be produced. Even if each individual step is accelerated by powerful hardware and parallel computation, the full output must still advance through a sequence of dependent steps. As models grow larger, contexts become longer, and reasoning chains become more complex, this sequential structure increasingly limits latency, memory efficiency, and energy consumption. At the same time, a large share of academic and engineering effort has concentrated on Transformer-based systems, while comparatively fewer resources have been devoted to GANs, VAEs, and other generative approaches capable of producing outputs in one shot or with much greater parallelism, as well as to more general non-autoregressive inference mechanisms. This concentration has accelerated the development of large models, but it has also created technological path dependence, to the point that token-by-token generation is often treated as the natural form of machine reasoning. It is not. It is only one implementation. If a future model can retain the expressive power and generalization capabilities of today’s large models while completing complex inference in one or a small number of parallel steps, the result may be more than a several-fold speedup. It could redefine the computational paradigm of AI itself. The infrastructure, hardware demand, and model designs built around autoregressive Transformers could then face fundamental disruption.
The Gap Between Token Representation and the Numerical World. A second limitation is that current AI represents information primarily through tokens. This makes it naturally well suited to language and other symbolic tasks, but much less naturally suited to the continuous numerical variables that define much of the physical world. Tokens are highly effective for representing text, code, and other discrete symbols, which is why large language models have become powerful tools for question answering, content generation, and programming. But the real world is not made of words. Temperature, pressure, current, rotational speed, vibration, flow rate, torque, energy consumption, inventory levels, and market prices are all continuous quantities in which small differences can carry precise and important meanings. Converting these values into text and passing them to a language model does not automatically give the model an accurate understanding of their numerical structure. To a language model, “1.01” and “1.02” may simply be different symbol sequences. In a high-speed motor, a chemical process, or a control system operating near a stability boundary, however, such a difference may correspond to a materially different system state. More importantly, many real-world problems depend not on individual values but on long-term, continuous, and dynamically coupled relationships among many variables: how a control action changes current and rotational speed, how those changes affect temperature several seconds later, and how all of this influences energy use and system stability. These relationships exist naturally in continuous state spaces and are difficult to capture fully through discrete token relationships alone. Large language models may be very good at talking about physical systems, explaining equations, reading papers, and interpreting experimental results, but that does not mean they are naturally suited for precise numerical prediction, real-time control, or continuous optimization. For manufacturing, energy systems, robotics, and scientific computing, future AI may need not only larger token-based models, but also new modeling methods that directly represent continuous variables, state spaces, and dynamic processes.
The Distance Between Internet Data and the Physical World. The third and more fundamental limitation lies in data. A major reason for AI’s rapid progress over the past decade is that the Internet contains enormous amounts of text, images, video, and audio. Over many years of producing and sharing information, humanity unintentionally created a vast training environment for today’s large models. Books, webpages, papers, code, photographs, recordings, and videos allow models to acquire substantial knowledge without directly interacting with the physical world. But this supply of data is not unlimited. As training scales continue to grow, high-quality, publicly available Internet data will be used ever more thoroughly. Over the coming years, better data curation, synthetic data, and greater use of images, video, and audio may continue expanding training sets, but the growth of these sources cannot indefinitely keep pace with the rising data demands of larger models. More importantly, even if Internet data were fully exploited, it would still represent only a limited projection of reality. Today’s large models mostly see information that humans have chosen to record, organize, and publish. Vast quantities of data generated by the operation of the physical world never reach the public Internet. Pressure, current, and vibration inside industrial equipment; temperature, humidity, and power consumption in air-conditioning systems; torque, position, and tactile feedback in robot joints; state changes during material processing; real-time loads in energy systems; and continuous process data on manufacturing lines all exist in enormous quantities, yet rarely become part of foundation-model training. A model may read nearly everything on the Internet about motors, air conditioners, or robots and watch large amounts of related video, but it is still learning primarily from human descriptions of those systems rather than from the full state transitions produced as the systems actually operate. As Internet data approaches the limits of what it can provide, the key question for AI will no longer be simply “How much more data can we find?” but “Where will the next kind of data come from?” If AI is to move deeply into manufacturing, energy, robotics, and scientific experimentation, it must go beyond passively reading the Internet and begin learning directly from sensors, machines, experiments, and real environments.
These three limitations ultimately point to the same underlying issue: today’s AI remains deeply rooted in an information world created by humans. The Transformer defines its dominant computational structure, tokens define its dominant representation, and Internet text, images, video, and audio define much of its data supply. This system has already demonstrated enormous value and will likely continue to improve for a long time, but it may not represent the final form of artificial intelligence. The next stage may require breaking through all three boundaries at once: moving from sequential inference toward more efficient parallel inference, from token-based representation toward direct representations of continuous variables and complex states, and from Internet data toward data generated by sensors, machines, experiments, and real environments. When models no longer merely read the text, images, and videos left behind by humans, but can directly observe system states, understand the consequences of actions, and improve through repeated experimentation, AI may finally evolve from a large model that lives on the Internet into an intelligent system capable of confronting real problems, understanding real processes, and discovering real solutions.
Comments
Post a Comment