The race to overcome the limitations of transformer-based large language models (LLMs) is gaining momentum. While transformers have powered the recent explosion in AI capabilities, their inherent constraints around memory and continual learning are increasingly recognized as roadblocks to achieving true artificial general intelligence (AGI). Two companies – Pathway and Mary Technology – are tackling these challenges from distinct, yet complementary, angles, as discussed in recent interviews at AWS re;Invent .
Pathway, a data company founded by Zuzanna Stamirowska, Jan Chorowski, and Adrian Kosowski, is developing a “post-transformer” architecture called Baby Dragon Hatchling (BDH). The core problem Pathway aims to solve is the inability of current LLMs to generalize over time. “Models that we are actually training right now will be capable of continual learning, capable of long-term reasoning, and of adaptation, imagine life AI,” explained Stamirowska. Unlike transformers, which rely on brute-force scaling – adding more data, compute, and layers – BDH takes inspiration from the structure of the human brain.
The brain, Stamirowska elaborated, is a highly efficient system of approximately 100 billion neurons connected by roughly 1,000 trillion synapses. This structure allows for efficient computation, long-term reasoning, and, crucially, the ability to learn continuously. BDH mimics this by utilizing artificial neurons and synapses, creating a modular structure that resembles the neocortex – the part of the brain responsible for higher-order cognitive functions. “We looked a little bit at the brain, how it works, and found a link between transformers and the brain,” Stamirowska said. “We published bits of what we were doing already… the BDH Dragon Hatchling Architecture.”
This architecture differs significantly from traditional transformers. Instead of relying on massive matrix calculations, BDH operates on local interactions between artificial neurons. When new information arrives, neurons “fire up” and send messages to their connected neighbors. The strength of these connections – the synapses – increases with repeated activation, creating intrinsic memory within the system. This approach, Stamirowska explained, is not only computationally efficient but also addresses the issue of energy consumption that plagues increasingly large transformer models.
A key difference lies in how BDH handles information. The model operates with positive and sparse activations, a concept rooted in the work of researchers like Jeff Hinton. This means the system focuses on strengthening positive connections rather than encoding negative ones, mirroring how humans learn – for example, understanding what *not* to do by observing a poor example rather than needing to experience failure repeatedly. “If you need to repaint, you tell a guy who’s renovating your house that he should do his job well by showing him how a job [is] badly done. This doesn’t give him information [on] how to do it well,” Stamirowska explained.
The implications of this architecture extend beyond simply improving memory. Pathway believes BDH will reduce the likelihood of “hallucinations” – the tendency of LLMs to generate factually incorrect or nonsensical outputs. By enabling longer attention spans and continuous learning, the model is better equipped to stay focused on a task and avoid straying from established knowledge. Pathway’s benchmark testing suggests BDH significantly outperforms current models in sustained reasoning tasks.
While Pathway focuses on the foundational architecture of AI, Mary Technology is applying LLMs to a specific, highly regulated domain: legal evidence review. Founded by Rowan McNamee, Mary Technology provides attorneys with a “fact management system” designed to handle the overwhelming volume of documents involved in litigation. “We help lawyers… with the thousands or tens of thousands of pages of evidence that they have in a legal case or a legal dispute that they’re managing,” McNamee said.
The system extracts facts from evidentiary documents, organizes them, and provides tools for lawyers to quickly identify relevant information. Mary Technology leverages both traditional machine learning and LLMs, emphasizing the importance of lawyer oversight and source verification. “We try not to provide any sort of legal interpretation of a fact, but if We see present in the evidence, it is an event,” McNamee stated. The system also includes “confidence tooling” – features designed to help lawyers assess the reliability of AI-generated insights, such as highlighting the source of a particular fact and explaining the reasoning behind its relevance.
The legal profession’s cautious approach to AI is understandable, given the potential for errors and the high stakes involved. McNamee acknowledged the recent cases of lawyers being sanctioned for relying on inaccurate information generated by LLMs. “The onus is still very much on the lawyer to check their sources,” he said. Mary Technology’s approach prioritizes transparency and provides lawyers with the tools to validate AI-generated insights, mitigating the risk of relying on flawed information.
Mary Technology is also addressing the challenge of data privacy and security. The company utilizes enterprise-grade models and ensures data sovereignty, complying with regulations in different jurisdictions. They are also developing methods for creating synthetic data to train their models without compromising the confidentiality of client information.
Both Pathway and Mary Technology highlight the need for responsible AI development and deployment. Pathway is focused on building more interpretable and efficient models, while Mary Technology prioritizes transparency and lawyer oversight. These efforts represent a shift towards AI systems that are not only powerful but also trustworthy and aligned with human values. The combination of foundational advancements like BDH and practical applications like Mary Technology’s fact management system suggests a future where AI can augment human capabilities without sacrificing accuracy, accountability, or control.
