Skip to content
Fast Company Executive Board

The Fast Company Executive Board is a private, fee-based network of influential leaders, experts, executives, and entrepreneurs who share their insights with our audience.

Role of memory technologies in the generative AI boom

Memory bandwidth, capacity, and proximity help build better AI

[Photo: Getty Images]

Profile photo of Shashank Chaurasia

Co-Founder & Chief AI Officer – MooresLabAI · MooresLabAI

Building Generative AI models depends heavily on how fast models can reach their data. Memory bandwidth, total capacity, and physical proximity to the processor determine how quickly data can move and how it impacts usability. Teams that prioritize these elements see faster products, steadier costs, and predictable scaling.

WHY MEMORY MATTERS

Modern AI systems, especially Large Language Models, move large volumes of data every millisecond. When this information can’t be accessed quickly enough, even powerful AI chips sit idle, waiting on memory, leading to slow training and inference.

Three ideas guide design and buying decisions:

  • Bandwidth: How quickly data can move between memory and the AI processor.
  • Capacity: How much of the model’s active information, such as learned knowledge, conversation history, and intermediate calculations, can stay in memory close to the processor without being moved or recomputed.
  • Proximity: How physically close that memory is to the AI processor itself. Shorter distances mean data reaches the processor faster and with less energy, allowing the model to respond more quickly and efficiently.

Think of memory like a busy factory with a warehouse next door. More loading lanes, a larger inventory, and shorter paths increase the speed at which the factory can create and move product.

WHAT’S ADVANCING

Generative AI is getting better, not just because chips are faster, but because the data that the model needs can reach the chip faster. That affects what your business cares about: speed, cost, and what features you can offer.

These recent advancements are driving faster models:

  • Memory is being placed right next to the AI chip. When memory is physically close, the chip spends less time waiting for data, which provides faster responses, higher user capacity per server, and better cost per AI interaction.
  • Newer memory generations are expanding what AI can handle in one go. More near-chip memory and faster transfer speeds can support longer documents, longer conversations, more tools, and higher throughput, which translates into better product experiences without spending more on infrastructure.
  • The way chips are assembled has become a business bottleneck. Advanced packaging is required to put memory close to compute at scale. The impact is practical; it can influence availability, lead times, and whether your AI roadmap ships on schedule.
  • A new approach lets servers share and expand memory more flexibly. CXL (Compute Express Link) is a standard that lets systems treat memory more like a shared pool, rather than being stuck inside one machine. For executives, the value is flexibility: You can handle spikes, run larger workloads without overbuying top-tier accelerators, and improve utilization of expensive hardware.

DESIGN PLAYBOOK FOR NEXT-GEN AI SYSTEMS

For C-Suite leaders building AI systems that are fast, reliable, and cost efficient, these four strategies can help you build faster products with steadier costs, and predictable scaling.

  1. Start from the customer promise, then map the infrastructure.
    Decide what you are promising users, such as response time, peak traffic, and how much context the system can handle. Those choices drive memory needs, and memory often determines real-world performance more than headline compute numbers.
  2. Treat hardware availability as a launch dependency.
    The best configuration is meaningless if you cannot get it in time. Build plans around lead times, secure allocations early, and avoid designs that depend on a single constrained component.
  3. Use a tiered approach.
    Keep the most frequently used information in the fastest memory, place less frequently used data in cheaper storage, and use shared memory pools when you need elasticity. This helps balance speed and cost and reduces painful redesigns as usage grows.
  4. Make power and cooling part of the business case.
    Faster memory and denser systems can raise power density. If you cannot cool it efficiently, you lose the gains through higher operating costs, lower reliability, or both.

STRATEGIC CHOICES IN AI HARDWARE

If you want AI performance that holds up in production, ask questions that map to business outcomes, not just peak specs.

  • Require performance at your real workloads. Ask vendors to show results at your typical prompt lengths and document sizes, and under real concurrency. Systems that look fast on short prompts can slow down sharply on long context workloads.
  • Measure outcomes that translate directly to cost and customer experience. Track responsiveness under load, throughput per server, and energy per unit of output. These are the numbers that determine how many customers you can serve, what it costs, and how predictable scaling will be.
  • Make supply chain and upgrade path part of the decision. Evaluate what you can actually procure, how quickly you can expand, and whether there is a clean path to the next generation, without a full rebuild. Availability and roadmap clarity often matter as much as raw speed.

FINAL THOUGHTS

The next wave of GenAI winners will be decided both by raw compute and by how well companies manage the “memory supply chain” inside their AI systems. If memory is too slow, too small, or too far from the processor, the AI chip spends expensive time waiting, and your product feels sluggish, your costs climb, and scaling becomes unpredictable.

For executives, the takeaway is practical:

  • Performance: Faster, more consistent response times, especially for long prompts and long documents.
  • Features: The ability to offer longer context, richer workflows, and more reliable tool use.
  • Unit economics: Lower cost per response, better utilization of expensive hardware.
  • Delivery risk: Fewer surprises tied to hardware availability and lead times.

Shashank Chaurasia is the Co-Founder and Chief AI Officer at MooresLab AI.