A Cross-Stack Approach to Efficient and Scalable Generative AI (Special Session Paper)

This paper presents a holistic, cross-stack vision for efficient and scalable generative AI, motivated by the view that efficiency is most readily improved when hardware, compiler, and algorithmic techniques are pursued together rather than in isolation. We connect four complementary perspectives: (i) foundational hardware innovations, including heterogeneous 2.5D/3D integration and in-memory computing, that reduce the compute, memory, and energy footprint of billion- parameter models; (ii) extensible compiler infrastructures, built around MLIR, that bridge emerging in-memory hardware to real applications and automate design-space exploration; (iii) cross-layer and algorithmic techniques –quantization, sparsification, and dataflow optimization– that compress and accelerate large language models from the cloud to the edge; and (iv) token reduction methods, such as token merging and clustering, that attack the quadratic cost of the transformer attention mechanism at the model level. By integrating hardware, compiler, and algorithm viewpoints into a single narrative, we outline how co-designed solutions across the stack can compound to enable affordable, scalable, and democratized access to generative intelligence.