Michael Karl Gschwind is an American computer scientist at Nvidia in Santa Clara, California. He is recognized for his seminal contributions to the design and exploitation of general-purpose programmable accelerators, as an early advocate of sustainability in computer design and as a prolific inventor.
Accelerators Gschwind led hardware and software architecture for the first general-purpose programmable accelerator Accelerators and is widely recognized for his contributionsHeterogeneous computing as architect of the Cell Broadband Engine processor used in the Sony PlayStation 3,Scarpino, M. (2008). Programming the cell processor: for games, graphics, and computation. Pearson Education. and RoadRunner, the first supercomputer to reach sustained Petaflop operation. As Chief Architect for IBM System Architecture, he led the integration of Nvidia GPUs and IBM CPUs to create the Summit and Sierra supercomputers.
Gschwind was an early advocate for accelerator virtualizationhttps://on-demand.gputechconf.com/gtc/2017/presentation/S7320-tim-kaldewey-optimizing-efficiency-of-deep-learning-workloads-through-gpu-virtualization.pdf, https://on-demand.gputechconf.com/gtc/2017/presentation/S7320-tim-kaldewey-optimizing-efficiency-of-deep-learning-workloads-through-gpu-virtualization.pdfOptimizing the efficiency of deep learning through accelerator virtualization, https://ieeexplore.ieee.org/document/8030299 and as IBM System Chief Architect led I/O and accelerator virtualization.I/O Vrtualization and System Acceleration in Power9, https://old.hotchips.org/wp-content/uploads/hc_archives/hc27/HC27.24-Monday-Epub/HC27.24.30-HP-Cloud-Comm-Epub/HC27.24.340-IO-Virtualization-POWER8-Gschwind-IBM.pdf
Gschwind has had a critical influence on the development of accelerator programming models with the development of APIs and best practices for accelerator programming,Chip Multiprocessing and the Cell Broadband Engine, https://computingfrontiers.org/2006/cf06-gschwind.pdfCBE Programming HandbookCBE Programming Tutorial, https://public.dhe.ibm.com/software/dw/cell/CBE_Programming_Tutorial_v3.1.pdf application studies for a diverse range of HPC and non-HPC applications.Cell GC: using the cell synergistic processor as a garbage collection coprocessor, ACM Virtual Execution Environments, https://dominoweb.draco.res.ibm.com/reports/rc24520.pdf and as co-editor of booksM. Gschwind, F. Gustavson, J. Prins (eds), High Performance Computing with the Cell Broadband Engine Scientific Programming 2009, https://www.semanticscholar.org/paper/High-Performance-Computing-with-the-Cell-Broadband-Gschwind-Gustavson/c6775765100eb3b9eb7b7bc003a8eba1ca90667f and journalsM. Gschwind, M. Perrone (Eds), Topical Issue On Hybrid Systems IBM Journal of Research and Development 53(5):1-2 September 2009, DOI:10.1147/JRD.2009.5429079 on practice and experience of programming accelerator-based systems.
AI acceleration Gschwind was an early advocate of AI Hardware Acceleration with GPUs and programmable accelerators. As IBM's Chief Engineer for AI, he led the development of IBM's first AI products and initiated the PowerAI project which brought to market AI-optimized hardware (codenamed "Minsky"), and the first prebuilt hardware-optimized AI frameworks. These frameworks were delivered as the firstfreely installable, binary package-managed AI software stacks paving the path for adoption.
At Facebook, Gschwind demonstrated accelerated Large Language Models (LLMs) for Facebook's First Generation ASIC accelerators and for GPUs, leading the first LLM production deployments at scale for embedding serving for content analysis and platform safety, and for numerous user surfaces such as Facebook Assistant, and FB Marketplace starting in 2020. Gschwind led the development of and is one of the architects of Multiray, an accelerator-based platform for serving foundation models and the first production system to serve Large Language Models at scale in the industry, serving over 800 billion queries per day in 2022.MultiRay: An Accelerated Embedding Service for Content Understanding, https://static.sched.com/hosted_files/pytorch2023/60/PyTorch_Conf_2023-Multiray.pdf
Gschwind led the company-wide adoption of ASICFirst-Generation Inference Accelerator Deployment at Facebook, https://arxiv.org/pdf/2107.04140.pdf and Facebook's subsequent "strategic pivot" to GPU Inference, deploying GPU Inference at scale, a move highlighted by FB CEO Mark Zuckerburg in his earnings call. Among the first recommendation models deployed with GPU Inference was a Reels video recommendation model which delivered a 30% user surge within 2 weeks of deployment, as reported by FB CEO Mark Zuckerburg in his Q1 2022 earnings call, and a subsequent $3B to $10B growth for REeels year-over-year.
Gschwind also led AI Accelerator Enablement for PyTorch with a particular focus on LLM acceleration, leading the development of Accelerated Transformers (formerly "Better Transformer") and partnered with companies such as HuggingFace to drive industry-wide LLM Acceleration to establish PyTorch 2.0 as the standard ecosystem for Large Language Models and Generative AI.
Gschwind subsequently led expanding LLM acceleration to on-device AI models with ExecuTorch, the PyTorch ecosystem solution for on-device AI, making on-device generative AI feasible for the first time. ExecuTorch LLM acceleration (across multiple surfaces including NPUs, MPS, and Qualcomm accelerators) delivered significant speedups making it practical to deploy Llama3 unmodified on servers and on-device (demonstrated on iOS, Android, and Raspberry Pi 5) at launch with developers reporting up to 5x-10x speedups over prior on-device AI solutions. Gschwind's contributions also extend to AI software stacks and frameworks, AI accelerators, mobile/embedded on-device AI and low-precision numeric representations in torchchat.
Gschwind is a pioneer and advocate of Sustainable AI.Sustainable AI: Environmental Implications, Challenges and Opportunities, https://arxiv.org/pdf/2111.00364.pdf
Supercomputer design Gschwind was a chief architect for hardware design and software architecture for several supercomputers, including three top-ranked supercomputer systems Roadrunner (June 2008 – November 2009), Sequoia (June 2012 – November 2012), and Summit (June 2018 – June 2020).
Roadrunner was a supercomputer built by IBM for the Los Alamos National Laboratory in New Mexico, USA. The US$100-million Roadrunner was designed for a peak performance of 1.7 petaflops. It achieved 1.026 petaflops on May 25, 2008, to become the world's first TOP500 LINPACK sustained 1.0 petaflops system. It was also the fourth-most energy-efficient supercomputer in the world on the Supermicro Green500 list, with an operational rate of 444.94 megaflops per watt of power used.
Sequoia was a petascale Blue Gene/Q supercomputer constructed by IBM for the National Nuclear Security Administration as part of the Advanced Simulation and Computing Program (ASC). It was delivered to the Lawrence Livermore National Laboratory (LLNL) in 2011 and was fully deployed in June 2012.NNSA awards IBM contract to build next generation supercomputer, February 3, 2009 Sequoia was dismantled in 2020, its last position on the top500.org list was #22 in the November 2019 list.
Summit is a supercomputer developed by IBM for use at Oak Ridge Leadership Computing Facility (OLCF), a facility at the Oak Ridge National Laboratory. It held the number 1 position from November 2018 to June 2020. Its current LINPACK benchmark is clocked at 148.6 petaFLOPS.
Many-core processor design Gschwind was an early advocate of many-core processor design to overcome the power and performance limitations of single-processor designs. Gschwind co-authored an analysis of the limitations of frequency scaling which arguably led to an industry-wide transition to many-core designs. Gschwind was a lead architect for several many-core designs, including the first commercial many-core processor Cell with 9 cores, BlueGene/Q with 18 cores, and several enterprise and mainframe processors (POWER7/POWER8/POWER9 with up to 24 cores; z10-z15 with up to 12 cores).
As chip chief architect and chief microarchitect, Gschwind was critical to the reboot of the POWER architecture after the POWER6 high-frequency high-power dead-end, leading revival of the POWER5-style out-of-order design with POWER7, serving as unit lead and chief microarchitect for the instruction fetch, decode and branch prediction unit (also including logical instruction execution), and as acting lead for most other units at one point during the design. In subsequent generations of the POWER architecture, integration of the VMX SIMD design and FPU into VSX, little-endian support in POWER8 laying the foundation for little-endian PowerLinux (used in the Google POWER prototype and for GPU integration for the Minsky PowerAI system), and integration of NVLink for optimized GPU/CPU integration; and native support for Linux-style hardware-managed radix page-tables in POWER9, used in the world-leading Summit and Sierra Power+Nvidia supercomputers; and the introduction of PC-relative addressing and prefix instructions to transcend the limitations of the 32-bit instruction encodings of RISC architectures in POWER 10.
As architecture lead/manager and cross-platform chief architect, Gschwind also led the reboot of system z mainframe, with introduction of compiled code efficiency (with a particular view to C, C++ and Java) in IBM z10, out-of-order execution, PCIe-based I/O in z196 and z114, support for transactional memory in IBM zEC12, introduction of hardware multithreading and z/Vector SIMD architecture (including shared software infrastructure with Power's VSX) in IBM z13; and the sunsetting of ESA390 for operating systemsCommon boot sequence for control utility able to be initialized in multiple architectures, US Patent 9,588,774, https://patents.google.com/patent/US9588774B2 substantially reducing verification and design complexity and improving time-to-market in IBM z14.
System reliability Gschwind coined the term "reliability wall" for obstacles to sustained operation of large-scale systems. He has made major contributions to system-level reliability modeling and improvements, with a particular view to enabling sustained supercomputing system operation. As chief architect of BlueGene/Q, he led system-level reliability and processor design in addition to being the chief ISA architect and QPU vector floating point unit design lead.
Gschwind led the first processor and chip-level architectural vulnerability modeling and selective hardening to achieve target MTBF, first implemented in BlueGene/Q using stacked DICE latches for critical state-holding latches. To increase system reliability while avoiding the performance and power cost associated with ECC-based designs, Gschwind proposed and led the design of register files and minor buses protected with parity with state recovery. In accordance with this approach, error detection is implemented in datapaths which may occur in parallel with initiating compute operations, with a recovery operation when a soft error is detected in parallel with the operation. Recovery then proceeds from good-state maintained in alternate copies of the register file commonly used to scale the number of register file read portsa and reduce wiring delay from register file reads to execution units.
Compiler technologies Gschwind has made seminal contributions to compiler technology, with a particular emphasis on pioneering contributions to just-in-time compilation, dynamic optimization, binary translation and compilers in supercomputing.
Just-in-time-compilation