Image of podcast

Expanding AI chip capabilities beyond Nvidia with Modular CEO Chris Lattner | E1808

This Week in Startups

Wed Sep 13 2023



  • Background of LLVM technology:

  • LLVM is a compiler technology used in devices like phones, laptops, and consoles.

  • It unified compute around CPUs by supporting multiple programming languages.

  • The need for a unified AI stack:

  • Machine learning lacks a solution like LLVM for hardware integration.

  • Modular aims to build an AI stack that can be plugged into any hardware, simplifying deployment and distribution of AI models.

  • Importance of software development in the AI space:

  • Building AI models involves training, deploying, and distributing them into products.

  • Existing ML systems focus on research and training, making the transition to production challenging.

  • Challenges in deploying ML models:

  • Deploying ML models requires addressing different problems than training.

  • Fragmentation and complexity in existing systems make it difficult to integrate models into various platforms.

  • Future of hardware and optimization:

  • Advancements in wearables, personal computing, AR/VR technologies are leading to more specialized and customized hardware.

  • Customized chips will play a significant role in future developments.

  • Open-source community and RISC-V:

  • RISC-V is an open-source instruction set architecture allowing anyone to create their own member of the RISC-V family.

  • This innovation has led to new hardware designs but highlights the need for scalable software solutions.

  • NVIDIA's dominance and partnership with Modular:

  • NVIDIA succeeded through programmability and support for general compute on GPUs.

  • Modular sees NVIDIA as an important partner due to complementary missions aimed at expanding the developer ecosystem.

AI wrapper debate:

  • Vertical AI apps vs. relying on a single language model like GPT-3.
  • Verticalized apps provide more specialized solutions compared to general-purpose models.
  • Different applications require different approaches, both have value depending on the use case.

Transition from nonprofit to for-profit by OpenAI:

  • Perceived as a money grab by some, while others understand the need for financial sustainability.
  • Expecting companies to invest billions of dollars into free products is unrealistic.

How much faster are developers getting, in your estimation?:

  • Developers have become significantly faster over time.
  • Improved tools and technologies contribute to increased development speed.

Comparison with hardware transitions:

  • Hardware transitions took a long time (e.g., two decades for PCs).
  • Software-based platforms like Google and Uber took around 10 years to deploy widely.
  • AI advancements may happen faster than hardware transitions due to lower manufacturing delays.

Challenges faced by startups:

  • Startups often believe they can quickly build successful products based on existing technology.
  • Intense competition arises if one startup can build something in a month, leading to the need for incremental progress and delivering value.

Modular's approach to tackling complexity:

  • Modular aims to simplify the complex AI infrastructure landscape.
  • They provide a unified solution that addresses fragmentation and infighting among different companies and groups.
  • Mojo, their programming language, allows Python developers to scale code effectively.

Mojo's role in addressing Python limitations:

  • Mojo enhances Python capabilities for high-performance tasks like AI and machine learning.
  • It enables customization while maintaining compatibility with the existing Python ecosystem.

AI Engine as a drop-in replacement for TensorFlow and PyTorch:

  • Modular offers the AI Engine, seamlessly replacing popular frameworks without code rewriting.
  • This consolidates various point solutions used by enterprises.

Compatibility with different hardware platforms:

  • Making Mojo compatible with each platform is challenging but crucial for scalability.
  • Modular's expertise allows them to bring up new architectures quickly, achieving performance uplifts on Intel CPUs, AMD CPUs, and ARM-based cloud servers like Graviton.

Apple's involvement in AI:

  • Apple focuses on client-side running of AI models on devices like laptops and phones.
  • They optimize their hardware and software for this purpose, providing a seamless user experience.

Modular's hiring and company culture:

  • Modular tackles a challenging technology problem in the AI infrastructure space.
  • They emphasize building high-quality production-ready solutions rather than rushing from demo to demo.
  • The company offers opportunities to impact millions of developers by enabling them to participate in AI development.