PodcastsLatent Space: The AI Engineer Podcast — Practitioners talking LLMs, CodeGen, Agents, Multimodality, AI UX, GPU Infra and all things Software 3.0High Agency Pydantic > VC Backed Frameworks — with Jason Liu of Instructor

High Agency Pydantic > VC Backed Frameworks — with Jason Liu of Instructor
Latent Space: The AI Engineer Podcast — Practitioners talking LLMs, CodeGen, Agents, Multimodality, AI UX, GPU Infra and all things Software 3.0Fri Apr 19 2024
Instructor - Patching LLM Provider SDKs and Use Cases:
- Instructor introduces a new response model option to enhance LLM provider SDKs, supporting models like OpenAI, Anthropic, Cohere, and others through LiteLLM.
- Core use cases of Instructor include extracting structured data from various inputs such as images or text, identifying nodes and edges in graphs for complex entity extraction, and defining schemas for API calls using language models.
Instructor - Transition to Embracing Transformers and Impact:
- Initially skeptical about transformers at Stitch Fix but transitioned after the release of Chanty PD.
- Shifted focus from computer vision recommendation systems to leveraging LLMs due to their significant potential.
- Developed a successful similarity search system utilizing Clip and GPT-3 embeddings within FICE that generated over $50 million in annual revenue.
Workflow Optimization with Language Models:
- Stresses the significance of workflows in AI applications for enhanced reasoning capabilities compared to reactive loops.
- Advocates for well-defined plans executed as workflows instead of relying on looping mechanisms to avoid inefficiencies.
Challenges in Model Integration and Performance Evaluation:
- Discusses challenges encountered when integrating different LLM frameworks into existing ecosystems due to varying performance metrics.
- Notes discrepancies between public benchmarks praising certain models versus real-world performance issues faced during production deployments.
Future Trends in AI Development Tools:
- Envisions a future trend towards workflow-based approaches for AI tasks rather than traditional looping mechanisms for improved efficiency.
- Foresees visual representations of Directed Acyclic Graphs (DAGs) playing a crucial role in effectively describing complex AI workflows.
Observability Tools and Consulting Strategy:
- Prefers simplicity by utilizing basic tools like Postgres, CurseCoder, PyTests, Datadog, or Sentry for observability needs over specialized observability startups.
- Opts for consulting services over establishing a venture-backed company despite having the capability to develop hosted versions of Instructor due to personal preferences towards free diving.
AI Engineering and Prompt Engineering:
- AI engineering involves developing agents, automations, and reports like private equity analyses based on user interviews.
- The focus is on structured outputs using prompt engineering techniques to enforce structured responses.
- Jason Liu created "Instructor," integrating Pydantic with LLM provider SDKs for data extraction, graph identification, and query comprehension.
- The goal of Instructor is to enhance LLM provider SDKs by offering a new response model option defined in Pydantic.
Challenges in Career Choices - Consulting vs. Startup Founder:
- Choosing between consulting and startup founding presents dilemmas related to career paths and financial expectations.
- Starting as a consultant can be financially rewarding without extensive funding or scaling efforts.
- Some individuals find happiness in raising small seed rounds, achieving profitability, and maintaining control over their ventures without external pressures.
Agency and Courage in Pursuing Goals:
- Having high agency involves demonstrating courage when pursuing challenging endeavors despite fear or uncertainty.
- Agency emphasizes taking action even when scared and focusing on process metrics rather than outcomes.
- It's crucial to start pursuing goals immediately instead of waiting for ideal conditions or feeling behind compared to others.
The Role of Prompts in AI Development:
- Prompts are vital for structuring interactions with language models (LLMs) by defining inputs and outputs as code objects.
- Tools like DSPy aim to streamline prompt engineering but may not align with all use cases requiring longer feedback loops for success measurement.
- Balancing business outcomes with prompt design is critical for effective AI development tailored towards specific objectives.
Talent Acquisition Challenges in AI Engineering:
- Companies face hurdles hiring machine learning engineers who may not fit the needs of AI engineering roles focused on prompt-based interactions.
- Motivated software engineers interested in AI engineering often struggle balancing traditional responsibilities with creative pursuits in AI development.
- Encouraging talent from various backgrounds such as data science into AI engineering can enhance problem-solving capabilities aligned with business objectives.
Elevating the Industry of AI Engineers:
- Initiatives like Latent Space's World's Fair aim to legitimize AI engineering as a subspecialty within software engineering by fostering creativity and innovation in the field.
- The industry shift towards quantifying business outcomes highlights the importance of measuring success based on real-world impact such as reduced churn rates.