Fri Jun 21 2024

Transition to AI Role at Elicit:

• James Brady made a significant career shift from a traditional VP Engineering role to an AI-focused position at Elicit due to the rapid advancements in AI capabilities, particularly language models like GPT-3.
• The hiring process for AI engineers at Elicit emphasizes the importance of candidates possessing conventional software engineering skills, a genuine curiosity and enthusiasm for machine learning and language models, and a fault-first mindset to handle the unpredictable nature of language models effectively.
• "The first thing to say is that we've effectively been hiring for this kind of role since before you coined the term... it was something that we backed into if you will. We didn't sit down and come up with a brand new role from scratch." - James

Challenges in Handling Language Models:

• Working with language models necessitates implementing defensive coding practices such as retries, fallbacks, timeouts, error handling, and parallelization due to their high variance in latency and unpredictable responses.
• Ensuring system reliability when dealing with chaotic language models is crucial to provide users with a stable experience despite the inherent unpredictability of model outputs.
• "Compared to normal APIs where normal... think of something like a Stripe API or search API or something like this, conventional search API, the latency when you're working with language models is wild." - James

Adapting to Unreliable Systems:

• Implementing defensive strategies like retries and fallbacks can be costly but essential when working with unreliable systems like language models.
• Balancing user experience against infrastructure costs poses a challenge for AI engineers when incorporating defensive measures in applications powered by language models.

Centralization vs. Decentralization in AI Development:

• Larger organizations may opt for centralized AI gateways to standardize technology choices across multiple teams developing AI products.

Enthusiasm for New Capabilities in Language Models:

• Maintaining curiosity about new capabilities in language models is vital for AI engineers at Elicit who constantly evaluate new releases like Anthropic Cloud 3.5.

Balancing Defensive Mindset with Innovation:

• Transitioning from code-centric architectures to core LLM (Language Model) apps requires balancing defensive strategies with innovative approaches to leverage evolving technologies effectively.

AI Engineer Hiring Process:

• The hiring process for AI engineers involves seeking individuals with conventional engineering skills and familiarity with language models, without requiring them to be full-fledged Machine Learning experts.
• Elicit has a technical team comprising both ML experts and what they term as AI engineers.
• The interview process at Elicit is structured to mirror actual work scenarios, serving as an assessment of technical capabilities while also allowing candidates to evaluate if the company aligns with their career objectives.
• --Implications of an ML-First Mindset:--
• Embracing an ML-first approach necessitates unlearning traditional software development patterns and adapting to interacting with opaque black box models.
• This mindset entails acknowledging inherent uncertainty when working with language models, leading to a different defensive programming style compared to standard software development practices.
• Adopting the ML-first mindset can result in unforeseen powerful outcomes by fully leveraging model capabilities rather than overly restricting them.

Balancing Systematic Control and Creativity in Engineering:

• Engineers must strike a balance between systematic control elements and creative, open-ended thinking when engaging with new technologies like language models.
• Finding this equilibrium is crucial for success in AI engineering, where qualities such as curiosity-driven exploration, artistic endeavors, systematizing processes, and maintaining control are all vital.

Sourcing AI Engineers:

• Effective strategies for sourcing AI engineers include active outbound methods like Twitter engagement, conference attendance, hackathon hosting, or releasing open-source projects.
• Sourcing approaches vary based on experience levels; seasoned professionals may be discovered through platforms like LinkedIn, while less experienced individuals might be identified through blog posts or challenges showcasing their skills.
• Specialized job boards linked to specific missions or affiliations can aid in targeting candidates interested in niche areas such as AI safety or effective altruism.