The Future Role of AI Engineers
How specialization, open source contributions, and focus on reliability are shaping the field
AI engineering is a unique domain experiencing immense changes daily; we witness this firsthand in our daily work as AI practitioners.
In many ways, the rule book is still being written regarding successfully building with AI. That's why we wanted to discuss the evolution of this emerging profession.
We will explore topics like the differentiating skills AI engineers need compared to data scientists, staying up-to-date without getting overwhelmed when to use APIs versus local models, getting involved in open source initiatives, and principles for creating reliable AI solutions.
Let's get started!
Talking about the AI Engineer
I had the pleasure of chatting with Ronak for the Destination Big Leagues podcast. We had an in-depth discussion about AI engineering, generative AI models, and the future of artificial intelligence.
Here are five key takeaways from our conversation:
1️⃣ Generative AI has seen massive proliferation recently with models like DALL-E, GPT-4, and others. These models generalize better and can be used for diverse tasks, and can be applied across industries, compared to classic machine learning models that are very narrow in scope and bound to their training data.
2️⃣ There are tradeoffs to consider between open-source and proprietary models. Cloud-based models offer more convenience, while open source allows for customization. Regulatory requirements also play a role.
3️⃣ Building custom AI models requires significant data, computing resources, and expertise. Using existing generative models and then specializing them for specific use cases is often a fast and more cost-effective approach, especially when starting out.
4️⃣ China is advancing quickly in AI research and applications, but increased fragmentation between countries means less sharing of information and models across borders.
5️⃣ AI will lead to enormous productivity gains but also potentially greater inequality. Ensuring access to technology and employment opportunities for all segments of society needs to be an area of focus.
I encourage you all to have a listen to the full episode, where we go deeper into these topics and more. I am adding the link in the comments section.
Office Hours this Week
I recently had an engaging consulting session with Omar, an AI engineer, where we discussed the evolving role of AI engineers and best practices for building AI solutions. Omar shared insights from his experience and I provided guidance on skills development and contributing to open source projects.
The Role of the AI Engineer
AI engineers now often start with a pre-trained model and focus more on building reliable products rather than optimizing machine learning algorithms.
Debugging skills are critical - AI engineers must handle inconsistent inputs/outputs and system failures gracefully.
Bespoke solutions with fine-tuned models still provide value even as publicly available models improve. Most clients want customized UX, data privacy, etc.
Specialize in a specific model stack rather than chase the latest research - focus on robustness over cutting edge.
Best Practices for Building AI Solutions
Start simple - completions, then streaming, then function calls. Take a step-by-step approach when learning.
Use local LLM solutions when possible - they provide more control, privacy, and security.
Contribute to open-source projects to accelerate learning. I aim to contribute significantly to a project soon.
Join communities like Discord to exchange ideas and get help. I'm starting a Discord server for engineering discussions.
The questions
1. What do you see as the evolving role of AI engineers compared to more traditional data scientists focused on machine learning?
The evolving role of AI engineers compared to traditional data scientists focused on machine learning can be delineated through three key distinctions:
Utilization of Pre-trained Models and Cloud Endpoints: AI engineers primarily engage with pre-trained models, often exclusively interacting with cloud endpoints. These models, essentially black boxes for AI engineers, differ from the more transparent models traditionally employed by machine learning engineers or data scientists.
Deviation from Traditional Data Science Processes: In contrast to the conventional data science approach involving data collection, cleaning, ETL processes, and model fitting, AI engineers may lack insight into the specifics of the data used to train a model. Consequently, classic benchmarking techniques for model evaluation, reliant on a clear understanding of training data, may not be applicable in their workflow.
Exclusive Emphasis on Deep Learning and Transformer Models: AI engineers work exclusively with deep learning models, primarily those within the transformer family. This specialization distinguishes them from machine learning engineers and data scientists who may employ a broader spectrum of modeling techniques.
As the AI engineering role continues evolving, it increasingly converges with software engineering. Foundation Models will become the equivalent of modern kernels or transpilers. This transformation implies two key aspects: enhanced accessibility for individuals to utilize and build upon these models and heightened complexity for experts seeking to modify their architectures.
This shift mirrors a scenario where front-end developers can easily leverage foundation models to create impressive applications, but fine-tuning the model's architecture becomes a more intricate task.
Contrastingly, data scientists are evolving into individuals adept at business understanding, with data science becoming more of a skill than a distinct profession. Analogous to the lack of a profession centered around "mastering Excel," data science skills are expected to be integrated into various roles. For instance, a marketing manager may possess significant data proficiency and employ different data science techniques without being formally titled a "data scientist."
In this evolving landscape, ML engineers stand out as a constant force. Their role persists in ushering machine learning models into production and maintaining the overarching "ML platform" infrastructure, maintaining a consistent and critical position in the ever-changing AI ecosystem.
2. What specific skills or knowledge do you think are most important for an AI engineer to be successful?
In AI engineering, key skills for success include curiosity and a willingness to learn from mistakes. With new libraries, publications, and techniques emerging daily, it's important to be adaptable, as not all of them may be suitable for your specific needs.
Currently, there aren't established "best practices" or textbooks in the field, although efforts are underway to create a body of literature. In this environment, success relies on a spirit of experimentation and a willingness to try new approaches.
While familiarity with the literature on transformers is valuable, a more significant focus in today's AI landscape is proficiency in the Modern AI Stack (🌽MAIS). This comprehensive understanding of the latest tools and technologies is crucial. Stay tuned for future discussions on 🌽MAIS in upcoming issues.
3. Do you think AI engineers should focus more on staying on top of the latest research versus building robust and reliable products?
It's easy to get sidetracked by constantly chasing the latest preprints and spending more time reading than taking action.
Preprints are often not validated by the broader academic community and may be limited to very specific cases, making them less applicable in general. Additionally, the threshold for implementation might be quite high.
For a more beneficial career and knowledge gain, spending an hour with LangChain to understand the fundamentals of generative AI outweighs the advantages of reading a preprint.
4. What best practices would you recommend for engineers just getting started with building AI solutions?
Imagine yourself as a software architect, but with the awareness that you're working with stochastic outputs. This means your testing and debugging approach will differ significantly from a conventional application.
A recommended starting point is using LangChain. The framework already integrates top-notch techniques and methodologies.
5. How valuable do you think it is to contribute to open-source AI projects?
Engaging in open-source projects is an excellent avenue to build expertise and gain "street credibility" with potential employers and partners. However, it's not a fit for everyone, as some developers prefer to avoid working in the open.
The decision is yours, but if given the choice, I always recommend getting involved in an open-source project.
6. What are your thoughts on using local LLMs versus relying on third-party APIs?
We've reached a stage where, for many typical office scenarios, open-source models stand as competitive and practical choices. However, in certain domains like agents or code completion, their effectiveness is still a matter of debate. Personally, I lean towards open-source models for learning purposes.
Yet, when considering company and project management aspects, the decision depends on factors like budget constraints and limitations tied to the specific use case and application you're working on.
7. How vibrant is the AI community in the UK and London specifically?
Very much so. Some weeks, I participate in 5-6 meetups and events, specifically industry-related. This count doesn't even include "paper reading clubs" or research group meetings. Currently, we're in a phase where individuals are generally at similar AI learning levels, all at the beginning of their AI journey. This environment fosters openness to collaboration and knowledge-sharing. London, in particular, stands out as one of the global AI hubs.
This is all for this week! Follow The AI Engineer on LinkedIn and Twitter/X. We are the AI community for hackers! We are the AI ducktypers!
Remember that we are having free office hours every Wednesday. Have you already signed up for yours? Do it now! Here