For the first three years at OpenAI, becoming a machine learning expert felt like a distant dream, despite my aspirations. However, in the last nine months, I successfully transitioned into a machine learning practitioner. It was a challenging journey, but definitely achievable. I believe that many proficient programmers with a solid grasp of mathematics (or the willingness to learn) can also make this leap. While numerous online resources are available for self-study, the biggest hurdle I faced was psychological – becoming comfortable with being a beginner once again.
Early Days and the Machine Learning Aspiration
A core principle at OpenAI is the equal importance of research and engineering. Our mission is to create functional systems that tackle previously insurmountable challenges, requiring both skill sets. Our team composition reflects this: 25% primarily software-focused, 25% primarily machine learning-focused, and 50% hybrid roles. From OpenAI’s inception, my software skills were highly valuable, and I kept postponing the development of my desired machine learning expertise.
After contributing to the creation of OpenAI Gym, I shifted my focus to Universe. As the Universe project concluded, we embarked on Dota, and we needed someone to develop a reinforcement learning environment for the game before any machine learning initiatives could commence.
Diving into Dota and the ML Learning Curve
Transforming such a complex game into a research environment without access to the source code was truly inspiring. The team’s enthusiasm with each obstacle I overcame was incredibly motivating. I successfully navigated Lua sandboxes, utilized LD_PRELOAD in a Go GRPC server for game control, incrementally extracted the complete game state into Protobuf, and developed a Python library with abstractions for future multi-agent configurations.
However, I felt like I was operating with limited vision. At Stripe, my previous role, while I leaned towards infrastructure solutions, my deep understanding of the product code allowed me to contribute across the entire stack. In the Dota project, my perspective was confined to software solutions. At times, this led me to attempt complex software solutions for problems that could have been more efficiently addressed with slight adjustments in the machine learning approach.
I aspired to emulate teammates like Jakub Pachocki and Szymon Sidor, the masterminds behind the core breakthrough that powered our Dota bot. They challenged the prevailing belief within OpenAI that reinforcement algorithms lacked scalability. They developed Rapid, a distributed reinforcement learning framework, scaling it exponentially, overcoming perceived limitations. I aimed to make similar impactful contributions, blending both software and machine learning expertise.
In July 2017, an opportunity arose. With the software infrastructure stabilized, I embarked on a machine learning project. My objective was to employ behavioral cloning to train a neural network using human training data. However, I was unprepared for the profound sense of being a novice.
I constantly struggled with minor workflow details, creating uncertainty about my progress. Simple issues like tracking the exact code used for an experiment or comparing results against improperly archived past data became major roadblocks. Worse, I kept uncovering subtle bugs that had been consistently skewing my results.
My lack of confidence was palpable, yet ironically, others seemed to have confidence in me. People would comment on the inherent difficulty of behavioral cloning from human data. I always made sure to correct them, emphasizing my beginner status and attributing any struggles to my inexperience rather than the problem’s complexity.
There was a fleeting moment of validation when my code was incorporated into the bot. Jie Tang used it as a foundation for creep blocking, later refining it with reinforcement learning. However, this success was short-lived. Jie achieved even better results without my code, leaving me with nothing tangible to show for my efforts.
After this experience, I didn’t attempt machine learning again within the Dota project.
Time for Focused Learning and Stepping Back
Following two defeats at The International in 2018, many observers concluded that we had reached the limits of our approach. However, our internal metrics indicated that we were close to a breakthrough, primarily needing more training. This shift meant my time demands lessened, and in November 2018, I saw an opportunity to dedicate three months to focused learning.
I learn most effectively when I have a specific project in mind. I decided to build a chatbot. I began self-studying the curriculum we had developed for our Fellows program, concentrating on the NLP-related modules. For instance, I developed and trained an LSTM language model and then a Transformer-based model. I also delved into topics like information theory and meticulously studied numerous research papers, ensuring complete comprehension.
Progress was slow, but this time, I anticipated it. I didn’t experience the flow state I was accustomed to in software engineering. It reminded me of my early days of programming, and I recalled the years it took to achieve a sense of mastery. Frankly, I lacked confidence that I would ever become proficient in machine learning. Yet, I persisted, driven by a desire to understand the entirety of my projects, not just a fraction of them.
My personal life played a crucial role in my perseverance. I had started a relationship with someone who fostered a sense of comfort with the possibility of failure. Our first holiday season together was spent with me grappling with machine learning challenges, but she remained supportive, even when it meant altering our plans.
A significant conceptual breakthrough was overcoming my reluctance to modify someone else’s machine learning code – a hesitation that had hindered me in the Dota project. I fine-tuned GPT-1 on chat datasets and implemented my own rudimentary sampling code. However, generating longer messages became incredibly slow, leading to frustration that surpassed my fear. I then implemented GPU caching, a modification that impacted the entire model.
I had to iterate several times, discarding changes that became too complex to manage mentally. After a few days, when I finally got it working, I realized a profound shift had occurred. I had grasped the architecture of the entire model, even down to subtle stylistic choices in the codebase, like the elegant handling of TensorFlow variable scopes. This level of understanding had previously seemed unattainable.
Re-Engaging and Moving Forward in Machine Learning
After three months of dedicated self-study, I felt prepared to tackle a real project. This was also the point where I recognized the value of leveraging the expertise within OpenAI. I was thrilled when Jakub and my co-founder Ilya Sutskever agreed to mentor me.
We started to achieve very promising results, and Jakub and Szymon joined the project full-time. It’s a source of pride every time I see their commits in the machine learning codebase I initiated.
I am now beginning to feel competent, although mastery is still a future goal. This growing competence is reflected in my focused work hours dedicated to machine learning, now reaching about 75% of my historical coding hours.
For the first time, I feel like I’m on the right path. Initially, the constant influx of new machine learning concepts felt overwhelming. Within six months, I realized I could progress without constantly learning entirely new fundamental concepts. I still need to gain more experience in areas like network initialization and learning rate scheduling, but the work now feels incremental and achievable, rather than potentially impossible.
From our Fellows and Scholars programs, I knew that software engineers with strong foundations in linear algebra and probability could transition into machine learning engineers with just a few months of dedicated study. Yet, I had convinced myself that I was an exception and incapable of Learning Machine Learning. I was wrong. Even within OpenAI, my inability to transition was due to my unwillingness to embrace being a beginner again.
You are likely not an exception either. If you are interested in becoming a deep learning practitioner, you absolutely can. You need to allow yourself the time and space to fail and learn from those failures. Success will follow, likely in a much shorter timeframe than you anticipate.
Eventually, surrounding yourself with experienced experts becomes crucial. This is where I consider myself incredibly fortunate. If you are a talented software engineer reaching this point, remember there’s a way to be surrounded by the same experts I am – apply to OpenAI!
