PhD and Data Science: Challenges and Opportunities with Startups

This article attempts to summarize potential expectation mismatches for graduates with a PhD degree experience while working in the software industry. This happens specifically in startups having a Machine Learning and Artificial Intelligence (AI) focus. Some of the opinions expressed here are based on personal experiences and recent attempts to hire engineers with a PhD degree.

Startups Embrace the Artificial Intelligence Revolution

In recent times, Startups have been churning out complex products involving techniques that have mathematical underpinnings. The industry loosely refers to these techniques as AI. Purists, however, would argue that prevalent techniques in natural language processing (NLP), computer vision, speech processing and image processing that depend on statistical algorithms and machine learning algorithms are not really AI.

A decade ago, startups hardly considered a PhD student with an AI background in optimization, machine learning, computer vision, NLP, or pattern recognition as an employee. However, over the last few years, there has been a remarkable change in the way these PhDs are being perceived, given the huge demand for Data Scientists in most spheres. And for the same reason, a PhD in computer science with a sprinkling of mathematics in their course work or in their thesis, seem to have an edge for the role of a Data Scientist today.

Hiring Demand for PhDs Soars

Given that a lot of startups are adopting AI, it is natural for engineering VPs driving to hire people who understand machine learning, or AI algorithms in the context of Data (images, speech, telemetry, text, etc) and Domain (FinTech, Wireless Networking, BioMedical imaging, etc). Most of the products do not lend to black-box machine learning. The ideas are so unique that one needs to dissect known machine learning algorithms, modify them, create newer techniques, and combine multiple algorithms in order to deliver the final result. As these algorithms have mathematical foundations, people with a PhD degree in mathematical subjects are in huge demand. They should understand these algorithms’ inner workings and create new algorithms for business problems, data problems, and available data. Further, they should complete exploratory work within time limits and develop software as an end product to go to production.

Despite the requirement for PhDs with mathematical backgrounds, the number hired in startups is still significantly lower. This is because the industry still prefers full-stack developers over those with a PhD degree. One of the reasons could be the skepticism in the minds of engineering VPs with regard to a PhD’s capability to deliver “productizable real-world solutions”. With so much open-source software available, decision makers pressed for time think that integrating available algorithms into the product is a practical option, and full-stack engineers are adept at it.

However, if open-source could solve an out-of-the box problem, then every startup would end up building a similar product there by cannibalizing each other. Thankfully, many startups have a clear vision to be market leaders. They engage themselves in solving problems where critical thinking and grass-roots innovation are required to build the product. Therein lies a huge opportunity for graduates with a PhD, provided they are hands-on.

What does being hands-on entail?

It is the ability to convert theory into working code that powers the product feature. However, to power a product feature startups should adopt effective engineering practices. In Startups, the role of a PhD is not limited to finding the solution to a problem. His job is to (A) build the solution in a working form, (B) adapt the solution to the platform and architecture (C) work with other engineers to get the solution into production (D) follow programming best practices (E) determine the solution approach within a short time span (F) find a solution that can be incrementally improved upon.

Note to fresh Graduates

Many fresh PhDs believe that their thesis work can be a game-changer for the industry. I believed the same many years ago. However, barring a few cases, the PhD thesis has no direct relevance to the immediate needs of the industry. What perhaps is relevant is the training that one undergoes during the PhD programme. One needs to look at the 5+ years spent on a PhD thesis as a journey involving (a) learning and building domain knowledge (b) exploration (c) taking risks (d) making mistakes, and changing course when one hits a roadblock. Organizations frequently look for experiences that demonstrate the following abilities like

• identifying the heart of a complex problem
• dissecting a problem into smaller parts
• practically implementing a solution for parts that stitch themselves into the whole
• improvising on a solution
• using knowledge in one domain to a different domain
• building a minimum viable solution as quickly as possible
• re-skilling oneself

Employers expect PhD hires to display an aptitude for incremental product feature rollout like the product development engineers. Designing a solution for incremental rollout with improving quality and accuracy is both a skill and an attitude. One needs to develop the skills to dissect a mathematical approach and build an approximate solution within the shortest time.

Summary

PhDs have to choose between two worlds. One is an engineering world requiring product development skills and an execution mindset. The other is an academic world that provides research opportunities and a career in teaching.

The expectations in these two worlds are very different. Those wishing to enter the startup engineering world should be aware of expectations in a product development setting. PhDs willing to work as individual contributors have an excellent opportunity for a long career in startups, even if it means moving between startups. They also have an opportunity to grow with the startup and move into leadership/execution roles if interested.

However, caution is prudent. Not all startups can be successful. But working in startups helps develop varied skills that other startups prefer in one’s resume!