PhD and Data Science: Challenges and Opportunities with Startups

This article attempts to summarize potential expectation mismatches graduates with a PhD degree experience while working in the software industry, specifically in startups having a Machine Learning and Artificial Intelligence (AI) focus. Some of the opinions expressed here are based on personal experiences and recent attempts to hire engineers with a PhD degree.

Startups Embrace the Artificial Intelligence Revolution

In recent times, Startups have been churning out complex products involving techniques that have mathematical underpinnings. The industry loosely refers to these techniques as AI. Purists, however, would argue that prevalent techniques in natural language processing (NLP), computer vision, speech processing and image processing that depend on statistical algorithms and machine learning algorithms are not really AI.

A decade ago, a PhD with an AI background in optimization, machine learning, computer vision, NLP, or pattern recognition was hardly considered for employment in Startups. However, over the last few years, there has been a remarkable change in the way these PhDs are being perceived, given the huge demand for Data Scientists in most spheres. And for the same reason, a PhD in computer science with a sprinkling of mathematics in their course work or in their thesis, seem to have an edge for the role of a Data Scientist today.

Hiring Demand for PhDs Soars

Given that a lot of startups are throwing in AI into their vision, it is natural that the engineering VPs driving these products wish to hire people who understand these data-driven or machine learning or AI algorithms in the context of Data (images, speech, telemetry, text etc) and Domain (FinTech, Wireless Networking, BioMedical imaging etc). Most of the products do not lend to black-box machine learning. The ideas are so unique that one needs to dissect known machine learning algorithms, modify them, create newer techniques, and combine multiple algorithms in order to deliver the final result. As these algorithms have mathematical foundations, people with a PhD degree in mathematical subjects are in huge demand. They are expected to not only understand the inner working of these algorithms, but are also expected to create new algorithms in the context of business problem, data platform and available data. Further, exploratory work is expected to be completed within time limits with the end result being software that can go to production.

Despite the requirement for PhDs with mathematical backgrounds, the number hired in startups is still significantly lower as full-stack developers are preferred over those with a PhD degree. One of the reasons could be the skepticism in the minds of engineering VPs with regard to a PhD’s capability to deliver “productizable real-world solutions”. With so much open-source software available, decision makers pressed for time think that integrating available algorithms into the product is a practical option, and full-stack engineers are adept at it.

However, if open-source could solve an out-of-the box problem, then every startup would end up building a similar product there by cannibalizing each other. Thankfully, many startups have a clear vision to be market leaders and are solving problems where critical thinking and grass-roots innovation are required in order to build the product. Therein lies a huge opportunity for graduates with a PhD, provided they are hands-on.

What does being hands-on entail?

It is the ability to convert theory into working code that powers the product feature. However, to power a product feature there are many steps to be followed and engineering practices to be adopted. In Startups, the role of a PhD is not limited to finding the solution to a problem. The PhD is expected to (A) build the solution in a working form, (B) adapt the solution to the platform and architecture (C) work with other engineers to get the solution into production (D) follow programming best practices (E) determine the solution approach within a short time span (F) find a solution that can be incrementally improved upon.

Note to fresh Graduates

Many fresh graduates on completion of their PhD believe that their thesis work is what the industry is currently looking for. I believed the same many years ago. However, barring a few cases, the PhD thesis has no direct relevance to the immediate needs of the industry. What perhaps is relevant is the training that one undergoes during the PhD programme. One needs to look at the 5+ years spent on a PhD thesis as a journey involving (a) learning and building domain knowledge (b) exploration (c) taking risks (d) making mistakes, and changing course when one hits a roadblock. Organizations frequently look for experiences that demonstrate the following

• ability to identify the heart of a complex problem
• ability to dissect a problem into smaller parts
• ability to practically implement a solution for parts that stitch themselves into the whole
• ability to improvise on a solution
• ability to use knowledge in one domain to a different domain
• ability to build a minimum viable solution as quickly as possible
• ability to re-skill oneself

PhD hires are expected to display aptitude for incremental product feature rollout pretty much like the product development engineers. Designing a solution for incremental rollout with improving quality and accuracy is as much a skill as it is an attitude. One needs to develop the skill to dissect a mathematical approach into parts such that an approximate solution can be built within the shortest time.

Summary

PhDs have to choose between two worlds. One is an engineering world requiring product development skills and an execution mindset. The other is an academic world that provides research opportunities and a career in teaching.

The expectations in these two worlds are very different. Those wishing to enter the engineering world of startups need to be aware of the expectations in a product development setting. PhDs who want to work as individual contributors have an excellent opportunity for a long career in startups, even if it means moving between startups. They also have an opportunity to grow with the startup and move into leadership/execution roles if interested.

However, caution is prudent. Not all startups can be successful, but working in startups leads to the development of varied skills that other startups prefer in one’s resume!

 

(1 votes, average: 4.00 out of 5)
Loading...