Problem Solving: Intro

Typically in data science education, there is heavy focus on technical skills, but once on the job the most important asset is being able to efficiently solve ambiguous problems. What is the best way to systematically and progressively solve data science problems?

Technical Problem Solving Series – Setting The Context

In “data science”, I, and I’m assuming many others, have focused primarily on the data component of the term: the modeling, tidying, exploring, programming, visualizing, etc. hard skills. — but what about the science part? Have I been missing this component? I mean.. what does it even mean to do proper science? What does it mean to do proper research? What is best way to take a general objective and without an obvious solution, take a scientific and methodical approach to achieve an answer that I’m either happy with, or know that I’ve done the best that could be accomplished in the given time?

In this series of posts I’m exploring the concepts of science, research, and problem solving, to come to conclusions about how this type of work should be done to achieve optimal outcomes.

But First – Here’s some background on myself to explain why this is interesting and important to me.

Education

I have a background in “data science”. As an undergraduate I studied Industrial Engineering which included courses in statistics, probability theory, simulation, stochastic modeling, deterministic modeling/optimization, time series and forecasting, calculus, differential equations, linear algebra, etc. On top of college courses, I took the popular moocs such as the Johns Hopkins DS course, Andrew NG’s ML course, MIT open courseware, etc. After undergrad, I continued education with a Masters in Analytics (both from NCSU) which included more linear algebra, simulation, regression, clustering, maths, etc.

The point being; I have spent many years studying common “data science” fundamentals, techniques, and applications.

Work

As of May 2020, I’m starting my 4th year as a working professional. I have had three different jobs in the data science field, and three very different roles that came with each.

KPMG

The first role was a “Data and Analytics Predictive Modeler” — consultant at KPMG. I helped an organization set up an analytics practice, and for a large portion of my time I worked on a research project where we simultaneously developed methodology and coded it up into a ‘production’ tool, with the objective to turn it into a product to sell externally as part of an engagement. This was my first taste of research. Why did it feel so stressful?

Visa

My second role was a “Decision Analytics Architect” which became “Data Scientist” at Visa. During my time there I worked with one of our marketing teams to provide analytics support. About 4-5 months into the role, my focus shifted and I was tasked on a research project. It lasted about 6-8 months. Similarly to KPMG, this was a simultaneous - develop a new methodology and turn it into a production tool - project. It sounds awesome — I get to work on developing cutting edge techniques with “big data” technologies like spark, Hadoop, etc. — so why was I experiencing the same stress that I had felt at KPMG? Why was I so unhappy?

I know I love ‘data science’ work — Maybe I was unhappy because there was too much research time and not enough focus on business impact.

Credit Karma

My current role (as of May 2020), is a “Senior Analyst” on the Partnership Analytics team. This role is a mix of: time critical work that has visibility into the C-Suite, Action oriented analyses, presentations to leadership, presentations & travel to our external CC partners, business monitoring, and analytical question answering. There are also some more advanced tasks I will take on such as identifying high risk populations through modeling.

Even at Credit Karma, When I take on a technical challenge I’ve noticed higher levels of stress than other work — why is this? Shouldn’t I be excited by the fact I get to work on something difficult and stimulating? In this scenario — I believe the stressor is knowing there are expectations of fast progress and either not seeing a clear path forward or not being confident that the path I am taking will lead to a successful outcome. This leads to a separate discussion.

Overall, I’ve spent a lot of time on the hard skills and business communication, but not much time on problem solving as a discipline itself. I think there is an opportunity for universities to teach more about how to solve these types of ambiguous and unknown problems and provide frameworks to help people get from unknown to solution in a systematic approach.

This series of posts will be exploring how to do so, and what a course could look like.