AI Product Management

AI/ML Product Management

80% of the ML Projects Fail.

1. https://hbr.org/2023/11/keep-your-ai-projects-on-track

2. https://www.kdnuggets.com/survey-machine-learning-projects-still-routinely-fail-to-deploy#:~:text=43%25%20say%20that%2080%25%20or,that%20their%20models%20usually%20deploy.&text=Key%3A,that's%20already%20been%20successfully%20deployed

Why?

1. Problem Framing

2. AI investment ROI

   1. Human Resource

   2. Compute Resource

3. AI Project expectations and inherent dissimilarities to software development

Problem Framing

1. Is there a customer/user problem?

1. End-user pain points

2. Operational improvements

2. Is there any business value in solving this problem?

Classify the business value as Low, medium or high

1. AI can either automate or Augment. Automate when risk is low and augment when risk is high. Risk is cost of being wrong. Medical diagnosis - high risk and therefore augment. Tech content writing - low risk and therefore, automate.

2. AI can automate, presonalize, predict

Dont use AI/ML because

• Everybody is doing it

• Leadership said so

• It’s a cool thing to do and we will have bragging rights

3. Can we solve this problem using ML/AI

Every problem can be solved by using heuristics (Non-ML) or Predictive AI or Gen AI.

For predictive AI or Gen AI solution, we have below questions to consider:

1. Classify as Easy, Hard, Impossible

AI can predict, classify or generate text or image. However, these capabilities are evolving fast.

B. Do we have the data to train AI in case of predictive AI?

4. What’s the ROI to solve this problem using AI?

1. High business value problem which can be easily solved with AI are the low hanging fruit

2. High business value problem which will hard to solve with AI is still worth pursuing

3. Consider the risk of using AI

   1. Environment change - frequent environment change required re-ingeering of ML Model. Adapting to changes in environment for ML can be costly and difficult

   2. Ethics - AI has no ethics or sense of wrong or right

   3. Accountability - who is responsible for decisions taken by AI? Developers?, service provider? 

4. Consider the following questions when comparing a non-ML approach to an ML one:

1. Quality. How much better do you think an ML solution can be? If you think an ML solution might be only a small improvement, that might indicate the current solution is the best one.

2. 
   

3. Cost and maintenance. How expensive is the ML solution in both the short- and long-term? In some cases, it costs significantly more in terms of compute resources and time to implement ML. Consider the following questions:

   1. Can the ML solution justify the increase in cost? Note that small improvements in large systems can easily justify the cost and maintenance of implementing an ML solution.

   2. How much maintenance will the solution require? In many cases, ML implementations need dedicated long-term maintenance.

   3. Does your product have the resources to support training or hiring people with ML expertise?

4. Predictions vs. actions: There's no value in predicting something if you can't turn the prediction into an action that helps users. That is, your product should take action from the model's output.

5. AI Project Management is costly because

1. Need Wider set of skills/team

2. Have higher technical risk

   1. 
      

   2. Data needs and quality

   3. Model limitations

3. Far more challenging to plan and estimate, 

   1. are not linear but experimentative in development

   2. Probabilistic and not deterministic

   3. Further work and experiments may not necessarily improve the model or accuracy

4. Harder to show progress

   1. Variance of model outputs

5. Require more ongoing support

ML Problem Solutioning

You frame a problem in ML terms by completing the following tasks:

• Define the ideal outcome and the model's goal.

  • Example: Predict when an item will be delivered to the customer-> Predict a data range by which customer will receive an item

• Identify the model's output.

  • Example: Min-Mac dates for delivery of an item

• Define success metrics.

  • Example: Accuracy of the predicted date range when compared to the actual delivery date.

AI Project Management is costly because

AI Projects:

6. Need Wider set of skills/team

7. Have higher technical risk

   1. Data needs and quality

   2. Model limitations

8. Far more challenging to plan and estimate, 

   1. are not linear but experimentative in development

   2. Probabilistic and not deterministic

   3. Further work and experiments may not necessarily improve the model or accuracy

9. Harder to show progress

   1. Variance of model outputs

10. Require more ongoing support

Team Structure

• Business Sponsor - Business strategy, goals and alignment because of high technical risk and project risk

• Proagram Manager who leads the below team members:

  • AI Product Manager

  • Data Engineer

  • Data Scientist

  • ML Engineer

============================

Designing AI Products and Preparing Business Cases: A Guide for Product Managers

In the realm of AI and machine learning, a staggering 80% of projects fail to achieve their objectives . For product managers, this underscores the importance of thoughtful AI product design and a solid business case. Here’s how to navigate this complex landscape effectively.

AI/ML Problem Framing

1. Assess the Need for AI/ML

Start by asking whether there is a genuine customer problem or operational improvement that AI can solve. Identify end-user pain points and classify the business value as low, medium, or high. Remember that AI can either automate or augment processes, depending on the risk involved. For example, in high-risk areas like medical diagnosis, AI should augment human decision-making, whereas in low-risk areas like tech content writing, it can automate tasks.

Avoid Using AI Just Because:

• Everyone else is doing it.

• Leadership demands it.

• It seems trendy or offers bragging rights.

2. Evaluate the AI/ML Feasibility

Not every problem needs AI. Consider whether a heuristic or non-ML solution might suffice. For problems where AI is appropriate, assess the data availability and classify the problem's complexity as easy, hard, or impossible.

3. ROI Consideration

Evaluate the return on investment (ROI) for AI projects. High-value problems that AI can solve easily are ideal. Harder-to-solve problems with high business value might still be worth pursuing, but consider the cost of adapting to environmental changes, ethical concerns, and accountability.

Implementing AI/ML Solutions

4. Defining AI/ML Solutions

Frame the problem in ML terms:

• Define the Ideal Outcome: What is the ultimate goal of the AI model? For example, predicting delivery dates.

• Model Output: What specific results should the model produce?

• Success Metrics: How will success be measured? This might include accuracy, precision, or user satisfaction.

5. Consideration of Costs and Maintenance

AI projects require significant resources and expertise. Assess the costs and long-term maintenance requirements, ensuring that your team has the capability to support the project.

6. Predictions vs. Actions

Ensure that AI predictions lead to actionable outcomes. There’s little value in a prediction if it cannot be translated into a useful action that benefits users.

AI Project Management Challenges

7. Managing Complexity

AI projects demand a broad set of skills and involve higher technical risks. They are non-linear, experimental, and require ongoing support, making planning and progress tracking more challenging.

8. Team Structure

Assemble a diverse team:

• Business Sponsor: Aligns AI efforts with business strategy and goals.

• Program Manager: Oversees the project and manages the team.

• AI Product Manager: Leads product development and strategy.

• Data Engineer, Data Scientist, ML Engineer: Provide technical expertise and implement AI solutions.

Conclusion

AI product design and business case preparation require careful consideration of problem framing, feasibility, and ROI. By addressing these factors, product managers can improve the likelihood of success in AI projects, ensuring they deliver real value to the business and its customers.