Sequential Decision Analytics and Modeling

Warren B. Powell
Professor Emeritus, Princeton University
Chief Innovation Officer, Optimal Dynamics


Sequential decision problems arise in virtually every form of human processes: transportation and logistics, supply chain management, energy, health (from public health to medical decision making), finance, e-commerce, laboratory sciences, … 

Sequential Decision Analytics and Modeling uses a teach-by-example style to illustrate a universal framework for modeling sequential decision problems.  The universal framework applies to any sequential decision problem, from active learning problems up through complex resource allocation problems.  Chapters are accompanied by python modules that have implemented the models, but the book should be of value even to people not interested in writing code.

There are two ways to get the book:

If you would like to make any comments (or if you find typos) please make them here.

I recently designed a new way of teaching introduction to optimization for undergraduates and masters students.  This strategy is summarized in a monograph which provides a course outline in the form of 11 topics, many of which reference Sequential Decision Analytics and Modeling. The new approach is given at:

 

Summary:

Chapter 1 illustrates the core modeling framework using two inventory examples, and introduces the four classes of policies that encompass any method used for making decisions.

Chapters 2-6 use a series of examples designed to illustrate each of the four classes of policies, along with different styles for modeling uncertainty.  We pause in chapter 7 to revisit the universal modeling framework, drawing on the applications from chapters 2-6 to illustrate state variables (including belief states), scalar and vector-valued decisions, uncertainty modeling, different types of objective functions, and the four classes of policies.

The book is designed to be used as a standalone introduction to the vast field of sequential decision analytics.  For more information, see the sequential decision analytics website. A youtube video introduction to sequential decision analytics is available here. A webpage of links to books, videos, discussions, and suggested courses is available here.

For advanced readers interested in developing models and algorithms, I recommend my new book Reinforcement Learning and Stochastic Optimization: A unified framework for sequential decisions. 

Audience

While the entire book is focused on sequential decisions under uncertainty, the book assumes no background in any form of stochastic optimization or dynamic programming. I taught out of an earlier version of this book in an undergraduate course at Princeton. The complete lecture notes are available at https://tinyurl.com/RLSOcourses  (scroll down to the “Undergraduate course”).  I also suggest an introductory weekly seminar on the same webpage (near the top).

Software

There is a series of python modules that accompany most of the chapters.  These can be accessed at

https://tinyurl.com/sdagithub/

These modules were developed by a group of students, reviewed by a staff member, and then used once to teach the course at Princeton (after which I retired).  I was very pleased to learn that Dennis Djanka, a professor at Karlsruhe University in Germany, has taken over the task of modernizing (“refactoring” in softwarespeak) the library, which is available at 

https://tinyurl.com/sdagithubnew/

In his words, the new library includes:

  • Introduction of abstract base classes SDPModel and SDPPolicy that make it easy to setup new models and policies with minimum amount of code.
  • Complete rewrite of the code for the AssetSelling, MedicalDecisionDiabetes  and StochasticShortestPath_static modules and created a Jupyter Notebook for each of the problems that walks the user from creating a model and policy to tuning policies and interpreting the results. 

You may contact Dennis at dennis.janka@h-ka.de if you have questions or suggestions about the software library. I am grateful to anyone willing to make contributions to this resource.

Supplementary material

There is additional material, such as spreadsheets and datasets, at https://castle.princeton.edu/sdamodelingsupplements/