Selected links for “sequential decision analytics”

Warren B. Powell
Professor Emeritus, Princeton University

I have been accumulating a number of links to pages on “sequential decision analytics” and my new book Reinforcement Learning and Stochastic Optimization.  Also, I have been using “tinyurl” links because they are easier to remember (and I can track traffic), but not everyone can use these links.  This page will list the most interesting resources, including both the “tinyurl” link and the original link.  Please use the “tinyurl” link if you can so I can see which links are attracting the most interest.

  •  Introduction to the fields of sequential decision problems
  • Some video introductions
  • Books
  • Courses and teaching materials
  • Short notes about sequential decision analytics
  • LinkedIn posts on sequential decision analytics

Introduction to the fields of sequential decision problems:

An introduction to the field I am calling “sequential decision analytics”:
Tinyurl link: https://tinyurl.com/sdafield/
Original link: https://castle.princeton.edu/sda/

My original “jungle of stochastic optimization” webpage:
Original link: https://castle.princeton.edu/jungle/

A discussion that bridges “reinforcement learning” to “sequential decision analytics”:
Tinyurl link: https://tinyurl.com/RLtoSDA/
Original link: https://castle.princeton.edu/RLtoSDA/

There is a lot of interest in “reinforcement learning” but a surprising lack of consensus on precisely what this means.  For my discussion of this topic go to
Tinyurl link: https://tinyurl.com/what-is-rl/
Original link: https://castle.princeton.edu/what-is-rl/

I prepared a webpage with a direct comparison of classical “reinforcement learning” (as captured by Sutton and Barto) and my RLSO book:
Tinyurl link: https://tinyurl.com/RLvsSD/
Original link: https://castle.princeton.edu/rlvssda/

Introductory articles that appeared in ORMS Today:

Some video introductions:

My best video introduction to sequential decision analytics was prepared for a Distinguished Speaker series in supply chain management:
Tinyurl link: https://tinyurl.com/sdafieldyoutube/
Original link: https://www.youtube.com/watch?v=lTUsr_-CpQM

Applications-oriented introductions – The videos below contain a typically shortened version of the universal framework (given in the video above) followed by illustrations in the context of specific applications:

A four part tutorial that I have given several times, most recently to a class at Oxford in December, 2022.  This provides a more in-depth introduction to the modeling framework and the four classes of policies:
Part I:
Tinyurl link: https://tinyurl.com/SDAPartI/
Original link: https://www.youtube.com/watch?v=XqBdBEv5_bs

Part II:
Tinyurl link: https://tinyurl.com/SDAPartII/
Original link: https://www.youtube.com/watch?v=gebjRw46AQw

Part III:
Tinyurl link: https://tinyurl.com/SDAPartIII/
Original link: https://www.youtube.com/watch?v=ZqtG1LZ5f-E

Part IV:
Tinyurl link: https://tinyurl.com/SDAPartIV/
Original link: https://www.youtube.com/watch?v=OOYKrsxEdmo 

 

Books:

The link to the page for Reinforcement Learning and Stochastic Optimization:
Tinyurl link: https://tinyurl.com/RLandSO/
Original link: https://castle.princeton.edu/RLSO/

The link to the page for Sequential Decision Analytics and Modeling. This is a companion book for RLSO written for an undergraduate level course (this is a free download, and has a python module for most of the chapters):
Tinyurl link: https://tinyurl.com/sdamodeling/
Original link: https://castle.princeton.edu/sdamodeling/

The companion book above has a Python module developed for most of the chapters (these are also used in the new RLSO book).  These are available at:
Tinyurl link: https://tinyurl.com/sdagithubnew/
Original link: https://github.com/djanka2/stochastic-optimization

One way to learn this material is to write your own chapters in the same style as the book Sequential Decision Analytics and Modeling.  If you have a student who wants to work in this area, encourage them to write a chapter on a problem of their choosing and add it to:
Tinyurl link: https://tinyurl.com/sdabookpublic/
Original link: https://www.overleaf.com/project/5c8add1d8faacf78eabd5761

I have written a monograph aimed at instructors called “A Modern Approach to Teaching an Introduction to Optimization.” The book describes 11 topics that approach optimization in a way that should be much more accessible and relevant to new students (undergraduates and masters). Starting with machine learning (a problem familiar to almost everyone), topics 2-5 focus on simpler decision problems that are sequential (each of these topics draws from the introductory book above).  Topic 6 gives a general overview of sequential decision problems.  Topics 7-11 cover linear, integer, and nonlinear programming, including both the familiar static versions of these problems as well as sequential versions.  There is much less emphasis on algorithms than is traditionally used in courses since any student needing this material would use a package.
Tinyurl link: https://tinyurl.com/TeachingOpt/
Original link: https://castle.princeton.edu/teachingoptimization/

Courses and teaching materials:

Suggested courses on sequential decision analytics:

  • A graduate-level course based on Reinforcement Learning and Stochastic Optimization – This course provides a thorough introduction to the materials needed to develop models and algorithms for practical applications.
  • An undergraduate/masters level course based on Sequential Decision Analytics and Modeling – This course focuses in developing a way of thinking about sequential decision problems using a rich series of examples.
  • Introducing the topic using a few lectures in an existing course.
  • An outline for a weekly seminar that could be run by a group of graduate students.
  • Teaching important modeling concepts in a non-analytical course for MBAs.
  • A new way of teaching introduction to optimization.  Instead of the typical course centered on linear programming, this course integrates simpler sequential decision problems before transitioning to static and sequential linear, integer and nonlinear programs.  

These course descriptions can be found at:

Tinyurl link: https://tinyurl.com/RLSOcourses/
Original link: https://castle.princeton.edu/RLSOcourses/

If you are thinking of teaching this material, please add your name to the signup list at:
Tinyurl link: https://tinyurl.com/RLSOsignup/
Original link: https://docs.google.com/spreadsheets/d/1mCCLM3E_gNw1I_bRHu3c-HLw-gEqk_xwISqHf1QXY6Q/edit#gid=0

I taught a course called Optimal Learning to undergraduates at Princeton.  Optimal learning problems are pure learning problems, but arise in a wide range of settings.  There is an online version of the 2nd edition of my book Optimal Learning along with all the lectures at:
Tinyurl link: https://tinyurl.com/optimallearningcourse/
Original link: https://castle.princeton.edu/orf-418/

A great way to teach about optimizing over policies is to tune the parameters of a simple PFA policy.  Below is a link to a spreadsheet where students can tune the parameters of a “buy low, sell high” policy for energy storage: 
Tinyurl link: https://tinyurl.com/energystorageoptimization
Original link: https://docs.google.com/document/d/1bHoHEXGZ3SEtZBeCqdgmU9OGSIkG6NfcOR2nYSd3IfM/edit#heading=h.mefhgf3fsks

I have prepared a new way of teaching introductory optimization for undergraduates and graduates.  The course evolves from basic machine learning, through simple sequential decision problems using parametric policies, to the familiar topics (in OR) of linear, integer and nonlinear programming, where each of these are presented initially as static problems, and then as sequential problems.
Tinyurl link: https://tinyurl.com/TeachingOpt
Original link: https://castle.princeton.edu/TeachingOpt

Short notes about sequential decision analytics

One of the most confusing topics in reinforcement learning is state variables (try to find a definition of a state variable in any MDP or RL book).  The webpage “On state variables” discusses this topic:
Tinyurl link: https://tinyurl.com/Onstatevariables/ 
Original link: https://castle.princeton.edu/statevariables/

Notes on notation for sequential decision problems: 
Tinyurl link: https://tinyurl.com/SDAnotation/
Original link: https://castle.princeton.edu/2022/05/06/sdanotation/

A discussion of the issue of tunable parameters that arise in any form of policy search:
Tinyurl link: https://tinyurl.com/tunableparameters/
Original link: https://castle.princeton.edu/2022/04/30/tunableparameters/

A quick introduction to the four classes of policies that I compiled from a series of posts on LinkedIn:
Tinyurl link: https://tinyurl.com/FourClassesofPolicies/
Original link: https://docs.google.com/document/d/1YqCgvraULSpKXFme6HXdKl2uVlp2Nkl8rc4x530PUGo/edit#heading=h.egpzjlkwh66q

This is a paper aimed at a transportation audience that provides a review of the universal framework and then focuses on two forms of direct lookahead approximations (DLA) that tend to be useful in transportation and logistics: stochastic lookaheads, and parameterized deterministic lookaheads.
Tinyurl link: https://tinyurl.com/PowellLookaheadPolicies/
Original link: https://ieeexplore.ieee.org/abstract/document/9702124

This is a brief discussion on the topic of “languages” for sequential decision problems:
Tinyurl link: https://tinyurl.com/SDAlanguages/
Original link: https://castle.princeton.edu/sdalanguages/

Almost entirely overlooked in the research literature, but widely used in practice (in an ad-hoc way) is the idea of creating policies using simplified (typically deterministic) optimization models.  This idea is outlined on the webpage:
Tinyurl link: https://tinyurl.com/CFAPolicy/
Original link: https://castle.princeton.edu/CFA/

LinkedIn posts on sequential decision analytics

I actively post on LinkedIn on the broad theme of sequential decision analytics. I have compiled the most educational posts, and instead of listing them in the order in which they are posted, I have organized them into various headings.  Take a look at

Tinyurl link: https://tinyurl.com/PowellLinkedInPosts/
Original link: https://docs.google.com/document/d/12CH6G_4MgOSwopgIrheW6DM9F0pn8UI2qod_LeKPBUA/edit#heading=h.7eckd8b872nb