Warren B. Powell
I have been accumulating a number of links to pages on “sequential decision
analytics” and my new book Reinforcement Learning and Stochastic Optimization. Also, I have been using “tinyurl” links because they are easier to remember (and I can track traffic), but not everyone can use these links. This page will list the most interesting resources, including both the “tinyurl” link and the original link. Please use the “tinyurl” link if you can so I can see which links are attracting the most interest.
Introduction to the fields of sequential decision problems:
An introduction to the field I am calling “sequential decision analytics”:
Tinyurl link: https://tinyurl.com/sdafield/
Original link: https://castle.princeton.edu/sda/
My original “jungle of stochastic optimization” webpage:
Original link: https://castle.princeton.edu/jungle/
A discussion that bridges “reinforcement learning” to “sequential decision analytics”:
Tinyurl link: https://tinyurl.com/RLtoSDA/
Original link: https://castle.princeton.edu/RLtoSDA/
There is a lot of interest in “reinforcement learning” but a surprising lack of consensus on precisely what this means. For my discussion of this topic go to
Tinyurl link: https://tinyurl.com/what-is-rl/
Original link: https://castle.princeton.edu/what-is-rl/
New! I prepared a webpage with a direct comparison of classical “reinforcement learning” (as captured by Sutton and Barto) and my RLSO book:
Tinyurl link: https://tinyurl.com/RLvsSD/
Original link: https://castle.princeton.edu/rlvssda/
Some video introductions:
My best video introduction to sequential decision analytics was prepared for a Distinguished Speaker series in supply chain management:
Tinyurl link: https://tinyurl.com/sdafieldyoutube/
Original link: https://www.youtube.com/watch?v=lTUsr_-CpQM
A four part tutorial that I have given several times, most recently to a class at Oxford in December, 2022. This provides a more in-depth introduction to the modeling framework and the four classes of policies:
Part I:
Tinyurl link: https://tinyurl.com/SDAPartI/
Original link: https://www.youtube.com/watch?v=XqBdBEv5_bs
Part II:
Tinyurl link: https://tinyurl.com/SDAPartII/
Original link: https://www.youtube.com/watch?v=gebjRw46AQw
Part III:
Tinyurl link: https://tinyurl.com/SDAPartIII/
Original link: https://www.youtube.com/watch?v=ZqtG1LZ5f-E
Part IV:
Tinyurl link: https://tinyurl.com/SDAPartIV/
Original link: https://www.youtube.com/watch?v=OOYKrsxEdmo
Books:
The link to the page for Reinforcement Learning and Stochastic Optimization:
Tinyurl link: https://tinyurl.com/RLandSO/
Original link: https://castle.princeton.edu/RLSO/
The link to the page for Sequential Decision Analytics and Modeling. This is a companion book for RLSO written for an undergraduate level course (this is a free download, and has a python module for most of the chapters):
Tinyurl link: https://tinyurl.com/sdamodeling/
Original link: https://castle.princeton.edu/sdamodeling/
One way to learn this material is to write your own chapters in the same style as the book Sequential Decision Analytics and Modeling. If you have a student who wants to work in this area, encourage them to write a chapter on a problem of their choosing and add it to:
Tinyurl link: https://tinyurl.com/sdabookpublic/
Original link: https://www.overleaf.com/project/5c8add1d8faacf78eabd5761
The companion book above has a Python module developed for most of the chapters (these are also used in the new RLSO book). These are available at:
Tinyurl link: https://tinyurl.com/sdagithub/
Original link: https://github.com/wbpowell328/stochastic-optimization
Courses and teaching materials:
Suggested courses on sequential decision analytics:
- A graduate-level course based on Reinforcement Learning and Stochastic Optimization – This course provides a thorough introduction to the materials needed to develop models and algorithms for practical applications.
- An undergraduate/masters level course based on Sequential Decision Analytics and Modeling – This course focuses in developing a way of thinking about sequential decision problems using a rich series of examples.
- Introducing the topic using a few lectures in an existing course.
- An outline for a weekly seminar that could be run by a group of graduate students.
- Teaching important modeling concepts in a non-analytical course for MBAs.
- A new way of teaching introduction to optimization. Instead of the typical course centered on linear programming, this course integrates simpler sequential decision problems before transitioning to static and sequential linear, integer and nonlinear programs.
These course descriptions can be found at:
Tinyurl link: https://tinyurl.com/RLSOcourses/
Original link: https://castle.princeton.edu/RLSOcourses/
If you are thinking of teaching this material, please add your name to the signup list at:
Tinyurl link: https://tinyurl.com/RLSOsignup/
Original link: https://docs.google.com/spreadsheets/d/1mCCLM3E_gNw1I_bRHu3c-HLw-gEqk_xwISqHf1QXY6Q/edit#gid=0
I taught a course called Optimal Learning to undergraduates at Princeton. Optimal learning problems are pure learning problems, but arise in a wide range of settings. There is an online version of the 2nd edition of my book Optimal Learning along with all the lectures at:
Tinyurl link: https://tinyurl.com/optimallearningcourse/
Original link: https://castle.princeton.edu/orf-418/
A great way to teach about optimizing over policies is to tune the parameters of a simple PFA policy. Below is a link to a spreadsheet where students can tune the parameters of a “buy low, sell high” policy for energy storage:
Tinyurl link: https://tinyurl.com/energystorageoptimization
Original link: https://docs.google.com/document/d/1bHoHEXGZ3SEtZBeCqdgmU9OGSIkG6NfcOR2nYSd3IfM/edit#heading=h.mefhgf3fsks
I have prepared a new way of teaching introductory optimization for undergraduates and graduates. The course evolves from basic machine learning, through simple sequential decision problems using parametric policies, to the familiar topics (in OR) of linear, integer and nonlinear programming, where each of these are presented initially as static problems, and then as sequential problems.
Tinyurl link: https://tinyurl.com/TeachingOpt
Original link: https://castle.princeton.edu/TeachingOpt
Short notes about sequential decision analytics
One of the most confused topics in reinforcement learning is state variables (try to find a definition of a state variable in any MDP or RL book). The webpage “On state variables” discusses this topic:
Tinyurl link: https://tinyurl.com/Onstatevariables/
Original link: https://castle.princeton.edu/statevariables/
Notes on notation for sequential decision problems:
Tinyurl link: https://tinyurl.com/SDAnotation/
Original link: https://castle.princeton.edu/2022/05/06/sdanotation/
A discussion of the issue of tunable parameters that arise in any form of policy search:
Tinyurl link: https://tinyurl.com/tunableparameters/
Original link: https://castle.princeton.edu/2022/04/30/tunableparameters/
A quick introduction to the four classes of policies that I compiled from a series of posts on LinkedIn:
Tinyurl link: https://tinyurl.com/FourClassesofPolicies/
Original link: https://docs.google.com/document/d/1YqCgvraULSpKXFme6HXdKl2uVlp2Nkl8rc4x530PUGo/edit#heading=h.egpzjlkwh66q
This is a paper aimed at a transportation audience that provides a review of the universal framework, and then focuses on two forms of direct lookahead approximations (DLA) that tend to be useful in transportation and logistics: stochastic lookaheads, and parameterized deterministic lookaheads.
Tinyurl link: https://tinyurl.com/PowellLookaheadPolicies/
Original link: https://ieeexplore.ieee.org/abstract/document/9702124
This is a brief discussion on the topic of “languages” for sequential decision problems:
Tinyurl link: https://tinyurl.com/SDAlanguages/
Original link: https://castle.princeton.edu/sdalanguages/
Almost entirely overlooked in the research literature, but widely used in practice (in an ad-hoc way) is the idea of creating policies using simplified (typically deterministic) optimization models. This idea is outlined on the webpage:
Tinyurl link: https://tinyurl.com/CFAPolicy/
Original link: https://castle.princeton.edu/CFA/