Selected links for “sequential decision analytics”

Warren B. Powell 
Professor Emeritus, Princeton University    

I have been accumulating a number of links to pages on “sequential decision analytics” and my new book, Reinforcement Learning and Stochastic Optimization.  Also, I have been using “tinyurl” links because they are easier to remember (and I can track traffic), but not everyone can use these links.  This page will list the most interesting resources, including both the “tinyurl” link and the original link.  Please use the “tinyurl” link if you can, so I can see which links are attracting the most interest.  

  •  New! Getting started  
  • Some video introductions
  • Selected books (introductory and advanced)
  • Introduction to optimal learning
  • Courses and teaching materials 
  • Educational web pages about sequential decision analytics
  • LinkedIn posts on sequential decision analytics 

 

Getting started

I am often asked how to get started by people just entering the field of sequential decision problems.  I recommend the following:

  • Continue to the webpage https://tinyurl.com/BridgingDecisionProblems/ and download Volume I of the monograph “Framing the Problem”.  You can download the pdf for free, or puchase it on Kindle for $5.  This is 150 pages, with no math.

 

 

 

 

 

  • If you are ready for a little math, start by viewing part 3 of the “framing” video above. I then recommend downloading the introductory book “Sequential Decision Analytics and Modeling” which uses a teach-by-example style.  With the exception of chapters 1 and 7, each chapter uses a different example to illustrate the universal modeling framework, along with all four classes of policies.  
  • For professors looking to teach an undergraduate or masterscourse on this topic, these two books would make a nice combination.  Encourage students to come up with their own examples of sequential decision problems.  They can either just perform framing or, if they have some math skills, they can model their problem following the style of the SDAM book.
  • I prepared a webpage where you can access a talk I gave in April, 2025 to Cornell’s ORIE department on a fresh approach to teaching optimization.  The presentation challenges how I think we should approach the contributions of George Dantzig and Richard Bellman:
    Tinyurl link: https://tinyurl.com/HowtoTeachOptimization/
    Original link: https://castle.princeton.edu/HowtoTeachOptimization/

Some video introductions:

My best introduction to sequential decision analytics is the tutorial (which I update periodically):
Tinyurl link: https://tinyurl.com/sdafieldyoutube/
Original link: https://youtu.be/rZBbYHgLl74

Applications-oriented introductions – The videos below contain a typically shortened version of the universal framework (given in the video above) followed by illustrations in the context of specific applications:

In July, 2025, I gave a major talk at Toyota’s North American headquarters for a general audience (over 300 attended).  I divided the 90 minute talk into five components, covering 1) the 7 levels of AI, 2) framing problems, 3) the universal modeling framework, 4) the four classes of policies, and 5) the path to implementation.  A webpage with links to the talks is available at:
Tinyurl link: https://tinyurl.com/PowellToyotaVideos/ 
Original link: https://castle.princeton.edu/Powellframingtheproblem/

In April, 2025, I gave a talk at Cornell’s Department of Operations Research and Information Engineering on a “Fresh Approach to Teaching Optimization”.  This talk is based on that presentation:
Tinyurl link: https://tinyurl.com/HowtoTeachOptimization/
Original link: https://castle.princeton.edu/howtoteachoptimization/

A brief introduction to a major field I call “optimal learning.” I plan on expanding this in the future:
Tinyurl link: https://tinyurl.com/PowellVideoOptimalLearning
Original link: https://youtu.be/x07Zh_WwS_w

In October, 2024, I gave a presentation for Rutgers’ Road for Supply Chain Leadership.  Below are two segments that present my universal modeling framework, emphasizing supply chain applications, and my universal modeling framework aimed at nontechnical business students:

Part III of the talk (the modeling framework)
Tinyurl link: https://tinyurl.com/PowellVideoRSCLPartIII/
Original link: https://youtu.be/1nO09d4JLos

Part IV of the talk (the problem solving process for business students)
Tinyurl link: https://tinyurl.com/PowellVideoRSCLPartIV/
Original link: https://youtu.be/jXg7bJ0m8gM

A general overview of sequential decision analytics given to SRI in 2024:

  • The seven levels of artificial intelligence
  • Modeling sequential decision problems
  • The problem solving process
  • Designing policies
  • A real-world inventory problem
  • Multi-agent supply chain modeling
  • A new educational field: sequential decision analytics

New link coming soon!

Selected books (introductory and advanced):

The following series of books is designed for readers starting with people who need to perform basic problem framing and modeling, up to people who want to develop their own models and algorithms.  There is also a (very thin) book for instructors with an outline of how to teach an introduction to optimization for undergraduates and master’s students.

Introductory series: Bridging from Problems to Models

This is a series of monographs on sequential decision problems, starting with Volume I: Framing the Problem.  This book has no math and instead focuses on the initial step of framing a problem, which starts with identifying performance metrics, types of decisions, and forms of uncertainty.  

The book is available on Kindle for $5 at

https://tinyurl.com/PowellFramingAmazon/

Intermediate book: Sequential Decision Analytics and Modeling.

This book (written for my undergraduate course at Princeton) uses a teach-by-example style. It can be used by itself, or as a companion to the advanced book (RLSO) below. 
Tinyurl link: https://tinyurl.com/sdamodeling/
Original link: https://castle.princeton.edu/sdamodeling/

The companion book above has a Python module developed for most of the chapters (these are also used in the advanced RLSO book).  These are available at:
Tinyurl link: https://tinyurl.com/sdagithubnew/
Original link: https://github.com/djanka2/stochastic-optimization

Advanced book: Reinforcement Learning and Stochastic Optimization: A unified framework for sequential decisions

This book is aimed at people with a technical background who are interested in writing their own models and algorithms.  It presents models and algorithms with the precision needed to write software, which means careful attention to details such as modeling time.  This book was used in a graduate course at Princeton, which was taken by 50 graduate students from 10 different departments.

https://tinyurl.com/RLandSO/

Teaching guide for instructors: A Modern Approach to Teaching an Introduction to Optimization

One way to learn this material is to write your own chapters in the same style as the book Sequential Decision Analytics and Modeling.  If you have a student who wants to work in this area, encourage them to write a chapter on a problem of their choosing and add it to:
Tinyurl link: https://tinyurl.com/sdabookpublic/
Original link: https://www.overleaf.com/project/5c8add1d8faacf78eabd5761

I have written a monograph aimed at instructors called “A Modern Approach to Teaching an Introduction to Optimization.” The book describes 11 topics that approach optimization in a way that should be much more accessible and relevant to new students (undergraduates and masters). Starting with machine learning (a problem familiar to almost everyone), topics 2-5 focus on simpler decision problems that are sequential (each of these topics draws from the introductory book above).  Topic 6 gives a general overview of sequential decision problems.  Topics 7-11 cover linear, integer, and nonlinear programming, including both the familiar static versions of these problems as well as sequential versions.  There is much less emphasis on algorithms than is traditionally used in courses since any student needing this material would use a package.
Tinyurl link: https://tinyurl.com/TeachingOpt/
Original link: https://castle.princeton.edu/teachingoptimization/

Introduction to optimal learning:

An introduction to a subfield of sequential decision that I like to call “Optimal Learning” (other names are stochastic search, multi-armed bandit, or just “intelligent trial-and-error”).  Optimal learning is a sequential decision problem where the impact of a decision now on the future is made by what we learn, which may affect the decisions we make.

Courses and teaching materials:

I have prepared a webpage dedicated to different ways for teaching sequential decision analytics (January, 2026).  

Tinyurl link: https://tinyurl.com/TeachingSDA/
Original link: https://castle.princeton.edu/2026/02/02/teachingsda/

I have also prepared a new way of teaching introductory optimization for undergraduates and graduates.  The course evolves from basic machine learning, through simple sequential decision problems using parametric policies, to the familiar topics (in OR) of linear, integer and nonlinear programming, where each of these are presented initially as static problems, and then as sequential problems.

Tinyurl link: https://tinyurl.com/TeachingOpt
Original link: https://castle.princeton.edu/TeachingOpt

Short notes about sequential decision analytics

Easily the biggest mistake made in the teaching of deterministic optimization is the failure to realize that most (almost all?) deterministic optimization problems are solved repeatedly over time, which means they are policies for solving a sequential decision problem, and they are virtually never optimal policies.  Overlooked is the ability to modify the model at a point in time to improve performance over time (illustrated in the image below).  To download a series of powerpoint slides on this point, go to:
Tinyurl link: https://tinyurl.com/PowellOptimizationtoPolicies/
Original link: https://castle.princeton.edu/wp-content/uploads/2025/07/Powell-From-static-to-sequential-MILP-July-2025.pptx

One of the most confusing topics in reinforcement learning is state variables (try to find a definition of a state variable in any MDP or RL book).  The webpage “On state variables” discusses this topic:
Tinyurl link: https://tinyurl.com/Onstatevariables/ 
Original link: https://castle.princeton.edu/statevariables/

Notes on notation for sequential decision problems: 
Tinyurl link: https://tinyurl.com/SDAnotation/
Original link: https://castle.princeton.edu/2022/05/06/sdanotation/

A discussion of the issue of tunable parameters that arise in any form of policy search:
Tinyurl link: https://tinyurl.com/tunableparameters/
Original link: https://castle.princeton.edu/2022/04/30/tunableparameters/

A quick introduction to the four classes of policies that I compiled from a series of posts on LinkedIn:
Tinyurl link: https://tinyurl.com/FourClassesofPolicies/
Original link: https://docs.google.com/document/d/1YqCgvraULSpKXFme6HXdKl2uVlp2Nkl8rc4x530PUGo/edit#heading=h.egpzjlkwh66q

This is a paper aimed at a transportation audience that provides a review of the universal framework and then focuses on two forms of direct lookahead approximations (DLA) that tend to be useful in transportation and logistics: stochastic lookaheads, and parameterized deterministic lookaheads.
Tinyurl link: https://tinyurl.com/PowellLookaheadPolicies/
Original link: https://ieeexplore.ieee.org/abstract/document/9702124

This is a brief discussion on the topic of “languages” for sequential decision problems:
Tinyurl link: https://tinyurl.com/SDAlanguages/
Original link: https://castle.princeton.edu/sdalanguages/

Almost entirely overlooked in the research literature, but widely used in practice (in an ad-hoc way) is the idea of creating policies using simplified (typically deterministic) optimization models.  This idea is outlined on the webpage:
Tinyurl link: https://tinyurl.com/CFAPolicy/
Original link: https://castle.princeton.edu/CFA/

From reinforcement learning to sequential decision analytics:

There is a lot of interest in “reinforcement learning” but a surprising lack of consensus on precisely what this means.  For my discussion of this topic go to
Tinyurl link: https://tinyurl.com/what-is-rl/
Original link: https://castle.princeton.edu/what-is-rl/

I prepared a webpage with a direct comparison of classical “reinforcement learning” (as captured by Sutton and Barto) and my RLSO book:
Tinyurl link: https://tinyurl.com/RLvsSD/
Original link: https://castle.princeton.edu/rlvssda/

LinkedIn posts on sequential decision analytics

I actively post on LinkedIn on the broad theme of sequential decision analytics. I have compiled the most educational posts, and instead of listing them in the order in which they are posted, I have organized them into various headings.  Take a look at

Tinyurl link: https://tinyurl.com/PowellLinkedInPosts/
Original link: https://docs.google.com/document/d/12CH6G_4MgOSwopgIrheW6DM9F0pn8UI2qod_LeKPBUA/edit#heading=h.7eckd8b872nb