Warren B. Powell
Executive-in-residence, Rutgers University
Chief Innovation Officer, Optimal Dynamics
Professor emeritus, Princeton University
Bridging Decision Problems is being planned as a series of monographs on the process of modeling decision problems and implementing the results. Unlike traditional presentations that start with a mathematical model, solve it, and then declare success, we are going to progress through the entire process:
- Start with a raw problem that features decisions using a process we call “framing the problem.”
- Model the process using the “universal modeling framework” that can be applied to any decision problem
- Translate the model to computer code, both as a simulator (which has several applications) and a method for making decisions (called the policy)
- Design the process for collecting the information needed to run the model.
- Implement the decisions in the field and track performance.
Volume I addresses the first step, framing the problem, which involves answering a series of questions in English that form the basis of a mathematical model that can be implemented on the computer. This webpage is a very brief overview of the framing process. At the bottom is a link to a new book that describes this process in much more detail (but no math).
Volume I is now available on Kindle here, or download the pdf from here.
In our presentation, we always assume that the “problem” involves improving one or more performance metrics. We operate under the principle:
The list of lists for framing
The graphic below contains a series of lists that can serve as a guide when you are identifying the key elements of a sequential decision problem. It includes the following lists:
- Categories of metrics
- Types of decisions
- Classes of uncertainty
- Behaviors of uncertainty
- The four classes of policies
- The functions that depend on the state variable
- State variables based on knowledge
The six steps for solving decision problems
The process of solving decision problems (and I mean any decision problem) can be envisioned in six stages comprised of:
- Summarizing the business problem in the language of the problem domain.
- Framing the problem, which means stating it in English, but using specific vocabulary that sets the stage for mathematical modeling, if required.
- Mathematical modeling – This is done with the universal modeling framework (UMF), which can capture any sequential decision problem.
- Identify the information needed to run the model, and creating the processes to acquire it. This process is increasingly being assisted by LLMs.
- Computer implementation – The UMF provides an explicit roadmap to computer implementation, which can be in a spreadsheet or sophisticated model.
- Implementing the decisions in the field, which typically raises fresh issues. This may or may not involve implementing the software itself in the field. This process is also starting to be assisted by LLMs.
We note that our guiding light (or “north star”) is using a computer to make decisions to improve performance, even if we do not develop a mathematical model or implement it on the computer. Our experience is that this process brings clarity to complex problems, which may provide enough guidance to make a decision without formal analysis. However, if additional help is needed, we always retain the option of progressing to a computer model, whether it is a spreadsheet or more sophisticated software.
We recognize that computers can be used in a variety of ways:
- Creating information so a human can make a decision – The universal modeling framework helps to identify what information is needed.
- Computer-assisted – The computer recommends, human reviews.
- Completely autonomous – The computer makes the decision on its own.
- Simulating decisions from the current state – We use a computer to project the downstream effect of a decision now, simulating decisions that are needed in the future.
- Strategic simulations – Here, we simulate a system over time, without needing to capture its current state. We use strategic simulators to help make design decisions.
The three stages of automation
The process of automating a decision process proceeds in three stages, as outlined below.
The universal modeling framework
The information provided by the framing process is guided by the universal modeling framework. This general framework can be used to represent any sequential decision problem, including both single and multi-agent applications. I like to present the universal modeling framework (for a single agent) using the slide:
Any problem modeled using the universal modeling framework (UMF) can be directly implemented on a computer. This does not mean that all decision problems benefit from being implemented on the computer. However, all decision problems benefit from being approached with this structure. At a minimum, it will help you think about any decision problem. If it is felt that a computer-assisted solution would add value, then there is an immediate path to computer implementation.
Making decisions
There are decades of research and countless books and papers that address the problem of how to make decisions in the presence of uncertainty. Since 2010 I have been refining the idea that any method for making decisions falls within four classes of policies divided into two broad strategies:
Strategy I: Policy search – These are functions that have to be tuned to work well over time.
- Policy function approximations (PFAs) – These are analytical functions, from order-up-to inventory policies, linear or nonlinear functions, even deep neural networks.
- Cost function approximations (CFAs) – These are parameterized versions of typically deterministic approximations that are parameterized to work well over time, under uncertainty.
Strategy 2: Lookahead policies – These policies approximate the impact of decisions now on the future.
- Policies based on value function approximations (VFAs) – This is where solutions based on Bellman’s equation are located.
- Direct lookahead approximations (DLAs) – These policies optimize into the future to make better decisions now. It is helpful to divide these into two classes: a) Deterministic lookahead (think of Google maps), and b) stochastic lookahead, which can come in a range of forms (see chapter 19 of RLSO).
A slightly longer description is provided here. I recommend the video tutorial here. Note that three of the four classes of policies (CFAs, VFAs and DLAs) involve solving imbedded optimization problems within the policies. These are usually (but not always) solved using deterministic methods.
References
This new process of framing the problem is described in detail in a new monograph. The book should be available for purchase in December 2025. The pdf should be available for free download by April, 2026. For now, the table of contents and preface (which contains an overview of the book) can be downloaded from:
Please feel free to leave comments here.
I am currently planning the next three volumes:
- Volume II: The Universal Modeling Framework – This volume will show how to translate the framing information from Volume I into a very general modeling framework called the “Universal Modeling Framework.” A substitute that you can access now is the book “Sequential Decision Analytics and Modeling,” which can be downloaded here.
- Volume III: Designing Policies – This volume will cover the four classes of policies in much more depth, and show how to evaluate them using the universal framework, and finally how to tune them. A substitute that you can access now is Chapter 11 of my advanced book which can be downloaded from here.
Potential future volumes (depending on what catches my interest):
- Uncertainty Modeling – Controllable dynamic systems are governed by two flows of information: decisions and exogenous information. This volume addresses in much more depth how we think about and model exogenous information processes. A substitute for now is Chapter 10 of my advanced book, which can be downloaded from here.
- Supply chain management – There is so much interest in this problem class. It will provide an opportunity for me to dip my toe into these waters.
- Navigating complex environments – This is where I want to really develop my ideas for multi-agent decision-making.
Other relevant books:
- Sequential Decision Analytics and Modeling (2022) – This book uses a teach-by-example style. Each chapter (other than chapters 1 and 7) describes a different problem, which is then modeled using the UMF. The problems are chosen to illustrate each of the four classes of policies.
- Reinforcement Learning and Stochastic Optimization: A unified framework for sequential decision problems (Wiley, 2022), 1100 pages. This is an advanced, graduate-level book designed for people who want to develop models and algorithms for sequential decision problems.
Webpages of possible interest:
“The Road to Digital Automation at Toyota: Learning How to Be an Informed Consumer,” Toyota’s North American Headquarters, July 10, 2025. This is a presentation in five parts given to a broad audience at Toyota that contrasts LLMs with using computers to make decisions.
Webpage for sequential decision analytics – This webpage is a page of resources for the following topics:
- Introduction to the field of sequential decision problems
- Some video introductions
- Downloadable books (introductory and advanced)
- Introduction to Optimal Learning
- Courses and teaching materials
- Educational web pages about sequential decision analytics
- LinkedIn posts on sequential decision analytics