Sequential Decision Analytics and Modeling
Professor Warren Powell
The next frontier of analytics involves making decisions over time in the presence of different forms of uncertainty. These challenges arise in internet advertising and auctions, transportation and logistics, health, energy, finance, and a wide range of problems in science and engineering. They occur in the lab (what experiment to run next) and in the field (what ad should I post to maximize ad-clicks), from the simple (when to stop testing a new drug) to the complex (optimizing thousands of vehicles in a ridesharing fleet).
The settings are so broad that these problems have been addressed by many different communities. I like to refer to these communities as the “jungle of stochastic optimization.” Each community offers powerful solution approaches, but only for narrow sets of applications. ORF 411 presents a unified framework that draws on the strengths of all of these fields, providing a set of tools that will solve any sequential decision problem.
ORF 411 focuses primarily on modeling. We use a canonical framework that allows us to model any sequential decision problem, in the presence of different sources of uncertainty. We solve these problems by working with four fundamental classes of policies which provide a path to computation for any problem setting. These are the tools that Google uses to suggest ads, that DeepMind uses to build AlphaGo, that Uber/Lyft use to price rides, and that Amazon and Apple use to manage their complex supply chains. Any company looking to bring advanced analytics to help make decisions will have to draw on these tools.
Statistical learning is an important component of sequential decision analytics. Learning arises in five settings: objective function approximation, policy function approximation, value function approximation, constraint parameterization, and learning transition functions. A key dimension is active/optimal learning, where we get to choose the inputs. Students with a background in machine learning will see new challenges, but no prior training is required beyond basic statistics.
Identifying good decision problems is part of the challenge in the use of analytics for sequential decisions. Students are invited to suggest their own decision problems:
Some of these decisions will be used as exercises in class.
The startup challenge
These analytic tools are at the heart of many products and services that can be the basis of a marketable app or startup. Given the diversity of applications, students (working in two-person teams), will propose a detailed design of a decision-analytic tool that could be the basis of an app or startup. The design will consist of a decision setting (including the decision maker), data requirements (the state variable), performance metrics (the objective function), uncertainties, and a policy that would optimize the performance metric.
The style of the course will be teaching through examples. The entire course will be set around a series of modern analytics challenges in energy, logistics, health, transportation and electronic commerce. The current list includes:
- Adaptive market planning (designing features, setting prices)
- Design and control of renewable energy systems
- Energy trading and storage (with extensions to finance, logistics,…)
- Forward contracts (lagged decisions)
- Supply chain management and the beer game (information sharing and adversarial planning)
- Learning in the laboratory (offline learning)
- Ad-click optimization (online learning)
- Medical decision making (contextual learning)
- Management of clinical trials (learning with a physical state)
- Surge pricing for ride sharing (high dimensional learning)
Click here for Excel spreadsheet listing topics and modeling features.
Audience, prerequisites and requirements
Audience: Any student (juniors, seniors and graduate) with an interest in making better decisions while managing uncertainty. Students are welcome from any engineering department, physical or social science, and economics. Students with some background from prior courses such as ORF 311/335/360/418, game theory, machine learning, or a background in a real problem that involves decisions under uncertainty, will benefit.
Prerequisites: The only prerequisite is a course in probability and statistics such as ORF 245 (ORF 309 is fine but not required). You need to understand random variables, distributions and Bayes theorem. Some python coding is necessary, but software will be provided as a starting point. One project will use linear programming, but the software will be provided and a course in optimization is not a prerequisite.
Textbook: Click here for web version of book.
Software: Click here for Python modules have been written for most of the chapters in the book. Each module is accompanied by a spreadsheet where you can run exercises by tuning parameters, or you can download the code and make changes on your own.
- 10 problem sets, combining modeling, design of policies, and testing policies in python code. Python code for basic models will be provided – students will need to implement model extensions, and test policies – 40%
- Midterm – 25%
- Take-home final – 35 %
First half of course (pdf). Second half of (pdf)
Individual lectures (powerpoint format):
- Lecture 1 – Introduction, applications, modeling (powerpoint)
- Lecture 2 – Adaptive market planning (powerpoint)
- Lecture 3 – Machine learning (powerpoint)
- Lecture 4 – Learning diabetes medications (powerpoint)
- Lecture 5 – Learning policies (powerpoint)
- Lecture 6 – Stochastic shortest paths I (powerpoint)
- Lecture 7 – Stochastic shortest paths II (powerpoint)
- Lecture 8 – Designing policies (powerpoint)
- Lecture 9 – Energy storage I (powerpoint)
- Lecture 10 – Energy storage II (powerpoint)
- Lecture 11 – Energy storage III (powerpoint)
- Lecture 12 – Uncertainty modeling (powerpoint)
- Lecture 13 – Uncertainty modeling (powerpoint)
- Lecture 14 – The beer game I – Introduction (powerpoint)
- Lecture 15 – The beer game II (powerpoint, spreadsheet)
- Lecture 16 – The beer game III – Discussion (powerpoint)
- Lecture 17 – The blood management problem (powerpoint)
- Lecture 18 – Hotel revenue management (powerpoint)
- Lecture 19 – Class decision problem – Booking flights (powerpoint)
- Lecture 20 – Clinicial drug trials (powerpoint)
- Lecture 21 – Student decision problems – Rowing strategies, twitter trading (powerpoint)
- Lecture 22 – Summary lecture (powerpoint)