Book reviews from the ETAIC Lab highlight influential texts in reinforcement learning, control, and intelligent systems. We hope this review helps bring broader international attention to outstanding work in the field.
Author: Professor Shengbo Eben Li
Review by: H. Eric Tseng
Professor Shengbo Eben Li’s Reinforcement Learning for Sequential Decision and Optimal Control is a timely, rigorous, and highly recommended textbook for readers seeking to understand reinforcement learning (RL) in relation to optimal control and approximate dynamic programming. Rather than presenting modern reinforcement-learning algorithms as a collection of disconnected techniques, Dr. Li develops a coherent framework in which foundational concepts—Markov decision processes, Bellman optimality, value functions, policy improvement, and dynamic programming—remain central to both classical and contemporary methods. The book is particularly valuable for readers who wish to understand not only how an algorithm is implemented, but also the theoretical principles and control perspectives that motivate it.
A major strength of the book is its logical organization. It begins with the basic formulation of reinforcement-learning problems and proceeds through Monte Carlo learning, temporal-difference learning, stochastic dynamic programming, function approximation, direct policy-gradient methods, and approximate dynamic programming. This sequence enables readers to see the relationships among model-free and model-based approaches, indirect RL and direct RL methods, value-based methods, policy-optimization approaches, and hybrid approaches. The text’s systematic development is especially useful for students encountering modern RL methods for the first time, since it shows how modern RL extends classical value learning and policy optimization through increasingly flexible forms of function approximation.
The book’s most distinctive contribution is its control-oriented treatment of RL. Li gives sustained attention to approximate dynamic programming, including infinite- and finite-horizon formulations, continuous-time systems, convergence and stability issues, and the relationship between RL and model predictive control. This perspective will be particularly appealing to readers in control, robotics, autonomous systems, and intelligent transportation. The book also uses examples involving autonomous driving, cleaning robots, vehicle control, lane keeping, and constrained decision-making, helping readers connect mathematical formulations with realistic engineering problems.
The chapter on state constraints and safety deserves particular recognition. Safety is often treated as an auxiliary issue in RL texts, but Li places it at the center of the discussion by examining feasibility, constrained optimization, penalty and Lagrange-multiplier methods, feasible-descent approaches, safe-policy training, and safety-shield mechanisms. The introduction of the actor-critic-scenery architecture further reflects the author’s interest in the practical problem of simultaneously learning a high-performing policy and identifying the region in which that policy can remain feasible. This emphasis makes the book unusually relevant to safety-critical applications, where reward maximization alone is an inadequate objective.
The book is well positioned relative to other major references. Sutton and Barto’s Reinforcement Learning: An Introduction remains the canonical text for the fundamental principles of reinforcement learning. Li’s book complements that classic by extending the discussion more fully toward approximate dynamic programming, deep RL, safety constraints, and control-oriented applications. It also complements Frank L. Lewis and Derong Liu’s Reinforcement Learning and Approximate Dynamic Programming for Feedback Control, an influential control-oriented reference by Frank Lewis, a pioneer in reinforcement learning and adaptive dynamic programming for feedback control. Whereas Lewis and Liu provide a focused and rigorous treatment of adaptive dynamic programming and feedback control, Li offers a broader textbook perspective on RL methods and their connections to contemporary sequential decision-making problems. The two books are therefore complementary: Lewis and Liu offer depth in adaptive dynamic programming and feedback control, while Li provides a wider view of modern RL algorithms and sequential decision-making.
The breadth of the book is impressive, but it also creates one of its few limitations. The final chapter introduces robust RL, partially observable MDPs, meta-RL, multi-agent RL, inverse RL, offline RL, software frameworks, and simulation platforms. This provides readers with an excellent map of important research directions, but these advanced subjects necessarily receive more compact treatment than the foundational chapters. Readers seeking research-level depth in offline RL, multi-agent RL, inverse RL, or robust RL will need to consult the references and specialized literature cited throughout the text.
A second limitation concerns prerequisites. Although systematic in presentation, the book is not aimed at complete beginners or readers without a solid mathematical background. Familiarity with Python and some prior exposure to deep learning—supplemented, where helpful, by a general reference such as Goodfellow, Bengio, and Courville’s Deep Learning—would help readers connect the theory with implementation.
Overall, Reinforcement Learning for Sequential Decision and Optimal Control is an excellent and strongly recommended textbook. Its primary achievement is to provide a unified account of reinforcement learning as both a learning paradigm and a framework for optimal control. For advanced undergraduate students, graduate students, researchers, and engineers working in machine learning, robotics, autonomous systems, transportation, or control, Li’s book is a valuable resource that rewards careful study and deserves a place alongside the leading foundational and control-oriented references in the field.