Reinforcement Learning: Learning Decision Policies from Delayed Feedback
Use MDPs, returns, value functions, Bellman equations, exploration, and policy gradients to understand how an RL agent optimizes long-term outcomes through interaction.
1 Interaction Rather Than a Fixed Answer Intuition
When supervised learning has no standard action, how can a machine learn from the consequences of its actions?
Reinforcement learning is a framework that allows an agent to learn a decision policy through continuous interaction, for problems where actions change future situations and there is no step-by-step standard answer: an action changes the next observed situation, and evaluating it must also account for delayed consequences. The party that executes decisions is called Agent, and the external system it is in and can influence is called environment.
At each step the current state is input, the policy outputs an action or action probability, and the environment returns a reward and the next state: the environment first provides state s(the current information relevant to the decision), and the Agent according to policy π chooses action a, then receives reward r and next state s′. This set of objects and their transition rules forms a Markov decision process (MDP). The final product of learning is not a fixed answer to a particular problem, but a policy for choosing actions in different states; the training process first collects such transitions, then estimates long-term outcomes, and finally updates the policy and repeats the interaction.
| Object | Refund assistant example | Role |
|---|---|---|
| state s | Order date, permissions, whether evidence is complete | Provides the information needed for the current decision |
| action a | Direct refund, request verification, transfer to human | Changes the environment and subsequent states |
| reward r | Correct resolution, user friction, loss from exceeding authority | Provides a computable objective proxy |
| policy π(a|s) | The selection probability for each handling action | Maps states to actions |
This differs from supervised learning with a fixed training set: the data is actively generated by the current policy, and actions also change the future visible states, so the training distribution changes as the policy updates. A high return indicates that the policy performs relatively well under the current environment and reward definition, but this holds only within those conditions; if the state omits key information such as permissions or evidence, or if the reward is misaligned or the environment changes, the conclusions may become invalid.
2Discounted ReturnMath
Single-step rewards may conflict with the final result; how do we combine future multi-step rewards into one goal?
The discounted return is a quantity that combines a sequence of rewards after the current time into a single decision objective, resolving the conflict between immediate score and long-term consequences. Its inputs are the future reward sequence and a discount factor, and its output is the weighted sum starting from the current time; it puts "pay a little cost now and gain greater benefit later" and "high immediate score but huge later cost" on the same scale.
Reading term by term:t is the current time;k indicates how many steps the reward is from now;rₜ₊ₖ₊₁ is the corresponding future reward;γ(gamma) is between 0 and 1discount factor,γᵏ makes the weight of more distant rewards progressively smaller;Gₜ is the discounted return starting from t. When calculating, first multiply by γ raised to the power according to distance, then add the rewards for each step.
For example, if you now get −1 and next get +8, take γ=0.9, the two-step return is G=−1+0.9×8=6.2. If we only look at the immediate reward, we would reject −1, but the return shows this step is worth it; a larger return means it is more worthwhile under the current time preference. A small γ is biased toward the short term, and a large γ gives more weight to the distant future; at the same time, longer causal chains increase the difficulty of credit assignment. For finite tasks, the sum can also be stopped directly at the terminal state. γ expresses time preference and the time scale of the task; it is not the case that a larger value is always more correct. Numbers under different γ, time units, and termination rules cannot be directly compared, nor can they prove that the reward itself represents true value.
3Value and Bellman RecursionPrinciple
Before the full trajectory is completed, how does the quality of the current state connect to the next step?
The value function is an expected estimate of future returns under a specific policy, used to address the problem of how to evaluate the current state or action when the trajectory has not yet ended.Vπ(s) answers, “Starting from state s and always following policy π, how much return can be obtained on average”;Qπ(s,a) also fixes the first-step action a, answering, “Now first take a, then follow π, how much return can be obtained on average.” Input a state or state–action pair, output V or Q.
Bellman Recursionis not another kind of reward, but rather decomposes a long problem into “one-step reward + remaining value of the next state”:
In the formula, s′ is the next state after executing the action,γ is the discount factor defined in the previous section, used to reduce the weight of more distant rewards in the next state;Eπ represents averaging over both policy choices and environmental randomness. This decomposition allows us not to wait until all trajectories end; instead, we first observe a one-step transition, then use r+γV(s′) to update the current estimate; temporal difference (TD) is exactly this “using estimates to update estimates” approach.
A higher value indicates a higher expected long-term return under the current policy and environment, not a factual label; sparse sampling, out-of-distribution actions, and function approximation can all produce bias. A single lucky trajectory from the same state does not equal high value. The optimal Bellman equation takes the maximum over actions, but when an approximate Q overestimates unseen actions too high, max will preferentially select the erroneous overestimation, requiring double estimation, target networks, or conservative offline methods to mitigate.
4Exploration and ExploitationDecision
Why might always choosing the current best action lead to never learning a better policy?
The root cause is that the current best is only “best estimated from existing samples” and is not the true best.The exploration–exploitation trade-off is a decision between current reward and information gathering: exploitation uses current evidence to obtain reward, while exploration tries actions with insufficient evidence that may change future choices. Action values come from rewards after trying an action; if an action is rarely tried, its estimate may be low due to chance bad outcomes, or it may have no data at all. If an RL agent then only selects the current first-place action, early errors are permanently fixed by its own choices.
At decision time, input action values, uncertainty, and risk constraints, and output the current step's action; at selection time, first evaluate existing rewards and evidence, then let ε-greedy, entropy bonus, or UCB decide how to leave exploration opportunities.
| Method | How to select an action each time | Why it can break the lock-in | Boundary |
|---|---|---|---|
| ε-greedy | Selects the current best action with probability 1−ε and randomly selects another action with probability ε. | Continuously leaves opportunities for underestimated or insufficiently tried actions to be observed. | Random exploration ignores information value and can waste trials when there are many actions or high cost. |
| Entropy bonus | In addition to the task reward, rewards more dispersed action probabilities, preventing the policy from prematurely becoming “select only one”. | Allows multiple candidate actions to still have sampling probability in early training, so their long-term consequences can still be compared. | Too large an entropy coefficient keeps randomness in the long term; it encourages diversity but does not guarantee trying the most informative action. |
| Upper Confidence Bound UCB | Compares “estimated return + uncertainty bonus”, temporarily adding points to less-tried actions due to their high uncertainty. | Gives priority to trying actions that “may be good but lack evidence”; as data increases, the bonus automatically shrinks. | Requires reliable uncertainty estimates; it is harder to implement in complex states and non-stationary environments. |
After exploration, the estimated ranking changes, showing that new evidence corrected old judgments. Real systems often explore within simulators, shadow mode, or safety constraints; repeating known bad actions only increases risk and does not reduce key uncertainty. In high-risk or irreversible environments, direct online trial-and-error is not allowed; instead, turn to offline experiments, human approval, or reversible environments. “Needing exploration” cannot justify unbounded trial-and-error.
5Value-based Methods and Policy GradientAlgorithms
After learning about returns and values, what methods can make the policy better?
Value-based methods, policy gradient, and Actor–Critic describe three computational routes for improving the policy using experience; they take as input states, actions, returns or value estimates, and output action values, policy probabilities, and updated parameters.Value-based methodsfirst learn action value Q, then choose actions with higher estimated value. DQN is a representative method that approximates Q using neural networks and is suitable for scenarios where actions can be enumerated.
Policy gradientdirectly lets a parameterized policy πθ(a|s) output action probabilities and adjust parameter θ. The core update signal can be summarized as:
J is the expected-return objective;∇logπθ(a|s) represents how changing the parameters changes the probability of the selected action;advantage A(s,a) represents how much better the action usually performs relative to the current state. If A is positive, increase the action probability; if negative, decrease it. It is not a new reward, but uses a value baseline to remove the influence of “this state being inherently easy or difficult.” In short, value-based methods first estimate Q and then choose actions, while policy gradient adjusts probabilities according to advantage.
Actor–Criticcombines the two routes: the Actor is the policy, responsible for choosing actions; the Critic is the value estimator, responsible for providing an advantage baseline. This has lower variance than directly using the entire stochastic return, but a wrong Critic can also give the Actor a biased update direction.
PPOlimits the probability ratio of the new policy to the old policy on sampled actions, avoiding a batch from causing a sudden policy change. It only limits update magnitude; it does not guarantee correct rewards or policy safety. If the advantage is contaminated by incorrect rewards, clipping only makes it go wrong more slowly. PPO's small-step constraint is not equivalent to a safety guarantee. Training should monitor return, KL, policy entropy, clipping ratio, and independent task metrics at the same time; only if these metrics improve together does it indicate that the current update is effective under multiple lines of evidence.
6Reward and Environment Are BoundariesRisk
Training return keeps rising—why can we still not assert that the real objective has been achieved?
Reward and environment together define the world that training can “see” and also constitute the boundary of the reinforcement learning objective. The inputs are the reward rule, state representation, transitions, and permissions; the output is the experience distribution that the policy can optimize. The policy first acts under these rules, then seeks behavioral patterns that improve return.Rewardis a computable proxy for the objective, not real-world value itself;Environmentdetermines which visible consequences actions will produce and may also omit real-world side effects. If the refund system only rewards immediate satisfaction, the policy may learn to make unauthorized commitments; if the simulator has no fraudsters, the policy may fail after being deployed to production.
| Boundary | What can be missed | Independent evidence needed |
|---|---|---|
| Reward definition | Unpriced harms, long-term consequences | Side-effect metrics, hidden end-to-end checks, manual review |
| State representation | Permissions, history, or context information | State sufficiency tests, failure slices |
| Simulated environment | Real-world noise and extreme participants | Offline replay, controlled low-traffic validation |
| Permission boundary | Irreversible or high-risk actions | Hard constraints, approvals, rollback-capable execution |
Therefore result interpretation must report both reward and real-task metrics, and check on environments, time periods, and risk slices that were not involved in training. If the real resolution rate, safety slices, or side effects deteriorate, this should be interpreted as objective misalignment. Constraints, offline replay, and human supervision are used to verify side effects, but they cannot prove that all unknown failures have been covered; omitted consequences, simulation gaps, and irreversible actions cannot be automatically recovered by continued training.
7How One Refund-Handling Process Becomes an MDPRunning Example
When the assistant faces a “35-day quality-issue refund,” why might the short-term reward of immediately promising be lower than the long-term return of verifying first and then processing?
Scroll horizontally to view the full diagram on small screens.
The refund case is a complete credit assignment calculation: input the order state, two candidate actions, two-step rewards, and the discount factor; output the return of each action and the first-step choice. The calculation first lists the immediate rewards, then discounts the subsequent rewards back to the first action, and finally compares −6.8 and 6.2 in the table.
| Action | r₁ | r₂ | G for γ=0.9 | Choice |
|---|---|---|---|---|
| Direct promise | +4 | −12 | 4−10.8=−6.8 | Reject |
| Verify first | −1 | +8 | −1+7.2=6.2 | Preferred |
| Only optimize immediate reward | +4 vs −1 | Ignored | Incorrectly chooses direct promise | Myopic |
Verification first comes out ahead indicates that delayed consequences can overturn immediate satisfaction. Credit assignment must attribute the subsequent −12 or +8 to the first action. Temporal-difference learning can use δ=r+γV(s′)−V(s) to update the value; if current V(s₀)=0 and the post-verification state V(s₁)=8, then the one-step TD target for the verification action is −1+0.9×8=6.2. The longer the reward delay and the more random the environment, the larger the estimation variance.
8How Exploration, Offline Data, and Policy Updates FailFailure Boundaries
Why can't the refund assistant randomly try unauthorized actions online to “explore”?
The safe reinforcement learning workflow places exploration, offline estimation, policy updates, and deployment gates into the same risk-control process, used to reduce the risks of dangerous exploration, offline extrapolation, and sudden policy changes. It takes as input historical logs, a simulator, a safe action set, candidate policies, and monitoring metrics, and outputs a constrained policy and a release or rollback decision. The value of exploration is to obtain the consequences of unknown actions, but real systems have unacceptable costs; offline RL also faces overestimation of out-of-distribution action values because the data never showed their true consequences. After policy improvement, the visited state distribution changes, so old evaluations may also fail.
| Failure | Mechanism | Defense |
|---|---|---|
| Dangerous exploration | Tries high-risk actions to obtain information | Safety set, simulator, approval |
| Offline extrapolation | Q-values for unseen actions are inflated | Conservative estimation, behavior constraints, real small-traffic validation |
| Reward hacking | Exploits proxy or environment loopholes | Hidden final validation, side-effect metrics, permission isolation |
| Policy collapse | Overly large updates cause abrupt behavior changes | PPO/KL, rollback points, staged gates |
| Non-stationary environment | Users/rules change over time | Drift monitoring and re-evaluation |
During implementation, first learn offline or in a sandbox, then use conservative estimation and small-traffic validation, and subsequently continuously monitor KL, key behavior regressions, and real-environment metrics. PPO’s clipped objective limits the change in a single probability ratio but does not prove long-term safety; even if each step update is small, many accumulated steps can still drift far from the reference policy. Small step updates and high offline scores only indicate that local evidence is better; they cannot prove long-term safety. Out-of-distribution actions, non-stationary environments, and reward hacking cannot be handled only by algorithm scores; they still require human oversight and hard constraints.
9Common Misconceptions and Learning PathMisconceptions and Path
Reinforcement learning is a sequential decision-making framework; it is not equivalent to “training whenever there is a reward” or to any one PPO algorithm.
| Common Misconceptions | More Accurate Understanding |
|---|---|
| Having the highest reward at every step is the optimal policy | It should maximize the long-term discounted return; actions change future states. |
| Exploration is just random trial-and-error | It should balance information value and risk, and be constrained by safety boundaries |
| Value functions are facts | It is an expectation estimate under a specific policy and data, and can be biased. |
| PPO guarantees policy safety | It limits update magnitude but does not guarantee correct rewards, environment, or permissions. |
| RLHF is reinforcement learning | RLHF is a class of applications that optimizes language policies using human preference proxies. |
| Level | Concept Dependencies and Extensions |
|---|---|
| Prerequisites | Probability, expectation, gradient descent, supervised learning |
| Core on this page | MDP, return, V/Q, Bellman, TD, exploration and policy gradient |
| Adjacent | RLHF, reward hacking, world models, agent planning |
| Engineering extensions | Offline RL, human-in-the-loop, test-time compute, safety constraints and observability |
10Connecting the Causal ChainSynthesis
How does this concept connect from the problem all the way to verifiable practice?
- The policy chooses actions
- The environment returns states and rewards
- Multi-step rewards form the return
- Value estimation handles credit assignment
- Exploration produces new experience
- Independent evaluation checks whether the reward represents the true goal
11Misconceptions and Self-TestSelf-test
Can you explain its mechanisms, boundaries, and validation methods without memorizing terminology?
- What objects does one step of interaction in reinforcement learning include? What are the input and output of the policy, respectively?
- The current reward is −1, the next step's reward is +8, and γ=0.9. Calculate the two-step return and explain why it might be better than an action with immediate reward +4 and next step −12.
- What is the difference between the inputs and meanings of Vπ(s) and Qπ(s,a)? Why can't a single lucky trajectory prove that the value is high?
- Explain in plain language what problem the Bellman recursion solves, and what γ and s′ represent in the equation.
- When the advantage A(s,a) in policy gradient is positive or negative, how does it update in each case? Why is it not a new reward?
- Why is exploration not equal to random trying? How should a refund assistant obtain evidence when facing unknown high-risk actions?
- What can PPO's clipping guarantee, and what can it not guarantee?
- After both training return and offline success rate have improved, design three acceptance slices and explain the failure boundaries that cannot be offset by averages.
- Assume that "Reinforcement Learning: Learning Decision Policies from Delayed Feedback" performs normally on offline examples, but its core results decline after launch. How would you locate the problem according to input, internal transformation, output feedback, and applicable boundaries?
Reference answers
- The environment provides state s, and the policy π(a|s) takes the state as input and outputs an action or action probability; after the agent executes action a, the environment returns reward r and next state s′. The learning result is a set of state-to-action decision rules, not a fixed answer to a particular problem.
- The former has G=−1+0.9×8=6.2; the latter has G=4+0.9×(−12)=−6.8. Looking only at immediate reward would choose +4, but the discounted return also includes delayed consequences, so one should choose the action that first pays a small verification cost and has a better long-term outcome.
- V takes a state as input and estimates the expected return from that state when following π; Q also fixes the current action and estimates the expected return from first taking that action and then following π. Both average over policy choices and environment randomness, so a single trajectory is only a noisy sample.
- It breaks the long future into "immediate one-step reward + remaining value of the next state", so there is no need to wait for the entire trajectory to end each time. s′ is the next state after taking the current action, and γ is the discount factor used to reduce the weight of more distant rewards in the current estimate.
- When A is positive, increase the probability of the selected action in that state; when negative, decrease it. The advantage is obtained by subtracting a state baseline from the action return, indicating how much better this action is than the typical level for that state; it re-centers the existing reward signal and does not define a new real-world objective.
- The purpose of exploration should be to reduce the key uncertainty that affects decisions; repeating known bad actions has only risk and no information value. High-risk actions should first be verified in a simulator, historical logs, or a restricted sandbox, and be subject to permissions, approval, and rollback constraints; they cannot be tested randomly on real users.
- Clipping limits the change in the probability ratio between the new and old policies in a single update, reducing the risk of a single batch causing abrupt behavior changes; it cannot guarantee that the reward represents the true objective, that state information is sufficient, that permissions are correct, or that long-term safety holds. Many small steps can also accumulate and move away from the reference policy.
- You can slice by permission risk, where the unauthorized action rate must be zero or below a hard upper limit; slice by order age and evidence completeness, reporting correct handling rate and human takeover rate respectively; slice by time or user group, checking returns and true resolution rate after environment changes. Red lines such as unauthorized actions and irreversible harm cannot be offset by high average scores on ordinary cases.
- First save the same failure sample and environment, and confirm that the input, permissions, and preconditions have not drifted; then record key intermediate states and check whether the mechanism completes the transformation as described on this page; then compare the raw output with independent metrics and manual final verification; finally retest with boundary examples and controlled experiments. Only after locating the first link that deviates from expectations can you decide whether to modify the data, mechanism, evaluation, or usage boundaries.
- Reinforcement Learning: An Introduction: MDP, value, and policy fundamentals
- Playing Atari with Deep Reinforcement Learning: DQN
- Proximal Policy Optimization Algorithms: PPO
- Conservative Q-Learning for Offline Reinforcement Learning: conservative value estimation for out-of-distribution actions in offline RL