Goal Displacement in Reward-Shaped Environments: Bundle Preservation versus Local Reward Optimization

Status: Working paper

Summary

This working paper studies a specific failure mode in reinforcement learning: local reward optimization without preservation of the larger action sequence required for task completion. Using a compact survival-crafting environment, the paper traces how naive PPO variants repeatedly converge to locally reinforced behaviors instead of maintaining the full escape objective.

The current draft also reports a structured counter-condition: a bundle-preserving macro policy that solves the base task reliably and transfers strongly under hostile pressure. The paper’s main result is therefore architectural rather than hype-driven: preserving the causal action bundle changes the task outcome in a way that reward shaping alone did not.

Current Boundary

This is still an active working paper. The public-facing summary should be read as a progress snapshot rather than a final publication record. In particular, the present result is about structured bundle preservation and policy behavior under the tested environment conditions; deeper learned arbitration remains ongoing work.