The off-policy issue arises when there is a discrepancy between the policy used to generate behaviour (or data) and the policy currently being evaluated or improved. This is significant in contexts where the agent needs to learn from past experiences or data generated from a different policy.