Peak, off-peak and super off-peak periods vary from line to line. That's why the National Rail site is 'vague' about it. For journeys between Leicester and Birmingham you can't use the return half of an off-peak day return (£10.90) on trains which leave Birmingham between 15:30 and 18:00. The last train that they can be used on before this is the 15:22 and the first train afterwards the 18:22.
You will not need to buy the Anytime Day Return (£16.30) for your journey though because, perhaps surprisingly, the off-peak return (£15.20) has no restriction on the return portion.
