When you distract from one subject, you should not totally withdraw from the subject. You have to gradually reduce concentration on the subject temporarily. If you think you are getting distracted from the subject, you have to focus again on the subject to maintain the minimum level of interest.

This may look easy but not. You have to practise more and more.