Session rewinds to earlier part of screen sharing activity

I'm having a really odd problem. When I connect to my Windows 10 PC from my MacBook when I am away from home the session starts up fine but after a few minutes the session appears to stall and then starts again but is running 5 or 6 minutes behind realtime. I spotted this when my local machines clock was 6 minutes ahead of the time showing on the Windows PC which was 5 miles away. This makes supporting the person using my Windows machine almost impossible.

Why does this happen and how do I either stop it happening or resend the session so it is in real time again? Both machines have the latest version of Parallels Access installed.