Cant use the hostonly IP for non 80 port

Hi
I have a windows 10 VM running on my mac host. I have set up bridged networking and also host only. I want to use the static host only IP as the bridged network IP can change. I am running a docker container in the windows VM on port 5001. I can access the container using http://BRIDGEDIP:5001 from my host machine and it responds nicely. However when I try and use the http://HOSTONLYIP:5001 it cant connect. If I try from inside the windows 10 VM it responds on both the bridged and hostonly IP.

I have opened the ports as without it the bridged check would not work. Is there something I have missed in my setup that I need to do here to enable communication from the host machine to the VM via the hostonly IP?

I should also add. It I just try http://HOSTONLYIP/ (so going via port 80) I get the default IIS response so it can definitely connect to the VM via that IP as well.

Thanks
Matt