I allowed a scammer to remote desktop into my routerr norton was installed but vpn was not insyalled. Should I be concerned.

If you know it was a scammer, be sure you run a full system scan with your Norton product. Then run a Norton Power Eraser,NPE, scan from the Norton scans section. Be careful here as NPE can be very aggressive. If it finds something, post back here with the information and we can guide you through it. Then I would suggest a second opinion scan using the FREE version of Malwarebytes. You can find it here https://www.malwarebytes.com/premium/

If you feel the scammer made any changes to your router, you should reset it to the manufacturer's default settings, and then set it up again for your own network.