A bit more Katharine Hepburn nautical than Kate Bosworth Blue Crush - think yar - it's the sea that calls to you, not the beach. The perfect dock of a village for boat shoe wearing, complicated rope tying, obviously privileged sailing enthusiasts, Sag Harbor is more than just that town everyone takes a short cut through to avoid the traffic getting further out East.
Though you're all for an escape, you've still a life to check on in Manhattan, so this not too close, not too far Goldie Locks situation of a town is just right.