In the case of Paris, the UNESCO perimeter is just the banks of the Seine, the Islands, the Louvre and few parks, squares and landmarks overlooking the Seine river.
It does not prevent the construction of buildings with modern design in Central Paris.
The Samaritaine is just outside the UNESCO perimeter.