We only need to build one "window" set, enough to cover the view of the camera.
Of course "the voyeur point of view", being stationary, would see each window on the building across from him from a slightly different angle, but we can simulate that by shooting the same window from many different angles. Some high, some low, some right, some left.
The set we distribute could have all the various camera angles built into it and everyone who joins in either claims one or is assigned one.
The transition from one window to the next would be covered by a blurry swish pan carefully edited in between.
The camera could have a rotoscope on it to simulate the "binoculars" view