With all of the high-rise condos, coops, and tall apartment buildings packed with millions of people, it’s difficult to believe that a hundred years ago there was enough room on this small island for the super-rich to build mansions all to themselves; single-family homes with the square footage of a castle. Many of these “Millionaire’s Row” mansions are located in the Upper East Side, but the question remains: What are the biggest buildings in Manhattan today that were built as single-family homes?

Felix Warburg Mansion - Built in 1908, this building has remained the largest mansion in Manhattan for over a hundred years. It is presently occupied by the Jewish Museum.

Harkness Mansion - Built on Standard Oil money, and presently owned by gallery owner Larry Gagosian, this 21,700 square foot building is in need of a lot of renovation.

Otto Kahn House - Finished in 1918, this building was modeled after the Palazzo della Cancelleria in Rome. This 50,316 square foot mansion is presently used as a Catholic girl’s school.

Carnegie Mansion - This building is so prominent that the surrounding UES neighborhood is called Carnegie Hill. Carnegie reportedly asked for the mansion to be “the most modest, plainest, and roomiest house in New York,” which seems ridiculous when you look at the gorgeous Georgian façade.

James Duke House - Built in 1912, at a relatively modest 30,000 square feet, the building was inspired by the Hotel Labottiere in Bordeaux. The building was donated to New York University’s Institute of Fine Arts, in 1952.