Here is what one of my favourtie reference books (Brewer's Dictionary of Phrase and Fable) has to say on the subject of Fitzrovia:

"A world of outsiders, down-and-outs, drunks, sensualists, homosexuals and eccentrics"
Times Literary Supplement, 1958

Sounds brilliant!