No, that's natural. First of all, the number of people in your city determine the quality/availability of the buildings therein.
As time passes without you interacting with the characters living inside your city, they leave, and the quality of your city decreases. Eventually, you get a practical ghost town. The only way to get back those people who left is to use Entralink and enter into someone else's White Forest, and invite NPCs back to your Black City. Then, the whole process starts over again. To maintain your city's population, you must interact with the NPCs often. Or they'll leave again.