For three years, a large brick building sandwiched between restaurants in the heart of Green Bay’s downtown sat empty.