As long as a buff is used (Tome or Moonshard) the scoreboard doesn't show building damage.
I think this is due to the extra column for abilities in the AD scoreboard and thus not being enough columns available.