Revert "Use previous build to speed up future builds"

This reverts commit 815b52da.

Though I thought that cleaning up the build directory with lb clean would
resolve any conflicts, I'm still getting strange build errors. Wiping the
build directory seems like the best option for now and I will revist this
later.