Fix Directory not empty error during cache move in build

We got this error when building

mv: cannot move '/builds/controls/picoscale-sdk/.ccache' to '/builds/controls/picoscale-sdk/../.ccache/.ccache': Directory not empty

Some debugging made us think that the problem is due to the temporary cache dir not being removed. This MR should fix it.