Even if mostly unused, the memory is not available to the host anymore, which causes problems when running many VMs. This partly reverts commit b88942f8.