Migrate Proxmox LXC to NVMe: The Safe, Tested Guide to Retiring a Failing SSD

migrate-proxmox-lxc-to-nvme.md
title Migrate Proxmox LXC to NVMe: The Safe, Tested Guide to Retiring a Failing SSD
date
author VahaC
read 9 min read
category Self-Hosting
tags #Backup #Homelab #Linux #lvm #lxc #Proxmox #Ssd #storage
migrate Proxmox LXC to NVMe

This guide shows you how to migrate Proxmox LXC to NVMe storage when the SATA SSD that holds your containers starts failing quietly. You move every container to a new LVM-thin pool, one at a time, and keep the old copy as an instant rollback until you are sure everything works.

My containers lived on a cheap DRAM-less 2 TB SATA SSD, set up as in my LVM-thin storage guide. One afternoon the thin pool flipped to read-only and every container lost write access at once. 💥

Why a Healthy-Looking SSD Can Still Be the Problem 🔍

SMART said PASSED with zero reallocated or pending sectors. Only a raw read test showed the problem: old data read at 30–70 MB/s, freshly written areas at 460 MB/s. That points to charge decay on cheap NAND: no errors, just slow read retries. Under load, one slow metadata write missed its timeout and the kernel froze the pool to protect it.

Rewriting the whole disk fixed the speed, but the decay comes back over months. A 4 TB NVMe drive was sitting on my shelf, so I decided to migrate Proxmox LXC to NVMe for good. (Swapping the boot drive instead? See how to replace a Proxmox SSD without reinstalling.)

Why Not Just Use pvmove?

There is more than one way to migrate Proxmox LXC to NVMe. LVM can move a whole physical volume online with pvmove, without stopping any container. I decided against it for three reasons:

  • It copies everything. pvmove copies the entire thin pool: 1.8 TB on my host instead of about 250 GB of real data, all read from the decaying drive.
  • It is hard to roll back halfway. A failure mid-move leaves you untangling LVM.
  • It keeps the old pool layout. My old pool had only 116 MB of metadata. A fresh pool is the right time to fix that.

With pct move-volume, each container is down for a few minutes, but every step can be reversed. Proxmox copies files with rsync, so only real data moves (see the pct manual).

Prerequisites ✅

Here is what you need to migrate Proxmox LXC to NVMe:

  • A Proxmox VE host with root shell access (this guide was tested on Proxmox VE 9.2)
  • The new NVMe drive installed and visible in lsblk
  • Containers stored on an LVM-thin pool (the old storage)
  • A working backup target, ideally Proxmox Backup Server
  • A maintenance window: each container stops for a few minutes while you migrate Proxmox LXC to NVMe

Step 1 — Inspect Both Drives

Before you migrate Proxmox LXC to NVMe, identify the new drive and make sure nothing on it is in use:

lsblk -o NAME,MODEL,SIZE,FSTYPE,MOUNTPOINTS
nvme list
smartctl -a /dev/nvme0n1

⚠️ Critical: Double-check the device name. Step 4 erases the whole drive. If lsblk shows a filesystem, LVM or ZFS label you do not recognise, mount it read-only and look inside first.

If you suspect the old SSD has the same read slowdown, measure it. This test is read-only and safe on a live disk. Here skip is in MB, so 200000 means “start reading 200 GB into the disk”:

for off in 0 200000 600000 1000000 1600000; do
  echo "offset ${off} MB:"
  dd if=/dev/sdX of=/dev/null bs=1M count=1024 skip=$off iflag=direct 2>&1 | tail -1
done

A healthy SSD reads every offset at roughly the same speed. Large gaps mean it is time to migrate Proxmox LXC to NVMe or another fresh drive. Scrutiny SMART monitoring helps, but SMART alone missed this problem for me.

Step 2 — Back Up Every Container

Before you migrate Proxmox LXC to NVMe, take a fresh backup of every container that lives on the old pool:

vzdump 101 102 103 --storage my-pbs --mode snapshot --compress zstd

Replace the IDs with your own and my-pbs with the name of your backup storage. If you do not have a backup server yet, see my dedicated Proxmox Backup Server setup. Avoid your scheduled backup window.

Step 3 — Clean Up Stale Snapshot Entries

Snapshots do not travel with the volume, and stale ones get in your way when you migrate Proxmox LXC to NVMe and clean up afterwards. List them:

for id in $(pct list | awk 'NR>1 {print $1}'); do echo "== $id"; pct listsnapshot $id; done

Proxmox refuses to move a volume and delete the source while a snapshot still points at it. The error is you can't move a volume with snapshots and delete the source. Delete snapshots you no longer need:

pct delsnapshot 103 my-old-snapshot

If the snapshot is listed but its LVM volume is already gone (for example after a manual pool rebuild), --force removes the config entry anyway:

pct delsnapshot 103 my-old-snapshot --force

The output will still show Failed to find logical volume. That is expected; check with pct listsnapshot that the entry is gone.

Step 4 — Build the NVMe Thin Pool ⚙️

⚠️ Data loss warning: The commands below permanently erase /dev/nvme0n1. Replace it with your device name from Step 1.

# Wipe old signatures and create one GPT partition of type Linux LVM
wipefs -a /dev/nvme0n1p1 /dev/nvme0n1
sgdisk -Z /dev/nvme0n1
sgdisk -n1:0:0 -t1:8e00 /dev/nvme0n1

# LVM physical volume, volume group and thin pool
pvcreate /dev/nvme0n1p1
vgcreate nvme-data /dev/nvme0n1p1
lvcreate --type thin-pool -l 95%FREE --poolmetadatasize 2G -n data nvme-data

This is the pool you will migrate Proxmox LXC to NVMe into, so size it well. --poolmetadatasize 2G gives a 3.5 TB pool plenty of metadata room; full metadata is a classic way to freeze a thin pool. -l 95%FREE leaves room to grow data or metadata later. The lvmthin manual explains both.

Register the pool in Proxmox:

pvesm add lvmthin nvme-data --vgname nvme-data --thinpool data --content rootdir,images
pvesm status

The new nvme-data storage should now show up as active. More options are in the Proxmox LVM-thin storage docs.

Step 5 — Migrate Proxmox LXC to NVMe One Container at a Time 🚚

This is the core of the job. To migrate Proxmox LXC to NVMe, repeat three commands for each container:

pct stop 101
pct move-volume 101 rootfs nvme-data
pct start 101

⚠️ Do not add --delete yet. Without it, the old volume stays behind as unused0 in the container config. If something breaks, stop the container, open /etc/pve/lxc/101.conf, put the old volume ID from unused0 back into the rootfs line (keep the size= part) and delete the unused0 line. You are back where you started.

Tips that saved me time:

  • Start with the least important container and move the critical one last.
  • If your VPN runs in an LXC, move it from the LAN, not through that VPN.
  • Containers with extra disks need one move per disk: pct move-volume 101 mp0 nvme-data. Bind mounts from a host path (for example mp0: /mnt/media,mp=/mnt/media) are not volumes; they stay as they are.
  • VMs use a different command: qm disk move <vmid> scsi0 nvme-data.

Each move takes as long as the real data in the container needs, not the disk size: under a minute for small containers, a few minutes for about 90 GB.

Step 6 — Verify Each Container ✔️

Each time you migrate Proxmox LXC to NVMe, confirm that the container really runs from the new pool:

grep -E '^(rootfs|unused)' /etc/pve/lxc/101.conf
pct exec 101 -- findmnt -no SOURCE,OPTIONS /

You want rootfs: nvme-data:vm-101-disk-0 in the config, and a source such as /dev/mapper/nvme--data-vm--101--disk--0 mounted rw. Then check the service itself: web UI, logs, health checks.

If Docker inside the LXC uses bind mounts from the host, check from inside Docker: docker exec <container> df -h /config must show the real disk, not the host’s root filesystem. If host storage mounts late, the container silently binds an empty directory.

After the last container, check the pool: lvs -o lv_name,data_percent,metadata_percent nvme-data/data.

Step 7 — Retire the Old SSD 🧹

Once you migrate Proxmox LXC to NVMe, give it a few days and let at least one scheduled backup run before you touch the old drive.

⚠️ Critical: Check every unusedN line before you delete it. Old containers sometimes carry leftover unused entries that point at other storages (for example local-lvm). Only remove the ones on the storage you are retiring.

# See which leftovers point at the old storage
grep -H 'unused.*ssd-data' /etc/pve/lxc/*.conf

# Remove one leftover volume (this deletes the old copy!)
pct set 101 --delete unused0

When no container references the old storage any more, take it out of Proxmox and clear the disk:

pvesm remove ssd-data
lvs ssd-data          # must list only the thin pool itself
vgremove ssd-data
pvremove /dev/sdX1
wipefs -a /dev/sdX1 /dev/sdX

pvesm remove only drops the storage definition. vgremove deletes data, so run it only after lvs shows nothing but the pool.

Troubleshooting

Errors you may hit when you migrate Proxmox LXC to NVMe:

  • cannot move volumes of a running container: stop the container first with pct stop.
  • you can't move a volume with snapshots and delete the source: remove the snapshots (Step 3) or move without --delete.
  • Configuration file ... does not exist: the ID is a VM, not a container. Use qm disk move.
  • Container stuck in starting: run systemctl list-jobs inside it. Mine waited for a DHCPv6 server that does not exist (ip6=dhcp). Unrelated to the move; IPv4 worked fine.
  • Something is broken after the move: roll back as described in the Step 5 warning. If that fails too, restore from your backup.

Final Thoughts

Using pct move-volume to migrate Proxmox LXC to NVMe costs a short stop for each container, but every step can be reversed. The old copy stays in place until you decide to delete it, and each container is a separate, small job.

One honest note: when you migrate Proxmox LXC to NVMe, you get speed, not redundancy. A single QLC NVMe can develop the same read slowdown over time. Keep backups running, test read speed now and then, and consider a mirror once a second drive is available. 🏠

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.