My god, that article is actually filled with terrible advice. I hope they don't do this professionally. It's bad enough they're misinforming people and directing them to do things in a worse, more fraught failure-prone way.
> First of all, 'dd' is going to try to read the entire drive, including the parts which currently do not contain any useful data. This happens because you have probably told it to read /dev/hdc. That refers to the entire readable portion of the disk, including the partition table and the parts which are not associated with living files.
Make a sparse image with dd. Then you have a 1:1 copy while skipping all the unused sectors to save space. You can then mount the image read-only or write it to a new disk for recovery. That means that you have all the time in the world and you can start over if need be without affecting the source data.
Doing all this on a possibly actively failing drive, particulary an SSD which can literally die 100% into an unrecoverable state without access to a lab in the blink of an eye? That's insanity. And it's even worse SUGGESTING that to random people who don't know any better. You want to read everything off the SSD or HDD once. As fast as possible. As completely as possible.
In fact, make sure you do a non-trim pass first, so you can be sure you have as much as possible before you try retrying sectors which might actually kill the drive dead.
Once all the recoverable data is off the failing drive as an image, you're free to do whatever you want however you want without fear of any data loss caused by possible mistakes, oversights, hardware issues, etc.
> Make a sparse image with dd. Then you have a 1:1 copy while skipping all the unused sectors to save space.
Filesystems don't zero out deleted data and dd isn't aware of the filesystem mapping, so unless it's a completely fresh drive, you'll still pull off garbage data.
ddrescue is one tool that combines dd with the ability to read the filesystem metadata to only extract out the allocated filesystem blocks.
Sorry, you're right, I definitely meant ddrescue. That's what I always use, no idea why I wrote dd. Especially since ddrescue -d -n -S /dev/sdb sdb.raw sdb.logfile is permanently burnt into my brain. I guess I just took it from the previous comment and article and didn't give it a second thought before posting my comment.
> Filesystems don't zero out deleted data and dd isn't aware of the filesystem mapping, so unless it's a completely fresh drive, you'll still pull off garbage data.
Sure, but if there's unused space left zeroed, such as partitions that you didn't use, then you get to skip it. It's not a huge priority but it doesn't hurt.
>Filesystems don't zero out deleted data and dd isn't aware of the filesystem mapping, so unless it's a completely fresh drive, you'll still pull off garbage data.
That may have been true a very long time ago.
Today, some filesystems such as ZFS can effectively, and automatically, zero out deleted data using trim on devices that support that. (I say "effectively" here because we don't actually know if the data is literally zero'd by the underlying device, and I say this even though that distinction doesn't actually matter in this context. Once trimmed, those logical sectors will always read as zeros until something new is written to them, and this is the functionality that is important in this context.)
This function is useful for SSDs, and is also useful for SMR spinny-disks.
Tons of other combinations of filesystems and operating systems also deal quite well with trimming of unused space, though this more-often happens as a scheduled task instead of something that is taken care of by the filesystem itself.
Trim (in various implementations) has been broadly used for well over a decade, and a trim'd device can lead dd to be able to produce sparse files.
---
Now, that said: It probably doesn't matter much if a particular dd-esque tool is set to create sparse output files or not. Sure, some space may be saved, and sparse files sure are cute and cuddly.
But it's probably a fool's errand to even plan such an operation on a machine that has less free space than the total maximum capacity of the thing being rescued: Either there's enough room to write a non-sparse image, or there isn't enough room to even think about starting the process since it might not be able to complete.
(If space becomes an issue later on down the road, the output file can be "sparsified" in-place using "fallocate --dig-holes" in instances where that makes sense.)
And I definitely want the whole disk imaged, which means that I definitely do not want ddrescue's interpretation of metadata to determine filesystem allocation and limit the scope of that image: This is the first step of a data rescue operation, and that makes it the worst place for data to be intentionally thrown away or disregarded.
If things are failing hard enough that any of this work is on the table, then obviously the combination of the source disk and filesystem is untrustworthy -- along with the metadata.
Getting all of the bits backed up -- regardless of their apparent lack of importance -- should always be the prime directive here. Any extra bits can always be tossed later if they're eventually deemed to be actually-unimportant.
There is no purpose in sending TRIM after each delete: at worst (with a naive implementation) you would get a solid write amplification and at least your drive would be doing GC instead of serving the data.
Most of the time it's just sent every once in a while, often triggered by the schedule and/or the amount of writes[0].
[0] my current machine says it was 9 days since 'the last retrim' and T440 which 'works' as a glorified dashboard (ie minuscule writes overall) says it's 24 days.
I'm pretty sure that OSes usually send TRIM right away and let the drive figure out optimization. Wikipedia says "some distributions" of Linux turn it off but my various ubuntu-ish systems have `discard` or `discard=async` in the output of `mount`.
"last retrim" on Windows is an extra feature, because TRIMs can get dropped when a drive is busy enough. It goes through all the free space and TRIMs it again once a month.
Also even if you did only TRIM once a month, I think most of your free space would still be zero.
> You want to read everything off the SSD or HDD once.
I mean, no, not really. You want to read everything off at least once. Because a "corrupt sector" is actually often in a non-deterministic state, reading differently with each read — but that state may still be floating just far enough toward logical 0 or 1, that you can get a bit-pattern out of it through statistical analysis, over multiple reads.
You do want to capture the whole disk once first, in case the disk spindle motor is about to die. (This isn't usually the problem with a dying disk... save for certain old known-faulty disk models — the "DeathStar" et al. But it's good to be safe!)
But after that, you want to read it again, and again, and again. And save the outputs of those reads. And then throw them into a tool like ddrescue that will put the results together.
Basically, it's the same principle that applies to archival preservation of floppy disks (see e.g. https://www.youtube.com/watch?v=UxsRpMdmlGo). Magnetic flux is magnetic flux! (And even NAND cells can end up with approximately the same "weak bits" problems.)
---
Mind you, it'd be even better if we could get several analogue dumps of the disk, and then merge those together using analogue-domain tools, like the ones used for floppy preservation mentioned in the video above.
Sadly, unlike with floppy preservation, it'd be very difficult to get an analogue dump of an HDD or SSD. With floppies, the flux is right there for the reading; you just need a special drive. HDDs and SSDs, meanwhile, are far more self-contained. For an HDD, there might be a path to doing it, at least on a disk-specific basis... it would likely involve tapping some signal lines between the disk controller and the read head. But for an SSD, I don't think it would be possible — IIRC with most NAND/NOR flash packages, the signal is already digitized by the time it comes off the chip.
---
Also, I should mention that if your HDD disk spindle motor is about to die, then you're actually on a bit of a ticking clock. You want to get your data off that disk as quickly as possible, before the motor locks up and the head crashes on the platter.
Tools like ddrescue have the option to first read a disk's filesystem metadata to build a map; and then use that map to only bother to read filesystem-allocated sectors, seeking past any unallocated ones.
Doing this can greatly speed up the time it takes to make a complete copy of "any potentially-useful data on the disk" (as opposed to "the entire disk".) Which means you might be able to get a capture done on a disk "just under the wire" using this method — where otherwise the disk would have died in the middle of reading some unused sectors, with some good data stored further down the disk left un-captured.
But of course, there's the counterpoint that the filesystem metadata itself might have holes in it, where that would mean that the map made by the tool will fail to capture some of the data.
I think this isn't a weighing-pros-and-cons situation, though, but pretty clearly a "right tool for the job" situation:
• If you're getting fsck errors, weird data corruptions, or files are just disappearing — and it's clear that you're going to need to run one of those forensic data recovery programs against the disk image to extract what you can from it — then you'll need the whole disk, including the "seemingly unallocated" parts.
• If all your data seems fine, but you're getting SMART errors, writes are slowing down, and your disk is making funny noises... then you're probably better off grabbing just the filesystem-allocated sectors ASAP, before your drive stops talking to you.
> Tools like ddrescue have the option to first read a disk's filesystem metadata to build a map; and then use that map to only bother to read used sectors, skipping the unused ones.
I don't know how many versions of "ddrescue" there might be, but GNU ddrescue doesn't know anything about filesystems.
The "map" file is just a list of device LBAs (extents) and recovery status.
I've seen a tool for NFTS that can pre-populate a GNU ddrescue map file with unused extents to avoid ddrescue reads of unused areas, but you have to find this tool and manage it yourself.
> First of all, 'dd' is going to try to read the entire drive, including the parts which currently do not contain any useful data. This happens because you have probably told it to read /dev/hdc. That refers to the entire readable portion of the disk, including the partition table and the parts which are not associated with living files.
Make a sparse image with dd. Then you have a 1:1 copy while skipping all the unused sectors to save space. You can then mount the image read-only or write it to a new disk for recovery. That means that you have all the time in the world and you can start over if need be without affecting the source data.
Doing all this on a possibly actively failing drive, particulary an SSD which can literally die 100% into an unrecoverable state without access to a lab in the blink of an eye? That's insanity. And it's even worse SUGGESTING that to random people who don't know any better. You want to read everything off the SSD or HDD once. As fast as possible. As completely as possible.
In fact, make sure you do a non-trim pass first, so you can be sure you have as much as possible before you try retrying sectors which might actually kill the drive dead.
Once all the recoverable data is off the failing drive as an image, you're free to do whatever you want however you want without fear of any data loss caused by possible mistakes, oversights, hardware issues, etc.