Most efficient way to generate and validate file checksums

I'd like to be able to capture and validate checksums for large-scale collections of files, typically nested within a complex directory hierarchy. Does every single file need a checksum? Are there ways to leverage the existing directory structure to, say, validate only a node in the file tree and not necessarily every file within?

I realize this is in part a computer science question but I'm hoping those with digital preservation expertise might have encountered a similar challenge.

Aaron Rubinstein

• file-management
• archive-format
• validation
• file-fixity

Comments

• Andy Jackson: As the answers note, it's important to distinguish the types of threat you are mitigating and checksum accordingly. A previous Library and Information Science Stack Overflow answer I contributed may be of interest, although it's mostly about HDFS.