Emulator

Such a hash doesn't exist in the format, and I'm reluctant to create a new version of the format just to add this. There is however information in the disc header at 0x58 (game ID, revision, disc number) which is sufficient for uniquely identifying any retail disc, as long as you don't need to distinguish bad dumps from good dumps.

Actually identifying dumps is what I'd like to do, that's why an sha1 of the input would be perfect as I can just check against redump DAT file to find bad dumps or rename files as needed.
Without such a hash I would need to decompress the whole image and check the slow way which I'd like to avoid.

Are there checksums for the compressed blocks?

To actually verify the dump you have to decompress the file anyway, as you have no way to prove the stored hash is correct otherwise.

Regarding the block checksums, Zstd has optional checksums, and I'm less sure about bzip2/LZMA/LZMA2. RVZ doesn't apply any additional checksum for each block on top of what the compression algorithm uses, so if you use no compression, you get no block checksum.

Pikachu025 wrote:

To actually verify the dump you have to decompress the file anyway, as you have no way to prove the stored hash is correct otherwise.

True but if I have verified my images against redump and then compress them I can assume the stored checksum matches, this would only apply to images from unknown sources.
This way I don't have to decompress all the data and can just quickly find new bad dumps or renamed files with a simple python script.

JosJuice wrote:

Regarding the block checksums, Zstd has optional checksums, and I'm less sure about bzip2/LZMA/LZMA2. RVZ doesn't apply any additional checksum for each block on top of what the compression algorithm uses, so if you use no compression, you get no block checksum.

But I need a checksum that's always the same for the same input data and not be affected by compression settings, similar to the chd format mame uses.
Block checksums would only work if they where for uncompressed data using the same algorithm at a fixed block size e.g. one crc per 32kb block this way the individual checksums could be combined into a bigger one for easy matching.
As long as these block checksums are in a single block somewhere I could easily read those without reading the whole file, a full sha1 of the input data would also do of course.

If this is too much of a breaking change to do now at least add something when changing the format in the future.

BTW is there a way to compress/decompress images from the command line?

JosJuice wrote:

whocares0101 wrote:

True but if I have verified my images against redump and then compress them I can assume the stored checksum matches, this would only apply to images from unknown sources.

We have no interest in helping you manage disc images from "unknown sources".

When did I say I'm using images from unknown sources?
I just said that if I compress the files myself I should be able to trust a stored hash.

Ah, you mean that the "this" in "this would only apply to images from unknown sources" refers to Pikachu025's statement? I apologize then.

Regarding converting images from the command line, Dolphin currently doesn't support this, but it's something I've considered adding.

Yes that was what I meant, reading my sentence again it does sounds ambiguous now.

So no checksum any time soon then?

Could you make this into a small external command line program like chdman from mame, this way the full emulator wouldn't be needed to convert between formats.

So no checksum any time soon then?

That's correct. I'll keep it in mind if we're ever going to do an update of the format, but I don't know if that's going to happen and if we'll actually add it if it happens.

Could you make this into a small external command line program like chdman from mame, this way the full emulator wouldn't be needed to convert between formats.

I believe Nanook is planning to add RVZ support to the NKit app sometime. I assume that would resolve it?

That's correct. I'll keep it in mind if we're ever going to do an update of the format, but I don't know if that's going to happen and if we'll actually add it if it happens.

Well that doesn't sound very promising, I still hope it happens though.

Here's another idea for this.
The reason you don't want to add a checksum is because it would make the binary format incompatible right?
So what if instead of changing the format repurpose the compression_level in WIAHeader2 and store a crc32 of the input data?
This shouldn't cause any problems and according to the comment isn't really used for anything, even the properties dialog in dolphin isn't showing this.
It's only a crc32 but it should still be good enough to identify the input data.

I believe Nanook is planning to add RVZ support to the NKit app sometime. I assume that would resolve it?

Sounds good.