Emulator

hmm I havent seen any memory leaks here, just compressed about 20 wii games with no
slowdown

It does this to me, while the game is compressing memory usage rises up to 99% - 100%
until the process finishes, then my system is sluggish until a reboot. I can't
reproduce the same exact behavior every time, sometimes it eats up less and sometimes
more.

Compressing Gamecube games doesn't seem to have this effect, only when compressing
Wii games and can be reproduced most of the time.

Perhaps it's a 64-bit Vista issue when compressing Wii games, I thought I would just
make a mention of it in case there is anything wrong. If anyone else is having the
same problem, I have seen it reported on the issue tracker.

um, this IS the issue tracker?

yes i know of this issue, dunno what causes it (haven't really looked). Should probably
do that sometime...

I meant that if someone else was having the same problem, I didn't see it reported
here, since I performed a search before filing the report.

i'm on vista64 as well, strange that I didnt see the leak

Hi there,

just tried on win 7 x64, works perfect. ram usage ~32/33MB while compressing.

Does it happen with all your wii games?

For the most part this happens every time I try to compress a Wii game, one time
memory shot up to 99% - 100% usage on Vista x64 when compressing a Wii game then
dropped after the compression was finished. My system continued to run sluggishly
afterward until I rebooted my machine.

so i've made the issue reproducible: the memleak happens when the scrubber is run, then
while the compressor loops, memory usage goes crazy. At the moment I have no idea why.

bleh, it fixes the memleak when scrubbing, but not when compressing...

It turns out it isn't a memory leak using up all the memory; it's the output buffer.

I guess when we get to data that doesn't compress very well, we start adding stuff to
the output buffer faster than it can write it to the disk. We keep filling it, and
eventually the computer runs out of memory.

This patch should fix it. You can adjust the value to make it flush the buffer more
or less frequently. Right now it flushes the buffer about every 16,000 blocks.

Also, I still get a big memory leak when scrubbing Wii games. If I scrub the game
then stop Dolphin, I can compress the game fine with no leaking.

Ehhh, this issue is about the memleak when scrubbing + compressing wii games. I haven't
noticed any large use of memory in other scenarios.
(ie - what does your patch fix?)

When compressing large Wii games, if your hard drive isn't fast enough (usually when
it's getting close to full) the output buffer grows faster than it can write to the
hard disk, causing extreme memory usage. (I think it does so for GameCube games too,
but because they're usually much smaller, nobody really notices because they never
use up that much RAM.) This patch periodically flushes the output buffer to free up
memory during the compression process.

The wierd thing is, it seems the scrubber keeps leaking memory even when the
scrubbing process ends and the actual compression begins. The scrubber memory
problems might also be file buffer related.

Yes I understood what you patch does, I was just saying that I had never noticed
extreme memory usage due to that :) (Which probably just means I wasn't looking hard
enough).

"The scrubber memory problems might also be file buffer related."
Certainly possible. Here is were all the fileIO in DiscScrubber is performed:
http://tinyurl.com/ydww2r6
Feel free to play around with that loop (I realize it's pretty tight...)
Probably the solution is to create a map of all the contiguous "free" area, and do
just a few writes (for each free section), instead of such an extremely large amount.

You probably never noticed it because your hard drive was fast. Mine is kind if slow
(and kind of full) so I got this problem compressing any Wii game. (Except NSMB Wii,
which compresses to only about 300mb.)

I'll look at the scrubber code to see if I can find the problem with it.

Turns out the issue with the scrubber is some weird conflict between Windows' native
CreateFile and C's file functions. PlainFileReader (which the scrubber uses) uses
CreateFile and pretty much everything else uses the C way. I changed PlainFileReader
to use C's standard file functions. This could have side effects, but I haven't
noticed any.

(this patch was originally a part of the one I made for Issue 2214, so tell me if you
can't apply it.)

oh yes! baby.lueshi this patch is so much better. Who needs those ifdefs anyway?

Nope, no sarcasm there. Just one too many exclamation marks and stupid questions.

Cross-platform file management is the way to go. I don't really know who even wrote
that windows only code in the first place and why...

The only problem it might have is opening large files, because from what I've read,
Windows sometimes has problems handling files over 2GB when not using its native file
functions. However, I didn't run into any problems testing both the 32 and 64 bit
versions.

From what I heard, there's functions such as fseek64 and more of the sort. It's even
used in the Common::File class IIRC.

I didn't notice this before, but Dolphin already overrides the 32-bit versions of
fseek/ftell with the respective 64-bit versions (in CommonFuncs.h)

So this patch SHOULD be fine to apply.

The fflush was really just left over from my previous attempt to fix this, so it
shouldn't be necessary.