Emulator

Based on a hardware test, unaligned writes to the 0x80000000 area behave as usual, and unaligned writes to the 0xC0000000 area seem to cause exceptions. This matches IBM's PowerPC documentation. Perhaps the funky "duplication" behavior is caused by the game's alignment exception handler?

I will try to implement this exception correctly and create a build you can test. (I only have NARP personally and not NARE, so I wouldn't be able to load your savestate.)

Oh interesting! I hadn't considered that possibility before. I wonder if the store is being implemented by the internal N64-to-Wii dynarec as a stwcx. That would definitely generate an alignment exception, and would certainly make sense, since for non-80/A0 address space, it simply adds the N64 address (C01C5557 in this case) to a fixed offset relative to the start of MEM1 where emulated RDRAM is stored (0x00F22FC0 for the JP version of our practice tool, "kz", and approximately that value for other versions). stwcx would be an efficient way to sum the N64 address and MEM1 offset and do a store in a single instruction.

Admittedly I have not researched the PPC recompiled assembly nearly as much as the MIPS original, so that's why a lot of this is conjecture for me. I can try to look into it more if needed, I'm just not as comfortable with Dolphin's debugging tools as PJ64's, but I'm sure I can figure it out with enough time.

In any case, I would be happy to test such a build for you! Unfortunately, the speedrunning community doesn't tend to make setups for PAL/NARP, since it's presumed slower by default due to 50 Hz instead of 60 Hz.

stwcx is documented as generating an alignment exception on misaligned addresses regardless of whether the memory mapping is cache-inhibited, but as you've shown, the behavior here is different for 0xC0000000 (which is cache-inhibited). So the instruction involved is probably one of the more "normal" store instructions, like stwx.

And yes, the PAL version is pretty slow. Even in casual play it feels sluggish unless I put on the Bunny Hood :)

Here's the pull request containing the changes I would like you to test: https://github.com/dolphin-emu/dolphin/pull/9865
Windows build: https://dl.dolphin-emu.org/prs/66/1f/pr-9865-dolphin-latest-x64.7z

Testing instructions:

1. Start Dolphin, then close it again (just to add the AlignmentExceptions line to Config/Dolphin.ini)
2. Open Config/Dolphin.ini and change AlignmentExceptions = False to AlignmentExceptions = True
3. Start Dolphin again and test if Majora's Mask behaves differently

Loading a savestate from a previous version of Dolphin is probably fine (as long as you know that the misaligned store didn't happen before making the savestate, of course), but if you can't manage to get the intended behavior when using an old savestate, please also try without using any old savestates.

Unfortunately, that build crashes when I do the unaligned write (even with making new savestates), complete with the characteristic Wii buzzing sound :) Is there any crash log or anything I can provide?

Something that may be worth mentioning... LightNode SRM actually does 2 writes. So while writing E02D0043 to the address C01C5557 is the goal, it comes with a spurious write of C01C5553 to the address E02D004B (which ends up as a DWW on real hardware, since it only crosses the singleword boundary of E02D004C). Obviously E0 is not a valid memory domain for the Wii (as far as I know), which is why I'm assuming it's something built into the internal N64-Wii emulator, likely to handle Paper Mario which uses the N64 E0xxxxxx domain heavily for its TLB. Also note, all of the above are N64 addresses, so add 0x00F1D0E0 to convert to Wii addresses.

It would be better if I could read the recompiled PPC asm for how the internal dynarec handles writes to N64 address space, but I can't seem to figure out breakpoints in Dolphin. When I set a write breakpoint on the address C01C5557, I expect it to stop execution when that address is written to for the first time, but that doesn't happen for some reason.

Breakpoints in Dolphin only work in MEM1, which is why that breakpoint doesn't work.

The OoT community has finally started playing around with this hardware quirk for their new Any% route. Usually once they get their hands on something, it gets figured out a lot faster since the OoT community is much bigger. Someone from that community suggested to me that this is likely a 60x bus behavior regarding memory transactions, not a CPU behavior regarding loads and stores. They linked this reference manual of the 60x bus: https://www.nxp.com/docs/en/reference-manual/MPC60XBUSRM.pdf

I've taken a look at it and while it does confirm that crossing a doubleword boundary with a memory access requires two separate accesses (one for each doubleword), it does not mention that each access will fill the entire doubleword if cache-inhibited. Maybe someone who understands the GC/Wii architecture better than me will get more out of this.

"Cache-inhibited" is a concept that exists in the CPU, not on the bus. However, whether an access is cache-inhibited does end up affecting the bus access patterns of the CPU. When caching is used, the CPU reads and writes entire cache lines at once (or at least I believe so?), and when caching isn't used, the CPU just reads and writes exactly what the program requested. So when caching is not inhibited, none of the CPU's accesses are misaligned as far as the bus can tell, even if the program is doing misaligned accesses from its own perspective.

So, yes, it's possible that this behavior has to do with how the bus works. I don't know if the fact that I managed to get what looks like a crash when doing misaligned writes to a cache-inhibited view of memory is a red herring or not...

While he doesn't go into much detail on the exact behavior, this thread from marcan seems closely related.

Has someone managed to reproduce this on real hardware with a minimal test case? e.g. a single "stw" to the uncached 1T-SRAM mapping which triggers the 128B overwrite. This would be a good starting point.

#3 mentions trying to reproduce but I suspect it wasn't done right -- JosJuice probably knows that now after having written the pull request, but "stw" shouldn't trigger an unaligned exception, even when writing to M=0 mappings. Instead the PPC should convert that to two single-beat 60x writes, with the right write byte enables to mask writes to the irrelevant areas within each 64b dword.

I've been brainstorming this with segher, the current most plausible hypothesis we have is that something in Hollywood is mishandling these write masks on the 60x transactions. The 1T-SRAM is probably not directly on the 60x bus, there's likely some crossbar in Hollywood itself and some 1T interface, and it's possible that at one of these levels the write masks get converted to some other system (e.g. access sizes). If this conversion logic pattern matches "known patterns" that only supports aligned accesses mask patterns, and defaults to full 64b write otherwise, then that might explain this situation.

It's known that Hollywood does weird stuff with these write masks in some situations: for example, they get ignored on some MMIO accesses. This clearly indicates that something somewhere drops that information. But clearly it's not entirely dropped, because otherwise no single-beat 60x memory transaction (aka. any cache-inhibited write) would ever behave properly.

Has someone managed to reproduce this on real hardware with a minimal test case? e.g. a single "stw" to the uncached 1T-SRAM mapping which triggers the 128B overwrite. This would be a good starting point.

I don't really have the Wii development skills necessary to write a program like this, but I'm reasonably sure what is happening in OoT/MM VC doesn't have any funny business beyond the single unaligned, uncached stw we know it does (see screenshot of the disassembly in my earlier post).

I've been brainstorming this with segher, the current most plausible hypothesis we have is that something in Hollywood is mishandling these write masks on the 60x transactions. The 1T-SRAM is probably not directly on the 60x bus, there's likely some crossbar in Hollywood itself and some 1T interface, and it's possible that at one of these levels the write masks get converted to some other system (e.g. access sizes). If this conversion logic pattern matches "known patterns" that only supports aligned accesses mask patterns, and defaults to full 64b write otherwise, then that might explain this situation.

This sounds very likely based on marcan's thread.

It's known that Hollywood does weird stuff with these write masks in some situations: for example, they get ignored on some MMIO accesses. This clearly indicates that something somewhere drops that information. But clearly it's not entirely dropped, because otherwise no single-beat 60x memory transaction (aka. any cache-inhibited write) would ever behave properly.

My guess is the catch here is that it's not any cache-inhibited write, it's cache-inhibited unaligned writes (perhaps also, cache-inhibited writes of less than 1 word). Each of these by itself is rare, both at once is probably something that almost never happened in retail software. When you're using a cache, even "unaligned" writes present themselves to the 1T-SRAM as an aligned 1+ word write.

Basic attempts at a repro failed, so it seems like there must be some kind of funny business involved. Here are the cases I've tested, for completeness. ubuffer here is a 16B aligned uncached MEM1 address:

	printf("aligned 32b to +4\n");
	reset_buffer();
	ubuffer[1] = 0x12345678;
	hexdump(ubuffer);
	
	printf("aligned 16b write to +6\n");
	reset_buffer();
	((u16*)ubuffer)[3] = 0x1234;
	hexdump(ubuffer);
	
	printf("aligned 8b write to +7\n");
	reset_buffer();
	((u8*)ubuffer)[7] = 0x12;
	hexdump(ubuffer);
		
	printf("unaligned 32b to +5\n");
	reset_buffer();
	*(u32*)((u8*)ubuffer + 5) = 0x12345678;
	hexdump(ubuffer);
	
	printf("unaligned 32b to +6\n");
	reset_buffer();
	*(u32*)((u8*)ubuffer + 6) = 0x12345678;
	hexdump(ubuffer);
		
	printf("unaligned 32b to +7\n");
	reset_buffer();
	*(u32*)((u8*)ubuffer + 7) = 0x12345678;
	hexdump(ubuffer);
			
	printf("unaligned 16b to +7\n");
	reset_buffer();
	*(u16*)((u8*)ubuffer + 7) = 0x1234;
	hexdump(ubuffer);
			
	printf("unaligned 32b to +3\n");
	reset_buffer();
	*(u32*)((u8*)ubuffer + 3) = 0x12345678;
	hexdump(ubuffer);

	printf("unaligned 16b to +3\n");
	reset_buffer();
	*(u16*)((u8*)ubuffer + 3) = 0x1234;
	hexdump(ubuffer);

All of these show the "expected" writes, nothing being overwritten that shouldn't be overwritten (https://i.imgur.com/VaRohkD.png). I've also tried matching the HID0/HID2/HID4 settings from OOT VC, in case some weird bit got enabled there, but that didn't change anything. DBAT mappings also don't seem unusual for the 0xC range in OOT VC. Running out of ideas right now, will come back to it tomorrow morning maybe :)

Good news: turns out my repro was broken because I accidentally dropped a volatile at a critical place and the compiler decided it should just make all the writes aligned...

Bad news: now the repro looks pretty much as simple as it could be ("stw rA, 7(rB)"). But when doing any kind of unaligned u32 write, or even aligned u16 writes to +2, we get a Broadway hang which doesn't even look like it triggers any exception (like, it's not a DSI/ISI/Alignment/MCE, it just freezes as far as we can tell). I'm entirely confused by this, because this is not a result that anyone else has been getting. I suspect I'm doing something stupid, but it's very unclear what, and aligned 32b writes do work with that same test harness...

Very interesting. At least we are finally getting some sort of abnormal behavior, though it's not exactly what we were expecting.

This is a long shot, but is there any chance that the behavior is somehow reliant on the b (branch) instruction immediately after the stw? I know on N64, we have to be particularly careful around branches and jumps due to the delay slot. Pretty sure PPC doesn't have one of those, but it's just an idea I had, since I'm still finding it hard to envision any shenanigans within the game itself, since the code is so simple (literally just the four instructions: and, add, stw, b)

It definitely seems like there's an interaction with either the instruction pipeline, the load/store execution units on the CPU, or something else altogether. So far we've found many ways in which it freezes, one very careful way where it doesn't freeze with just the right sequence of instructions... but then it still doesn't repro the behavior you've been seeing with 32b->128b, instead the unaligned 32b write just get executed normally. eigenform is working on trying to refactor and minimize the working unaligned write instruction sequence, then we want to try it on MEM2 to see if the behavior differs there (marcan claims it should?).

This is a rabbit hole of wtf, and with 4 experienced Wii reverse engineers / emulator developers / PPC experts brainstorming this for now almost 6h we've yet to even figure out a theory as to what's going on.

Alright, I think GCC threw delroth, phire and I for a loop: it was coalescing stores to avoid the misaligned one. :^(
Avoiding all compiler magic, misaligned accesses to the uncacheable MEM1 mapping seem to consistently crash in my tests built with libogc.
Not clear what's happening yet. Have tests and notes at https://github.com/eigenform/broadway-misaligned-accs

My current unproven theory is the following:

1. Broadway supports uncached, unaligned writes perfectly fine (documented, but not proven)
2. Hollywood's implementation of the x60 bus is incomplete and doesn't support unaligned writes.
3. On receiving the two-beat, two part unaligned bus transfer, Hollywood's Processor Interface interprets it as a regular four-beat 32byte cacheline transfer.
4. Hollywood is stuck waiting for the next two beats, which never come.
5. Broadway is stuck waiting for the write compete acknowledgement, which never comes.
6. Deadlock. Broadway is hung.

And on our homebrew testing, this is where we are stuck, any unaligned write end up in a broadway hang.

But Zelda VC has a bunch of other activity happening within hollywood. 3d rendering, efb write-backs, audio rendering, IOS commands etc. My theory is that one of those (probably a write to MEM1) is causing PI and Broadway to exit their deadlock and continue execution.

I'm currently setting up some tooling so I can examine this broadway hang from starlet, which I hope will tell us more about the nature of the hang.

Edit: Wild theory disproved below.

Bad news: now the repro looks pretty much as simple as it could be ("stw rA, 7(rB)"). But when doing any kind of unaligned u32 write, or even aligned u16 writes to +2, we get a Broadway hang which doesn't even look like it triggers any exception (like, it's not a DSI/ISI/Alignment/MCE, it just freezes as far as we can tell). I'm entirely confused by this, because this is not a result that anyone else has been getting. I suspect I'm doing something stupid, but it's very unclear what, and aligned 32b writes do work with that same test harness...

It is possible that I in fact was getting a hang. I was just assuming that it was an exception that wasn't getting handled properly, since getting an actual hang of the CPU is quite strange. However I was only getting hangs when doing unaligned writes to the 0xC0 area (with libogc's default mappings), not the 0x80 area.

But Zelda VC has a bunch of other activity happening within hollywood. 3d rendering, efb write-backs, audio rendering, IOS commands etc. My theory is that one of those (probably a write to MEM1) is causing PI and Broadway to exit their deadlock and continue execution.

I just asked our resident expert on the N64 VC emulator, and he said that the Gfx fifo and likely the frame buffers are both in MEM1. He doesn't think that the GPU would write to the Gfx fifo, but is positive it writes to the frame buffers. So the GPU writing to MEM1 in Zelda 64 VC is highly plausible. I'm under the impression (perhaps incorrectly) that MEM2 is usually used for the GPU in most games, but in N64 VC it's basically only used to store the N64 ROM, meaning that MEM1 is the general purpose RAM for both CPU and GPU.

Ok, turns out the reason why meta's, josjuice's and delroth's reproduction attempts were failing was stupid.

libogc enables the ERROR interrupt from PI for some reason (likely bad copy/paste). It then never handles or clears the interrupt flag, resulting in the interrupt continually triggering.

Thanks to Extrems for pointing this out.

In my mini/ppcskel environment, I've replicated the both the original behaviour of this issue (both DWW and QWW) as described, along with broadway hanging when you enable (but don't handle) ERROR interrupts.

There are no special conditions for replicating the bug, it should be super-easy for someone to actually implement. It was just libogc's faulty exception handling throwing everyone off.

Oh awesome! Glad it was just a compiler issue, and it really is as simple as it sounds (unaligned uncached stw = DWW/QWW with no VC emu funny business). So is the behavior likely due to faulty write-masking between the MEM1 and the 60x bus as delroth suggested in the quote below?

I've been brainstorming this with segher, the current most plausible hypothesis we have is that something in Hollywood is mishandling these write masks on the 60x transactions. The 1T-SRAM is probably not directly on the 60x bus, there's likely some crossbar in Hollywood itself and some 1T interface, and it's possible that at one of these levels the write masks get converted to some other system (e.g. access sizes). If this conversion logic pattern matches "known patterns" that only supports aligned accesses mask patterns, and defaults to full 64b write otherwise, then that might explain this situation.

Also, do we think this behavior would happen on GCN as well? Like I mentioned previously, the MM community thinks it probably does, but it's difficult for us to test since we don't have practice tools for the GCN version of the game. Would anyone be able to run a basic test like this on an actual GCN (without using upstream libogc due to the bug)?

Actually, I can point directly to the patent which show the issue (at least for the gamecube)

https://patents.google.com/patent/US8098255

If you check out figure 11, it shows the exact internal bus layout between the Processor Interface (which terminates the x60 bus) and Memory Controller (which connects to the 1T-SRAM). There are only two bits of mask infomation labelled as pi_mem_msk(1:0), which allow masking just the top 32bit or bottom 32bit of data in this internal 64bit transfer.

The text of the patent confirms this. The QWW behaviour is coming from the fact that they cross the 8 bit boundary, and two of these 64bit transfers are needed.

Also, if you check out figure 6A/6B, the Memory controller is connected to four Memory Access Controllers, each controlling 32bits of memory. It's likely that the memory controller can only mask by disabling a whole Memory Access Controller.

It looks like the Wii has inherited this behaviour (Hollywood is a faster flipper with a few extra things bolted on). I did some testing and looks like the behaviour is the same on both MEM1 and MEM2.

Yes, I'm 99.9% sure this will apply to the gamecube too.

BTW, I also did some quick tests with byte and halfword writes. It seems they simply ignore the size and write the full 32bits of the register, and then the same DWW and QWW behaviour happens.

I'm curious what happens when you do unaligned writes to registers within flipper. I don't think they are going over this internal PI <--> MI bus and the behaviour could be different.

If you do get this working in Dolphin, I request some footage of the glitch in action on Dolphin for the progress report :)

I'm not 100% sure who's doing the word replication behaviour.

I suspect it's actually Boardway, who just fills out the rest of the bus with duplicated bits. From the point of the x60 bus, those bits are undefined and it might have been easier internally to just replicate.

The other option is that PI is replicating the words. Would seem a bit weird, since PI clearly knows that Broadway has messed up and done an unaligned write. It triggers an interrupt to let the code know it messed up, and even has address and cause registers with details of the error (Note to self, Batman: Vengeance is a debug build that contains code to print out these error registers)

But despite notifying Broadway of the error, PI then lets the write pass though to Memory interface. I guess in their quest for performance, ArtX/ATI decided to not add hardware to actually block the request.

I'm excited to see this in action in Dolphin.

I've posted a working hardware test here: https://github.com/dolphin-emu/hwtests/pull/42

Unfortunately I don't understand the pattern behind when PI ERROR interrupts are raised and when they aren't. I was wondering if it was due to the interrupt not happening immediately, but (back when I was running the tests for 32-bit stores before 16-bit and 8-bit stores) I always got an interrupt after the very first misaligned write, and adding an eieio did not appear to change the results whatsoever (though I am not certain if this is sufficient for avoiding race conditions). Because of this, I think I will focus on implementing the QWW/DWW behavior in Dolphin without implementing PI ERROR interrupts.

Wow this is fascinating. So it really is basically just like how I guessed it was in my initial video on the subject. The Broadway just bitshifts the word to be written by 1 to 3 bytes, then tries to copy paste that a couple times, expecting the irrelevant parts will be masked, however, the MI only has the capability to mask entire words (this is the part I couldn't possibly foresee). Without any masking, the full doubleword is written, and in the case of crossing a doubleword boundary, it does these same shenanigans twice.

If you do get this working in Dolphin, I request some footage of the glitch in action on Dolphin for the progress report :)

Consider it done. I would be thrilled to represent yet another N64 community that discovered bizarre hardware behavior via our broken-ass VC games not working the same in Dolphin as on real Wiis (the SM64 floating platform was legendary, amazing work on that).

I'm excited to see this in action in Dolphin.

Me three :)

phire wrote:

I modified JosJuice's hardware test to be standalone for gamecube.

Confirmed that the DWW/QWW hardware behaviour is identical on gamecube. Though that doesn't guarantee it's actually exploitable, as the gamecube version of the N64 emulator might be different enough to not trigger an unaligned uncached write.

That's wonderful to know! The GameCube version of the N64 emulator was written by the same guy as VC was, so I expect it very likely would trigger that same write (barring any issues with the code being dynarec'd more frequently due to the GCN's smaller available memory) [Bounds checking is only done during dynarec phase, but if the code is already dynarec'd it will just let you write wherever you want.]

So it really is basically just like how I guessed it was in my initial video on the subject. The Broadway just bitshifts the word to be written by 1 to 3 bytes, then tries to copy paste that a couple times, expecting the irrelevant parts will be masked, however, the MI only has the capability to mask entire words (this is the part I couldn't possibly foresee). Without any masking, the full doubleword is written, and in the case of crossing a doubleword boundary, it does these same shenanigans twice.

The main other things I would personally add to the explanation:

• Processor Interface totally knows about unaligned, uncached accesses and will actually raise an error exception
• But release builds of the official gamecube/wii SDKs mask these exceptions, ignoring them.
• Debug builds will print out an error message with the error type and address.
• The doubleword boundary crossing behaviour is a fully documented and defined feature of Broadway.
• Despite erroring, PI will totally pass on an undefined write. It seems whenever there is any kind of error, it just throws open both mask bits.

That's any kind of error. Even a byte or half write on an aligned word boundary will cause PI to throw open both mask bits.

It's possible opening both mask bits is a hardware bug, that they ARTX/ATI actually intended to close both mask bits to suppress the write. Or MI doesn't understand a masked write and flips back to writing the entire dword.

It's also possible that ARTX/ATI decided to do this for performance reasons. That the error exception is intended to crash the program immediately and there is no expiation that the program could or should continue. Supporting this argument, the Memory Controller supports "memory protection exceptions" which are documented to act in a similar way, throwing an exception but allowing the read/write to continue anyway.

FWIW, the error logic being distinct from the actual interface logic is not uncommon. On the Apple M1, if you use <32bit writes to the UART peripheral, it throws an SError, but it works anyway (in particular the TX/RX registers are only 8 bits wide so it's fine for those).

At the point where you're throwing around bad memory ops, blocking them doesn't really matter much. The error is supposed to tell you that something is definitely wrong so you can fix it and make it not happen. That said, throwing open both mask bits seems like a poor default in this particular case.

phire wrote:

I'm not 100% sure who's doing the word replication behaviour.

I suspect it's actually Boardway, who just fills out the rest of the bus with duplicated bits. From the point of the x60 bus, those bits are undefined and it might have been easier internally to just replicate.

("60x bus"... it means 601, 603, 604).

The other option is that PI is replicating the words. Would seem a bit weird, since PI clearly knows that Broadway has messed up and done an unaligned write. It triggers an interrupt to let the code know it messed up, and even has address and cause registers with details of the error (Note to self, Batman: Vengeance is a debug build that contains code to print out these error registers)

Yes, it almost certainly is the 750, because it has to drive something
there, and just the rotated data is the simplest thing to do.

Btw, unaligned accesses on the 60x bus are perfectly fine... it is the
flipper / hwood that does not support this.

But despite notifying Broadway of the error, PI then lets the write pass though to Memory interface. I guess in their quest for performance, ArtX/ATI decided to not add hardware to actually block the request.

This is perfectly fine if this is meant as a debug feature (instead of as
something that will be used during normal operation).

Nice work everyone :-)

The other LNSRM researching for the MM Community, Türkenheimer, just uploaded a video showing off some strange behavior he has encountered from uncached loads: https://youtu.be/3mIyHJ47IvA

Our community doesn't understand this behavior nearly as well, but it appears as though uncached loads are causing some sort of zero-fill behavior (the place he keeps warping to is Scene 0, Spawn 0, with Params 0, aka a 32 bit word of all zeros. Türk claims that alignment doesn't matter in the case of the uncached loads.

Admittedly, I understand the loads behavior far less than the stores behavior because I have personally worked on routes that encounter the stores behavior, but he is really the only one who has worked on a route that encounters the loads behavior. More investigation is obviously needed. I'm not even sure at this point whether Dolphin does or does not emulate the loads correctly. I just wanted to throw it out there though, because it likely is related.

In the meantime though, I would be perfectly satisfied if only the unaligned uncached store behavior were implemented in Dolphin, as that's the behavior that is actually useful for speedrunning. The uncached loads behavior Türk describes is actually a mild inconvenience.

I might have distracted JosJuice from fixing this earlier by tricking them into fixing a different issue first ;)

It's really interesting seeing all of these uncached things. It's nice having a REAL case to look at that uses them, rather than just hardware tests.

If this odd behavior with loads happen even with aligned 32-bit loads, and the behavior does not happen in Dolphin, it's likely that it's the old problem of the CPU's data cache containing stale data. Dolphin does not emulate the CPU's data cache correctly because it would make the performance much worse.

In case you're not familiar with the problem: Let's say that you have some piece of memory at 0x80001000/0xC0001000 which starts out having the value 0. At some point, the CPU reads from 0x80001000 and retrieves the value 0. This has a side effect of placing an entry into the CPU's data cache that says that the memory at 0x80001000 contains the value 0 (and also further entries for some additional memory adjacent to 0x80001000, in case the program will want to access that later). Then at some later point, you write the value 1 to 0xC0001000. This causes the value in memory to change to 1, but if that entry in the data cache is still there, that entry does not get updated, because the write was not to 0x80001000. If you then try to read from 0x80001000 and the entry is still in the data cache, the CPU will find the entry in the data cache and return the value 0, skipping accessing main memory to save time, even though the value in main memory is now actually 1.

The intended way for game developers to avoid this kind of issue is to use the instruction dcbi to get rid of the entry from the data cache at some point. This is most likely not feasible in a speedrun setting unless you have ACE. The less reliable alternative would be to make the CPU access so many other addresses in between the first 0x80001000 read and the second 0x80001000 read that the data cache fills up and evicts the entry for 0x80001000. Though, I suppose it's likely that you don't have much control of this either in a speedrun setting.

Dolphin basically acts as if the data cache contains nothing, but while giving the emulated game the same performance as if the data cache contained every part of memory.

Sorry, for some reason when coming up with the example scenario above I was thinking of an uncached store instead of an uncached load even though you had specified otherwise. There is an equivalent problem with uncached loads, though.

Let's once again have 0x80001000/0xC0001000 in memory contain the value 0 to begin with. Now the CPU writes the value 1 to 0x80001000. This does not immediately cause the value in memory to update – rather, an entry gets added to the data cache that says that the value 1 is at 0x80001000. If the CPU now were to read from 0xC0001000, it would get the value 0 back because the value written to the cache has not been written back to main memory yet.

This too can be avoided by either using dcbi or ensuring that enough other addresses have entered the data cache in between the write and the read.

I got a second opinion on this, and we think the uncached loads behavior is due to the VC emulator. We think it probably is just that C0/E0 is not a valid memory domain on the N64, so if the code isn't already dynarec'd, it will bounds check that load and realize it's invalid and replace with zeros. If it is already dynarec'd, it should work normally, since it will bypass the bounds check.

So the unaligned uncached stores behavior is likely separate. That's a hardware "feature", whereas the loads behavior is a VC emu "feature".

Though it does seem to always not be dynarec'd in Dolphin (uncached loads always invalid -> zero filled), whereas on a real Wii it's variable. That could be due to any number of things though. No cache emulation, using JIT64, who knows. Definitely not worth chasing down. We'll just avoid those uncached loads since they only cause us grief.

I've figured out why the interrupts weren't behaving as expected in my hardware test. Before each test, I was writing zeroes to my buffer to get rid of what the previous test wrote. However, I was doing so using std::fill with a volatile u8* pointer, and this was triggering the DWW behavior. Changing the pointer type to volatile u32* fixed the problem, and now I'm getting a very consistent pattern: There is an interrupt after every write to uncached memory, even ones that are 32-bit and aligned. But in my original iteration of the hardware test (before we figured out what was going on with the interrupts), I can't recall 32-bit aligned writes ever causing hangs...

JosJuice wrote:

and now I'm getting a very consistent pattern: There is an interrupt after every write to uncached memory, even ones that are 32-bit and aligned.

I can't replicate this on my setup. No interrupts on 32bit aligned, uncached writes. Interrupts for everything else.

Just ran through the entire credits warp setup (QWW), start to finish, and yep, it works on the new build. I tested DWW as well and that seems to work as expected too. I also got someone else to confirm DWW/QWW is working for them.

Would it be possible for us to get a video in a public place that we could link to in the progress report following this getting merged?

How much performance would it cost to enable this feature by default?

If the cost is low (less than 1% in most games) it might be worth enabling it by default. Do games even do that many uncached writes by default?

Like phire wrote in the PR, I too am curious what the performance cost really is. I highly doubt many games at all do unaligned uncached writes, because if they did, they'd be overwriting so much memory they didn't intend to due to DWW/QWW. That sounds like a recipe for crashing or glitchy behavior.

On the flip side, maybe there were a small handful of games that were aware of this behavior and exploited it to their advantage, and so some previous glitches caused by running in Dolphin could be fixed? (complete speculation)

JMC4789 wrote:

Would it be possible for us to get a video in a public place that we could link to in the progress report following this getting merged?

Yes, absolutely. I just wanted to do an offline test first. I'm gonna have to think how I want to make that video though, because doing such a precise setup on a controller on Dolphin without savestates (which I made heavy use of in my offline test) is a bit difficult. Maybe I'll use keyboard for the first part.

You could use movie recording to essentially "TAS" it.

Oh I just realized that Dolphin has a "virtual notches" feature. That's probably gonna fix the issue I was having doing the setup entirely on controller with no savestates. So I'll probably just do it live in that case :)

Rylie wrote:

I highly doubt many games at all do unaligned uncached writes, because if they did, they'd be overwriting so much memory they didn't intend to due to DWW/QWW. That sounds like a recipe for crashing or glitchy behavior.

The problem is we take the performance hit on both unaligned aligned uncached writes, we can't tell them apart ahead of time.

Though I do suspect the number of games doing many aligned uncached writes is low. It's a large performance hit on real hardware to not use cached writes (or something else, like the write gather pipe or locked L1 cache DMA)

On the flip side, maybe there were a small handful of games that were aware of this behavior and exploited it to their advantage, and so some previous glitches caused by running in Dolphin could be fixed?

That's my main motivation to wanting it on by default if the performance hit is low.

The problem is we take the performance hit on both unaligned aligned uncached writes, we can't tell them apart ahead of time.

Can we not slowmem the uncached memory space?

Here's the video :)

https://youtu.be/ayyyDKCGXqw

Rylie wrote:

You can't do a % 4 once you see it's uncached? I mean sure that would hurt performance for aligned uncached writes, but less than slowmem would I assume? (I don't really understand the difference between fastmem and slowmem).

At the point we see it's uncached, we are already in the slowmem handler and have taken a page fault. (Unless we introduce an explicit check in the fastmem handler for whether memory is uncached, but then we would be hurting performance for all stores, so that's a no-go.) I suppose it isn't impossible to add a % 4 in the slowmem handler and then based on the result either do a simple store or branch to Write_U*, but now we have the problem that this simple store inside the slowmem handler in itself may trigger a page fault and have to go through slowmem again, which... Well, it's probably possible to solve it somehow, but it's a ton of complexity for very little gain. So I don't think it's a workable idea.

delroth wrote:

The problem is we take the performance hit on both unaligned aligned uncached writes, we can't tell them apart ahead of time.

Can we not slowmem the uncached memory space?

That is what my pull request does. I think phire's comment was missing an "and" – what he meant is that all uncached writes take a performance hit, regardless of whether they are aligned. And that is precisely because we're not allowing fastmem to be used with the uncached memory space.