Emulator

If you're the one who can reproduce the issue, bisect it to an exact build. Asking devs who have never played the game to go after a bug that takes an indeterminate amount of time is a good way to get the issue back burnered. If we have an exact build where the issue started, we can potentially provide more information on what to do and ways to debug it.

Also, are you using single core or dual core?

If you can get a process dump of Dolphin in the frozen state (Task Manager -> Details -> Dolphin.exe -> Create dump file), that is also likely to be helpful.

e: Wait, is the emulated game freezing, or the whole Dolphin process? If it's just the emulated game that might not be too useful...

Based on what I remember from the early HLE BBA testing, this is the emulated game freezing.

If it's the emulated game freezing a savestate may be useful?

I can try getting a dump file and save state during a freeze. Bisecting will continue, I narrowed down to ~11 PRs or so.

Great :)

If it is one of the more complicated BBA HLE pull requests, it's possible we can narrow it down to the exact commit through compiling the individual commits of the pull request too.

Using single core

Got a freeze on 17260, process dump is here:
https://drive.google.com/file/d/1MCSvBKRD8YJV-vQrG4PT5KmyMBXOaIuj/view?usp=sharing

Was unable to get a save state or FIFO, this crashed the entire emulator.

Will try 17253 next.

I'll start testing PR 11083 to confirm it fixes the bug. I'm not sure further bisection will be productive.

From my bisection testing:
17245 (20 hrs) and 17253 (27 hrs) appear stable.
17260 (15 hrs) and 17271 (10 hrs) have produced freezes.

The two builds in between 17253 and 17260 don't seem like they changed something that would cause this sporatic freezing issue.

Well it's impossible to prove that a sporadic bug doesn't exist in any given revision, you can only prove that it does exist. So I'm guessing that 17253 has the bug and you just got lucky, and that the bug was introduced in either 17249 or 17251 (because those builds touch network code).

I got another freeze, I'd been back on the latest dev build (post-latest BBA fix) (172445). save state and memory dump are here:
https://drive.google.com/file/d/1ducyt2UG0IhNemlFFMux6pzPUOiW7eOk/view?usp=sharing

If this issue didn't occur on the PRs I touched the network code, it could mean that it was pure luck or that it was already there before. It might be some situational network issues as well.

@notaloop

1. Do you know if the game produces interesting network log messages or maybe debug OSReport messages?
2. Do you know if a crash happens when you have network issue?
3. What happen when you're online and disconnect from your Internet access point (by disconnecting Wi-Fi or removing the Ethernet cable)? Does it freeze/crash?
4. What happen when you're online, disconnect and reconnect but to a different access point (i.e. switch from ethernet to Wifi, or Wifi to 3g/4g hotspot)?

BTW, I can't access your files from your Google drive link.

@sepalani I’ve updated sharing on the files. You should have access now.

1. Where would I look for this? Currently the game basically freezes with no error messages.

2. I can specifically check this next time. My network is usually reliable . The 17260 crash was on Wi-Fi, 172445 crash on Ethernet. Internet is fiber, 600mbs and I have all modern hardware (<3 years old).

3. I can try this and get back to you.

4. I can also try this. Start on Wi-Fi then plug in the Ethernet cable.

This look likes some kind of memory corruption which results in a crash. Do you know if the bug happens with the build 5.0-17249 and 5.0-17245? I see that you're using a modded game so do you know if such crash can happen on real hardware? Do this bug occur during relatively short sessions (1~2 hours)?

I'm suspecting that my PRs allow more packets to be handled and are increasing the sequence number more quickly. Then a bug I haven't identified is occurring. My assumption might be incorrect but that's the only think I can see that might make the issue occur more frequently.

I put like 20 hours into 17245 and was not able to get the crash. I can test it some more and report back.

People can also crash on real hardware, but its generally regarded as a lot more stable than using Dolphin. People play on hardmodded gamecubes, or softmodded wii's to patch their iso. Default behavior on improper disconnect or crash is to delete all unequipped items, so people patch that out.

Can't say if its more common on short or long sessions. The 17253 crash I shared above was 35 mins into a session. Its kinda random? Dolphin definitely loads quests (<100 kb) faster than real hardware now, though the crashes tend to happen during a quest, not during quest loading.

Testing https://dolp.in/v5.0-16838 is a good idea. If that version is working properly then https://dolp.in/v5.0-10871 might be the PR causing it. However, if the original PR still has the issue, you might want to try with the other BBA backend to see if the issue is still happening.

If it still is, then the issue might not be strictly network related. It could be a timing issue or a crash happening on real hardware. Do you happen to notice anything in common for all the crashes you had?

I meant https://dolp.in/v5.0-16871 not 10871.

Thanks for the additional pieces of information and sorry for the late reply. Do you recall a Dolphin build where you never had this issue happening? Can it occur while using another BBA adapter (using a TAP or Xlink BBA)? A network dump might help if it's network emulation issue, however, it can also be a hardware/EXI emulation issue. AFAICT from the log, the crash is EXI related for sure.

Based on the signature database I have, I was able to find some known symbols (they might be false-positives, though, I doubt it):

Address:      Back Chain    LR Save
0x81295760:   0x812958a8    0x8042aa6c -> ??? (after EXIUnlock)
0x812958a8:   0x81295ba0    0x8043005c -> EXIIntrruptHandler
0x81295ba0:   0x81295bc8    0x80372a00 -> __OSDispatchInterrupt
0x81295bc8:   0x81295bd0    0x8042f600 -> ExiDma (after OSRestoreInterrupts)
0x81295bd0:   0x81295c10    0x8042f5c4 -> ExiDma (after __OSUnmaskInterrupts)
0x81295c10:   0x81295c28    0x8042b424
0x81295c28:   0x81295c30    0x8042b360
0x81295c30:   0x81295c50    0x80425d60
0x81295c50:   0x81295c58    0x803f9ce0
0x81295c58:   0x00000000    0x803755b8

When I'll have some more time, I'll try to investigate it in more details. It's unfortunate that there are no easy way to replicate this issue. Regarding testing, you should definitely use the stock ROM since the logs aren't clobbered and your 16290 build or the original build (https://dolphin-emu.org/download/dev/master/5.0-16838/) should be fine.

I too have encountered this issue multiple times while playing online using the HLE BBA (haven't tested with TAP or XLink BBA). Within about an hour of gameplay (but often sooner), Dolphin suddenly freezes while emitting a buzzing noise that doesn't stop until you force-close the emulator. One difference in my case: I do not get the in-game message "The line was disconnected" as seen in screenshots above, as the entire emulator freezes. On restart, you re-enter the game to find that all of your character's unequipped items are gone (apparently the freeze somehow corrupts the memory card save). I imagine this can be quite serious if the character has items from thousands of hours of gameplay, though in my case it has only been a few.

@sepalani As @notaloop may have moved on, please let me know if I can assist further. I have not tested any other builds or BBAs against this issue but I can try if you still think it can help. I'm surprised others have not encountered this issue as well, as quite a few people play this game online using Dolphin.

Test system:

• OS: Linux Mint 21.2 / kernel 6.2
• CPU: Intel i7-10870H
• GPU: Nvidia GTX 3070 / driver 535.129
• Build tested: 5.0-20290

Interesting to see that this issue also occurs on Linux.

@shoegazer
Do you mind testing the TAP and XLink BBA implementations to see if the issue is also occurring with them?

I hadn't realized this until I read further, but I don't currently have the means to test either of those BBA implementations unfortunately. I'm happy to provide any other information you may need though. If there are any differences with Dolphin logging facilities in Linux that might provide further clues just let me know and I'll get them for you.

Also on a related note for other testers: Dolphin's default IP address for the Schthack server, as found in the HLE BBA settings, is outdated and no longer works. The new address is 3.18.217.27, and is what I've been using in my tests.

Are you able to check if the issue also happens on a real Wii with both Nintendont and Devolution?

I managed to trigger that packet corruption multiple times, afaict, it's a concurrency issue regarding the packet logging and shouldn't impact the emulation: https://github.com/dolphin-emu/dolphin/pull/12304

So it's most likely unrelated to the issue. I did notice some TCP window warnings on Wireshark, I'll have a look at it.

My apologies for missing this earlier - for some reason I'm not getting notifications of changes to the thread even though it's on in my profile. They're not in junk either. Ah well, I'll try to check more frequently.

My GC hasn't worked in nearly two years now sadly, though perhaps someone else with a BBA-equipped GC can help to confirm.

Interesting packet analysis, though it's too bad it turned out to be a red herring. This is such a tricky issue to catch in the game itself since there's no apparent way to trigger it; and because you can play for many hours without it happening, and then it can happen multiple times within an hour. I hope there's some way to trap the condition outside the game as you've attempted. At least it seems you're getting closer, and hopefully the TCP window warnings provide some clues. Thanks for working through this.

As a side-note, my PR to update the Schthack server IP was merged, so my above post about that is no longer relevant now for anyone testing.