Emulator

If you're the one who can reproduce the issue, bisect it to an exact build. Asking devs who have never played the game to go after a bug that takes an indeterminate amount of time is a good way to get the issue back burnered. If we have an exact build where the issue started, we can potentially provide more information on what to do and ways to debug it.

Also, are you using single core or dual core?

If you can get a process dump of Dolphin in the frozen state (Task Manager -> Details -> Dolphin.exe -> Create dump file), that is also likely to be helpful.

e: Wait, is the emulated game freezing, or the whole Dolphin process? If it's just the emulated game that might not be too useful...

Based on what I remember from the early HLE BBA testing, this is the emulated game freezing.

If it's the emulated game freezing a savestate may be useful?

I can try getting a dump file and save state during a freeze. Bisecting will continue, I narrowed down to ~11 PRs or so.

Great :)

If it is one of the more complicated BBA HLE pull requests, it's possible we can narrow it down to the exact commit through compiling the individual commits of the pull request too.

Using single core

Got a freeze on 17260, process dump is here:
https://drive.google.com/file/d/1MCSvBKRD8YJV-vQrG4PT5KmyMBXOaIuj/view?usp=sharing

Was unable to get a save state or FIFO, this crashed the entire emulator.

Will try 17253 next.

I'll start testing PR 11083 to confirm it fixes the bug. I'm not sure further bisection will be productive.

From my bisection testing:
17245 (20 hrs) and 17253 (27 hrs) appear stable.
17260 (15 hrs) and 17271 (10 hrs) have produced freezes.

The two builds in between 17253 and 17260 don't seem like they changed something that would cause this sporatic freezing issue.

Well it's impossible to prove that a sporadic bug doesn't exist in any given revision, you can only prove that it does exist. So I'm guessing that 17253 has the bug and you just got lucky, and that the bug was introduced in either 17249 or 17251 (because those builds touch network code).

I got another freeze, I'd been back on the latest dev build (post-latest BBA fix) (172445). save state and memory dump are here:
https://drive.google.com/file/d/1ducyt2UG0IhNemlFFMux6pzPUOiW7eOk/view?usp=sharing

If this issue didn't occur on the PRs I touched the network code, it could mean that it was pure luck or that it was already there before. It might be some situational network issues as well.

notaloop

1. Do you know if the game produces interesting network log messages or maybe debug OSReport messages?
2. Do you know if a crash happens when you have network issue?
3. What happen when you're online and disconnect from your Internet access point (by disconnecting Wi-Fi or removing the Ethernet cable)? Does it freeze/crash?
4. What happen when you're online, disconnect and reconnect but to a different access point (i.e. switch from ethernet to Wifi, or Wifi to 3g/4g hotspot)?

BTW, I can't access your files from your Google drive link.

sepalani I’ve updated sharing on the files. You should have access now.

1. Where would I look for this? Currently the game basically freezes with no error messages.

2. I can specifically check this next time. My network is usually reliable . The 17260 crash was on Wi-Fi, 172445 crash on Ethernet. Internet is fiber, 600mbs and I have all modern hardware (<3 years old).

3. I can try this and get back to you.

4. I can also try this. Start on Wi-Fi then plug in the Ethernet cable.

This look likes some kind of memory corruption which results in a crash. Do you know if the bug happens with the build 5.0-17249 and 5.0-17245? I see that you're using a modded game so do you know if such crash can happen on real hardware? Do this bug occur during relatively short sessions (1~2 hours)?

I'm suspecting that my PRs allow more packets to be handled and are increasing the sequence number more quickly. Then a bug I haven't identified is occurring. My assumption might be incorrect but that's the only think I can see that might make the issue occur more frequently.

I put like 20 hours into 17245 and was not able to get the crash. I can test it some more and report back.

People can also crash on real hardware, but its generally regarded as a lot more stable than using Dolphin. People play on hardmodded gamecubes, or softmodded wii's to patch their iso. Default behavior on improper disconnect or crash is to delete all unequipped items, so people patch that out.

Can't say if its more common on short or long sessions. The 17253 crash I shared above was 35 mins into a session. Its kinda random? Dolphin definitely loads quests (<100 kb) faster than real hardware now, though the crashes tend to happen during a quest, not during quest loading.

Testing https://dolp.in/v5.0-16838 is a good idea. If that version is working properly then https://dolp.in/v5.0-10871 might be the PR causing it. However, if the original PR still has the issue, you might want to try with the other BBA backend to see if the issue is still happening.

If it still is, then the issue might not be strictly network related. It could be a timing issue or a crash happening on real hardware. Do you happen to notice anything in common for all the crashes you had?

I meant https://dolp.in/v5.0-16871 not 10871.

Thanks for the additional pieces of information and sorry for the late reply. Do you recall a Dolphin build where you never had this issue happening? Can it occur while using another BBA adapter (using a TAP or Xlink BBA)? A network dump might help if it's network emulation issue, however, it can also be a hardware/EXI emulation issue. AFAICT from the log, the crash is EXI related for sure.

Based on the signature database I have, I was able to find some known symbols (they might be false-positives, though, I doubt it):

Address:      Back Chain    LR Save
0x81295760:   0x812958a8    0x8042aa6c -> ??? (after EXIUnlock)
0x812958a8:   0x81295ba0    0x8043005c -> EXIIntrruptHandler
0x81295ba0:   0x81295bc8    0x80372a00 -> __OSDispatchInterrupt
0x81295bc8:   0x81295bd0    0x8042f600 -> ExiDma (after OSRestoreInterrupts)
0x81295bd0:   0x81295c10    0x8042f5c4 -> ExiDma (after __OSUnmaskInterrupts)
0x81295c10:   0x81295c28    0x8042b424
0x81295c28:   0x81295c30    0x8042b360
0x81295c30:   0x81295c50    0x80425d60
0x81295c50:   0x81295c58    0x803f9ce0
0x81295c58:   0x00000000    0x803755b8

When I'll have some more time, I'll try to investigate it in more details. It's unfortunate that there are no easy way to replicate this issue. Regarding testing, you should definitely use the stock ROM since the logs aren't clobbered and your 16290 build or the original build (https://dolphin-emu.org/download/dev/master/5.0-16838/) should be fine.