Emulator Issues #13032
openUnrecoverable game freezes on Phantasy Star Online
0%
Description
Game Name?
Phantasy Star Online I & II Plus
Game ID? (right click the game in the game list, Properties, Info tab)
GPOE8P
MD5 Hash? (right click the game in the game list, Properties, Verify tab, Verify Integrity button)
36a7f90ad904975b745df9294a06baea
What's the problem? Describe what went wrong.
When playing online the game will have an unrecoverable freeze, requiring a restart. Progress since the last save is lost.
What steps will reproduce the problem?
There isn't a consistent way of triggering them. Generally my frequency of freezing has been daily while keeping up with the latest dev version.
Is the issue present in the latest development version? For future reference, please also write down the version number of the latest development version.
17271 is the latest tested with instability.
Is the issue present in the latest stable version?
Latest stable version is 17245, 20+ hours of gameplay without freezing.
If the issue isn't present in the latest stable version, which is the first broken version? (You can find the first broken version by bisecting. Windows users can use the tool https://forums.dolphin-emu.org/Thread-green-notice-development-thread-unofficial-dolphin-bisection-tool-for-finding-broken-builds and anyone who is building Dolphin on their own can use git bisect.)
Somewhere between 17245 and 17271.
If your issue is a graphical issue, please attach screenshots and record a three frame fifolog of the issue if possible. Screenshots showing what it is supposed to look like from either console or older builds of Dolphin will help too. For more information on how to use the fifoplayer, please check here: https://wiki.dolphin-emu.org/index.php?title=FifoPlayer
[Attach any fifologs if possible, write a description of fifologs and screenshots here to assist people unfamiliar with the game.]
What are your PC specifications? (CPU, GPU, Operating System, more)
Windows 10 Pro x64
Ryzen R5 3600
RTX 3060 Ti
16 GB RAM
512 GB SSD
Using GC adapter and GC controller
Is there anything else that can help developers narrow down the issue? (e.g. logs, screenshots,
configuration files, savefiles, savestates)
These freezes are usually suspected of being caused by network changes. PRs 10985/10920 may have changed network behavior in a way that increases freeze frequency. If back-to-back testing w/logging is possible, that might help confirm these PR are impacting network behavior on PSO.
Alternatively, re-testing 17271 but w/o those two PRs.
Files
Updated by JMC4789 7 months ago
If you're the one who can reproduce the issue, bisect it to an exact build. Asking devs who have never played the game to go after a bug that takes an indeterminate amount of time is a good way to get the issue back burnered. If we have an exact build where the issue started, we can potentially provide more information on what to do and ways to debug it.
Updated by AdmiralCurtiss 7 months ago
If you can get a process dump of Dolphin in the frozen state (Task Manager -> Details -> Dolphin.exe -> Create dump file), that is also likely to be helpful.
e: Wait, is the emulated game freezing, or the whole Dolphin process? If it's just the emulated game that might not be too useful...
Updated by AdmiralCurtiss 7 months ago
If it's the emulated game freezing a savestate may be useful?
Updated by notaloop 6 months ago
Got a freeze on 17260, process dump is here:
https://drive.google.com/file/d/1MCSvBKRD8YJV-vQrG4PT5KmyMBXOaIuj/view?usp=sharing
Was unable to get a save state or FIFO, this crashed the entire emulator.
Will try 17253 next.
Updated by notaloop 6 months ago
I'll start testing PR 11083 to confirm it fixes the bug. I'm not sure further bisection will be productive.
From my bisection testing:
17245 (20 hrs) and 17253 (27 hrs) appear stable.
17260 (15 hrs) and 17271 (10 hrs) have produced freezes.
The two builds in between 17253 and 17260 don't seem like they changed something that would cause this sporatic freezing issue.
Updated by AdmiralCurtiss 6 months ago
Well it's impossible to prove that a sporadic bug doesn't exist in any given revision, you can only prove that it does exist. So I'm guessing that 17253 has the bug and you just got lucky, and that the bug was introduced in either 17249 or 17251 (because those builds touch network code).
Updated by notaloop 6 months ago
I got another freeze, I'd been back on the latest dev build (post-latest BBA fix) (172445). save state and memory dump are here:
https://drive.google.com/file/d/1ducyt2UG0IhNemlFFMux6pzPUOiW7eOk/view?usp=sharing
Updated by sepalani 6 months ago
If this issue didn't occur on the PRs I touched the network code, it could mean that it was pure luck or that it was already there before. It might be some situational network issues as well.
- Do you know if the game produces interesting network log messages or maybe debug OSReport messages?
- Do you know if a crash happens when you have network issue?
- What happen when you're online and disconnect from your Internet access point (by disconnecting Wi-Fi or removing the Ethernet cable)? Does it freeze/crash?
- What happen when you're online, disconnect and reconnect but to a different access point (i.e. switch from ethernet to Wifi, or Wifi to 3g/4g hotspot)?
BTW, I can't access your files from your Google drive link.
Updated by notaloop 6 months ago
sepalani I’ve updated sharing on the files. You should have access now.
Where would I look for this? Currently the game basically freezes with no error messages.
I can specifically check this next time. My network is usually reliable . The 17260 crash was on Wi-Fi, 172445 crash on Ethernet. Internet is fiber, 600mbs and I have all modern hardware (<3 years old).
I can try this and get back to you.
I can also try this. Start on Wi-Fi then plug in the Ethernet cable.
Updated by notaloop 6 months ago
- File GPOE8P_2022-09-29_19-08-45.png GPOE8P_2022-09-29_19-08-45.png added
- File GPOE8P_2022-09-29_19-12-50.png GPOE8P_2022-09-29_19-12-50.png added
So after some testing, the outcome of 3 and 4 is the same. The game throws a 100 error pretty immediately. The game stops accepting movement/action inputs, but does not freeze. Attached a couple of sample screenshots.
Updated by notaloop 6 months ago
- File 17253 freeze log.txt 17253 freeze log.txt added
I managed to get a freeze on 17253, link to save state, log, and memory dump is here:
https://drive.google.com/file/d/15iy0yY6gwO0F3TnNyEKXP351ORCGrxap/view?usp=sharing
I'm also attaching a copy of the log here, for brevity.
Updated by sepalani 5 months ago
This look likes some kind of memory corruption which results in a crash. Do you know if the bug happens with the build 5.0-17249 and 5.0-17245? I see that you're using a modded game so do you know if such crash can happen on real hardware? Do this bug occur during relatively short sessions (1~2 hours)?
I'm suspecting that my PRs allow more packets to be handled and are increasing the sequence number more quickly. Then a bug I haven't identified is occurring. My assumption might be incorrect but that's the only think I can see that might make the issue occur more frequently.
Updated by notaloop 5 months ago
I put like 20 hours into 17245 and was not able to get the crash. I can test it some more and report back.
People can also crash on real hardware, but its generally regarded as a lot more stable than using Dolphin. People play on hardmodded gamecubes, or softmodded wii's to patch their iso. Default behavior on improper disconnect or crash is to delete all unequipped items, so people patch that out.
Can't say if its more common on short or long sessions. The 17253 crash I shared above was 35 mins into a session. Its kinda random? Dolphin definitely loads quests (<100 kb) faster than real hardware now, though the crashes tend to happen during a quest, not during quest loading.
Updated by notaloop 5 months ago
- File 17245 log.txt 17245 log.txt added
Well that didn't take long.
17245 save state, memory dump, and log here. Log duplicated + attached here. Playtime was around 30 mins into that session.
https://drive.google.com/file/d/1pUSdx5veCL3C2sfjDlvT1q9Kdgq_Hmc4/view?usp=sharing
So at this point I'm not sure what would be the most insightful thing to test to help figure out what's going on. Is there a particular build you want me to try? 16838 was the first one with the BBA HLE added, so that's as far back as I'd be able to test.
Updated by sepalani 5 months ago
Testing https://dolp.in/v5.0-16838 is a good idea. If that version is working properly then https://dolp.in/v5.0-10871 might be the PR causing it. However, if the original PR still has the issue, you might want to try with the other BBA backend to see if the issue is still happening.
If it still is, then the issue might not be strictly network related. It could be a timing issue or a crash happening on real hardware. Do you happen to notice anything in common for all the crashes you had?
Updated by notaloop 5 months ago
- File 16838 Crash.txt 16838 Crash.txt added
Crashed on 16838, attached link here to crash dump, save state, and error log.
https://drive.google.com/file/d/1eWWlREvShDqksJRTX1SQth7f6-_oQnES/view?usp=sharing
I had started a solo boss rush and was probably <20 mins into that session.
Updated by notaloop 5 months ago
- File Perfect Tapless log.txt Perfect Tapless log.txt added
Dump from a early build of the BBA adapter, from around May 9th. Usual files (dump, log, savestate) are here:
https://drive.google.com/file/d/1rPmS-06Tjd4QWEmcrjVNrYiFdh-fUBS1/view?usp=share_link
Log attached as well. By the way, this one was using D3D11 as the backend. The other crashes are a mix of Vulkan or D3D12.
I've been given another early build from late April that I'll try next.
Updated by notaloop 5 months ago
- File 16290 pre CPU + Patch Rom.txt 16290 pre CPU + Patch Rom.txt added
- File 16290 Pre-CPU and Stock ROM.txt 16290 Pre-CPU and Stock ROM.txt added
Okay so couple more dumps with an even earlier commit of the BBA tapless adapter. Build is 16290, idk the exact commit but it was labelled as " pre-cup usage lower" on 4/29.
I did something a little different here. Once I got it to crash and got the usual dump files, I also tried using the stock ROM and was able to get get a crash with a lot more detail that what is usual seen on the log. Hopefully that's helpful!
16290 with patched ROM
https://drive.google.com/file/d/1Kx1Fi4bbFOKcB6WGT8KQgeSNrTMeUvi4/view?usp=share_link
16290 with stock ROM (with a lot more detail in the log):
https://drive.google.com/file/d/1jVLUXnSaaXwzhJCAQbrjjZs2poQPXM-A/view?usp=share_link
I'll also attach both sets of logs to this.
PS - I also played for about an hour in offline mode with the stock ROM and was not able to get another crash. The two crashes above happened within a few minutes of playing in online mode.
sepalani
I see there's network dump settings under the debugging menu. Would logging/dumping be helpful for figuring this out? What settings and build should I use for that purpose?
Updated by notaloop 5 months ago
- File 16788 log.txt 16788 log.txt added
16788, this is yet another dev build. Log attached.
https://drive.google.com/file/d/16zVgrpB5Vh0na4YYXXs-vllp0flLB_oz/view?usp=share_link
Updated by sepalani 3 months ago
Thanks for the additional pieces of information and sorry for the late reply. Do you recall a Dolphin build where you never had this issue happening? Can it occur while using another BBA adapter (using a TAP or Xlink BBA)? A network dump might help if it's network emulation issue, however, it can also be a hardware/EXI emulation issue. AFAICT from the log, the crash is EXI related for sure.
Based on the signature database I have, I was able to find some known symbols (they might be false-positives, though, I doubt it):
Address: Back Chain LR Save 0x81295760: 0x812958a8 0x8042aa6c -> ??? (after EXIUnlock) 0x812958a8: 0x81295ba0 0x8043005c -> EXIIntrruptHandler 0x81295ba0: 0x81295bc8 0x80372a00 -> __OSDispatchInterrupt 0x81295bc8: 0x81295bd0 0x8042f600 -> ExiDma (after OSRestoreInterrupts) 0x81295bd0: 0x81295c10 0x8042f5c4 -> ExiDma (after __OSUnmaskInterrupts) 0x81295c10: 0x81295c28 0x8042b424 0x81295c28: 0x81295c30 0x8042b360 0x81295c30: 0x81295c50 0x80425d60 0x81295c50: 0x81295c58 0x803f9ce0 0x81295c58: 0x00000000 0x803755b8
When I'll have some more time, I'll try to investigate it in more details. It's unfortunate that there are no easy way to replicate this issue. Regarding testing, you should definitely use the stock ROM since the logs aren't clobbered and your 16290 build or the original build (https://dolphin-emu.org/download/dev/master/5.0-16838/) should be fine.