Project

General

Profile

Emulator Issues #10129

Resident Evil 0 GBZP08 crashes at certain events

Added by Maverick_Hunter_X over 2 years ago. Updated 6 months ago.

Status:
Fixed
Priority:
High
Assignee:
% Done:

0%

Operating system:
N/A
Issue type:
Bug
Milestone:
Current
Regression:
Yes
Relates to usability:
No
Relates to performance:
No
Easy:
No
Relates to maintainability:
No
Regression start:
Fixed in:
5.0-10039

Description

Game Name?

Resident Evil 0

Game ID? (right click the game in the game list, properties, info tab)

GBZP08

MD5 Hash? (right click the game in the game list, properties, info tab, MD5 Hash: Compute)

5ea73a9328c549c74c4feaf68abae7bc (verified good dump http://redump.org/disc/4659/)

What's the problem? Describe what went wrong.

When you're still on the train in the beginning of the game and you have the manhole opener and attempt to use it with either Billy or Rebecca on the hatch in the kitchen area, Dolphin (5.0-2535 tested at the time) crashes.
If I however use an older build of Dolphin (5.0-895) it works and the cut-scene triggers correctly.
There is also a crash later on in the Training Facility where you need to operate the small elevator with a crank and this is all I have tested before I ended up completing the game with 5.0-895

What steps will reproduce the problem?

  1. Start a new game
  2. Play til you have to use the manhole opener in order to proceed with the game (takes around 10-15 minutes to get there)
  3. Dolphin crashes

Which versions of Dolphin did you test on? Does using an older version of Dolphin solve your issue? If yes, which versions of Dolphin used to work?

The newest version I could test this on were 5.0-2535 at the time but with 5.0-895 the game works 100% for sure.

What are your PC specifications? (CPU, GPU, Operating System, more)

i7 4790K @ stock
ASUS Nvidia 1060 6GB
16 GB RAM
Windows 10 x64

Is there any other relevant information? (e.g. logs, screenshots,
configuration files)

It crashed without any detailed log information unfortunately.

MemoryCardA.EUR.zip (52.6 KB) MemoryCardA.EUR.zip Maverick_Hunter_X, 03/02/2017 06:38 AM

History

#1 Updated by Maverick_Hunter_X over 2 years ago

Here's a screenshot of the crash and a memory card save attached right before you have to use the manhole opener in the kitchen on the train.

#2 Updated by Maverick_Hunter_X over 2 years ago

I've been able to see what the error message says, turned out I had Panic Handlers set to off.

The first set of errors:

Invalid read from 0x01219138, PC = 0x801f5b80
Invalid read from 0x0121913c, PC = 0x801f5ba4
etc. etc.

Then after those errors this pops up before Dolphin crashes:
BackPatch: no register use entry for address 000000001393BDD6

#3 Updated by Maverick_Hunter_X over 2 years ago

Forgot to mention that I just tested this on 5.0-3019 x64

#4 Updated by Maverick_Hunter_X over 2 years ago

It's a "JIT Recompiler (recommended)" issue!

Just tried "Cached Interpreter (slower)" and "JITIL Recompiler (slow, experimental)" and it worked perfectly fine with no crashes.

So there is definitely something wrong with "JIT Recompiler (recommended)" with the manhole opener sequence on the train and the lift sequence in the Training Facility.

#5 Updated by JMC4789 over 2 years ago

  • Assignee set to degasus

If you can compile builds, can you try playing the fallback game? Try disabling various instructions in the JIT until it starts working, until you figure out which instruction is the issue?

#6 Updated by degasus over 2 years ago

It could also be some kind of timing issue as the recommended JIT is by far the fastest: Can you try in single core, too?

As it is a regression, may you check which build is the first broken one? 1000 builds can be checked by 10 runs by bisecting the range, so this should be quite fast.

#7 Updated by Maverick_Hunter_X over 2 years ago

I just tried Single Core with JIT Recompiler and it crashed unfortunately.
I'm gonna try to find which build started causing this, 5.0-895 works so it shouldn't take too long to find out.

I'll be back in a bit.

#8 Updated by Maverick_Hunter_X over 2 years ago

OK this went fast, first build I tried was 5.0-910 and it crashed with JIT Recompiler.

Between 5.0-895 and 5.0-910 I don't see any JIT related changes at all (from download section https://dolphin-emu.org/download/list/master/36/)

5.0-897 - InputCommon: Fix formatting issues in Quartz (PR #4282 from leoetlino)
5.0-899 - VideoCommon: Minor changes (PR #4280 from shuffle2)
5.0-901 - Fix a few warnings (PR #4283 from leoetlino)
5.0-910 - Add ability to passthrough a Bluetooth adapter (PR #4195 from leoetlino)

I'm gonna try 5.0-897 and 5.0-901 out as they're the only 2 being downloadable between 5.0-895 and 5.0-910

#9 Updated by Maverick_Hunter_X over 2 years ago

I have a new discovery to report, the game also crashed on 5.0-895 but the reason was because I had "Enable Idle Skipping" ON and once I turned it OFF it worked again.

I know "Enable Idle Skipping" is removed in newer versions of Dolphin (at least from the GUI) but turning this option OFF eliminates the crash.

According to this article:
https://dolphin-emu.org/blog/2016/11/01/dolphin-progress-report-october-2016/
You say, and I quote:
"Idle skipping is something you always want on at this stage of Dolphin's development. There are no known downsides, and it makes the emulator run faster by skipping instruction loops that do nothing."

Is it safe to assume, while the option is removed from the GUI, is it still enabled/ON code behind because this is what causing Dolphin to crash at these events.

#10 Updated by Maverick_Hunter_X over 2 years ago

I'm now 99,99% sure this issue is "Enable Idle Skipping" related.

And digging deeper it seems this option is always turned ON according to these pull requests since it were removed from the GUI:

All versions I have tested which still have this option and where you can set it to OFF I experience no crash at all but as soon as it's ON it crash.

I know reinstating this option is out of the question but this is likely the culprit and as far as I can tell of all the games I have and tested, Resident Evil 0 only seems to crash at these certain events.

So in conclusion, Resident Evil 0 doesn't play well with "Enable Idle Skipping"

#11 Updated by Maverick_Hunter_X over 2 years ago

Now I'm 100% sure.

Test base: 5.0-910 before "Enable Idle Skipping" got removed.

1st test case scenario:
"JIT Recompiler (recommended)" selected with "Enable Idle Skipping" ON = CRASH

2nd test case scenario:
"JIT Recompiler (recommended)" selected with "Enable Idle Skipping" OFF = NO CRASH

3rd test case scenario:
"JITIL Recompiler (slow, experimental)" selected with "Enable Idle Skipping" ON = NO CRASH

4th test case scenario:
"JITIL Recompiler (slow, experimental)" selected with "Enable Idle Skipping" OFF = NO CRASH

Since "Enable Idle Skipping" is always ON these days that's why it crashes with "JIT Recompiler (recommended)" selected.

I think this is as far as I can narrow down the problem at this point.

#12 Updated by JMC4789 over 2 years ago

It's some optimization/timing issue in the JIT triggering this.

The key is figuring out what/why it is happening.

#13 Updated by JMC4789 over 2 years ago

  • Status changed from New to Accepted
  • Priority changed from Normal to High

Oh, btw, thank you so much for your hard work narrowing this down.

#14 Updated by Maverick_Hunter_X over 2 years ago

No problem, thanks to you too!

I'm glad I decided to play through this game again so the issue could be caught relatively soon and who knows, if a fix is made it might fix other games too whichever they might be that might have the same issue.

I don't have much time these days to replay all the games in my library and I'm glad if I have the time to complete 10 games a year.

There's a lot of games, a lot of emulators and platforms so I usually just check the basic stuff with 10-15 minute sessions.

Well, anyway I hope you guys can find out what's wrong and if you need more testing just let me know.

#15 Updated by Maverick_Hunter_X about 2 years ago

@JMC4789

How are things coming with this?

JITIL Recompiler is gone in Dolphin these days so there is no way in progressing in the game since it crashes on JIT Recompiler.
It's a real shame because this makes me have to stick with an older build right now.

I just didn't see the point in removing the "Enable Idle Skipping" option really, it's fine with me that you removed D3D12, JITIL etc. but the "Enable Idle Skipping" option is absolutely essential for Resident Evil 0 and maybe a few other games.

Please reconsider in re-instating it since this game is utterly broken if you can't advance in it.

#16 Updated by JosJuice over 1 year ago

  • Milestone set to Current

#17 Updated by zlice over 1 year ago

in "Source/Core/Core/PowerPC/Jit64/Jit_LoadStore.cpp"
around line 130
~~~ c
SafeLoadToReg(gpr.RX(d), gpr.R(a), accessSize, offset, CallerSavedRegistersInUse(), signExtend);
~~~

Seems to be not so safe.
Got rid of this and was able to use the "man hole opener" without a fuss.

Gona look more but I'm not sure if I'll find what's wrong.

#18 Updated by JMC4789 over 1 year ago

  • Assignee changed from degasus to Lioncash

Maybe lioncash can look into this since he's been doing some interpreter testing. I assume this happens on interpreter too?

#19 Updated by zlice over 1 year ago

Seems to be just on JIT.
Tried "Cached Interpreter" and didn't get anything, didn't try plain "Interpreter"

#20 Updated by JMC4789 over 1 year ago

you could fallback individual instructions until you find the broken one if you can compile.

#21 Updated by zlice over 1 year ago

i tried that in SafeLoadToReg() but i'm not sure the problem is there because what i could comment out changes nothing and the other parts seem vital and crash dolphin before games launch

in "Jit_LoadStore.cpp" where i originally pointed out that call is what causes the crash
the condition for the call is

 if (!CPU::IsStepping() && inst.OPCD == 32 && CanMergeNextInstructions(2) &&
      (inst.hex & 0xFFFF0000) == 0x800D0000 &&
      (js.op[1].inst.hex == 0x28000000 || (SConfig::GetInstance().bWii && .....)

CPU::IsStepping is called twice here because CanMergeNextInstructions does that check also
CanMergeNextInstructions only looks like it checks if the next 2 instructions are breakpoints or branches
the other condition here is "OPCD == 32" which is lwz

the only way this code is reached seems to be
if "OPCD == 32" (and it can pass CanMergeNextInstructions)
||
SConfig::GetInstance().bWii (so running in GameCube mode and lwz instruction that does a clear to zero{?})

so, i'm not sure why that's there...or what it's doing exactly that helps with what

is it trying to condense a potential "2 emulated instructions" into "1 emulated instruction" and not do an extra "clear to zero" ?
but it's checking like 5 things and executing extra code to do that?
maybe i'm not emulator savvy
wonder if this snip or code saves other game(s) or is just breaking RE0

#22 Updated by zlice over 1 year ago

took another look after testing for a few
i was wrong partially about the "if OPCD == 32" part
it looks like this has been in here as long as the file has been in git (2013)

here's what i got (if my assembly is correct and there's no endian funkiness)

!CPU::IsStepping() && inst.OPCD == 32 && CanMergeNextInstructions(2) &&

CanMergeNextInstructions redoes IsStepping() but is essentially useless because of the next couple conditions
"OPCD == 32" also seems to be checked twice

 (inst.hex & 0xFFFF0000) == 0x800D0000 &&

checks if the current instruction is lwz (which was already checked)
for source register 13 (0xd)

(js.op[1].inst.hex == 0x28000000 ||
       (SConfig::GetInstance().bWii && js.op[1].inst.hex == 0x2C000000))

checks if the next instruction (CanMergeNextInstruction recheck) is lhz with no args (essentially an 'idle' nop command)
(assuming the bWii portion is the same code on w/e chip wii uses)

js.op[2].inst.hex == 0x4182fff8)

checks the instruction after that (again, CanMergeNextInstruction)
it's a very specific branch operation. seems to be set up from the 2 proceeding ops to make the conditional jump

pg 23 (if this is the same as gecko instructions)
http://math-atlas.sourceforge.net/devel/assembly/ppc_isa.pdf

Branch Conditional B-Form
16 (chops bits from 0x41) (didn't figure out the rest)

there were comments from ector and m2b that were removed somewhere down the line
if they're still around maybe they know what this very specific idle skip is for

it seems too damn specific to be scattered across a bunch of games, but who knows
guess it could also have been a fix for RE0 at some point, too, and now it crashes it

maybe i'm completely wrong about most of this. but i'm commenting out line 122-154 of "Jit_LoadStore.cpp" since i have to use an old build for another bug
hopefully someone else can comment that out and test other games with it

thanks Maverick_Hunter_X for narrowing it down

#23 Updated by JosJuice 8 months ago

Could this be related to issue 11580? I don't know if anyone has tested this issue on JitArm64 or on 4.0.2 (with idle skipping).

#24 Updated by Maverick_Hunter_X 7 months ago

There have been quite some time since I paid attention to this issue, as of today it still happens as of 5.0-9705 which is almost 9000 commits since 5.0-895 the build I use to play this game on.

One question, would it be possible to make "Enable Idle Skipping" as a hidden game ini option?

Also just a little PSA, Resident Evil Archives - Resident Evil 0 for the Wii hangs at the "This game contains scenes of explicit violence and gore" screen no matter what settings you use.

This is a game that also works great on 5.0-895.

#25 Updated by zlice 6 months ago

can confirm

and the worst part is since everything has been updating, my old "patches" don't compile on new software versions =/

#26 Updated by zlice 6 months ago

JosJuice wrote:

Could this be related to issue 11580? I don't know if anyone has tested this issue on JitArm64 or on 4.0.2 (with idle skipping).

i think it is (see #22 where i did the same)

trying latest(git) now and then gona try to mess with the idle skip again .-.

#27 Updated by zlice 6 months ago

yep - just commented out that /if/ section for OPCD == 32 and i can open the damn hole...

#28 Updated by zlice 6 months ago

so i read the https://dolphin-emu.org/blog/2016/11/01/dolphin-progress-report-october-2016/ that maverick posted - which says "need for speed: hot pursuit 2" wont run without idle skip.

found need for speed - and it runs fine with idle skip disabled (commented out in code)

i'm doubting some of the claims in that changelog. maybe with the older rendered there was an issue? (i found the specific commit/date that torched Code Veronica and it was when the backend rendering related changes were made) maybe somewhere else in the interpreter? i'm just throwing out random guesses.

i haven't noticed any performance difference in games. i could test more but i'm on a intel 620 so if it was noticeable i'd think the barely powerful enough iGPU would have major slow down.

#29 Updated by JMC4789 6 months ago

  • Assignee changed from Lioncash to degasus
  • Regression changed from No to Yes

Idle skipping is a huge performance up. But, if there's a change in behavior from turning it on and off, there's a bug. I tried pinging degasus to look into this but he's been busy.

#30 Updated by zlice 6 months ago

it might be for some games (guessing that is why it use to have the enable/disable check box)

i also added a printf in the idle skip block and got maybe 2 or 3 playing RE games (usually right at boot intro or when starting a game), RE0 a few more at the manhole part before it crashes.

#31 Updated by JMC4789 6 months ago

There's a difference between idleskipping and sync on idleskipping.

Again, it's hard to really debug this because the change should effectively do nothing so there isn't much to work from. Hacking around it isn't really an option, we don't want more hacks without understanding the underlying cause of the issue.

#32 Updated by zlice 6 months ago

just did some quick testing

evolution worlds
pso
(already have done with re0/1/2)
skies of arcadia
zelda : wind waker
and...need for speed: hot pursuit 2 (the game that at least used to need idle skip)

with the print in idle skip, there were only the few at boot and load like i said.
all of the games had no difference as far as i could tell...except
need for speed
which was grinding gears with "idle skip" block enabled.

unless the thread was grabbed by something else and not printing to console, the idle skip wasn't really used in 'need for speed' which was said to need it.
like i said above, i don't know how the emulation works. have never wrote more than a simple javascript bs emulator. but to me, it doesn't make much sense to check, ever single operation, for 5 or more things, and then perform a "skip" of sorts on 2 or so instructions.

maybe i'm wrong. would love if someone else had the same results.

#33 Updated by zlice 6 months ago

JMC4789 wrote:

There's a difference between idleskipping and sync on idleskipping.

having trouble finding where sync idle is in the code. did a grep and see configuration/settings stuff, but not it being used

#34 Updated by JMC4789 6 months ago

If you're using single core none of these games will ever need Idleskipping. You're confusing idleskipping was a hack that dolphin has to make dualcore more stable called "sync on idleskip." Basically, Dolphin will synchronize the GPU/CPU thread on idleskipping. And yes, Need For Speed is affected by that because I tested it personally. There has to be some kind of flaw with your detection.

#35 Updated by zlice 6 months ago

just commented out everything but the printf and things go fine (although even in single core, printf happens only on boot and game load like i said). so the actual code trying to skip the OPCD 32 block is what slows things down and happens in the background somewhere since printf isn't printing

i tried single and dual core.

did find the sync on idle gpu code, not what i'm looking at directly but it is called here (https://github.com/dolphin-emu/dolphin/blob/master/Source/Core/Core/PowerPC/Jit64/Jit_LoadStore.cpp#L142) through ABI_CallFunction(CoreTiming::Idle); i believe

setting "SyncOnSkipIdle = False" for 'Need for Speed' doesn't help - so clearly not related to the OPCD 32 skip

#36 Updated by JMC4789 6 months ago

I'm probably misunderstanding something on some level.

There could have been a mistake in the progress report, the commit, or the testing. I am 100% certain that not syncing on idleskip makes Dolphin run in a much more messed up state in games where idleskipping is enabled. There should be stats that allow you to see idleskipping stats in the titlebar while a game is running, I used to use it when playing/testing these games.

I'd really prefer someone other than me implement/submit a fix because I don't have any knowledge of the JIT and syncing outside of testing and observing behavior.

#37 Updated by degasus 6 months ago

zlice idle skipping is a big preformance boost and so it affects the timing a lot. So there is really no point is reporting dual core differences here. Please provide here only infos about games which have different behavior on single core. And SyncOnSkipIdle has no effect on single core at all.

#38 Updated by zlice 6 months ago

i'm more confused now

i was not originally even looking at the "SYNC" on idle to gpu

i believe past code says these skips (like lwz) are specifically for dual core

now degasus says it's only for single core

games crash or run poorly with the specific code i'm talking on dual or single core

as i said above somewhere, i think the problem is x64 "safeloadtoreg"
i noticed JosJuice's link says Arm64 works while x64 does not, which seems to back that theory up. i started comparing code, Arm64 has its own safeloadtoreg, need to compare between ISAs

is there any reason to believe my commenting out of the lzw skip and leaving it in is a bad test?

#39 Updated by JosJuice 6 months ago

Idle skipping is both for dual core and single core. But, due to how unpredictable timings are when using dual core, any timing bugs that happen when using dual core aren't really worth reporting since the only proper fix for them is to not use dual core.

SyncOnSkipIdle is not the same thing as idle skipping, but it interacts with idle skipping. SyncOnSkipIdle only has an effect when you are using dual core.

Please note that nobody has tested whether the Resident Evil 0 issue happens on JitArm64 yet. I only tested it on CSI and don't have this game.

#40 Updated by zlice 6 months ago

true, may be correlating the safelodatoreg

but i did find this, which is interesting
`
// void EmuCodeBlock::SafeLoadToReg(X64Reg reg_value, const Gen::OpArg& opAddress, int accessSize,
// s32 offset, BitSet32 registersInUse, bool signExtend, int flags)

SafeLoadToReg(Rd, Ra, accessSize, offset, CallerSavedRegistersInUse(), signExtend);

`

no flags ??? so that value is random when the function is called i would assume?

#42 Updated by zlice 6 months ago

  (inst.hex & 0xFFFF0000) == 0x800D0000 && (inst.hex != 0x800D9C20) &&

with this RE0 works, not sure why yet

#43 Updated by degasus 6 months ago

zlice: PR https://github.com/dolphin-emu/dolphin/pull/7287 rewrites the idle skip implementation. Is RE0 accidently fixed there?

#44 Updated by zlice 6 months ago

yes, removing that if OPCD == 32 is what i was doing manually

with the inst.hex & 0xFFFF0000 == 0x800D000 - it would seem loads won't execute
is that right? that seems important

800D9C20

10000000000011011001110000100000

https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.alangref/idalangref_lwz_lwo_instrs.htm
Bits Value
0 - 5 32
6 - 10 RT
11 - 15 RA
16 - 31 D

10000 lwz
00000 r0
00110 r6
11001110000100000 (pointer adjustment value)

#45 Updated by JosJuice 6 months ago

Does that "yes" mean that you tested the PR and it makes the game work properly?

#46 Updated by zlice 6 months ago

not the entire PR, only the jit64/jit_loadstore.cpp around line 122 (looks like pr commits changed)

#47 Updated by zlice 6 months ago

so i think WriteExceptionExit() is clobbering registers that are in use with instruction 800D9C20

commented out and all was fine

also, that whole lwz OPCD 32 skip can be shortened and moved to the end of the lXXx void. looks like the arm64 does the same, worked np for me.

not sure what the WriteExceptionExit() function is for but it uses scratch registers and does a push/pop to check powerpc checkexceptions

#48 Updated by JosJuice 6 months ago

  • Status changed from Accepted to Fixed
  • Fixed in set to 5.0-10039

#49 Updated by Maverick_Hunter_X 6 months ago

It is most definitely fixed now, thank you so much from the bottom of my heart!

I can finally ditch 5.0-895, hallelujah!!!!!

Good bye issue, it been over two long years....

Also available in: Atom PDF