Emulator Issues #6538
Would be cool if Dolphin had some kind of actual benchmarking mode. Dolphin could be included in benchmark suites like Phoronix Test Suite that way. Also, it could be used by end users to easily determine if their hardware is powerful enough to run Dolphin.
- automated control (batch mode + some way to make the benchmark stop automatically at some point)
- FPS counting (already possible via FPS logging, but the user has to manually enable it and it's not the most convenient thing to work with as it doesn't calculate average/min/max FPS automatically)
- standardized settings or setting profiles (possibly requires lots of work)
- actual benchmarks. I don't think we're allowed to distribute dffs of commercial games with Dolphin (nor am I sure if fifo playback is a suitable benchmark), so we'd need to write some kind of homebrew application involving at the very least things like EFB copies.
- possibly other things that I'm forgetting about right now.
#1 Updated by nigel.harris.uk over 7 years ago
It would be cool if it worked like this:
Benchmarks runs a non-configurable homebrew application with EFB copy to ram tests xfcb testes etc.
The test should be 'calibrated' to the recommended 'Excellent' hardware specs listed in the FAQ: https://dolphin-emu.org/docs/faq/
for example the 'Excellent' hardware specs has a score of 100 after running benchmark. Anything below that the Benchamrk can advise to set XFB to disabled / EFB to Texture etc... etc... depending on how much lower the score is to the 'Ecellent' hardware.
#2 Updated by nigel.harris.uk over 7 years ago
continuing on from the above....
basically the benchmark mode can advise on the optimal setting for Dolphin to run at a decent speed for lower spec'd machines and vice versa for higher spec'd machines (assuming the user has the least compatible options selected).
#6 Updated by NeoBrainX over 7 years ago
The point is we should to be able to distribute all file required to run the benchmark with the Dolphin source. We likely can't do that with DFFs if they are recorded from commercial games. Similar, one cannot use TAS input files or savestates without the required game ISO.
#8 Updated by nigel.harris.uk over 7 years ago
Wickmark (sorry I cant see rest of your username) thank you for the clarification!
As JMC4789 says though I think it be good to get a little useful info back from running the benchmark?
I think the benchmark would be more useful for lower end machines users. TBH a guy with a 3960x etc will know his machine is fast enough.
The test could be into groups like EFB test / XFB tes / Texture cache test / CPU test etc. For example if the EFB test was really really slow then at least the user knows ok to get a decent speed I have to disable EFB.
But all in all a benchmark mode would be a cool feature. I dont recall any other emu having such a thing. Cant wait to see what you guys can come up with.
#11 Updated by delroth over 7 years ago
We aren't going to maintain a database of benchmark scores for every single game, and make suggestions based on it.
Actually we could, and that would be an interesting idea. Compute average FPS over 1min periods the whole time Dolphin is playing, get all required info like config, CPU, OS, driver version, etc. and send that online. Then make a good UI for it.
It's not even that hard to do imho, something like 400 lines of code at most + ~50-100 lines of code to add a dialog box asking people if they want to give us data when launching Dolphin.
#12 Updated by nigel.harris.uk over 7 years ago
Delroth: Now that really does sound good.
I'm probably getting overly excited about this but if a database of FPS were set up online then it could be linked to the WIKI pages of each game?
For example the user can right click the game within dolphin and select 'WIKI'. The WIKI page of that game can contain a Benchmark section containing the performance database. The user can then filter out cpu config, video cards/ ram / dolphin config etc and get a rough idea on performance on his/her machine.
Just thinking out loud.
#13 Updated by NeoBrainX over 7 years ago
delroth: We seem to be thinking in different directions.
You seem to prefer some database for users to look up their hardware in so they can judge what games they will be able to run
I, on the other hand, would prefer letting users run a small number of different benchmarks (which are bundled with Dolphin) and give them a score for each one, so they can look up their score in a forum thread and see how well-suited their hw is for Dolphin.
I guess your suggestion is better if people want to run specific games. It's going to be tougher to maintain comparable results though, since we have to make sure that actually the uploaded scores of a single game are actually comparable against each other (i.e. people playing zelda TP at the beginning of the game in hyrule field would report much better scores than people playing in hyrule field at late-game).
However, I guess making users look up their hardware in a database requires a much higher technical knowledge on the end user side. If we only had a fixed number of well-defined benchmarks and returned a score for each of them that people could look up in a forum thread which then tells them how well-suited their hardware is for Dolphin, that'd be much easier for them to do.
That said, I guess both would work, although at the moment I'd still prefer mine.
#15 Updated by mbc07 over 7 years ago
What about separate scores? For example, Dolphin is bunbled with a few homebrew apps for benchmarking, one testing CPU fucntions, other testing EFB Copies, etc. Then, at the end of every test, a system score (like CPU: 75, EFB Copies: 20, XFB: 60) is set.
Initially, users could use the benchmark mode also while running games to get scores that can be submitted to the wiki (or other place), at least to populate a benchmark database.
At the end, after the database is set, a new user could run the homebrew benchmark to get their system score, and then he/she could go to the game thread or wiki page and here we would have a section like "Game score requirements: CPU 50, EFB Copies 70, etc" so the user discover if their computer is able to handle the game. If not, some suggestions would appear (game-specific), like "this game uses several EFB Copies, using EFB to Texture or disabling it should provide better performance at cost of accuracy, etc, etc"...
#17 Updated by JMC4789 about 7 years ago
Somewhat mitigated for the time being? https://forums.dolphin-emu.org/Thread-new-dolphin-cpu-benchmark-no-game-required
#18 Updated by delroth almost 7 years ago
- Milestone changed from Current to Next
Moving to next milestone since these issues are not critical to a new release.
If you feel strongly about this issue and want it to block the next Dolphin release, please assign yourself as owner and re-add Milestone-Current.
#19 Updated by ZephyrSurfer over 6 years ago
Can you discuss how this is supposed to work since benchmarks are static versions and results are not comparable between versions.
If there was a benchmark mode in Dolphin a future revision could be dramatically faster and thus comparisons between speed of CPUs could not be made.
I think this is quite vague and needs some more specifics if it is to be advanced.
#20 Updated by Buddybenj over 6 years ago
I think a good benchmark would need to be:
- built in to Dolphin (maybe in tools menu)
- low margin of error (i.e. if you run the benchmark multiple times you should get very similar results)
- proportional to real world Dolphin performance (i.e. a higher benchmark score should almost always mean better real world Dolphin performance)
- take about 3-5 minutes
I hope this will make it into Dolphin 5.0 :)
#23 Updated by nintendomaniac64 about 6 years ago
One thing to note is that if we do integrate something like luabench into Dolphin then while running said bench the "pause emulation" function should be disabled since it screws up the resulting time (basically the benchmark still counts the time while emulation was paused, I'm guessing the realtime clock in involved in some way)
#28 Updated by nintendomaniac64 about 3 years ago
I just remembered this after nearly having a livestreamer (who got shipped a Ryzen 2400G early by Amazon on accident) to run luabench in Dolphin 5.0. However, he completely "noped" out when he noticed that my unofficial "for dummies" edition was hosted on Google Drive, but as he was someone that hasn't used Dolphin before I couldn't have entirely expected him to try doing things the manual way.
Having luabench.dol (which is smaller than DSPTool.exe mind you) bundled into an official Dolphin 5.5 or 6.0 with an easy-to-access "run CPU benchmark" menu option in the 'Tools' menu or the like would have completely avoided the issue of having to pick between "untrustworthy download" and "not-trivial-for-beginners download".
And luabench.dol seems to run out-of-the-box without any settings adjustment on new builds of Dolphin now (unlike Dolphin 5.0), so that's always a good sign.