Project

General

Profile

Emulator Issues #6538

Benchmark Mode

Added by NeoBrainX almost 7 years ago. Updated over 2 years ago.

Status:
Accepted
Priority:
Low
Assignee:
-
% Done:

0%

Operating system:
N/A
Issue type:
Feature request
Milestone:
Regression:
No
Relates to usability:
No
Relates to performance:
No
Easy:
No
Relates to maintainability:
No
Regression start:
Fixed in:

Description

Would be cool if Dolphin had some kind of actual benchmarking mode. Dolphin could be included in benchmark suites like Phoronix Test Suite that way. Also, it could be used by end users to easily determine if their hardware is powerful enough to run Dolphin.

Requirements:
- automated control (batch mode + some way to make the benchmark stop automatically at some point)
- FPS counting (already possible via FPS logging, but the user has to manually enable it and it's not the most convenient thing to work with as it doesn't calculate average/min/max FPS automatically)
- standardized settings or setting profiles (possibly requires lots of work)
- actual benchmarks. I don't think we're allowed to distribute dffs of commercial games with Dolphin (nor am I sure if fifo playback is a suitable benchmark), so we'd need to write some kind of homebrew application involving at the very least things like EFB copies.
- possibly other things that I'm forgetting about right now.

History

#1 Updated by nigel.harris.uk almost 7 years ago

It would be cool if it worked like this:

Benchmarks runs a non-configurable homebrew application with EFB copy to ram tests xfcb testes etc.

The test should be 'calibrated' to the recommended 'Excellent' hardware specs listed in the FAQ: https://dolphin-emu.org/docs/faq/

for example the 'Excellent' hardware specs has a score of 100 after running benchmark. Anything below that the Benchamrk can advise to set XFB to disabled / EFB to Texture etc... etc... depending on how much lower the score is to the 'Ecellent' hardware.

#2 Updated by nigel.harris.uk almost 7 years ago

continuing on from the above....

basically the benchmark mode can advise on the optimal setting for Dolphin to run at a decent speed for lower spec'd machines and vice versa for higher spec'd machines (assuming the user has the least compatible options selected).

#3 Updated by rachelbryk almost 7 years ago

No, that's silly.

#4 Updated by degasus almost 7 years ago

nigel: I'm sorry, but this won't work as the performance of every feature depend on the game. eg some use more efb copys so efb2ram is a bigger slowdown.

#5 Updated by JMC4789 almost 7 years ago

It'd be nice to have games marked as fast or slow in regards of the benchmark score, maybe.

Neobrain: Can't we use TAS input files, Savestates, etc? DFF files are definitely not a good source, but a homebrew app could work.

#6 Updated by NeoBrainX almost 7 years ago

The point is we should to be able to distribute all file required to run the benchmark with the Dolphin source. We likely can't do that with DFFs if they are recorded from commercial games. Similar, one cannot use TAS input files or savestates without the required game ISO.

#7 Updated by NeoBrainX almost 7 years ago

Should've probably mentioned this as a separate requirement:
- needs to run out of the box without requiring acquisition of external resources.

#8 Updated by nigel.harris.uk almost 7 years ago

Wickmark (sorry I cant see rest of your username) thank you for the clarification!

As JMC4789 says though I think it be good to get a little useful info back from running the benchmark?

I think the benchmark would be more useful for lower end machines users. TBH a guy with a 3960x etc will know his machine is fast enough.

The test could be into groups like EFB test / XFB tes / Texture cache test / CPU test etc. For example if the EFB test was really really slow then at least the user knows ok to get a decent speed I have to disable EFB.

But all in all a benchmark mode would be a cool feature. I dont recall any other emu having such a thing. Cant wait to see what you guys can come up with.

#9 Updated by rachelbryk almost 7 years ago

That doesn't make any sense at all. On a slow cpu, every test is going to be slow. We aren't going to maintain a database of benchmark scores for every single game, and make suggestions based on it.

#10 Updated by degasus almost 7 years ago

nigel: I also think we should test every features on each own as this would benchmark the differences of our backends. But this still isn't useful for users to see which feature should be disabled.

#11 Updated by delroth almost 7 years ago

We aren't going to maintain a database of benchmark scores for every single game, and make suggestions based on it.

Actually we could, and that would be an interesting idea. Compute average FPS over 1min periods the whole time Dolphin is playing, get all required info like config, CPU, OS, driver version, etc. and send that online. Then make a good UI for it.

It's not even that hard to do imho, something like 400 lines of code at most + ~50-100 lines of code to add a dialog box asking people if they want to give us data when launching Dolphin.

#12 Updated by nigel.harris.uk almost 7 years ago

Delroth: Now that really does sound good.

I'm probably getting overly excited about this but if a database of FPS were set up online then it could be linked to the WIKI pages of each game?

For example the user can right click the game within dolphin and select 'WIKI'. The WIKI page of that game can contain a Benchmark section containing the performance database. The user can then filter out cpu config, video cards/ ram / dolphin config etc and get a rough idea on performance on his/her machine.

Just thinking out loud.

#13 Updated by NeoBrainX almost 7 years ago

delroth: We seem to be thinking in different directions.

You seem to prefer some database for users to look up their hardware in so they can judge what games they will be able to run

I, on the other hand, would prefer letting users run a small number of different benchmarks (which are bundled with Dolphin) and give them a score for each one, so they can look up their score in a forum thread and see how well-suited their hw is for Dolphin.

I guess your suggestion is better if people want to run specific games. It's going to be tougher to maintain comparable results though, since we have to make sure that actually the uploaded scores of a single game are actually comparable against each other (i.e. people playing zelda TP at the beginning of the game in hyrule field would report much better scores than people playing in hyrule field at late-game).

However, I guess making users look up their hardware in a database requires a much higher technical knowledge on the end user side. If we only had a fixed number of well-defined benchmarks and returned a score for each of them that people could look up in a forum thread which then tells them how well-suited their hardware is for Dolphin, that'd be much easier for them to do.

That said, I guess both would work, although at the moment I'd still prefer mine.

#14 Updated by delroth almost 7 years ago

Both would work, but making a useful homebrew program that reproduces the workload of Dolphin accurately enough to provide good benchmark results seems hard to me. Never said that these two ideas were exclusive.

#15 Updated by mbc07 almost 7 years ago

What about separate scores? For example, Dolphin is bunbled with a few homebrew apps for benchmarking, one testing CPU fucntions, other testing EFB Copies, etc. Then, at the end of every test, a system score (like CPU: 75, EFB Copies: 20, XFB: 60) is set.

Initially, users could use the benchmark mode also while running games to get scores that can be submitted to the wiki (or other place), at least to populate a benchmark database.

At the end, after the database is set, a new user could run the homebrew benchmark to get their system score, and then he/she could go to the game thread or wiki page and here we would have a section like "Game score requirements: CPU 50, EFB Copies 70, etc" so the user discover if their computer is able to handle the game. If not, some suggestions would appear (game-specific), like "this game uses several EFB Copies, using EFB to Texture or disabling it should provide better performance at cost of accuracy, etc, etc"...

#16 Updated by delroth almost 7 years ago

  • Milestone changed from Next to Current

4.0 was released, moving Milestone-Next to Milestone-Current.

#18 Updated by delroth about 6 years ago

  • Milestone changed from Current to Next

Moving to next milestone since these issues are not critical to a new release.

If you feel strongly about this issue and want it to block the next Dolphin release, please assign yourself as owner and re-add Milestone-Current.

#19 Updated by ZephyrSurfer almost 6 years ago

Can you discuss how this is supposed to work since benchmarks are static versions and results are not comparable between versions.

If there was a benchmark mode in Dolphin a future revision could be dramatically faster and thus comparisons between speed of CPUs could not be made.

I think this is quite vague and needs some more specifics if it is to be advanced.

#20 Updated by Buddybenj almost 6 years ago

I think a good benchmark would need to be:
- built in to Dolphin (maybe in tools menu)
- low margin of error (i.e. if you run the benchmark multiple times you should get very similar results)
- proportional to real world Dolphin performance (i.e. a higher benchmark score should almost always mean better real world Dolphin performance)
- take about 3-5 minutes

I hope this will make it into Dolphin 5.0 :)

#21 Updated by ZephyrSurfer over 5 years ago

Ok so we've got FPS counting now I guess with min, dev, max -all that.
I'd imagine this would need to be attempted to be completed before 5.0 then for users to have reproducable results in the benchmark.
Thoughts on using Pov-ray in the benchmark?

#22 Updated by nintendomaniac64 over 5 years ago

Delroth PM'd me a few months back stating that povray is actually somewhat inaccurate and that he made a more accurate benchmark that is also considerably smaller in filesize (379KB):
http://delroth.net/luabench.dol

Alternatively, see attachment.

#23 Updated by nintendomaniac64 over 5 years ago

One thing to note is that if we do integrate something like luabench into Dolphin then while running said bench the "pause emulation" function should be disabled since it screws up the resulting time (basically the benchmark still counts the time while emulation was paused, I'm guessing the realtime clock in involved in some way)

#24 Updated by nintendomaniac64 about 5 years ago

Well, I guess this didn't make it into version 5.0...

#25 Updated by JMC4789 about 5 years ago

There's not enough demand for a benchmark mode, and the implementation of it would have to be agreed upon. Also, how do we benchmark GPU stuff? It requires a more complicated benchmark than we have.

#26 Updated by JosJuice about 4 years ago

  • Milestone changed from Next to Current

5.0 was released, moving Milestone-Next to Milestone-Current.

#27 Updated by JosJuice over 2 years ago

  • Milestone deleted (Current)

#28 Updated by nintendomaniac64 over 2 years ago

I just remembered this after nearly having a livestreamer (who got shipped a Ryzen 2400G early by Amazon on accident) to run luabench in Dolphin 5.0. However, he completely "noped" out when he noticed that my unofficial "for dummies" edition was hosted on Google Drive, but as he was someone that hasn't used Dolphin before I couldn't have entirely expected him to try doing things the manual way.

Having luabench.dol (which is smaller than DSPTool.exe mind you) bundled into an official Dolphin 5.5 or 6.0 with an easy-to-access "run CPU benchmark" menu option in the 'Tools' menu or the like would have completely avoided the issue of having to pick between "untrustworthy download" and "not-trivial-for-beginners download".

And luabench.dol seems to run out-of-the-box without any settings adjustment on new builds of Dolphin now (unlike Dolphin 5.0), so that's always a good sign.

Also available in: Atom PDF