Emulator

It would be cool if it worked like this:

Benchmarks runs a non-configurable homebrew application with EFB copy to ram tests xfcb testes etc.

The test should be 'calibrated' to the recommended 'Excellent' hardware specs listed in the FAQ: https://dolphin-emu.org/docs/faq/

for example the 'Excellent' hardware specs has a score of 100 after running benchmark. Anything below that the Benchamrk can advise to set XFB to disabled / EFB to Texture etc... etc... depending on how much lower the score is to the 'Ecellent' hardware.

continuing on from the above....

basically the benchmark mode can advise on the optimal setting for Dolphin to run at a decent speed for lower spec'd machines and vice versa for higher spec'd machines (assuming the user has the least compatible options selected).

No, that's silly.

nigel: I'm sorry, but this won't work as the performance of every feature depend on the game. eg some use more efb copys so efb2ram is a bigger slowdown.

It'd be nice to have games marked as fast or slow in regards of the benchmark score, maybe.

Neobrain: Can't we use TAS input files, Savestates, etc? DFF files are definitely not a good source, but a homebrew app could work.

The point is we should to be able to distribute all file required to run the benchmark with the Dolphin source. We likely can't do that with DFFs if they are recorded from commercial games. Similar, one cannot use TAS input files or savestates without the required game ISO.

Should've probably mentioned this as a separate requirement:

• needs to run out of the box without requiring acquisition of external resources.

Wickmark (sorry I cant see rest of your username) thank you for the clarification!

As JMC4789 says though I think it be good to get a little useful info back from running the benchmark?

I think the benchmark would be more useful for lower end machines users. TBH a guy with a 3960x etc will know his machine is fast enough.

The test could be into groups like EFB test / XFB tes / Texture cache test / CPU test etc. For example if the EFB test was really really slow then at least the user knows ok to get a decent speed I have to disable EFB.

But all in all a benchmark mode would be a cool feature. I dont recall any other emu having such a thing. Cant wait to see what you guys can come up with.

That doesn't make any sense at all. On a slow cpu, every test is going to be slow. We aren't going to maintain a database of benchmark scores for every single game, and make suggestions based on it.

nigel: I also think we should test every features on each own as this would benchmark the differences of our backends. But this still isn't useful for users to see which feature should be disabled.

We aren't going to maintain a database of benchmark scores for every single game, and make suggestions based on it.

Actually we could, and that would be an interesting idea. Compute average FPS over 1min periods the whole time Dolphin is playing, get all required info like config, CPU, OS, driver version, etc. and send that online. Then make a good UI for it.

It's not even that hard to do imho, something like 400 lines of code at most + ~50-100 lines of code to add a dialog box asking people if they want to give us data when launching Dolphin.

Delroth: Now that really does sound good.

I'm probably getting overly excited about this but if a database of FPS were set up online then it could be linked to the WIKI pages of each game?

For example the user can right click the game within dolphin and select 'WIKI'. The WIKI page of that game can contain a Benchmark section containing the performance database. The user can then filter out cpu config, video cards/ ram / dolphin config etc and get a rough idea on performance on his/her machine.

Just thinking out loud.

@delroth: We seem to be thinking in different directions.

You seem to prefer some database for users to look up their hardware in so they can judge what games they will be able to run

I, on the other hand, would prefer letting users run a small number of different benchmarks (which are bundled with Dolphin) and give them a score for each one, so they can look up their score in a forum thread and see how well-suited their hw is for Dolphin.

I guess your suggestion is better if people want to run specific games. It's going to be tougher to maintain comparable results though, since we have to make sure that actually the uploaded scores of a single game are actually comparable against each other (i.e. people playing zelda TP at the beginning of the game in hyrule field would report much better scores than people playing in hyrule field at late-game).

However, I guess making users look up their hardware in a database requires a much higher technical knowledge on the end user side. If we only had a fixed number of well-defined benchmarks and returned a score for each of them that people could look up in a forum thread which then tells them how well-suited their hardware is for Dolphin, that'd be much easier for them to do.

That said, I guess both would work, although at the moment I'd still prefer mine.

Both would work, but making a useful homebrew program that reproduces the workload of Dolphin accurately enough to provide good benchmark results seems hard to me. Never said that these two ideas were exclusive.

What about separate scores? For example, Dolphin is bunbled with a few homebrew apps for benchmarking, one testing CPU fucntions, other testing EFB Copies, etc. Then, at the end of every test, a system score (like CPU: 75, EFB Copies: 20, XFB: 60) is set.

Initially, users could use the benchmark mode also while running games to get scores that can be submitted to the wiki (or other place), at least to populate a benchmark database.

At the end, after the database is set, a new user could run the homebrew benchmark to get their system score, and then he/she could go to the game thread or wiki page and here we would have a section like "Game score requirements: CPU 50, EFB Copies 70, etc" so the user discover if their computer is able to handle the game. If not, some suggestions would appear (game-specific), like "this game uses several EFB Copies, using EFB to Texture or disabling it should provide better performance at cost of accuracy, etc, etc"...

Somewhat mitigated for the time being? https://forums.dolphin-emu.org/Thread-new-dolphin-cpu-benchmark-no-game-required

Can you discuss how this is supposed to work since benchmarks are static versions and results are not comparable between versions.

If there was a benchmark mode in Dolphin a future revision could be dramatically faster and thus comparisons between speed of CPUs could not be made.

I think this is quite vague and needs some more specifics if it is to be advanced.

I think a good benchmark would need to be:

• built in to Dolphin (maybe in tools menu)
• low margin of error (i.e. if you run the benchmark multiple times you should get very similar results)
• proportional to real world Dolphin performance (i.e. a higher benchmark score should almost always mean better real world Dolphin performance)
• take about 3-5 minutes

I hope this will make it into Dolphin 5.0 :)

Ok so we've got FPS counting now I guess with min, dev, max -all that.
I'd imagine this would need to be attempted to be completed before 5.0 then for users to have reproducable results in the benchmark.
Thoughts on using Pov-ray in the benchmark?

Delroth PM'd me a few months back stating that povray is actually somewhat inaccurate and that he made a more accurate benchmark that is also considerably smaller in filesize (379KB):
http://delroth.net/luabench.dol

Alternatively, see attachment.

One thing to note is that if we do integrate something like luabench into Dolphin then while running said bench the "pause emulation" function should be disabled since it screws up the resulting time (basically the benchmark still counts the time while emulation was paused, I'm guessing the realtime clock in involved in some way)

Well, I guess this didn't make it into version 5.0...

There's not enough demand for a benchmark mode, and the implementation of it would have to be agreed upon. Also, how do we benchmark GPU stuff? It requires a more complicated benchmark than we have.

I just remembered this after nearly having a livestreamer (who got shipped a Ryzen 2400G early by Amazon on accident) to run luabench in Dolphin 5.0. However, he completely "noped" out when he noticed that my unofficial "for dummies" edition was hosted on Google Drive, but as he was someone that hasn't used Dolphin before I couldn't have entirely expected him to try doing things the manual way.

Having luabench.dol (which is smaller than DSPTool.exe mind you) bundled into an official Dolphin 5.5 or 6.0 with an easy-to-access "run CPU benchmark" menu option in the 'Tools' menu or the like would have completely avoided the issue of having to pick between "untrustworthy download" and "not-trivial-for-beginners download".

And luabench.dol seems to run out-of-the-box without any settings adjustment on new builds of Dolphin now (unlike Dolphin 5.0), so that's always a good sign.