Emulator

Below is i-ninja using opengl, real xfb and 8x antialiasing.
http://prntscr.com/kk5xp
and below is direct 3d 11 with real xfb and 8x antialiasing.
http://www.mediafire.com/view/?3h329vguu4q3f#waavtpde3jj2rx9

Why the hell did you use two different upload-services for the images?

I had wifi issues and the first one was rather slow and stopped uploading repeatedly. So i used mediafire for the second pic. Lightshot is a great tool put their file hosting is rather slow...

Try the attached patch (pastie.org is 503'ing on me).

Even if it doesn't help your specific issue, it still fixes some XFB scaling issues with both DX9 and DX11.

And if this patch doesn't get added, I'll create a new issue and pester the devs until they do. :p

Why didn't you just set the BorderColor in http://msdn.microsoft.com/en-us/library/ff476207%28v=vs.85%29 to that value?

Because I'm a total D3D n00b, that's why. :p Thanks for the tip, but can I go back to writing Avisynth filters now? The digital video headaches are less migraine-y than the 3D ones...

Turns out the culprit was D3D::SetLinearCopySampler(), which Renderer::Swap() calls right before it calls Television::Submit(). I copied the code for creating linear_copy_sampler into Television, and either setting the border to YUYV black or setting the filter to point fixes the issue (I chose the former, of course.)

Anyway, about the actual scaling fixes (the border thing is a separate issue but it's in my xfb-fixes branch, so into the patch it went, if only by accident), here are a couple of screenshots that clearly demonstrate the issue (both DX9, 1xIR, EFB to RAM, real XFB):
b0rken: http://s8.postimage.org/4h7fl3unp/RMCE01_4.png
patched: http://s13.postimage.org/twwndgqnb/RMCE01_5.png

Please use tabs for indenting instead :)

Is the "0.499 * x_step" thing still necessary? If so, could you explain a bit more what specifically it's supposed to achieve?

I knew you'd bring up the tab thing! ;) Is there a per-solution setting for that in VS2010?

The 0.499 part used to be 0.5, but I found that at higher IRs rounding errors would make the point sampler sometimes land on one side of a texel edge, but on another side of an edge later in the same line, creating beat patterns (long story; lots of alternating-line test patterns were involved). The 0.499 introduces just enough compensating error to always(?) make the point sampler land on an actual texel, but not so much error that it would noticeably affect linear sampling (which isn't done right now anyway but could be). It's actually just me being anal; 0.5 works just as well for anything that isn't a test pattern (ie games).

The need to subtract the 0.499/0.5 in the first place comes from the fact that the original 0.5 texel offset (WHY, MS, WHY?) gets effectively doubled due to the 2:1 subsampling (even though each output pixel contains two Y values, the shader is still only outputting 320 actual pixels). For example, at 1xIR the sampler would run from 1...639 (instead of 0.5...638.5 like it should) because it was doubling 0.5...319.5. Same for higher IRs, just different ratios, hence the need to multiply 0.499 by x_step (which is really just the IR ratio).

If you're confused by the above explanation, don't worry; that's the expected reaction to one of my ramblings. -_-

tl;dr - I can't swear my patch doesn't break something, but I did do a lot of testing (my OCD over stuff like this ensured that!)

Nice results there, unfortunately i get "Patch format detection failed" when trying to apply it with tortoisegit.

If I understand you correctly, then you'll likely want to add something like what I've added in the pixel shader gen at http://code.google.com/p/dolphin-emu/source/browse/Source/Core/VideoCommon/Src/VertexShaderGen.cpp#496 .

That 0.5 texel offset crap basically gets more complicated if a) the texture gets distorted by perspective (not the case for here) b) the texture size isn't equal to the viewport size (that might be the case here?). Anyway, if it's just the texel->pixel mapping that you're trying to correct (I wasn't sure if that was already done in the shader encoders, that's why I didn't try to fix it yet) you'll likely want to either modify the vertex shader or the source vertices themselves.

The patch had Unix line endings, maybe that was the issue. I've attached another one CRLF-style. (Am I the only one getting 503's uploading to pastie.org? It worked a few days ago.)

So basically you're suggesting giving the YUYV encoder its own vertex shader (right now it's using VertexShaderCache::GetSimpleVertexShader(0))? Hmm, not sure I'm up to that quite yet, but I'll look into it.

The modded pixel shader works, but I can see how fixing the coordinates up in the vertex shader would be preferable.