Emulator

Does not happen on Software Renderer if you actually play there rather than using the fifo-log.

.. post it? (not sure why you ask anyway)

Forget it, I was a bit sleepy yesterday, should have posted the code anyway.

In Core/VideoCommon/VertexLoader.cpp, replace method UpdateBoundingBox() with this one:

void LOADERDECL UpdateBoundingBox()
{
if (!PixelEngine::bbox_active)
return;

// reset videodata pointer
VertexManager::s_pCurBufferPointer = s_bbox_pCurBufferPointer_orig;

// copy vertex pointers
memcpy(VertexManager::s_pCurBufferPointer, s_bbox_vertex_buffer, 12);
VertexManager::s_pCurBufferPointer += 12;

// We must transform the just loaded point by the current world and projection matrix - in software.
// Then convert to screen space and update the bounding box.
float p[3] = {s_bbox_vertex_buffer[0], s_bbox_vertex_buffer[1], s_bbox_vertex_buffer[2]};

const float *world_matrix  = (float*)xfmem + MatrixIndexA.PosNormalMtxIdx * 4;

// We need to get the raw projection values for the bounding box calculation
// to work properly. That means, no projection hacks!
float *rawProjection = xfregs.projection.rawProjection;
float proj_matrix[16];

switch (xfregs.projection.type)
{
case GX_PERSPECTIVE:
    proj_matrix[0]  = rawProjection[0]; proj_matrix[1]  = 0.0f;             proj_matrix[2]  = rawProjection[1]; proj_matrix[3]  = 0.0f;
    proj_matrix[4]  = 0.0f;             proj_matrix[5]  = rawProjection[2]; proj_matrix[6]  = rawProjection[3]; proj_matrix[7]  = 0.0f;
    proj_matrix[8]  = 0.0f;             proj_matrix[9]  = 0.0f;             proj_matrix[10] = rawProjection[4]; proj_matrix[11] = rawProjection[5];
    proj_matrix[12] = 0.0f;             proj_matrix[13] = 0.0f;             proj_matrix[14] = -1.0f;            proj_matrix[15] = 0.0f;
    break;

case GX_ORTHOGRAPHIC:
    proj_matrix[0]  = rawProjection[0]; proj_matrix[1]  = 0.0f;             proj_matrix[2]  = 0.0f;             proj_matrix[3]  = rawProjection[1];
    proj_matrix[4]  = 0.0f;             proj_matrix[5]  = rawProjection[2]; proj_matrix[6]  = 0.0f;             proj_matrix[7]  = rawProjection[3];
    proj_matrix[8]  = 0.0f;             proj_matrix[9]  = 0.0f;             proj_matrix[10] = rawProjection[4]; proj_matrix[11] = rawProjection[5];
    proj_matrix[12] = 0.0f;             proj_matrix[13] = 0.0f;             proj_matrix[14] = 0.0f;             proj_matrix[15] = 1.0f;
    break;

default:
    ERROR_LOG(VIDEO, "Unknown projection type: %d", xfregs.projection.type);
}


float t[3];
t[0] = p[0] * world_matrix[0] + p[1] * world_matrix[1] + p[2] * world_matrix[2] + world_matrix[3];
t[1] = p[0] * world_matrix[4] + p[1] * world_matrix[5] + p[2] * world_matrix[6] + world_matrix[7];
t[2] = p[0] * world_matrix[8] + p[1] * world_matrix[9] + p[2] * world_matrix[10] + world_matrix[11];

float o[3];
o[0] = t[0] * proj_matrix[0]  + t[1] * proj_matrix[1]  + t[2] * proj_matrix[2] + proj_matrix[3];
o[1] = t[0] * proj_matrix[4]  + t[1] * proj_matrix[5]  + t[2] * proj_matrix[6] + proj_matrix[7];
o[2] = t[0] * proj_matrix[12] + t[1] * proj_matrix[13] + t[2] * proj_matrix[14] + proj_matrix[15];

o[0] /= o[2];
o[1] /= o[2];

// Max width is 608, while max height is 480
// Could this be register dependant? Seems to work this way, though
o[0] = (o[0] + 1.0f) * 304.0f;
o[1] = (1.0f - o[1]) * 240.0f;

if (o[0] < PixelEngine::bbox[0]) PixelEngine::bbox[0] = (u16) std::max(0.0f, o[0]);
if (o[0] > PixelEngine::bbox[1]) PixelEngine::bbox[1] = (u16) o[0];
if (o[1] < PixelEngine::bbox[2]) PixelEngine::bbox[2] = (u16) std::max(0.0f, o[1]);
if (o[1] > PixelEngine::bbox[3]) PixelEngine::bbox[3] = (u16) o[1];

}

This might also fix many minor issues with Super Paper Mario, and hopefully also the hang on world 6-1. Needs testing...

Can you provide a diff instead for easier review?

Sorry, I don't have git installed on this computer... I will attach the diff later when I get home, though.

I can confirm it's all working pretty well. There's some problems on the shadows around them, but peeling posters and pageflip stuff is all working properly.

As requested, here is a diff with the latest changes.

diff --git a/Source/Core/VideoCommon/VertexLoader.cpp b/Source/Core/VideoCommon/VertexLoader.cpp
index 4a389bc..e4542a9 100644
--- a/Source/Core/VideoCommon/VertexLoader.cpp
+++ b/Source/Core/VideoCommon/VertexLoader.cpp
@@ -119,25 +119,39 @@ void LOADERDECL UpdateBoundingBox()
float p[3] = {s_bbox_vertex_buffer[0], s_bbox_vertex_buffer[1], s_bbox_vertex_buffer[2]};

const float *world_matrix  = (float*)xfmem + MatrixIndexA.PosNormalMtxIdx * 4;

• const float *proj_matrix = &g_fProjectionMatrix[0];

  float t[3];
  t[0] = p[0] * world_matrix[0] + p[1] * world_matrix[1] + p[2] * world_matrix[2] + world_matrix[3];
  t[1] = p[0] * world_matrix[4] + p[1] * world_matrix[5] + p[2] * world_matrix[6] + world_matrix[7];
  t[2] = p[0] * world_matrix[8] + p[1] * world_matrix[9] + p[2] * world_matrix[10] + world_matrix[11];

• float o[3];

• o[0] = t[0] * proj_matrix[0] + t[1] * proj_matrix[1] + t[2] * proj_matrix[2] + proj_matrix[3];

• o[1] = t[0] * proj_matrix[4] + t[1] * proj_matrix[5] + t[2] * proj_matrix[6] + proj_matrix[7];

• o[2] = t[0] * proj_matrix[12] + t[1] * proj_matrix[13] + t[2] * proj_matrix[14] + proj_matrix[15];

• o[0] /= o[2];

• o[1] /= o[2];

• // We need to get the raw projection values for the bounding box calculation

• // to work properly. That means, no projection hacks!

• float * const proj_matrix = xfregs.projection.rawProjection;

• // Max width seems to be 608, while max height is 480

• // Here height is set to 484 as BBox bottom always seems to be off by a few pixels

• o[0] = (o[0] + 1.0f) * 304.0f;

• o[1] = (1.0f - o[1]) * 242.0f;

• float o[2];
  +

• // Only calculate what we need, discard the rest

• switch (xfregs.projection.type)

• {

• case GX_PERSPECTIVE:

•   o[0] = (t[0] * proj_matrix[0] + t[2] * proj_matrix[1]) / (-t[2]);
  

•   o[1] = (t[1] * proj_matrix[2] + t[2] * proj_matrix[3]) / (-t[2]);
  

•   break;
  

  +

• case GX_ORTHOGRAPHIC:

•   o[0] = t[0] * proj_matrix[0] + proj_matrix[1];
  

•   o[1] = t[1] * proj_matrix[2] + proj_matrix[3];
  

•   break;
  

  +

• default:

•   ERROR_LOG(VIDEO, "Unknown projection type: %d", xfregs.projection.type);
  

• }
  +

• // Computed pixel position depends on viewport register setting

• o[0] = (o[0] + 1.0f) * xfregs.viewport.wd;

• o[1] = (o[1] - 1.0f) * xfregs.viewport.ht;

  if (o[0] < PixelEngine::bbox[0]) PixelEngine::bbox[0] = (u16) std::max(0.0f, o[0]);
  if (o[0] > PixelEngine::bbox[1]) PixelEngine::bbox[1] = (u16) o[0];

How did you chose the value 304.0f? Is there any noticeable change by making it 324.0f instead?

That 304.0f value is wrong. If you check the diff I posted, I changed it so the width is obtained from xfregs.viewport.wd.

Doing it this way fixes the white shadow in Mickey Magical Mirror,while still working correctly in Paper Mario.

Right, I was in a hurry and misread the diff, sorry!

Either way, looks good. Does the projection hack stuff even matter though? AFAIK paper mario doesn't even use PHacks after all.

Hm, 304.0f vs xfregs.viewport.wd, did you miss a *0.5f ?

Should this fix 6193 as well?

xfregs.viewport.wd is always set to half the viewport value (304.0f)
xfregs.viewport.ht is both half, and negative (-240.0f)

No idea why, but the values fit and it works well, even when the viewport register values are changed.

Removing the hack stuff is what solved the bounding box issues. Besides, using the raw values means less matrix multiplication, and faster code ;)

Oh and I haven't checked if it fixes 6193. Does anyone have a save file next to that area?

It doesn't fix either the glitched punies, or the hang in chapter 6-1 of Super Paper Mario, however.

Glitched punies can be fixed by exporting their texture and then making a texture pack with the fixed sprites. Maybe it's similar to the Wind Waker problem that delroth fixed?

I don't think it is. If you look closely to screenshots of the bug, the correct textures are loaded, but they are very warped, almost beyond recognition.