I'm not an expert, but I believe if you have a 3D matrix then you just multiply the point by it? Coordinate transformations are fundamentally matrix multiplications.

Some other random thoughts:

• The picture as drawn is not enough information, I believe. You need to specify the origin and directions of axes in both planes to get an unambiguous matrix. But if you already have a matrix this is moot.
• A good first step is to transform the origin to the first plane.

As Kartik Agaram says, I think there is missing piece of information to uniquely specify a point on the plane. If you have a location of a "camera", then you can use the ray that connects the camera position to the position on the screen. The ray will also intersect with your transformed plane. Then you just need to figure out the line-plane intersection: en.wikipedia.org/wiki/Line%E2%80%93plane_intersection

homogeneous coordinates, you actually needs more than a 3x3 because matrixes can't represent linear offsets.

en.wikipedia.org/wiki/Homogeneous_coordinates

so you pad a 3d coordinate with a 1 to make a 4d coordinate [x, y, x, 1]

That 1 gets uses as the offset in the resulting math.

en.wikipedia.org/wiki/Camera_matrix

ok for projection from a screen space to 3d space the homogeneous coordinates are the pixels, so from what you would naturally think of as a 2d coordinate becomes a 3d coordinate [x, y, 1], multiplied by the camera matrix.

After you do math with homogeneous coordinates you often end with a vector that does not have a 1 in the last position, and you have to normalize it (divide everything by the magnitude of the last element) to make the pixel parts usable

homogeneous coordinates are a critical trick in computer graphics/vision

tomdalling.com/blog/modern-opengl/explaining-homogenous-coordinates-and-projective-geometry

Its something I would not think of myself but it transforms projective geometry (I assume thats what you are doing) into a linear space and makes all the matrix math work, which wouldn't the naive way.

E.g. you can't translate a 2d point with a 2x2 matrix, a 2x2 matrix center's rotations, scales and shears around the 0,0 point. So lots of trivial movement like panning the canvas cannot be represented.

you can translate a [x, y, 1] point with a 3x3 matrix if you normalize it after. Thats the homogeneous coordinate trick that underpins pretty much all computer graphics.

Thanks for the pointers! I’ve noticed through trail and error that transforming the point on the screen by the inverse of matrix of the transformed plane works for when the planes are parallel (e.g. translation, scale, rotation around the z-axis)

@John Christensen i guess the camera is just orthogonal to the screen? In this case it’s not really something you interact with like you might in a video game

three.js has to do these calculations a lot if you are looking for code. Here is someone asking a similar question

discourse.threejs.org/t/projecting-an-image-on-a-perspective-wall-in-another-image/35825

Thanks for the pointers Tom Larkworthy ! Homography has popped up and ive yet to wrap my mind about it 😅

But ill try again now, its reassuring to know that’s the right direction. im still pretty new to the terminology so its been hard to search for these things

its really just adding a 1 to the your coordinates. in 2d [x, y] is [x, y, 1]. In 3d [x, y, z] is [x, y, x, 1], and that adds a handhold for your spatial transform matrices to handle translation

so a 2d translation transform becomes becomes something like

[

[1, 0, dx]

[0, 1, dy]

[0, 0,  1]

]

its good to do the steps manually and you will see how the extra 1 hits the last column and manipulates the real coordinates directly.

The resultant of multiplying [x, y, 1] with the above transform is

[x + dx, y + dy, 1] i.e. its original coordinate + dy, dy.

happy to collaberate with you on getting what you need done, I am quite comfortable with this area of math but I probably can't write anything up until the weekend

const projectedPoint = matrix.inverse().transformPoint({x: pointerEvent.pageX, y: pointerEvent.pageY })

This is what i currently have, where matrix is a DOMMatrix of the transformed plane. From what i can tell this only works when the planes are parallel to each other

The DOMMatrix constructor creates a new DOMMatrix object which represents 4x4 matrices, suitable for 2D and 3D operations.

so if you are working with 3d spatial transforms, you need a 4x4 matrix which is instanciated with 16 elements

Appreciate it Tom Larkworthy 😄

I will play with it later, I did not know the DOM had these features!

You've asked ChatGPT, right?

Right?

Duncan Cragg yep, tried coercing it into knowing what we need and no luck, it was very confidently incorrect, many times 😛

Got it working for affine transformations! now to figure out perspective...

If anyone wants to see this working its at folk.systems/canvas/space/space-transform

Really struggling to get perspective working, I think there's some important gaps in my knowledge somewhere

kinda raw I can try to refine it at the weekend

observablehq.com/@tomlarkworthy/perspective-transform

(suggestions welcome if you correct the katex)

this screenshot demonstrates its a perspective transform as the back side is fits inside the front side

Oh I just understood your diagram as trying to ray cast. from pixel coords to intesect one of the planes

took a few goes but I added ray casting calculations observablehq.com/@tomlarkworthy/perspective-transform#projectedIntersection

Here's a demo drawing some rope between shapes on 2 transformed planes, and also dragging shapes around (which is where raycasting comes in)

the missing piece is handling of CSS perspective... Tom Larkworthy it seems like you may have figured out some (or all?) of that problem, gonna try and grok the observable you linked, would love to figure out how to account for perspective here

yeah the raycasting works under perspective. I will try to get the katex formulas reactively linked to the math to make it easier to follow over the weekend. If you need some more degrees of freedom added LMK what you would like.

yeah if you were able to add a visual for some transformed plane to see it in action that would be super useful

Christopher Shank if I recall CSS behaviour correctly, then your code should also account for element position before applying matrix inverse. CSS transforms operate in element's local coordinate space (transform origin, specifically). It's probably just about subtracting origin from pageX and pageY.

ok added a 2nd camera that orbits the cube so you have a better angle of the intersection ray. Also fixed the domains and aspects of the plots so the FoV parameter properly zooms