Learn By Doing #1: Forced Perspective in three.js

Today's goal will be to obtain a basic functioning understanding of Three.js by completing a coding challenge. The challenge is to create a forced perspective illusion similar to the game Superliminal. In Superliminal, the player can pick up objects and resize them by moving them closer or further away using the player camera. This creates the illusion that the object is either larger/smaller or further/closer than it actually is. The player can then use these objects to solve puzzles and progress through the game.

Table of contents

• 0. Demo
• 1. Setup
• 2. Camera Controls and Player Movement
• 3. Picking up the Object
• 4. Rotating the Object
• 5. Scaling the Object
• 6. Push Object to Background Wall
• 7. Conclusion
• Appendix
  • PivotPoint Concept in p5.js
  • Different Approaches

0. Demo

Since our primary goal is to learn enough Three.js to form a functional understanding, we will not be focusing on creating a full-fledged game (If making a Triple-A game from scratch is what you were going for, Three.js may not be what you were looking for anyway). Instead, we will create a simple scene with a single object that we can resize by moving it closer or further away, creating the visual illusion that we desire. We will also add a wall to the scene to give us a reference point to compare the object's size. Here is a simple functioning demo of what we hope to achieve.

Instructions:

• Click into the scene to start capturing the mouse
• Hit 'Escape' to release the mouse
• Use the 'W', 'A', 'S', 'D' keys to move the camera
• Hold down the left mouse button to pick up the object, and release to release it
• Move the player around while holding the left mouse button to see the effect

🔗 Check out the Demo here!

1. Setup

Let's begin coding. We will first bootstrap our Three.js project by following the official 🔗 Installation Guide from Three.js. Three.js offers a few ways to get started, but we will be using the npm method (Option 1). Using the CDN method with plain HTML is also viable, but we will not be using it in this tutorial, due to the undeniable conveniences that Vite and npm puts on the table. This means that you'll need Node.js and NPM installed.

1. Create a new directory for your project and navigate into it. Then, create these two files:

2. Install Three.js and Vite using npm:

# three.js
npm install --save three
 
# vite
npm install --save-dev vite
 
# To start the development server
npx vite
 
# To build the project (optional, run after project is done)
# npx vite build
# npx serve dist

.
├── index.html
├── main.js
├── package-lock.json
└── package.json
 
1 directory, 4 files (excluding node_modules)

3. Creating the Scene

We will now create a basic scene with the ball and the wall. The code below is just a modified version of the boilerplate detailed in 🔗 Three.js > Creating a Scene. We've modified the boilerplate starter code to include the ball (SphereGeometry) and wall (PlaneGeometry) objects.

You should now see this in your browser.

2. Camera Controls and Player Movement

Next, let's add 🔗 Pointer Lock Controls to our scene. This will allow us to look around by moving our mouse cursor.

Great! Now, I want you to go to the browser and click on the screen. You should now be able to control the camera by moving your mouse around! 🎉 Next, let us work on the player movement. We will add the ability to move the player forward, backward, left, and right using the 'W', 'A', 'S', and 'D' keys, respectively.

First, we create state to keep track of which keys are currently being pressed.

The reason why I chose to store these states as properties of an object instead of just having them as individual variables extends beyond just organization and readibility. I did this with foresight, as I anticipate that we will be writing code in other files that will need to access these states. By storing them in an object, we can pass this object to other functions and files, and they will be able to modify the states as well. This is a concept known as pass by reference. Check out this article for more information: 🔗 Pass By Reference.

Tldr, we will use this mechanism to modify the state properties from inside functions of many different files, and the changes will reflect in every file.

Next, we will create a new file called camera.js in the same folder as main.js:

Remember when I was talking about pass by reference? This is where it comes into play. We are passing the camera and state objects to the updateCameraPosition function. This function will then modify the camera object's position based on the state object's properties, and it all works even if the code were refactored into multiple different files. This is a very common pattern in programming, and it is a good practice to follow. It makes your code more modular and easier to maintain. This way, we can refactor the code that handles the camera movement logic to its own file, and not blow up main.js with too much code.

Also, I just want to note the use of the Cross Product to compute the normal of the plane which is formed by the camera's forward and up direction. (The 'normal' is the vector which is 90° to the two vectors that form the plane). Here is a quick illustration:

Just in case that wasn't enough, here is 🔗 Mechanics Map (Open Textbook) > Cross Product, that's where I got the original diagram from.

Lastly, notice that I return cameraDir on the last line of this function. That's just a convenient way to get the camera's direction vector so that we can reuse it in other parts of our code.

Great! Now we have the logic to update the camera's position based on the state object. We'll just need to create the keydown and keyup event listeners to update the state object itself. Create a new file called eventListeners.js:

You will now be able to move your player around using the 'WASD' keys! Finally, we'll tweak the index.html file to add a crosshair to the center of the screen:

Try moving your player around now using the 'WASD' keys! (You may need to click on the screen to lock the cursor first)

3. Picking up the Object

Next, we will work on the ability to pick up the object by holding down the left mouse button. In order to interact the object, we first have to determine when the object lies under our cursor crosshair. To do this, we will make use of 🔗 THREE.Raycaster().

Shameless self-plug to another relevant tutorial article: 🔗 Raycasting to cover the front or back surface of an object

We will shoot a ray out from our camera into the world, and detect if the ray intersects with our target object (the ball). If it does, we will bring the object closer within a certain hard-coded distance to our camera (say, 1 unit, for example). However, we must take care to translate/move the object towards our camera in a careful fashion, by taking into account the exact point where our ray first intersects the object. We will call this point the "pivot/handle". In the diagram below, this pivot point is highlighted by a red dot on the surface of the ball object.

Observe that we cannot simply 'teleport' our object closer to the camera by setting the object.position, because that updates the object's center position directly. Imagine grabbing a hold of the object's top right corner, and having the object snap abruptly to the center of your crosshair. That would indeed be quite a jarring experience.

We would instead have to first anchor our object to the red pivot/handle point, and then move the red pivot point to our target location. Then, the object will 'follow' the pivot point to the target location smoothly, preserving the relative distance between the crosshair selection and the center of the object.

In other words, we will create a virtual and invisible pivot point object. Then, we'll 'attach' the ball object to the pivot point object as a child. We will manipulate the position of the pivot point object (parent) and the ball object (child) will inherit the change. This way, we avoid directly modifying the position of the ball (we instead modify it through the parent 'handle' object). If you're not familiar with how Parent/Child relationships work in three.js, I strongly recommend reading 🔗 SBCode.net > Object3D Hierarchy (it even contains an interactive demo to illustrate how changing a parent's position will affect the descendants!).

In the first frame that the user holds down the left mouse button, we will use a THREE.Raycaster to send out a ray from the camera into the center of the screen (that's where the cursor/crosshair is located). If that ray intersects the ball, then we will 'set' the anchor position to be the intersection point of the ray with the ball (this will be the ray's entry point on the ball's surface).

Then, we attach the ball object to the anchor object (called the pivot parent - this is the invisible 'virtual' handle, represented by the red dot in the diagram above). We will then reposition the pivot parent to the coordinates of the point 1 unit away from our camera (using camera.position and cameraDir, we compute the position vector oneUnitAwayFromCamera). Clicking on the ball will look like the ball is pulled towards the camera, and that is because we have not implemented scaling the ball proportionally yet.

However, the crosshair's location on the ball will be preserved. If you initially clicked on the ball's top right corner ⌝, the ball will move towards from the camera in such a way that the top right corner ⌝ will still be under the crosshair post-repositioning. Without using this 'pivot point' method, we would've had to reposition the ball by directly setting the ball's Object3D.position property, which would have centered it directly under the crosshair. This is undesireable.

In the following frames, we will repeatedly recompute the target position oneUnitAwayFromCamera, and reposition the pivot point there, until the user releases the mouse. And when they eventually click and hold the ball again, we will repeat the same process, resetting the anchor/pivot point to the intersection point of the ray with the ball every time.

From the demo above, observe that the crosshair turns from white to blue when the left mouse button is held down (apologies if it's tiny for smaller screens). Picking up the ball works, as far as moving the ball together with the camera goes. However, the ball still appears to snap forward (towards the camera). We will fix this abrupt movement soon, I promise. But for now, we have another challenge to tackle.

Observe this next demo closely:

Notice that I am not moving the camera's position with 'WASD' in this clip. Instead, as we rotate the camera, the ball appears to swivel around the pivot point.

I've given you a little help by drawing a blue bounding box around the invisible pivotParent object.

This isn't quite the desired effect that we were looking for. We want to simulate actually picking up the object, meaning the object shouldn't appear to be swiveling around the pivot point like that.

Extra exercise: For the hungry learners, I encourage you to add some code to draw a helper indicator point by adding another instance of Three.Object3D to the scene (could be any geometry, like Sphere or Box), and repeatedly set this object's position to mirror pivotParent.obj.position on every frame! This will help you visualize the pivot point's position in the scene, and see where on the ball's surface does it land. This way, the pivot point will be visible to you, and you can see how it moves around as you move the ball.

4. Rotating the Object

I have created a diagram to illustrate the concept of counter-rotation. The key concept to understand, is that as we rotate the camera angle by $\theta$ degrees, we want to apply the same amount of rotation to the object in the same direction to preserve the amount of relative rotation.

Think of it this way: the moon always shows the same face to the Earth, because it rotates at the same rate that it orbits the Earth. This is called synchronous rotation. This is also where the notion of "the dark side of the moon" comes from. The moon does have a 'dark side', but it is not always dark. It is just that we never see it from Earth, because it is always facing away from us.

Here comes the (only very slightly) tricky part: ✨ Quaternion ✨ rotations. 🔗 Quaternions have become the industry standard way of representing and dealing with rotations and orientation in 3D space. This is because they do not suffer from 🔗 Gimbal Lock, which is a problem that 🔗 Euler angles have.

If you're not familiar with quaternions, do not fret. I have seen many developers who have been working with various 3D engines for years, and still do not fully understand the intricacies of quaternions (myself included, most definitely). If you are interested, 3b1b has an amazing video on 🔗 Visualizing Quaternions with Stereographic projection. But fair warning: it is not for the faint of heart, for it involves mind bending sorcery like visualizing 4D numbers (humans struggle a lot with this).

☁️ Here's the silver lining: We can treat quaternions the same way we treat transformation matrices, as they share many similar properties. Like matrices, quaternions can be multiplied to combine rotations.

Note: while both can rotate objects, quaternions apply rotations through quaternion-vector multiplication (sandwich multiplication), rather than standard matrix multiplication. But in Three.js, we do not need to worry about any of these details, because we can use the THREE.Quaternion class to handle all the quaternion math for us.

Every THREE.Object3D object has a .quaternion property, which is what we will use to manipulate the 3D orientation of that object. I will define some very light math to illustrate this concept. Ideally, you have a little bit of experience with basic linear algebra. If not, you may still be able to follow along, but the intuition will feel more foreign. Otherwise, feel free to skip this section.

I'll define $x_{\text{anchor}}$ as the orientation of the camera during the instance the object is picked up. This is the exact moment when we set the anchor point. I'll further define $x$ as the orientation of the camera at any given frame after the object has been picked up (the anchor has been set in a previous frame).

In order to go from the initial orientation $x_{\text{anchor}}$ to the current orientation $x$, we need to apply a rotation $A$ to the object. This rotation $A$ is same the rotation that we need to apply to the object in order to ensure that the same side of the object always faces the camera (remember the previous "dark side of the moon" example: the moon always rotates to face the Earth on the same side).

We denote $q_{\text{anchor}}$ as the quaternion that describes the ball's orientation at the moment pivotParent is anchored, and $q$ as the object's orientation at any given frame after it has been anchored, picked up, and is being dragged around. It is necessary to capture $q_{\text{anchor}}$ because the object may already have a initial rotation prior to being picked up, and we want to apply the camera's rotation on top of this initial rotation.

The equations below illustrate:

• How $x_{\text{anchor}}$ becomes $x$.
• How we can compute the transformation quaternion $A$.
• How we can apply $A$ to the object's orientation quaternion $q_{\text{anchor}}$ to get the object's new orientation quaternion $q$.

A good thing to understand is that even though the camera and object orientations are represented as quaternions, the rotation transformation, $A$, is also a quaternion by itself (this resembles transformation vector in linear algebra, and indeed it can be helpful to think of quaternions in a similar way for simplication and familiarity. Though one must be careful - quaternions behave distinctly from vectors in certain cases).

Try and get used to the notion of using $A \cdot z = z'$. For those new to linear algebra, think of it the same way you think of functions in mathematics. It is equivalent to saying $A(z) = z'$, where $A$ is a function that applies a rotation or transformation to $z$ and returns $z'$.

Summary:

• We use the camera rotation quaternions $x$ and $x_{\text{anchor}}$ to compute the transformation quaternion $A$.
• Then, for all subsequent frames after the object has been picked up (and the anchor has been captured), we apply the transformation quaternion $A$ to the object's orientation quaternion $q_{\text{anchor}}$ to get the object's new orientation quaternion $q$.
• $x_{\text{anchor}}$ = PivotPoint.capturedCameraQuaternion, $x$ = PivotPoint.camera.quaternion
• $q_{\text{anchor}}$ = PivotPoint.capturedObjQuaternion, $q$ = PivotPoint.obj.quaternion

Now, let us write code to rotate the ball by as much as the camera has rotated since pivotParent was anchored.

Look at that, we've been able to get rid of all the swiveling! The ball now rotates in sync with the camera, and it looks like we're actually picking up the object.

Here, we stumble into another implicit benefit of using a pivot parent object to handle the ball's movement: the center of rotation is the pivot point itself, and the ball is rotated correctly around this point. Try and imagine how this would've looked if we had directly rotated the ball object itself, with the ball's origin being the center of rotation. If the ball was a perfect sphere, this type of rotation would've accomplished absolutely nothing, as a perfect sphere will look the same from all sides no matter how you rotate it (around its own center).

5. Scaling the Object

Finally, we can work on making the "picking up" look seamless by proportionally scaling the ball with the distance the ball has travelled from its initial position. This will make the ball appear to have not moved, when in reality, we have repositioned it closer to the camera.

Where:

• $h_2$ = PivotPoint.obj.scale
• $h_1$ = PivotPoint.capturedObjScale
• $d_2$ = PivotPoint.getDistance()
• $d_1$ = PivotPoint.capturedObjDistance

Just as before, the center of scaling is also the pivot point (and not the center of the ball itself). It is easy to take such conveniences for granted, and this is why having a pivot parent object is so useful. It allows us to perform transformations around a point that is not the object's center, and this is a very powerful concept in 3D graphics.

After implementing the code changes above, when dragging the ball you will now notice that the ball doesn't appear to have changed in shape, even though it has moved closer to the camera. This is because we are scaling the ball to compensate for the distance it has moved from its initial position.

6. Push Object to Background Wall

Now, instead of leaving the ball in front of the camera, we want to reposition the ball to be touching the wall when we let go of it.

• If this diff view is confusing to look at, try using the "View full changes" link.
• You can browse the full project state including the complete files instead of just the changes using this.

The code above shoots a ray and reflects it off of the wall to capture the point on the backside surface of the ball. Then, we reposition the ball so that its backside surface is touching the wall. However, if you watch the animation below, you will notice that the ball experiences "clipping" still, and the effect is not perfect every time.

The animation also provides a solution to this problem. The orange dot and arrow towards the end of the animation represents this extra "pull back" step that we need to go through. Every time we release the ball, we want to save a snapshot of the orange arrow (the opposite direction of the camera: cameraDir.negate(), because the direction is facing backwards). We save this direction vector into state.pullback.direction.

Then, even if the player's camera has moved away from that spot, we can still continuously pull the ball backwards towards the camera's old direction until the ball is no longer clipping with the wall.

7. Conclusion

That's it!! 🎉🥳👏

If you made it all the way here, really give yourself a nice generous pat on the back. Congratulations.

Appendix

PivotPoint Concept in p5.js

Place your mouse on the canvas, hold the left mouse button, and drag.

Different Approaches

I have a confession to make. Initially, when I first started, I had made many, many separate version of this project, using a completely different approaches. I was trying different things, inspired by other people's attempts like 🔗 Remaking Superliminal's Mindbending Illusions... by Mujj and 🔗 Superliminal Perspective Scaling by BojanV03.

I've tried using virtual objects by making an invisible clone of the ball to compute the intersection points ahead of time before actually moving the real ball. I've tried casting many rays to project a shadow of the ball to the back wall using randomization and/or downsampling, and then computing the ray which has the least distance. Then I used this ray to move the ball backwards without the need of the "pull back" step.

These attemps all worked to a certain degree of effectiveness, but the one I chose for this tutorial is the one that I found to be the most robust and reliable. It is also the one that I found to be the most intuitive to understand and implement.

Believe me, here is a glimpse into one of my previous attempts: