A whale of a Demo

In this post, we’ll break down how we created the above demo, from its simplest form, to a jaw-dropping semi-immersive experience.

By Michael Gutensohn

A 3D whale breaking out of water, on a wall

In this post, we’ll break down how we created the above demo, from its simplest form, to a jaw-dropping semi-immersive experience.

Responsiveness

Right off the bat, Developers can create responsive 2D UI without learning an entirely new syntax. If you know how to use HTML & CSS, you know how to use mrjs.

<mr-panel class="columns">

    ...

    <mr-div class="information">
        <mr-text class="title">Humpback whale</mr-text>
        <mr-text class="animal-order">Megaptera novaeangliae</mr-text>
        <mr-text class="description">The humpback whale is a species of baleen whale. It is a rorqual (a member of the family Balaenopteridae) and is the only species in the genus Megaptera. Adults range in length from 14-17 m (46-56 ft) and weigh up to 40 metric tons (44 short tons).</mr-text>
    </mr-div>

    ...

</mr-panel>
.columns {
    display: grid;
    grid-template-columns: 1fr 1fr;
    height: 100vh;
}

.information {
    display: flex;
    flex-direction: column;
    align-items: center;
    justify-content: end;
    gap: 20px;
    ...
}

Responsiveness is a key element of any UI library. If your app can only run on one platform, the return on investment and effort is pretty low. From the beginning, we’ve invested much of our engineering effort to ensure that apps built using mrjs apps can adapt to any screen.

Code on GitHub

Depth & Interaction

Building off the last demo, We ease into spatial UI by introducing the <mr-model> tag and the data-comp-animation attribute, which gives access and control to a given 3D model animations.

<mr-model src="./assets/whale.glb"></mr-model>

Which can be loaded using vanilla JavaScript.

let model = document.createElement("mr-model");
model.setAttribute("src", "./assets/" + fish.model);
model.object3D.visible = false;
model.dataset.name = fish.name;
model.dataset.rotation = fish.rotation;
model.dataset.position = fish.position;

model.onLoad = () => {
    model.components.set('animation', fish.animation)
}

Object.assign(model.style, {
    scale: fish.scale
})

document.querySelector("#models").append(model);

This same app looks great in a headset, and mrjs has hand and controller interaction built in, so there’s no need to learn how to incorporate touch, or point and pinch.

Code on GitHub

Laying it all out on the table

With our third demo, we make the short leap from panels to true mixed reality. This process is relatively simple using the data-comp-anchor attribute, we can pin content to our environment, in this case we anchor an aquarium to a table. We also introduce mr-volume, an element that fits to the plane it’s anchored to, and keeps spatial content contained within its boundary.

<mr-volume
    id="volume"
    data-comp-anchor="type: plane; label: table;">
    <mr-aquarium id="aquarium"></mr-aquarium>
    <mr-entity id="aquarium-models"></mr-entity>
</mr-volume>

You’ll notice we have an element called mr-aquarium. This is a simple example of how to extend mrjs by creating your own custom elements.

class MRAquarium extends MREntity {

    constructor() {
        super()

        const geometry = new THREE.BoxGeometry(0.99, 0.99, 0.99);
        const material = new THREE.MeshPhongMaterial({
            color: '#0235ff',
            side: 2,
            transparent: true,
            opacity: 0.2,
            specular: '#7989c4',
            clipping: true
        })

        this.mesh = new THREE.Mesh(geometry, material)
        this.object3D.add(this.mesh)
    }
}

customElements.define('mr-aquarium', MRAquarium);

Those familiar with THREE.js will notice that we have an object3D that can be manipulated using the THREE.js API.

Code on GitHub

Breaking Down Barriers, One Wall at a Time

In our final demo, we finish off with a bang by creating a more complex scene using the same tools of the last demo, and adding in skyboxes, plane occlusion, spatial audio, and shaders.

<mr-entity id="ceiling" data-comp-anchor="type: plane; label: ceiling;">
    <mr-skybox
        data-rotation="0 0 0"
        src="./assets/ocean.png"></mr-skybox>
    <mr-model
        id="whale"
        data-position="-6 -2.7 -1"
        data-comp-animation="clip: 0;"
        src="./assets/whales/whale_circle_2.glb">
    </mr-model>
</mr-entity>

Let’s start with skyboxes, mr-skybox is a useful tag that allows you to create semi-immersive and fully immersive experiences using 360 images or cube maps.

We also set the position of the whale model using the data-position data attribute. This can also be set from JavaScript using the dataset API.

model.dataset.position = "-6 -2.7 -1"

Creating a Semi-immersive Effect with Occlusion

On its own, a skybox would fully immerse you, removing you from your world and planting you in another. This doesn’t quite fit the spirit of Mixed Reality, we want to mix realities, not replace them.

To achieve this, we use scene occlusion. By default, mrjs utilizes the WebXR API to detect surface planes (walls, ceiling, floor, tables), and creates occlusion planes to block off any content underneath, above, or beyond the boundaries of the room.

Occlusion can be disabled using the occlusion flag in the data-comp-anchor attribute.

Creating a more Complex Scene and Adding audio

Using the same simple syntax we’ve introduced, we can create more complex scenes.

<mr-entity id="wall" data-comp-anchor="type: plane; label: wall; ">
    <mr-water id="water"></mr-water>
    <mr-model
        data-position="0 0 -15"
        data-comp-animation="clip: 0; action: play;"
        src="./assets/hammerhead/hammerhead_circle_swim1.glb">
    </mr-model>

    <mr-entity
        id="whalesounds"
        data-comp-audio="src: ./assets/whales/audio/whale_sounds.mp3; loop: true;"
        data-position="0 0 -10"></mr-entity>

    <mr-model
        data-position="0 0 -0.5"
        style="scale: 0.05"
        data-comp-animation="clip: 1; action: play;"
        src="./assets/koi/koifish.glb">
    </mr-model>
</mr-entity>

In less than 20 lines of HTML, we’ve added a custom element, 2 animated models, and spatial audio. Setting scale with in line styling, and position, audio, and animations using data attributes, all anchored to the wall closest to the user. This is all hidden until we toggle the occlusion flag, which can be done using our JS components API. We can also play the audio and animations.

wall.components.set('anchor', {occlusion : false})
whalesounds.components.set('audio', {state: 'play'})
whale.components.set('animation', {action: 'play'})

Adding a Little Wonder

Finally, we cap off the experience by creating another custom element with a Water shader, giving a more immersive, under sea experience.

class MRWater extends MREntity {

    constructor() {
        super()

        const waterGeometry = new THREE.PlaneGeometry(0.9, 0.9);
        this.water = new Water(waterGeometry, {
            color: '#00ccff',
            scale: 0.25,
            flowDirection: new THREE.Vector2(0.5, 0.5),
            normalMap0: new THREE.TextureLoader().load('./assets/Water_1_M_Normal.jpg'),
            normalMap1: new THREE.TextureLoader().load('./assets/Water_2_M_Normal.jpg'),
            textureWidth: 1024,
            textureHeight: 1024
        });

        this.water.material.clipping = true;
        this.object3D.add(this.water);
    }
}

customElements.define('mr-water', MRWater);

This element differs only slightly from our previous one, adding in the water mesh we borrowed from THREE.js and optimized for MR.

Wrap up

In this tutorial, we created 4 demos, each building off the last. Creating a responsive app, and added 3D models, a tabletop aquarium, and a window into the under sea world.

The rest of the code can be found on GitHub.

We’re constantly making improvements to mrjs, so be sure to star the repo on GitHub and join our discord for the latest updates.