Toronto’s iVirtual wants to make the digital you, and it starts with a GIF

Toronto-based virtual, augmented and extended reality (VR, AR and XR) company iVirtual announced the launch of its social app, Heymoji, which lets users create digital versions of themselves.

And I know what you’re thinking because I thought the same thing: “Didn’t Apple do that with Memoji?”

Or maybe, “What about Snapchat’s Bitmoji?”

When I sat down with the iVirtual team, I asked CEO and co-founder Sherif Khair, as well as CTO and chief scientist Fernando Flores-Mangas, just that.

“I think our big differentiation is [Heymoji] isn’t a cartoon or a cartoon version of you. It’s really you,” Khair said.

“On the social side, we aren’t directly competing because we don’t think they can do what we can do.”

It was big talk for a small Canadian company, but Khair and the team have a broad vision and a working app to back it up.

Heymoji, which is available for free on iOS now, offers a simple premise. Users can take two pictures, and then iVirtual’s software uses that to create a 3D digital model that can be fully animated. Once the company’s machine learning algorithms form your digital self, you can add it to a variety of GIFs — iVirtual calls them tiles — that you can easily share with friends on social media or in messaging apps.

While that might not sound like a lot, this simple social platform is the first step in iVirtual’s big plan.

Heymoji is more than just a tool to GIF yourself

“The social aspect is what makes our brand, or at least the first piece of it, really relevant,” Flores-Mangas said.

“If I sent a GIF of something funny, it’s funny, but if it’s a GIF of something funny that you know is me, like I’m slipping on the floor, it’s actually properly funny. So you take the social aspect of it and imprint it on the product, it has this catch that it wouldn’t normally.”

In other words, iVirtual chose to pursue the social side first so it could show the world what it was capable of.

To understand what iVirtual has really achieved with Heymoji, you have to know where it started.

The Toronto company began with physical booths packed with expensive cameras. People could step inside, and the booth would scan them and create an accurate 3D model.

“[iVirtual] scanned over 10,000 people with booths, learned how the human body looks and also learned that the business model of physical rigs isn’t scalable. It isn’t financial exciting for us or investors,” Flores-Mangas said. So, they ditched the rigs — and it wasn’t easy.

Learning to see in 3D

“The trick is, with the rigs, we knew exactly where all the cameras were pointing, the relationships between each other, the resolutions and the lens. All those things we know. When you submit a picture with your phone, we know nothing,” Flores-Mangas said.

The company had to write roughly 100,000 lines of computer learning code to get around that issue. One of the solutions iVirtual used was structure from motion (SfM) to estimate 3D from a 2D image. Flores-Mangas described it as the notion of how your eyes perceive 3D.

Most people learned as kids that if you hold your thumb up about arm’s length from your face and close one eye, then switch and close the other, your thumb appears to jump to a new position. That’s because your eyes have distance between them and actually see different images. Your brain is really good at interpreting those images and stitching them together so you can see in three dimensions.

Heymoji uses a similar technique to estimate the dimensions of a person when you snap the photo, but it’s not quite as fast as your brain.

Flores-Mangas explained that they exploit a lot of the data they’ve captured over time and do some prior reasoning. For example, if you look at the front of someone’s shirt, you have a pretty good idea what the back will look like. Further, since Heymoji scans humans, it can assume certain things such as the proportions of a person.

Thanks to the tens of thousands of body scans iVirtual has collected, it was able to train the machine learning algorithms and tune them. When a user submits their images to the app, it can pick a generic model from the pre-collected data and fine-tune it for that specific user. It tweaks every detail as much as possible within a computation budget, which ensures the process doesn’t take forever. Finally, the 3D model is added to tiles, which are sent securely to the Heymoji app.

In practice, I found the process took two or three minutes to generate tiles from a scan. While it’s not lightning fast, the upside to performing the tasks off-device is it helps make Heymoji device agnostic.

Works everywhere

Circling back to the Memoji comparison, that relies on tech exclusively found in Apple’s X series phones, like the iPhone X, XS, XS Max and XR. If you don’t have one of those phones with the 3D camera and other sensors, you can’t create Memoji.

“We don’t need an iPhone X, you can do this with an iPhone 5, 6 or 7,” Khair said.

“If you look at our demographic, looking at the kids that are ten or twelve, they’re usually passed down older phones. That was one of the big drivers for us.”

That said, iVirtual does plan to leverage features like the 3D camera sensors in the iPhone X in the future, as the more data the company has, the better the models get.

For the Android users wondering about this app, iVirtual had the app on the Play Store ahead of the launch, but pulled it after discovering some bugs. When I spoke with the team, they were hard at work fixing it so it could go back up before the launch. At the time of writing, the Android app hasn’t returned to the Play Store.

A question of security

With all the ongoing privacy and security issues, concerns over data are turning up with just about every new app. With Heymoji’s use of photography and machine learning, privacy concerns take on new depth, especially when you factor in the 3D model.

When I asked about these issues, the iVirtual team was quick to affirm that it’s working to ensure it protects user data, but also acknowledged it still has areas to improve.

To start, Flores-Mangas said that iVirtual doesn’t share the 3D models it makes. Users only get the pre-rendered GIF created by the process. Further, the 3D model, and the data used to create it, is stored in encrypted spaces.

Looking to the long term, Khair believes people will want to have a virtual version of themselves and will want to be able to authenticate it. Additionally, Khair said that the iVirtual team believes users should own their own data and use it how and where they want, and be able to take it back afterward.

That ability to share their data will be an integral part of iVirtual going forward, according to Khair. He has a broad vision for applications of the company’s technology in e-commerce, gaming and more.

“We’ll have to have a serious conversation with the partners we engage with to ensure the privacy of the 3D models is respected,” said Flores-Mangas.

Looking ahead

Khair’s grand vision is to turn these virtual models in a repository of data stored on your device that you can use when shopping, communicating, gaming and more.

Imagine storing your shoe or shirt size with the virtual model. When you shop for clothes online, you could virtually try them on and see how it looks on your 3D model. When you find something you like, you just tap order — the data handles the rest.

Khair also pointed to massively popular games like Fortnite. He envisioned a partnership that could let players scan themselves into the game, so when they play, they play as themselves. There are broader applications as well, in games like Grand Theft Auto or RPGs like Skyrim.

And, of course, communication is a big area as well, beyond the ability to send custom GIFs. The iVirtual team hopes that their tech can be used in VR meetings in the future or video chats. The team also has plans to build more customization into the Heymoji app, including a ’tile builder’ so users can create their own tiles. Despite the lack of a tile builder at the moment, iVirtual can still assemble new tiles quickly. Within a few hours of leaving the iVirtual office, I received three custom MobileSyrup tiles the team put together.

Despite the potential and the hope iVirtual has, it’s important to remember that Heymoji is really more of a proof of concept at the moment.

“For us right now, it’s a tool to show what we can do,” said Flores-Mangas.

“It’s showing that we’re first at being able to take pictures and make a 3D model of you that’s animated, that looks like you and has your own proportions and such. This is real, you can download the app.”

“We’re in beta mode, in a way.” – Flores-Mangas

“The technology is in its infancy. It definitely still needs to be perfected. And we want to show people we can do this, and as we progress, as we look for investment, we know where we want to put it,” said Khair.

One of those areas is hair. Flores-Mangas noted that when he saw my hair, he was rather concerned — and my hair isn’t even that crazy. Khair said that bald people are the easiest to virtualize, and you can see some of the issues when you look at the GIFs created using my model.

Flores-Mangas took me through the work he’s doing on developing hair models that the system can pull from. It isn’t possible to create a unique hairstyle for every user, but Flores-Mangas is working on a system that can look at someone’s hair, compare it to what’s in the database, and merge some pre-rendered styles to match length and cut better.

For what Heymoji is, though, the fact that it created a model of me by taking two photos is impressive, even if it didn’t get my hair or proportions quite right. And that’s the point: the app is a start, a showcase for what’s to come.

“We’re in beta mode, in a way,” said Flores-Mangas.

“We want to calibrate the expectations of our users. It’s good now, but it’s only going to get better.”

If you want to try out Heymoji for yourself, you can download the app here.