Let's Dress Up
Using the real-time virtual try-on application, you can turns any images into a 3D asset that allows you to try it on.
Author Kong Chun Yung
Supervisor Dr. Wong, Kenneth K.Y.
About This Project
As e-commerce becomes mature, online shopping becomes a mainstream method to buy clothes. Especially during COVID-19, more and more shops transited to online platform. Fitting is an indispensable part of fashion to check whether the clothes suit you, but it becomes challenging when the clothes are not physically present. A method allowing customers to try the clothes on virtually is essential for online shopping.
Current available services/products in the market has the following problems:
- Single take – you need to take a lot of pictures for different postures
- Using avatars – not realistic enough
- Brand-exclusive – you cannot try other clothes
A better way is needed. By making use of computer vision and augmented reality, this project aims at creating an Android app to make real-time virtual try-on possible for any clothes of any kinds from any brands with just a single image of the desired clothes.
Features
3D Garment Reconstruction
It reconstructs a 3D static mesh from a monocular RGB image provided the alignment information.
Skinning Weights Transfer
It adds bones and weights to the static mesh to animate the reconstructed mesh.
Real-time Body Tracking
It uses the mobile camera to capture user's body movement and predict the user's pose in real-time.
3D Garment Reconstruction
Using a CNN-based encoder-decoder neural network, the 2D RGB image can be transformed into a 3D garment static mesh with textures. Input images with black background and of size 512x512 are preferred. The model will output the result as a point cloud, and reconstruct the mesh using Poisson surface reconstruction.
Skinning Weights Transfer
Using a robust skinning weights transfer algorithm to add dynamics to the static garment mesh reconstructed from the previous module. This allows the garment mesh to move along with the human body and deform accordingly. It preserves the original shape and smooths out loose region at the same time to produce a more realistic animation.
Real-Time Body Tracking
Using the pretrained pose estimation model ROMP, the mobile device can obtain real-time pose information so that the deformation of the garment mesh can be handled accordingly. A RPC server is set up to compensate the lack of computation resources in mobile device and camera captures are streamed to the server.
Technologies / Frameworks