Microsoft has released a new version of TRELLIS, a generative 3D model framework.
There are several tools of this sort now available, but TRELLIS uses a different approach. They explain:
“The cornerstone is a unified Structured LATent (SLAT) representation which allows decoding to different output formats, such as Radiance Fields, 3D Gaussians, and meshes. This is achieved by integrating a sparsely populated 3D grid with dense multiview visual features extracted from a powerful vision foundation model, comprehensively capturing both structural (geometry) and textural (appearance) information while maintaining flexibility during decoding.”
The framework provides for several different generative activities:
- Text to 3D Asset
- Image to 3D Asset
- Generate 3D Variants
- Editing 3D Assets
Their process involves multiple steps, as shown in this diagram:
Their explanation:
“SLAT marries sparse structures with powerful visual representations. It defines local latents on active voxels intersecting the object’s surface. The local latents are encoded by fusing and processing image features from densely rendered views of the 3D asset, while attaching them onto active voxels. These features, derived from powerful pretrained vision encoders, capture detailed geometric and visual characteristics, complementing the coarse structure provided by the active voxels. Different decoders can then be applied to map SLAT to diverse 3D representations of high quality.”
On their project page, they show multiple examples of TRELLIS capabilities, and the results seem quite impressive. Quite complex images are fed into TRELLIS, and correspondingly complex and seemingly accurate 3D models are produced.
They provide a simple demonstration system whereby you can try TRELLIS using your own images. My quick results were nowhere near as impressive as the ones shown on the project page.
For example, I submitted an image of a pair of pliers, and it produced this:
It seems that it interpreted the shadow of the pliers as part of its structure. I tried several photographic inputs and had poor results, and even an error when using an image of a vase. Images of humans don’t work very well, either.
However, when I submitted generated images of random machines, even complex machines, the resulting 3D models were quite astounding. Here you can see an image I generated for submission.
The 3D model generated for this image turned out to be eerily accurate, and tells me that TRELLIS is trained on machinery images, not organic photographs.
TRELLIS is quite intriguing, and could become the basis for some very interesting 3D generative services in the future.
Via Trellis
