A new approach for generating 3D models from 2D images has been announced: SPAR3D.
2D to 3D generation via AI methods has been a hot topic recently, and there have been multiple approaches presented. There are even several available where you can try this approach yourself.
I’ve tried a bunch, and while it’s quite amazing to have anything generated — I’ll never get over that — the results are very often less than optimal.
Typically, you’ll see two problems.
One issue is that the 2D image doesn’t show the rear of the object, and the generation technique doesn’t properly fill it in. Usually, you’ll get some weird approximation that basically invalidates the model entirely.
The other problem is accuracy. While the obscured part of the object is reasonably generated, the object overall is just not quite what the original 2D image presented. This method also is very time-consuming and expensive.
These two problems are the result of two different algorithmic approaches to 3D model generation: regression and diffusion. Diffusion works very well when generating 2D images but suffers from the accuracy issues described above.
Researchers devised a way around these dual issues by combining them together. They first generate a rough sparse point cloud using a diffusion 3D model. Then, the original image and the generated point cloud are combined with a regression algorithm to create a much more detailed 3D mesh.
This allows the process to complete both quickly and with better quality results. Two different approaches are successfully combined.
There’s another interesting aspect: because the approach produces an intermediate point cloud, it is possible for user edits to take place before the regression step.
This means that a “draft” version of the generation can be presented and edited as a point cloud, and then the edited version is sent off for full meshing.
This is quite an interesting approach that could dramatically improve the 3D models generated in the future.
SPAR3D is the output of a research paper, but no doubt it will inspire those operating 2D to 3D generation services to consider changing their approach to be more like SPAR3D.
Via GitHub
