NerfStudio: Game-Changing Software for 3D Capture with Neural Radiance Fields (NeRF)

NerfStudio is going to change many things in the 3D world.

It’s all about NeRF, or “Neural Radiance Fields”. It’s an entirely new way to capture 3D scenes and geometries that has been implemented in only a few commercial systems to date.

Most 3D captures are done using either structured light, depth sensing or photogrammetry approaches, but NeRF methods produce far better results with less effort.

How does NeRF work? It’s one of the many AI-style tools recently emerging. Basically it interprets a series of 2D images to compute the radiosity — the reflectivity of light paths through the scene. This is then used to generate a full 3D scene, even though far fewer images were used as compared to, say, photogrammetry. Another way to explain it is that NeRF “learns” the scene from the images, and can then generate new images of the scene that were not originally provided.

This approach is still so new that it does not yet have a Wikipedia article. However, there is a small draft version in the works.

There are few tools using NeRF. One of them is the Luma AI scanner. It’s a smartphone app that uses NeRF to capture 3D models. I’ve used this particular app many times and the results are uncannily better than any other smartphone-based photogrammetry app.

One of the reasons NeRF hasn’t shown up in many apps is because it’s hard to set up in a program. But that will definitely change with the introduction of NerfStudio.

Researchers from Berkeley have developed NerfStudio, a PyTorch framework for NeRF applications. This development tool should allow many more parties quickly integrate NeRF technology directly into their applications.

What exactly does it do? They explain:

“Nerfstudio provides a simple API that allows for a simplified end-to-end process of creating, training, and testing NeRFs. The library supports a more interpretable implementation of NeRFs by modularizing each component. With more modular NeRFs, we hope to create a more user-friendly experience in exploring the technology.”

NerfStudio enables incredible feats of 3D, as seen in their video:

The software has been generously provided on GitHub, which can be downloaded at no charge and put to immediate use. There’s also a Discord to share questions and answers regarding NerfStudio.

You must have an NVIDIA graphics card installed with CUDA on your computer to use NerfStudio, along with Python 3.8.

The software looks quite easy to use. You simply supply images that are used to train the software’s “nerfacto” base model. After that you can generate point clouds for conversion into 3D models, render new image angles or even video streams.

This tool appears to be quite powerful, and in the hands of competent programmers, we are certain to see an explosion of NeRF-powered 3D capture apps in the very near future.

Via GitHub and ACM