The aixiv column is a column of the machine's heart that publishes academic, technical content. Over the past few years, the aixiv column of the machine has received over 2,000 stories covering top-level laboratories in universities and enterprises around the world, effectively facilitating academic exchange and dissemination. If you have a good job to share, you're welcome to contribute or contact the story. Mailbox for contributors: liyazhou@jiqizhixin. Com; zhaoyunfeng@jiqizhixin. Com
The present paper was prepared in collaboration with researchers from the institute of computing technology of the chinese academy of sciences, the university of california at st. Barbara and kira innovations。
Three-dimensional reconstruction is a classic task of computer graphics and has great utility. In recent years, hidden field approaches, such as neuro-radiation fields, have become widely used in reconstruction missions。
These methods would allow for the reconstruction of scenes with both permeable and smooth reflections without additional input such as a mask. However, the reconstruction of scenes with retroactivity or transparent materials, or even embedded phenomena (i. E., there are other objects within transparent objects, which can also be transparent) is difficult to resolve, both in hidden field and traditional methods。
It is true that there have been some efforts to explore the reconstruction of transparent objects, but these efforts have not been able to rebuild embedded objects, and they need additional input information to reduce the duality of transparent objects, such as masks or requests for scenes to be filmed in particular contexts。
In order to address this problem, a team of teachers from calculus calculus, professor zhang jia of the university of california at santa barbara, and kiri reconstructions have worked together to propose a way to rebuild the embedded transparent objects... Of nested transparent objects with uncoI'm sorry。
This method enables the re-establishment of embedded transparent objects without the need for additional input or special capture scenarios. The study has already been commissioned by acm tog and will be reported at sigraph asia 2024。
Figures 1 and 2 show the re-establishment effect of the nu-nerf set of embedded and transparent scenarios。
Figure 1. Nu-nerf reconstruction of live scenes and their promotion in new scenes
Figure 2. Nu-nerf re-establishment of live and synthetic scenes and rendering of new scenes
Research objectives
Transparent materials such as plastics and glass are one of the most common in everyday life, yet the task of rebuilding them is challenging. The underlying reason is that light is refractioned on the surface of transparent materials, resulting in highly discontinuous surface colour, which is easily confused with the background。
The basic idea behind what has already been done to address this issue is to impose additional constraints on it. Early methods the method of restraint is to capture, by means of special capture devices, information such as dilution and radiance of light, and to use certainty algorithms for reconstruction. There are also computer-based visual and machine-based learning methods that use pre-posterous data to learn how to predict transparent objects from images。
In recent years, the method based on neuro-radiation fields has been used to re-establish by placing behind objects a background with known patterns to obtain the exact position of the light after refraction, using this a priori design loss function。
However, these restrictions are limited by two limitations: (a) the need for additional capture equipment, capture of the environment or input of information (e. G., coverings), which does not allow the user to recaptulate the camera in a random environment; and (b) the assumption that the light is not shielded and reflected inside the object in the process of using a priori, and has only been rereflected twice, making it impossible to rebuild the embedded object。
In response to these questions, the author of the paper presented the nu-nerf. It's a new embedded set of transparent objects to rebuild pipes. As shown in figure 3, the nu-nerf input is a picture of the same set of transparent objects with different perspectives, with the output being the internal, external geometry reconstruction of the scene and a certain degree of decomposition. Reconstructing reconciliation results can be re-advertised by importing rendering software (e. G. Blender, etc.) to digitize real objects。
Figure 3. Nu-nerf internal and external reconstruction and replicating of different types of scene
Research methodology
The nu-nerf pipe consists of two steps. The first step is to rebuild outer geometry. The reconstruction of geometry in the outer layer is a crucial step, as it directly affects the geometry of the inner layer of the second step. The first question to be addressed is the two-dimensional nature of the reflection。
The nu-nerf solution to this problem is simple: the reflection and reflection of the transparent surface of the modelling. The reflect colour is accurately modelled, but the reflect colour is projected directly using an mlp network. The bottom logic of this strategy is that no precise modeling of refraction colours is needed in the reconstruction process, but only an "average " estimate of refractions is provided。
The second step is aimed at a second reconstruction within geometry, taking advantage of the obvious outer layers geometry of reconstruction. This step involves a visible light tracking of geometry in the outer layer and separate modelling of different geometry types in the outer layer (the difference being whether the thickness of the surface is negligible)。
The overall framework of the nu-nerf is shown in figure 4。
Figure 4. Nu-nerf methodology diagram
Outer geometric reconstruction and light model
As figure 4 shows, from left to right, in the outer geometry process, the nu-nerf uses neuro-rendering to rebuild. Physically based rendering is used for each sampling point in the neurotranslation process. Specifically, the colours of surface reflections are divided into reflections and reflections, and they are modelled separately。
For reflection, nu-nerf uses nero 's modelling method to divide the smooth reflection of the traditional rendering equation into two different fractions l and m:
The elements l and m correspond to the nature of the light and the material itself, respectively. Of these, m can be expected, l can be predicted through the network。
In nu-nerf, except for the color loss and eiko common in the neurotranslation methodIn addition to the loss of nal, a loss function has been added: lc (incident light consistency)。
The source of this loss is that nu-nerf uses a neuro-radiation field to approximate the re-establishment of scenes outside the object (e. G., the table where the object is placed, the vision, etc.), while the loss of irradiation coherence encourages the prediction of light in l to match the color of the neuro-irradiation field, thus improving the quality of reconstruction。
As shown in figure 4, for an entry light from the direction of a sample point, the vision colour of the direction is calculated by rendering and the l2 loss is used to encourage the same。
Figure 5
For retrospect, nu-nerf uses a very simple strategy: direct use of neural networks for predictions, compared to the photo-tracking process of previous methods。
The input for this neural network is the coordinates of the sample point p and the light direction ω, the output is the rgb colour. The natural output of the nervous network is a `average' reflection colour because of its inherent low-transform filter nature. The results of the experiment show that this simple strategy will yield better results。
The reconstruction results of the first step, as shown in figure 4, in the second half of the second block, can recreate accurate geometry and light in the outer layer, but the colour of the reflection is blurred because it is predicted directly by the network. As a result of the first phase of reconstruction, new perspectives cannot be directly synthesized. This is to reduce the "cost" of the two。
Visible light tracking and inner geometry reconstruction
Nu-nerf carries out the interior geometry reconstruction, as shown in figure 4, from the left to the right, following the geometry of the outer layer. In this step, the outer geometry of the first step is extracted from the hidden field into a visible grid and fixed。
For each neural rendering sample light, it tracks the intersection of geometry with the outer layer and calculates the direction of reflection into the interior using the refraction law (snell law). Real sampling and rendering within the outer geometry. Please note that during this light tracking, the refractive rate is defined at the outer geometry and obtained through network predictions. Figure 4 also shows visualized images of the refractive rates of learning and shows higher consistency of learning。
Modelling of surface during visible light tracking
As noted in the previous section, the light-tracking process is mainly concerned with the calculation of reflections on the outer geometric surface. However, the snell law applies only in cases where the internal and outer layers are two different materials。
In reality, there may be three different materials at the interface, a typical example being containers. Plastic bottles, glass bottles, such container walls and inner and outer layers of materials are different, and objects such as "containers" are common in real life and therefore need to be considered more closely。
Figure 6. Nu-nerf sun colour a priori
As shown in figure 6, nu-nerf takes into account a variety of different types of materials in surface modelling. Figure 6 (a) depicts the interface normally considered by snell's law, where the sine ratio between the input angle and the output angle is the last of the reflection ratio。
Figure 6 (b) (b2) describes the material of a container of negligible thickness whose method of entry and exit point can be considered the same, so that the margin between the input angle and the output angle depends only on the refraction rate of the internal and outer materials。
The scene in figure 6 (c) is an incoherent packaging with a different input point and an output point method line. For this material, nu-nerf uses a spherical shape to approximate the local geometry of the entry and exit points, the radius of which is determined by the curvature of the object。
Finally, as shown in figure 6 (d), in order to prevent multiple refractions of this thickness material within the wall of the container at the edge of the object at some angles, the nu-nerf removes samples from the edge directly using a mask (mask)。
Experimental effects
To test the validity of the proposed methodology, nu-nerf has experimented on synthetic and actual data sets, where the synthetic data sets have real values (ground truth) and parts of the actual data sets are collected from the web and therefore have no real values. The way to rebuild the experiment is to compare it with the two methods available。
Reconstruction
Figure 7 illustrates the effects of different approaches in reconstructing a synthetic scene, and it can be seen that the previous approach performed better when there was no nesting, but that there was a more severe decrease in performance when the embedded geometry and non-transparent and transparent materials were mixed. Figure 8 figure 9 illustrates the results of different approaches to re-establishing the scene. As you can see, the nu-nerf is able to accurately recreate the outer geometry in a variety of scenarios and, more precisely, the inner geometry
Figure 7
Figure 8
Figure 9
Summary and outlook
The core idea of nu-nerf is to simplify the issue of the reconstruction of complex nested and transparent objects in two steps. In response to the inherent duality of transparent objects, the nu-nerf does not select options for direct light tracking using similar methods previously used, but uses the network to predict the color of the reflection directly, thus increasing geometric reconstruction with the aim of sacrificing the accuracy of the synthesis of new perspectives。
Accurate reconstruction of outer geometry removes most of the duality of the problem and allows for the use of visible light tracking for inner geometry. At the same time, the nu-nerf has proposed an approximation formula for refraction calculations for container-type objects, which allows for the reconstruction of more complex objects without too much decreasing the speed of operation。
