3d reconstruction has been a major area of research in computer graphics and computer visualization。
In short, 3d reconstruction is the restoration of 3d scene structures based on 2d images。
It was reported that after the accident, his facial reconstruction programme was used for three dimensions of reconstruction。
Different technical routes for 3d reconstruction, hopefully integrated
In fact, three-dimensional rehabilitation techniques have been widely applied in the fields of games, films, mapping, positioning, navigation, autopiloting, vr/ar, industrial manufacturing and consumer goods。
With the development of gpu and distributed computing, as well as hardware, deep-seated cameras such as microsoft kinect, the gorgeous xtion and intel's realsense have matured, the cost of three-dimensional reconstruction has tended to decrease。
Operationally, the 3d reconstruction process can generally be divided into five steps。
Step one, image acquisition。
As the 3d reconstruction is a back-up operation for the camera, a first two-dimensional image of the 3d object is required。
This step cannot be ignored, as light conditions, geometry of cameras, etc. Have a significant impact on subsequent image processing。
Step two, camera marking。
This step is to restore objects in space using images taken by cameras。
It is usually assumed that there is a linear relationship between the images taken by the camera and the objects in the three-dimensional space, and this process of solving the parameters of the linear relationship is referred to as the identification of the camera。
Step three, character extraction。
Features mainly include feature points, feature lines and areas。
In most cases, the basis is matched by a feature point and the form in which the feature point is extracted is closely linked to the matching strategy used。
It is therefore necessary to determine which matching method is used when extracting a feature point。
Step four, stereo match。
Stereo-matching refers to the creation of a correspondence between the image pairs based on the characteristics extracted, i. E. The matching of the physical space point with the imaging point in the two different images。
Step five, three-dimensional reconstruction。
With more precise matching results, combined with the internal and external parameters marked by the camera, three-dimensional scene information can be restored。
These five steps, rounded together, can only be designed for a more precise stereovision system with high precision and small error at each link。
In algorithms, three-dimensional reconstruction can be broadly divided into two types, one based on traditional multi-view geometry。
Another three-dimensional reconstruction algorithm based on deep learning。
Currently, thanks to the enormous advantages of cnn in character matching images, an increasing number of researchers are beginning to turn their eyes to three-dimensional reconstruction based on deep learning。
However, most of this approach is one of monitoring learning and relying heavily on data sets。
The collection and labelling of data sets has been a source of problems for monitoring learning and, as a result, the three-dimensional reconstruction based on in-depth learning has been more studied in the direction of reconstruction of smaller objects。
In addition, three-dimensional reconstruction based on in-depth learning, although high, has good accuracy。
However, training models take a great deal of time and the 3d volume layer for 3d reconstruction is very expensive。
As a result, researchers have begun to revisit the traditional three-dimensional approach to reconstruction。
The traditional three-dimensional approach to reconstruction, while deficient in performance, is relatively sophisticated。
A certain integration of the two approaches would then have better results。
No 3d volume layer, no 3d reconstruction
Researchers from the university of london, oxford university, google and niantic (the unicorn company for the study of ar from google) have explored a 3d reconstruction approach that does not involve 3d volumes。
They proposed a simple state-of-the-art multi-view depth estimater。
The multiview depth estimate has two breakthroughs。
The first is a well-designed two-dimensional cnn, which can be pre-tested with powerful images, and which can also obtain a flat-scan character and geometric loss
The second is the ability to integrate key frames and geometric metadata into cost volumes, thus achieving informed deep-level plane scores。
According to researchers, their methods have a clear lead in in-depth estimates over the current state-of-the-art methods。
The 3d reconstruction on scannet and 7-scenes is close or better, but allows real-time low memory reconstruction online。
Moreover, the pace of reconstruction is very rapid, with only about 73 ms per frame。
Researchers believe that this makes it possible to rebuild precisely through rapid and deep integration。
According to researchers, their method is to use the image encoder to extract matching features from reference and source images, then enter them into a post volume, then use the 2d volume code/code decoder network to process the output of post volume。
The study was carried out using pytorch, which was matched with resnet18, and two 40gb a100 gpus, which were completed after 36 hours。
In addition, while the model does not use 3d volumes, it is better than the baseline model in depth projection indicators。
This shows that a well-designed and trained 2d network is sufficient for high-quality in-depth estimates。
Interested readers can read the original paper:
Https://nianticlabs. Github. Io/simplerecon/resources/simplerecon. PDF
It should be recalled, however, that there is a professional threshold for reading the paper and that some details may not be easily noticed。
Let's see what foreign netizens found in this paper。
One of the web users, “stickshipplease”, said that “although the reasoning on a100 is about 70 ms, this can be shortened by various techniques, and memory requirements do not have to run the memory for 40gb, with the smallest model running 2. 6gb”。
Another netizens, irreverenthippie, stated, "please note that this study is still sampled by a deep sensor based on lidar. This is why this method has acquired such good quality and accuracy."。
A comment by a friend named "nickthorpie" was longer, saying, "the strengths and weaknesses of the tof camera are well documented. Tof addressed the problems that had plagued the processing of original images. Of these, two main issues are scalability and detail. It is always difficult for tof to identify small details such as the edge of the table or the pole. This is critical for autonomous or semi-autonomous applications。
In addition, as tof is an active sensor, the mass of the picture is rapidly reduced when multiple sensors are used together, for example in crowded crossroads or self-built warehouses。
Obviously, the more data you gather in a scene, the more accurate the description you create. Many researchers prefer to study raw image data because it is more flexible."。
References:
Https://www. Reddit. Com/r/machinelearning/comments/xbj6cn/r simplerecon 3d reconstruation without 3d/
Https://nianticlabs. Github. Io/simplerecon/resources/simplerecon. PDF
