Difference between revisions of "Structure from motion (SfM)"

From FIThydrowiki
Jump to navigation Jump to search
 
(18 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
=Quick summary=
 
=Quick summary=
[[file:adcp_example_units.png|thumb|250px|Figure 1:]]
+
[[file:sfm_drone_picture.png|thumb|250px|Figure 1: Drone picture taken in a height of 50 m over ground showing the residual flow area in Anundsjø HPP, ice cover and the drone controller (Kordula Schwarzwälder, NTNU).]]
[[file:adcp_qboat.png|thumb|250px|Figure 2: ]]
+
[[file:sfm_evaluation.png|thumb|250px|Figure 2: First run of a structure from motion evaluation in the lab of NTNU. The blue field indicates the camera position (Kordula Schwarzwälder, NTNU).]]
[[file:adcp_wse.png|thumb|250px|Figure 3: ]]
+
[[file:sfm_workflow.png|thumb|250px|Figure 3: Screenshot of the Argisoft Photoscan Workflow.]]
[[file:adcp_workflow.png|thumb|250px|Figure 4: ]]
+
[[file:sf_mesh.png|thumb|250px|Figure 4: Mesh generated from SfM from a section of the residual flow reach of Anundsjø (Kordula Schwarzwälder, NTNU).]]
[[file:adcp_output.png|thumb|250px|Figure 5: ]]
 
  
 
Developed by: Various Companies
 
Developed by: Various Companies
  
Date:  
+
Date: -
  
 
Type: [[:Category:Devices|Device]], [[:Category:Tools|Tool]]
 
Type: [[:Category:Devices|Device]], [[:Category:Tools|Tool]]
 
Suitable for the following [[:Category:Measures|measures]]:
 
  
 
=Introduction=
 
=Introduction=
Acoustic Doppler Current Profiler (ADCP) allows quick, easy and accurate measurements of 3D velocity time series and bathymetry, and computation of discharges in rivers, estuaries, lakes and reservoirs as well as oceans. ADCP data can be used for calibration of numerical models, hydraulic studies (for example, flow field around hydraulic structures), habitat quality assessment and modelling, hydro-morphologic surveys and sediment studies.
+
The technique of structure from motion was developed for the video-game industry to allow fast and easy 3D detection and evaluation of bodies. With this technique it is possible, based on pictures of an object taken with a camera, to combine these pictures. There are several approaches to generate a 3D model from SfM. In incremental SFM (Schönberger & Frahm 2016) camera poses are solved and added one by one. In global SFM (Govindu 2001) the poses of all cameras are solved at the same time.
 
   
 
   
The ADCP is equipped with multi-beams (three up to nine beams, Figure 1), which emit acoustic energy at a known frequency and record the frequency of the acoustic energy backscattered by the particles in the water column. The velocity of the water flow along each beam is computed based on the change in the frequency of the emitted and backscattered acoustic energy, i.e. the Doppler shift. Detailed information on the ADCP working principle and its limitations are described by Simpson (2002). The ADCP beams are positioned to 20 or 30 degree away from the vertical axis. By using a simple trigonometry, 3D velocity components are computed from the Doppler shifts measured with three or four sonar beams. In the latter, a redundant, fourth beam is used to compute error
+
The best result can be achieved when taking pictures from as many positions as possible 360 degree around the object of interest. However, this is not possible for rivers. Depending on the riverbank vegetation and the specific location of each river (such as located in a canyon etc.) only a flight directly above, but with no relevant angle to the sides, might be possible (Figure 1).
velocity, which is the difference between a velocity measured by one set of three beams and a velocity measured by another set of three beams at the same time (Simpson, 2002). The error velocity is used to evaluate the assumption of horizontal homogeneity. The frequency of the ultrasonic sound transmitted by commercially available ADCPs ranges from 30 kHz to 3000 KHz (Simpson, 2002). ADCP can be used at a fixed position, i.e. stationary, or mounted to a tethered boat, manned boat or a remote-controlled boat (Mueller et al., 2013). Non-stationary i.e. moving boat ADCP measurements yield the flow velocity and direction relative to the boat and hence the velocity of the boat should be accounted for by using either bottom tracking or global positioning system (GPS) to determine true flow velocity.  
+
The same problem might appear when using the technique in the lab (Figure 2).
 +
 
 +
For a sufficient positioning accuracy of the results, at least a camera with a high-quality GPS sensor needs to be used. The use of targets or even coded targets to allow a more accurate positioning would be more favourable, however. Without such an accurate positioning of the pictures in the space, the result would not be correct as the cameras could not be located correctly in dependency to each other. Targets on the ground, especially for field measurements, are highly recommended. The position of the targets can be measured with a GPS and the targets can be redetected later in the pictures.  
 +
 
 +
Depending on the software used this process of “re-finding” can be done automatically or needs to be done manually.
 +
In general are there different types of software available, commercial and non-commercial ones. As there is a very fast development and most non-commercial tools need a lot of experience with picture modifications etc. it is recommended, despite the costs, to use a commercial one such as Agisoft Photoscan for instance. In this software, the user can follow relatively easy the workflow, provided by the program (Figure 3), to produce a DEM. It further offers a quite comprehensive manual.
  
  
 
=Application=
 
=Application=
Within the scope of FIThydro, high resolution 3D velocity, as well as bathymetry measurements, have been conducted using an ADCP mounted on a high speed remote-controlled boat at two hydropower plants (HPP) in Switzerland since the beginning of 2018. The models of the ADCP and the boat are River Pro 1200 kHz including piston style four-beam transducer with a 5th, independent 600 kHz vertical beam and Q-Boat purchased from Teledyne Marine, USA, respectively (Figure 2). An external Differential GPS (DGPS) system from A326 AtlasLink (Hemisphere) was used to accurately measure the positions of the ADCP. One set of the battery for the Q-boat allowed us to make measurements for 4 hours up to 10 hours depending on the flow velocity and field conditions i.e. temperature.
+
As for other picture-based evaluation methods, the data processing is the most demanding part of the measurement. When starting with the measurements in the field, it is highly recommended to use targets. The rough workflow is then as follows:
  
Compass calibration and moving bed tests are conducted before each ADCP measurement at the case study HPPs. The Test Case study HPP Schiffmühle is located on the 35 km long river Limmat between in Untersiggenthal and Turgi near Baden in Switzerland (see the Test Case presentation file for HPP Schiffmühle). Two transects of ADCP at each densely spaced cross-section along the river were enough but high accuracy of altitude data was required for the bathymetry measurements at the HPP and in general. The present DGPS system resulted in ±1m of errors in altitude measurements (Figure 3, black line). Therefore, use of a total station, which is time consuming, or real-time kinematic (RTK) GPS is recommended to accurately determine water surface and hence bathymetry (Figure 3, red line from total station measurements).  
+
* Putting targets on the ground with sufficient visibility (from an aerial view). The quality of the results increases in case there are two to three targets visible in every picture. Hence the distribution of the targets depends on the flight height of the camera.
 +
* The camera is usually mounted on a drone, however other constructions as a crane etc. are possible.
 +
* Speed and height over ground depend on the area to be evaluated.
 +
* In case large areas need to be covered an automated flight route programmed for the drone would be useful.
 +
* Starting and landing the drone can be a crucial point, especially in the case this is done automatic and the connection is nut sufficient.
 +
* Further problems can be caused by wind (for the flying performance) and by reflections (for the resulting pictures).
 +
* Always be aware of any kind of regulation, law etc. which might restrict or limit the use and handling of a drone!
  
Furthermore, the test results from the HPP Bannwil located on River Aare in canton Bern indicated that averaging of at least 8 transects or even more at each cross-section is needed to obtain robust and smooth velocity field and accurate discharge data at highly turbulent and 3D flows occurring in rivers, turbine inlet and outlets or other hydraulic structures (see the Test Case presentation file for HPP Bannwil).
+
Once the measurements are done the evaluation is the most crucial point. Depending on the number of pictures evaluated the result quality de- or increases and in a reverse way the computer performance and power needed in- or decreases (Figure 4).
  
The ADCP data from both HPPs Schiffmühle and Bannwil are post-processed according to the workflow sketched in Figure 4 using the software WinRiver II (Teledyne software) and velocity mapping toolbox (VMT, Matlab based software for processing and visualizing ADCP data provided by U.S. Geological Survey). Figure 5 shows the depth-averaged velocities at the HPP Bannwil plotted with VMT. VMT can be used with the output files from Sontek ADCPs. For further data analysis and presentation on the maps like river bed changes, Q-GIS (free software) or ARC-GIS (Commercial software) are also recommended.
 
 
The present system based on the remote-controlled boat platform has advantages over the tethered boat ADCP application. These are less man-power needed, faster and more measurements in a shorter time, no flow disturbance and interference with beams and smoother movement of the boat.
 
  
 +
=Relevant mitigation measures and test cases=
 +
{{Suitable measures for 3D fish tracking system}}
  
 
=Other information=
 
=Other information=
The total costs for the geophone and accelerometer sensors amount to approx. 885-1'330 €. The costs for the field computer, the analog-digital-converter, and the 3G modem are approx. 5'300-6'200 €. The total costs for the Teledyne RiverPro 1200 kHz, Teledyne Q-boat and DGPS from Hemisphere Atlas link amount to approx. 22’000 €, 21’200 € and 3’340 € respectively. The costs of shipping, VAT, some mounting apparatus and long-range radio modem are excluded. For current costs of the equipment, we recommend to ask the corresponding supplier. Note that Q-boat can also house Sontek RiverSurveyor M9. Furthermore, a rugged laptop for field use is recommended.
+
The highest cost is the software license in case a commercial product is used.
  
 
=Relevant literature=
 
=Relevant literature=
*Mueller, D.S., Wagner, C.R., Rehmel, M.S., Oberg, K.A., Rainville, F. (2013). Measuring discharge with acoustic Doppler current profilers from a moving boat (ver. 2.0, December 2013), U.S. Geological Survey Techniques and Methods, book 3, chap. http://dx.doi.org/10.3133/tm3A22.
+
*Alfredsen, K., Haas, C., Tuthan, J., Zinke, P., (2018). Brief Communication: Mapping river ice using drones and structure from motion.
 
+
*Bouhoubeiny, E., Germain, G., Druault, P., (2011). Time-Resolved PIV investigations of the flow field around cod-end net structures. Fisheries Research 108, 344–355. https://doi.org/10.1016/j.fishres.2011.01.010
*Simpson, M.R. (2002). Discharge measurements using a broadband acoustic Doppler current profiler. Open-file Report 2001-1, https://doi.org/10.3133/ofr011.
+
*Buscombe, D., (2016a). Spatially explicit spectral analysis of point clouds and geospatial data. Computers & Geosciences 86, 92–108. https://doi.org/10.1016/j.cageo.2015.10.004
 
+
*Buscombe, D., (2016b). Spatially explicit spectral analysis of point clouds and geospatial data. Computers & Geosciences 86, 92–108. https://doi.org/10.1016/j.cageo.2015.10.004
<b>Links to the suppliers of equipment:</b>
+
*Cunliffe, A.M., Brazier, R.E., Anderson, K., (2016). Ultra-fine grain landscape-scale quantification of dryland vegetation structure with drone-acquired structure-from-motion photogrammetry. Remote Sensing of Environment 183, 129–143. https://doi.org/10.1016/j.rse.2016.05.019
 
+
*Flammang, B.E., Lauder, G.V., Troolin, D.R., Strand, T.E., (2011). Volumetric imaging of fish locomotion. Biol. Lett. 7, 695–698. https://doi.org/10.1098/rsbl.2011.0282
*Teledyne Marine, ADCP RiverPro: http://www.teledynemarine.com/riverpro-adcp?ProductLineID=13
+
*Govindu, V.M. (2001). Combining two-view constraints for motion estimation. IEEE Computer Society Conference on Computer Vision and Pattern Recognition
 
+
*Koci, J., Jarihani, B., Leon, J.X., Sidle, R., Wilkinson, S., Bartley, R., (2017). Assessment of UAV and Ground-Based Structure from Motion with Multi-View Stereo Photogrammetry in a Gullied Savanna Catchment. ISPRS International Journal of Geo-Information 6, 328. https://doi.org/10.3390/ijgi6110328
*Teledyne Marine, Q-Boat: http://www.teledynemarine.com/Lists/Downloads/Q-Boat_1800_Datasheet.pdf
+
*Kothnur, P.S., Tsurikov, M.S., Clemens, N.T., Donbar, J.M., Carter, C.D., (2002). Planar imaging of CH, OH, and velocity in turbulent non-premixed jet flames. Proceedings of the Combustion Institute 29, 1921–1927. https://doi.org/10.1016/S1540-7489(02)80233-4
 
+
*Langhammer, J., Lendzioch, T., Miřijovský, J., Hartvich, F., (2017). UAV-Based Optical Granulometry as Tool for Detecting Changes in Structure of Flood Depositions. Remote Sensing 9, 240. https://doi.org/10.3390/rs9030240
*Hemisphere Atlas DPS: https://hemispheregnss.com/Atlas/atlaslinke284a2-gnss-smart-antenna-1226
+
*Li, S., Cheng, W., Wang, M., Chen, C., (2011). The flow patterns of bubble plume in an MBBR. Journal of Hydrodynamics, Ser. B 23, 510–515. https://doi.org/10.1016/S1001-6058(10)60143-6
 
+
*Lükő, G., Baranya, S., Rüther, D.N., (2017). UAV Based Hydromorphological Mapping of a River Reach to Improve Hydrodynamic Numerical Models.19th EGU General Assembly, EGU2017, proceedings from the conference held 23-28 April, 2017 in Vienna, Austria., p.13850
*Sontek ADCP M9: https://www.sontek.com/riversurveyor-s5-m9
+
*Morgan, J.A., Brogan, D.J., Nelson, P.A., (2017). Application of Structure-from-Motion photogrammetry in laboratory flumes. Geomorphology 276, 125–143. https://doi.org/10.1016/j.geomorph.2016.10.021
 
+
*Paterson, D.M., Black, K.S., (1999). Water Flow, Sediment Dynamics and Benthic Biology, in: D.B. Nedwell and D.G. Raffaelli (Ed.), Advances in Ecological Research. Academic Press, pp. 155–193.
<b>Software for ADCP data analysis:</b>
+
*Schönberger, J.L & Frahm, J.M. (2016). Structure-from-Motion Revisited. IEEE Computer Society Conference on Computer Vision and Pattern Recognition.
 
+
*Tytell, E.D., (2011). Buoyancy, locomotion and movement fishes | Experimental Hydrodynamics, in: Anthony P. Farrell (Ed.), Encyclopedia of Fish Physiology. Academic Press, San Diego, pp. 535–546.
*Velocity Mapping Toolbox: https://hydroacoustics.usgs.gov/movingboat/VMT/VMT.shtml
+
*Vázquez-Tarrío, D., Borgniet, L., Liébault, F., Recking, A., (2017). Using UAS optical imagery and SfM photogrammetry to characterize the surface grain size of gravel bars in a braided river (Vénéon River, French Alps). Geomorphology 285, 94–105. https://doi.org/10.1016/j.geomorph.2017.01.039
 
+
*Westoby, M.J., Brasington, J., Glasser, N.F., Hambrey, M.J., Reynolds, J.M., (2012). ‘Structure-from-Motion’ photogrammetry: A low-cost, effective tool for geoscience applications. Geomorphology 179, 300–314. https://doi.org/10.1016/j.geomorph.2012.08.021
*Q-GIS: https://qgis.org/en/site/
 
 
 
*ARC-GIS: https://www.esri.com/en-us/arcgis/about-arcgis/overview
 
  
 
=Contact information=
 
=Contact information=
  
  
[[Category:Devices]]
+
[[Category:Devices]][[Category:Methods]]

Latest revision as of 07:36, 30 September 2020

Quick summary

Figure 1: Drone picture taken in a height of 50 m over ground showing the residual flow area in Anundsjø HPP, ice cover and the drone controller (Kordula Schwarzwälder, NTNU).
Figure 2: First run of a structure from motion evaluation in the lab of NTNU. The blue field indicates the camera position (Kordula Schwarzwälder, NTNU).
Figure 3: Screenshot of the Argisoft Photoscan Workflow.
Figure 4: Mesh generated from SfM from a section of the residual flow reach of Anundsjø (Kordula Schwarzwälder, NTNU).

Developed by: Various Companies

Date: -

Type: Device, Tool

Introduction

The technique of structure from motion was developed for the video-game industry to allow fast and easy 3D detection and evaluation of bodies. With this technique it is possible, based on pictures of an object taken with a camera, to combine these pictures. There are several approaches to generate a 3D model from SfM. In incremental SFM (Schönberger & Frahm 2016) camera poses are solved and added one by one. In global SFM (Govindu 2001) the poses of all cameras are solved at the same time.

The best result can be achieved when taking pictures from as many positions as possible 360 degree around the object of interest. However, this is not possible for rivers. Depending on the riverbank vegetation and the specific location of each river (such as located in a canyon etc.) only a flight directly above, but with no relevant angle to the sides, might be possible (Figure 1). The same problem might appear when using the technique in the lab (Figure 2).

For a sufficient positioning accuracy of the results, at least a camera with a high-quality GPS sensor needs to be used. The use of targets or even coded targets to allow a more accurate positioning would be more favourable, however. Without such an accurate positioning of the pictures in the space, the result would not be correct as the cameras could not be located correctly in dependency to each other. Targets on the ground, especially for field measurements, are highly recommended. The position of the targets can be measured with a GPS and the targets can be redetected later in the pictures.

Depending on the software used this process of “re-finding” can be done automatically or needs to be done manually. In general are there different types of software available, commercial and non-commercial ones. As there is a very fast development and most non-commercial tools need a lot of experience with picture modifications etc. it is recommended, despite the costs, to use a commercial one such as Agisoft Photoscan for instance. In this software, the user can follow relatively easy the workflow, provided by the program (Figure 3), to produce a DEM. It further offers a quite comprehensive manual.


Application

As for other picture-based evaluation methods, the data processing is the most demanding part of the measurement. When starting with the measurements in the field, it is highly recommended to use targets. The rough workflow is then as follows:

  • Putting targets on the ground with sufficient visibility (from an aerial view). The quality of the results increases in case there are two to three targets visible in every picture. Hence the distribution of the targets depends on the flight height of the camera.
  • The camera is usually mounted on a drone, however other constructions as a crane etc. are possible.
  • Speed and height over ground depend on the area to be evaluated.
  • In case large areas need to be covered an automated flight route programmed for the drone would be useful.
  • Starting and landing the drone can be a crucial point, especially in the case this is done automatic and the connection is nut sufficient.
  • Further problems can be caused by wind (for the flying performance) and by reflections (for the resulting pictures).
  • Always be aware of any kind of regulation, law etc. which might restrict or limit the use and handling of a drone!

Once the measurements are done the evaluation is the most crucial point. Depending on the number of pictures evaluated the result quality de- or increases and in a reverse way the computer performance and power needed in- or decreases (Figure 4).


Relevant mitigation measures and test cases

Relevant measures
Baffle fishways
Bottom-type intakes (Coanda screen, Lepine water intake, etc)
Bypass combined with other solutions
Complete or partial migration barrier removal
Fish guidance structures with narrow bar spacing
Fish lifts, screws, locks, and others
Fish-friendly turbines
Fish refuge under hydropeaking conditions
Fishways for eels and lampreys
Mitigating rapid, short-term variations in flow (hydro-peaking operations)
Nature-like fishways
Operational measures (turbine operations, spillway passage)
Other types of fine screens
Pool-type fishways
Sensory, behavioural barriers (electricity, light, sound, air-water curtains)
Skimming walls (fixed or floating)
Truck transport
Vertical slot fishways
Relevant test cases Applied in test case?
Anundsjö test case Yes
Guma and Vadocondes test cases -

Other information

The highest cost is the software license in case a commercial product is used.

Relevant literature

  • Alfredsen, K., Haas, C., Tuthan, J., Zinke, P., (2018). Brief Communication: Mapping river ice using drones and structure from motion.
  • Bouhoubeiny, E., Germain, G., Druault, P., (2011). Time-Resolved PIV investigations of the flow field around cod-end net structures. Fisheries Research 108, 344–355. https://doi.org/10.1016/j.fishres.2011.01.010
  • Buscombe, D., (2016a). Spatially explicit spectral analysis of point clouds and geospatial data. Computers & Geosciences 86, 92–108. https://doi.org/10.1016/j.cageo.2015.10.004
  • Buscombe, D., (2016b). Spatially explicit spectral analysis of point clouds and geospatial data. Computers & Geosciences 86, 92–108. https://doi.org/10.1016/j.cageo.2015.10.004
  • Cunliffe, A.M., Brazier, R.E., Anderson, K., (2016). Ultra-fine grain landscape-scale quantification of dryland vegetation structure with drone-acquired structure-from-motion photogrammetry. Remote Sensing of Environment 183, 129–143. https://doi.org/10.1016/j.rse.2016.05.019
  • Flammang, B.E., Lauder, G.V., Troolin, D.R., Strand, T.E., (2011). Volumetric imaging of fish locomotion. Biol. Lett. 7, 695–698. https://doi.org/10.1098/rsbl.2011.0282
  • Govindu, V.M. (2001). Combining two-view constraints for motion estimation. IEEE Computer Society Conference on Computer Vision and Pattern Recognition
  • Koci, J., Jarihani, B., Leon, J.X., Sidle, R., Wilkinson, S., Bartley, R., (2017). Assessment of UAV and Ground-Based Structure from Motion with Multi-View Stereo Photogrammetry in a Gullied Savanna Catchment. ISPRS International Journal of Geo-Information 6, 328. https://doi.org/10.3390/ijgi6110328
  • Kothnur, P.S., Tsurikov, M.S., Clemens, N.T., Donbar, J.M., Carter, C.D., (2002). Planar imaging of CH, OH, and velocity in turbulent non-premixed jet flames. Proceedings of the Combustion Institute 29, 1921–1927. https://doi.org/10.1016/S1540-7489(02)80233-4
  • Langhammer, J., Lendzioch, T., Miřijovský, J., Hartvich, F., (2017). UAV-Based Optical Granulometry as Tool for Detecting Changes in Structure of Flood Depositions. Remote Sensing 9, 240. https://doi.org/10.3390/rs9030240
  • Li, S., Cheng, W., Wang, M., Chen, C., (2011). The flow patterns of bubble plume in an MBBR. Journal of Hydrodynamics, Ser. B 23, 510–515. https://doi.org/10.1016/S1001-6058(10)60143-6
  • Lükő, G., Baranya, S., Rüther, D.N., (2017). UAV Based Hydromorphological Mapping of a River Reach to Improve Hydrodynamic Numerical Models.19th EGU General Assembly, EGU2017, proceedings from the conference held 23-28 April, 2017 in Vienna, Austria., p.13850
  • Morgan, J.A., Brogan, D.J., Nelson, P.A., (2017). Application of Structure-from-Motion photogrammetry in laboratory flumes. Geomorphology 276, 125–143. https://doi.org/10.1016/j.geomorph.2016.10.021
  • Paterson, D.M., Black, K.S., (1999). Water Flow, Sediment Dynamics and Benthic Biology, in: D.B. Nedwell and D.G. Raffaelli (Ed.), Advances in Ecological Research. Academic Press, pp. 155–193.
  • Schönberger, J.L & Frahm, J.M. (2016). Structure-from-Motion Revisited. IEEE Computer Society Conference on Computer Vision and Pattern Recognition.
  • Tytell, E.D., (2011). Buoyancy, locomotion and movement fishes | Experimental Hydrodynamics, in: Anthony P. Farrell (Ed.), Encyclopedia of Fish Physiology. Academic Press, San Diego, pp. 535–546.
  • Vázquez-Tarrío, D., Borgniet, L., Liébault, F., Recking, A., (2017). Using UAS optical imagery and SfM photogrammetry to characterize the surface grain size of gravel bars in a braided river (Vénéon River, French Alps). Geomorphology 285, 94–105. https://doi.org/10.1016/j.geomorph.2017.01.039
  • Westoby, M.J., Brasington, J., Glasser, N.F., Hambrey, M.J., Reynolds, J.M., (2012). ‘Structure-from-Motion’ photogrammetry: A low-cost, effective tool for geoscience applications. Geomorphology 179, 300–314. https://doi.org/10.1016/j.geomorph.2012.08.021

Contact information