DEM Processing – who’s the beast

(yes, I meant to say Beast).  Today I began to experiment with processing some large raster DEMs, and making contours.  I used ArcGIS, Manifold 9, and GDAL.  This was a nice initial test, and you’ll see some of the results and what I had to wrestle with below.  I think a lot of my friends could offer me advice on how to improve this test, and I would welcome that.

Data

This was a fairly good sized DEM, so it can give us a nice challenge.  The particulars about the DEM are as follows:

Garrett County, MD (2015, 1m DEM) 51187, 60832 pixels (for those of you keeping score, that’s 3.1 billion pixels!) Size: 13.5GB Format: ESRI Coverage

Initial Results

I ran the contouring tests on a 64-bit Dell Precision T1700, i-7 3.4GHz processor with 18GB of RAM.  I used GDAL, Manifold Viewer (Edge version), and ArcGIS 10.3.

GDAL

For GDAL, I simply used the following command to convert the DEM to a shapefile:

gdal_contour -a elev c:\temp\countywide\garrett15_1m\hdr.adf c:\temp\contour.shp -i 10.0

The process completed in 53m.   You’ll notice the CPU’s that were firing.  They weren’t flat, and it seemed that more than one was firing:

Manifold 9

The next test was to use Manifold.  I had Manifold 9 import the DEM data.  It took about 2 minutes to import the 13.5GB raster into Manifold.  From there, I used the contouring transform tool with “Run in parallel” checked.

The process completed in 1561s (24m 1s).  Twice as fast as GDAL.

ArcGIS

Finally, I fired up ArcGIS and used the Contour (Spatial Analyst Tool).  You can see that ArcGIS does not use all the cores on the CPU:

However, I was surprised to see that the contours were completed in 7m 45 seconds – by far the fastest result.  I thought I must have done something wrong, so I ran it again, and instead of creating a geodatabase, I wrote the results out to a shapefile.  The timings were almost identical.

Display and Alignment

The next question, of course, was to see any differences in the results.  As you can see, Garrett County, even at 10m has fairly dense contours.

I decided to bring all three results into a single window.  You’ll see the red, green, and yellow (I changed the line thicknesses).  All  three products yield the same results.  So, they are obviously using a similar algorithm.

What does it all mean?

I was surprised that GDAL was so much slower.  Most people really like the speed of GDAL, but ArcGIS totally crushed it.  Also, seeing Manifold totally pinning the CPU in the parallel processing capabilities, I was surprised that ArcGIS was many times faster.  I wonder if using Manifold 9, rather than Manifold Viewer (Edge) might make a difference.   If anyone wants access to the dataset I used, feel free to email me.

0 views

© 2020, Arthur J. Lembo, Jr.