Link to FWS banner of FWS,NWI,BHA link to FWS link to BHA link to NWI

Technical Help Document #1:

Data Processing, Review and Wetlands Verification Tool

Table of Contents

Getting Started

Large Dataset Problems

Data Processing Steps to Avoid

Data Review and Feedback Tips



Getting Started

The following solutions remedy some of the common problems that have been observed when verifying digital map data.  To prevent these problems from occurring run through this list and verify that your data does not violate any of these situations before using the Wetlands Verification Tool.

File Permissions

The QAQC tool only works on geodatabases that can be edited.  After the data is copied from a CD or DVD make sure the properties to the file are not read-only.  This is done in Windows Explorer by right clicking the file or folder, clicking ‘Properties’ and making sure the ‘Read-only’ box is NOT checked.  Also, if the data is copied to a network drive make sure that drive is shared and you have write access to that drive.  You may need to see your System Administrator to do this.

Ghost Polygons

While editing polygon geodatabases in ArcGIS some edit functions (clip, merge, dissolve, delete, etc.) periodically leave ‘ghost’ polygons.  These ‘ghost’ polygons have an attribute, but do not have any length or area.  This occurs when a polygon is automatically ‘deleted’ by the Arc software when using some of ESRI edit functions.  The deleted polygon retains the attribute but the length and area fields become Null.  Manual deletes by the user remove both the spatial and attribute components of the record. Use of the Repair Geometry tool in ArcCatalog will also correct this problem.

The following is the error message that occurs when running the verification tools on a data set that has polygons with no Shape_Length or Shape_Area

To check for this problem view the table in the preview tab of ArcCatalog.  Right click either the Shape_Length or Shape_Area field header and select ‘Sort Ascending’.  If there are records at the top of the table with no values for Shape_Length or Shape_Area they need to be deleted.  To delete them do one of the following:

Method 1:

  1. Open the geodatabase in ArcMap
  2. Start Editing
  3. Open the attribute table.
  4. Right click either the Shape_Length or Shape_Area field header.
  5. Select ‘Sort Ascending’
  6. Select all records that have ‘0’ for Shape_Length or Shape_Area, and delete them.
    1. Click the gray box at the left side of the table to select a record.
    2. Click and drag down the gray boxes to select more than one record or hold shift and click each record’s gray box that you want selected.
  7. Save edits.
  8. Stop editing.

Method 2:

Compacting Personal Geodatabases

When a lot of edits are done to a geodatabase the tables in Microsoft Access that store the Personal geodatabase can become large.  The edits do not get completely flushed from Access so the geodatabase becomes larger and larger in size.  This is of particular concern with larger datasets as they soon approach the 2GB limit of Microsoft Access.  To try and keep the size of a geodatabase under control it is a good practice to compact it after any intense edit sessions, before running the QAQC tool and before copying the data to a CD.

To compact a Personal Geodatabase, right click on the PGD in ArcCatalog and select Compact Database or follow these steps to add the tool to a menu:

  1. Open ArcCatalog.
  2. Select Tools > Customize.
  3. Select the Toolbars tab on the Customize window.
  4. Check the Context Menus box.
    1. Notice a Context Menus window appears on your screen somewhere.
  5. Select the Commands tab on the Customize window.
  6. In the Categories window on the left, scroll down and click Geodatabase tools.
    1. Notice the commands appear in the ‘Commands’ window on the right.
  7. Click, drag and hold the Compact Database command icon over the Context Menu pop-up window, but do not drop.
  8. The pull down menu will expand, then drag it up to the Database Context Menu and hold.
  9. Another window will expand from the Database Context Menu, drag and drop it to a location on that menu.  Somewhere below the Import option.
  10. Click close on the Customize window.
  11. In ArcCatalog navigate to the Personal Geodatabase you want to compact.
  12. Right click.
  13. Select Compact Database.

Compacting the database will take a few seconds to a few minutes depending on the size of the geodatabase.  To monitor the results of the Compact Database you need to find the size of the personal geodatabase (.mdb) in Windows Explorer before and after the Compact Database is ran. 

Extra Fields

Extra fields can also cause the geodatabase to become quite large.  Everyone may do their processing slightly different and track different things with user defined attributes.  As far as the MGD is concerned the only necessary fields are:  Attribute, and the personal geodatabase fields of ObjectID, Shape_Length, and Shape_Area.  The QAQC_CODE is not necessary before you begin as the first verification check on the Wetlands Verification tool actually builds QAQC_CODE field and populates it.  If it is already there you can leave it.  All other fields can be deleted if they are unnecessary; as they will be deleted before they are incorporated into the MGD.  Deleting extra fields may reduce the size of the geodatabase considerably.  The geodatabase should be compacted again after any of the fields are deleted.

Attribute Field

The Attribution and Verification tools populate and query the standard NWI field named “Attribute”, which is a text data type and has the length of 20.  This field holds the Cowardin Wetland Code and must be in the geodatabase for the tools to work.  If this field is not present you must create and populate a text field named “Attribute” with the length of 20.

QAQC_CODE Field

The Wetland Verification Tool uses a field named “QAQC_CODE” to store the results of the various tests. This field needs to be properly defined and populated in order for the Wetlands Verification Tool to work.  In fact, the field is intended to be modified only by the Wetlands Verification Tool, NOT by users. There have been occasions when users have modified the “QAQC_CODE” field or the field has been compromised by process steps conducted by the user (i.e. conversions between data storage types) or other editing steps.  When this situation occurs the user should simply delete the QAQC_CODE and run the first verification test again. This procedure allows the Wetlands Verification Tool to properly define, populate, and manage the field. 



Large Dataset Problems

Large datasets push the computing capabilities of both the CPU and the software.  The following steps can be followed to remedy some of these problems that occur when running verification on large datasets.  Large datasets include datasets larger than 1GB in size, datsets with more than 50,000 records, or datasets with large complex polygons with high connectivity with other polygons.

Large Geodatabases and Topological Verification

The Wetlands Verification Tool runs topological checks (Improper Island Polygons, and Overlapping Polygons) that were designed for ArcGIS 8.1, which did not support intrinsic topological verification operations.  ESRI added support for topological operations at ArcGIS version 8.3. With ArcGIS version 8.3 or later the user now has the ability to define topological rules such as ‘Polygons Must Not Overlap’.  This functionality is now intrinsic to ArcGIS.

Keep in mind that the Wetlands Verification tool was designed to be used on small datasets (i.e. quad sized).   It is much more efficient to use the ‘Polygons Must Not Overlap’ topology, than run the last two verification checks.  As it currently stands, however, this prevents the user from running the ‘Pass/Fail’ function of the tool as the last two checks are automatically run during the Pass/Fail operation.

Currently, all data checked out from the MGD has a ‘Polygons Must Not Overlap’ topology layer included.  This essentially negates the need for the last two verification checks. However, some data must still be accepted that did not originate as a ‘check-out’ from the master geodatabase. In that situation, the users need to use the Wetlands Verification Tool to check for overlaps.

It is highly likely that future versions of the Wetlands Verification Tool will not include the last two topological checks for the reasons previously described.  In the future, the topological integrity of the data will depend solely on the ‘Polygons Must Not Overlap’ topology layer associated with the checked out data, or that topology layer must be built for any ‘new’ data.

How to build Topology

Topology can only be built in a Feature Dataset that contains the Feature classes that are to be included in the topology. 

  1. Right Click the Feature Dataset that has the wetland polygon Feature class in it.
  2. Select New > Topology…
  3. Click Next in the New Topology Window that pops up.
  4. Enter the name ‘No_Overlaps’ in the ‘Enter a name for your topology’ window.
  5. Accept the default value in the ‘Enter a cluster tolerance:’ window.
  6. Click Next.
  7. Check the wetland polygon Feature Class that will participate in the topology.
  8. Click Next.
  9. Accept the default rank value in the next window.
  10. Click Next.
  11. In the ‘Add Rule’ window the wetland feature class should be the default feature class as it is the only one checked as participating earlier.
  12. The Rule should also default as ‘Must Not Overlap’, if not, select it as the rule.
  13. The ‘Show Errors’ box should also be checked by default.
  14. Click OK.
  15. Click Next.
  16. Click Finish.
  17. Click Yes to validate the topology.

How to use Topology

  1. Open an ArcMap Session.
  2. Add the ‘No_Overlaps’ Topology layer.
    1. This can be found in the Feature Dataset that contains the wetlands polygon Feature Class.
  3. Click Yes, to add the Feature Class that participates in the ‘No_Overlaps’ topology.
  4. The topological errors will be highlighted in red.
    1. When these areas are fixed the red area is removed.
    2. If the red does not disappear, or more polygons have been added to the Wetlands Feature Class, the Topology needs to be validated again.

How to Validate Topology

  1. Add the Topology Toolbar to your ArcMap session.
    1. In ArcMap click Tools > Customize.
    2. In the Toolbars tab check the box next to Topology.
    3. The Topology toolbar appears, and this can be anchored in a convenient location.
    4. Close the Customize window.
  2. Start Editing.
  3. Select any of the three Topology Validation buttons on the Topology toolbar.
    1.  Validates topology in specific areas defined by a user defined selection box.
    2.  Validates topology in current extent.
    3.   Validates topology of entire dataset.

MaxFileLocks Limitations in ArcMap

When working with large datasets ArcMap will reach the default MaxFileLocks limit assigned by Microsoft Windows Operating Systems.  The error message that appears in the ArcMap session is:

AllNWILayers Error: 2147467250 Occurred
File sharing lock count exceeded.  Increase MaxLocksPerFile registry entry.

In order to remedy this situation the following steps need to be followed.  NOTE:  You need to have administrator permissions on your computer to make this change.


Steps to Increase MaxFileLocks

  1. Open Windows Explorer (not Internet Explorer).  Be sure to exit ArcMap.
  2. Navigate to the folder c:\Program Files\ArcGIS\Utilities

  1. Double Click the AdvancedArcMapSettings.exe

 


  1. Choose the Editor Tab Along the Top


 

  1. In the Box to the right of the title ‘JET engine max # of records to calculate:’ change the default number of 9000 to 2,000,000.

 

  1. Hit Apply, Close the window.
  2. Restart ArcMap in order for your changes to take effect.

Data Processing Steps to Avoid

The following processing methods create errors in the data.  These errors cause QA/QC verification to fail upon submission and therefore should not be used.

Geodatabase to Coverage to Geodatabase conversion

Because of significant differences in the way features are stored, geodatabases should not be converted to coverages and then back to geodatabases in order to solve topological problems. The process introduces or potentially introduces a wide range of both geometric and attribute problems that are unwarranted and unnecessary.  The following is one example.

 

The new polygons were automatically created by the coverage model.  The coverage model does not reference default values set in the geodatabase schema, thus all user created fields are not populated.  The fact that the QAQC_CODE field is empty for these ‘new’ polygons prevents the Wetlands Verification Tool from working properly.

Map Topology in ArcMap

The use of topology in ArcGIS should be restricted to the Topology Layer which is created in ArcCatalog.  This Topology Layer does nothing more than identify topological problems and relies on the user to use standard editing tools to correct the problems.  The topology toolbar in ArcMap allows the user to create ‘Map Topology’ and ‘Construct Features’.  This ‘Map Topology’ actually modifies the geometry of the polygons based on the tolerances set and the ‘Construct Features’ tool has been known to duplicate polygons in certain circumstances.  The use of this ‘Map Topology’ should be avoided as it introduces errors into the dataset.

Changing Projections

The QA/QC tools perform three verifications based on area. In order for these tests to be valid, the tools assume a map projection that preserves area.  Many of the wetlands data not checked out from the MGD are stored in a UTM map projection, which does NOT preserve area. Therefore, in order for the verification tools to provide appropriate results, it is recommended that the data be re-projected to the Albers Equal-Area projection used by the master geodatabase.

The spatial reference information is:

Projected Coordinate System:

Geographic Coordinate System:

X/Y Domain:



Data Review and Feedback Tips

Viewing Wetlands, and Reference DRG’s and OrthoPhotos in ArcMap

When reviewing NWI wetlands data it is invaluable to have imagery and USGS quadrangle maps to use as reference material.  This can be achieved without having the data stored locally, but through the use of Web Mapping Services (WMS) and Internet Map Services (IMS).  All that is needed is internet access and some simple customization of ArcMap (verion 9.1 or later) to view Open Geographic Consortium (OGC) layers.

How to Add Wetlands Geodatabase wetlands to ESRI's ArcGIS

How to Add a imported OGC Layer to ArcGIS

  1. Launch ArcCatalog.
  2. Under the “GIS Servers” directory in the explorer window on the left, double click on “Add WMS Server” to open the Add WMS Server dialog box.
  3. For URL of server enter “http://terraservice.net/ogccapabilities.ashx?”.
  4. Click on the “Get Layers” button to view the available layers and then click OK.
  5. The connected WMS server name should now show up under “GIS Servers” in the explorer window in ArcCatalog.
  6. You may now drag the WMS connection into ArcMap to view the layers.

Error Feedback from Reviewer to Image Interpreter

There are some instances when the data being reviewed needs to go through a feedback loop with the image interpreters that did the work.  In some cases a discussion over the phone may not be enough.  It may be necessary for the image interpreters to have the data in front of them.  Instead of mailing the data back to them on a CD the polygons in question can be selected and exported as a Personal Geodatabase or shapefile and emailed to the image interpreters.  Remember any changes the image interpreters make to this ‘exported’ dataset only happen on that stand alone file.  This data can be emailed back to the Reviewer, and the Reviewer needs to make those changes on the final dataset they made the export from.

Exporting to a Feature Class

1.      In ArcCatalog navigate to a folder where you want to store the data.

2.      Right click folder and select New > Personal Geodatabase.

3.      Right click the ‘New Personal Geodatabase’ and select Rename.

4.      Rename to desired name (e.g. NV_errors)

5.      View your wetlands data in ArcMap.

6.      Select the polygon(s) you wish to export to the ‘error’ geodatabase.

a.       If the Verification tool has been run you can select all polygons with a certain type of error.

                                                               i.      Click ‘Selection’ at the top of ArcMap.

                                                             ii.      Click ‘Select by Attributes…’

                                                            iii.      In the pop-up window double-click the QAQC_CODE in the ‘Fields’ window.  This adds it to the Calculations window.

                                                           iv.      Click the ‘=’ sign function button.

                                                             v.      Double-click the desired QAQC_CODE in the ‘Unique values’ window.

Select by Attributes Example

 

                                                           vi.      Notice in example above the selection equation that is built in the lower window will select any polygon from CONUS_wet_poly where the QAQC_CODE equals ‘NNNNNNLNN’.  The 7th letter in this attribute is an ‘L’ which stands for the 7th verification and represents a Lacustrine wetland that is less than 20 acres in size.

1.      For more information on the QAQC_CODE attribute click on the question mark on the Verification Tool under the ‘Codes’ column.

  

 

                                                          vii.      Notice the highlighted ‘Unique value’ in the figure above also has an ‘L’ in it.  The selection equation can be modified to select all polygons with an ‘L’ in the attribute in two ways.

1.      [QAQC_CODE] = 'NNNNNNLNN' OR [QAQC_CODE] = 'NNANNNLNN'

2.      [QAQC_CODE] LIKE '*L*'

 

7.      Once you have the polygons selected that you want to export Right click the polygon layer in the ArcMap session.

8.      Select Data > Export Data.

9.      Accept the default for the Export pull down menu ‘Selected features’

10.  Accept the default radio button selection of ‘Use the same Coordinate System as this layer’s source data.’

11.  In the output feature class window, select the open folder button and navigate to the personal geodatabase you created earlier (e.g. NV_errors).

12.  You can then change the default ‘Export_Output’ feature class name to something more descriptive (e.g. small_L1UBs).

13.  This personal geodatabase of just the error polygons can now be emailed or burned to CD and mailed. 

14.  NOTE:   Any changes to this ‘error’ personal geodatabase will not change the original data you created it from, nor can they be checked back in.  Those changes must be transcribed back to the original data. 

Link to USGS  In Cooperation with the U.S. Geological Survey

Wetlands Home || Privacy, Disclaimer, Copyrights and Logo Use || FOIA || Contact Us


icon