Where’d My Property Go: Finding data during Splits and Merges

One of the challenges when working with geospatial systems is managing the data attributes when the object is split or combined with another. For example, if I have two parcels that are joining, and they have different Assessor Parcel Numbers (APNs), how do I get just one new APN from the 2 previous APNs. And what happens to the area field? Or when I split a parcel, how does the system name the 2 new parcels? More importantly, that about the ID field that serves as a key to link to other databases. The answer is that you can pretty much set it to do what you want. AutoCAD Map incorporated split and merge tools so that you can manage them. The following example should give you an idea of how to get started. The important thing is that like most things GIS, putting some thought into it prior to trying to do the work will give you the best results. In other words, design is very important.

My example will use a proposed forests data set. I have attributes for the proposed name, the area, perimeter, and the area in square kilometers and hectares. During the join, I want to rename to proposed forest, and update the area and perimeter fields.

NOTE:  As I go through this, keep in mind the terminology gets wonky. The same words can be used to describe multiple elements of these objects. For example, the attributes of a feature can be called attributes, properties, fields and columns (to us database geeks), and the properties describing the said attributes, such as field size and type, can all use the same names. So, try not to read too much in the wording and I’ll try to match AutoCAD Map’s terminology.

To set the Split and Merge Rules, I will highlight the target data set in the Display Manager and open the Data Table. 

 Once I get the Data Table open, I’ll select the Options, and select Set Split and Merge Rules.

At that point, I’ll get the Split and Merge Rules dialog box.

At the left of the box, I get a list of all of the feature properties of the selected data set (attributes or database columns). As I select each of these properties, I get the various attributes of that property. It identifies whether the property is an Identifier, the data type of the property, whether it is autogenerated, read-only or nullable. In addition, I can set Split and Merge Rules for each property attribute. Keep in mind, the available Split and Merge Rules are context sensitive based on the data type (it’s a little tough to sum text fields).

The data set I’m using is an ESRI shape file, so there are certain feature properties that are inherent because of the type of data set. The FeatID is autogenerated and read only, so I won’t be able to set any rules for this one.

My ID field is an identifier for the individual forest polygons. I’ll set my Merge Rule to Empty. When merging, I’m going to create a new and distinct record from the previous record. This is a business rule I’ve decided upon so I can keep a history of forest proposals, even if they are not actually implemented.

My area field shows the area of the polygon. I could add the polygons, but for better results I can use an expression to calculate the actual area of the result. To do this, I set my Split rule to Calculation, and select the Expression Builder button (next to the Exporession box). I’ll select the Area2D from the Geometric pull down,

And then Geometry from the Property pulldown.

The resulting expression, Area2D(Geometry) , will calculate the area of the new polygon (if you know the expression, you could just type it in the box rather than going through the expression builder – but if I had done that, you wouldn’t have seen it, right?). That expression will go into both the split and merge rules.
My next feature property is the Perimeter, and guess what? There’s a calculation for that as well. Select Length2D in the Geometric pull down to get this expression:  Length2D(Geometry)
On the Name feature property, I will generally not use the existing names – again a business decision. There are cases where I would want to keep one of the names (Using the FirstSelected or LastSelected rule) or concatenate the two names, just not for this case. In my example, I would need to add the name manually after doing the merge (or split).
My next feature property is AREASQKM, or the area in square kilomters. I can use the same expression as before, but include the conversion to square kilometers, giving me this expression: Area2D(Geometry)*0.00000009290304. Again this will apply to both split and merge.
My last standard property is HECTARES, which is the area in hectares. This will match the previous bit with the appropriate factor:  Area2D(Geometry)*0.000009290304

Of course the last entry is Geometry, and you can’t use rules on geometry,

Adding Custom Linetypes

  My last post was on the linetypes for feature objects in map 3D. There are a lot of options, but it isn’t as open as using AutoCAD linetypes. You can create additional stylization, but it takes rolling up your sleeves. My next post was going to do that, but I don’t have to – Murph did. So check out his post on adding custom linetypes, and I’ll work on something else.

Finding your “Type”

One of the common questions I after folks start working with Feature Data Object connections in AutoCAD Map 3D is how to use interesting linetypes in their maps. To add linetypes to FDO-connected data sources, select the ellipses next to the Thematic Rule in question, and the Style Line dialog box opens. There you can select the thickness, color and pattern for the line.

The pattern selection provides a set of linetypes – not the AutoCAD Linetypes (there is no direct way for users to modify this list, although it can be programmatically edited – more on this another time).  This will provide a number of options and style combinations.
You can also stack line patterns to create more complex styles. For example, many road maps will use a red line with black borders for a highway, sometimes with a back center line or dashed line to show divided highways. This can be modeled by using several lines patterns overlying each other.
To do this, select the composite lines option. The Style Line dialog box will then expand to add a composite style box where you can add multiple line components.

To make the example highway style, add 2 new lines by clicking the New button at the top of the composite box. Select the line at the bottom, change it to a .2 cm thickness at black with a continuous pattern. Change the middle line in the box to a continuous red line with a width of .15 cm. Change the top line to a black line with a dashed pattern at a .1 cm thickness. The resulting line style will be apply to this theme rule throughout the drawing.

 You can have different composite lines for each theme rule, allowing fairly complex maps. Combine these with some annotation patterns and you can make maps that look like the standard road maps from various publishers.

Get Your Head in the Clouds

Geospatial applications are renowned devourers of information. As new technologies for capturing data are developed, the geospatial applications soon follow. It makes sense as we try to find better and cheaper methods of modeling our world. One of these technologies is LIDAR (Light Detection And Ranging). LIDAR is becoming a common remote sensing technique. It sends out and measures the return of laser pulses to determine distances (as well as other information) in 3 dimensions to essentially generate a 3D view of the scanners environment. The resulting data is a very large group, or cloud of X, Y and Z values (along with other descriptive information depending on the equipment and technique). These point clouds have required specialized software to view and capture information.
Now, subscription AutoCAD Map 3D and Civil 3D users have some new tools available to work with these point clouds. The new Subscription Advantage Packs include tools for viewing and classifying Point Clouds. This can be incredibly useful to geospatial users. At a large private electric utility I used to work for, we used LIDAR as a method to survey electric transmission lines and surrounding vegetation. We were looking for trees that were growing into the lines and cause potential outages.

After installing the new Subscription Advantage packs in AutoCAD Map 3D, some new options become available. In the Task Pane’s Display Manager Tab, there is a new option in the Data menu – “Add Point Cloud Data.” Selecting this option opens a file select dialog prompting for an ISD file – a Point Cloud index, or data store file. You create the index from the new Point Cloud Manager.

 
Another new option is in the Tools menu – “Create Point Cloud Index.” This option opens the Point Cloud Manager.
From the Point Cloud Manager, we can add LIDAR data files to create a Point Cloud data store, merge them into groups, set coordinate systems, use filters to classify, create elevation ranges and even create spatial filters.
 

Adding the Point Cloud Data will create an entry in the Display Manager tab, and add a new Point Cloud ribbon giving us tools for stylization, exporting and creating surfaces. 

Keep in mind, these data sets are big, and will tax your system resources, so be prepared.
So, get your head in the clouds and check it out. The view is great.

I went to AU, and all I got was this caricature

Back to work today after several days at Autodesk University. I always find it time well spent – and yes, I did get a caricature done by an artist from TradeshowDraw.com – he was doing the sketches on a tablet PC using Autodesk’s SketchBook Pro as a way to demonstrate that product. I’m not sure about his capture of my cheeks, but he did say he was going to have some fun with it…

My caricature?

In reality, though, this year’s Autodesk University was, as I expected, a very productive use of my time. I was able to connect with some old friends and professional contacts, meet some new ones, and put faces with some of those folks I’ve only had phone or email (facebook, linkedin, etc) contact with.

I attended some really good workshops as well. A couple of highlights were seeing the SDSFIE model updates and tools for AutoCAD Map – these are essentially GIS standards adopted by the DOD for a common model (they call it Common Installation Picture) across the services. It is from an outgrowth of the old Tri-Services CADD group (now the CADBIM Technology Center) - I was involved in the Tri-Services standardization efforts was back in the 80s before there was a Tri-Services.  Back then working with CAD and GIS standards was like the Lewis and Clark expedition. The SDSFIE standards are pretty interesting, and to my mind would be a good model for municipalities as well. More on that later.

One of the other labs I enjoyed most gave me the chance to sit and work with the new AutoCAD Map Subscription Advantage Pack tools for working with point cloud data. Point Clouds are the result from LIDAR surveys that send out light waves in multiple directions and measure the time for the light to return from whatever is in its path. It’s kind of like a laser range finder in every direction. Point clouds are becoming more common as a way to survey existing information. At a utility company I used to work for, I was peripherally involved (working on related GIS data models) in a project using LIDAR to collect vegetation and electrical transmission data to identify potential interference. Not real exciting stuff on the surface, but when you look at how we did that work just a few years ago, it’s pretty amazing. I’ll add some details early next week on the new tools.

Happy GIS Day! Here’s some Arizona GIS Data Sites

Today is National GIS Day!

CADsoft Consulting’s CAD Camp 2009 is well underway.  We had a very successful Architectural/BIM day yesterday, and the Civil day is in full swing. Tomorrow will be the Geospatial Day (so it doesn’t interfere with any GIS Day activities). This morning we’ve had a presentation from Autodesk’s Civil 3D maven, Lucy Kuhns, and our own Ron Coulliard is doing a workshop on grading as I type. During Lucy’s presentation, I was asked about local Arizona GIS data, so I promides to share some of the sites I use/am aware of. The list is by no means exhaustive, and there’s some dupplication within the sites, but here you go anyway. I’ll continue to identify sites I run across in the future. If you’ve got some good ones you want to share, add them to the comments or email to me and I’ll add them to the list.

GIS Data Sites for Arizona

Arizona State Cartographer’s Office 
They maintain the Arizona GeoServer, with aerial photos and statewide features served through web mapping services (WMS) and web feature services (WFS). They also maintin links to other data sources throughout the state

The AGIC (Arizona Geographic Information Council) GeoData Portal
AGIC is a state sponsored group working with GIS across the state. They sponsor an annual GIS educational conference every year. We just finished the 2009 conference in Tucson. There was great attendance. I presented 3 hands on worksops this year.  They have County boundaries, tribal boundaries, cities, wilderness areas, political boundaries, voting districts, school districts, census information, environmental and natural resource data, interstates and roads

US Fish and Wildlife
USFW maintains larger scale data sets covering National Wetlands Inventory and area boundaries

U of A Library
The U of A Libraries maintains the Arizona Electronic Atlas and the Arizona Regional Image Archive (ARIA) as well as links to other data sites

ASU
The ASU Libraries also has spatial data and links available

ADEQ
Arizona Department of Environmental Quality (ADEQ) has water quality data, surface, drinking and groundwater

Local division sites

Maricopa County
The County Assessor’s Office GIS Department maintains data for the county including parcels, detailed topographic data, floodplains, and survey network.

Pima County
Pima County is really one of the long runners in GIS. They have had data available for as long as anyone in the state. They maintain over 273 data layers in ESRI shape files as well as landbase section maps in AutoCAD format.

City of Phoenix
Phoenix has an extensive collection of GIS data.  They have Engineering Quarter-Section maps in DXF format CAD files

Nationally-based sites

US Forest Service

FEMA
The Federal Emergency Management Agency maintains flood hazard data sets which are available as GIS data sets or through a Web Mapping Service (WMS) 

NSGIC
The National States Geographic Information Council maintains an inventory of data and its currency in the Ramona GIS Inventory. Arizona’s page is here:

Natural Resources Conservation Service (NRCS)
The NRCS maintains the Soil Data Mart with soil data available by state. They also collect other data such as water supply and snowpack

United States Geological Servey (USGS)
The USGS maintains large scale data sets for the US. They have digitial orthoquads and photos, land cover, elevation model and other data sets.

US Census Bureau
The Census maintains census and popluation data for the US

United States Department of Agriculture

USDA has Forest Coverage

Geography Network
The Geography Network maintains various spatial data sets for the US

ESRI Geoportal Extension
ESRI has a beta site with downloadable GIS data

National Center for Atmospheric Research
Atmospheric data fot the US

New Mexico

Lucy Kuhns mentioned the New Mexico Resource GIS program site. Here’s the link:
http://rgis.unm.edu/

Losing Your Religion: Interoperability with AutoCAD Map 3D and ESRI

I’ve been speaking at conferences for several years about CAD and GIS interoperability. It’s one of those topics where there’s a lot of interest and a lot of misinformation. Or at least, it seems to be much more difficult than it really is. I’ve been moving data between both systems for years, with very few real challenges. What I’ve found, is that the real issue is not the technical aspect of moving data back and forth, but the differences in how the software is generally used. AutoCAD (and other CAD systems) are primarily used for doing design work, and GIS (ostensibly ESRI, but it could be any GIS system) is primarily used for managing as-built facilities and systems. The real challenges are working between the design and as-built management processes. In other words, the issue isn’t CAD to GIS, the issue is Design to As-Built.

I’m going to make several blogs here in a series of the issues and some methods to make the process easier. This first post, I’m going to discuss the a bit about the similarities in the technologies. Following that, I’ll be posting on the differences, barriers, myths, and other issues involved.

Both AutoCAD and ESRI are built on basic primitive elements that are combined to create representations of real objects. Both systems include:

Points - a representation of a single location. It could represent a physical object such as a pole, manhole or brass cap in the ground, or it could be a non-physical point, such as a crime scene location or the corner of a property line. In any case, the systems both record a coordinate consisting of an X and a Y and possible a Z (if elevations are being included). The X and Y value could represent any projection or coordinate system, such as degrees of latitude and longitude or northings and eastings from a state plane.

Lines – a representation of of a connected set of coordinate pairs. Every line is going to have a start point defined by X, Y and/or Z, and and end point defined by an X, Y and/or Z. It could represent the centerline of a road, the edge of a building, or a buried pipe.

The line may be defined the system by coordinate pairs, such as point A and point B, or it could have the actual coordinate values in the line definition, such as this example from AutoCAD (a listing of a line – the start point is the set of parenthesis with the 10 X Y Z, and the end point is 11 X Y Z):

In some cases, for example, ESRI, the actual coordinates of the line are stored within an object “envelope”, which is a rectagle enclosing the object.

 Polygons – a representation of an area. It could be a representation of a parcel, a building footprint, or an animal migratory zone. It is defined by lines and so by a series of bounding coordinates. Generally, in vector systems (save that discussion for another time), polygons are defined by their boundaries. They could be defined by groups of lines, or it could have the coordinate values built into the definition, simlar to the lines (as shown above).

In ESRI, the coordinate pairs are contained in an envelope bounding the entire object:

Attributes -  data associated with an object. Associated data could be an identification number, a name, a description of the object, the color, size, diameter, etc. This is what turns a simple point, line or polygon into a representation of a fire hydrant, electric line or county. Attributes may be stored and linked to the object in a myriad of methods. It could be based on a common identifier stored in the object definition and the attribute list, as in a primary-foreign key relationship, or the definition of the object may be created to include certain atribute sets intrinsically. In some cases there may be a mixture of methods. For example, in AutoCAD, objects have intrinsic attributes (such as blocks attributes), extended entity data (attribute values associted to an individual object), or object data (data tables stored internally in the drawing and linked to objects). Additionally, both systems include methods to link objects to externally associated databases to extens the attributes of an object.

Understanding these similarities is key to understanding how to integrate these two systems. The next post, I’ll discuss the primary differences between the two.

Breaking Out of the DWG

Professionally, I’ve always been square on the fence between CAD and GIS. The funny thing is that GIS folks always call me a CAD guy, and the CAD guys always call me a GIS guy. Personally, I’ve always believed in using the right tool for the job, regardless of the technology. The best part for me is that I learn from both technologies and apply them wherever it fits. One of those “learnings” has been to break free from the DWG in AutoCAD Map.

Traditionally, when working in AutoCAD, we work in a drawing. The drawing maintains the model as well as any associated annotation, labeling, details, etc. At most we’ll have a group of drawing files, and some external references to share drawings. It’s much like using Word. We work in documents.

Traditionally, working in a GIS is a bit of a different model. While you have views, or map documents (depending on the particular flavor of GIS) your documents are really little more than a collection of links to data with some instructions on symbolizing the data. It really is more like working in a database, like Access, than in a document.
Both approaches provide benefits, so why not mix them? AutoCAD Map 3D allows me to bridge the gap between technologies. I can use my current drawing file as a pointer to various data sets, while having additional drawing information in my drawing file. For example, I can have my model (the actual drawing representation, whether it be a design of a road, or a city water system) linked from external data sources, and have multiple drawing sheets, complete with one or more title blocks in my current drawing. It allows me to have specific views, sheets or plans with project-specific data and take advantage of permanent data stores that are dynamic and updated each time I open them.

This allows me to use my enterprise GIS as part of my design tools without having to go through a conversion process. The great part is that it doesn’t really matter what my data source is. I can use other drawings through the attach and query tools, as well as Feature Data Objects (FDO) connectors to systems such as ArcSDE, Oracle or MySQL. I’ll blog more on that GIS-Design integration another time. The point being, is that the individual drawing can lose it’s importance and become essentially a snapshot of your model.

When using drawings as your data source, you can either use the ribbon or the Task Pane (as well as the Classic Menus). On the Ribbon, Attach is right on the Data Panel on the Home Ribbon for both task and tool-based workspaces. The Map Explorer tab in the Task Pane provides a visual of the drawings attached – to access the commands right click on drawings.

FDO data sources can be connected using Connect on the Data Panel in the Home Ribbon as well (next door to the Attach commands), or from the data button at the top of the Task Pane.

The short story is that you don’t have to be a slave to your DWG file, and treating your DWG as a view to your model, you have some real power to take advantage of a whole new range of possibilities. The great part is that this is not new technology (the FDO process is relatively recent). Map has incorporated this capability since it was acquired.

So, unleash the shackles and get connected.

New License transfer in 2010

For those folks (like myself) who work with multiple machines and aren’t using network licenses, the Autodesk Portable License Utility has been a relatively easy way to shift licenses around. Not to mention providing a way to do a planned reimage of a machine.
We have had several people who have called asking where the Portable License Utility shortcut is in the 2010 versions. Well, the answer is that it’s not there. The Portable License Utility has been replaced by a web tool that allows you to perform your license transfer. Now, your shortcut, called simply “License Transfer Utility” connects to the web and allows transfer your stand-alone licenses. Each shortcut is product specific, so you need to run the utility from the folder for the product you want to transfer.
Autodesk has a video discussing the tools and a demonstration of the tool on YouTube. Check it out:

AutoCAD Map 3D Tip: Data from the Web

One of the challenges for GIS users is finding data, and another is maintaining that data once you find it. Many times, another organization will have a data set that they are willing to share. Getting a copy of the data and incorporating it into your system is a relatively easy process. When that data is regularly being updated, keeping your copy of the data current is another matter altogether. The challenge is greater as the data becomes more dynamic.

Web Mapping Services (WMS – shares image data) and Web Feature Services (WFS – shares feature data) are a way to distribute, and receive data over the web. An organization can publish their data to the web into this format and make their data available to other organizations. Then any authorized user with the right tools can have a live connection to that web data. When data is updated, all of the connected users will have those updates as well.

AutoCAD Map 3D can use both of these services to allow the user to connect to these data sets and make them part of their drawings. The great part is that this data is a connection to live data rather than objects stored in the drawing, so they will not require space in the drawing (the other side of that is that the connection to the data must be live to see the data).

To add this data, you right click on the data icon in the Display Manager tab of the Map Task Pane and select “Connect to Data” (this also works in Civil 3D).

This will open the FDO Data Connect palette. Select Add WMS (or WFS) Connection.

The palette will show boxes to add the connection information.

Add a name for the connection, the web address, and the WMS standard version (which is usually part of the address) and select the Connect button.

Once you are connected, you have the option to select which data (image sets or layers) to add to the map. Select the “add to map” button and the data will appear in the drawing and the feature set will show up in the Display manager.

Now, you have a live connection to the data. You can save the drawing file, and the next time you open it, the connection will be reestablished.
This makes a relatively painless way to share GIS data. The user has immediate access to the data, any changes, and can’t modify the data in any way, but can use it as a part of their geographic data.

In addition, the Open Geospatial Consortium, Inc., a non-profit standards organization, maintains a list of some of these public web-enabled data sets. So next time you’re needing some data, check it out. Someone may have posted just what you need.