ArcGIS 10 Pre-Release finding duplicates

[haydena]

Basically, the info in the article http://resources.arcgis.com/content/...leShow&d=31854 has worked for me in 9.3 and prior. In pre-release 10 I get:

General error executing calculator.
ERROR 999999: Error executing function.
Expected statement
Failed to execute (CalculateField).
Failed at Thu May 06 10:46:18 2010 (Elapsed Time: 1.00 seconds)

Something with CreateObject("Scripting.Dictionary") perhaps? Any other thoughts?

[esrikmarten]

Hi Haydena,

In ArcGIS 10, the Field Calculator has been enhanced to work with Python scripting. You can no longer use VBA code blocks; instead, both VB Script and Python are fully supported. The code sample you are trying to use will not work because it uses VBA. There is not a VBScript alternative for this VBA sample because you cannot declare Static variables.

You may be interested in learning more about using Python with the field calculator. Here is a Help Topic to get you started: http://help.arcgis.com/en/arcgisdesk...000002m000000/

I'll try and find some more information for you regarding this specific workflow and using Python.

Kent M.
esri

[esrikmarten]

Hi Haydena,

Using the Field Calculator in ArcGIS 10. You can use Python to easily find duplicate values in a given field. In Python you can declare a list outside of the function so that it can retain added values. Here is a code sample:

uniqueList = []
def isDuplicate(inValue):
  if inValue in uniqueList:
    return 1
  else:
    uniqueList.append(inValue)
    return 0

At the bottom of the Field Calculator, you will have to enter in the function and field name like this:

isDuplicate(!FIELD_NAME!)

Thanks for your post.

Kent M.
esri

[haydena]

Works perfect.

Thanks for the response Kent.

[ColterSikora]

Hello!

I am looking to find duplicate attributes in my shapefile as shown in this forum. However, when I try the Python code example I get the following error message: "Error 000539: Error running expression: uniqueList = Error running expression: uniqueList = [] "

Here is the code that I entered:

uniqueList = [ !Duplicate! ]
def isDuplicate(inValue):
if inValue in uniqueList:
return 1
else:
uniqueList.append(inValue)
return 0

isDuplicate(!Calculate!)


---

"Duplicate" is the field I want the results entered into, "Calculate" is the field that I want duplicates selected from.

Hopefully this is enough code to explain my situation.

Thank you for your help in advance!

Sincerely,

Colter

[jcmolina]

Hi All,

First, thanks Kent M. for making this script available, it works great.

Colter, this is how I made it work.

1) Right click on the column you wish to populate > field calculator>check in "Show Codeblock" and select "Python" at the top let corner of the calculator
2) Copy the first piece of Kent's code in the upper box of the calculator (Do not write anything on that part, do not add the column name, just leave it as it is)
3) Copy the second part of the code (---isDuplicate(!Calculate!)-----) in the lower box and replace the "Calculate" word with the column name that you are exploring for duplicates, leave the exclamation marks in.

I hope that works for you all.

JCMolina

[leftyb]

Hi there, this works great, but I was wondering - is there a way to modify the script so it shows the first record of the duplicate as a 1 also?

[mse2k11]

I'm getting the following error when I run this script:

"There was a failure during processing, check the Geoprocessing Results window for details."

However, no Geoprocessing Results window is evident.

Any suggestions?

[tguthrie]

You can also use the frequency analysis geoprocessing tool to find all duplicates, including the first one. Anything resulting in a frequency > 1 for a given field would be a duplicate.

http://help.arcgis.com/en/arcgisdesk...000001w000000/

[davidbrett]

Thanks for the code, works great.

Any insight on modifying it to mark the duplicates with an incremental number per occurrence?

Like what EzCalc did in VBA:
field_Mark_Duplicates_2.cal Calculates a unique number for each occurence of a value in a field. Long
Example:
a==>1
b==>1
c==>1
b==>2
b==>3
a==>2

[Jamal432@gmail.com]

Hi Haydena,

In ArcGIS 10, the Field Calculator has been enhanced to work with Python scripting. You can no longer use VBA code blocks; instead, both VB Script and Python are fully supported. The code sample you are trying to use will not work because it uses VBA. There is not a VBScript alternative for this VBA sample because you cannot declare Static variables.

You may be interested in learning more about using Python with the field calculator. Here is a Help Topic to get you started: http://help.arcgis.com/en/arcgisdesk...000002m000000/

I'll try and find some more information for you regarding this specific workflow and using Python.

Kent M.
esri

Hi Kent,

i couldn't manage how to rewrite the code below (if statement) in VBscript or python.
could you please help me how to do so?

and in case we have so many codes in VBA, are we required to switch them to VBscript or python? is that reasonable?

thank you

the code:
-------------------------
if [FC_mm] <=50 then
op = 0

else
if [FC_mm]>50 and [FC_mm]<=90 then
op = 50

else
if [FC_mm] >90 and [FC_mm] <= 140 then
op = 125

else
if [FC_mm] >140 and [FC_mm] < = 200 then
op = 250

else
if [FC_mm] > 200 and [FC_mm] < = 250 then
op = 500

else
if [FC_mm] > 250 then
op = 750

end if
end if
end if
end if
end if
end if
-----------------------------------------------------

[davidbrett]

In the field calculator, check 'show code block', then paste the following in the 'Pre-Logic Script Code' section:

def myVal(FC_mm):
  rec = 0
  if FC_mm>50 and FC_mm<=90:
    rec = 50
  if FC_mm >90 and FC_mm <= 140:
    rec = 125
  if FC_mm >140 and FC_mm <= 200:
    rec = 250
  if FC_mm > 200 and FC_mm <= 250:
    rec = 500
  if FC_mm > 250:
    rec = 750
  return rec

In the box below, paste:

myVal( !FC_mm! )

[hami1981]

#Here's a script that flags duplicates as "1" including the first occurence:
#It works fine for strings and integers, but doesn't work for float values, to overcome the float
#problem convert the values to strings.
#Paste the following code in the Field Calculator:

import arcpy
inFeatures = "C:\\Folder\\Subfolder\\yourshapefile.shp"

inField = "YOUR_FIELD" #The field you want to analyse
rows = arcpy.SearchCursor(inFeatures)
row = rows.next()
ValueList = list()
while row:
currentValue = row.getValue(inField)
ValueList.append(currentValue)
row = rows.next()

def isDuplicate(inValue):
if ValueList.count(inValue) > 1:
return 1
else:
return 0


##In the box below, paste:
## isDuplicate(!YOUR_FIELD!)

[olau]

Hami1981,

I was unable to process the alternative duplicate script that can label the first occurrence as "1"

import arcpy
inFeatures = "C:\\Folder\\Subfolder\\yourshapefile.shp"

inField = "!AnalysisField!"
rows = arcpy.SearchCursor(inFeatures)
row = rows.next()
ValueList = list()
while row:
currentValue = row.getValue(inField)
ValueList.append(currentValue)
row = rows.next()

def isDuplicate(inValue):
if ValueList.count(inValue) > 1:
return 1
else:
return 0

The error I received was "Failed to execute. Parameters are not valid"
Error 000989: Python syntax error: Parsing error <type 'exceptions.SyntaxError'>: invalid syntax (line 9)
Failed to execute (CalculateField)

Any thoughts?

[hami1981]

Hi Olivia,

First, remember python is tab sensitive, so a four-space tab indent width is the prefered coding style for Python. In your code i'm going to repace space/tabs with "_". But do replace it again with spaces in your code!

Also, don't forget to set the right path to your shapefile and keep the double \\ (because \ is an scape character)

I think what your problem is that you put inField = "!AnalysisField!", it should be:
inField = "AnalysisField" without the !.


import arcpy
inFeatures = "C:\\Folder\\Subfolder\\yourshapefile.shp" #the path to your shapefile

inField = "!AnalysisField!"
rows = arcpy.SearchCursor(inFeatures)
row = rows.next()
ValueList = list()
while row:
____currentValue = row.getValue(inField)
____ValueList.append(currentValue)
____row = rows.next()

def isDuplicate(inValue):
____if ValueList.count(inValue) > 1:
________return 1
____else:
________return 0

In the codeblock below write:

inValue(!AnalysisField!)

Hope it works now!

Hugo

[jsnider@wwfcanada.org]

For finding duplicate records using ArcGIS 10, this Python code might be slightly faster than those codes listed above, at least for large datasets, since it only passes through the list once to complete the value count rather than looping to count for each item in the list...

import arcpy 
from collections import defaultdict

inft = r"C:\GIS\yourshape.shp"
inField = "Your_field"

value_list = []
rows = arcpy.SearchCursor(inft)
for row in rows:
value = row.getValue(inField)
value_list.append(value)
duplicate_list = []
dict1 = defaultdict(int)
for value in value_list:
dict1(value) +=1
for key, val in dict1.iteritems():
if val > 1:
 dup_list.append(key)
print "duplicate list length = ", len(duplicate_list)

[mike.weaver]

For finding duplicate records using ArcGIS 10, this Python code might be slightly faster than those codes listed above, at least for large datasets, since it only passes through the list once to complete the value count rather than looping to count for each item in the list...

How do I get this to work using the Calculate field tool. I'm kinda new to python and running these scripts. Is there something i need to put in the in the box below the pre-logic code box?

I want to use the script you posted because I am working with large datasets and need it run faster.

Thanks in advance.

[jsnider@wwfcanada.org]

Hi Mike -- I'm not positive that the Calculate Value tool is the best solution here, but I that will depend on what exactly you are trying achieve -- for instance, are you trying to calculate a value for a field based on the number of times that value occurs?

The Python code I've posted above it designed to be run either in the Python window within ArcGIS 10 or from an IDLE GUI interface for Python.

The output of the script, as it is currently written, is a list of values that are duplicates. So, in order for this to be useful for a calculation, you would likely have to refer to the items within the list.

Maybe if you provide a better sense of your task, perhaps then I, or someone else here, can provide a more constructive reply.

Thanks,
James

[mike.weaver]

Thanks for the quick reply. What I'm trying to do is remove duplicate telephone records in a database. So if I get a database with 600k records about 20% of those are duplicates. I can de-dup this database in MS access easily and quickly. But I'm trying to incorporate this de-dup process in a model. I've tried your script in the Python window in ArcGIS 10 and keep getting an error: Runtime error <type 'exceptions.SyntaxError'>: illegal expression for augmented assignment (<string>, line 13)

I've used other code that will flag the duplicates, but have proven to be very time consuming comparative to the append query method in MS Access.

[jsnider@wwfcanada.org]

Mike, if it were me, I would proceed by using the code as written, using the Python window in Arc 10.

From there, I would use an updateCursor to delete the duplicate row from the table.

More on updateCursors, including the `deleteRow`expression here:
http://help.arcgis.com/en/arcgisdesk...0000001q000000

However in doing so, you need to be careful not to delete the value entirely -- i.e. make sure to retain one instance of each value in the duplicate list.

Unfortunately, I`m not much help in terms of integrating this into a modelbuilder-based approach.

[ErinK]

Any insight on modifying it to mark the duplicates with an incremental number per occurrence?

DavidBrett,

Python code block:

UniqueDict = {}
def isDuplicateIndex(inValue):
    UniqueDict.setdefault(inValue,0)
    UniqueDict[inValue] += 1
    return UniqueDict[inValue]

Expression:

isDuplicateIndex( !YOUR_FIELD! )

Hope that helps, Erin.

[davidbrett]

Thanks Erin! This is exactly what I need.

[mbd7878]

This is great! Can you help me put this in a stand alone python script? Everything runs great until the Calculate Field management...thats where it fails. Can anyone see where I am going wrong?

fclayer = arcpy.mapping.Layer(C:/data/BaseData.gbd/myLayer)
fieldName = "Duplicate"
expression2 = "isDuplicate(!soil_desc!)"
uniqueList = []
codeblock = """def isDuplicate(inValue):
(INDENT)if inValue in uniqueList:
(INDENTx2)return 1
(INDENT)else:
(INDENTx2)uniqueList.append(inValue)
(INDENTx2)return 0"""

arcpy.AddField_management(myLayer,fieldName,"SHORT")

arcpy.CalculateField_management(myLayer,fieldName,expression2,"PYTHON",codeblock)

[davidbrett]

The attached will increment duplicates in a user specified column. It is configured to run from ArcToolbox. You can hard code the variable if necessary.

[mbd7878]

I get the following error.

<type 'exceptions.TypeError'>: 'Layer' object is unsubscriptable
Failed to execute (Script)

any ideas why?

Got it figured out. Here is worked for me.

dupField = "soil_desc"
markField = "Duplicate"

arcpy.AddField_management(soilssortlayer,markField,"FLOAT")

rows = arcpy.UpdateCursor(soilssortlayer,"","","",dupField)

mylist = []

i = -1
for row in rows:
if i == -1:
value = row.getValue(dupField)
mylist.append(value)
i += 1
if row.getValue(dupField) != value:
value = row.getValue(dupField)
mylist.append(value)
i = 0
row.Duplicate = i
rows.updateRow(row)

[biniamts]

Hi there,
I am wandering for your help. I would like to copy selected rows of an attribute table based on the values of a single field. and duplicate (paste) the copied rows in the attribute table itself with some editing afterwards on the object id for example. can some one provide me information(idea ) on how to put it in python script.
thanx
B. Weldmeichael

[mbd7878]

feature = "C://Data//Feature"
newField = "NEWField"

# Create the new Field
newField = arcpy.AddField_management(feature,newField,"TEXT","50")

# Do your Select by Attribute normally
expression = "your select by attribute expression"
arcpy.SelectLayerByAttribute_management(feature,"NEW_SELECTION",expression)

# Calculate the new field
arcpy.CalculateField_management(feature,newField,"!OLDField!","PYTHON")


I am new to these Scripts, so I am not sure if when you calculate the new field it will honor those just selected or if it will update all the rows, but I think it will work. Depending on how complex the "some editing afterwards" is, but if it is simple you can add onto your expression on the Calculation (the !OLDField! part).
This has not been tested, and like I said I am a beginner, but that should give you a place to start.

[CJCarsley]

Hello!

I am looking to find duplicate attributes in my shapefile as shown in this forum. However, when I try the Python code example I get the following error message: "Error 000539: Error running expression: uniqueList = Error running expression: uniqueList = [] "

Here is the code that I entered:

uniqueList = [ !Duplicate! ]
def isDuplicate(inValue):
if inValue in uniqueList:
return 1
else:
uniqueList.append(inValue)
return 0

isDuplicate(!Calculate!)


---

"Duplicate" is the field I want the results entered into, "Calculate" is the field that I want duplicates selected from.

Hopefully this is enough code to explain my situation.

Thank you for your help in advance!

Sincerely,

Colter

I used this script to identify duplicates once, last week, and it worked perfectly. For some reason, I'm trying to do it again this week, and I'm being consistently presented with a syntax error for line 1. The only contrast between my two sets of data is size (some 43,000 records vs. 233,000). Could this difference in size be enough to cause the script to fail? I've noticed the process gets about 70%-80% complete before it 'fails to execute'. I still get some results, but a large portion of the "Dup" field is populated with <null> values.

Any guidance would be greatly appreciated!

[pppbpnst]

thank s to Mark its work..... and for JCMolina thanks for the step of the script...

this for Phyton, if anyone know script for the VBA please ...

[olcoil]

Hi Guys, I didn't want to start a new thread, instead I'll just post my problem here. Sorry if it seems like I am not understanding it all, our department is just starting to learn Python on ArcMap 10!

I have 3 fields:
1. "isDuplicate" (I want this field flag as 1 for duplicates and 0 for unique values)
2. x (coordinate calculated by Field Calculator)
3. y (coordinate calculated by Field Calculator)

I want the script do something like this:

For All_Objects
If x is unique AND y is unique,
isDuplicate = 0
Else
isDuplicate = 1
End

But I know that doesn't make much sense. Basically this script should identify all points that have the same X/Y coordinates and label the fields as such. There may be other easier ways to do this, but there are other functions that I will need to add to this script anyways.

I just very new with Python sorry

[mbd7878]

I think this will work, but I have not tested it. Also there might be an easier way. But what I did was create a new field that combines the values from the x & y fields (you can delete this new field at the end if you want). Then search that one field for duplicates and add the values to the isDuplicate field.
You also might need to make sure when you created isDuplicate it is a FLOAT field.

ObjectLayer = "Name of Layer to be flagged"

arcpy.AddField_management(ObjectLayer,"Working","TEXT","50")

expression = "x + y"
arcpy.CalculateField_management(ObjectLayer,"Working",expression,"PYTHON")

dupField = "Working"
Duplicate = "isDuplicate" 

rows = arcpy.UpdateCursor(ObjectLayer,"","","",dupField)

mylist = []

i = -1
for row in rows:
    if i == -1:
        value = row.getValue(dupField)
        mylist.append(value)
    i += 1
    if row.getValue(dupField) != value:
        value = row.getValue(dupField)
        mylist.append(value)
        i = 0
    row.isDuplicate = i
    rows.updateRow(row)

[davidbrett]

You can use the following in the field calculator

Parser = Python
check 'Show Codeblock'

In the Pre-Logic Script Code:

uniqueList = []
def isDuplicate(inValue):
  if inValue in uniqueList:
    return 1
  else:
    uniqueList.append(inValue)
    return 0

in the Box below where it says FIELD_NAME =

isDuplicate ( !FieldName1! + !FieldName2! )

tested and works.

[angelagotch]

Juan, that worked perfectly. Thanks! ~Angela

[cgraves73]

Hi Guys,

Further to this thread I would like select duplicate records in a field and then make these unique using the Field Calculator. For example I have parcel IDs (e.g. F555 A30) which might occur 3 times. I want to select these and change the name of these with either 'F555 A30 B' or 'F555 A30 1'. Also I would like the change to occur in the field the original records are in and not create a new field.

Thanks in advance,

Chris

[pputri]

You can use the following in the field calculator

Parser = Python
check 'Show Codeblock'

In the Pre-Logic Script Code:

uniqueList = []
def isDuplicate(inValue):
  if inValue in uniqueList:
    return 1
  else:
    uniqueList.append(inValue)
    return 0

in the Box below where it says FIELD_NAME =

isDuplicate ( !FieldName1! + !FieldName2! )

tested and works.

Hi David!

I have the same issue with Peter and use your code. It works great actually when there are duplicate XY in the data. However, when there are no such data, the script's output highlighting records that has duplicate X value AND/OR duplicate Y value (screenshot attached).

How to make a script that find only the absolute duplicate value of both X AND Y?

Thank you.

Prima.

[davidbrett]

Hi David!

I have the same issue with Peter and use your code. It works great actually when there are duplicate XY in the data. However, when there are no such data, the script's output highlighting records that has duplicate X value AND/OR duplicate Y value (screenshot attached).

How to make a script that find only the absolute duplicate value of both X AND Y?

Thank you.

Prima.

Hi,

Let's try another way

uniqueList = []
i = 0
def isDuplicate(inValue):
    global i
    if inValue in uniqueList:
        i += 1
        return i
    else:
        uniqueList.append(inValue)
    return 0

This will increment the dup field per the matching duplicates. You can test how this works by using one inValue field at a time, and then both.

Example 1:
dup =
isDuplicate( !x!)

Example 2:
dup =
isDuplicate( !y!)

Example 3:
dup =
isDuplicate( !x! + !y!)

[davidbrett]

Hi Guys,

Further to this thread I would like select duplicate records in a field and then make these unique using the Field Calculator. For example I have parcel IDs (e.g. F555 A30) which might occur 3 times. I want to select these and change the name of these with either 'F555 A30 B' or 'F555 A30 1'. Also I would like the change to occur in the field the original records are in and not create a new field.

Thanks in advance,

Chris

Hi Chris,

The safest way to do this is to create a new field and not update the field you are calculating during the calculation...logic: don't change the data your analyzing while analyzing the data.

You can use the code I just posted to increment the dup field, then calculate the new id based on the old ID + dup value.

[rblash]

Has anyone figured out how to simply select the duplicates without creating a new field or using summarize?

Thanks,

Rob