+ Reply to Thread
Results 1 to 46 of 46

Thread: ArcGIS 10 Pre-Release finding duplicates

  1. #1

    Join Date
    Apr 2010
    Posts
    2
    Points
    0
    Answers Provided
    0


    0

    Default ArcGIS 10 Pre-Release finding duplicates

    Basically, the info in the article http://resources.arcgis.com/content/...leShow&d=31854 has worked for me in 9.3 and prior. In pre-release 10 I get:

    General error executing calculator.
    ERROR 999999: Error executing function.
    Expected statement
    Failed to execute (CalculateField).
    Failed at Thu May 06 10:46:18 2010 (Elapsed Time: 1.00 seconds)

    Something with CreateObject("Scripting.Dictionary") perhaps? Any other thoughts?
    Last edited by haydena; 05-06-2010 at 08:32 AM.

  2. #2
    Kent Marten
    Join Date
    Oct 2009
    Posts
    95
    Points
    17
    Answers Provided
    0


    0

    Lightbulb Re: ArcGIS 10 Pre-Release finding duplicates

    Hi Haydena,

    In ArcGIS 10, the Field Calculator has been enhanced to work with Python scripting. You can no longer use VBA code blocks; instead, both VB Script and Python are fully supported. The code sample you are trying to use will not work because it uses VBA. There is not a VBScript alternative for this VBA sample because you cannot declare Static variables.

    You may be interested in learning more about using Python with the field calculator. Here is a Help Topic to get you started: http://help.arcgis.com/en/arcgisdesk...000002m000000/

    I'll try and find some more information for you regarding this specific workflow and using Python.

    Kent M.
    esri

  3. #3
    Kent Marten
    Join Date
    Oct 2009
    Posts
    95
    Points
    17
    Answers Provided
    0


    5

    Thumbs up Re: ArcGIS 10 Pre-Release finding duplicates

    Hi Haydena,

    Using the Field Calculator in ArcGIS 10. You can use Python to easily find duplicate values in a given field. In Python you can declare a list outside of the function so that it can retain added values. Here is a code sample:

    Code:
    uniqueList = []
    def isDuplicate(inValue):
      if inValue in uniqueList:
        return 1
      else:
        uniqueList.append(inValue)
        return 0
    At the bottom of the Field Calculator, you will have to enter in the function and field name like this:
    Code:
    isDuplicate(!FIELD_NAME!)
    Thanks for your post.

    Kent M.
    esri

  4. #4

    Join Date
    Apr 2010
    Posts
    2
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Works perfect.

    Thanks for the response Kent.

  5. #5

    Join Date
    Sep 2010
    Posts
    1
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Hello!

    I am looking to find duplicate attributes in my shapefile as shown in this forum. However, when I try the Python code example I get the following error message: "Error 000539: Error running expression: uniqueList = Error running expression: uniqueList = [] "

    Here is the code that I entered:

    uniqueList = [ !Duplicate! ]
    def isDuplicate(inValue):
    if inValue in uniqueList:
    return 1
    else:
    uniqueList.append(inValue)
    return 0

    isDuplicate(!Calculate!)


    ---

    "Duplicate" is the field I want the results entered into, "Calculate" is the field that I want duplicates selected from.

    Hopefully this is enough code to explain my situation.

    Thank you for your help in advance!

    Sincerely,

    Colter

  6. #6
    Juan Carlos Molina
    Join Date
    Aug 2010
    Posts
    1
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Hi All,

    First, thanks Kent M. for making this script available, it works great.

    Colter, this is how I made it work.

    1) Right click on the column you wish to populate > field calculator>check in "Show Codeblock" and select "Python" at the top let corner of the calculator
    2) Copy the first piece of Kent's code in the upper box of the calculator (Do not write anything on that part, do not add the column name, just leave it as it is)
    3) Copy the second part of the code (---isDuplicate(!Calculate!)-----) in the lower box and replace the "Calculate" word with the column name that you are exploring for duplicates, leave the exclamation marks in.

    I hope that works for you all.

    JCMolina

  7. #7
    Bret Allphin
    Join Date
    Apr 2010
    Posts
    8
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Hi there, this works great, but I was wondering - is there a way to modify the script so it shows the first record of the duplicate as a 1 also?

  8. #8
    M Elliott
    Join Date
    Jan 2011
    Posts
    2
    Points
    1
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    I'm getting the following error when I run this script:

    "There was a failure during processing, check the Geoprocessing Results window for details."

    However, no Geoprocessing Results window is evident.

    Any suggestions?

  9. #9
    Tyrone Guthrie
    Join Date
    Feb 2011
    Posts
    1
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    You can also use the frequency analysis geoprocessing tool to find all duplicates, including the first one. Anything resulting in a frequency > 1 for a given field would be a duplicate.

    http://help.arcgis.com/en/arcgisdesk...000001w000000/

  10. #10
    David Brett
    Join Date
    Nov 2009
    Posts
    33
    Points
    1
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Thanks for the code, works great.

    Any insight on modifying it to mark the duplicates with an incremental number per occurrence?

    Like what EzCalc did in VBA:
    field_Mark_Duplicates_2.cal Calculates a unique number for each occurence of a value in a field. Long
    Example:
    a==>1
    b==>1
    c==>1
    b==>2
    b==>3
    a==>2

  11. #11
    Jamal NUMAN
    Join Date
    Apr 2010
    Posts
    2,362
    Points
    33
    Answers Provided
    5


    0

    Question Re: ArcGIS 10 Pre-Release finding duplicates

    Quote Originally Posted by esrikmarten View Post
    Hi Haydena,

    In ArcGIS 10, the Field Calculator has been enhanced to work with Python scripting. You can no longer use VBA code blocks; instead, both VB Script and Python are fully supported. The code sample you are trying to use will not work because it uses VBA. There is not a VBScript alternative for this VBA sample because you cannot declare Static variables.

    You may be interested in learning more about using Python with the field calculator. Here is a Help Topic to get you started: http://help.arcgis.com/en/arcgisdesk...000002m000000/

    I'll try and find some more information for you regarding this specific workflow and using Python.

    Kent M.
    esri
    Hi Kent,

    i couldn't manage how to rewrite the code below (if statement) in VBscript or python.
    could you please help me how to do so?

    and in case we have so many codes in VBA, are we required to switch them to VBscript or python? is that reasonable?

    thank you

    the code:
    -------------------------
    if [FC_mm] <=50 then
    op = 0

    else
    if [FC_mm]>50 and [FC_mm]<=90 then
    op = 50

    else
    if [FC_mm] >90 and [FC_mm] <= 140 then
    op = 125

    else
    if [FC_mm] >140 and [FC_mm] < = 200 then
    op = 250

    else
    if [FC_mm] > 200 and [FC_mm] < = 250 then
    op = 500

    else
    if [FC_mm] > 250 then
    op = 750

    end if
    end if
    end if
    end if
    end if
    end if
    -----------------------------------------------------
    ------------------------------------------------------------------
    Jamal NUMAN, GIS & Civil Engineer, MSc
    e-mail: JamaL9722060@yahoo.com
    P.O.Box: 731| Ramallah| West Bank| Palestine.
    ------------------------------------------------------------------

    ArcGIS Desktop 10.2 (Advanced)
    ArcGIS Server 10.2 (Standard/Enterprise)
    ArcGIS Viewer for Silverlight 3.1/Flex 3.0, Geocortex 3.15
    AutCAD 2014
    Windows 8.1 Pro 64 bits (Build 9200)
    Firefox 24.0
    Kaspersky 2014

  12. #12
    David Brett
    Join Date
    Nov 2009
    Posts
    33
    Points
    1
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    In the field calculator, check 'show code block', then paste the following in the 'Pre-Logic Script Code' section:

    Code:
    def myVal(FC_mm):
      rec = 0
      if FC_mm>50 and FC_mm<=90:
        rec = 50
      if FC_mm >90 and FC_mm <= 140:
        rec = 125
      if FC_mm >140 and FC_mm <= 200:
        rec = 250
      if FC_mm > 200 and FC_mm <= 250:
        rec = 500
      if FC_mm > 250:
        rec = 750
      return rec
    In the box below, paste:
    Code:
    myVal( !FC_mm! )

  13. #13
    Hugo de Alba
    Join Date
    Jun 2011
    Posts
    3
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    #Here's a script that flags duplicates as "1" including the first occurence:
    #It works fine for strings and integers, but doesn't work for float values, to overcome the float
    #problem convert the values to strings.
    #Paste the following code in the Field Calculator:

    import arcpy
    inFeatures = "C:\\Folder\\Subfolder\\yourshapefile.shp"

    inField = "YOUR_FIELD" #The field you want to analyse
    rows = arcpy.SearchCursor(inFeatures)
    row = rows.next()
    ValueList = list()
    while row:
    currentValue = row.getValue(inField)
    ValueList.append(currentValue)
    row = rows.next()

    def isDuplicate(inValue):
    if ValueList.count(inValue) > 1:
    return 1
    else:
    return 0


    ##In the box below, paste:
    ## isDuplicate(!YOUR_FIELD!)
    Last edited by hami1981; 06-02-2011 at 06:29 AM.

  14. #14
    Olivia Lau
    Join Date
    Jun 2011
    Posts
    1
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Hami1981,

    I was unable to process the alternative duplicate script that can label the first occurrence as "1"

    import arcpy
    inFeatures = "C:\\Folder\\Subfolder\\yourshapefile.shp"

    inField = "!AnalysisField!"
    rows = arcpy.SearchCursor(inFeatures)
    row = rows.next()
    ValueList = list()
    while row:
    currentValue = row.getValue(inField)
    ValueList.append(currentValue)
    row = rows.next()

    def isDuplicate(inValue):
    if ValueList.count(inValue) > 1:
    return 1
    else:
    return 0

    The error I received was "Failed to execute. Parameters are not valid"
    Error 000989: Python syntax error: Parsing error <type 'exceptions.SyntaxError'>: invalid syntax (line 9)
    Failed to execute (CalculateField)

    Any thoughts?
    Last edited by olau; 06-07-2011 at 10:25 AM.

  15. #15
    Hugo de Alba
    Join Date
    Jun 2011
    Posts
    3
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Hi Olivia,

    First, remember python is tab sensitive, so a four-space tab indent width is the prefered coding style for Python. In your code i'm going to repace space/tabs with "_". But do replace it again with spaces in your code!

    Also, don't forget to set the right path to your shapefile and keep the double \\ (because \ is an scape character)

    I think what your problem is that you put inField = "!AnalysisField!", it should be:
    inField = "AnalysisField" without the !.


    import arcpy
    inFeatures = "C:\\Folder\\Subfolder\\yourshapefile.shp" #the path to your shapefile

    inField = "!AnalysisField!"
    rows = arcpy.SearchCursor(inFeatures)
    row = rows.next()
    ValueList = list()
    while row:
    ____currentValue = row.getValue(inField)
    ____ValueList.append(currentValue)
    ____row = rows.next()

    def isDuplicate(inValue):
    ____if ValueList.count(inValue) > 1:
    ________return 1
    ____else:
    ________return 0

    In the codeblock below write:

    inValue(!AnalysisField!)

    Hope it works now!

    Hugo
    Last edited by hami1981; 07-01-2011 at 08:59 AM.

  16. #16
    James Snider
    Join Date
    Jun 2010
    Posts
    26
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    For finding duplicate records using ArcGIS 10, this Python code might be slightly faster than those codes listed above, at least for large datasets, since it only passes through the list once to complete the value count rather than looping to count for each item in the list...

    Code:
    import arcpy 
    from collections import defaultdict
    
    inft = r"C:\GIS\yourshape.shp"
    inField = "Your_field"
    
    value_list = []
    rows = arcpy.SearchCursor(inft)
    for row in rows:
    value = row.getValue(inField)
    value_list.append(value)
    duplicate_list = [] dict1 = defaultdict(int) for value in value_list:
    dict1(value) +=1
    for key, val in dict1.iteritems():
    if val > 1:
    dup_list.append(key)
    print "duplicate list length = ", len(duplicate_list)

  17. #17
    mike weaver
    Join Date
    Aug 2011
    Posts
    2
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    For finding duplicate records using ArcGIS 10, this Python code might be slightly faster than those codes listed above, at least for large datasets, since it only passes through the list once to complete the value count rather than looping to count for each item in the list...

    How do I get this to work using the Calculate field tool. I'm kinda new to python and running these scripts. Is there something i need to put in the in the box below the pre-logic code box?

    I want to use the script you posted because I am working with large datasets and need it run faster.

    Thanks in advance.

  18. #18
    James Snider
    Join Date
    Jun 2010
    Posts
    26
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Hi Mike -- I'm not positive that the Calculate Value tool is the best solution here, but I that will depend on what exactly you are trying achieve -- for instance, are you trying to calculate a value for a field based on the number of times that value occurs?

    The Python code I've posted above it designed to be run either in the Python window within ArcGIS 10 or from an IDLE GUI interface for Python.

    The output of the script, as it is currently written, is a list of values that are duplicates. So, in order for this to be useful for a calculation, you would likely have to refer to the items within the list.

    Maybe if you provide a better sense of your task, perhaps then I, or someone else here, can provide a more constructive reply.

    Thanks,
    James

  19. #19
    mike weaver
    Join Date
    Aug 2011
    Posts
    2
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Thanks for the quick reply. What I'm trying to do is remove duplicate telephone records in a database. So if I get a database with 600k records about 20% of those are duplicates. I can de-dup this database in MS access easily and quickly. But I'm trying to incorporate this de-dup process in a model. I've tried your script in the Python window in ArcGIS 10 and keep getting an error: Runtime error <type 'exceptions.SyntaxError'>: illegal expression for augmented assignment (<string>, line 13)

    I've used other code that will flag the duplicates, but have proven to be very time consuming comparative to the append query method in MS Access.

  20. #20
    James Snider
    Join Date
    Jun 2010
    Posts
    26
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Mike, if it were me, I would proceed by using the code as written, using the Python window in Arc 10.

    From there, I would use an updateCursor to delete the duplicate row from the table.

    More on updateCursors, including the `deleteRow`expression here:
    http://help.arcgis.com/en/arcgisdesk...0000001q000000

    However in doing so, you need to be careful not to delete the value entirely -- i.e. make sure to retain one instance of each value in the duplicate list.

    Unfortunately, I`m not much help in terms of integrating this into a modelbuilder-based approach.
    Last edited by jsnider@wwfcanada.org; 09-22-2011 at 01:24 PM.

  21. #21
    Erin Kenna
    Join Date
    Jun 2010
    Posts
    2
    Points
    2
    Answers Provided
    0


    2

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Quote Originally Posted by davidbrett View Post
    Any insight on modifying it to mark the duplicates with an incremental number per occurrence?
    DavidBrett,

    Python code block:
    Code:
    UniqueDict = {}
    def isDuplicateIndex(inValue):
        UniqueDict.setdefault(inValue,0)
        UniqueDict[inValue] += 1
        return UniqueDict[inValue]
    Expression:
    Code:
    isDuplicateIndex( !YOUR_FIELD! )
    Hope that helps, Erin.

  22. #22
    David Brett
    Join Date
    Nov 2009
    Posts
    33
    Points
    1
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Thanks Erin! This is exactly what I need.

  23. #23
    matthew driscoll
    Join Date
    Aug 2010
    Posts
    25
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    This is great! Can you help me put this in a stand alone python script? Everything runs great until the Calculate Field management...thats where it fails. Can anyone see where I am going wrong?

    fclayer = arcpy.mapping.Layer(C:/data/BaseData.gbd/myLayer)
    fieldName = "Duplicate"
    expression2 = "isDuplicate(!soil_desc!)"
    uniqueList = []
    codeblock = """def isDuplicate(inValue):
    (INDENT)if inValue in uniqueList:
    (INDENTx2)return 1
    (INDENT)else:
    (INDENTx2)uniqueList.append(inValue)
    (INDENTx2)return 0"""

    arcpy.AddField_management(myLayer,fieldName,"SHORT")

    arcpy.CalculateField_management(myLayer,fieldName,expression2,"PYTHON",codeblock)
    Last edited by mbd7878; 01-18-2012 at 04:13 AM. Reason: typos

  24. #24
    David Brett
    Join Date
    Nov 2009
    Posts
    33
    Points
    1
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    The attached will increment duplicates in a user specified column. It is configured to run from ArcToolbox. You can hard code the variable if necessary.
    Attached Files

  25. #25
    matthew driscoll
    Join Date
    Aug 2010
    Posts
    25
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    I get the following error.

    <type 'exceptions.TypeError'>: 'Layer' object is unsubscriptable
    Failed to execute (Script)

    any ideas why?

    Got it figured out. Here is worked for me.

    dupField = "soil_desc"
    markField = "Duplicate"

    arcpy.AddField_management(soilssortlayer,markField,"FLOAT")

    rows = arcpy.UpdateCursor(soilssortlayer,"","","",dupField)

    mylist = []

    i = -1
    for row in rows:
    if i == -1:
    value = row.getValue(dupField)
    mylist.append(value)
    i += 1
    if row.getValue(dupField) != value:
    value = row.getValue(dupField)
    mylist.append(value)
    i = 0
    row.Duplicate = i
    rows.updateRow(row)
    Last edited by mbd7878; 01-18-2012 at 11:03 AM.

  26. #26
    Biniam Weldemichael
    Join Date
    May 2011
    Posts
    8
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Hi there,
    I am wandering for your help. I would like to copy selected rows of an attribute table based on the values of a single field. and duplicate (paste) the copied rows in the attribute table itself with some editing afterwards on the object id for example. can some one provide me information(idea ) on how to put it in python script.
    thanx
    B. Weldmeichael
    Biniam W.Michael

  27. #27
    matthew driscoll
    Join Date
    Aug 2010
    Posts
    25
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    feature = "C://Data//Feature"
    newField = "NEWField"

    # Create the new Field
    newField = arcpy.AddField_management(feature,newField,"TEXT","50")

    # Do your Select by Attribute normally
    expression = "your select by attribute expression"
    arcpy.SelectLayerByAttribute_management(feature,"NEW_SELECTION",expression)

    # Calculate the new field
    arcpy.CalculateField_management(feature,newField,"!OLDField!","PYTHON")


    I am new to these Scripts, so I am not sure if when you calculate the new field it will honor those just selected or if it will update all the rows, but I think it will work. Depending on how complex the "some editing afterwards" is, but if it is simple you can add onto your expression on the Calculation (the !OLDField! part).
    This has not been tested, and like I said I am a beginner, but that should give you a place to start.
    Last edited by mbd7878; 01-20-2012 at 10:33 AM. Reason: typo

  28. #28
    Craig Carsley
    Join Date
    Jan 2012
    Posts
    48
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Quote Originally Posted by ColterSikora View Post
    Hello!

    I am looking to find duplicate attributes in my shapefile as shown in this forum. However, when I try the Python code example I get the following error message: "Error 000539: Error running expression: uniqueList = Error running expression: uniqueList = [] "

    Here is the code that I entered:

    uniqueList = [ !Duplicate! ]
    def isDuplicate(inValue):
    if inValue in uniqueList:
    return 1
    else:
    uniqueList.append(inValue)
    return 0

    isDuplicate(!Calculate!)


    ---

    "Duplicate" is the field I want the results entered into, "Calculate" is the field that I want duplicates selected from.

    Hopefully this is enough code to explain my situation.

    Thank you for your help in advance!

    Sincerely,

    Colter
    I used this script to identify duplicates once, last week, and it worked perfectly. For some reason, I'm trying to do it again this week, and I'm being consistently presented with a syntax error for line 1. The only contrast between my two sets of data is size (some 43,000 records vs. 233,000). Could this difference in size be enough to cause the script to fail? I've noticed the process gets about 70%-80% complete before it 'fails to execute'. I still get some results, but a large portion of the "Dup" field is populated with <null> values.

    Any guidance would be greatly appreciated!

  29. #29
    Pengaturan Penataan Pertanahan
    Join Date
    Mar 2012
    Posts
    2
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    thank s to Mark its work..... and for JCMolina thanks for the step of the script...

    this for Phyton, if anyone know script for the VBA please ...
    Last edited by pppbpnst; 03-12-2012 at 11:40 PM.

  30. #30
    Peter Peng
    Join Date
    Apr 2012
    Posts
    11
    Points
    0
    Answers Provided
    1


    0

    Post Re: ArcGIS 10 Pre-Release finding duplicates

    Hi Guys, I didn't want to start a new thread, instead I'll just post my problem here. Sorry if it seems like I am not understanding it all, our department is just starting to learn Python on ArcMap 10!

    I have 3 fields:
    1. "isDuplicate" (I want this field flag as 1 for duplicates and 0 for unique values)
    2. x (coordinate calculated by Field Calculator)
    3. y (coordinate calculated by Field Calculator)

    I want the script do something like this:

    For All_Objects
    If x is unique AND y is unique,
    isDuplicate = 0
    Else
    isDuplicate = 1
    End

    But I know that doesn't make much sense. Basically this script should identify all points that have the same X/Y coordinates and label the fields as such. There may be other easier ways to do this, but there are other functions that I will need to add to this script anyways.

    I just very new with Python sorry

  31. #31
    matthew driscoll
    Join Date
    Aug 2010
    Posts
    25
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    I think this will work, but I have not tested it. Also there might be an easier way. But what I did was create a new field that combines the values from the x & y fields (you can delete this new field at the end if you want). Then search that one field for duplicates and add the values to the isDuplicate field.
    You also might need to make sure when you created isDuplicate it is a FLOAT field.



    Code:
    ObjectLayer = "Name of Layer to be flagged"
    
    arcpy.AddField_management(ObjectLayer,"Working","TEXT","50")
    
    expression = "x + y"
    arcpy.CalculateField_management(ObjectLayer,"Working",expression,"PYTHON")
    
    dupField = "Working"
    Duplicate = "isDuplicate" 
    
    rows = arcpy.UpdateCursor(ObjectLayer,"","","",dupField)
    
    mylist = []
    
    i = -1
    for row in rows:
        if i == -1:
            value = row.getValue(dupField)
            mylist.append(value)
        i += 1
        if row.getValue(dupField) != value:
            value = row.getValue(dupField)
            mylist.append(value)
            i = 0
        row.isDuplicate = i
        rows.updateRow(row)
    Last edited by mbd7878; 04-04-2012 at 12:11 PM.

  32. 04-04-2012 12:05 PM

    Reason
    duplicate post

  33. 04-04-2012 12:09 PM

    Reason
    duplicate

  34. #32
    David Brett
    Join Date
    Nov 2009
    Posts
    33
    Points
    1
    Answers Provided
    0


    1

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    You can use the following in the field calculator

    Parser = Python
    check 'Show Codeblock'

    In the Pre-Logic Script Code:
    Code:
    uniqueList = []
    def isDuplicate(inValue):
      if inValue in uniqueList:
        return 1
      else:
        uniqueList.append(inValue)
        return 0
    in the Box below where it says FIELD_NAME =

    Code:
    isDuplicate ( !FieldName1! + !FieldName2! )
    tested and works.

  35. #33
    Angela Gotch
    Join Date
    Nov 2011
    Posts
    1
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Juan, that worked perfectly. Thanks! ~Angela

  36. #34
    Chris Graves
    Join Date
    Mar 2010
    Posts
    34
    Points
    1
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Hi Guys,

    Further to this thread I would like select duplicate records in a field and then make these unique using the Field Calculator. For example I have parcel IDs (e.g. F555 A30) which might occur 3 times. I want to select these and change the name of these with either 'F555 A30 B' or 'F555 A30 1'. Also I would like the change to occur in the field the original records are in and not create a new field.

    Thanks in advance,

    Chris

  37. #35
    Primaristianti Putri
    Join Date
    Feb 2013
    Posts
    1
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Quote Originally Posted by davidbrett View Post
    You can use the following in the field calculator

    Parser = Python
    check 'Show Codeblock'

    In the Pre-Logic Script Code:
    Code:
    uniqueList = []
    def isDuplicate(inValue):
      if inValue in uniqueList:
        return 1
      else:
        uniqueList.append(inValue)
        return 0
    in the Box below where it says FIELD_NAME =

    Code:
    isDuplicate ( !FieldName1! + !FieldName2! )
    tested and works.

    Hi David!

    I have the same issue with Peter and use your code. It works great actually when there are duplicate XY in the data. However, when there are no such data, the script's output highlighting records that has duplicate X value AND/OR duplicate Y value (screenshot attached).

    How to make a script that find only the absolute duplicate value of both X AND Y?

    Thank you.

    Prima.
    Attached Thumbnails Attached Thumbnails Click image for larger version

Name:	Rec_2425_Duplicate X value.jpg‎
Views:	112
Size:	126.9 KB
ID:	22034   Click image for larger version

Name:	Rec_2425_Duplicate Y value.jpg‎
Views:	154
Size:	162.8 KB
ID:	22035  

  38. #36
    David Brett
    Join Date
    Nov 2009
    Posts
    33
    Points
    1
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Quote Originally Posted by pputri View Post
    Hi David!

    I have the same issue with Peter and use your code. It works great actually when there are duplicate XY in the data. However, when there are no such data, the script's output highlighting records that has duplicate X value AND/OR duplicate Y value (screenshot attached).

    How to make a script that find only the absolute duplicate value of both X AND Y?

    Thank you.

    Prima.

    Hi,

    Let's try another way
    Code:
    uniqueList = []
    i = 0
    def isDuplicate(inValue):
        global i
        if inValue in uniqueList:
            i += 1
            return i
        else:
            uniqueList.append(inValue)
        return 0
    This will increment the dup field per the matching duplicates. You can test how this works by using one inValue field at a time, and then both.

    Example 1:
    dup =
    isDuplicate( !x!)

    Example 2:
    dup =
    isDuplicate( !y!)

    Example 3:
    dup =
    isDuplicate( !x! + !y!)

  39. #37
    David Brett
    Join Date
    Nov 2009
    Posts
    33
    Points
    1
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Quote Originally Posted by cgraves73 View Post
    Hi Guys,

    Further to this thread I would like select duplicate records in a field and then make these unique using the Field Calculator. For example I have parcel IDs (e.g. F555 A30) which might occur 3 times. I want to select these and change the name of these with either 'F555 A30 B' or 'F555 A30 1'. Also I would like the change to occur in the field the original records are in and not create a new field.

    Thanks in advance,

    Chris
    Hi Chris,

    The safest way to do this is to create a new field and not update the field you are calculating during the calculation...logic: don't change the data your analyzing while analyzing the data.

    You can use the code I just posted to increment the dup field, then calculate the new id based on the old ID + dup value.

  40. #38
    Robert Blash
    Join Date
    Apr 2010
    Posts
    26
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Has anyone figured out how to simply select the duplicates without creating a new field or using summarize?

    Thanks,

    Rob

  41. #39
    Andrea B, GISP
    Join Date
    Jul 2010
    Posts
    360
    Points
    7
    Answers Provided
    3


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Quote Originally Posted by rblash View Post
    Has anyone figured out how to simply select the duplicates without creating a new field or using summarize?

    Thanks,

    Rob
    This is what I really need to do too. I need to export the records that are duplicates, but I need to have the duplicate with it. I.e. If I have 2 records with the same location number I want to export both the first record and the duplicate record. Am I missing something fairly easy to select these?

    Thanks,
    Andrea B, GISP
    GIS Coordinator

  42. #40
    Robert Blash
    Join Date
    Apr 2010
    Posts
    26
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Take a look at this add-in. Someone recycled the tool we were using in 9.X, seems to be reliable so far.

    http://www.arcgis.com/home/item.html...a2a9f7f519eb5a

  43. #41

    Join Date
    Sep 2013
    Posts
    1
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Good Morning! Is there a way this can this be done using the field calculator with a geodatabase feature class? It isn't working on my end... Thanks.

    Quote Originally Posted by hami1981 View Post
    #Here's a script that flags duplicates as "1" including the first occurence:
    #It works fine for strings and integers, but doesn't work for float values, to overcome the float
    #problem convert the values to strings.
    #Paste the following code in the Field Calculator:

    import arcpy
    inFeatures = "C:\\Folder\\Subfolder\\yourshapefile.shp"

    inField = "YOUR_FIELD" #The field you want to analyse
    rows = arcpy.SearchCursor(inFeatures)
    row = rows.next()
    ValueList = list()
    while row:
    currentValue = row.getValue(inField)
    ValueList.append(currentValue)
    row = rows.next()

    def isDuplicate(inValue):
    if ValueList.count(inValue) > 1:
    return 1
    else:
    return 0


    ##In the box below, paste:
    ## isDuplicate(!YOUR_FIELD!)

  44. #42
    Andrew Oakley
    Join Date
    Apr 2013
    Posts
    4
    Points
    1
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Is it possible to give weight to a feature with this code? Basically I have two sets of data that I have combined. One comes from GPS points and the other comes from geocoded points. I have a field named 'info_src" that distinguishes this for me. I want to erase the duplicate geocoded points NOT the GPS points. The script I have been using is pasted below. I want the duplicates marked but I want the geocoded point to be marked as the duplicate instead of the GPS point and just the opposite is occuring. Help!!

    uniqueList = []
    def isDuplicate(inValue):
    if inValue in uniqueList:
    return 1
    else:
    uniqueList.append(inValue)

    isDuplicate( !Account! )

  45. #43
    Iraida Rodriguez
    Join Date
    Mar 2011
    Posts
    1
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    I'm a Python beginner and there a couple things I don't understand in the script below.

    1. What does invalue mean?
    2. What is .setdefault(invalue,0)?
    3. What does += mean?


    Thanks


    Quote Originally Posted by ErinK View Post
    DavidBrett,

    Python code block:
    Code:
    UniqueDict = {}
    def isDuplicateIndex(inValue):
        UniqueDict.setdefault(inValue,0)
        UniqueDict[inValue] += 1
        return UniqueDict[inValue]
    Expression:
    Code:
    isDuplicateIndex( !YOUR_FIELD! )
    Hope that helps, Erin.

  46. #44
    David Brett
    Join Date
    Nov 2009
    Posts
    33
    Points
    1
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    There's a function named isDuplicateIndex and it takes one parameter. The name of that parameter = InValue.

    Using Field Calculator, Expression:
    Code:

    isDuplicateIndex( !YOUR_FIELD! )

    isDuplicateIndex is the name of function. InValue is a variable and is set to whatever is in the parenthesis: !YOUR_FIELD!

    the .setdefault method ...set's the default value of the dictionary. You can read about that here http://docs.python.org/2/library/stdtypes.html

    setdefault(key[, default])If key is in the dictionary, return its value. If not, insert key with a value of default and return default. default defaults to None.

    += is an increment: x = x + 1 is the same as x += 1.

  47. #45
    William Park
    Join Date
    Dec 2013
    Posts
    1
    Points
    0
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    I have a question and would like some input on how to go on with my concept

    So I used the Python code provided above and it works well.
    What I am curious as to doing is:
    -- Is there a way to create a script / something that is continuous to check for duplicates
    -- Have some kind of "validation rule" that will give you an error/warning message if you enter a "duplicate value" that is already within the database. (or attributes table feature list)

    Example


    DC_ID : 1
    DC_ID : 2
    DC_ID : 3
    DC_ID : 4
    DC_ID : null <---- It will warn you if you put any value for 1-4 within this field.

  48. #46
    Holly Copeland
    Join Date
    May 2011
    Posts
    9
    Points
    1
    Answers Provided
    0


    0

    Default Re: ArcGIS 10 Pre-Release finding duplicates

    Hi Kent,
    The code below is what I'm looking for, with the exception that I want to find duplicates across 4 fields... in my case, Latitude, Longitude, BirdID and DateTime. Can you give me pointers for how to make that modification? Thanks much, Holly

    ====================
    Using the Field Calculator in ArcGIS 10. You can use Python to easily find duplicate values in a given field. In Python you can declare a list outside of the function so that it can retain added values. Here is a code sample:

    Code:
    uniqueList = []
    def isDuplicate(inValue):
      if inValue in uniqueList:
        return 1
      else:
        uniqueList.append(inValue)
        return 0
    At the bottom of the Field Calculator, you will have to enter in the function and field name like this:
    Code:
    isDuplicate(!FIELD_NAME!)
    Thanks for your post.

    Kent M.
    esri[/QUOTE]

+ Reply to Thread

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts