+ Reply to Thread
Results 1 to 19 of 19

Thread: Simple Merge of Shapefiles in a single folder

  1. #1
    Tyler Jones
    Join Date
    Oct 2012
    Posts
    13
    Points
    0
    Answers Provided
    0


    0

    Default Simple Merge of Shapefiles in a single folder

    Hello,

    I am very new to Python scripting and apparently not very good at it. I have a single folder with many shapefiles that I need to merge into a single shapefile. They are in the same projection and will not hit the 2gb limit so I assume this would be the easiest type of script to write. I have been trying to execute from PythonWin since that is the IDE the ESRI training uses but at this point I don't care where I execute it just that it works. I assume the workflow to be something like: import arcpy, set the environment to that particular folders location, apply overwrite to true, create a list of the shapefiles in the folder (i assume by telling it to look for all files with the ending ".shp"), the using the Merge_management to execute the merge.
    So with that said, could anyone offer some help. I have found a few other scripts online but they have some extra arguements for parsing and what not and I could not get them to work. All help appreciated

  2. #2
    Caleb Mackey

    Join Date
    Jul 2012
    Posts
    440
    Points
    463
    Answers Provided
    54


    0

    Default Re: Simple Merge of Shapefiles in a single folder

    Try this:

    Code:
    import arcpy, os
    
    out = r'Output\folder\location' # change this
    arcpy.env.workspace = r'Some\Path\Name'  #change this
    shplist =  arcpy.ListFeatureClasses('*.shp')
    arcpy.Merge_management(shplist, os.path.join(out, 'Merged_fc.shp'))
    print 'done'

  3. #3
    Tyler Jones
    Join Date
    Oct 2012
    Posts
    13
    Points
    0
    Answers Provided
    0


    0

    Default Re: Simple Merge of Shapefiles in a single folder

    I'm getting an error for invalid input. This seems to be the same problem I have had with the other scripts. Can't find any extra info on the input parameters in the help section.

  4. #4
    Caleb Mackey

    Join Date
    Jul 2012
    Posts
    440
    Points
    463
    Answers Provided
    54


    0

    Default Re: Simple Merge of Shapefiles in a single folder

    Weird, this works for me. The input parameters need to be supplied as a list (in square brackets []). Can you add a print statement to show the list? Perhaps you can find the problem there. Here is a script converting coverages to shapefiles and merging at the end. I have a similar merge operation below:

    Code:
    import arcpy, os, sys, traceback
    arcpy.env.overwriteOutput = True
    
    # Workspace for interchange files
    ws = r'C:\IGIC\Beginner\Unit_1\DATA\EffigyMounds\InterchangeFiles'
    arcpy.env.workspace = ws
    
    # Local Variables
    out = r'C:\IGIC\Beginner\Unit_1\DATA\EffigyMounds\Fire'
    
    try:
        # Creates a folder if it doesn't already exist
        if not os.path.exists(out):
            os.makedirs(out)
    
        # List all ArcInfo Interchange files (.e00)
        for cov in arcpy.ListFiles('*.e00'):
            arcpy.ImportFromE00_conversion(cov, ws, cov.split('.')[0])
            print 'Converted %s' %cov
    
        # Exclude Interchange files and info folder
        for cov in arcpy.ListFiles():
            if not '.e00' in cov and cov != 'info':
                arcpy.env.workspace = os.path.join(ws,cov)
                for fc in arcpy.ListFeatureClasses('*polygon'):
                    arcpy.FeatureClassToFeatureClass_conversion(fc, out, '%s.shp' %cov)
                    shp = os.path.join(out, '%s.shp' %cov)
                    print 'Converted Coverage to %s' %shp
    
        # List shapefiles in folder to create input list for merge
        arcpy.env.workspace = out
        shplist =  arcpy.ListFeatureClasses()
        arcpy.Merge_management(shplist, os.path.join(out, 'All_Fires.shp'))
        print 'done'
        
    except:
        
        # Get the traceback object
        tb = sys.exc_info()[2]
        tbinfo = traceback.format_tb(tb)[0]
        pymsg = "PYTHON ERRORS:\nTraceback info:\n" + tbinfo + \
                "\nError Info:\n" + str(sys.exc_info()[1])
        msgs = "ArcPy ERRORS:\n" + arcpy.GetMessages(2) + "\n"
        arcpy.AddError(pymsg)
        arcpy.AddError(msgs)
    
        # Print Python error messages for use in Python / Python Window
        print pymsg + "\n"
        print msgs
    Last edited by Caleb1987; 12-05-2012 at 03:08 PM.

  5. #5
    R.D. Harles
    Join Date
    Apr 2010
    Posts
    220
    Points
    5
    Answers Provided
    1


    0

    Default Re: Simple Merge of Shapefiles in a single folder

    This is the ultra simplified version. Merge all fcs in the given directory into a shapefile called "Merged.shp"

    Code:
    import arcpy
    arcpy.env.workspace = "c:/junk/mergeMe"
    fcs = arcpy.ListFeatureClasses()
    arcpy.Merge_management(fcs, "Merged.shp")
    Last edited by rdharles; 12-06-2012 at 04:31 AM.
    ---------------
    R.D. Harles
    ADCi
    www.adci.com

  6. #6
    Michael Volz
    Join Date
    May 2010
    Posts
    1,381
    Points
    186
    Answers Provided
    23


    0

    Post Re: Simple Merge of Shapefiles in a single folder

    Tyler:

    What exactly is the error message that you are receiving?

    Also, can you post the code that you are trying to run?

  7. #7
    Tyler Jones
    Join Date
    Oct 2012
    Posts
    13
    Points
    0
    Answers Provided
    0


    0

    Default Re: Simple Merge of Shapefiles in a single folder

    So this is what my code looks like (copied from the python window). I am sure I have some simple error but it is hard to fix when your still at the starting gate. Is there some way that I can actually see what the Listing looks like? As in have it create a simple .txt or something similar so I can see if the ListFeatureClasses command is even finding what it is supposed to find? I tried to simply "print Listing" but all I get are "[]" as a result which makes me think it is not seeing the file in the first place for some reason. This file has about 15 shapefiles (and their associated files) generated by the eCognition image processing software and I am using it for testing because I have much bigger ones with thousands of very small shapefiles that I am really gonna need Python for. Anyway, my error message below is talking about input parameters. Really wish they offered a python course here at my university. Again I appreciate the help from you guys.


    Code:
    import arcpy
    >>> arcpy.env.workspace = "D:eCognition Results/Landcover/Buildings"
    >>> Listing = arcpy.ListFeatureClasses()
    >>> arcpy.Merge_management(Listing,"MergedFile.shp")
    Runtime error <class 'arcgisscripting.ExecuteError'>: Failed to execute. Parameters are not valid. ERROR 000735: Input Datasets: Value is required Failed to execute (Merge).

  8. #8
    Caleb Mackey

    Join Date
    Jul 2012
    Posts
    440
    Points
    463
    Answers Provided
    54


    0

    Default Re: Simple Merge of Shapefiles in a single folder

    It appears that you are getting an empty list. I would double check your path to make sure it is correct. Also, not sure if this is the issue, but I would try putting an underscore between Cognition and Results (Cognition_Results) for that folder. ArcGIS does not like spaces in paths. I would do that then use your print statement (your print statement is correct).

  9. #9
    Michael Volz
    Join Date
    May 2010
    Posts
    1,381
    Points
    186
    Answers Provided
    23


    0

    Post Re: Simple Merge of Shapefiles in a single folder

    Tyler:

    It looks like this line might show the problem.

    arcpy.env.workspace = "D:eCognition Results/Landcover/Buildings"

    Should it read something like this instead:

    arcpy.env.workspace = "D:/eCognition Results/Landcover/Buildings"

    where you are currently missing a slash.

  10. #10
    Tyler Jones
    Join Date
    Oct 2012
    Posts
    13
    Points
    0
    Answers Provided
    0


    0

    Default Re: Simple Merge of Shapefiles in a single folder

    You are right. Im missing a forward slash after "D:". Now I just feel silly. So I was able to print the Listing file but now at the mergeManagment I get the same error but with the actual filenames I was looking for at least.
    Code:
    arcpy.Merge_management(Listing,"MergedFile.shp")
    Runtime error <class 'arcgisscripting.ExecuteError'>: Failed to execute. Parameters are not valid. ERROR 000732: Input Datasets: Dataset campus_ortho.tiles.tile00.v5.shp;campus_ortho.tiles.tile01.v8.shp;campus_ortho.tiles.tile02.v5.shp;campus_ortho.tiles.tile03.v6.shp;campus_ortho.tiles.tile04.v5.shp;campus_ortho.tiles.tile05.v9.shp;campus_ortho.tiles.tile06.v9.shp;campus_ortho.tiles.tile07.v9.shp;campus_ortho.tiles.tile08.v10.shp;campus_ortho.tiles.tile09.v8.shp;campus_ortho.tiles.tile10.v8.shp;campus_ortho.tiles.tile11.v8.shp;campus_ortho.tiles.tile12.v8.shp;campus_ortho.tiles.tile13.v9.shp;campus_ortho.tiles.tile14.v9.shp;campus_ortho.tiles.tile15.v8.shp;campus_ortho.tiles.tile17.v8.shp;campus_ortho.tiles.tile18.v9.shp does not exist or is not supported Failed to execute (Merge). 
    >>>
    An example of the shapefile names would be "campus_ortho.tiles.tile00.v5.shp" whereas each tile from my project was assigned a unique name upon export. Does this name have anything to do with the error I am recieving? I am not sure what characters are reserved in filenames when it comes to this.

  11. #11
    Caleb Mackey

    Join Date
    Jul 2012
    Posts
    440
    Points
    463
    Answers Provided
    54


    0

    Default Re: Simple Merge of Shapefiles in a single folder

    It may have something to do with the periods before the .shp extension on the shapefiles. I would rename them by replacing the periods with underscores. Try this:

    Code:
    import arcpy, os
    arcpy.env.workspace = "D:/eCognition Results/Landcover/Buildings"
    for file in arcpy.ListFiles('*.shp'):
        os.rename(os.path.join(ws,file), os.path.join(ws,file.replace('.','_')[:-4] + '.shp'))
    Listing = arcpy.ListFeatureClasses()
    arcpy.Merge_management(Listing,"MergedFile.shp")
    If this doesn't work, I'm stumped.

  12. #12
    Michael Volz
    Join Date
    May 2010
    Posts
    1,381
    Points
    186
    Answers Provided
    23


    0

    Post Re: Simple Merge of Shapefiles in a single folder

    Are these raster datasets? The merge_management documentation says that this data type is not supported.

  13. #13
    Tyler Jones
    Join Date
    Oct 2012
    Posts
    13
    Points
    0
    Answers Provided
    0


    0

    Default Re: Simple Merge of Shapefiles in a single folder

    Caleb,

    I went and renamed the files by hand and it worked like a charm. Thanks for the help. Now I just have to get the rename code to work so I don't have to do them all by hand.

  14. #14
    Caleb Mackey

    Join Date
    Jul 2012
    Posts
    440
    Points
    463
    Answers Provided
    54


    0

    Default Re: Simple Merge of Shapefiles in a single folder

    Quote Originally Posted by tjonest3 View Post
    Caleb,

    I went and renamed the files by hand and it worked like a charm. Thanks for the help. Now I just have to get the rename code to work so I don't have to do them all by hand.
    I guess I should have mentioned that code was not thoroughly tested. I copied several of the names of your shapefiles and renamed them as text files and ran this code. It did successfully convert those txt files to shp, but I do not know how that would work with real shapefiles. It may work with taking out the wild card that I had in the first time. I also changed it to ListFeatureClasses() which I do not know why I didn't do that the first time.

    Code:
    import arcpy, os
    arcpy.env.workspace = "D:/eCognition Results/Landcover/Buildings"
    for file in arcpy.ListFeatureClasses():
        os.rename(os.path.join(ws,file), os.path.join(ws,file.replace('.','_')[:-4] + '.shp'))
    Listing = arcpy.ListFeatureClasses()
    arcpy.Merge_management(Listing,"MergedFile.shp")
    Last edited by Caleb1987; 12-06-2012 at 10:31 AM.

  15. #15
    Caleb Mackey

    Join Date
    Jul 2012
    Posts
    440
    Points
    463
    Answers Provided
    54


    0

    Default Re: Simple Merge of Shapefiles in a single folder

    If THAT doesn't work (renaming shapefiles may not have been a good idea to begin with), here is my last idea:

    Code:
    import arcpy, os
    ws = "D:/eCognition Results/Landcover/Buildings"
    arcpy.env.workspace = ws
    out = "D:/eCognition Results/Landcover/Buildings/Outputs"
    for file in arcpy.ListFeatureClasses():
        arcpy.CopyFeatures_management(file, os.path.join(out,file.replace('.','_')[:-4] + '.shp'))
    arcpy.env.workspace = out   
    Listing = arcpy.ListFeatureClasses()
    arcpy.Merge_management(Listing,"MergedFile.shp")
    If this does not work, Python: 1, Me: 0
    Last edited by Caleb1987; 12-06-2012 at 10:56 AM.

  16. #16
    Tyler Jones
    Join Date
    Oct 2012
    Posts
    13
    Points
    0
    Answers Provided
    0


    0

    Default Re: Simple Merge of Shapefiles in a single folder

    I tried the first one and the logic seems to make sense to me but I get an 'exeptions.NameError'>: name 'ws' is not defined' So I am looking over the rename function to see what those parameters are all about. Can't be far from it now. Again man thanks for all the help.

  17. #17
    Caleb Mackey

    Join Date
    Jul 2012
    Posts
    440
    Points
    463
    Answers Provided
    54


    0

    Default Re: Simple Merge of Shapefiles in a single folder

    Quote Originally Posted by tjonest3 View Post
    I tried the first one and the logic seems to make sense to me but I get an 'exeptions.NameError'>: name 'ws' is not defined' So I am looking over the rename function to see what those parameters are all about. Can't be far from it now. Again man thanks for all the help.
    Ah! That is my fault! I had a ws variable in my test script on my end. Try this:

    Code:
    import arcpy, os
    ws = "D:/eCognition Results/Landcover/Buildings"
    arcpy.env.workspace = ws
    for file in arcpy.ListFeatureClasses():
        os.rename(os.path.join(ws,file), os.path.join(ws,file.replace('.','_')[:-4] + '.shp'))
    Listing = arcpy.ListFeatureClasses()
    arcpy.Merge_management(Listing,"MergedFile.shp")
    The problem was I had a variable set for the workspace and used that to append the paths and forgot to do that in yours. If you copied and pasted what I had, that was probably the problem the whole time! Sorry!
    Last edited by Caleb1987; 12-06-2012 at 10:57 AM.

  18. #18
    Tyler Jones
    Join Date
    Oct 2012
    Posts
    13
    Points
    0
    Answers Provided
    0


    0

    Default Re: Simple Merge of Shapefiles in a single folder

    Works!
    Only thing is that it only renames the .shp files and none of the associated files (.dbf,.prj, ect.) so the merge gives an error saying it cannot be completed probably because it cannot see the associated filetypes. However, I assume that I can just rewrite this for each of the filetypes and once they are all renamed the merge should work as expected? Fingers crossed.

  19. #19
    Caleb Mackey

    Join Date
    Jul 2012
    Posts
    440
    Points
    463
    Answers Provided
    54


    0

    Default Re: Simple Merge of Shapefiles in a single folder

    Quote Originally Posted by tjonest3 View Post
    Works!
    Only thing is that it only renames the .shp files and none of the associated files (.dbf,.prj, ect.) so the merge gives an error saying it cannot be completed probably because it cannot see the associated filetypes. However, I assume that I can just rewrite this for each of the filetypes and once they are all renamed the merge should work as expected? Fingers crossed.
    I would just copy the shapefiles with a different name into another folder, much safer:

    Code:
    import arcpy, os
    ws = "D:/eCognition Results/Landcover/Buildings"
    arcpy.env.workspace = ws
    out = "D:/eCognition Results/Landcover/Buildings/Outputs"
    for file in arcpy.ListFeatureClasses():
        arcpy.CopyFeatures_management(file, os.path.join(out,file.replace('.','_')[:-4] + '.shp'))
    arcpy.env.workspace = out   
    Listing = arcpy.ListFeatureClasses()
    arcpy.Merge_management(Listing,"MergedFile.shp")
    This *should work.

+ Reply to Thread

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts