GEDCOM Crawler

Download your entire family tree

version 1.1
© 2006 Antone Roundy
All Rights Reserved
Introduction | Requirements | License | Limitations | Support | Instructions
More Information sources
I have not used these sites, so I can't vouch for them personally, but they are popular, so I imagine you might find them useful.

"197 GEDCOMs downloaded, 4832 to go...Ugh!"

Have you ever tried to download all the GEDCOMs for your ancestors to import into PAF? How many did you finish before you gave up? I'd finished all of maybe 2 or 3 and immediately realized that doing it manually would take forever. So I put off getting started for years...until I realized that I could write a script to download them all automatically.

GEDCOM Crawler generates a single GEDCOM file containing all of the information from the LDS Ancestral File for the ancestors of the indicated family. In about 45 minutes, it downloaded data on 5,525 families in my family tree, stretching all the way back to 124 A.D.!

NOTE: Because the data comes from the Ancestral File, it does not contain LDS ordinance data. Look for other tools to help you import this data into PAF.


Genealogy News

Newsfeed display by CaRP

Return of Random Genealogy?
17 Mar 2010 at 10:23pm
Random Genealogy:-
[from EOGN Other News - powered by KickRSS]

Next event: Whittier, California 27 February 2010
17 Mar 2010 at 10:23pm
Paula's Genealogical Eclectica:- My next speaking engagement is February 27th for the great folks at the Whittier Area Genealogical Society (WAGS) in California. I know several people in this group...
[from EOGN Other News - powered by KickRSS]

A great woman and genealogist has passed away -- Marsha Hoffman Rising, CG, FASG
17 Mar 2010 at 10:23pm
Paula's Genealogical Eclectica:- If you never had the chance to interact with Marsha, you have missed out on some special genealogical education from an awesome person. Marsha passed away peacefull...
[from EOGN Other News - powered by KickRSS]

FGS 2010 Conference program and registration is live online!
17 Mar 2010 at 10:23pm
Paula's Genealogical Eclectica:- The full program, including meal events, special events, keynote speakers, special lectures, and information on the updated Focus on Societies Day is now online at ...
[from EOGN Other News - powered by KickRSS]

Even figure skaters have an archives
17 Mar 2010 at 10:23pm
Paula's Genealogical Eclectica:- While I sit here watching the figure skating at the 2010 Olympics, I decided to check out the World Figure Skating Musuem and Hall of Fame. There is an Archives! I ...
[from EOGN Other News - powered by KickRSS]

Cemetery cannot be destroyed for O'Hare International Airport.
17 Mar 2010 at 10:23pm
Paula's Genealogical Eclectica:- For some time now, I have been reading about Chicago's plans to add a runway to O'Hare International Airport. Sadly, this necessitated the moving of many bodies bur...
[from EOGN Other News - powered by KickRSS]

My Irish surnames on this St. Patrick's Day
17 Mar 2010 at 10:23pm
Paula's Genealogical Eclectica:- I live in the U.S. but I have ancestral ties to Canada, Denmark, England, France, Ireland, Germany, Scotland, and Sweden but the Irish ancestry is prevelant. They a...
[from EOGN Other News - powered by KickRSS]

Requirements

Because of the amount of bandwidth GEDCOM Crawler uses when downloading large family trees, I do not run it on my server on behalf of anyone but family members. GEDCOM Crawler can run on a computer that meets the following requirements:
  • Internet connection (preferably high-speed)
  • Perl
  • Command line access (ie. if you are connecting to the computer over a network rather than sitting at the terminal, you will need something like Telnet or SSH access)

License

By downloading GEDCOM Crawler, you agree to the terms of this license.
  • Subject to the terms of this license, you may download and use GEDCOM Crawler at no charge.
  • You may make modifications to your own copy.
  • You may NOT redistribute GEDCOM Crawler in whole or in part, nor modified copies or other scripts based on GEDCOM Crawler, whether commercially or for free without prior written consent. Refer people to this website instead.
  • This script is provided as-is with no warranty whatsoever. Use it at your own risk. If you live in a jurisdiction that does not allow complete disclaimer of warranty, then you may NOT download nor use GEDCOM Crawler.
The data from the Ancestral File is covered by its own copyright terms. See the FamilySearch.org website for more information.

Known Limitations

Support

Sorry, I'm way too busy--no technical support is available at this time.

Instructions

  1. Download and unzip GEDCOM-Crawler.zip (By downloading GEDCOM Crawler, you agree to the above license terms). NOTE: The files use UNIX style linebreaks. When opening the files on other operating systems, ...things may look weird. I may make additional packages with different linebreak styles later.
  2. Open gedcom-crawler-conf.pl in a text editor like Notepad, or better yet, a programmer's text editor.
  3. If the computer on which GEDCOM Crawler will run has a host name (for example, if you are going to run it on your webserver), enter the host name in $myHostName. Otherwise you will need to find out the IP address of the computer and enter it in $myHostIP. (If you enter the host name, you may leave $hostIP blank.)
  4. Save your changes (be sure to save as a plain text file).
  5. If you are going to run GEDCOM Crawler on a different computer than the one you have it on, upload all four files to that computer.
  6. If necessary, make all four files executable. For example, on a UNIX or Linux computer, you might type the following command on the command line from within the directory where the files are located: chmod 700 gedcom-crawler*
  7. Find the family ID for the most recent ancestor on the line whose data you wish to download:
    1. Go to the Ancestral File Search Form
    2. Find the ancestor
    3. Go to their individual record
    4. Copy the "Family" link on the right side of the page across from their name
    5. (don't go to that page and copy the URL in your browser's address bar--that won't be the right one--instead, right click the "Family" link and select "copy URL" or something like that from that menu)
    6. The number after "familyid=" in that URL is the family ID.
  8. From the directory where GEDCOM Crawler is installed, execute the following command, substituting in the desired family ID: ./gedcom-crawler.pl 123456 Here, the fun begins! GEDCOM Crawler will start downloading and combining GEDCOM files for your ancestors. Each time it downloads a file, it will tell you how many families' data it has processed, and how many family IDs it has found but not processed. Watch both numbers go up for a while as it finds more and more families! After the numbers, it will display one dot for each line in the GEDCOM file--some families are bigger than others.

    IMPORTANT: GEDCOM Crawler will create a file whose name is the family ID specified on the command line plus ".ged". If you already have a file by that name in the same directory as GEDCOM Crawler, it will be overwritten.
  9. If you wish to download other family lines without getting duplicates, run GEDCOM Crawler again with the new family ID, adding a space and an "x" to the command line like this: ./gedcom-crawler.pl 567890 x This will prevent GEDCOM Crawler from clearing the family IDs it has already downloaded from the database.
  10. If you have already cleared the family IDs from the database and wish to avoid duplicates, or already have a GEDCOM file containing Ancestral File family IDs which you don't wish to duplicate, run gedcom-crawler-skip-families.pl as follows to put the family IDs into the database (substituting in the name of your GEDCOM file): ./gedcom-crawler-skip-families.pl gedcom-file.ged
  11. As with gedcom-crawler.pl, you may tell gedcom-crawler-skip-families.pl not to empty the database before it runs by adding a space and an "x" to the command line: ./gedcom-crawler-skip-families.pl gedcom-file.ged x
  12. Send your GEDCOM files to your family members!