GEDCOM Crawler

Download your entire family tree

version 1.1
© 2006 Antone Roundy
All Rights Reserved
Introduction | Requirements | License | Limitations | Support | Instructions
More Information sources
I have not used these sites, so I can't vouch for them personally, but they are popular, so I imagine you might find them useful.

"197 GEDCOMs downloaded, 4832 to go...Ugh!"

Have you ever tried to download all the GEDCOMs for your ancestors to import into PAF? How many did you finish before you gave up? I'd finished all of maybe 2 or 3 and immediately realized that doing it manually would take forever. So I put off getting started for years...until I realized that I could write a script to download them all automatically.

GEDCOM Crawler generates a single GEDCOM file containing all of the information from the LDS Ancestral File for the ancestors of the indicated family. In about 45 minutes, it downloaded data on 5,525 families in my family tree, stretching all the way back to 124 A.D.!

NOTE: Because the data comes from the Ancestral File, it does not contain LDS ordinance data. Look for other tools to help you import this data into PAF.


Genealogy News

Newsfeed display by CaRP


[CaRP] XML error: junk after document element at line 6 - Unknown document format.
Personal Ancestry Writer II Version 81 for Macintosh
3 Jul 2008 at 3:07pm
I have written before about Personal Ancestry Writer II (PAWriter II), a free genealogy program for Macintosh. I find it to be a great program, especially considering its price tag of zero. PAWrite...
[from Eastman's Online Genealogy Newsletter]

(+) Throw Your Telephone Away
3 Jul 2008 at 9:00am
NOTE: The following article has nothing to do with genealogy. However, I have written about telephone cost reduction methods before, and some people seem to appreciate the articles; so, I'll publis...
[from Eastman's Online Genealogy Newsletter]

Southern California Genealogy Jamboree 2008: Video Introductions
2 Jul 2008 at 9:06pm
Roots Television took a camera to the recent Southern California Genealogy Jamboree held in Burbank, California. They made a number of videos of the events and people there, and those videos are no...
[from Eastman's Online Genealogy Newsletter]

The Tables Are Turned: Matthew Poe Interviews Dick Eastman
2 Jul 2008 at 9:03pm
If you have been reading this newsletter for a while, you may know that Roots Television and I have found many interesting genealogists. I have been fortunate enough to interview a number of people...
[from Eastman's Online Genealogy Newsletter]

Ancestry.com Launches New Homepage
2 Jul 2008 at 8:26pm
The following announcement was written by The Generations Network, the parent company of Ancestry.com: Ancestry.com today launched a brand-new homepage that logged-in members will see the next time...
[from Eastman's Online Genealogy Newsletter]

St. George Family History Expo Call for Papers
2 Jul 2008 at 6:33am
The following announcement was written by Holly Hansen, President, My Ancestors Found: Family History Expo ?Digging Roots: It's that easy!? Call for Papers Dixie Convention Center 1835 Convention C...
[from Eastman's Online Genealogy Newsletter]

Southern California Genealogy Jamboree is a Success
1 Jul 2008 at 6:26pm
You may have noticed that I haven't posted many new articles in the past few days. I am also late in sending out the weekly Plus Edition newsletter by e-mail. The reason is simple: I've been busy. ...
[from Eastman's Online Genealogy Newsletter]

Requirements

Because of the amount of bandwidth GEDCOM Crawler uses when downloading large family trees, I do not run it on my server on behalf of anyone but family members. GEDCOM Crawler can run on a computer that meets the following requirements:
  • Internet connection (preferably high-speed)
  • Perl
  • Command line access (ie. if you are connecting to the computer over a network rather than sitting at the terminal, you will need something like Telnet or SSH access)

License

By downloading GEDCOM Crawler, you agree to the terms of this license.
  • Subject to the terms of this license, you may download and use GEDCOM Crawler at no charge.
  • You may make modifications to your own copy.
  • You may NOT redistribute GEDCOM Crawler in whole or in part, nor modified copies or other scripts based on GEDCOM Crawler, whether commercially or for free without prior written consent. Refer people to this website instead.
  • This script is provided as-is with no warranty whatsoever. Use it at your own risk. If you live in a jurisdiction that does not allow complete disclaimer of warranty, then you may NOT download nor use GEDCOM Crawler.
The data from the Ancestral File is covered by its own copyright terms. See the FamilySearch.org website for more information.

Known Limitations

Support

Sorry, I'm way too busy--no technical support is available at this time.

Instructions

  1. Download and unzip GEDCOM-Crawler.zip (By downloading GEDCOM Crawler, you agree to the above license terms). NOTE: The files use UNIX style linebreaks. When opening the files on other operating systems, ...things may look weird. I may make additional packages with different linebreak styles later.
  2. Open gedcom-crawler-conf.pl in a text editor like Notepad, or better yet, a programmer's text editor.
  3. If the computer on which GEDCOM Crawler will run has a host name (for example, if you are going to run it on your webserver), enter the host name in $myHostName. Otherwise you will need to find out the IP address of the computer and enter it in $myHostIP. (If you enter the host name, you may leave $hostIP blank.)
  4. Save your changes (be sure to save as a plain text file).
  5. If you are going to run GEDCOM Crawler on a different computer than the one you have it on, upload all four files to that computer.
  6. If necessary, make all four files executable. For example, on a UNIX or Linux computer, you might type the following command on the command line from within the directory where the files are located: chmod 700 gedcom-crawler*
  7. Find the family ID for the most recent ancestor on the line whose data you wish to download:
    1. Go to the Ancestral File Search Form
    2. Find the ancestor
    3. Go to their individual record
    4. Copy the "Family" link on the right side of the page across from their name
    5. (don't go to that page and copy the URL in your browser's address bar--that won't be the right one--instead, right click the "Family" link and select "copy URL" or something like that from that menu)
    6. The number after "familyid=" in that URL is the family ID.
  8. From the directory where GEDCOM Crawler is installed, execute the following command, substituting in the desired family ID: ./gedcom-crawler.pl 123456 Here, the fun begins! GEDCOM Crawler will start downloading and combining GEDCOM files for your ancestors. Each time it downloads a file, it will tell you how many families' data it has processed, and how many family IDs it has found but not processed. Watch both numbers go up for a while as it finds more and more families! After the numbers, it will display one dot for each line in the GEDCOM file--some families are bigger than others.

    IMPORTANT: GEDCOM Crawler will create a file whose name is the family ID specified on the command line plus ".ged". If you already have a file by that name in the same directory as GEDCOM Crawler, it will be overwritten.
  9. If you wish to download other family lines without getting duplicates, run GEDCOM Crawler again with the new family ID, adding a space and an "x" to the command line like this: ./gedcom-crawler.pl 567890 x This will prevent GEDCOM Crawler from clearing the family IDs it has already downloaded from the database.
  10. If you have already cleared the family IDs from the database and wish to avoid duplicates, or already have a GEDCOM file containing Ancestral File family IDs which you don't wish to duplicate, run gedcom-crawler-skip-families.pl as follows to put the family IDs into the database (substituting in the name of your GEDCOM file): ./gedcom-crawler-skip-families.pl gedcom-file.ged
  11. As with gedcom-crawler.pl, you may tell gedcom-crawler-skip-families.pl not to empty the database before it runs by adding a space and an "x" to the command line: ./gedcom-crawler-skip-families.pl gedcom-file.ged x
  12. Send your GEDCOM files to your family members!