GEDCOM Crawler

Download your entire family tree

version 1.1
© 2006 Antone Roundy
All Rights Reserved
Introduction | Requirements | License | Limitations | Support | Instructions
More Information sources
I have not used these sites, so I can't vouch for them personally, but they are popular, so I imagine you might find them useful.

"197 GEDCOMs downloaded, 4832 to go...Ugh!"

Have you ever tried to download all the GEDCOMs for your ancestors to import into PAF? How many did you finish before you gave up? I'd finished all of maybe 2 or 3 and immediately realized that doing it manually would take forever. So I put off getting started for years...until I realized that I could write a script to download them all automatically.

GEDCOM Crawler generates a single GEDCOM file containing all of the information from the LDS Ancestral File for the ancestors of the indicated family. In about 45 minutes, it downloaded data on 5,525 families in my family tree, stretching all the way back to 124 A.D.!

NOTE: Because the data comes from the Ancestral File, it does not contain LDS ordinance data. Look for other tools to help you import this data into PAF.


Genealogy News

Newsfeed display by CaRP

Stagecoach Mary: the Black Cowgirl
9 Feb 2010 at 1:00am
America's Old West was undoubtedly a Wild West before an ex-slave named Mary Fields arrived in 1885 at a small railroad town in present-day Montana. Yet she certainly made things more interesting. ...
[from Eastman's Online Genealogy Newsletter]

Worth Tucker Diary – 8 Feb 1903
8 Feb 2010 at 10:18pm
ThinkGenealogy:-
[from EOGN Other News - powered by KickRSS]

TV: Genealogy shows we want to see
8 Feb 2010 at 10:18pm
Tracing the Tribe: The Jewish Genealogy Blog:- Our favorite creative geneablogger, Thomas MacEntee, has produced a list of genealogy shows that we'd all like to see.
[from EOGN Other News - powered by KickRSS]

JewishGen: ViewMate improvements
8 Feb 2010 at 10:18pm
Tracing the Tribe: The Jewish Genealogy Blog:- New social networking features will be added to JewishGen's ViewMate feature from February 20.
[from EOGN Other News - powered by KickRSS]

Boston: Jews of the American Revolution
8 Feb 2010 at 10:18pm
Tracing the Tribe: The Jewish Genealogy Blog:- In Concord, Massachusetts, Dr. Joseph L. Andrew, a retired physician and local historian, is descended from several Jewish colonists who battled the B...
[from EOGN Other News - powered by KickRSS]

Los Angeles: The Jewish Body, Feb. 15
8 Feb 2010 at 10:18pm
Tracing the Tribe: The Jewish Genealogy Blog:- Emory University professor anthropology Melvin Konner will explore traditional Jewish and non-Jewish views about the Jewish body at the next meeting o...
[from EOGN Other News - powered by KickRSS]

See You at the St. George Family History Expo
8 Feb 2010 at 10:18pm
The Ancestry Insider:- #FHExpo stands for this month?s St. George Family History Expo, to be held in? (don?t get ahead of me now) ?Timbuktu! Just kidding. It?s in St. George, Utah on 26th and 27th,...
[from EOGN Other News - powered by KickRSS]

Requirements

Because of the amount of bandwidth GEDCOM Crawler uses when downloading large family trees, I do not run it on my server on behalf of anyone but family members. GEDCOM Crawler can run on a computer that meets the following requirements:
  • Internet connection (preferably high-speed)
  • Perl
  • Command line access (ie. if you are connecting to the computer over a network rather than sitting at the terminal, you will need something like Telnet or SSH access)

License

By downloading GEDCOM Crawler, you agree to the terms of this license.
  • Subject to the terms of this license, you may download and use GEDCOM Crawler at no charge.
  • You may make modifications to your own copy.
  • You may NOT redistribute GEDCOM Crawler in whole or in part, nor modified copies or other scripts based on GEDCOM Crawler, whether commercially or for free without prior written consent. Refer people to this website instead.
  • This script is provided as-is with no warranty whatsoever. Use it at your own risk. If you live in a jurisdiction that does not allow complete disclaimer of warranty, then you may NOT download nor use GEDCOM Crawler.
The data from the Ancestral File is covered by its own copyright terms. See the FamilySearch.org website for more information.

Known Limitations

Support

Sorry, I'm way too busy--no technical support is available at this time.

Instructions

  1. Download and unzip GEDCOM-Crawler.zip (By downloading GEDCOM Crawler, you agree to the above license terms). NOTE: The files use UNIX style linebreaks. When opening the files on other operating systems, ...things may look weird. I may make additional packages with different linebreak styles later.
  2. Open gedcom-crawler-conf.pl in a text editor like Notepad, or better yet, a programmer's text editor.
  3. If the computer on which GEDCOM Crawler will run has a host name (for example, if you are going to run it on your webserver), enter the host name in $myHostName. Otherwise you will need to find out the IP address of the computer and enter it in $myHostIP. (If you enter the host name, you may leave $hostIP blank.)
  4. Save your changes (be sure to save as a plain text file).
  5. If you are going to run GEDCOM Crawler on a different computer than the one you have it on, upload all four files to that computer.
  6. If necessary, make all four files executable. For example, on a UNIX or Linux computer, you might type the following command on the command line from within the directory where the files are located: chmod 700 gedcom-crawler*
  7. Find the family ID for the most recent ancestor on the line whose data you wish to download:
    1. Go to the Ancestral File Search Form
    2. Find the ancestor
    3. Go to their individual record
    4. Copy the "Family" link on the right side of the page across from their name
    5. (don't go to that page and copy the URL in your browser's address bar--that won't be the right one--instead, right click the "Family" link and select "copy URL" or something like that from that menu)
    6. The number after "familyid=" in that URL is the family ID.
  8. From the directory where GEDCOM Crawler is installed, execute the following command, substituting in the desired family ID: ./gedcom-crawler.pl 123456 Here, the fun begins! GEDCOM Crawler will start downloading and combining GEDCOM files for your ancestors. Each time it downloads a file, it will tell you how many families' data it has processed, and how many family IDs it has found but not processed. Watch both numbers go up for a while as it finds more and more families! After the numbers, it will display one dot for each line in the GEDCOM file--some families are bigger than others.

    IMPORTANT: GEDCOM Crawler will create a file whose name is the family ID specified on the command line plus ".ged". If you already have a file by that name in the same directory as GEDCOM Crawler, it will be overwritten.
  9. If you wish to download other family lines without getting duplicates, run GEDCOM Crawler again with the new family ID, adding a space and an "x" to the command line like this: ./gedcom-crawler.pl 567890 x This will prevent GEDCOM Crawler from clearing the family IDs it has already downloaded from the database.
  10. If you have already cleared the family IDs from the database and wish to avoid duplicates, or already have a GEDCOM file containing Ancestral File family IDs which you don't wish to duplicate, run gedcom-crawler-skip-families.pl as follows to put the family IDs into the database (substituting in the name of your GEDCOM file): ./gedcom-crawler-skip-families.pl gedcom-file.ged
  11. As with gedcom-crawler.pl, you may tell gedcom-crawler-skip-families.pl not to empty the database before it runs by adding a space and an "x" to the command line: ./gedcom-crawler-skip-families.pl gedcom-file.ged x
  12. Send your GEDCOM files to your family members!