Gecko Tribe Family
Gecko Tribe Family
twitter youtube linkedin google-plus

GEDCOM Crawler

Download your entire family tree

version 1.1
© 2006 Antone Roundy
All Rights Reserved
Introduction | Requirements | License | Limitations | Support | Instructions
More Information sources
I have not used these sites, so I can't vouch for them personally, but they are popular, so I imagine you might find them useful.

"197 GEDCOMs downloaded, 4832 to go...Ugh!"

Have you ever tried to download all the GEDCOMs for your ancestors to import into PAF? How many did you finish before you gave up? I'd finished all of maybe 2 or 3 and immediately realized that doing it manually would take forever. So I put off getting started for years...until I realized that I could write a script to download them all automatically.

GEDCOM Crawler generates a single GEDCOM file containing all of the information from the LDS Ancestral File for the ancestors of the indicated family. In about 45 minutes, it downloaded data on 5,525 families in my family tree, stretching all the way back to 124 A.D.!

NOTE: Because the data comes from the Ancestral File, it does not contain LDS ordinance data. Look for other tools to help you import this data into PAF.


Genealogy News

Newsfeed display by CaRP


[CaRP] Can't open remote newsfeed [401].

[CaRP] Failed to open file: /www/shared/carp/autocache/de80e3b4122038a187678f60e28aadbf

[CaRP] Can't open remote newsfeed [-1].

[CaRP] Can't open remote newsfeed [401].

[CaRP] Failed to open file: /www/shared/carp/autocache/0e9d5d4b9fc29507e0b50793bf20da0f

[CaRP] Can't open remote newsfeed [-1].
EOGN web site is back up
20 Apr 2014 at 4:45pm
This has been a tough week! I have traveled more than 12,000 miles and the eogn.com web site has been up and down like a yo-yo during much of that time. It seems to fail immediately after I board a...
[from Eastman's Online Genealogy Newsletter]

Want a Cheap Laptop? Add a Keyboard to an iPad or Android Tablet Computer
19 Apr 2014 at 1:04am
Many people own and love their tablet computers. I have an Pad Mini and it has become my primary traveling computer. I hear similar statements from owners of various Android tablets as well. As use...
[from Eastman's Online Genealogy Newsletter]

FamilySearch Seeks Feedback from Volunteer Indexers
18 Apr 2014 at 10:09pm
The FamilySearch method of creating indexes of old records has been described many times in this newsletter and many other genealogy publications. FamilySearch is now planning to introduce a new pr...
[from Eastman's Online Genealogy Newsletter]

Woman Claims She's the Virgin Mary's Cousin 65 Times Removed
18 Apr 2014 at 10:00pm
I won't offer any opinion about the accuracy of this article but I certainly would like to see the source citations. A Pennsylvania woman claims she is the 64th great-granddaughter of Saint Joseph ...
[from Eastman's Online Genealogy Newsletter]

2015 FGS National Conference Call for Presentations
17 Apr 2014 at 5:50am
The following announcement was written by the Federation of Genealogical Societies: 2015 FGS NATIONAL CONFERENCE CALL FOR PRESENTATIONS Submissions will be accepted between May 1, 2014 and May 31, ...
[from Eastman's Online Genealogy Newsletter]

92-Year-Old Man Writes His Own 3-Word Obituary
17 Apr 2014 at 3:38am
Several articles have been posted here about humorous obituaries, many of them written by the deceased in advance of the "final event." Now Stig Kernell, a Swedish resident, gave his funeral home i...
[from Eastman's Online Genealogy Newsletter]

This Web Site May Be Running Slowly
17 Apr 2014 at 3:35am
Some newsletter readers may find the www.EOGN.com web site is running slowly today. The hosting company is having networking problems in their Provo, Utah, data center, one of several data centers ...
[from Eastman's Online Genealogy Newsletter]

Requirements

Because of the amount of bandwidth GEDCOM Crawler uses when downloading large family trees, I do not run it on my server on behalf of anyone but family members. GEDCOM Crawler can run on a computer that meets the following requirements:
  • Internet connection (preferably high-speed)
  • Perl
  • Command line access (ie. if you are connecting to the computer over a network rather than sitting at the terminal, you will need something like Telnet or SSH access)

License

By downloading GEDCOM Crawler, you agree to the terms of this license.
  • Subject to the terms of this license, you may download and use GEDCOM Crawler at no charge.
  • You may make modifications to your own copy.
  • You may NOT redistribute GEDCOM Crawler in whole or in part, nor modified copies or other scripts based on GEDCOM Crawler, whether commercially or for free without prior written consent. Refer people to this website instead.
  • This script is provided as-is with no warranty whatsoever. Use it at your own risk. If you live in a jurisdiction that does not allow complete disclaimer of warranty, then you may NOT download nor use GEDCOM Crawler.
The data from the Ancestral File is covered by its own copyright terms. See the FamilySearch.org website for more information.

Known Limitations

Support

Sorry, I'm way too busy--no technical support is available at this time.

Instructions

  1. Download and unzip GEDCOM-Crawler.zip (By downloading GEDCOM Crawler, you agree to the above license terms). NOTE: The files use UNIX style linebreaks. When opening the files on other operating systems, ...things may look weird. I may make additional packages with different linebreak styles later.
  2. Open gedcom-crawler-conf.pl in a text editor like Notepad, or better yet, a programmer's text editor.
  3. If the computer on which GEDCOM Crawler will run has a host name (for example, if you are going to run it on your webserver), enter the host name in $myHostName. Otherwise you will need to find out the IP address of the computer and enter it in $myHostIP. (If you enter the host name, you may leave $hostIP blank.)
  4. Save your changes (be sure to save as a plain text file).
  5. If you are going to run GEDCOM Crawler on a different computer than the one you have it on, upload all four files to that computer.
  6. If necessary, make all four files executable. For example, on a UNIX or Linux computer, you might type the following command on the command line from within the directory where the files are located: chmod 700 gedcom-crawler*
  7. Find the family ID for the most recent ancestor on the line whose data you wish to download:
    1. Go to the Ancestral File Search Form
    2. Find the ancestor
    3. Go to their individual record
    4. Copy the "Family" link on the right side of the page across from their name
    5. (don't go to that page and copy the URL in your browser's address bar--that won't be the right one--instead, right click the "Family" link and select "copy URL" or something like that from that menu)
    6. The number after "familyid=" in that URL is the family ID.
  8. From the directory where GEDCOM Crawler is installed, execute the following command, substituting in the desired family ID: ./gedcom-crawler.pl 123456 Here, the fun begins! GEDCOM Crawler will start downloading and combining GEDCOM files for your ancestors. Each time it downloads a file, it will tell you how many families' data it has processed, and how many family IDs it has found but not processed. Watch both numbers go up for a while as it finds more and more families! After the numbers, it will display one dot for each line in the GEDCOM file--some families are bigger than others.

    IMPORTANT: GEDCOM Crawler will create a file whose name is the family ID specified on the command line plus ".ged". If you already have a file by that name in the same directory as GEDCOM Crawler, it will be overwritten.
  9. If you wish to download other family lines without getting duplicates, run GEDCOM Crawler again with the new family ID, adding a space and an "x" to the command line like this: ./gedcom-crawler.pl 567890 x This will prevent GEDCOM Crawler from clearing the family IDs it has already downloaded from the database.
  10. If you have already cleared the family IDs from the database and wish to avoid duplicates, or already have a GEDCOM file containing Ancestral File family IDs which you don't wish to duplicate, run gedcom-crawler-skip-families.pl as follows to put the family IDs into the database (substituting in the name of your GEDCOM file): ./gedcom-crawler-skip-families.pl gedcom-file.ged
  11. As with gedcom-crawler.pl, you may tell gedcom-crawler-skip-families.pl not to empty the database before it runs by adding a space and an "x" to the command line: ./gedcom-crawler-skip-families.pl gedcom-file.ged x
  12. Send your GEDCOM files to your family members!