Overview
You may create your own web archive collections and generate ARCH datasets by collecting data with the web archiving software service, Archive-It.
|
âšī¸ Archive-It is a subscription service available to any ARCH user. Learn more about service options and get started with your own account: Archive-It Products & Services. |
Use the guide below to create your first web archive collection and utilize Archive-It's User Guide and FAQs to learn much more about tools and features.
In this guide:
- Archive-It overview
- Accessing your Archive-It account
- Creating a collection
- Adding websites and pages to your collection
- Running a web crawl
- Reviewing your crawl results
- Analyzing your collection with ARCH
Archive-It overview
Watch the video below for a <10-minute tour of the Archive-It software suite. This video can help you to get acquainted with your Archive-It account, point out where features are located, and demonstrate why you might want to use them.
(Note that this video refers to the legacy "ARS" suite of collection data features that other Archive-It partners may still use while you have access to the full and superseding ARCH platform).
Accessing your Archive-It account
Follow the directions in your invitation email from Archive-It to create your Archive-It account's password. You may then access your Archive-It account anytime here: https://partner.archive-it.org/login
More resources when you need them:
đ How to set up and administer your account
â Forgot password?
Creating a collection
Once logged in, you can create a new collection by clicking on the Create a Collection button and assigning your new collection a name:
You may create as many collections and add as many sites and pages to each as you wish. The only limitation for ARCH users is that you will be capped at 256 GB of total data archived among your collections during the pilot program period. This data can be tracked in real-time using the subscription graphic on the left-hand side of the home screen above. (Follow the instructions below for conducting test crawls in order to make the most efficient use of your data budget).
More resources when you need them:
đ How to create and manage a collection
đ How to monitor your data budget
Adding websites and pages to your collection
Click the Add Seeds button on your new collection's "Overview" page to add your first websites, feeds, and/or pages:
Each "seed" in your collection directs Archive-It's automated web crawling technology to a URL on the live web, where it can find the material that you want to collect and archive. This may be the home page of a website, a user's social media profile, a single news article or Wikipedia entry, or more.
Each seed can bear its own or share settings like public/private accessibility and recurring crawl schedule. Type or paste one seed URL per line in the "Add Seeds" dialog and select the following from the drop-down menus:
- Access: Whether the seeds' archives should be visible to the general public or only to you.
- Frequency: When any seeds should be archived automatically.
- Seed type: Select the "Standard" seed type for most websites with multiple pages or directories. Select the "One Page" option for individual web pages.
Setting your "scope"
Because web crawlers could potentially follow links forever if left to their own devices, you can apply some rules to limit how much they collect from each seed in your collection.
After applying one of the seed types above, you can tell the crawler more specifically what is "in scope" with how you format your seed URL. The crawler will read the URL string and scope-in anything that includes that same full string and additional location information to the right, for instance:
- https://www.mla.org/ â Archive all pages throughout MLA's entire website.
- https://www.mla.org/MLA-Style/ â Archive only the pages in the MLA Style Handbook section of the website, not pages in the "About Us," "Convention," or other sections.
- https://www.mla.org/content/download/190800/file/Prizewinners%202022.pdf â Archive only this PDF file about 2022 publication prize winners.
When you need to, you can also apply more specific scoping rules to seeds or to your entire collection. Click on the "Collection Scope" tab or on any specific seed, followed by the "Seed Scope" tab, to select rule options from the drop-down menu:
For instance, the rule above enables the crawler to archive the entire MLA website except for the MLA Style Handbook section. You may also expand the scope to include more URLs, impose a data limit, or ignore a robots.txt directive.
More resources when you need them:
đ How to select your "seed" URLs and seed types
đ How Archive-It crawlers determine scope
đē Pre-crawl scoping
đ Scoping guidance for specific types of sites
Running a web crawl
When you are ready to collect and archive them, select one or more of the seeds in your list and click the Run Crawl button. Configure your crawl job in the dialog box and click Crawl:
If this is the first time collecting a seed, we recommend selecting the option above to run a "Test Crawl." This option enables you to review the results of your crawl and either save or discard them before using any of your account's data budget.
Other best practices for first-time crawl configurations:
- Select ⤠10 seeds at a time if possible, to maximize efficiency and ease post-crawl review.
- Leave the data and document limits blank in order to run your crawl for as long as needed.
- Select the time limit of 7 days (the crawl will end and you will be notified if it collects everything in its scope before this limit).
Once launched, you can monitor the progress of your crawl under the "Crawls" > "Current Crawls" tabs. You will receive an email notification when the crawling process completes, including a link to a full report on everything collected. Follow the instructions below to review the results and determine if any changes need be made to collect your seed/s more successfully.
If your crawl is a "One-Time" or recurring crawl (rather than a test crawl), then your data is stored at this point and can be analyzed with ARCH.
If and when you wish to collect and archive your seeds on a recurring basis, navigate to the "Crawls" > "Scheduled Crawls" tabs and click the "Schedule Crawl" button. You may start crawling immediately or select a specific date and time to begin the recurring process:
More resources when you need them:
đē How and why to run a test crawl
đ How to schedule recurring crawls
Reviewing your crawl results
Once a crawl job completes, you can review it for completeness. This is recommended especially for test crawls and first permanent crawls before any scheduled recurring crawls commence. Follow the directions below to glean the most important information from your crawls' reports, identify any issues, and make interventions where they are helpful.
Reading crawl reports
Follow the link in your email notification or click on the unique ID number under the "Crawls" tab in order to view any crawl job's complete report. There is much more information contained in these reports than you should need at any one time, so you can follow these steps to ensure that the crawl ran properly:
- Review the information in your crawl's "Overview" tab in order to make sure that the crawl completed successfully (see "Status") and that the volume of material archived meets your expectations. Errant or obstructed crawls may require that you modify the scope of your collection in order to more accurately target the crawler more accurately.
If the crawl status indicates that the crawl ended due to a time, document, or data limit, then you can extend your crawl or run a longer test crawl.
Archive-It de-duplicates data as it crawls, so you can expect to see a difference between the "total" and "new" data collected each time, especially as crawls recur. - Review the information in your crawl's Seeds report, particularly each "Seed Status" in order to make sure that all of your seeds were crawled successfully, or if alternatively any robots.txt exclusions or other errors prevented seeds from being crawled.
- Review the information in your crawl's Hosts report in order to determine if any valuable hosts were blocked from crawling by robots.txt exclusions or deemed as outside the scope of your crawl. "Queued" documents that do not point to a crawler trap may indicate that the time, document, or data limit on the crawl should be extended in order to capture missing elements.
Saving or deleting test crawls
You will have 60 days after the completion of any test crawl to decide whether to save its contents permanently or delete them. After 60 days, Archive-It deletes them automatically. Make your selection with the banner options at the top of your test crawl's "Overview" tab:
Like the One-Time and scheduled crawls that are stored in your account automatically, test crawls can take up to 24 hours to process completely and then replay again in Wayback mode after you save them. Once stored completely, these crawl data can also be analyzed with ARCH.
More resources when you need them:
đē Getting the most from your post crawl reports
đ What should I check first in my post crawl reports?
đē Understanding your Hosts Report
đ Modify scope and run patch crawls from your report
Analyzing your collection with ARCH
Once in your Archive-It account, you may explore, analyze, and extract the new web archive collection data with ARCH.
More resources when you need them:
đ ARCH Help Center
Comments
Please sign in to leave a comment.