ia

Internet Archive CLI Skill

This skill enables interaction with the Internet Archive (archive.org) using the ia command-line tool from the internetarchive Python package.

Items

An item is the fundamental unit on archive.org - a logical grouping of related files sharing common metadata. An item can be a book, a song, an album, a dataset, a movie, an image or set of images, etc. Each item has a unique identifier across the entire archive.

Every item contains:

• Original uploaded files
• Derivative files (automatically generated by archive.org)
• <identifier>_meta.xml - item-level metadata
• <identifier>_files.xml - file-level metadata

Items must belong to a collection.

Item Limits

Constraint	Recommended	Hard Limit
Item total size	Under 100GB	~1TB
Files per item	Under 10,000	250,000 (performance degrades >10,000)
Single file size	Under 50GB	500-700GB
Daily upload	Under 1,000 files	5,000 files (zips count as 1)

Permanent URL patterns:

• Details page: https://archive.org/details/<identifier>
• Download directory: https://archive.org/download/<identifier>
• Specific file: https://archive.org/download/<identifier>/<filename>
• Item history: https://archive.org/history/<identifier>

Warning: Never link to server-specific URLs like ia802304.us.archive.org - these break when items migrate between servers. Always use the canonical archive.org URLs above.

For more details, see: https://archive.org/developers/items.html

Derivatives

When you upload files to the Internet Archive, the system automatically generates derivative files - converted versions in different formats and resolutions. For example:

• Video: Transcoded to h.264, Ogg, and various bitrates
• Audio: Converted to MP3 (multiple bitrates), Ogg Vorbis, FLAC
• Text/Books: OCR processing, searchable PDFs, EPUB, DjVu
• Images: Thumbnails, JPEG 2000, different resolutions

Derivatives make content accessible across different devices and bandwidths. You can identify derivatives in ia list output - they have an original field pointing to their source file.

To skip derivative generation during upload, use --no-derive:

ia upload my-item file.mp4 --metadata="mediatype:movies" --no-derive

For the complete list of source formats and their generated derivatives, see: https://archive.org/help/derivatives.php

Metadata Schema

Internet Archive items use XML-based metadata. Key points:

• Required fields: identifier, mediatype
• Recommended fields: title, description, creator, date, subject, collection, language
• Repeatable fields: collections, creators, subjects, languages support multiple values
• Custom fields: You can define unlimited custom metadata fields (must follow XML naming rules)

Identifier requirements:

• ASCII alphanumeric, underscores, dashes, or periods only
• Must begin with alphanumeric character
• 5-100 characters (5-80 recommended)
• Unique and unchangeable once set

For the complete metadata schema reference, see: https://archive.org/developers/metadata-schema

Collections

Collections group related items together. Key points:

• Only IA staff can create collections - users must request creation
• Minimum 50 items required for a new collection
• Items must be related and typically same media type
• Collection creation takes up to two weeks after request

To request a collection, contact Internet Archive with:

• List of item identifiers or search query identifying items
• Desired collection identifier (5-80 chars, alphanumeric only)
• Collection title and description
• At least one subject tag

Public upload collections (anyone can upload to):

• opensource_movies, opensource_audio, opensource_media - general media
• community_texts, community_video, community_audio - community contributions

Other collections restrict uploads to designated uploaders only.

Tool Detection and Installation

Before using any ia commands, check if the tool is installed:

ia --version

If the ia command is not found, install it using uv:

uv tool install internetarchive

Alternative installation methods:

• pipx install internetarchive
• pip install internetarchive

After installation, verify it works with ia --version.

Global Options

These options work with all ia commands:

Option	Description
-h, --help	Show help message
-v, --version	Display version
-c FILE, --config-file	Path to config file
-l, --log	Enable logging
-d, --debug	Enable debug output

Configuration and Authentication

Check if ia is configured:

ia configure --whoami

If not configured (shows error or empty), the user needs to set up credentials:

1. Interactive setup: Run ia configure and follow prompts
2. Get credentials: IA-S3 keys from https://archive.org/account/s3.php
3. Config location: Saves to ~/.config/ia.ini

Configure Options

Option	Description
--whoami	Print current authenticated user
--show	Print current config as JSON
--check	Validate IA-S3 keys (exit 0 if valid, 1 otherwise)

# Show current config
ia configure --show

# Validate keys (useful in scripts)
ia configure --check && echo "Keys valid"

Environment Variables

Alternative to config file:

export IA_ACCESS_KEY_ID="your-access-key"
export IA_SECRET_ACCESS_KEY="your-secret-key"

Note: Configuration is required for uploads and metadata modifications. Searching and downloading public items works without authentication.

User-Agent Identification (Required)

All requests to the Internet Archive must include a proper User-Agent string that clearly identifies the source of the request. This applies to every request made via any tool - the ia CLI, Python library, direct API calls, curl, or any other HTTP client. This is critical for AI agents, bots, and automated tools.

The ia CLI automatically includes a default User-Agent with your access key:

internetarchive/5.7.2 (Linux x86_64; N; en; ACCESS_KEY) Python/3.11.0

When using Claude Code or other AI/LLM agents, you must append a custom suffix that includes:

• The tool/agent name and version (e.g., "Claude Code/1.0.0")
• The model being used if applicable (e.g., "claude-sonnet-4-20250514")
• Any relevant context about the automation

The --user-agent-suffix CLI option and user_agent_suffix config setting require internetarchive version 5.7.2 or newer. The default User-Agent (including access key) is always sent - your suffix is appended to it.

CLI:

ia --user-agent-suffix "Claude Code/1.0.0 (claude-sonnet-4-20250514)" download my-item

INI file (~/.config/internetarchive/ia.ini):

[general]
user_agent_suffix = Claude Code/1.0.0 (claude-sonnet-4-20250514)

Python API:

from internetarchive import get_session

session = get_session(config={
    'general': {'user_agent_suffix': 'Claude Code/1.0.0 (claude-sonnet-4-20250514)'}
})

The resulting User-Agent will look like:

internetarchive/5.7.2 (Linux x86_64; N; en; ACCESS_KEY) Python/3.11.0 Claude Code/1.0.0 (claude-sonnet-4-20250514)

This helps the Internet Archive track usage patterns, troubleshoot issues, and maintain service quality. Always be specific - include version numbers, model identifiers, and enough detail to distinguish your tool from others.

Search Operations

Search the Internet Archive catalog:

ia search '<query>'

Search Parameters

Parameter	Description
--itemlist	Output identifiers only, one per line
-n, --num-found	Print only the count of results
-s, --sort	Sort results: --sort='field desc' or --sort='field asc'
-f, --field	Return specific metadata fields (repeatable)
-F, --fts	Full-text search (search within text content, not just metadata)
--parameters	Raw query parameters: --parameters="page=N&rows=N"

# Get result count only
ia search 'collection:nasa' -n

# Sort by date descending
ia search 'mediatype:texts' --sort='date desc'

# Return specific fields
ia search 'collection:nasa' --field=identifier --field=title

Sort Fields

Common sort fields for use with --sort:

Field	Description
date	Content date
publicdate	When item was published to archive.org
addeddate	When added to archive
updatedate	Last updated
title / titleSorter	Alphabetical by title
creator / creatorSorter	Alphabetical by creator
downloads	Total downloads
week	Downloads this week
month	Downloads this month
num_reviews	Number of reviews
num_favorites	Number of favorites
item_size	Total item size
files_count	Number of files

Use asc or desc suffix:

ia search 'mediatype:audio' --sort='downloads desc'
ia search 'collection:books' --sort='publicdate asc'
ia search 'creator:NASA' --sort='title asc'

Search Query Syntax

The Internet Archive uses Apache Lucene query syntax. By default, the operator is AND (all terms must be present).

Query Operators

Operator	Description
AND	All terms must be present (default)
OR	Any of the terms can be present
NOT	Exclude documents with term (requires at least one positive term)
( )	Group clauses to form subqueries

Field-Specific Searches

Use field:value syntax to search specific metadata fields:

Query	Description
'title:"search text"'	By title
'creator:"Author Name"'	By creator/author
'subject:"topic"'	Search by subject
'description:"text"'	By description
'collection:name'	Items in a collection
'mediatype:texts'	By media type (texts, movies, audio, software, image, data)
'contributor:smithsonian'	By contributor
'language:eng'	By language code
'format:pdf'	Items containing specific file format
'isbn:9780123456789'	By ISBN
'licenseurl:http*by-nc*'	By Creative Commons license

Range Queries

Search values between bounds using brackets or parentheses:

Syntax	Description
[1000 TO 2000]	Inclusive range (includes bounds)
{1000 TO 2000}	Exclusive range (excludes bounds)
[1000 TO null]	Open-ended range (1000 or greater)
[null TO 2000]	Open-ended range (2000 or less)

Date Fields

Searchable date fields: addeddate, createdate, date, indexdate, publicdate, reviewdate, updatedate, oai_updatedate

Query	Description
'date:[2020-01-01 TO 2024-12-31]'	Date range
'publicdate:[2024-01-01 TO 2024-06-30]'	By publication date
'indexdate:[2024-01-01T00:00:00Z TO 2024-12-31T23:59:59Z]'	With timestamp
'date:2024*'	Wildcard for year (non-range)

Fuzzy Queries

Append ~ for approximate spelling matches:

ia search 'title:buttonwood~'

# Boost fuzzy matches with weights
ia search '(title:buttonwood~)^150 OR (subject:buttonwood~)^100'

Searching for Missing Fields

Find items where a field doesn't exist:

ia search 'collection:microfiche AND NOT _exists_:creator'

Searching by Uploader

Search by uploader's user item, screen name, or email:

ia search '_uploader_useritem:@username'
ia search '_uploader_screenname:"Display Name"'
ia search 'uploader:[email protected]'

Additional Searchable Fields

Beyond standard metadata, you can search by:

• downloads - download count
• item_size - total item size in bytes
• files_count - number of files
• collection_size - size of collection
• item_count - items in collection

ia search 'collection:opensource AND downloads:[1000 TO null]'
ia search 'mediatype:movies AND item_size:[1000000000 TO null]'

Combined Queries

# AND is implicit between terms
ia search 'collection:nasa mediatype:image'

# Explicit operators
ia search 'collection:nasa AND mediatype:image'
ia search 'mediatype:texts OR mediatype:audio'
ia search 'collection:opensource NOT mediatype:software'

# Grouped subqueries
ia search '(mediatype:texts OR mediatype:audio) AND creator:"Mark Twain"'

Full-Text Search

Use the -F (or --fts) flag to search within the actual text content of items rather than just metadata. This is particularly powerful for searching text collections like books, documents, and OCR'd materials.

Basic full-text search:

ia search -F 'collection:collection_name "search phrase"'

How it works:

• Searches inside the full text of documents (OCR'd PDFs, text files, etc.)
• More powerful than metadata-only search for finding specific quotes or passages
• Requires items to have searchable text (OCR or text files)
• Can be combined with collection and metadata filters

Full-text search syntax:

• Use quotes for exact phrases: "complete phrase"
• Combine with metadata filters: collection:name AND "text to find"
• Works best with text collections that have been OCR'd

Examples

# Search NASA images
ia search 'collection:nasa mediatype:image' --parameters="rows=10"

# Search public domain books
ia search 'subject:"public domain" mediatype:texts'

# Get just identifiers
ia search 'creator:"Mark Twain"' --itemlist

# Full-text search within a text collection
ia search -F 'collection:books "climate change"'

# Full-text search for a specific quote in public domain texts
ia search -F '"to be or not to be" mediatype:texts'

# Full-text search with collection filter and pagination
ia search -F 'collection:usgovernmentdocuments "artificial intelligence"' --parameters="rows=20"

Download Operations

Download files from an Internet Archive item:

ia download <identifier>

Download Parameters

Parameter	Description
--glob="*.ext"	Download only matching files (use | for multiple: '*.mp4|*.webm')
--exclude="*pattern*"	Exclude files matching pattern
--format="FORMAT"	Download specific derivative format
--source=SOURCE	Filter by source: original, derivative, metadata
--exclude-source=SOURCE	Exclude by source type
--destdir=path	Download to specific directory
--no-directories	Flatten directory structure
-s, --stdout	Write file to stdout (for piping)
--dry-run	Show what would be downloaded
--checksum	Skip files that already exist with correct checksum
--on-the-fly	Download on-the-fly files (generated derivatives)
--search="QUERY"	Download from search results
--itemlist=FILE	Download items listed in file

Filtering by Source Type

Use --source and --exclude-source to filter by file origin:

# Download only original files (skip all derivatives)
ia download my-item --source=original

# Download originals and metadata, skip derivatives
ia download my-item --exclude-source=derivative

# Download only metadata files
ia download my-item --source=metadata

Examples

# Download all files from an item
ia download TripDown1905

# Download specific files by name
ia download TripDown1905 file1.mp4 file2.ogv

# Download only MP4 files
ia download TripDown1905 --glob="*.mp4"

# Download MP4s but exclude low-quality versions
ia download TripDown1905 --glob="*.mp4" --exclude="*512kb*"

# Download specific format
ia download TripDown1905 --format='512Kb MPEG4'

# Download to specific directory
ia download TripDown1905 --destdir=./downloads

# Download from search results
ia download --search 'collection:opensource_movies' --glob="*.mp4"

# Download items from a list file
ia search 'collection:glasgowschoolofart' --itemlist > itemlist.txt
ia download --itemlist itemlist.txt

# Preview what will be downloaded
ia download my_item --dry-run

Upload Operations

Upload files to the Internet Archive (requires authentication):

ia upload <identifier> file1 file2 --metadata="mediatype:value"

Required Metadata

The mediatype field is required. Common values:

• texts - Books, documents, PDFs
• movies - Video files
• audio - Music, podcasts, sound
• software - Programs, games
• image - Photos, graphics
• data - Datasets, archives

Upload Parameters

Parameter	Description
--metadata="key:value"	Set metadata (repeatable)
--header="key:value"	Set HTTP header
--checksum	Skip files already uploaded
-v, --verify	Verify data wasn't corrupted after upload
--no-derive	Skip derivative processing
--retries=N	Number of retry attempts
--remote-name=NAME	Set remote filename (for stdin uploads)
--keep-directories	Preserve directory structure in remote filename
-o, --open-after-upload	Open item in browser after upload
--file-metadata=FILE	File-level metadata from JSONL file
--spreadsheet=FILE	Bulk upload from CSV spreadsheet

Common Metadata Fields

--metadata="title:My Document Title"
--metadata="creator:Author Name"
--metadata="description:A description of the content"
--metadata="subject:topic1;topic2"
--metadata="collection:community_texts"
--metadata="date:2024-01-15"
--metadata="language:eng"

Examples

# Upload a PDF document
ia upload my-document-2024 document.pdf \
  --metadata="mediatype:texts" \
  --metadata="title:My Document" \
  --metadata="creator:John Doe"

# Upload multiple files
ia upload my-archive file1.pdf file2.pdf file3.pdf \
  --metadata="mediatype:texts" \
  --metadata="title:Document Collection"

# Upload with checksum verification and retries
ia upload my-item large-file.zip \
  --metadata="mediatype:data" \
  --checksum \
  --retries=10

# Upload from stdin
cat data.gz | ia upload my-item - \
  --remote-name=data.gz \
  --metadata="mediatype:data"

# Bulk upload using spreadsheet
ia upload --spreadsheet=metadata.csv

Notes:

• Items receive data mediatype by default if not specified
• Mediatype can only be changed after upload with admin support
• Derivative generation takes seconds to days depending on file type and system load
• Items typically appear in search within minutes, but can take up to 24 hours

Test Collection

Upload to test_collection for validation - items are automatically removed after ~30 days:

ia upload my-test-item file.pdf \
  --metadata="mediatype:texts" \
  --metadata="collection:test_collection"

Identifier Guidelines

• Use lowercase letters, numbers, and hyphens
• No spaces or special characters
• Keep it descriptive but concise
• Check if identifier exists: ia metadata <identifier> --exists

Item Thumbnail Image

To set a custom thumbnail for an item, upload an image named <identifier>_itemimage.jpg:

ia upload my-item my-item_itemimage.jpg

Restricting Downloads

To make files streamable but not downloadable, add the item to the stream_only collection:

ia metadata <identifier> --append-list="collection:stream_only"

Metadata Operations

View and modify item metadata:

# View metadata (JSON output)
ia metadata <identifier>

# Extract specific field with jq
ia metadata <identifier> | jq '.metadata.date'

# List file formats contained in an item
ia metadata <identifier> --formats

# Modify metadata (set or replace)
ia metadata <identifier> --modify="title:New Title"
ia metadata <identifier> --modify="foo:bar" --modify="baz:value"

# Remove a metadata field
ia metadata <identifier> --modify="fieldname:REMOVE_TAG"

# Append value to existing field
ia metadata <identifier> --append="title:Subtitle Here"

# Append to list field (e.g., subjects)
ia metadata <identifier> --append-list="subject:new topic"

# Remove specific value from list field
ia metadata <identifier> --remove="subject:old topic"

# Modify file-level metadata
ia metadata <identifier> --target="files/foo.txt" --modify="title:My File"

# Bulk updates from spreadsheet
ia metadata --spreadsheet=metadata.csv

List Operations

List files in an Internet Archive item:

ia list <identifier>

Shows all files with details (name, size, format).

List Parameters

Parameter	Description
--columns=name,size	Specify columns to show
--glob="*.pdf"	Filter by pattern
-l, --location	Print full URLs for each file
-a, --all	List all available file information
-v, --verbose	Print column headers

# List with full URLs
ia list my-item --location

# List all file info with headers
ia list my-item --all --verbose

# List specific columns
ia list my-item --columns=name,size,format

Tasks and Jobs

Check status of catalog tasks (uploads, derives, etc.):

# Check tasks for a specific item
ia tasks <identifier>

# Check all your tasks
ia tasks

Darking and Undarking Items

To make an item dark (hidden from public access) or undark it:

# Dark an item (requires comment)
ia tasks <identifier> --cmd=make_dark.php --comment="Reason for darking"

# Undark an item
ia tasks <identifier> --cmd=make_undark.php --comment="Reason for undarking"

Bulk Operations with GNU Parallel

For batch processing many items, use GNU Parallel to run ia commands concurrently.

Installation

# macOS
brew install parallel

# Debian/Ubuntu
apt install parallel

Basic Usage

Pipe item identifiers to parallel, using {} as placeholder:

# Fetch metadata for many items
cat itemlist.txt | parallel 'ia metadata {}'

# Download multiple items
cat itemlist.txt | parallel 'ia download {}'

Careful Batch Processing

For reliable bulk operations, use job logging to track progress and handle failures:

# Step 1: Create item list
ia search 'collection:myproject' --itemlist > itemlist.txt

# Step 2: Run with job logging
cat itemlist.txt | parallel --joblog job.log 'ia download {}'

# Step 3: Check for failures
echo $?  # 0 = all succeeded

# Step 4: Retry only failed jobs
parallel --retry-failed --joblog job.log

Job Log Benefits

The --joblog file tracks each command's exit status, allowing you to:

• Resume interrupted batch jobs
• Retry only failed items without re-processing successes
• Audit what succeeded and failed

Dry Run First

Always preview before bulk execution:

cat itemlist.txt | parallel --dry-run 'ia download {}'

Rate Limiting

Control concurrency to avoid overwhelming the server:

# Limit to 4 concurrent jobs
cat itemlist.txt | parallel -j4 'ia download {}'

# Add delay between jobs
cat itemlist.txt | parallel --delay 1 'ia download {}'

See: https://archive.org/developers/internetarchive/parallel.html

Best Practices

1. Always configure before uploading - Run ia configure first
2. Use meaningful identifiers - Descriptive, lowercase, hyphenated
3. Include proper metadata - At minimum: mediatype, title, creator
4. Check before uploading - Verify identifier doesn't exist: ia metadata <id>
5. Use checksums - Add --checksum for large uploads to enable resume
6. Respect rate limits - Don't spam requests; add delays for bulk operations
7. Test with dry-run - Use --dry-run to preview operations
8. Use test_collection first - Validate uploads before committing to permanent collections
9. Zip large file sets - Bundle many small files into archives before uploading
10. Specify language - Set language metadata for proper OCR processing on texts

Error Handling

Error	Solution
"not configured"	Run ia configure or set environment variables
"identifier exists"	Choose a different identifier
"permission denied"	Check credentials at https://archive.org/account/s3.php
"network error"	Retry the operation; check internet connection
"item not found"	Verify the identifier spelling
"429 Too Many Requests"	Rate limited; wait and retry with Retry-After header value
Item not appearing in search	Usually appears within minutes; check ia tasks <identifier> for pending jobs
Derive task failed	Check filename characters, file format, language metadata

Quick Reference

# Search
ia search 'query'
ia search 'query' --itemlist

# Download
ia download <identifier>
ia download <identifier> --glob="*.pdf"

# Upload (requires auth)
ia upload <identifier> files --metadata="mediatype:texts"

# Metadata
ia metadata <identifier>
ia metadata <identifier> --modify="title:New Title"

# List files
ia list <identifier>

# Tasks
ia tasks <identifier>

# Config
ia configure
ia configure --whoami

# Install
uv tool install internetarchive

API Reference

For programmatic access beyond the CLI, see the full developer documentation: https://archive.org/developers

Core APIs

API	Description
Items	Understanding item structure and access
Metadata Schema	Complete metadata field reference
Metadata Read	Retrieve item metadata via API
Metadata Write	Modify item metadata via API
IAS3	S3-compatible API for uploads
Tasks	Task queue management

Additional APIs

API	Description
Changes	Track item modifications across the archive
Views	Access viewing and download statistics
Reviews	Manage item reviews
Simple Lists	Create item relationships and lists
OCR Service	Text recognition service
PDF Service	PDF generation and processing

Python Library

For Python integration: internetarchive library

TypeScript Library (Third-Party)

A community-maintained TypeScript port is available: internetarchive-ts (docs)

Note: This is a work in progress and not officially maintained by the Internet Archive.