Here is a brain dump:
Surf to https://archive.org/ and create an account.
Then:
pip install internetarchive
ia configure
this is what ia configure writes:
~/.config/internetarchive/ia.ini
[s3]
access = ...
secret = ...
[cookies]
logged-in-user = lindahl%40pbm.com; expires=Sat, 28-Aug-2027 19:39:52 GMT; Max-Age=31536000; path=/; domain=.archive.org
logged-in-sig = ...
[general]
screenname = Greg Lindahl
to read the above file
import configparser
config = configparser.ConfigParser()
config.read('example.ini')
config['s3']['access']
config['s3']['secret']
env var configuration if you want to pretend that IA is s3, without changing any source code
(but this is ugly, so keep on going)
export AWS_ACCESS_KEY_ID="your_access_key"
export AWS_SECRET_ACCESS_KEY="your_secret_key"
export AWS_S3_ENDPOINT_URL="https://archive.org"
export AWS_DEFAULT_REGION="us-east-1"
and if you want to do the same with ugly source code
import fsspec
IA_ACCESS_KEY = "your_access_key"
IA_SECRET_KEY = "your_secret_key"
BUCKET_NAME = "your_item_identifier"
FILE_PATH = "example_file.txt"
fs = fsspec.filesystem(
"s3",
key=IA_ACCESS_KEY,
secret=IA_SECRET_KEY,
endpoint_url="https://archive.org",
client_kwargs={"region_name": "us-east-1"},
)
with fs.open(f"s3://{BUCKET_NAME}/{FILE_PATH}", "rb") as f:
print(f.read().decode("utf-8"))
But wouldn't it be better if we could just say ia:// and it worked?
This is a fsspec helper that makes ia:// uris work
import os
from fsspec.implementations.s3fs import S3FileSystem
class InternetArchiveFileSystem(S3FileSystem):
protocol = "ia"
def __init__(self, ia_access_key=None, ia_secret_key=None, *args, **kwargs):
"""
Initializes the IA filesystem by injecting Internet Archive's S3 endpoint
and authenticating via provided keys or environment variables.
"""
kwargs["client_kwargs"] = kwargs.get("client_kwargs", {})
kwargs["client_kwargs"].setdefault("endpoint_url", "https://archive.org")
access_key = ia_access_key or os.environ.get("IA_ACCESS_KEY") # if None, will be anonymous
secret_key = ia_secret_key or os.environ.get("IA_SECRET_KEY") # ditto
if access_key and secret_key:
kwargs["key"] = access_key
kwargs["secret"] = secret_key
else:
# XXX read from ~/.config/internetarchive/ia.ini
pass
super().__init__(*args, **kwargs)
@classmethod
def _strip_protocol(cls, path):
"""
Removes 'ia://' from the path so the underlying S3 client
can parse the bucket and key properly.
"""
if path.startswith("ia://"):
return path[5:]
return super()._strip_protocol(path)
and in main(), register the helper
register_implementation("ia", InternetArchiveFileSystem, clobber=True)
public IA WARC to test with
https://archive.org/download/EOT24PRE-20240926175758-crawl808/EOT24PRE-20240926175758-00032.warc.gz # this already works
ia://EOT24PRE-20240926175758-crawl808/EOT24PRE-20240926175758-00032.warc.gz # this should work for both non-logged-in and logged-in users
once that is working in warc2zip, add a PR to fsspec
git clone git@github.com:fsspec/filesystem_spec.git
ls filesystem_spec/fsspec/implementations/
Here is a brain dump:
Surf to https://archive.org/ and create an account.
Then:
this is what ia configure writes:
to read the above file
env var configuration if you want to pretend that IA is s3, without changing any source code
(but this is ugly, so keep on going)
and if you want to do the same with ugly source code
But wouldn't it be better if we could just say ia:// and it worked?
This is a fsspec helper that makes ia:// uris work
and in main(), register the helper
public IA WARC to test with
https://archive.org/download/EOT24PRE-20240926175758-crawl808/EOT24PRE-20240926175758-00032.warc.gz # this already works
ia://EOT24PRE-20240926175758-crawl808/EOT24PRE-20240926175758-00032.warc.gz # this should work for both non-logged-in and logged-in users
once that is working in warc2zip, add a PR to fsspec