Environment variables¶
This notebook reads its settings from environment variables. Set them before starting Jupyter:
S3_BUCKET: name of an S3 bucket you can access
Before you run Examples 1-4, upload two archives of your own to that bucket: any 7z archive as ganymede-demo/archive.7z and any zip archive as ganymede-demo/example.zip. Example 5 downloads a public archive over HTTPS.
Example 1¶
List the contents of a remote 7z file on s3
In [ ]:
Copied!
import os
from fsspec import filesystem
# Open the 7z archive in your S3 bucket as a filesystem
libarchive_fs = filesystem("libarchive", fo=f"s3://{os.environ['S3_BUCKET']}/ganymede-demo/archive.7z")
# See what's in it: the path of every file in the archive
archive_files = libarchive_fs.find("")
archive_files
import os
from fsspec import filesystem
# Open the 7z archive in your S3 bucket as a filesystem
libarchive_fs = filesystem("libarchive", fo=f"s3://{os.environ['S3_BUCKET']}/ganymede-demo/archive.7z")
# See what's in it: the path of every file in the archive
archive_files = libarchive_fs.find("")
archive_files
Example 2¶
Download one file in a remote 7z archive on s3 to local
In [ ]:
Copied!
from pathlib import Path
tmp = Path("/tmp/ganymede_demo")
tmp.mkdir(parents=True, exist_ok=True)
# Download the first file in the archive from s3 to local
member = archive_files[0]
local_member = tmp / Path(member).name
libarchive_fs.get(rpath=member, lpath=str(local_member))
from pathlib import Path
tmp = Path("/tmp/ganymede_demo")
tmp.mkdir(parents=True, exist_ok=True)
# Download the first file in the archive from s3 to local
member = archive_files[0]
local_member = tmp / Path(member).name
libarchive_fs.get(rpath=member, lpath=str(local_member))
Example 3¶
See info about an object in the archive and open the local copy
In [ ]:
Copied!
# Get info about the object inside the archive
print(libarchive_fs.info(path=member))
# Open the local copy and read its first bytes
with open(local_member, "rb") as f:
print(f.read(64))
# Get info about the object inside the archive
print(libarchive_fs.info(path=member))
# Open the local copy and read its first bytes
with open(local_member, "rb") as f:
print(f.read(64))
Example 4¶
Download a remote zip file from S3 to local and see what's in it
In [ ]:
Copied!
import os
from fsspec import filesystem
from pathlib import Path
libarchive_fs_s3 = filesystem("libarchive", fo=f"s3://{os.environ['S3_BUCKET']}/ganymede-demo/example.zip")
print(libarchive_fs_s3.ls("/", detail=False))
Path("/tmp/ganymede_demo").mkdir(parents=True, exist_ok=True)
libarchive_fs_s3.get(
rpath="/",
lpath="/tmp/ganymede_demo/unarchived",
recursive=True
)
local_fs = filesystem("local")
local_fs.ls("/tmp/ganymede_demo/unarchived", detail=False)
import os
from fsspec import filesystem
from pathlib import Path
libarchive_fs_s3 = filesystem("libarchive", fo=f"s3://{os.environ['S3_BUCKET']}/ganymede-demo/example.zip")
print(libarchive_fs_s3.ls("/", detail=False))
Path("/tmp/ganymede_demo").mkdir(parents=True, exist_ok=True)
libarchive_fs_s3.get(
rpath="/",
lpath="/tmp/ganymede_demo/unarchived",
recursive=True
)
local_fs = filesystem("local")
local_fs.ls("/tmp/ganymede_demo/unarchived", detail=False)
Example 5¶
Download a remote sqlite database locally
In [ ]:
Copied!
from pathlib import Path
from fsspec import filesystem
# Create a new dir
Path("/tmp/ganymede_demo/sqlite").mkdir(parents=True, exist_ok=True)
# Get the zip archive from URL
libarchive_fs = filesystem("libarchive", fo="https://www.sqlitetutorial.net/wp-content/uploads/2018/03/chinook.zip")
# Download it locally
libarchive_fs.get(lpath="/tmp/ganymede_demo/sqlite/chinook.db", rpath="chinook.db")
local_fs = filesystem("local")
local_fs.ls("/tmp/ganymede_demo/sqlite/chinook.db", detail=False)
from pathlib import Path
from fsspec import filesystem
# Create a new dir
Path("/tmp/ganymede_demo/sqlite").mkdir(parents=True, exist_ok=True)
# Get the zip archive from URL
libarchive_fs = filesystem("libarchive", fo="https://www.sqlitetutorial.net/wp-content/uploads/2018/03/chinook.zip")
# Download it locally
libarchive_fs.get(lpath="/tmp/ganymede_demo/sqlite/chinook.db", rpath="chinook.db")
local_fs = filesystem("local")
local_fs.ls("/tmp/ganymede_demo/sqlite/chinook.db", detail=False)