ArrowFs¶
Description:
- Work with data in pyarrow. Arrow manages data in arrays, which can be grouped in tables, to represent columns of data in tabular data.
Resources:
Environment variables¶
This notebook reads its settings from environment variables. Set them before starting Jupyter:
S3_BUCKET: name of an S3 bucket you can access
Only Example 3 needs it; Examples 1 and 2 use the local filesystem.
Example 1¶
Write data to local filesystem
In [ ]:
Copied!
from pyarrow import fs
from pathlib import Path
tmp_path = "/tmp/ganymede_demo"
p = Path(tmp_path).mkdir(exist_ok=True)
tmp_fp = f"{tmp_path}/local_fs.dat"
# Instantiate local fs
local = fs.LocalFileSystem()
# Write data to the file as output stream
with local.open_output_stream(tmp_fp) as stream:
stream.write(b'data')
# Alternatively, write data to the file as input stream
with local.open_input_stream(tmp_fp) as stream:
print(stream.readall())
from pyarrow import fs
from pathlib import Path
tmp_path = "/tmp/ganymede_demo"
p = Path(tmp_path).mkdir(exist_ok=True)
tmp_fp = f"{tmp_path}/local_fs.dat"
# Instantiate local fs
local = fs.LocalFileSystem()
# Write data to the file as output stream
with local.open_output_stream(tmp_fp) as stream:
stream.write(b'data')
# Alternatively, write data to the file as input stream
with local.open_input_stream(tmp_fp) as stream:
print(stream.readall())
Example 2¶
Instantiate ArrowFSWrapper for a local filesystem and list the contents of a folder
In [ ]:
Copied!
from fsspec.implementations.arrow import ArrowFSWrapper
from pyarrow import fs
fs = fs.LocalFileSystem()
arrow_fs = ArrowFSWrapper(fs)
arrow_fs.ls("/tmp/ganymede_demo")
from fsspec.implementations.arrow import ArrowFSWrapper
from pyarrow import fs
fs = fs.LocalFileSystem()
arrow_fs = ArrowFSWrapper(fs)
arrow_fs.ls("/tmp/ganymede_demo")
Example 3¶
Create a filesystem object from a url
In [ ]:
Copied!
import os
from pyarrow import fs
s3, path = fs.FileSystem.from_uri(f"s3://{os.environ['S3_BUCKET']}")
print(s3)
print(path)
import os
from pyarrow import fs
s3, path = fs.FileSystem.from_uri(f"s3://{os.environ['S3_BUCKET']}")
print(s3)
print(path)