Environment variables¶
pyarrow talks to HDFS through Hadoop's libhdfs, which runs on Java. Set these before starting Jupyter:
JAVA_HOME: path of your Java installationHADOOP_HOME: path of your Hadoop installationCLASSPATH: the Hadoop jars, for exampleexport CLASSPATH="$("$HADOOP_HOME/bin/hdfs" classpath --glob)"ARROW_LIBHDFS_DIR: optional, the directory that holdslibhdfs.soif it is not in$HADOOP_HOME/lib/nativeHDFS_HOST: hostname of the HDFS NameNodeHDFS_PORT: optional, defaults to8020HDFS_USER: optional, the HDFS user name (defaults to your login user)HDFS_KERBEROS_TICKET_CACHE: used by Example 2: path of your Kerberos ticket cacheHDFS_FOLDER: optional, the folder Example 2 lists (defaults to/tmp)
Example 1¶
List contents of the filesystem
In [ ]:
Copied!
import os
from pyarrow import fs
host = os.environ["HDFS_HOST"]
port = os.environ.get("HDFS_PORT", "8020")
user = os.environ.get("HDFS_USER")
# The user (and replication, buffer_size, default_block_size) go in the URI's query string
uri = f"hdfs://{host}:{port}/" + (f"?user={user}" if user else "")
hdfs = fs.HadoopFileSystem.from_uri(uri)
# List the root directory
for info in hdfs.get_file_info(fs.FileSelector("/")):
print(info.path, info.type)
import os
from pyarrow import fs
host = os.environ["HDFS_HOST"]
port = os.environ.get("HDFS_PORT", "8020")
user = os.environ.get("HDFS_USER")
# The user (and replication, buffer_size, default_block_size) go in the URI's query string
uri = f"hdfs://{host}:{port}/" + (f"?user={user}" if user else "")
hdfs = fs.HadoopFileSystem.from_uri(uri)
# List the root directory
for info in hdfs.get_file_info(fs.FileSelector("/")):
print(info.path, info.type)
Example 2¶
Get file info in a folder, recursively, on a cluster that uses Kerberos
In [ ]:
Copied!
import os
from pyarrow import fs
hdfs = fs.HadoopFileSystem(
os.environ["HDFS_HOST"],
port=int(os.environ.get("HDFS_PORT", "8020")),
user=os.environ.get("HDFS_USER"),
kerb_ticket=os.environ["HDFS_KERBEROS_TICKET_CACHE"],
)
hdfs.get_file_info(fs.FileSelector(os.environ.get("HDFS_FOLDER", "/tmp"), recursive=True))
import os
from pyarrow import fs
hdfs = fs.HadoopFileSystem(
os.environ["HDFS_HOST"],
port=int(os.environ.get("HDFS_PORT", "8020")),
user=os.environ.get("HDFS_USER"),
kerb_ticket=os.environ["HDFS_KERBEROS_TICKET_CACHE"],
)
hdfs.get_file_info(fs.FileSelector(os.environ.get("HDFS_FOLDER", "/tmp"), recursive=True))