Hdfs.Files

What is Power Query?

Power Query is a data connectivity and transformation tool that allows users to connect to various data sources, transform the data, and load it into Excel or Power BI. It provides a user-friendly interface making it easy to perform complex data transformations without the need for coding. Power Query supports a wide range of data sources, including databases, Excel files, text files, and web services.

What is Hdfs.Files?

Hdfs.Files is a Power Query M function that allows users to connect to Hadoop Distributed File System (HDFS) and retrieve a list of files from a directory in HDFS. HDFS is a distributed file system that is commonly used in big data environments. The Hdfs.Files function makes it easy to connect to HDFS and retrieve files for further processing.

The M Code Behind Hdfs.Files

The M code behind Hdfs.Files is relatively simple. The function takes two parameters: the URL of the HDFS instance and the directory path to retrieve the files from. The function uses the Web.Contents function to connect to the HDFS instance and retrieve the file list. The file list is then parsed using the Xml.Tables function to extract the file names and other information.

Here is an example of the M code for the Hdfs.Files function:

let

Source = (url as text, path as text) =>

let

hdfsUrl = url & “/webhdfs/v1” & path & “?op=LISTSTATUS”,

fileContent = Web.Contents(hdfsUrl),

xml = Xml.Tables(fileContent),

files = xml{0}[#”Table1″],

fileList = Table.FromList(files, Splitter.SplitByNothing(), null, null, ExtraValues.Error),

fileTable = Table.TransformColumnTypes(fileList, {{“Column1”, type text}})

in

fileTable

in

Source

Hdfs.Files(“http://hdfs-instance:50070”, “/path/to/directory”)