• GUFI Documents
• Github
• Videos
• Mailing List

Modern High-Performance Computing (HPC) data centers routinely store massive data sets resulting in millions of directories and billions of files. To efficiently search and sift through these files and directories we have developed  the Grand Unified File Index (GUFI), a novel file system metadata index that enables both privileged and regular users to rapidly locate and characterize data sets of interest. GUFI uses a hierarchical index that preserves file access permissions such that the index can be securely accessed by users while still enabling efficient, advanced analysis of storage system usage by cluster administrators. Compared with the current state-of-the-art indexing for file system metadata, GUFI is able to provide speedups of 1.5× to 230× for queries executed by administrators on a real production file system namespace. Queries executed by users, which typically cannot rely on cluster-wide indexing, see even greater speedups using GUFI.

WHAT DOES GUFI DO? 

Every research institution accumulates documents over years – SOPs, policy memos, training decks, how-to guides, incident reports, lab notebooks. This collection is critical; it encodes how the organization operates. It is also, almost invariably, a mess – outdated, conflicting, and hard to navigate. For a complete examination of a resource, extra metadata, beyond just file paths/names/dates/sizes/extended attributes/etc. to be stored for querying must be separately extracted from the files being indexed. We use an external metadata extraction package, VaultIQ, to pull metadata from these files. It works for Microsoft files, pdfs, all image types, movies, audio, code (e.g., python/C/...), etc., as well as many new types of extraction capabilities constantly being added. Using alternative external deep extraction mechanisms is also possible if you prefer your own/other. Any amount of metadata extracted from files can be indexed, and then queried by GUFI. Querying extracted data and metadata involves building a request-specific evidence plan, and combines semantic search, exact-term retrieval, document-structure signals, and follow-up retrieval as needed, following the source file access controls, allowing data holdings to be turned into a RAG/GraphRAG that follows access control rules.