GRAND UNIFIED FILE INDEX

We live in a time of massive data generation. Never have we been tasked with storing, sorting, extracting metadata, indexing, protecting, and organizing so much critical information at such a vast scale. This has created major challenges for large-scale data centers, where data is growing more rapidly than the capabilities of the storage-systems that house that data. Our data is useless, if we can’t find what we are looking efficiently. Queries can have a devastating impact on the performance of ongoing computations/operations, which must themselves have unimpeded storage-system access to avoid wasting computational resources. Furthermore, security requires that results of queries must be constrained to include only data that a given user or artificial intelligence entity is allowed to see, including even the filenames. AI has the potential to be an incredibly force multiplier for science/defense, but one of its main tenants is aggregation of lots of information, which in a need-to-know world makes for issues. Leveraging something like AI while containing its aggregation scope is a difficult problem in a highly shared but need-to-know controlled environment. It is also useful to consider the differences in types of queries done by users versus by systems/data management professionals for a variety of tasks and try to accommodate all areas in a comprehensive indexing capability.

Storage systems have need-to-know access technologies such as POSIX permissions used in basically all file systems, access control lists (ACL) used in file systems and object systems like in the cloud, Multi-Level-Security (MLS) systems which use labeling and compartmented/policy driven access control. The problem is, storage-systems are typically not good at metadata oriented queries over the file/object name space or over metadata extracted from the files/objects. In order to find what you are looking for and to leverage things like AI, one needs to be able to easily find and leverage extracted metadata. What’s worse, each storage-system, which emphasizes a different strength for the storage aspect of the solution, often different tools for metadata handling. There are a few tools in industry that help with metadata indexing and query but essentially none specialize in honoring user-oriented security, most are tools meant for system administrators to use and user access is something of an afterthought.

The Grand Unified File Index (GUFI) is a solution to finding and utilizing extracted metadata that honors access control of the source information and as users change their need-to-know the access is adjusted the same as the source information. GUFI is an index that holds storage-system metadata (i.e. filenames, access and creation dates, file attributes, extended attributes, etc.) and extracted metadata from those files like text and other attributes, pulled from huge file/data storage-systems using full and incremental index update GUFI allows for rapid searches that don’t have to impact the file-systems themselves, supporting both users and system/data managers. GUFI is a very fast software solution that offers speed and security, while minimizing impact on supercomputing and source storage-system resources.

GUFI can be applied to a variety of file, object, and archive systems (tape archives, and file-system, object systems, and others), making it a ubiquitous solution for indexing that supports arbitrary queries and unifying information from all the places where a stored information might reside. Further, GUFI can dynamically query other information from things like databases, key value stores, etc. acting as the user. GUFI is a one-stop shop for access-controlled query and consumption of metadata from all your holdings. GUFI allows the users, artificial intelligent systems/agents, and system administrators to focus on doing the actual work for which these queries are just prerequisites.

GUFI Requirements

The following requirements were all considered in the design choices made in producing this utility:

• Unified index over multiple heterogeneous storage systems including shared storage like home and project space, scratch for the very hot tier, warm tier like LANL’s campaign store/MarFS, Object Systems, and Archives.
• Enable dynamic access to other non-storage-system information
• Obey and support full POSIX tree attributes, permissions, hierarchy representation as well as access control lists and compartmented multi-level access control security etc.
• Stores/indexes metadata from derived from source folders, buckets, and files
• Shared index for users and admins, users can only see metadata derived from files/folders/buckets/trees they could read from the source files/folders/trees
• Parallel search capabilities that are very fast (minutes for billions of files/dirs) including parallelism both threads and cross node
• Parallel metadata extraction with both full and incremental updating of the index, make the updating of this index as painless as possible
• The index should be highly composable/decomposable to allow multi-site and multi-storage-system indexing. This includes sub setting and aggregating parts of the index, all while honoring user access controls.
• Index can live in a separate space or within the source file/storage systems themselves
• Can appear as mounted file system where you get a virtual image of your file metadata based on query input and also a pre-run query can also appear as a mounted file system
• Provide a way to work in a pure POSIX environment requiring only a POSIX interface to storage systems for full and incremental metadata extraction in addition to other non-POSIX techniques.
• Exploit special interfaces from source file/storages systems provided by some storage systems for mass metadata manipulation/extraction like GPFS ILM, Lustre activity logs, extraction from Robinhood SQL, Extraction from HPSS DB2, etc.
• Be open-source software and leverage other open source software and/or open interfaces
• Very transparent and simple so one can easily understand/enhance/administrate with simple to understand formats. Avoid black box anything.
• Extensible capabilities, especially in the query area, for example enable outputs from query to be consumable by humans or other programs and ability to connect in external data sources into query results. Ability to store both base POSIX information and potentially source storage system unique metadata per entry. Ensure connectivity with the Apache analytics ecosystem and emerging standards like Model Context Protocol (MCP) for AI consumption. Additionally, enable additional indexing and search types as they emerge, like full text search on extracted text and vector similarity over text or other extracted metadata, all while obeying user access control.
• The intent is to provide a periodically consistent index (doesn’t have to be a perfect snapshot or continuously keeping up with source file/storage systems).
• The intent is not to produce a policy management system for users or admins but of course it could provide the index usable for policy management function.
• Keep the code base small by leveraging existing technology as much as possible both hardware and software including:
  • flash storage: Assume the index would be small enough to fit in flash storage or even perhaps memory with a few hundred bytes of index per entry. Assume metadata mount per directory will typically be a few kilobytes so parallel access ends up looking like multi-kilobyte random reads which is well suited to flash devices which can provide millions of read IOPS.
  • both process/multi-machine and thread parallelism: where possible enable both types of parallelism for speed and efficiency
  • A standard and powerful basis for search: enable the exploitation of the power and stability of an existing basis for search even if the interface to the user doesn’t export that interface (like SQL or other)
  • commercial database technology: no need to invent our own underlying indexing/database technology given the abundance of solutions available
  • commercial file system technology: leverage commercial file system technology and its strengths including extremely fast traversal and access control which is probably of the most optimized code in the world
  • open source software: leverage open interfaces/software where possible
  • agnostic to leveraged parts where possible: if depending on external function where possible enable use of more than one provider of that external function
  • Very transparent and simple so one can easily understand/enhance/administrate.

Most file/storage-systems employ the concept of a “tree” of directories containing sub-directories, containing sub-sub-directories, etc. Even with storage systems that are not inherently tree based, users still implement folder like organizations. GUFI partitions storage-system metadata and metadata extracted from files/folders into many small databases, kept in a directory-tree having the same structure and access-controls (POSIX/ACL/MLS/etc.) as that of the original storage-system or in the source storage system itself. Each folder/bucket in the source storage-system directory-tree has a matching folder in the GUFI directory-tree, with the same access permissions. Due to the ability to do breadth first traversal of trees, this allows for fast parallel searches across the GUFI databases (one per folder). Furthermore, because the directory access-controls are replicated from the original storage-system, regular users can be permitted to conduct their own searches, because they will simply not have access to the GUFI DBs that live in folders they can’t access. This mimics POSIX security/access control exactly and with access control lists mimics cloud object access control as well.