Latent semantic hierarchies derived from foundation model embeddings yield higher downstream utility for dataset selection than explicit metadata-based hierarchies.

Motivation

The original paper relies on existing hierarchical structures (likely metadata or source-based), which may not reflect the true statistical distribution or transferability of the data. Metadata is often noisy, missing, or irrelevant to the learning task, whereas content-based latent structures could group datasets by feature alignment.

Proposed Method

1. Aggregate samples from candidate datasets and generate embeddings using a pre-trained foundation model (e.g., CLIP for images, BERT for text). 2. Perform hierarchical clustering on these embeddings to construct a synthetic 'latent hierarchy' tree. 3. Apply the paper's selection algorithm on this synthetic tree versus the original metadata tree. 4. Evaluate downstream model performance on a target task using data selected from both methods.

Expected Contribution

Demonstrating that data-driven, content-aware hierarchies outperform human-labeled metadata hierarchies would significantly broaden the applicability of this method to unstructured data lakes lacking metadata.

Required Resources

Access to pre-trained foundation models, GPU compute for embedding generation and clustering, and a diverse collection of datasets (e.g., DataComp or HuggingFace Hub subsets).

Source Paper

Hierarchical Dataset Selection for High-Quality Data Sharing

View Paper Details →