Datasets:

Smith42
/

mmu_desi_edr_sv3_x_mmu_ssl_legacysurvey_north

image_order int8	image_pixel int64	image_id string	spectrum_order int8	spectrum_pixel int64	spectrum_id string
6	9,144	12952683	8	146,319	39632936147880205
6	9,144	2015893	8	146,319	39632936147880659
6	9,144	13804158	8	146,319	39632936152073158
6	9,144	5388607	8	146,319	39632936152073395
6	9,144	986338	8	146,319	39632936147880966
6	9,144	4106747	8	146,319	39632936147880919
6	9,144	1623079	8	146,319	39632936147880778
6	9,144	7643606	8	146,319	39632936147880777
6	9,144	11837759	8	146,319	39632936152072340
6	9,144	4748686	8	146,319	39632936152072601
6	9,144	371600	8	146,319	39632936152072633
6	9,144	4673711	8	146,319	39632936152072293
6	9,145	1724879	6	9,145	39632941243958815
6	9,145	13285548	6	9,145	39632941243959658
6	9,145	12336627	6	9,145	39632941243959488
6	9,145	8042889	6	9,145	39632941243959779
6	9,145	6053055	6	9,145	39632941243959724
6	9,145	1604460	6	9,145	39632941243960750
6	9,145	1744612	6	9,145	39632941243960555
6	9,145	1135408	6	9,145	39632941243960371
6	9,145	2947627	6	9,145	39632941243960650
6	9,145	5926113	6	9,145	39632941243961287
6	9,145	5862514	6	9,145	39632941243961452
6	9,145	8118217	6	9,145	39632941243961025
6	9,145	5683314	6	9,145	39632941243960492
6	9,145	4643282	6	9,145	39632941243960744
6	9,145	2853519	6	9,145	39632941243960918
6	9,145	2227623	6	9,145	39632941243960943
6	9,145	1694119	6	9,145	39632941248152269
6	9,145	4326287	6	9,145	39632941248151856
6	9,145	4594353	6	9,145	39632941248152174
6	9,145	13555418	6	9,145	39632941248152693
6	9,145	3115307	6	9,145	39632941248152621
6	9,145	12882359	6	9,145	39632941248152914
6	9,145	2173132	6	9,145	39632941243961528
6	9,145	13767791	6	9,145	39632941248151802
6	9,145	1915693	6	9,145	39632941248151998
6	9,145	2463625	6	9,145	39632941243961319
6	9,145	7741817	6	9,145	39632941243961292
6	9,145	5644178	6	9,145	39632941243961474
6	9,145	266043	6	9,145	39632941243961236
6	9,145	8676705	6	9,145	39632941248151603
6	9,145	6289273	6	9,145	39632941248151765
6	9,145	6100555	6	9,145	39632941248152345
6	9,145	1346720	6	9,145	39632941248152587
6	9,145	3365608	6	9,145	39632941248152501
6	9,145	4271439	6	9,145	39632941248152670
6	9,145	8760759	6	9,145	39632941248152102
6	9,145	1664705	6	9,145	39632946310679271
6	9,145	2390690	6	9,145	39632946310679353
6	9,145	69338	6	9,145	39632946310679358
6	9,145	12354896	6	9,145	39632936156267376
6	9,145	12991253	6	9,145	39632936156268000
6	9,145	4777548	6	9,145	39632936156267539
6	9,145	8053410	6	9,145	39632936156267626
6	9,145	6998368	6	9,145	39632936156268734
6	9,145	5328108	6	9,145	39632936156269476
6	9,145	6615998	6	9,145	39632936156268939
6	9,145	12819493	6	9,145	39632936156269445
6	9,145	3291938	6	9,145	39632936156269474
6	9,145	3391988	6	9,145	39632936156268141
6	9,145	13309176	6	9,145	39632936156267994
6	9,145	4826188	6	9,145	39632936156268310
6	9,145	8582782	6	9,145	39632936156268700
6	9,145	3331695	6	9,145	39632936156269286
6	9,145	2933060	6	9,145	39632936156270109
6	9,145	1558067	6	9,145	39632936156270149
6	9,145	5499148	6	9,145	39632941239765500
6	9,145	6906177	6	9,145	39632941239765115
6	9,145	4293987	6	9,145	39632936156269553
6	9,145	8244741	6	9,145	39632941239763989
6	9,145	3455407	6	9,145	39632941239764400
6	9,145	5020524	6	9,145	39632941239764837
6	9,145	8020587	6	9,145	39632936152075823
6	9,145	7823466	6	9,145	39632936152074985
6	9,145	6226495	6	9,145	39632936152074543
6	9,145	1032971	6	9,145	39632936152073945
6	9,145	12288869	6	9,145	39632936152074214
6	9,145	10324544	6	9,145	39632936152074255
6	9,145	5450573	6	9,145	39632936152074710
6	9,145	6214968	6	9,145	39632936152074781
6	9,145	7485738	6	9,145	39632936152075027
6	9,145	372324	6	9,145	39632936152074555
6	9,145	7948049	6	9,145	39632936152074895
6	9,145	564343	6	9,145	39632936152075857
6	9,145	2427904	6	9,145	39632936152076039
6	9,145	13913062	6	9,145	39632936152075893
6	9,145	3117089	6	9,145	39632936152075576
6	9,145	2178168	6	9,145	39632936156267491
6	9,145	8483623	6	9,145	39632936156267256
6	9,145	13059470	6	9,145	39632936156267284
6	9,145	7373804	6	9,145	39632936156266834
6	9,145	8613516	6	9,145	39632936156266718
6	9,145	13480408	6	9,145	39632936152075338
6	9,145	1463866	6	9,145	39632936152075215
6	9,145	8462615	6	9,145	39632936152075172
6	9,145	5164190	6	9,145	39632936152075335
6	9,145	13338618	6	9,145	39632936152075785
6	9,145	1151060	6	9,145	39632936152075304
6	9,145	1245279	6	9,145	39632936152075100

End of preview. Expand in Data Studio

mmu_desi_edr_sv3 × mmu_ssl_legacysurvey_north

A lightweight match-index that pairs LegacySurvey imaging objects with DESI spectra: ~138k matched pairs, published as ~tens of MB of id pointers, not pixels. It is part of the Multimodal Universe crossmatch tooling (see the HATS crossmatch blogpost) and is the fast-path source for the AstroPT v3 image×spectrum pairs stream.

This is an index of object ids + HATS partition cells. Every row is one matched pair: where to find the image row, where to find the spectrum row. Fetch the actual image/spectrum by joining back to the two source catalogs (see Usage).

Columns

name	type	description
`image_order`	int8	HEALPix partition order of the matched image in `mmu_ssl_legacysurvey_north`
`image_pixel`	int64	HEALPix partition pixel of the matched image
`image_id`	string	`object_id` of the matched image in the LegacySurvey catalog (stored as string)
`spectrum_order`	int8	HEALPix partition order of the matched DESI spectrum
`spectrum_pixel`	int64	HEALPix partition pixel of the matched DESI spectrum
`spectrum_id`	string	`object_id` of the matched spectrum in the DESI catalog (stored as string)

Partitions are addressed by their HEALPix (order, pixel) cell, not by position in any catalog listing, so this index resolves to the right partitions regardless of how the source catalogs are re-sharded.

Usage

The index is a single parquet; pyarrow resolves hf:// natively through huggingface_hub, so no bulk download is needed. To fetch one matched pair's actual image and spectrum, join back to the two source HATS catalogs with LSDB:

import pyarrow.parquet as pq
import lsdb

# 1. read the index
idx = pq.read_table(
    "hf://datasets/Smith42/mmu_desi_edr_sv3_x_mmu_ssl_legacysurvey_north/match_index.parquet"
).to_pandas()

# 2. open the two source HATS catalogs
img = lsdb.open_catalog("hf://datasets/UniverseTBD/mmu_ssl_legacysurvey_north")
spec = lsdb.open_catalog("hf://datasets/UniverseTBD/mmu_desi_edr_sv3")

# 3. pick a matched pair and fetch the rows by object_id / partition cell
row = idx.iloc[0]
img_row = img.get_partition(row.image_order, row.image_pixel).compute()
image = img_row[img_row["object_id"] == int(row.image_id)]
spec_row = spec.get_partition(row.spectrum_order, row.spectrum_pixel).compute()
spectrum = spec_row[spec_row["object_id"] == int(row.spectrum_id)]

The *_id columns are stored as strings here; cast to the source catalog's object_id type when matched. To match the full collection, use the same pointers on every row, or run the live LSDB streaming crossmatch yourself for fresh results (see the blogpost above).

Source catalogs

side	catalog	partition order
image	`UniverseTBD/mmu_ssl_legacysurvey_north`	variable HATS (use the cell in `image_order`/`image_pixel`)
spectrum	`UniverseTBD/mmu_desi_edr_sv3`	variable HATS (use the cell in `spectrum_order`/`spectrum_pixel`)

How it was built

Built offline by scripts/build_match_index.py using LSDB's crossmatch. Headline parameters:

radius_arcsec = 1.0 (pinned in astropt3/data/streaming.py)
n_neighbors = 1 — one nearest match per image object
how = "inner" — only objects present in both surveys
left suffix "", right suffix "_desi"

Publish to the hub with:

hf upload Smith42/mmu_desi_edr_sv3_x_mmu_ssl_legacysurvey_north match_index.parquet --repo-type=dataset

See the script for the full partition-resolution logic and a smoke build.

Caveats

Id index, not joined rows. You must join back to the source catalogs to get pixels/fluxes; this index only locates the matched rows.
One match per image. n_neighbors=1 returns the single nearest spectrum within 1.0 arcsec per image object.
Bounded by the spectroscopic side. how="inner" means only objects with a DESI spectrum appear; objects with imaging only are not listed here.
Redundancy is intentional. A matched image also appears standalone in mmu_ssl_legacysurvey_north; the AstroPT v3 stream counts both, by design (ADR 0006 §1).

AstroPT v3 consumption

The AstroPT v3 pairs stream reads this index directly from the hub and joins by id at train time (no spatial join in the data loader):

export ASTROPT3_MATCH_INDEX="hf://datasets/Smith42/mmu_desi_edr_sv3_x_mmu_ssl_legacysurvey_north/match_index.parquet"

See astropt3.data.streaming (load_match_index, _pairs_dataset) for the exact join contract.

Citation

If you use this crossmatch, please cite:

The Multimodal Universe — The Multimodal Universe: Enabling Large-Scale Machine Learning with 100TBs of Astronomical Scientific Data, arXiv:2412.02527.
DESI — DESI Early Data Release, SV3.
LegacySurvey — The DESI Legacy Imaging Surveys (DECaLS North).

Acknowledgements

This match-index was built with HATS and LSDB, on top of the Multimodal Universe v1.5 HATS release. See the HATS crossmatch blogpost for the full tool stack and acknowledgements.

Downloads last month: 28

Paper for Smith42/mmu_desi_edr_sv3_x_mmu_ssl_legacysurvey_north

The Multimodal Universe: Enabling Large-Scale Machine Learning with 100TB of Astronomical Scientific Data

Paper • 2412.02527 • Published Dec 3, 2024 • 13