Vaex
What it is: Out-of-core DataFrame library for visualizing and exploring billion-row datasets. Uses memory mapping and lazy evaluation to work with data larger than RAM.
What It Does Best
Instant statistics. Calculate mean, std, histograms on billion rows in seconds. Memory mapping means no data loading time. Zero-copy operations.
Built-in visualization. Plot histograms and heatmaps on massive datasets interactively. Samples intelligently for responsive plots. Explore data visually before cleaning.
Lazy everything. Expressions evaluated only when needed. Create virtual columns, filter, transform—all free until you compute. Optimize query automatically.
Pricing
Free. Open source, MIT license.
When to Use It
✅ Exploring massive datasets (billion+ rows)
✅ Need quick statistics without loading data
✅ Interactive visualization of big data
✅ Data stored in HDF5/Arrow/Parquet
When NOT to Use It
❌ Complex data transformations (limited API vs pandas)
❌ Need distributed computing (single machine only)
❌ Heavy string operations (optimized for numerics)
❌ Small data (pandas/Polars simpler)
Bottom line: Unique niche—interactive exploration of huge data on single machine. If you need to visualize and understand billion-row datasets before cleaning, Vaex is magic. Less mature than Dask, but incredibly fast for its use case.