Federal Data Users

Hello colleagues, 

Anyone here have experience migrating SAS datasets into Parquet format? I may need to do this for many very large datasets.

I would appreciate any expertise. Tools, workflows, etc.

Also would love to hear from anyone doing app development using DuckDB with Parquet storage.

Exciting stuff!

Thanks,

Glenn

Hi Glenn. Jon Rothbaum at Census wrote a python package for converting SAS files to parquet. I've used it for internal work with very large data sets, and it's worked quite well. It's still a work-in-progress, though.

https://github.com/jrothbaum/polars_readstat

Thanks for this! Looks like great stuff indeed.

If the benchmarks are to be believed, this will be much faster than saspy (which uses Pandas DFs). Also it appears to support reading SAS metadata, including formats, which are a major component of our flow.

I'll be digging into this for sure. I appreciate the suggestion.

You should be able to do this for low memory/fast conversion: 

scan_readstat("/path/to/myfile.sas7bdat").sink_parquet("/path/to/my/file.parquet")

If you want to guarantee the order is preserved, you can do preserve_order=True (won't be low memory, though) or threads=1 (keeps order, slower, but stays low ram).