Parquet + PySpark (= Speed)

It was mentioned in passing in my previous post that I needed to setup `winutils.exe` in order to save my dataframe to a parquet file. It was not, however, mentioned why I wanted to save the dataframe as a parquet file. In short, the answer is for speed.

First, let's load a dataset using pyspark, save it as a parquet file, load it back, and compare the time it takes to count the number of items in the file.

Select your favorite medium-sized data file. I happened to be working with a CSV file that was about 7 GB, "data.csv" (composed of about 6.5 million rows and 300 columns).

My data had a header, so I ran
`df = spark.read.option("header", "true").csv("data.csv")`

The default headings had characters not allowed by parquet when saving, so I looped through the columns and use regex to find and replace the set of illegal characters (" ,;{}()\n\t="):
import re

df2 = df
for column in df.columns:
new_column = re.sub("[ ,;{}()\n\t=]", "", column)
df2 = df2.withColumnRenamed(column, new_column)

To save the file as a parquet, we run
`df2.write.parquet("data.parquet")`
.

Once it's been saved, we can load it back into the notebook with the following:
`parquetFile = spark.read.parquet("data.parquet")`

Finally, for an example comparison, we compare `%timeit df.count()` and `%timeit parquetFile.count()`. Over 7 runs, 1oop each, they ran 20.6 s +/- 1.11 s per loop and 1.01 s +/- 123 ms per loop, respectively. That's a huge difference!*

Note that we although we spent an initial upfront cost of 7 min 15 s to save the dataframe as a parquet file, we benefit in the long-run after sufficiently many operations. In terms of the `count()` method, it would take about 20 uses of `count()` to even out on the initial cost.

*I later ran a second set and the second set came up 26.2 s +/- 389 ms per loop and 933 ms +/- 86.4 ms per loop, respectively.
Load times were similar. Loading from the CSV would take 163 +/- 29.7 ms per loop while loading from the parquet file takes 153 ms +/- 52.8 ms per loop.

While the original CSV is about 7 GB, the parquet file is just 770 MB.
On June 25, 2018, I added a draft post to my notes blog titled "Parquet is Efficient." The draft never got filled. However, I remember working with Spark and performing some joins which took a long time to run. Luckily, my coworker somehow figured out that saving to and loading from a parquet file made certain operations incredibly fast. In particular, my join which originally took several hours before the use of parquet became just several minutes after the use of parquet - if I recall correctly (assuming the same 20 factor difference, 8 hours would reduce to 24 minutes).

Comments

Post a Comment

Popular posts from this blog

PySpark + Anaconda + Jupyter (Windows)

It seems like just about every six months I need to install PySpark and the experience is never the same. Note that this isn't necessarily the fault of Spark itself. Instead, it's a combination of the many different situations under which Spark can be installed, lack of official documentation for each and every such situation, and me not writing down the steps I took to successfully install it. So today, I decided to write down the steps needed to install the most recent version of PySpark under the conditions in which I currently need it: inside an Anaconda environment on Windows 10. Note that the page which best helped produce the following solution can be found here (Medium article). I later found a second page with similar instructions which can be found here (Towards Data Science article). Steps to Installing PySpark for use with Jupyter This solution assumes Anaconda is already installed, an environment named `test` has already been created, and Jupyter has already...
Read more

Using tensorflowjs_converter

A friend and I were looking to convert a Tensorflow model to a TensorFlow.js model. My friend had done some prior research and started me off with the following page: Importing a Keras model into TensorFlow.js . The first step, and one which I messed up on, is installing the `tensorflowjs` library into a Python environment: `pip install tensorflowjs`. Although I'm not sure on the reason for the mistake, the fact of the matter is that I ended up installing `tensorflow` (`pip install tensorflow`) and spent a good amount of time wondering why the bash command `tensorflowjs_converter` wasn't working. In any case, in the tutorial linked above, the command takes the following structure: # bash tensorflowjs_converter --input_format keras \ path/to/my_model.h5 \ path/to/tfjs_target_dir Meanwhile, the readme for the tfjs COCO-SSD model supplies a converter command for removing the post process graph as a way of improving performance ...
Read more

pyspark: the number of entries in a column which are null or not

There's were a handful of resources describing how to count the number of null values in a pyspark dataframe. For example this Stack Overflow thread which asks for the number of null (and nan) values for each column in a pyspark dataframe. Thus given the total number of entries in the dataframe, one could indirectly compute the number of not null values (by taking the difference). However, I sought after a solution which would simultaneously output both the number of null and not null values (for a single column). Eventually, I wrote the following possible solution: ( df .select(df.MY_COLUMN_NAME.isNull().alias(MY_COLUMN_NAME)) .groupby(MY_COLUMN_NAME) .count() .show() ) In contrast, the following is the solution with just the count of null values: ( df .select(count(when(df.MY_COLUMN_NAME.isNull()).alias(MY_COLUMN_NAME)) .show() ) Of course the advantage of the latter was it's ability to summarize across every column: ( df .select([count(when(col(c)...
Read more