AoM: Data Science

It was mentioned in passing in my previous post that I needed to setup `winutils.exe` in order to save my dataframe to a parquet file. It was not, however, mentioned why I wanted to save the dataframe as a parquet file. In short, the answer is for speed. First, let's load a dataset using pyspark, save it as a parquet file, load it back, and compare the time it takes to count the number of items in the file. Select your favorite medium-sized data file. I happened to be working with a CSV file that was about 7 GB, "data.csv" (composed of about 6.5 million rows and 300 columns). My data had a header, so I ran `df = spark.read.option("header", "true").csv("data.csv")` The default headings had characters not allowed by parquet when saving, so I looped through the columns and use regex to find and replace the set of illegal characters (" ,;{}()\n\t="): import re df2 = df for column in df.columns: new_column = re.sub(...

It seems like just about every six months I need to install PySpark and the experience is never the same. Note that this isn't necessarily the fault of Spark itself. Instead, it's a combination of the many different situations under which Spark can be installed, lack of official documentation for each and every such situation, and me not writing down the steps I took to successfully install it. So today, I decided to write down the steps needed to install the most recent version of PySpark under the conditions in which I currently need it: inside an Anaconda environment on Windows 10. Note that the page which best helped produce the following solution can be found here (Medium article). I later found a second page with similar instructions which can be found here (Towards Data Science article). Steps to Installing PySpark for use with Jupyter This solution assumes Anaconda is already installed, an environment named `test` has already been created, and Jupyter has already...

I recently took a look at Dash which seems great for data visualization, but I have a project for which I'd love to build a user interface. I previously used PySimpleGUI and the result was to my satisfaction. However, there was something left to be desired. In any case, that brings me to Qt (pronounced "cute") for Python. Let's check out the documentation and see what we can do with it. Headings in this post follow the headings in the documentation. Qt for Python Quick start `pip install PySide2` Right off the bat. The `hello_world.py` seems more difficult to create something simple (versus PySimpleGUI ). However, I imagine all this complexity is what eventually enables a more advanced user interface (I later learn about QtQuick/QML). Remark: I encountered an error for which I tried various environment variable fixes. Eventually the solution which worked for me was this answer from stackoverflow. But I wasn't happy with just implementing the solution...