Q: Pyspark going extremely slow on Windows? So, I'm trying to build a program that goes through all of the files in a directory, and then uses Spark to read into these files and count the instances of a given word (let's say simply the word "word").
I have a Python function get_files that receives a path as an input and returns a string with all of the files in that directory separated by a comma.
I've tried doing something like this:
import pyspark
import get_files

sc = pyspark.SparkContext('local[*]')
files = get_files(<path/to/my/files>)
num = sc.flatMap(lambda x: x.split(' ')).filter(lambda x: 'word' in x).count()

This whole process takes around 40 minutes for a folder with around 120Mb. If  instead I simply build a function with Python (no Spark) that iterates through each of these files, opens them, and counts the words, it just takes a couple of seconds. In case it's important, I'm running Windows with an Intel i5 with 4 cores and 8 logical processors, and with 8Gb RAM.
I've tried several different methods: this was ran on a Jupyter Notebook, but I've tried running it from a Python script, and also from the command prompt. I've also tried configuring the number of cores manually (doing like local[8] or local[7] instead of local[*]) or also tried changing some configuration values regarding the number of workers, or adjusting the number of partitions it creates, but either way it goes super slow.
I also try running localhost:4040 and I can see that it does generate several partitions and it is using all cores. When going to the "Executors" sections, I can see there's one executor running, using all 8 cores. strong text
