Hi HN,
Personally, one of the most dreadful aspects of working on HDFS + Hive + Spark data pipelines has been the “wait” game while spinning up a cluster on cloud. So one day I said I am going to run things locally; and after a week of trial and error, I got the pipelines running all in my laptop. This was honestly a game-changer, increasing velocity and enabling full agentic workflow.
I built local-data-platform because it took me a week to setup HDFS + Hive + Spark all working together on my local machine and how messy it was to switch between different settings (with or without HDFS).
I am sharing the repo in case anyone is stuck in the same predicament.
0 comments