Apache Spark Optimisation Techniques Data Science by Sunny Srinidhi - February 23, 2023February 23, 20230 Apache Spark is one of the most popular big data processing tools today. It’s used extensively for data sizes small to large. The availability of Spark in more than one programming language makes it a favourite tool for data engineers and data scientists coming from various backgrounds. Read more... “Apache Spark Optimisation Techniques”
Optimising Hive Queries with Tez Query Engine Data Science by Sunny Srinidhi - June 13, 2022June 13, 20220 Hive provides us the option of executing SQL queries with a few different query engines. It ships with the native MapReduce engine. But we can switch that to Tez which has gained popularity since its launch, or we can also use Apache Spark as well. Read more... “Optimising Hive Queries with Tez Query Engine”
Understanding Apache Hive LLAP Data Science by Sunny Srinidhi - November 18, 2021November 18, 20210 Apache Hive is a complex system when you look at it, but once you go looking for more info, it’s more interesting than complex. There are multiple query engines available for Hive, and then there’s LLAP on top of the query engines to make real-time, interactive queries more workable. Read more... “Understanding Apache Hive LLAP”
Installing Hadoop on the new M1 Pro and M1 Max MacBook Pro Data Science by Sunny Srinidhi - November 5, 2021November 5, 20213 In the previous series of posts, I wrote about how to install the complete Hadoop stack on Windows 11 using WSL 2. And now that the new MacBook Pro laptops are available with the brand new M1 Pro and M1 Max SOCs, here’s a guide on how to install the same Hadoop stack on these laptops. Read more... “Installing Hadoop on the new M1 Pro and M1 Max MacBook Pro”
Installing Hadoop on Windows 11 with WSL2 Data Science by Sunny Srinidhi - November 1, 2021November 1, 20213 In the previous post, we saw how to install a Linux distro on Windows 11 using WSL2 and then how to install Zsh and on-my-zsh to make the terminal more customizable. In this post, we’ll see how we can install the complete Hadoop environment on the same Windows 11 machine using WSL. Read more... “Installing Hadoop on Windows 11 with WSL2”
Installing Zsh and Oh-my-zsh on Windows 11 with WSL2 Tech by Sunny Srinidhi - October 27, 2021October 27, 20211 Before we begin, you might ask, why am I writing on something this trivial? I sold off my old MacBook Pro because I’m super excited about the new M1 Pro MacBook Pros. I have pre-ordered one of those and am waiting for it to come. Read more... “Installing Zsh and Oh-my-zsh on Windows 11 with WSL2”
Getting Started With Apache Airflow Data Science by Sunny Srinidhi - October 11, 2021October 11, 20210 Apache Airflow is another awesome tool that I discovered just recently. Just a couple of months after discovering it, I can’t imagine not using it now. It’s reliable, configurable, and dynamic. Because it’s all driven by code, you can version control it too. Read more... “Getting Started With Apache Airflow”
Querying Hive Tables From a Spring Boot App Data Science Tech by Sunny Srinidhi - June 30, 2021June 30, 20211 In this post, we’ll see how we can query tables that reside in Hive using a Spring Boot application. As always, I’m going to use a Spring Boot web app with a few GET APIs to show how we can query data from Hive. Read more... “Querying Hive Tables From a Spring Boot App”
Getting Started With JanusGraph Data Science by Sunny Srinidhi - February 25, 2021February 25, 20211 JanusGraph is a graph processing tool that can process graphs stored on clusters with multiple nodes. JanusGraph is designed for massive clusters and for real-time traversals and analytics queries. In this post, we’ll look at a few queries that you would want to run the very first time you install JanusGraph and start playing with the Gremlin console. Read more... “Getting Started With JanusGraph”
Null Hypothesis and the P-Value Data Science by Sunny Srinidhi - November 8, 2019November 8, 20195 When you're starting your machine learning journey, you'll come across null hypothesis and the p-value. At a certain point in your journey, it becomes quite important to know what these mean to make meaningful decisions while designing your machine learning models. So in this post, I'll try to explain what these two things mean, and you try to understand that. Now, if you don't have a background in statistics, the definitions of null hypothesis and p-value will make no sense to you. It's just gibberish going way over your head. That's what happened to me the first few times I tried to understand them. It took me a good couple of days to get an idea of what they mean. I