Apache Spark Optimisation Techniques Data Science by Sunny Srinidhi - February 23, 2023February 23, 20230 Apache Spark is one of the most popular big data processing tools today. It’s used extensively for data sizes small to large. The availability of Spark in more than one programming language makes it a favourite tool for data engineers and data scientists coming from various backgrounds. Read more... “Apache Spark Optimisation Techniques”
Optimising Hive Queries with Tez Query Engine Data Science by Sunny Srinidhi - June 13, 2022June 13, 20220 Hive provides us the option of executing SQL queries with a few different query engines. It ships with the native MapReduce engine. But we can switch that to Tez which has gained popularity since its launch, or we can also use Apache Spark as well. Read more... “Optimising Hive Queries with Tez Query Engine”
Understanding Apache Hive LLAP Data Science by Sunny Srinidhi - November 18, 2021November 18, 20210 Apache Hive is a complex system when you look at it, but once you go looking for more info, it’s more interesting than complex. There are multiple query engines available for Hive, and then there’s LLAP on top of the query engines to make real-time, interactive queries more workable. Read more... “Understanding Apache Hive LLAP”
Installing Hadoop on the new M1 Pro and M1 Max MacBook Pro Data Science by Sunny Srinidhi - November 5, 2021November 5, 20213 In the previous series of posts, I wrote about how to install the complete Hadoop stack on Windows 11 using WSL 2. And now that the new MacBook Pro laptops are available with the brand new M1 Pro and M1 Max SOCs, here’s a guide on how to install the same Hadoop stack on these laptops. Read more... “Installing Hadoop on the new M1 Pro and M1 Max MacBook Pro”
Installing Hadoop on Windows 11 with WSL2 Data Science by Sunny Srinidhi - November 1, 2021November 1, 20213 In the previous post, we saw how to install a Linux distro on Windows 11 using WSL2 and then how to install Zsh and on-my-zsh to make the terminal more customizable. In this post, we’ll see how we can install the complete Hadoop environment on the same Windows 11 machine using WSL. Read more... “Installing Hadoop on Windows 11 with WSL2”
Installing Zsh and Oh-my-zsh on Windows 11 with WSL2 Tech by Sunny Srinidhi - October 27, 2021October 27, 20211 Before we begin, you might ask, why am I writing on something this trivial? I sold off my old MacBook Pro because I’m super excited about the new M1 Pro MacBook Pros. I have pre-ordered one of those and am waiting for it to come. Read more... “Installing Zsh and Oh-my-zsh on Windows 11 with WSL2”
Fake (almost) everything with Faker Data Science by Sunny Srinidhi - September 30, 2021September 30, 20210 I was recently tasked with creating some random customer data, with names, phone numbers, addresses, and the usual other stuff. At first, I thought I’ll just generate random strings and numbers (some gibberish) and call it a day. But then I remembered my colleagues using a package for that. Read more... “Fake (almost) everything with Faker”
Emulating Apache Kafka with Amazon SNS and SQS Tech by Sunny Srinidhi - January 22, 2020January 24, 20200 I have already written quite a few posts about Apache Kafka. It’s an awesome tool for parallel and asynchronous processing. You can have multiple producers and multiple consumers listening to a topic to process each and every message coming out of the topic. Read more... “Emulating Apache Kafka with Amazon SNS and SQS”
Stack Implementation example in Java Tech by Sunny Srinidhi - December 20, 2019December 23, 20192 More in The Data Structures series. A stack is one of the most simplest data structure to understand. If you had data structures in your academia, you already know what it means. It’s a simple Last In First Out (LIFO) queue. What that means is the last element to enter the stack will be first element to go out of the stack. Let’s try to understand the concept first with a few illustrations. The concept Suppose we have an empty container which looks like the container shown in the image below: Empty stack That’s pretty simple to understand. Now suppose again that we “push” a string with value “string1” to this empty stack. The stack now looks like this: Stack with one element That’s pretty simple to
Understanding Word N-grams and N-gram Probability in Natural Language Processing Data Science by Sunny Srinidhi - November 26, 2019December 19, 20192 More in The fastText Series. N-gram is probably the easiest concept to understand in the whole machine learning space, I guess. An N-gram means a sequence of N words. So for example, “Medium blog” is a 2-gram (a bigram), “A Medium blog post” is a 4-gram, and “Write on Medium” is a 3-gram (trigram). Well, that wasn’t very interesting or exciting. True, but we still have to look at the probability used with n-grams, which is quite interesting. Why N-gram though? Before we move on to the probability stuff, let’s answer this question first. Why is it that we need to learn n-gram and the related probability? Well, in Natural Language Processing, or NLP for short, n-grams are used for a variety of things.