Understanding the Battle of Database Storage: Row-Oriented vs. ColumnarData Science by Sunny Srinidhi - March 8, 2024March 8, 20241 In the realm of database storage, row-wise and columnar approaches stand as stalwarts with distinct advantages. Row-wise storage excels in transactional operations, ensuring data integrity with simplicity. Conversely, columnar storage revolutionizes analytical querying, leveraging vertical organization for rapid attribute retrieval. Understanding their nuances is pivotal in crafting efficient, tailored database solutions for diverse data-driven needs.
Connect Apache Spark with MongoDB database using the mongo-spark-connectorData ScienceTech by Sunny Srinidhi - April 3, 2019February 28, 20200 A couple of days back, we saw how we can connect Apache Spark to an Apache HBase database and query the data from a table using a catalog. Today, we’ll see how we can connect Apache Spark to a MongoDB database and get data directly into Spark from there. MongoDB provides us a plugin called the mongo-spark-connector, which will help us connect MongoDB and Spark without any drama at all. We just need to provide the MongoDB connection URI in the SparkConf object, and create a ReadConfig object specifying the collection name. It might sound complicated right now, but once you look at the code, you’ll understand how extremely easy this is. So, let’s look at an example. The Dataset Before we look
Connect Apache Spark to your HBase database (Spark-HBase Connector)Data ScienceTech by Sunny Srinidhi - April 1, 2019January 31, 20202 There will be times when you’ll need the data in your HBase database to be brought into Apache Spark for processing. Usually, you’ll query the database, get the data in whatever format you fancy, and then load that into Spark, maybe using the `parallelize()`function. This works, just fine. But depending on the size of the data, this could cause delays. At least it did for our application. So after some research, we stumbled upon a Spark-HBase connector in Hortonworks repository. Now, what is this connector and why should you be considering this? The Spark-HBase Connector (shc-core) The SHC is a tool provided by Hortonworks to connect your HBase database to Apache Spark so that you can tell your Spark context to pickup the
Real-Time Data Processing: Understanding the What, Why, Where, Who, and HowData ScienceTech by Sunny Srinidhi - October 22, 20240 In today’s data-driven world, businesses and organizations are continuously generating massive amounts of data. While processing data in batch mode remains useful, the need for instant decision-making has led to an increasing focus on real-time data processing. This article delves into what real-time data processing is, why it's essential, its various applications, the tools used to achieve it, trends shaping its evolution, and real-world use cases. What is Real-Time Data Processing? Real-time data processing refers to the capability to continuously ingest, process, and output data as soon as it is generated, with minimal latency. Unlike batch processing, which collects and processes data in large groups at set intervals (e.g., daily or hourly), real-time processing works with data immediately as it becomes available,
Exploring the Inner Workings of Google BigQuery: A Deep Dive into Design, Competitors, Use Cases, and Pros/ConsData Science by Sunny Srinidhi - March 13, 2024March 13, 20240 Discover the inner workings of Google BigQuery, a game-changer in big data analytics. Unravel its architecture, including the prowess of its distributed query engine, Dremel, and the innovative Capacitor technology. Compare it with competitors, explore diverse use cases from real-time analytics to healthcare, and weigh its pros and cons. Join us on a journey into the heart of data analytics excellence.
Getting Started With Apache AirflowData Science by Sunny Srinidhi - October 11, 2021October 11, 20210 I recently started working with Apache Airflow. And as is tradition, I’m telling you everything about it here.
Querying Hive Tables From a Spring Boot AppData ScienceTech by Sunny Srinidhi - June 30, 2021June 30, 20211 In this post, we’ll see how to connect to a Hive database and run queries on that database from a Spring Boot application.
Why caching is important to improve your system’s performanceTech by Sunny Srinidhi - May 5, 2020December 18, 20210 We often hear about caching data on servers, but exactly does that mean? And how do we cache data? What are the different approaches?
Explore your Amazon S3 data online using FilestashTech by Sunny Srinidhi - April 29, 2020April 29, 20200 Filestash is a very handy tool in your browser which helps you nativage your S3 buckets and folders easily, and even edit files online.
Redundancy in a distributed systemTech by Sunny Srinidhi - April 13, 20200 We don’t think of introducting redundancy in our systems early enough. This is bad. Let’s see why with an example, and how to fix it.