My personal webspace ! https://sathiyapk.github.io Spark New Optimization Rule - ReplaceExceptWithNotFilter Spark community decided to replace Except logical operator using left anti-join in SPARK-12660. It facilitates to take advantage of all the benefits of the join operations such as managed memory, code generation and broadcast joins, cc. SPARK-12660. Except is one of the mostly used... Mon, 18 Sep 2017 00:00:00 -0700 https://sathiyapk.github.io/2017-09-18-spark-replace-except-with-filter/ https://sathiyapk.github.io/2017-09-18-spark-replace-except-with-filter/ Spark Adding Custom Optimization Rules One of the main benefits of spark-sql as mentioned in their sigmod paper is its ability to easily define and plug in user defined adhoc rules for better optimization. Spark-sql provides api for adding set of adhoc rules that can be plugged into the query... Sat, 16 Sep 2017 00:00:00 -0700 https://sathiyapk.github.io/2017-09-16-spark-custom-optimization/ https://sathiyapk.github.io/2017-09-16-spark-custom-optimization/ Spark Catalyst Internals Spark catalyst is one of the secret sauce of Spark’s Operations on the structured data. Let’s take a deep look into its internals. TreeNode Abstraction TreeNode is the fundamental data type abstraction for the catalyst internals. This abstraction brings methods (such as foreach, map, flatmap, collect etc.)... Fri, 15 Sep 2017 00:00:00 -0700 https://sathiyapk.github.io/2017-09-15-spark-catalyst-internals/ https://sathiyapk.github.io/2017-09-15-spark-catalyst-internals/ Distributed Hash Table (DHT) traditionally found enough applicabilities in decentralized distributed systems. In DHT data are distributed among several nodes in the system via hashing techniques. Currently, DHT shows increasing popularity in the modern storage systems aka NoSql Systems. This post reveals some of the uses and applications of DHT in today's modern... Sat, 06 Dec 2014 00:00:00 -0800 https://sathiyapk.github.io/2014-12-06-distributed-hash-table/ https://sathiyapk.github.io/2014-12-06-distributed-hash-table/ Colum-Family data stores are a special class of NoSql system that facilitates to store wide range of data types keeping the design in between traditional Relational Database System and Modern Key-Value stores. Cassandra is a Column-Family data store on its underlying peer-to-peer architecture. One of the important features of any data... Sat, 25 Oct 2014 00:00:00 -0700 https://sathiyapk.github.io/2014-10-25-cassandra-cql-client-limitations/ https://sathiyapk.github.io/2014-10-25-cassandra-cql-client-limitations/ The term BigData is buzzing everywhere in every sectors from past few years. One of the the joint term that is buzzed along with the BigData is NoSql. NoSql is considered to be the solution for the challenges introduced by BigData. So what is meant by NoSql..? Actually, nobody is sure about its... Sun, 12 Oct 2014 00:00:00 -0700 https://sathiyapk.github.io/2014-10-12-nosql-movement/ https://sathiyapk.github.io/2014-10-12-nosql-movement/ Hi Forks, Today in this post am gonna tell how to open Cassandra source in Netbeans.. It’s been so long, since i started working on Cassandra till date whenever i tend to meet some students who newly started working on Cassandra, the common question which i came across is “How you managed to run... Mon, 06 Oct 2014 00:00:00 -0700 https://sathiyapk.github.io/2014-10-06-running-cassandra-netbeans/ https://sathiyapk.github.io/2014-10-06-running-cassandra-netbeans/ We are currently in an Era of Data Explosion..! Image Source: http://visual.ly/data-never-sleeps-20 So what this era all about, what all the potential causes for these, what are its consequences..? I was speculating it was due to technological outgrowth or change in our life style. One of my friend threw me some examples... Sat, 27 Sep 2014 00:00:00 -0700 https://sathiyapk.github.io/2014-09-27-an-era-of-data-explosion/ https://sathiyapk.github.io/2014-09-27-an-era-of-data-explosion/