Cassandra anyone?

Been some time after my last post. Blogging has not been an option for the past few weeks with other commitments going on. But you cannot stop doing what you love the most. So here I am back again fervently nudging away on the keyboard. I was analyzing the use of Cassandra for one of the projects I am working on. This post will focus on the getting started aspect with related to Cassandra and mainly in terms of Windows because fortunately or unfortunately that is the operating system I am stuck with at the moment (no pun intended).

The latest Cassandra distribution can be downloaded at here. 

I wanted to have the ability to run Cassandra as a Windows service. Here are the steps on how to get that done;

• Download Apache commons daemon from here
• Extract the Cassandra distribution to a location of your choice
• Go into the “bin” directory of the Cassandra distribution you just extracted
• Create a folder called “daemon” inside the “bin’ directory
• Extract the Apache commons daemon distribution to a location of your choice
• Copy the “prunsrv.exe” related to your architecture (32bit/64bit) to the “daemon” directory created inside the “bin” directory in your Cassandra distribution extract
• Go into your “bin” directory via the command line and execute the following command;

 cassandra.bat install

 That’s it. Now you have Cassandra installed as a windows service which you can set to start automatically if that is what you prefer.

By default, authentication and authorization is disabled on Cassandra. Enabling this is a breeze with the Cassandra YAML configuration file. Go to the “conf” directory of your Cassandra distribution where you will find a file named as “cassandra.yaml”. Open up this with your favourite text editor (Notepad++ is awesome. Just saying). Make the following changes to the already existing configuration;

• authenticator: org.apache.cassandra.auth.PasswordAuthenticator
• authorizer: org.apache.cassandra.auth.CassandraAuthorizer

Now that you have enabled authentication and authorization, you can go in and login to Cassandra. The super user in Cassandra has the following credentials which you must use to log in;

• User name : cassandra
• Password : cassandra

To log in, let us go back to the “bin” directory of the Cassandra distribution where you will find a batch file called “cqlsh”. Bring up your command line again and execute the following command;

cqlsh -u cassandra -p cassandra

Now that you are in, if required you can go and change your super user password. One of the advantages of Cassandra for me is how similar the syntax is to relational databases. You can get up and running in a very short amount of time as the commands are very similar to traditional SQL. Cassandra has its own query language called CQL.

Next up, I was looking for a convenient GUI to interact with Cassandra. Browsing around, the most user friendly interface I found was DBeaver which you can download from here.

On the next article, I will focus on functionality with respect to Cassandra and how your thinking pattern should change when working with Cassandra as it will not be the same as how you design a traditional database schema.

Performance monitoring and profiling – Part 1

Performance monitoring and profiling are two different aspects. The former takes a more of a proactive measure whereas the latter is a reactive approach. In my experience, performance is an afterthought in most cases rather than being built into the software development life cycle. I see the same thing in most cases when it comes to security as well but let us not go there today as that lends itself to a different post.

Performance monitoring

In order to monitor your application, you first need to understand what performance aspects the end users of your application expects. This would include throughput, response time, uptime etc. After you collate these information, the next step would be to create scripts for load testing your application. Coming primarily from a Java background, I normally use JMeter which comes with a user friends UI to create load testing depending on your use case be it database, Web Service, JMS etc.

So you have your load testing scripts and it is running fine. What next?

Well now that you have the load scripts in place, the next step would be note down what parameters you should monitor. Generally you would start out monitoring the CPU and Memory usage while your load test is in progress.

When you monitor CPU, you would be looking at the CPU usage as the application is being load tested. If there is high CPU usage, you need to further drill down to find out what issue might cause this. If there is low CPU usage, then probably you can try and increase the load to see what the optimum load that uses efficient CPU.

If you are on a Windows machine, by default you will open the task manager and go into the Performance tab in order to find out what your CPU is up to. A snippet of mine is as follows;

I’m running on a Core i5 with 8GB of RAM. In the upper right hand corner you can see the CPU usage history. The four boxes here represent the four CPUs. You will get the same value if you run Runtime.getRuntime().availableProcessors() within a Java program. 

By default it only shows the total CPU time. But you can easily add the kernel CPU time by clicking onn View-> Show Kernal Times. You will then be presented with the following graphs with a red line;

The space between the red line and the green line is the user CPU time. So what is the difference between the user and the kernel CPU time you might ask.

User CPU Time : The amount of time the CPU spends on running your application code.

Kernel CPU Time : The amount of time the CPU spends on operating system related activities. For example if your application does a lot of reads/writes to the disk, you will see high kernel CPU as it requires the operating system related functionality to be invoked.

But what if you needed further information on what your CPU is doing at any given moment of time? PerfMon to the rescue. Open up a Run tab and type in perfmon and you will be presented with the following screen;

You can add any counter which you are interested in monitoring by right clicking on the graph area and selecting the Add Counters menu option. If you want to monitor the user CPU and kernel CPU, select the Performance object and under that you will find the User time and Privileged time (kernel CPU time) counters.

Another key indicator to watch out for when monitoring CPU is the CPU schedule run queue. This is where all light weight processes needed to run are queued up waiting for CPU. As a rule of thumb, if your run queue depth is more than four times the available virtual processors on your machine, then you need to investigate your application further more to see what could be done. In the era of high memory machines, people often overlook the fact of using and writing efficient algorithms and data structures as there is plenty of memory available on high end servers. But the issue here is that many a times CPU is limited. If your algorithms/data structures do not scale well with added load, you will end up over loading your CPU with the only alternative being is to scale your application into different servers. So if you are having issues with the scheduler run queue depth, it is always best to see the possibility of writing certain sections of your application code in a more efficient manner so as to utilize your CPU better. The run queue depth can also be monitored using Perfmon by selecting Add Counters->System-> Processor Queue Length.

Now that we have covered some of the important factors to look into in terms of CPU when monitoring your application, in the next post we will look at what aspects we need to consider when monitoring the memory usage.

Please do leave by your comments and suggestions which is as always much appreciated as I love learning from the experience of others which I deem as invaluable.

A little bit on the JVM and JIT

As you might be aware, the JVM(Java Virtusal Machine) is what makes it possible for Java to adhere to the write-once-run-anywhere paradigm. At its core, the JVM consists of the following components;

• Heap
• Stack
• PermGen and Method Area
• JIT Compiler
• Code cache

The heap is where memory is allocated for every new operator you use during the application code development stage. Stack will store the local variables that you will assign within the scope of a method. One thing to note is that the variables defined within the scope of a method will be removed after the completion of the method. If for example, a String is assigned within the scope of a method, and its scope is guaranteed to be of local scope, then this will be stored in the stack which would otherwise be assigned within the heap.
The PermGen space will store class and method level data as well as static variables that are defined in your application. The method area is actual an area within the PermGen space where it will store all method,field, constant pool level details of your application.

The JIT compiler and the code cache go hand in hand. The JVM at its core interprets the Java Byte Code into assembly code at runtime. Interpreting can be a slow process because the code needs to be converted from byte code to machine code at runtime every time a portion of your application code is executed. This is where the JIT compiler comes into action, with its super awesome compilation of methods which it then stores in the code cache.

The JIT compiler analyzes the application code at runtime to understand which methods can be categorized as hot methods. Hot in this context meaning code fragments that are accessed more frequently. At a very high level, what the JIT compiler does is that it will have a counter for each method executed in order to understand the frequency of its usage. When the counter reaches a defined threshold value, the method then becomes eligible to be compiled by the JIT compiler to its respective assemble code which will then be stored within the code cache. What happens is that now, whenever the JIT compiler comes across calls to those methods which were compiled and stored within the code cache, it will not try to interpret them yet again but will use the already compiled assembly code available within the code cache. This gives your application a performance boost because using the compiled code is much faster than interpreting it during runtime.

When talking about the JIT compiler, there are mainly two flavors of it which we are mostly oblivious of due to the fact of the lack of documentation around them. The two types are;

• Client
• Server

The default compiler used will defer according to the machine architecture and the JVM version (32bit or 64bit) that you are running on. Let us briefly see what each one does.

The client compiler starts compiling your byte code to assembly code at the application startup time. What this indirectly means is that your application will have a much improved startup time. But the main disadvantage this brings along with it is that your code cache will run out of memory faster. Most optimizations can be made only after your application has run for a brief period of time. But since the client compiler already took up the code cache space, you will not have space to store the assembly code for these optimizations. This is where the server cache excels.

Unlike the client compiler, the server compiler will not start compiling at the start of your application. It will allow the application code to run for some time (which is often referred to as the warm-up period) after which it will start compiling the byte code to assembly code which it will then store within the code cache.
In my next post I will discuss how we can actually mix and match the client and server compilation and also introduce you to a few more JVM flags that we seldom come across but are vital for increasing the performance of your application.

Finding the Equilibrium index of an array

I wanted to do a brain teaser today so i took up an algorithmic question to give a shot at. Now i know this question already has many answer on the internet if you do a quick search. But i wanted to try it out with the solution that came to my mind. The problem statement is as follows;

Equilibrium index of an array is an index such that the sum of elements at lower indexes is equal to the sum of elements at higher indexes. For example, in an array A:

A[0] = -7, A[1] = 1, A[2] = 5, A[3] = 2, A[4] = -4, A[5] = 3, A[6]=0

3 is an equilibrium index, because:

A[0] + A[1] + A[2] = A[4] + A[5] + A[6]

6 is also an equilibrium index, because sum of zero elements is zero, i.e., A[0] + A[1] + A[2] + A[3] + A[4] + A[5]=0

7 is not an equilibrium index, because it is not a valid index of array A.

Write a function int equilibrium(int[] arr); that given a sequence arr[] of size n, returns an equilibrium index (if any) or -1 if no equilibrium indexes exist.

The solution i came up with is as follows;

 
public static int solution(int[] A) {

  if (A == null || A.length < 3)
   throw new RuntimeException("Cannot find equilirbium");
  int pointer = 1;
  int lowerIndCount = A[0];
  int upperIndCount = 0;

  for (int i = 2; i < A.length; i++) {
   upperIndCount += A[i];
   if (lowerIndCount < 0) {
    if (upperIndCount > lowerIndCount && i != A.length - 1)
     continue;
    if (upperIndCount == lowerIndCount && i == A.length - 1)
     break;
    lowerIndCount += A[pointer];
    upperIndCount = 0;
    ++pointer;
    i = pointer;
   } else {
    if (upperIndCount > lowerIndCount) {
     lowerIndCount += A[pointer];
     upperIndCount = 0;
     ++pointer;
     i = pointer;
    }
   }
  }

  if (upperIndCount == lowerIndCount)
   return pointer;

  return -1;
 }

If this is the optimum or not im not sure. Im also not sure if this will work on very large data sets. But for the data sets i tried(both negative and positive) it worked fine. And also this has O(N) complexity as I am iterating through the array only once.

Enter the Node-Hood

So today we will explore the node neighborhood in order to see what awesome goodies lie in the node-land. Ok I am no expert artist, but I remember well when I jot down something I learn into a diagram and utter those famous words “A picture speaks a thousand words”. And of course I love to learn by adding some humor to it which again makes me remember things better. So this is my lame attempt in doing just that to introduce the world of Node to anyone who might be interested.

If you are a node pro, I will save you the time and ask you to close this tab right now… Ah so you did not close it, great, then come on in. So let us see what we will find today on node-land. Following is a picture I just drew in order to emphasize what we will focus on today.

So this is the node home as of now for me. You are greeted with a door mat which leads to the site where you can get more information on the same. And as always you can see just right to the door, there is a CD( who uses CDs nowadays, I know) which states the famous command you will encounter many a times when you are working with node.

npm install <module_name>

Add a –save to the end of the command, and it will save the module you just installed in your package.json file. More on that in the future when we discuss web development with express.

The today’s special bored in the house highlights some core modules that you will find when working with node. They include;

•  fs
• crypto
• process
• http

This is not the whole list and you will indeed find many more as we go along with the journey through nodejs.

So we have a friend inside our nodehood who is asking for a cool beer. His best friend as you can see is approaching the Node Fridge to get him one. In node land, you can ask for external elements if they are willing to expose their functionality. And this is done through the use of exports. Let us see more closely what is happening inside the Node Fridge;

 
var  drink = function(name){
 return "Cool "+name;
}

module.exports.drink = drink;

So the Node Fridge is exposing a drink function which returns any drink that is passed into it with an added coolness to the drink. Of course there is no point in the fridge having the cold drinks if no one is going to be drinking it, and hence we use a mode.exports at the end to expose this coolness to the whole world.

So his friend has to open up and drink this up. Since he requested this first from his friend, he added the following;

 
var beer = request('./nodefridge');

console.log("Chilling with a ”+beer.drink("beer");

As he is making the request for the drink from the same house itself, he can do as it has been highlighted in the above code. But if he was asking this from the backyard of the house, then he will need to request for the same as follows;

 
var beer = request('./backyard/nodefridge');

The assumption here is that the nodefridge.js is residing in the backyard directory. You can download this simple example from here and run it using node main.js.

So today we will cover how file streams work in the node world. The main point here is that node file streams work with the use of Event Emitters which is a way of notifying results as and when an event occurs. It is slightly different from callbacks where you can get partial results even if an error occurs midway through. It acts as more of a Pulish/Subscribe model as opposed to the Request/Reply  model in callbacks.

Before diving into how streaming works in Node, let us see how they actual work within this new found Event emitter paradigm. Again a little comic strip will better help you understand the interaction between the Readable Streams and Writtable Streams. And no surprises with their names where RS Dude is representing Readable Streams and WS Dude will be representing Writable Streams. Let us see how this turns out;

The story is self explanatory I believe. But if you want to go all technical, here is it in technical terms.

·         The read stream will pipe itself to the writable stream where by the writable stream will it’s write method internally.

·         As it keeps on writing, the write() method can send false which means it cannot accept any more data for the moment. At this moment, the readable stream will call it’s pause() method.

·         After the writable stream is ready again, it will call it’s drain() method and in turn the readable stream will call it’s resume() method and the data will again start flowing.

·         When the end of the stream is reached by the readable stream, it will call it’s end() method and the writable stream will call it’s end() method as well and close the stream.

I have included a sample on using readable streams which is available in the same download I mentioned earlier. You can run the examples as follows;

node readable.js

node wriestream.js

Note that before you run this, please run npm install request as this example requires this module.

Templating with KnockoutJS

Another nice feature of KnockoutJS is the ability to have templates to display your data. For instance, say that if you have a collection of data and you want them to be displayed in a particular format without having to do much DOM manipulation. This can be achieved in two ways when it comes to KnockoutJS;

• Using jQuery based templating
• Using native knockout templating

In this tutorial i will show you both the ways and you will see why you would always want to use the native KnockoutJS templating given its unobtrusive nature.

So first off let us look at the example below;

 
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
 <title>Superhero Registration</title>
</head>

<body>

 <div data-bind="template:'superHeroTemplate'">
  
  
 </div>


<script src="http://code.jquery.com/jquery-1.10.1.min.js"></script>
<script src="jquery-tmpl.js"></script>
<script type="text/javascript" src="http://cdnjs.cloudflare.com/ajax/libs/knockout/2.3.0/knockout-min.js"></script>

 <script type="text/javascript">
 
 $(function(){
   
   var data = [
    new superHero('Superman'),
    new superHero('Spiderman')
   ];
   
   var viewModel = {
    superHeroes:ko.observableArray(data)
   };
   
   function superHero(name){
    return {
     name:ko.observable(name)
    };
   }
   
   ko.applyBindings(viewModel);
  }
  );
 </script>
<script id="superHeroTemplate" type="text/html">
 <ul>
  {{each superHeroes}} 
   <li>${name}</li>
  {{/each}}
  
 </ul>
</script>
</body>
</html>

Ok i am obsessed with superheroes hence the use of examples follow the same pattern. Here what i am doing is constructing an observable array of super heroes.

 
function superHero(name){
    return {
     name:ko.observable(name)
    };
   }

The above function wraps the passed in input parameter to an observable object and passes it back. This is not required, but i have done as such so that Knockout will manage not only the observable array, but also the elements within the array since these are now observable objects themselves.

Here i am using jQuery templating and as such you need one additional script file to be included which you can dowload from here. Then you need to include it on your path in order for this example to work. I have saved this script file as jquery-tmpl.js. Note that the order you include these scripts are important as well. First it should be the jQuery script, then the jQuery template script and finally the KnockoutJS script.

First in the root element we need to define the template we are going to use. This is done as follows;

 
<div data-bind="template:'superHeroTemplate'">

Here you can see in the data-bind attribute i have defined the template as superHeroTemplate. Now this name is the name i have used as the id attribute within the template script i have used below;

 
<script id="superHeroTemplate" type="text/html">
 <ul>
  {{each superHeroes}} 
   <li>${name}</li>
  {{/each}}
  
 </ul>
</script>

Note that i have specified the type as text/html here since i do not want the browser to interpret this script tag since we are using this for the sole purpose of templating. The superHeroes attribute i have defined within the each tag is the name in my viewModel which is shown below. Also the ${name} attribute is what i am returning from the superHero function that returns obserable objects when passed in the name of the super hero.

 
 var viewModel = {
    superHeroes:ko.observableArray(data)
   };

This is all that is required for this to work. One thing you might have noticed is that this is a bit too obtrusive in terms of using additional DOM and script tags to make this work. This looks more of a hack since we are specifying script tags with a type calls text/html which is not valid just to make the templating work. This is where the native Knockout templating comes in handy. Let us look at how you can do the same using that and you will see why that is much cleaner.

 
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
 <title>Superhero Registration</title>
</head>

<body>

 <div>
  <ul>
   <!-- ko foreach:superHeroes -->
    <li data-bind="text: $data.name"></li>
   <!--/ko-->
  </ul>
  
 </div>


<script src="http://code.jquery.com/jquery-1.10.1.min.js"></script>
<script src="jquery-tmpl.js"></script>
<script type="text/javascript" src="http://cdnjs.cloudflare.com/ajax/libs/knockout/2.3.0/knockout-min.js"></script>

 <script type="text/javascript">
 
 $(function(){
   
   var data = [
    new superHero('Superman'),
    new superHero('Spiderman')
   ];
   
   var viewModel = {
    superHeroes:ko.observableArray(data)
   };
   
   function superHero(name){
    return {
     name:ko.observable(name)
    };
   }
   
   ko.applyBindings(viewModel);
  }
  );
 </script>

</body>
</html>

As you can see, no script tags were used. You directly work on the DOM and only need to add HTML comments block and write your code within that. You can even get rid of writing it within comments and directly embed the foreach loop within the DOM element as follows;

 
<div>
  <ul data-bind="foreach:superHeroes">
   <li data-bind="text:$data.name"></li>
  </ul>
  
 </div>

Much cleaner isnt it? No more hacks, just direct DOM manipulation. That is the power this library gives you. You can write it according to what you prefer.

So that ends my article on how to use templating with KnockoutJS. If you have any queries or feedback, please do leave by a comment which is as always highly appreciated.

Have a nice day everyone!!

Getting knocked out

Everyone is saying JavaScript is the future and i was wondering what all the hype was. Was never a big fan of JavaScript to begin with since i was involved mainly with writing back-end code. But i am always open for new things and being the enthusiast i am, thought to try out one of the plethora of JavaScript libraries out there.

Since MVC knowledge is inherent in anyone who would have written any serious web application, i wanted to try out something closer to my comfort zone. This is where i stumbled upon knockoutjs . It advocates more of a MVVM(Model-View-View-Model) architectural pattern which i believe is a very handy way to handle your front-end logic with a clear layer of separation.

I remember the code we wrote prior to these libraries where we used a separate JSON array for each page to hold all the elements in that page and do all validations based on the attributes we define on each JSON attribute.

With Knockout, this is all handled by the library. It handles the data bindings to the elements, changes to the elements as well dependency tracking. I did some reconnaissance of my own on the library and this article is just an introduction to what you can achieve with this library.

The code is simple, i enter my first name and last name and i have a separate field that displays the full name by concatenating the first name and the last name. Then i have a simple button to generate the JSON string which we could use to send the data back to the server.

Let us look at the code;

 
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
 <title>Personal Information</title>
</head>

<body>

 <div>
  <table>
   <tr>
    <td>First Name : </td>
    <td><input type="text" name="txtName" id="txtName" data-bind="value:firstname,valueUpdate:'afterkeydown'"></td>
   </tr>
   
   <tr>
    <td>Last Name : </td>
    <td><input type="text" name="txtLName" id="txtLName" data-bind="value:lastname,valueUpdate:'afterkeydown'"></td>
   </tr>
   
   <tr>
    <td colspan="2"><span id="lblFullName" data-bind="text:fullname"></span></td>
   </tr>
   
   <tr>
    <td colspan="2"><button data-bind="click: generateJSON">Generate JSON String</button></td>
   </tr>
  </table>
 </div>

<script src="http://code.jquery.com/jquery-1.10.1.min.js"></script>
<script type="text/javascript" src="http://cdnjs.cloudflare.com/ajax/libs/knockout/2.3.0/knockout-min.js"></script>

 <script type="text/javascript">
 
 $(function(){
   var person = {
    firstname:ko.observable(''),
    lastname:ko.observable(''),
    generateJSON:function(){
     alert(ko.toJSON(person));
    }
   };
   
   person.fullname = ko.computed(function(){
    
    return "The full name is : "+this.firstname()+" "+this.lastname();
   },person);
  
   ko.applyBindings(person);
  
  }
  );
 </script>
</body>
</html>

As you can see from the code this is just a normal HTML mark-up. The magic happens within the code between the script tags. For ease of use of this code i have included the knockout and jquery scripts directly from the CDN locations because otherwise you would have to manually download them and include the scripts.

In Knockout, everything revolves around your Model. This is the main point of interaction in terms of Knockout. Here i have defined by model as person. One thing to note is that i have defined the firstname and lastname properties as observable. In Knockout, observables are JavaScript functions since not all browsers support getters/setters.

Then i need to tell how to map each property in my model to the DOM elements. This is done through the data-bind attribute. As you can see, in each input element, i have defined the data-bind attribute and stated that the value property should be equal to the value of the property in my Model.

Knockout also has a concept called computed observables which was previously called Dependent-Observables. What this does is that it allows me to aggregate any fields within the model and show them as part of the result as well and any change in the properties will be reflected in the computed observable variable as well. You can see that if you keep typing values to the firstname and last name input boxes, the full name string also changes accordingly. I have specified another value in the data-bind attribute as valueUpdate:'afterkeydown' which allows the changes typed into the input boxes to be reflected as the user types in which otherwise would only have reflected if the user tabs into the fields.

To bind our Model to the Knockout library, we call ko.applyBindings passing in our Model. There are helper methods that allow us to generate a JSON string of our model which is what i have done in the click event function defined within my Model.

I hope this introduction would get you interested to try this library out if you have not already. I will follow up this post with more advanced topics such as templating, observable arrays, parent binding context etc..

As always your comments and suggestions are most welcome.