File | Directory Monitoring Service

Log monitoring is prime requirement for any batch execution application. Often batch process get hung an we are not able to check if this happened.

To achieve this we need to have some sort of File/Directory Change Notification service is required, which should be able to detect any changes. One way of doing is to poll for file system but this is an in-efficient way of doing so.
One way to do so is to poll the file system looking for changes, but this approach is inefficient. It does not scale to applications that have hundreds of open files or directories to monitor.

Java provides java.nio.file package provides a file notification API called watch service API. This api provides a mechanism to register a directory with the watch service. While registering, you tell the service which events you are interested in:
  • File Creation
  • File deletion
  • File Modification
When Watchservice finds any such event it forwards to the regitered process.

Implementation of a simple DirectoryMonitor is as follows:
public class DirectoryWatcher {

 public static void doMonitor(Path path) {
  
  try {
   Boolean isFolder = (Boolean) Files.getAttribute(path,
     "basic:isDirectory", NOFOLLOW_LINKS);
   if (!isFolder) {
    throw new IllegalArgumentException("Path: " + path + " is not a folder");
   }
  } catch (IOException ioe) {
   // Folder does not exists
   ioe.printStackTrace();
  }
  
  System.out.println("Watching path: " + path);
  
  // We obtain the file system of the Path
  FileSystem fs = path.getFileSystem ();
  
  // We create the new WatchService using the new try() block
  try(WatchService service = fs.newWatchService()) {
   
   // We register the path to the service
   // We watch for creation events
   path.register(service, ENTRY_CREATE);
   
   // Start the infinite polling loop
   WatchKey key = null;
   while(true) {
    key = service.take();
    
    // Dequeueing events
    Kind kind = null;
    for(WatchEvent watchEvent : key.pollEvents()) {
     // Get the type of the event
     kind = watchEvent.kind();
     if (OVERFLOW == kind) {
      continue; //loop
     } else if (ENTRY_CREATE == kind) {
      // A new Path was created 
      Path newPath = ((WatchEvent) watchEvent).context();
      // Output
      System.out.println("New path created: " + newPath);
     }
    }
    
    if(!key.reset()) {
     break; //loop
    }
   }
   
  } catch(IOException ioe) {
   ioe.printStackTrace();
  } catch(InterruptedException ie) {
   ie.printStackTrace();
  }
  
 }

 public static void main(String[] args) throws IOException,
   InterruptedException {
  // Folder we are going to watch
  Path folder = Paths.get(System.getProperty("user.home"));
  doMontor(folder);
 }
}

Output

Watching path: /home/ram
New path created: a




Machine Learning | Introduction - 1


Machine Learning deals with the set of algorithm which can learn a pattern in different types of user beahviour and help developer to solve various problems:


  • Collaborative Filtering (Basket Analysis)
  • Classification (Auto Tagging, Spam Filtering)
  • Prediction
  • Pattern recognition (Clustering)
  • Text extraction (Named Entity Recognition)
We can distribute Machine Learning Algorithm in tow categories

  1. Supervised Learning
  2. Unsupervised Learning

Supervised Learning

In simple words Supervised learning means Learn By Examples. We have  prior knowledge about certain domain. Using Supervised Learning we can create a model from existing knowledge which helps us to classify new data. For example:

Spam Classification

We mark existing text as HAM or SPAM and generate a model using this model new text can be categorised in one of the category.

Un-Supervised Learning 

In unsupervised Learning we do not have any supervisor, rather algorithm tries to learn a mapping from existing data and applies to new data. For example:

Document Clustering

In document clustering we try to group similar documents like we have different types are getting posted to some site using this algorithm we can group them in politics, business, technology etc categories.

List of Algorithms

  • Classification
    • Logistic Regression (SGD)
    • Bayesian
    • Support Vector Machines (SVM) 
    • Hidden Markov Model
  • Clustering
    • Canopy Clustering 
    • K-Means Clustering 
    • Fuzzy K-Means 
    • Expectation Maximization 
    • Mean Shift Clustering 
    • Hierarchical Clustering 
    • Dirichlet Process Clustering 
    • Latent Dirichlet Allocation 
    • Spectral Clustering 
    • Minhash Clustering 
    • Top Down Clustering

Tools/Libraries

Mongo Cheat Sheet

These days NoSQL concept is hot. I am doing some POC on mongodb.

I am using java driver to access mongodb, to use same DAO pattern and ORM like interface I am using mongo-morphia library.

Quick access to mongo commands, you can go through mongo cheat sheet

JSP 2.0 EL and Functions

In my last project I need to call some utility methods from my JSP to manipulate model data. I didn't wanted to use JSP scriplets within the JSP, so the solution was to use utility method via JSP2.0 EL feature.

Approach I folowed was:
1. Declare function in a tag library descriptor.




xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee
http://java.sun.com/xml/ns/j2ee/web-jsptaglibrary_2_0.xsd"
version="2.0">

Image Utility functions
Image Utility Tag Library
1.0
ImageUtility

getImagePath
com.til.hk.utils.CommonUtils
java.lang.String getImagePath(java.lang.String, java.lang.String)


getRelativeImagePath
com.til.hk.utils.CommonUtils
java.lang.String getImagePath(java.lang.String)




2. Store TLD under WEB-INF directory (/WEB-INF/tld/)
3. Specify the function's prefix and the library's Uniform Resource Identifier (URI) in JSP file

<%@ taglib uri="/WEB-INF/tld/ImageManip.tld" prefix="image"%>

Using Alert conditionally

In my last project I faced a lot of Bugs filed by QC team related to test Alert pop-ups coming which were introduced by developers for debugging and they forgot to remove those.

To get rid of this I used a wrapper function over alert() function which checks the value of a test variable if that is true then only alert() function will get called, this way I was able to control test alert pop-ups across the web application with single test variable.


DEBUG = false;
function prompt(text) {
if(DEBUG) {
alert(text);
}
}

Message Property Like functionality in Javascript

In my last project I was using JavaScript heavily which was using messages extensively.

Repetition of same kind of messages from multiple places was becoming unmanageable to handle that I have written a JavaScript class which provides Text message with respect to the message code provided.

This function works in two ways:
1. With code only (Messages.getText("error.text"))
2. With code and args (Messages.getText("error.invalid",args))
Here args is replacement for place holders like:
error.invalid = "Invalid input [1] for [0]"
So Invalid error message can be reused for multiple places where [1] will replace by value and [0] wil be replaced for Field name

For function overloading I have used addMethod - By John Resig as reference


function addMethod(object, name, fn){
var old = object[ name ];
if ( old )
object[ name ] = function(){
if ( fn.length == arguments.length )
return fn.apply( this, arguments );
else if ( typeof old == 'function' )
return old.apply( this, arguments );
};
else
object[ name ] = fn;
}

function MessageProvider() {
this.msg = new Object();
this.msg['error.Login'] = 'Please login to submit link';
this.msg['error.invalid'] = 'Invalid input [1] for [0] [0] is invalid';

addMethod(this, "getText", function(code){
return this.msg[code];
});
addMethod(this, "getText", function(code,args){
var text = this.msg[code];
for (var i = 0; i < args.length; i++) {
var re = new RegExp('\\[' + i + '\\]', 'g');
var dest = args[i];
//text = oldtext.replace(re,'’);
text = text.replace(re,dest);
}
return text;

});

}

var Messages = new MessageProvider();


Now to use centralized messaging in javascript you need to add this code to .js File and include in your html
Add all your messages to msg Object

like :



Conflicts faced While Load Testing

I was working on load testing of a web application last month, results were very conflicting when I was accessing application from browser response was very good but while executing script from Jmeter throughput was very less.

I had applied all possible optimizations like :
1. Query Caching
2. Output compression (mod_deflate)
3. Minify JS and CSS

but result was same.
I enabled logs to see compression ratio and realized that Jmeter was not sending accept-encoding HTTP header so Apache was not sending compressed result and that was affecting throughput.
After adding accept-encoding header throughput was as desired.