Wednesday, July 23, 2008

Maven crash course in Singe HTML page.

Maven Tutorial

 
Maven is a powerful build tool for Java software projects. Actually, you can build software projects using other languages too, but Maven is developed in Java, and is thus historically used more for Java projects.
The purpose of this Maven tutorial is to make you understand how Maven works. Therefore this tutorial focuses on the core concepts of Maven. Once you understand the core concepts, it is much easier to lookup the fine detail in the Maven documentation, or search for it on the internet.
Actually, the Maven developers claim that Maven is more than just a build tool. You can read what they believe it is, in their document Philosophy of Maven. But for now, just think of it as a build tool. You will find out what Maven really is, once you understand it and start using it.

Maven Version

The first version of this Maven tutorial is based on Maven 3.0.5. However, this tutorial has been updated in several places since the first version of this tutorial. The updates were tested with Maven 3.3.3.

Maven Website

The Maven website is located here:
From this website you can download the latest version of Maven and follow the project in general.

What is a Build Tool?

A build tool is a tool that automates everything related to building the software project. Building a software project typically includes one or more of these activities:
  • Generating source code (if auto-generated code is used in the project).
  • Generating documentation from the source code.
  • Compiling source code.
  • Packaging compiled code into JAR files or ZIP files.
  • Installing the packaged code on a server, in a repository or somewhere else.
Any given software project may have more activities than these needed to build the finished software. Such activities can normally be plugged into a build tool, so these activities can be automated too.
The advantage of automating the build process is that you minimize the risk of humans making errors while building the software manually. Additionally, an automated build tool is typically faster than a human performing the same steps manually.

Installing Maven

To install Maven on your own system (computer), go to the Maven download page and follow the instructions there. In summary, what you need to do is:
  1. Set the JAVA_HOME environment variable to point to a valid Java SDK (e.g. Java 8).
  2. Download and unzip Maven.
  3. Set the M2_HOME environment variable to point to the directory you unzipped Maven to.
  4. Set the M2 environment variable to point to M2_HOME/bin (%M2_HOME%\bin on Windows,$M2_HOME/bin on unix).
  5. Add M2 to the PATH environment variable (%M2% on Windows, $M2 on unix).
  6. Open a command prompt and type 'mvn -version' (without quotes) and press enter.
After typing in the mvn -version command you should be able to see Maven execute, and the version number of Maven written out to the command prompt.
Note: Maven uses Java when executing, so you need Java installed too (and the JAVA_HOME environment variable set as explained above). Maven 3.0.5 needs a Java version 1.5 or later. I use Maven 3.3.3 with Java 8 (u45).
I have a tutorial about installing the Java SDK in case you are not familiar with that. Remember, it has to be an SDK (Software Developer Kit), not just a JRE (Java Runtime Environment). The JRE does not contain a Java compiler. Only the SDK does.

Maven Overview - Core Concepts

Maven is centered around the concept of POM files (Project Object Model). A POM file is an XML representation of project resources like source code, test code, dependencies (external JARs used) etc. The POM contains references to all of these resources. The POM file should be located in the root directory of the project it belongs to.
Here is a diagram illustrating how Maven uses the POM file, and what the POM file primarily contains:
Overview of Maven core concepts.
Overview of Maven core concepts.
These concepts are explained briefly below to give you an overview, and then in more detail in their own sections later in this tutorial.
POM Files
When you execute a Maven command you give Maven a POM file to execute the commands on. Maven will then execute the command on the resources described in the POM.
Build Life Cycles, Phases and Goals
The build process in Maven is split up into build life cycles, phases and goals. A build life cycle consists of a sequence of build phases, and each build phase consists of a sequence of goals. When you run Maven you pass a command to Maven. This command is the name of a build life cycle, phase or goal. If a life cycle is requested executed, all build phases in that life cycle are executed. If a build phase is requested executed, all build phases before it in the pre-defined sequence of build phases are executed too.
Dependencies and Repositories
One of the first goals Maven executes is to check the dependencies needed by your project. Dependencies are external JAR files (Java libraries) that your project uses. If the dependencies are not found in the local Maven repository, Maven downloads them from a central Maven repository and puts them in your local repository. The local repository is just a directory on your computer's hard disk. You can specify where the local repository should be located if you want to (I do). You can also specify which remote repository to use for downloading dependencies. All this will be explained in more detail later in this tutorial.
Build Plugins
Build plugins are used to insert extra goals into a build phase. If you need to perform a set of actions for your project which are not covered by the standard Maven build phases and goals, you can add a plugin to the POM file. Maven has some standard plugins you can use, and you can also implement your own in Java if you need to.
Build Profiles
Build profiles are used if you need to build your project in different ways. For instance, you may need to build your project for your local computer, for development and test. And you may need to build it for deployment on your production environment. These two builds may be different. To enable different builds you can add different build profiles to your POM files. When executing Maven you can tell which build profile to use.

Maven vs. Ant

Ant is another popular build tool by Apache. If you are used to Ant and you are trying to learn Maven, you will notice a difference in the approach of the two projects.
Ant uses an imperative approach, meaning you specify in the Ant build file what actions Ant should take. You can specify low level actions like copying files, compiling code etc. You specify the actions, and you also specify the sequence in which they are carried out. Ant has no default directory layout.
Maven uses a more declarative approach, meaning that you specify in the Maven POM file what to build, but now how to build it. The POM file describes your project resources - not how to build it. Contrarily, an Ant file describes how to build your project. In Maven, how to build your project is predefined in the Maven Build Life Cycles, Phases and Goals.

Maven POM Files

A Maven POM file (Project Object Model) is an XML file that describe the resources of the project. This includes the directories where the source code, test source etc. is located in, what external dependencies (JAR files) your projects has etc.
The POM file describes what to build, but most often not how to build it. How to build it is up to the Maven build phases and goals. You can insert custom actions (goals) into the Maven build phase if you need to, though.
Each project has a POM file. The POM file is named pom.xml and should be located in the root directory of your project. A project divided into subprojects will typically have one POM file for the parent project, and one POM file for each subproject. This structure allows both the total project to be built in one step, or any of the subprojects to be built separately.
Throughout the rest of this section I will describe the most important parts of the POM file. For a full reference of the POM file, see the Maven POM Reference.
Here is a minimal POM file:

    4.0.0

    com.jenkov
    java-web-crawler
    1.0.0

The modelVersion element sets what version of the POM model you are using. Use the one matching the Maven version you are using. Version 4.0.0 matches Maven version 2 and 3.
The groupId element is a unique ID for an organization, or a project (an open source project, for instance). Most often you will use a group ID which is similar to the root Java package name of the project. For instance, for my Java Web Crawler project I may choose the group ID com.jenkov. If the project was an open source project with many independent contributors, perhaps it would make more sense to use a group ID related to the project than an a group ID related to my company. Thus, com.javawebcrawlercould be used.
The group ID does not have to be a Java package name, and does not need to use the . notation (dot notation) for separating words in the ID. But, if you do, the project will be located in the Maven repository under a directory structure matching the group ID. Each . is replaced with a directory separator, and each word thus represents a directory. The group ID com.jenkov would then be located in a directory calledMAVEN_REPO/com/jenkov. The MAVEN_REPO part of the directory name will be replaced with the directory path of the Maven repository.
The artifactId element contains the name of the project you are building. In the case of my Java Web Crawler project, the artifact ID would be java-web-crawler. The artifact ID is used as name for a subdirectory under the group ID directory in the Maven repository. The artifact ID is also used as part of the name of the JAR file produced when building the project. The output of the build process, the build result that is, is called an artifact in Maven. Most often it is a JAR, WAR or EAR file, but it could also be something else.
The versionId element contains the version number of the project. If your project has been released in different versions, for instance an open source API, then it is useful to version the builds. That way users of your project can refer to a specific version of your project. The version number is used as a name for a subdirectory under the artifact ID directory. The version number is also used as part of the name of the artifact built.
The above groupIdartifactId and version elements would result in a JAR file being built and put into the local Maven repository at the following path (directory and file name):
MAVEN_REPO/com/jenkov/java-web-crawler/1.0.0/java-web-crawler-1.0.0.jar
If your project uses the Maven directory structure, and your project has no external dependencies, then the above minimal POM file is all you need to build your project.
If your project does not follow the standard directory structure, has external dependencies, or need special actions during building, you will need to add more elements to the POM file. These elements are listed in the Maven POM reference (see link above).
In general you can specify a lot of things in the POM which gives Maven more details about how to build your projects. See the Maven POM reference for more information about what can be specified.

Super POM

All Maven POM files inherit from a super POM. If no super POM is specified, the POM file inherits from the base POM. Here is a diagram illustrating that:
Super POM and POM inheritance.
Super POM and POM inheritance.
You can make a POM file explicitly inherit from another POM file. That way you can change the settings across all inheriting POM's via their common super POM. You specify the super POM at the top of a POM file like this:

    4.0.0
    
        
        org.codehaus.mojo
        my-parent
        2.0
        ../my-parent
        
    

    my-project
    ...

An inheriting POM file may override settings from a super POM. Just specify new settings in the inheriting POM file.
POM inheritance is also covered in more detail in the Maven POM reference.

Effective POM

With all this POM inheritance it may be hard to know what the total POM file looks like when Maven executes. The total POM file (result of all inheritance) is called the effective POM. You can get Maven to show you the effective POM using this command:
mvn help:effective-pom
This command will make Maven write out the effective POM to the command line prompt.

Maven Settings File

Maven has two settings files. In the settings files you can configure settings for Maven across all Maven POM files. For instance, you can configure:
  • Location of local repository
  • Active build profile
The settings files are called settings.xml. The two settings files are located at:
  • The Maven installation directory: $M2_HOME/conf/settings.xml
  • The user's home directory: ${user.home}/.m2/settings.xml
Both files are optional. If both files are present, the values in the user home settings file overrides the values in the Maven installation settings file.
You can read more about the Maven settings files in the Maven Settings Reference.

Running Maven

When you have installed Maven and have created a POM file and put the POM file in the root directory of your project, you can run Maven on your project.
Running Maven is done by executing the mvn command from a command prompt. When executing themvn command you pass the name of a build life cycle, phase or goal to it, which Maven then executes. Here is an example:
mvn install
This command executes the build phase called install (part of the default build life cycle), which builds the project and copies the packaged JAR file into the local Maven repository. Actually, this command executes all build phases before install in the build phase sequence, before executing theinstall build phase.
You can execute multiple build life cycles or phases by passing more than one argument to the mvncommand. Here is an example:
 mvn clean install
This command first executes the clean build life cycle, which removes compiled classes from the Maven output directory, and then it executes the install build phase.
You can also execute a Maven goal (a subpart of a build phase) by passing the build phase and goal name concatenated with a : in between, as parameter to the Maven command. Here is an example:
mvn dependency:copy-dependencies
This command executes the copy-dependencies goal of the dependency build phase.

Maven Directory Structure

Maven has a standard directory structure. If you follow that directory structure for your project, you do not need to specify the directories of your source code, test code etc. in your POM file.
You can see the full directory layout in the Introduction to the Maven Standard Directory Layout.
Here are the most important directories:
- src
  - main
    - java
    - resources
    - webapp
  - test
    - java
    - resources

- target
The src directory is the root directory of your source code and test code. The main directory is the root directory for source code related to the application itself (not test code). The test directory contains the test source code. The java directories under main and test contains the Java code for the application itself (under main) and the Java code for the tests (under test).
The resources directory contains other resources needed by your project. This could be property files used for internationalization of an application, or something else.
The webapp directory contains your Java web application, if your project is a web application. The webappdirectory will then be the root directory of the web application. Thus the webapp directory contains theWEB-INF directory etc.
The target directory is created by Maven. It contains all the compiled classes, JAR files etc. produced by Maven. When executing the clean build phase, it is the target directory which is cleaned.

Project Dependencies

Unless your project is small, your project may need external Java APIs or frameworks which are packaged in their own JAR files. These JAR files are needed on the classpath when you compile your project code.
Keeping your project up-to-date with the correct versions of these external JAR files can be a comprehensive task. Each external JAR may again also need other external JAR files etc. Downloading all these external dependencies (JAR files) recursively and making sure that the right versions are downloaded is cumbersome. Especially when your project grows big, and you get more and more external dependencies.
Luckily, Maven has built-in dependency management. You specify in the POM file what external libraries your project depends on, and which version, and then Maven downloads them for you and puts them in your local Maven repository. If any of these external libraries need other libraries, then these other libraries are also downloaded into your local Maven repository.
You specify your project dependencies inside the dependencies element in the POM file. Here is an example:

    4.0.0

    com.jenkov.crawler
    java-web-crawler
    1.0.0
    
      

        
          org.jsoup
          jsoup
          1.7.1
        

        
          junit
          junit
          4.8.1
          test
        

      
    

    
    


Notice the dependencies element in bold. Inside it are two dependency elements. Each dependencyelement describes an external dependency.
Each dependency is described by its groupIdartifactId and version. You may remember that this is also how you identified your own project in the beginning of the POM file. The example above needs theorg.jsoup group's jsoup artifact in version 1.7.1, and the junit group's junit artifact in version4.8.1.
When this POM file is executed by Maven, the two dependencies will be downloaded from a central Maven repository and put into your local Maven repository. If the dependencies are already found in your local repository, Maven will not download them. Only if the dependencies are missing will they be downloaded into your local repository.
Sometimes a given dependency is not available in the central Maven repository. You can then download the dependency yourself and put it into your local Maven repository. Remember to put it into a subdirectory structure matching the groupIdartifactId and version. Replace all dots (.) with / and separate thegroupIdartifactId and version with / too. Then you have your subdirectory structure.
The two dependencies downloaded by the example above will be put into the following subdirectories:
MAVEN_REPOSITORY_ROOT/junit/junit/4.8.1
MAVEN_REPOSITORY_ROOT/org/jsoup/jsoup/1.7.1

External Dependencies

An external dependency in Maven is a dependency (JAR file) which is not located in a Maven repository (neiterh local, central or remote repository). It may be located somewhere on your local hard disk, for instance in the lib directory of a webapp, or somewhere else. The word "external" thus means external to the Maven repository system - not just external to the project. Most dependencies are external to the project, but few are external to the repository system (not located in a repository).
You configure an external dependency like this:

  mydependency
  mydependency
  system
  1.0
  ${basedir}\war\WEB-INF\lib\mydependency.jar

The groupId and artifactId are both set to the name of the dependency. The name of the API used, that is. The scope element value is set to system. The systemPath element is set to point to the location of the JAR file containing the dependency. The ${basedir} points to the directory where the POM is located. The rest of the path is relative from that directory.

Snapshot Dependencies

Snapshot dependencies are dependencies (JAR files) which are under development. Instead of constantly updating the version numbers to get the latest version, you can depend on a snapshot version of the project. Snapshot versions are always downloaded into your local repository for every build, even if a matching snapshot version is already located in your local repository. Always downloading the snapshot dependencies assures that you always have the latest version in your local repository, for every build.
You can tell Maven that your project is a snapshot version simply by appending -SNAPSHOT to the version number in the beginning of the POM (where you also set the groupId and artifactId). Here is aversion element example:
1.0-SNAPSHOT
Notice the -SNAPSHOT appended to the version number.
Depending on a snapshot version is also done by appending the -SNAPSHOT after the version number when configuring dependencies. Here is an example:

    com.jenkov
    java-web-crawler
    1.0-SNAPSHOT

The -SNAPSHOT appended to the version number tells Maven that this is a snapshot version.
You can configure how often Maven shall download snapshot dependencies in the Maven Settings File.

Maven Repositories

Maven repositories are directories of packaged JAR files with extra meta data. The meta data are POM files describing the projects each packaged JAR file belongs to, including what external dependencies each packaged JAR has. It is this meta data that enables Maven to download dependencies of your dependencies recursively, until the whole tree of dependencies is download and put into your local repository.
Maven repositories are covered in more detail in the Maven Introduction to Repositories, but here is a quick overview.
Maven has three types of repository:
  • Local repository
  • Central repository
  • Remote repository
Maven searches these repositories for dependencies in the above sequence. First in the local repository, then in the central repository, and third in remote repositories if specified in the POM.
Here is a diagram illustrating the three repository types and their location:
Maven Repository Types and Location.
Maven Repository Types and Location.
Local Repository
A local repository is a directory on the developer's computer. This repository will contain all the dependencies Maven downloads. The same Maven repository is typically used for several different projects. Thus Maven only needs to download the dependencies once, even if multiple projects depends on them (e.g. Junit).
Your own projects can also be built and installed in your local repository, using the mvn installcommand. That way your other projects can use the packaged JAR files of your own projects as external dependencies by specifying them as external dependencies inside their Maven POM files.
By default Maven puts your local repository inside your user home directory on your local computer. However, you can change the location of the local repository by setting the directory inside your Maven settings file. Your Maven settings file is also located in your user-home/.m2 directory and is calledsettings.xml. Here is how you specify another location for your local repository:

    
        d:\data\java\products\maven\repository
    

Central Repository
The central Maven repository is a repository provided by the Maven community. By default Maven looks in this central repository for any dependencies needed but not found in your local repository. Maven then downloads these dependencies into your local repository. You need no special configuration to access the central repository.
Remote Repository
A remote repository is a repository on a web server from which Maven can download dependencies, just like the central repository. A remote repository can be located anywhere on the internet, or inside a local network.
A remote repository is often used for hosting projects internal to your organization, which are shared by multiple projects. For instance, a common security project might be used across multiple internal projects. This security project should not be accessible to the outside world, and should thus not be hosted in the public, central Maven repository. Instead it can be hosted in an internal remote repository.
Dependencies found in a remote repository are also downloaded and put into your local repository by Maven.
You can configure a remote repository in the POM file. Put the following XML elements right after the element:

   
       jenkov.code
       http://maven.jenkov.com/maven2/lib
   

Maven Build Life Cycles, Phases and Goals

When Maven builds a software project it follows a build life cycle. The build life cycle is divided into build phases, and the build phases are divided into build goals. Maven build life cycles, build phases and goals are described in more detail in the Maven Introduction to Build Phases, but here I will give you a quick overview.
Build Life Cycles
Maven has 3 built-in build life cycles. These are:
  1. default
  2. clean
Each of these build life cycles takes care of a different aspect of building a software project. Thus, each of these build life cycles are executed independently of each other. You can get Maven to execute more than one build life cycle, but they will be executed in sequence, separately from each other, as if you had executed two separate Maven commands.
The default life cycle handles everything related to compiling and packaging your project. The cleanlife cycle handles everything related to removing temporary files from the output directory, including generated source files, compiled classes, previous JAR files etc. The site life cycle handles everything related to generating documentation for your project. In fact, site can generate a complete website with documentation for your project.
Build Phases
Each build life cycle is divided into a sequence of build phases, and the build phases are again subdivided into goals. Thus, the total build process is a sequence of build life cycle(s), build phases and goals.
You can execute either a whole build life cycle like clean or site, a build phase like install which is part of the default build life cycle, or a build goal like dependency:copy-dependencies. Note: You cannot execute the default life cycle directly. You have to specify a build phase or goal inside thedefault life cycle.
When you execute a build phase, all build phases before that build phase in this standard phase sequence are executed. Thus, executing the install build phase really means executing all build phases before the install phase, and then execute the install phase after that.
The default life cycle is of most interest since that is what builds the code. Since you cannot execute thedefault life cycle directly, you need to execute a build phase or goal from the default life cycle. Thedefault life cycle has an extensive sequence of build phases and goals, ,so I will not describe them all here. The most commonly used build phases are:
Build PhaseDescription
validateValidates that the project is correct and all necessary information is available. This also makes sure the dependencies are downloaded.
compileCompiles the source code of the project.
testRuns the tests against the compiled source code using a suitable unit testing framework. These tests should not require the code be packaged or deployed.
packagePacks the compiled code in its distributable format, such as a JAR.
installInstall the package into the local repository, for use as a dependency in other projects locally.
deployCopies the final package to the remote repository for sharing with other developers and projects.
You execute one of these build phases by passing its name to the mvn command. Here is an example:
mvn package
This example executes the package build phase, and thus also all build phases before it in Maven's predefined build phase sequence.
If the standard Maven build phases and goals are not enough to build your project, you can create Maven plugins to add the extra build functionality you need.
Build Goals
Build goals are the finest steps in the Maven build process. A goal can be bound to one or more build phases, or to none at all. If a goal is not bound to any build phase, you can only execute it by passing the goals name to the mvn command. If a goal is bound to multiple build phases, that goal will get executed during each of the build phases it is bound to.

Maven Build Profiles

Maven build profiles enable you to build your project using different configurations. Instead of creating two separate POM files, you can just specify a profile with the different build configuration, and build your project with this build profile when needed.
You can read the full story about build profiles in the Maven POM reference under Profiles. Here I will give you a quick overview though.
Maven build profiles are specified inside the POM file, inside the profiles element. Each build profile is nested inside a profile element. Here is an example:

  4.0.0

  com.jenkov.crawler
  java-web-crawler
  1.0.0

  
      
          test
          ...
          ...
          ...
          ...
          ...
          ...
          ...
          ...
          ...
      
  



A build profile describes what changes should be made to the POM file when executing under that build profile. This could be changing the applications configuration file to use etc. The elements inside theprofile element will override the values of the elements with the same name further up in the POM.
Inside the profile element you can see a activation element. This element describes the condition that triggers this build profile to be used. One way to choose what profile is being executed is in thesettings.xml file. There you can set the active profile. Another way is to add -P profile-name to the Maven command line. See the profile documentation for more information.

Maven Plugins

Maven plugins enable you to add your own actions to the build process. You do so by creating a simple Java class that extends a special Maven class, and then create a POM for the project. The plugin should be located in its own project.
To keep this tutorial short, I will refer to the Maven Plugin Developers Centre for more information about developing plugins.

Thursday, February 28, 2008

FIX protocol messages latency calculation using awk

#  awk based script to calculate the latency of FIX protocoll messages.
#  usage: awk -f $someroot/ackdelta.awk [VERBOSE=1] [STAT=1] $someroot/FIX.log 

# Author : Sakthivel Kathirvel
# last.modified=Thursday, Mar 28, 2007

#This program is free software; you can redistribute it and/or
#modify it under the terms of the GNU General Public License
#as published by the Free Software Foundation; either version 2
#of the License, or (at your option) any later version.

#This program is distributed in the hope that it will be useful,
#but WITHOUT ANY WARRANTY; without even the implied warranty of
#MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#GNU General Public License for more details.

#You should have received a copy of the GNU General Public License
#along with this program; if not, write to the Free Software
#Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.#  


# sample output => line format => ,, ,   
#   D,AAA 0002/02222007,50,10:40:13.329,10:40:13.379
#   D,AAA 0003/02222007,47,10:51:22.845,10:51:22.892
#   D,AAA 0004/02222007,4,10:58:46.246,10:58:46.250
#   D,AAA 0005/02222007,5,11:00:31.500,11:00:31.505
#   D,AAA 0006/02222007,4,11:01:35.909,11:01:35.913

#  sample output with VERBOSE ON
#    timeline=>_Feb_22_10:44:38.431068::mesgtype=>35=D::clordid=>11=AAA 0001/02222007
#    timeline=>_Feb_22_10:44:38.452347::mesgtype=>35=8::clordid=>11=AAA 0001/02222007::exetype=>150=0::ordstatus=>39=0
#    

#  sample output with STAT ON
############### Stats ###############
#   Min delay=33 ms.
#   Max delay=211 ms.
#   Aver. delay=79 ms. of 1216 acks
#   No of ClOrdId processed=1216
#   Process Started...Thu Feb 28 08:50:17 CST 2007
#   Process Ended ...Thu Feb 28 08:50:23 CST 2007
#   Processing took 6 Secs

# Preconditions and assumptions
#   assump.1=the trade log timestamp format = Fri Feb 22 10:50:07.604341
#   assump.2= trade log format has the datestamp appears exactly 2 lines above trademesg lines containing order[35=D] and ack[35=8].
#   assump.3= trademesg lines containing FIX tags uses 'SPACE' or '^A' as line delimiter
#   assump.4=ClOrdId tag value contains space. this space should be accounted for while reading 'full' clordid value while tokenzing the lines based on space as delimiter.
#   assump.5= in tradelog files, trademesg lines containing FIX tags may appear in different order
#   assump.6= the execution report 35=8 can have ExecType 150=0 or 150=A

# last.modified=Thursday, Mar 28, 2007

# impls details
# to speed up processing associative array is used. ClordId is used as index.
# lineBuffer is flushed frequently to optimize the memory usage while processing Laaarge log flies, only last 4 lines read are kept in memory. .
# the input line is normalized to use ' ' as delimiter. ^A characters are normalized to space
# dup clordid with 35=D are scrubbed out.
# clordid with 35=D that dont have match 35=8 with 150=0 or A are aggregaged and printed in stat; STAT switch need to be enabled.
# order and ack timestamp are scaled from micro to mill secs [ rounded off to 3 digits ]
BEGIN {
  linePtr=1;
  clordidCtr=1; #unique clordid with msgtype 35=D
  timeformat = "%a %b %d %H:%M:%S %Z %Y";
  startTimeSec=systime();
  startTS=sprintf("Process Started...%s", strftime(timeformat,systime()));
}
{
  gsub(/ /, "|", $0); # handles if the line delimiter is ascii control character ^A (SOH)
  aline[linePtr++]=$0;
}

# catches (In)coming New Order[35=D] line and extracts ClOrdId fields
/.*35=D.*11=.*59=3.*/{
    mesgtype=getMsgType(aline[linePtr-1]);
    orderLineClOrdID=getClordId(aline[linePtr-1]);
    orderTimeLine=getTimeLine(aline[linePtr-1]);
    if (VERBOSE ~ /[0-9]+/) printf("orderTimeLine=>%s::mesgtype=>%s::clordid=>%s\n", orderTimeLine, mesgtype,orderLineClOrdID )
    # convert the seconds part in the orderTimeLine from microsec to millisec [ rounding to 3 digit precision ]
    inSecPart=substr(orderTimeLine, index(orderTimeLine, ".")+1)
    orderLineClOrdIDValue=substr(orderLineClOrdID, index(orderLineClOrdID, "=")+1);
    # check for duplicate ClOrdID with 35=D and take only unique ones.
    if (length(clordid[orderLineClOrdIDValue]) > 0 ) {
        dupclordid[orderLineClOrdIDValue]="";
    } else {
        clordid[orderLineClOrdIDValue]=sprintf("%s,%s%d",
                                                      substr(mesgtype,index(mesgtype,"=")+1), 
                                                      substr(orderTimeLine, 1, index(orderTimeLine, ".")), 
                                                      inSecPart);
    }
    #flushes out lineBuffer
    delete aline;linePtr=1;

}
# scans a line for execution report[35=8] that are [150=0] or [150=A]  new Orders for any of ClordId-with-[35=D] 
/.*35=8.*150=[0A].*59=3.*/ {
    ackLineClOrdID=getClordId(aline[linePtr-1]);
    ackLineClOrdIDValue=substr(ackLineClOrdID, index(ackLineClOrdID, "=")+1);
    if (length(clordid[ackLineClOrdIDValue]) > 0 ) {
        ackTimeLine=getTimeLine(aline[linePtr-1]);
        if (VERBOSE ~ /[0-9]+/) printf("ACKtimeline=>%s::mesgtype=>%s::clordid=>%s::exetype=>%s::ordstatus=>%s\n", ackTimeLine, getMsgType(aline[linePtr-1]),ackLineClOrdID, getExecType(aline[linePtr-1]),getOrderStatus(aline[linePtr-1]))
        split(clordid[ackLineClOrdIDValue], PARTS, ",");
        inTimeLineinMilSec=PARTS[2];
        # convert the seconds part in the ackTimeLine from microsec to millisecs [ rounding to 3 digit precision ]
        ackSecPart=substr(ackTimeLine, index(ackTimeLine, ".")+1);
        ackTimeLineinMilSec=sprintf("%s%d",substr(ackTimeLine, 1, index(ackTimeLine,".")), ackSecPart);
        clordid[ackLineClOrdIDValue]=sprintf("match, %s,%s,%d",
                                                    clordid[ackLineClOrdIDValue],
                                                    ackTimeLineinMilSec,
                                                    getDelta(inTimeLineinMilSec,ackTimeLineinMilSec));
                                                    
    }
    #flushes out lineBuffer
    delete aline;linePtr=1;
}

END {
  min=9999;max=-1;aver=0;
  for (assocIndex in clordid ) {
      split(clordid[assocIndex],PARTS,",");
      if ("match" ~ PARTS[1]) {
          clordidCtr++;
          deltaPart=PARTS[5];
          if (int(deltaPart) >= int(max)) {max=deltaPart; }
          if (int(deltaPart) < int(min)) {min=deltaPart;}
          aver=aver+deltaPart;
          sub(/ /,"",PARTS[2]);
          printf("%s,%s,%s,%s,%s\n", PARTS[2],assocIndex,deltaPart,PARTS[3],PARTS[4]); 
      } else {
         nomatch[assocIndex]="";
      }
  }
  
  
  if (STAT ~ /[0-9]+/) {
      print "############### Stats ###############"
      printf ("Min delay=%d ms.\n", min)
      printf ("Max delay=%d ms.\n", max)
      printf ("Aver. delay=%d ms. of %i acks\n", aver/(clordidCtr-1),(clordidCtr-1) )
      printf ("No of ClOrdId processed=%d\n",clordidCtr-1);
      print startTS 
      endTimeSec=systime();
      printf("Process Ended ...%s\n", strftime(timeformat,systime()));
      printf("Processing took %d Secs\n",  (endTimeSec-startTimeSec));
      
      for (assocIndex in nomatch) {
            print assocIndex;
      }

      for (assocIndex in dupclordid ) {
            print assocIndex;
      }
  }
}

function getTimeLine(aLine) {
    linefields = split(aLine, FIELDS, "|");
    return sprintf("%s",FIELDS[1]);
}

function getClordId(aLine) {
   linefields = split(aLine, FIELDS, "|");
   for(ctr=1; ctr<=linefields; ctr++) {
     if (FIELDS[ctr] ~ "11=") {
       return sprintf("%s %s",  FIELDS[ctr], FIELDS[ctr+1]);
     }
   }
}

function getMsgType(aLine) {
   linefields = split(aLine, FIELDS, "|");
   for(ctr=1; ctr<=linefields; ctr++) {
     if (FIELDS[ctr] ~ "35=") {
       return sprintf("%s",  FIELDS[ctr]);
     }
   }
}

function getOrderStatus(aLine) {
   linefields = split(aLine, FIELDS, "|");
   for(ctr=1; ctr<=linefields; ctr++) {
     if (FIELDS[ctr] ~ "39=") {
       return sprintf("%s",  FIELDS[ctr]);
     }
   }
}

function getExecType(aLine) {
   linefields = split(aLine, FIELDS, "|");
   for(ctr=1; ctr<=linefields; ctr++) {
     if (FIELDS[ctr] ~ "150=") {
       return sprintf("%s",  FIELDS[ctr]);
     }
   }
}

function getStats() {
# todo
#find min, max and average of the deltas
}

function getDelta(inTime,outTime) {

  split(inTime, inTimeParts, ".")
  split(outTime, outTimeParts, ".")
  split(inTimeParts[1],inTimeHMSParts, ":")
  split(outTimeParts[1],outTimeHMSParts, ":")

  inTimeMilsec=(inTimeHMSParts[1]*60*60*1000) + (inTimeHMSParts[2]*60*1000) + (inTimeHMSParts[3]*1000) + inTimeParts[2]
  outTimeMilSec=(outTimeHMSParts[1]*60*60*1000) + (outTimeHMSParts[2]*60*1000) + (outTimeHMSParts[3]*1000) + outTimeParts[2]
  return (outTimeMilSec-inTimeMilsec)
}

Saturday, July 28, 2007

Software Patterns Made Simple


Construction Types:

Abstract factory, Factory, Builder, Prototype, Singleton (5)

Adapter Types: 

Bridge, composite, deco, facade, flyweight, proxy (7)

Behavior Types:

Visitor, observer, interpreter, iterator, mediator (5)

Observer pattern:

  • There are two kinds of object Subject and Observer. Whenever there is change on subject's state observer will receive notification. Common example of observer pattern is Social Network notifications - Facebook, Linkedin - notification mechanism. Subject is you and observers are your friends. 

Execution Types:

command, chain of resp.(2)

Planning Types:
state, strategy, template (3)

  • State design pattern is used to change Object's behavior based on it’s internal state.
  • Context is the class that has a State reference to one of the concrete implementations of the State and forwards the request to the state object for processing. 

Strategy pattern:
  • Comes handy when you want to accomplish the same goal with different strategies. One good example of Strategy pattern is sorting a bunch of objects.  JDK's Collections.sort() method and Comparator interface, which is a strategy interface and defines strategy for comparing objects. Because of this pattern, we don't need to modify sort() method (closed for modification) to compare any object, at same time we can implement Comparator interface to define new comparing strategy (open for extension).
Template Pattern
Reward Types:
 momento,




Monday, July 23, 2007

Chain of Responsibility Pattern

Follow the Chain of Responsibility

Run through server-side and client-side CoR implementations

Jul 29, 2007 1:00 AM PT
I recently switched to Mac OS X from Windows and I'm thrilled with the results. But then again, I only spent a short five-year stint on Windows NT and XP; before that I was strictly a Unix developer for 15 years, mostly on Sun Microsystems machines. I also was lucky enough to develop software under Nextstep, the lush Unix-based predecessor to Mac OS X, so I'm a little biased.
Aside from its beautiful Aqua user interface, Mac OS X is Unix, arguably the best operating system in existence. Unix has many cool features; one of the most well known is the pipe, which lets you create combinations of commands by piping one command's output to another's input. For example, suppose you want to list source files from the Struts source distribution that invoke or define a method namedexecute(). Here's one way to do that with a pipe:
   grep "execute(" `find $STRUTS_SRC_DIR -name "*.java"` | awk -F: '{print }'
The grep command searches files for regular expressions; here, I use it to find occurrences of the stringexecute( in files unearthed by the find command. grep's output is piped into awk, which prints the first token—delimited by a colon—in each line of grep's output (a vertical bar signifies a pipe). That token is a filename, so I end up with a list of filenames that contain the string execute(.
Now that I have a list of filenames, I can use another pipe to sort the list:
  grep "execute(" `find $STRUTS_SRC_DIR -name "*.java"` | awk -F: '{print }' | sort
This time, I've piped the list of filenames to sort. What if you want to know how many files contain the string execute(? It's easy with another pipe:
  grep "execute(" `find $STRUTS_SRC_DIR -name "*.java"` | awk -F: '{print }' | sort -u | wc -l
The wc command counts words, lines, and bytes. In this case, I specified the -l option to count lines, one line for each file. I also added a -u option to sort to ensure uniqueness for each filename (the -u option filters out duplicates).
Pipes are powerful because they let you dynamically compose a chain of operations. Software systems often employ the equivalent of pipes (e.g., email filters or a set of filters for a servlet). At the heart of pipes and filters lies a design pattern: Chain of Responsibility (CoR).
Note: You can download this article's source code from Resources.

CoR introduction

The Chain of Responsibility pattern uses a chain of objects to handle a request, which is typically an event. Objects in the chain forward the request along the chain until one of the objects handles the event. Processing stops after an event is handled.
Figure 1 illustrates how the CoR pattern processes requests.

Figure 1. The Chain of Responsibility pattern
In Design Patterns, the authors describe the Chain of Responsibility pattern like this:
Avoid coupling the sender of a request to its receiver by giving more than one object a chance to handle the request. Chain the receiving objects and pass the request along the chain until an object handles it.
The Chain of Responsibility pattern is applicable if:
  • You want to decouple a request's sender and receiver
  • Multiple objects, determined at runtime, are candidates to handle a request
  • You don't want to specify handlers explicitly in your code
If you use the CoR pattern, remember:
  • Only one object in the chain handles a request
  • Some requests might not get handled
Those restrictions, of course, are for a classic CoR implementation. In practice, those rules are bent; for example, servlet filters are a CoR implementation that allows multiple filters to process an HTTP request.
Figure 2 shows a CoR pattern class diagram.

Figure 2. Chain of Responsibility class diagram
Typically, request handlers are extensions of a base class that maintains a reference to the next handler in the chain, known as the successor. The base class might implement handleRequest() like this:
   public abstract class HandlerBase {
         ...
         public void handleRequest(SomeRequestObject sro) {
            if(successor != null)
                  successor.handleRequest(sro);
         }
   }
So by default, handlers pass the request to the next handler in the chain. A concrete extension ofHandlerBase might look like this:
   public class SpamFilter extends HandlerBase {
      public void handleRequest(SomeRequestObject mailMessage) {
         if(isSpam(mailMessage))   { // If the message is spam
            // take spam-related action. Do not forward message.
         }
         else { // Message is not spam.
            super.handleRequest(mailMessage); // Pass message to next filter in the chain.
         }
      }
   }
The SpamFilter handles the request (presumably receipt of new email) if the message is spam, and therefore, the request goes no further; otherwise, trustworthy messages are passed to the next handler, presumably another email filter looking to weed them out. Eventually, the last filter in the chain might store the message after it passes muster by moving through several filters.
Note the hypothetical email filters discussed above are mutually exclusive: Ultimately, only one filter handles a request. You might opt to turn that inside out by letting multiple filters handle a single request, which is a better analogy to Unix pipes. Either way, the underlying engine is the CoR pattern.
In this article, I discuss two Chain of Responsibility pattern implementations: servlet filters, a popular CoR implementation that allows multiple filters to handle a request, and the original Abstract Window Toolkit (AWT) event model, an unpopular classic CoR implementation that was ultimately deprecated.

Servlet filters

In the Java 2 Platform, Enterprise Edition (J2EE)'s early days, some servlet containers provided a handy feature known as servlet chaining, whereby one could essentially apply a list of filters to a servlet. Servlet filters are popular because they're useful for security, compression, logging, and more. And, of course, you can compose a chain of filters to do some or all of those things depending on runtime conditions.
With the advent of the Java Servlet Specification version 2.3, filters became standard components. Unlike classic CoR, servlet filters allow multiple objects (filters) in a chain to handle a request.
Servlet filters are a powerful addition to J2EE. Also, from a design patterns standpoint, they provide an interesting twist: If you want to modify the request or the response, you use the Decorator pattern in addition to CoR. Figure 3 shows how servlet filters work.

Figure 3. Servlet filters at runtime

A simple servlet filter

You must do three things to filter a servlet:
  • Implement a servlet
  • Implement a filter
  • Associate the filter and the servlet
Examples 1-3 perform all three steps in succession:

Example 1. A servlet

import java.io.PrintWriter;
import javax.servlet.*;
import javax.servlet.http.*;
public class FilteredServlet extends HttpServlet {
   public void doGet(HttpServletRequest request, HttpServletResponse response)
                         throws ServletException, java.io.IOException {
      PrintWriter out = response.getWriter();
      out.println("Filtered Servlet invoked");
   }
}

Example 2. A filter

import java.io.PrintWriter;
import javax.servlet.*;
import javax.servlet.http.HttpServletRequest;
public class AuditFilter implements Filter {
   private ServletContext app = null;
   public void init(FilterConfig config) { 
      app = config.getServletContext();
   }
   public void doFilter(ServletRequest request, ServletResponse response,
                        FilterChain chain) throws java.io.IOException,
                                               javax.servlet.ServletException {
      app.log(((HttpServletRequest)request).getServletPath());
      chain.doFilter(request, response);
   }
   public void destroy() { }
}

Example 3. The deployment descriptor


   
      auditFilter
      AuditFilter
   
   <filter-mapping>
      auditFilter
      /filteredServlet
   </filter-mapping>
     
     
       filteredServlet
       FilteredServlet
     
     
       filteredServlet
       /filteredServlet
     
   ...

If you access the servlet with the URL /filteredServlet, the auditFilter gets a crack at the request before the servlet. AuditFilter.doFilter writes to the servlet container log file and callschain.doFilter() to forward the request. Servlet filters are not required to call chain.doFilter(); if they don't, the request is not forwarded. I can add more filters, which would be invoked in the order they are declared in the preceding XML file.
Now that you've seen a simple filter, let's look at another filter that modifies the HTTP response.

Filter the response with the Decorator pattern

Unlike the preceding filter, some servlet filters need to modify the HTTP request or response. Interestingly enough, that task involves the Decorator pattern. I discussed the Decorator pattern in two previous Java Design Patterns articles: "Amaze Your Developer Friends with Design Patterns" and "Decorate Your Java Code."
Example 4 lists a filter that performs a simple search and replace in the body of the response. That filter decorates the servlet response and passes the decorator to the servlet. When the servlet finishes writing to the decorated response, the filter performs a search and replace within the response's content.

Example 4. A search and replace filter

import java.io.*;
import javax.servlet.*;
import javax.servlet.http.*;
public class SearchAndReplaceFilter implements Filter {
   private FilterConfig config;
   public void init(FilterConfig config) { this.config = config; }
   public FilterConfig getFilterConfig() { return config; }
   public void doFilter(ServletRequest request, ServletResponse response,
                        FilterChain chain) throws java.io.IOException,
                                               javax.servlet.ServletException {
      StringWrapper wrapper = new StringWrapper((HttpServletResponse)response);
      chain.doFilter(request, wrapper);
      String responseString = wrapper.toString();
      String search = config.getInitParameter("search");
      String replace = config.getInitParameter("replace");
      if(search == null || replace == null)
         return; // Parameters not set properly
      int index = responseString.indexOf(search);
      if(index != -1) {
         String beforeReplace = responseString.substring(0, index);
         String afterReplace=responseString.substring(index + search.length());
         response.getWriter().print(beforeReplace + replace + afterReplace);
      }
   }
   public void destroy() {
      config = null;
   }
}
The preceding filter looks for filter init parameters named search and replace; if they are defined, the filter replaces the first occurrence of the search parameter value with the replace parameter value.
SearchAndReplaceFilter.doFilter() wraps (or decorates) the response object with a wrapper (decorator) that stands in for the response. When SearchAndReplaceFilter.doFilter() callschain.doFilter() to forward the request, it passes the wrapper instead of the original response. The request is forwarded to the servlet, which generates the response.
When chain.doFilter() returns, the servlet is done with the request, so I go to work. First, I check for the search and replace filter parameters; if present, I obtain the string associated with the response wrapper, which is the response content. Then I make the substitution and print it back to the response.
Example 5 lists the StringWrapper class.

Example 5. A decorator

import java.io.*;
import javax.servlet.*;
import javax.servlet.http.*;
public class StringWrapper extends HttpServletResponseWrapper {
   StringWriter writer = new StringWriter();
   public StringWrapper(HttpServletResponse response) { super(response); }
   public PrintWriter getWriter() { return new PrintWriter(writer); }
   public String       toString() { return writer.toString(); }
}
StringWrapper, which decorates the HTTP response in Example 4, is an extension ofHttpServletResponseWrapper, which spares us the drudgery of creating a decorator base class for decorating HTTP responses. HttpServletResponseWrapper ultimately implements theServletResponse interface, so instances of HttpServletResponseWrapper can be passed to any method expecting a ServletResponse object. That's why SearchAndReplaceFilter.doFilter()can call chain.doFilter(request, wrapper) instead of chain.doFilter(request,response).
Now that we have a filter and a response wrapper, let's associate the filter with a URL pattern and specify search and replace patterns:
Page 2
Page 2 of 2

Example 6. A deployment descriptor


   ...
   
      searchAndReplaceFilter
      SearchAndReplaceFilter
      
         search
         Blue Road Inc.
      
      
      
         replace
         Red Rocks Inc.
      
   
   
      searchAndReplaceFilter
      /*
   
   ...

I've now wired the search and replace filter to all requests by associating it with the URL pattern /* and specified that Red Rocks Inc. will replace the first occurrence of Blue Road Inc.. Let's try it on this JavaServer Pages (JSP) page:

Example 7. A JSP page

Welcome to Blue Road Inc.
Figure 4 shows the preceding JSP page's output. Notice that Red Rocks Inc. has replaced Blue Road Inc..


Figure 4. Using a search and replace filter. Click on thumbnail to view full-size image.
Of course, my search and replace filter serves little practical use other than demonstrating how filters can modify the response with a wrapper. However, useful freely available servlet filters are easy to find. For example, Tomcat 4.1.X comes with the following filters: a compression filter that zips responses larger than a threshold (that you can set as a filter parameter); an HTML filter; and a filter that dumps information about a request. You can find those filters under Tomcat's examples directory.
Servlet filters represent a popular CoR pattern variation where multiple objects in the chain may handle a request. Let's wrap up the CoR pattern with a brief look at a classic CoR implementation that was deprecated.

The AWT event model

The AWT originally used the CoR pattern for event handling. This is how it worked:
import java.applet.Applet;
import java.awt.*;
public class MouseSensor extends Frame {
   public static void main(String[] args) {
      MouseSensor ms = new MouseSensor();
      ms.setBounds(10,10,200,200);
      ms.show();
   }
   public MouseSensor() {
      setLayout(new BorderLayout());
      add(new MouseSensorCanvas(), "Center");
   }
}
class MouseSensorCanvas extends Canvas {
   public boolean mouseUp(Event event, int x, int y) {
      System.out.println("mouse up");
      return true; // Event has been handled. Do not propagate to container.
   }
   public boolean mouseDown(Event event, int x, int y) {
      System.out.println("mouse down");
      return true; // Event has been handled. Do not propagate to container.
   }
}
The preceding application creates a canvas and adds it to the application. That canvas handles mouse up and down events by overriding mouseUp() and mouseDown(), respectively. Notice those methods return a boolean value: true signifies that the event has been handled, and therefore should not be propagated to the component's container; false means the event was not fully handled and should be propagated. Events bubble up the component hierarchy until a component handles it; or the event is ignored if no component is interested. This is a classic Chain of Responsibility implementation.
Using the CoR pattern for event handling was doomed to failure because event handling requires subclassing components. Because an average graphical user interface (GUI) uses many components, and most components are interested in at least one event (and some are interested in many events), AWT developers had to implement numerous component subclasses. Generally, requiring inheritance to implement a heavily used feature is poor design, because it results in an explosion of subclasses.
The original AWT event model was eventually replaced with the Observer pattern, known as the delegation model, which eliminated the CoR pattern. Here's how it works:
import java.awt.*;
import java.awt.event.*;
public class MouseSensor extends Frame {
   public static void main(String[] args) {
      MouseSensor ms = new MouseSensor();
      ms.setBounds(10,10,200,200);
      ms.show();
   }
   public MouseSensor() {
      Canvas canvas = new Canvas();
      canvas.addMouseListener(new MouseAdapter() {
         public void mousePressed(MouseEvent e) {
            System.out.println("mouse down");
         }
         public void mouseReleased(MouseEvent e) {
            System.out.println("mouse up");
         }
      });
      setLayout(new BorderLayout());
      add(canvas, "Center");
   }
}
The delegation model—where a component delegates event handling to another object—doesn't require extending component classes, which is a much simpler solution. With the delegation event model, event methods return void because events are no longer automatically propagated to a component's container.
The CoR pattern was not applicable for AWT events because GUI events are much too fine-grained; because there are so many events, and components handle them so frequently, the CoR pattern resulted in an explosion of subclasses—an average GUI could easily have upwards of 50 component subclasses for the single purpose of handling events. Finally, propagation of events was rarely used; typically, events are handled by the component in which they originated.
On the other hand, the CoR pattern is a perfect fit for servlet filters. Compared to GUI events, HTTP requests occur infrequently, so the CoR pattern can better handle them. And event propagation is very useful for filters because you can combine them—much like the Unix pipes discussed at the beginning of this article—to produce a myriad of effects.

Last link

The Chain of Responsibility pattern lets you decouple an event's sender from its receiver with a chain of objects that are candidates to handle the event. With the classic CoR pattern, one or none of the objects in the chain handles the event; if an object doesn't handle the event, it forwards it to the next object in the chain. In this article, we examined two CoR pattern implementations: the original AWT event model and servlet filters.
David Geary is the author of Core JSTL Mastering the JSP Standard Tag Library (Prentice Hall, 2002; ISBN: 0131001531), Advanced JavaServer Pages (Prentice Hall, 2001; ISBN: 0130307041), and the Graphic Java series (Prentice Hall). David has been developing object-oriented software with numerous object-oriented languages for almost 20 years. Since reading the GOF Design Patterns book in 1994, David has been an active proponent of design patterns, and has used and implemented design patterns in Smalltalk, C++, and Java. In 1997, David began working full-time as an author and occasional speaker and consultant. David is a member of the expert groups defining the JSP Standard Tag Library and JavaServer Faces, and is a contributor to the Apache Struts JSP framework. David is currently working on Core JavaServer Faces, which will be published in the spring of 2004.

Learn more about this topic


Follow the Chain of Responsibility <!-- Google Analytics

Shak.blog.notes 08/20/2024

Thinking Like an Architect - InfoQ tags: blog ...