Data gives clarity to Decision Making. : July 2015

.@NYSE grinds to a halt for nearly 4 hrs on July 8 2015. http://t.co/jEAYSqWyNJ @PostGraphics pic.twitter.com/RDeWfGTYyn
— Shak Kathir (@ShakKathir) July 19, 2015

.@NYSE #nyse July 8 2015 blackout is just a blip. Take a look at YTD US #stocks exchanges market #Share pic.twitter.com/J8KcPEh9Qa
— Shak Kathir (@ShakKathir) July 19, 2015

Spark 1.4.1 startup script debugged.

/home/hduser/spark-1.4.1-bin-hadoop2.6/bin/run-example
|_
    /home/hduser/spark-1.4.1-bin-hadoop2.6/bin/spark-submit
    |_
        /home/hduser/spark-1.4.1-bin-hadoop2.6/bin/spark-class
        
hduser@shaklubix1:~/spark-1.4.1-bin-hadoop2.6/bin$ ./run-example JavaWordCount /home/hduser/spark-1.4.1-bin-hadoop2.6/README.md
|_
    Running /home/hduser/spark-1.4.1-bin-hadoop2.6/bin/spark-submit
    shak debug: /home/hduser/spark-1.4.1-bin-hadoop2.6/bin/spark-class: SPARK_HOME=/home/hduser/spark-1.4.1-bin-hadoop2.6
    shak debug: /home/hduser/spark-1.4.1-bin-hadoop2.6/bin/spark-class: ASSEMBLY_DIR=/home/hduser/spark-1.4.1-bin-hadoop2.6/lib
    shak debug: /home/hduser/spark-1.4.1-bin-hadoop2.6/bin/spark-class: $@
    org.apache.spark.deploy.SparkSubmit
        --master
        local[*]
        --class
        org.apache.spark.examples.JavaWordCount
        /home/hduser/spark-1.4.1-bin-hadoop2.6/lib/spark-examples-1.4.1-hadoop2.6.0.jar
        /home/hduser/spark-1.4.1-bin-hadoop2.6/README.md
    shak debug: -----

    shak debug: /home/hduser/spark-1.4.1-bin-hadoop2.6/bin/spark-class: LAUNCH_CLASSPATH
        /home/hduser/spark-1.4.1-bin-hadoop2.6/lib/spark-assembly-1.4.1-hadoop2.6.0.jar

    shak debug: /home/hduser/spark-1.4.1-bin-hadoop2.6/bin/spark-class: ${CMD[@]}
        /usr/lib/jvm/java-7-oracle/bin/java
        -cp
        /home/hduser/spark-1.4.1-bin-hadoop2.6/conf/
        /home/hduser/spark-1.4.1-bin-hadoop2.6/lib/spark-assembly-1.4.1-hadoop2.6.0.jar
        /home/hduser/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar
        /home/hduser/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar
        /home/hduser/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar
        -Xms512m
        -Xmx512m
        -XX:MaxPermSize=256m
        org.apache.spark.deploy.SparkSubmit
        --master
        local[*]
        --class
        org.apache.spark.examples.JavaWordCount
        /home/hduser/spark-1.4.1-bin-hadoop2.6/lib/spark-examples-1.4.1-hadoop2.6.0.jar
        /home/hduser/spark-1.4.1-bin-hadoop2.6/README.md
        shak debug: -----

    Running /home/hduser/spark-1.4.1-bin-hadoop2.6/bin/spark-class
    exec /usr/lib/jvm/java-7-oracle/bin/java -cp /home/hduser/spark-1.4.1-bin-hadoop2.6/conf/:/home/hduser/spark-1.4.1-bin-hadoop2.6/lib/spark-assembly-1.4.1-hadoop2.6.0.jar:/home/hduser/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar:/home/hduser/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar:/home/hduser/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar -Xms512m -Xmx512m -XX:MaxPermSize=256m org.apache.spark.deploy.SparkSubmit --master local[*] --class org.apache.spark.examples.JavaWordCount /home/hduser/spark-1.4.1-bin-hadoop2.6/lib/spark-examples-1.4.1-hadoop2.6.0.jar /home/hduser/spark-1.4.1-bin-hadoop2.6/README.md

JAVA 8 FLATMAP EXAMPLE

A developer knows a set of programming languages:


public class Developer {

    private String name;
    private Set languages;

    public Developer(String name) {
        this.languages = new HashSet<>();
        this.name = name;
    }

    public void add(String language) {
        this.languages.add(language);
    }

    public Set getLanguages() {
        return languages;
    }
}

A team has more than one developers. Now we would like to know the aggregate programming language skills for a given team. Stream#flatMap is perfect for "flattening" collections:


import java.util.ArrayList;
import java.util.List;
import java.util.stream.Collectors;
import static org.junit.Assert.assertTrue;
import org.junit.Test;

public class FlatMapTest {

    @Test
    public void flatMap() {
        List team = new ArrayList<>();
        Developer polyglot = new Developer("esoteric");
        polyglot.add("clojure");
        polyglot.add("scala");
        polyglot.add("groovy");
        polyglot.add("go");

        Developer busy = new Developer("pragmatic");
        busy.add("java");
        busy.add("javascript");

        team.add(polyglot);
        team.add(busy);

        List teamLanguages = team.stream().
                map(d -> d.getLanguages()).
                flatMap(l -> l.stream()).
                collect(Collectors.toList());
        assertTrue(teamLanguages.containsAll(polyglot.getLanguages()));
        assertTrue(teamLanguages.containsAll(busy.getLanguages()));
    }
}

Historical returns...

Visual Capitalist

NYSE July 8 2015 Trading system Outage

Spark 1.4.1 startup script debugged.

JAVA 8 FLATMAP EXAMPLE

JAVA 8 FLATMAP EXAMPLE

Historical returns...

Big picture on BigData