[AI] MLFlow

MLflow, a new open source project from Databricks the simplifies the process of Building and deploying a machine learning model. MLflow provides APIs for tracking experiment runs between multiple users within a reproducible environment and for managing the deployment of models to production. Moreover,

MLflow is designed to be an open, modular platform—you can use it with any existing ML library and incorporate it incrementally into an existing ML development process.

Spark 1.4.1 startup script debugged.

/home/hduser/spark-1.4.1-bin-hadoop2.6/bin/run-example
|_
    /home/hduser/spark-1.4.1-bin-hadoop2.6/bin/spark-submit
    |_
        /home/hduser/spark-1.4.1-bin-hadoop2.6/bin/spark-class
        
hduser@shaklubix1:~/spark-1.4.1-bin-hadoop2.6/bin$ ./run-example JavaWordCount /home/hduser/spark-1.4.1-bin-hadoop2.6/README.md
|_
    Running /home/hduser/spark-1.4.1-bin-hadoop2.6/bin/spark-submit
    shak debug: /home/hduser/spark-1.4.1-bin-hadoop2.6/bin/spark-class: SPARK_HOME=/home/hduser/spark-1.4.1-bin-hadoop2.6
    shak debug: /home/hduser/spark-1.4.1-bin-hadoop2.6/bin/spark-class: ASSEMBLY_DIR=/home/hduser/spark-1.4.1-bin-hadoop2.6/lib
    shak debug: /home/hduser/spark-1.4.1-bin-hadoop2.6/bin/spark-class: $@
    org.apache.spark.deploy.SparkSubmit
        --master
        local[*]
        --class
        org.apache.spark.examples.JavaWordCount
        /home/hduser/spark-1.4.1-bin-hadoop2.6/lib/spark-examples-1.4.1-hadoop2.6.0.jar
        /home/hduser/spark-1.4.1-bin-hadoop2.6/README.md
    shak debug: -----

    shak debug: /home/hduser/spark-1.4.1-bin-hadoop2.6/bin/spark-class: LAUNCH_CLASSPATH
        /home/hduser/spark-1.4.1-bin-hadoop2.6/lib/spark-assembly-1.4.1-hadoop2.6.0.jar

    shak debug: /home/hduser/spark-1.4.1-bin-hadoop2.6/bin/spark-class: ${CMD[@]}
        /usr/lib/jvm/java-7-oracle/bin/java
        -cp
        /home/hduser/spark-1.4.1-bin-hadoop2.6/conf/
        /home/hduser/spark-1.4.1-bin-hadoop2.6/lib/spark-assembly-1.4.1-hadoop2.6.0.jar
        /home/hduser/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar
        /home/hduser/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar
        /home/hduser/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar
        -Xms512m
        -Xmx512m
        -XX:MaxPermSize=256m
        org.apache.spark.deploy.SparkSubmit
        --master
        local[*]
        --class
        org.apache.spark.examples.JavaWordCount
        /home/hduser/spark-1.4.1-bin-hadoop2.6/lib/spark-examples-1.4.1-hadoop2.6.0.jar
        /home/hduser/spark-1.4.1-bin-hadoop2.6/README.md
        shak debug: -----

    Running /home/hduser/spark-1.4.1-bin-hadoop2.6/bin/spark-class
    exec /usr/lib/jvm/java-7-oracle/bin/java -cp /home/hduser/spark-1.4.1-bin-hadoop2.6/conf/:/home/hduser/spark-1.4.1-bin-hadoop2.6/lib/spark-assembly-1.4.1-hadoop2.6.0.jar:/home/hduser/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar:/home/hduser/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar:/home/hduser/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar -Xms512m -Xmx512m -XX:MaxPermSize=256m org.apache.spark.deploy.SparkSubmit --master local[*] --class org.apache.spark.examples.JavaWordCount /home/hduser/spark-1.4.1-bin-hadoop2.6/lib/spark-examples-1.4.1-hadoop2.6.0.jar /home/hduser/spark-1.4.1-bin-hadoop2.6/README.md    

What are closures ?

In a simplest sense, a Closure is a function instance,  just like class instance, with each function instance is instantiated with the specific value. In Python for example where everything is an object - modules, classes, functions, attributes etc., a closure is an object returned by a function which is instantiated with parameters it is invoked with. The object returned in this case is a  function  ( an inner function ) instance returned by another function (an outer function).

Now a more canonical explanation definition of closure.
"A closure is data attached to code " . A Closure is about  maintaining state of some kind in functions. 

Many functional languages ( Lisp, Haskel, Scala and other )  rely heavily on closures.  Python, support it, but it's not at the very core of the language. Python, however, is a fine language for getting to know closures. So, even if your expertise is in some other language, you might learn a thing or two here.

An Example

Now consider this :

def make_log(level):
    ''' a closure that manufactures different logs'''
    def _anon(message):
        print ("{}: {}".format(level, message))
    return _anon

if __name__ == '__main__' :
    main()
    print(main.__doc__)
    # log_info closure
    log_info = make_log("info")
    # log warn closure
    log_warn = make_log("warn")
    # log error closure
    log_error = make_log("error")
    
    log_info("loading configuration file")  #invoking log_info closure
    log_warn("config value missing. using default value") # invoking log_warn closure
    log_error("network socket error. can not connect to server") # invoking log_error closure

Spark - RDD how do they work ?

Original RDD research publication by Matei Zharia - Creater of Apache Spark