Class04 Answer:

Work through the first page of the Spark Quick-Start:

I found the Spark Tutorial at the URL listed below:

http://spark.apache.org/docs/latest/quick-start.html

I tried shell and Scala commands listed below:

cd ~/spark
bin/spark-shell
val textFile = sc.textFile("README.md")
textFile.count()
textFile.first()
val linesWithSpark = textFile.filter(line => line.contains("Spark"))
textFile.filter(line => line.contains("Spark")).count()
textFile.map(line => line.split(" ").size).reduce((a, b) => if (a > b) a else b)
import java.lang.Math
textFile.map(line => line.split(" ").size).reduce((a, b) => Math.max(a, b))
val wordCounts = textFile.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey((a, b) => a + b)
wordCounts.collect()
linesWithSpark.cache()
linesWithSpark.count()
linesWithSpark.count()

I saw some output:


dan@h80:~ $ 
dan@h80:~ $ 
dan@h80:~ $ cd ~/spark
dan@h80:~/spark $ 
dan@h80:~/spark $ ls -la
total 112
drwxrwxr-x 12 dan dan  4096 Nov 24 15:31 .
drwxr-xr-x 43 dan dan  4096 Jan 11 12:52 ..
drwxrwxr-x  2 dan dan  4096 Nov 24 15:31 bin
drwxrwxr-x  2 dan dan  4096 Jan 11 04:42 conf
drwxrwxr-x  5 dan dan  4096 Nov 24 15:31 data
drwxrwxr-x  4 dan dan  4096 Nov 24 15:31 examples
drwxrwxr-x  2 dan dan 12288 Nov 24 15:31 jars
-rw-rw-r--  1 dan dan 17881 Nov 24 15:31 LICENSE
drwxrwxr-x  2 dan dan  4096 Nov 24 15:31 licenses
-rw-rw-r--  1 dan dan 24645 Nov 24 15:31 NOTICE
drwxrwxr-x  8 dan dan  4096 Nov 24 15:31 python
drwxrwxr-x  3 dan dan  4096 Nov 24 15:31 R
-rw-rw-r--  1 dan dan  3809 Nov 24 15:31 README.md
-rw-rw-r--  1 dan dan   128 Nov 24 15:31 RELEASE
drwxrwxr-x  2 dan dan  4096 Nov 24 15:31 sbin
drwxrwxr-x  2 dan dan  4096 Nov 24 15:31 yarn
dan@h80:~/spark $ 
dan@h80:~/spark $ 



dan@h80:~/spark $ bin/spark-shell
Spark context Web UI available at http://192.168.1.80:4041
Spark context available as 'sc' (master = local[*], app id = local-1515715964418).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.2.1
      /_/
         
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_152)
Type in expressions to have them evaluated.
Type :help for more information.

scala> 
scala> 
scala> 



scala> val textFile = sc.textFile("README.md")
textFile: org.apache.spark.rdd.RDD[String] = README.md MapPartitionsRDD[1] at textFile at <console>:24
scala> 
scala> 
scala> 



scala> textFile.count()
res0: Long = 103
scala> 
scala> 
scala> 



scala> textFile.first()
res1: String = # Apache Spark
scala> 
scala> 
scala> 


scala> val linesWithSpark = textFile.filter(line => line.contains("Spark"))
linesWithSpark: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[2] at filter at <console>:26
scala> 
scala> 
scala> 



scala> textFile.filter(line => line.contains("Spark")).count()
res2: Long = 20
scala> 
scala> 
scala> 



scala> textFile.map(line => line.split(" ").size).reduce((a, b) => if (a > b) a else b)
res3: Int = 22
scala> 
scala> 
scala> 


scala> import java.lang.Math
import java.lang.Math
scala> 
scala> 
scala> 



scala> textFile.map(line => line.split(" ").size).reduce((a, b) => Math.max(a, b))
res4: Int = 22
scala> 
scala> 
scala> 



scala> val wordCounts = textFile.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey((a, b) => a + b)
wordCounts: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[8] at reduceByKey at <console>:27
scala> 
scala> 
scala> 



scala> wordCounts.collect()
res5: Array[(String, Int)] = Array((package,1), (this,1), (Version"](http://spark.apache.org/docs/latest/building-spark.html#specifying-the-hadoop-version),1), (Because,1), (Python,2), (page](http://spark.apache.org/documentation.html).,1), (cluster.,1), (its,1), ([run,1), (general,3), (have,1), (pre-built,1), (YARN,,1), (locally,2), (changed,1), (locally.,1), (sc.parallelize(1,1), (only,1), (several,1), (This,2), (basic,1), (Configuration,1), (learning,,1), (documentation,3), (first,1), (graph,1), (Hive,2), (info,1), (["Specifying,1), ("yarn",1), ([params]`.,1), ([project,1), (prefer,1), (SparkPi,2), (<http://spark.apache.org/>,1), (engine,1), (version,1), (file,1), (documentation,,1), (MASTER,1), (example,3), (["Parallel,1), (are,1), (params,1), (scala>,1), (DataFrames,,1), (provides,...
scala> 
scala> 
scala> 

scala> linesWithSpark.cache()
res6: linesWithSpark.type = MapPartitionsRDD[2] at filter at <console>:26
scala> 
scala> 
scala> 



scala> linesWithSpark.count()
res7: Long = 20
scala> 
scala> 
scala> 



scala> linesWithSpark.count()
res8: Long = 20
scala> 
scala> 
scala> 



scala> :quit
dan@h80:~/spark $ 
dan@h80:~/spark $ 
dan@h80:~/spark $ 

I thought the logging output was too verbose; I made it less so.

I control the logging level by creating a file here:

~/spark/conf/log4j.properties

I created the file from this file:

~/spark/conf/log4j.properties.template

Now the file looks like this:


#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# Set everything to be logged to the console
log4j.rootCategory=ERROR, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=ERROR

# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark_project.jetty=ERROR
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=ERROR
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=ERROR
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR

# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR

When I start spark-shell, the output is much cleaner:


dan@h80:~/spark $ bin/spark-shell 
Spark context Web UI available at http://192.168.1.80:4041
Spark context available as 'sc' (master = local[*], app id = local-1515721081501).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.2.1
      /_/
         
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_152)
Type in expressions to have them evaluated.
Type :help for more information.

scala> 
scala>

After fixing the log-level, I created a folder:

mkdir ~/spark10

Then I created a file named ~/spark10/build.sbt:

name := "Simple Project"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.2.1"

Then I created a file named ~/spark10/src/main/scala/SimpleApp.scala


/* SimpleApp.scala */
import org.apache.spark.sql.SparkSession

object SimpleApp {
  def main(args: Array[String]) {
    val logFile = "/etc/passwd" // Should be some file on your system
    val spark = SparkSession.builder.appName("Simple Application").getOrCreate()
    val logData = spark.read.textFile(logFile).cache()
    val numAs = logData.filter(line => line.contains("a")).count()
    val numBs = logData.filter(line => line.contains("b")).count()
    println(s"Lines with a: $numAs, Lines with b: $numBs")
    spark.stop()
  }
}

The quick-start shows that I need a utility named 'sbt'.

But the quick-start fails to describe what sbt is or how to install it.

After 10 minutes with Google I learned that I use sbt to compile scala scripts.

Also I found a page which serves information about sbt:

http://www.scala-sbt.org

I used some shell commands to install sbt:


cd ~/Downloads

wget https://github.com/sbt/sbt/releases/download/v1.1.0/sbt-1.1.0.tgz

cd ~

tar xf Downloads/sbt-1.1.0.tgz 

echo 'export PATH=${HOME}/sbt/bin:${PATH}' >> ~/.bashrc

bash

which sbt

sbt -help

Next, I ran simple shell commands:

cd ~/spark10/
sbt package

I saw this:


dan@h80:~ $ 
dan@h80:~ $ cd spark10
dan@h80:~/spark10 $ 
dan@h80:~/spark10 $ 
dan@h80:~/spark10 $ 



dan@h80:~/spark10 $ which sbt
/home/dan/sbt/bin/sbt
dan@h80:~/spark10 $ 
dan@h80:~/spark10 $ 
dan@h80:~/spark10 $ 



dan@h80:~/spark10 $ ll
total 16
drwxrwxr-x  3 dan dan 4096 Jan 11 17:46 ./
drwxr-xr-x 46 dan dan 4096 Jan 11 18:06 ../
-rw-rw-r--  1 dan dan  137 Jan 11 17:45 build.sbt
drwxrwxr-x  3 dan dan 4096 Jan 11 17:46 src/
dan@h80:~/spark10 $ 
dan@h80:~/spark10 $ 
dan@h80:~/spark10 $ 



dan@h80:~/spark10 $ sbt package
[info] Updated file /home/dan/spark10/project/build.properties: set sbt.version to 1.1.0
[info] Loading settings from idea.sbt ...
[info] Loading global plugins from /home/dan/.sbt/1.0/plugins
[info] Loading project definition from /home/dan/spark10/project
[info] Updating ProjectRef(uri("file:/home/dan/spark10/project/"), "spark10-build")...
[info] Done updating.
[info] Loading settings from build.sbt ...
[info] Set current project to Simple Project (in build file:/home/dan/spark10/)
[info] Updating ...
[info] downloading https://repo1.maven.org/maven2/commons-digester/commons-digester/1.8/commons-digester-1.8.jar ...
[info] downloading https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-core/3.1.2/metrics-core-3.1.2.jar ...
[info] downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-app/2.6.5/hadoop-mapreduce-client-app-2.6.5.jar ...
[info] downloading https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.11.8/scala-library-2.11.8.jar ...
[info] 	[SUCCESSFUL ] io.dropwizard.metrics#metrics-core;3.1.2!metrics-core.jar(bundle) (83ms)
[info] downloading https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-jvm/3.1.2/metrics-jvm-3.1.2.jar ...
[info] 	[SUCCESSFUL ] io.dropwizard.metrics#metrics-jvm;3.1.2!metrics-jvm.jar(bundle) (54ms)
[info] downloading https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-json/3.1.2/metrics-json-3.1.2.jar ...
[info] 	[SUCCESSFUL ] io.dropwizard.metrics#metrics-json;3.1.2!metrics-json.jar(bundle) (166ms)
[info] downloading https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-graphite/3.1.2/metrics-graphite-3.1.2.jar ...
[info] 	[SUCCESSFUL ] commons-digester#commons-digester;1.8!commons-digester.jar (458ms)
[info] downloading https://repo1.maven.org/maven2/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar ...
[info] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-mapreduce-client-app;2.6.5!hadoop-mapreduce-client-app.jar (723ms)
[info] 	[SUCCESSFUL ] io.dropwizard.metrics#metrics-graphite;3.1.2!metrics-graphite.jar(bundle) (442ms)
[info] downloading https://repo1.maven.org/maven2/com/fasterxml/jackson/module/jackson-module-scala_2.11/2.6.5/jackson-module-scala_2.11-2.6.5.jar ...
[info] downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-api/2.6.5/hadoop-yarn-api-2.6.5.jar ...
[info] 	[SUCCESSFUL ] commons-beanutils#commons-beanutils-core;1.8.0!commons-beanutils-core.jar (618ms)
[info] downloading https://repo1.maven.org/maven2/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar ...
[info] 	[SUCCESSFUL ] org.scala-lang#scala-library;2.11.8!scala-library.jar (1233ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/spark/spark-sql_2.11/2.2.1/spark-sql_2.11-2.2.1.jar ...
[info] 	[SUCCESSFUL ] com.fasterxml.jackson.module#jackson-module-scala_2.11;2.6.5!jackson-module-scala_2.11.jar(bundle) (521ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar ...
[info] 	[SUCCESSFUL ] commons-beanutils#commons-beanutils;1.7.0!commons-beanutils.jar (196ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/directory/server/apacheds-kerberos-codec/2.0.0-M15/apacheds-kerberos-codec-2.0.0-M15.jar ...
[info] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-yarn-api;2.6.5!hadoop-yarn-api.jar (774ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.6.5/hadoop-mapreduce-client-core-2.6.5.jar ...
[info] 	[SUCCESSFUL ] org.apache.directory.server#apacheds-kerberos-codec;2.0.0-M15!apacheds-kerberos-codec.jar(bundle) (580ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/curator/curator-framework/2.6.0/curator-framework-2.6.0.jar ...
[info] 	[SUCCESSFUL ] org.apache.ivy#ivy;2.4.0!ivy.jar (873ms)
[info] downloading https://repo1.maven.org/maven2/oro/oro/2.0.8/oro-2.0.8.jar ...
[info] 	[SUCCESSFUL ] org.apache.curator#curator-framework;2.6.0!curator-framework.jar(bundle) (322ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/directory/server/apacheds-i18n/2.0.0-M15/apacheds-i18n-2.0.0-M15.jar ...
[info] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-mapreduce-client-core;2.6.5!hadoop-mapreduce-client-core.jar (703ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.6.5/hadoop-mapreduce-client-jobclient-2.6.5.jar ...
[info] 	[SUCCESSFUL ] oro#oro;2.0.8!oro.jar (293ms)
[info] downloading https://repo1.maven.org/maven2/net/razorvine/pyrolite/4.13/pyrolite-4.13.jar ...
[info] 	[SUCCESSFUL ] org.apache.directory.server#apacheds-i18n;2.0.0-M15!apacheds-i18n.jar(bundle) (320ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/directory/api/api-asn1-api/1.0.0-M20/api-asn1-api-1.0.0-M20.jar ...
[info] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-mapreduce-client-jobclient;2.6.5!hadoop-mapreduce-client-jobclient.jar (330ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-annotations/2.6.5/hadoop-annotations-2.6.5.jar ...
[info] 	[SUCCESSFUL ] net.razorvine#pyrolite;4.13!pyrolite.jar (219ms)
[info] downloading https://repo1.maven.org/maven2/net/sf/py4j/py4j/0.10.4/py4j-0.10.4.jar ...
[info] 	[SUCCESSFUL ] org.apache.directory.api#api-asn1-api;1.0.0-M20!api-asn1-api.jar(bundle) (177ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/directory/api/api-util/1.0.0-M20/api-util-1.0.0-M20.jar ...
[info] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-annotations;2.6.5!hadoop-annotations.jar (135ms)
[info] downloading https://repo1.maven.org/maven2/com/google/guava/guava/11.0.2/guava-11.0.2.jar ...
[info] 	[SUCCESSFUL ] net.sf.py4j#py4j;0.10.4!py4j.jar (91ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/commons/commons-crypto/1.0.0/commons-crypto-1.0.0.jar ...
[info] 	[SUCCESSFUL ] org.apache.spark#spark-sql_2.11;2.2.1!spark-sql_2.11.jar (1521ms)
[info] downloading https://repo1.maven.org/maven2/com/univocity/univocity-parsers/2.2.1/univocity-parsers-2.2.1.jar ...
[info] 	[SUCCESSFUL ] org.apache.directory.api#api-util;1.0.0-M20!api-util.jar(bundle) (104ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar ...
[info] 	[SUCCESSFUL ] org.apache.commons#commons-crypto;1.0.0!commons-crypto.jar (317ms)
[info] downloading https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar ...
[info] 	[SUCCESSFUL ] com.univocity#univocity-parsers;2.2.1!univocity-parsers.jar (300ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/spark/spark-sketch_2.11/2.2.1/spark-sketch_2.11-2.2.1.jar ...
[info] 	[SUCCESSFUL ] org.apache.zookeeper#zookeeper;3.4.6!zookeeper.jar (319ms)
[info] downloading https://repo1.maven.org/maven2/jline/jline/0.9.94/jline-0.9.94.jar ...
[info] 	[SUCCESSFUL ] com.google.guava#guava;11.0.2!guava.jar (454ms)
[info] downloading https://repo1.maven.org/maven2/commons-cli/commons-cli/1.2/commons-cli-1.2.jar ...
[info] 	[SUCCESSFUL ] org.apache.spark#spark-sketch_2.11;2.2.1!spark-sketch_2.11.jar (87ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.11/2.2.1/spark-core_2.11-2.2.1.jar ...
[info] 	[SUCCESSFUL ] org.codehaus.jackson#jackson-core-asl;1.9.13!jackson-core-asl.jar (102ms)
[info] downloading https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar ...
[info] 	[SUCCESSFUL ] jline#jline;0.9.94!jline.jar (85ms)
[info] downloading https://repo1.maven.org/maven2/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar ...
[info] 	[SUCCESSFUL ] commons-cli#commons-cli;1.2!commons-cli.jar (285ms)
[info] downloading https://repo1.maven.org/maven2/xmlenc/xmlenc/0.52/xmlenc-0.52.jar ...
[info] 	[SUCCESSFUL ] org.mortbay.jetty#jetty-util;6.1.26!jetty-util.jar (521ms)
[info] downloading https://repo1.maven.org/maven2/xerces/xercesImpl/2.9.1/xercesImpl-2.9.1.jar ...
[info] 	[SUCCESSFUL ] org.codehaus.jackson#jackson-mapper-asl;1.9.13!jackson-mapper-asl.jar (578ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/commons/commons-compress/1.4.1/commons-compress-1.4.1.jar ...
[info] 	[SUCCESSFUL ] xmlenc#xmlenc;0.52!xmlenc.jar (519ms)
[info] downloading https://repo1.maven.org/maven2/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar ...
[info] 	[SUCCESSFUL ] org.apache.commons#commons-compress;1.4.1!commons-compress.jar (785ms)
[info] downloading https://repo1.maven.org/maven2/org/tukaani/xz/1.0/xz-1.0.jar ...
[info] 	[SUCCESSFUL ] commons-httpclient#commons-httpclient;3.1!commons-httpclient.jar (711ms)
[info] downloading https://repo1.maven.org/maven2/commons-io/commons-io/2.4/commons-io-2.4.jar ...
[info] 	[SUCCESSFUL ] xerces#xercesImpl;2.9.1!xercesImpl.jar (1032ms)
[info] downloading https://repo1.maven.org/maven2/xml-apis/xml-apis/1.3.04/xml-apis-1.3.04.jar ...
[info] 	[SUCCESSFUL ] org.tukaani#xz;1.0!xz.jar (574ms)
[info] 	[SUCCESSFUL ] xml-apis#xml-apis;1.3.04!xml-apis.jar (394ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-common/2.6.5/hadoop-mapreduce-client-common-2.6.5.jar ...
[info] downloading https://repo1.maven.org/maven2/org/apache/avro/avro-ipc/1.7.7/avro-ipc-1.7.7-tests.jar ...
[info] 	[SUCCESSFUL ] commons-io#commons-io;2.4!commons-io.jar (479ms)
[info] downloading https://repo1.maven.org/maven2/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar ...
[info] 	[SUCCESSFUL ] org.apache.spark#spark-core_2.11;2.2.1!spark-core_2.11.jar (2056ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/spark/spark-catalyst_2.11/2.2.1/spark-catalyst_2.11-2.2.1.jar ...
[info] 	[SUCCESSFUL ] org.apache.avro#avro-ipc;1.7.7!avro-ipc.jar (371ms)
[info] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-mapreduce-client-common;2.6.5!hadoop-mapreduce-client-common.jar (358ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-shuffle/2.6.5/hadoop-mapreduce-client-shuffle-2.6.5.jar ...
[info] downloading https://repo1.maven.org/maven2/org/apache/avro/avro-ipc/1.7.7/avro-ipc-1.7.7.jar ...
[info] 	[SUCCESSFUL ] commons-collections#commons-collections;3.2.2!commons-collections.jar (403ms)
[info] downloading https://repo1.maven.org/maven2/commons-lang/commons-lang/2.6/commons-lang-2.6.jar ...
[info] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-mapreduce-client-shuffle;2.6.5!hadoop-mapreduce-client-shuffle.jar (390ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.6.5/hadoop-yarn-common-2.6.5.jar ...
[info] 	[SUCCESSFUL ] org.apache.avro#avro-ipc;1.7.7!avro-ipc.jar (396ms)
[info] downloading https://repo1.maven.org/maven2/com/esotericsoftware/kryo-shaded/3.0.3/kryo-shaded-3.0.3.jar ...
[info] 	[SUCCESSFUL ] commons-lang#commons-lang;2.6!commons-lang.jar (422ms)
[info] downloading https://repo1.maven.org/maven2/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar ...
[info] 	[SUCCESSFUL ] com.esotericsoftware#kryo-shaded;3.0.3!kryo-shaded.jar(bundle) (638ms)
[info] downloading https://repo1.maven.org/maven2/com/esotericsoftware/minlog/1.3.0/minlog-1.3.0.jar ...
[info] 	[SUCCESSFUL ] commons-configuration#commons-configuration;1.6!commons-configuration.jar (589ms)
[info] downloading https://repo1.maven.org/maven2/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar ...
[info] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-yarn-common;2.6.5!hadoop-yarn-common.jar (685ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-client/2.6.5/hadoop-yarn-client-2.6.5.jar ...
[info] 	[SUCCESSFUL ] com.esotericsoftware#minlog;1.3.0!minlog.jar(bundle) (274ms)
[info] downloading https://repo1.maven.org/maven2/org/objenesis/objenesis/2.1/objenesis-2.1.jar ...
[info] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-yarn-client;2.6.5!hadoop-yarn-client.jar (260ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-server-common/2.6.5/hadoop-yarn-server-common-2.6.5.jar ...
[info] 	[SUCCESSFUL ] com.google.protobuf#protobuf-java;2.5.0!protobuf-java.jar(bundle) (283ms)
[info] downloading https://repo1.maven.org/maven2/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar ...
[info] 	[SUCCESSFUL ] org.apache.spark#spark-catalyst_2.11;2.2.1!spark-catalyst_2.11.jar (1665ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/spark/spark-tags_2.11/2.2.1/spark-tags_2.11-2.2.1.jar ...
[info] 	[SUCCESSFUL ] org.objenesis#objenesis;2.1!objenesis.jar (84ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/2.6.5/hadoop-common-2.6.5.jar ...
[info] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-yarn-server-common;2.6.5!hadoop-yarn-server-common.jar (152ms)
[info] downloading https://repo1.maven.org/maven2/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar ...
[info] 	[SUCCESSFUL ] org.apache.spark#spark-tags_2.11;2.2.1!spark-tags_2.11.jar (159ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-column/1.8.2/parquet-column-1.8.2.jar ...
[info] 	[SUCCESSFUL ] com.google.code.gson#gson;2.2.4!gson.jar (194ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-auth/2.6.5/hadoop-auth-2.6.5.jar ...
[info] 	[SUCCESSFUL ] javax.xml.bind#jaxb-api;2.2.2!jaxb-api.jar (384ms)
[info] downloading https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-jaxrs/1.9.13/jackson-jaxrs-1.9.13.jar ...
[info] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-auth;2.6.5!hadoop-auth.jar (373ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/curator/curator-client/2.6.0/curator-client-2.6.0.jar ...
[info] 	[SUCCESSFUL ] org.apache.parquet#parquet-column;1.8.2!parquet-column.jar (427ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-hadoop/1.8.2/parquet-hadoop-1.8.2.jar ...
[info] 	[SUCCESSFUL ] org.codehaus.jackson#jackson-jaxrs;1.9.13!jackson-jaxrs.jar (125ms)
[info] downloading https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-xc/1.9.13/jackson-xc-1.9.13.jar ...
[info] 	[SUCCESSFUL ] org.apache.curator#curator-client;2.6.0!curator-client.jar(bundle) (96ms)
[info] downloading https://repo1.maven.org/maven2/org/htrace/htrace-core/3.0.4/htrace-core-3.0.4.jar ...
[info] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-common;2.6.5!hadoop-common.jar (637ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/2.6.5/hadoop-hdfs-2.6.5.jar ...
[info] 	[SUCCESSFUL ] org.codehaus.jackson#jackson-xc;1.9.13!jackson-xc.jar (138ms)
[info] downloading https://repo1.maven.org/maven2/com/google/inject/guice/3.0/guice-3.0.jar ...
[info] 	[SUCCESSFUL ] org.apache.parquet#parquet-hadoop;1.8.2!parquet-hadoop.jar (164ms)
[info] downloading https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.6.5/jackson-databind-2.6.5.jar ...
[info] 	[SUCCESSFUL ] org.htrace#htrace-core;3.0.4!htrace-core.jar (449ms)
[info] downloading https://repo1.maven.org/maven2/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar ...
[info] 	[SUCCESSFUL ] com.google.inject#guice;3.0!guice.jar (793ms)
[info] downloading https://repo1.maven.org/maven2/javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar ...
[info] 	[SUCCESSFUL ] com.fasterxml.jackson.core#jackson-databind;2.6.5!jackson-databind.jar(bundle) (833ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/xbean/xbean-asm5-shaded/4.4/xbean-asm5-shaded-4.4.jar ...
[info] 	[SUCCESSFUL ] javax.servlet#javax.servlet-api;3.1.0!javax.servlet-api.jar (535ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/commons/commons-lang3/3.5/commons-lang3-3.5.jar ...
[info] 	[SUCCESSFUL ] javax.xml.stream#stax-api;1.0-2!stax-api.jar (424ms)
[info] downloading https://repo1.maven.org/maven2/javax/inject/javax.inject/1/javax.inject-1.jar ...
[info] 	[SUCCESSFUL ] org.apache.xbean#xbean-asm5-shaded;4.4!xbean-asm5-shaded.jar(bundle) (398ms)
[info] downloading https://repo1.maven.org/maven2/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar ...
[info] 	[SUCCESSFUL ] org.apache.commons#commons-lang3;3.5!commons-lang3.jar (374ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/commons/commons-math3/3.4.1/commons-math3-3.4.1.jar ...
[info] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-hdfs;2.6.5!hadoop-hdfs.jar (1393ms)
[info] 	[SUCCESSFUL ] javax.inject#javax.inject;1!javax.inject.jar (63ms)
[info] downloading https://repo1.maven.org/maven2/net/jpountz/lz4/lz4/1.3.0/lz4-1.3.0.jar ...
[info] downloading https://repo1.maven.org/maven2/aopalliance/aopalliance/1.0/aopalliance-1.0.jar ...
[info] 	[SUCCESSFUL ] org.spark-project.spark#unused;1.0.0!unused.jar (78ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/avro/avro/1.7.7/avro-1.7.7.jar ...
[info] 	[SUCCESSFUL ] aopalliance#aopalliance;1.0!aopalliance.jar (314ms)
[info] downloading https://repo1.maven.org/maven2/org/sonatype/sisu/inject/cglib/2.2.1-v20090111/cglib-2.2.1-v20090111.jar ...
[info] 	[SUCCESSFUL ] net.jpountz.lz4#lz4;1.3.0!lz4.jar (327ms)
[info] downloading https://repo1.maven.org/maven2/org/roaringbitmap/RoaringBitmap/0.5.11/RoaringBitmap-0.5.11.jar ...
[info] 	[SUCCESSFUL ] org.apache.avro#avro;1.7.7!avro.jar(bundle) (314ms)
[info] 	[SUCCESSFUL ] org.apache.commons#commons-math3;3.4.1!commons-math3.jar (393ms)
[info] downloading https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar ...
[info] downloading https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.7.7/avro-mapred-1.7.7-hadoop2.jar ...
[info] 	[SUCCESSFUL ] com.google.code.findbugs#jsr305;1.3.9!jsr305.jar (86ms)
[info] downloading https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.16/slf4j-api-1.7.16.jar ...
[info] 	[SUCCESSFUL ] org.roaringbitmap#RoaringBitmap;0.5.11!RoaringBitmap.jar(bundle) (110ms)
[info] downloading https://repo1.maven.org/maven2/commons-net/commons-net/2.2/commons-net-2.2.jar ...
[info] 	[SUCCESSFUL ] org.sonatype.sisu.inject#cglib;2.2.1-v20090111!cglib.jar (122ms)
[info] downloading https://repo1.maven.org/maven2/org/fusesource/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.jar ...
[info] 	[SUCCESSFUL ] org.apache.avro#avro-mapred;1.7.7!avro-mapred.jar (138ms)
[info] downloading https://repo1.maven.org/maven2/com/twitter/chill_2.11/0.8.0/chill_2.11-0.8.0.jar ...
[info] 	[SUCCESSFUL ] org.slf4j#slf4j-api;1.7.16!slf4j-api.jar (192ms)
[info] downloading https://repo1.maven.org/maven2/org/slf4j/jul-to-slf4j/1.7.16/jul-to-slf4j-1.7.16.jar ...
[info] 	[SUCCESSFUL ] commons-net#commons-net;2.2!commons-net.jar (203ms)
[info] downloading https://repo1.maven.org/maven2/org/json4s/json4s-jackson_2.11/3.2.11/json4s-jackson_2.11-3.2.11.jar ...
[info] 	[SUCCESSFUL ] org.fusesource.leveldbjni#leveldbjni-all;1.8!leveldbjni-all.jar(bundle) (227ms)
[info] downloading https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.6.5/jackson-annotations-2.6.5.jar ...
[info] 	[SUCCESSFUL ] com.twitter#chill_2.11;0.8.0!chill_2.11.jar (157ms)
[info] downloading https://repo1.maven.org/maven2/com/twitter/chill-java/0.8.0/chill-java-0.8.0.jar ...
[info] 	[SUCCESSFUL ] org.slf4j#jul-to-slf4j;1.7.16!jul-to-slf4j.jar (61ms)
[info] downloading https://repo1.maven.org/maven2/org/slf4j/jcl-over-slf4j/1.7.16/jcl-over-slf4j-1.7.16.jar ...
[info] 	[SUCCESSFUL ] org.json4s#json4s-jackson_2.11;3.2.11!json4s-jackson_2.11.jar (56ms)
[info] downloading https://repo1.maven.org/maven2/org/glassfish/jersey/core/jersey-client/2.22.2/jersey-client-2.22.2.jar ...
[info] 	[SUCCESSFUL ] com.fasterxml.jackson.core#jackson-annotations;2.6.5!jackson-annotations.jar(bundle) (61ms)
[info] downloading https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.6.5/jackson-core-2.6.5.jar ...
[info] 	[SUCCESSFUL ] com.twitter#chill-java;0.8.0!chill-java.jar (63ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.6.5/hadoop-client-2.6.5.jar ...
[info] 	[SUCCESSFUL ] org.slf4j#jcl-over-slf4j;1.7.16!jcl-over-slf4j.jar (86ms)
[info] downloading https://repo1.maven.org/maven2/log4j/log4j/1.2.17/log4j-1.2.17.jar ...
[info] 	[SUCCESSFUL ] org.glassfish.jersey.core#jersey-client;2.22.2!jersey-client.jar (98ms)
[info] downloading https://repo1.maven.org/maven2/org/glassfish/jersey/core/jersey-common/2.22.2/jersey-common-2.22.2.jar ...
[info] 	[SUCCESSFUL ] com.fasterxml.jackson.core#jackson-core;2.6.5!jackson-core.jar(bundle) (219ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/httpcomponents/httpcore/4.3.3/httpcore-4.3.3.jar ...
[info] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-client;2.6.5!hadoop-client.jar (214ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/spark/spark-launcher_2.11/2.2.1/spark-launcher_2.11-2.2.1.jar ...
[info] 	[SUCCESSFUL ] log4j#log4j;1.2.17!log4j.jar(bundle) (223ms)
[info] downloading https://repo1.maven.org/maven2/org/slf4j/slf4j-log4j12/1.7.16/slf4j-log4j12-1.7.16.jar ...
[info] 	[SUCCESSFUL ] org.glassfish.jersey.core#jersey-common;2.22.2!jersey-common.jar (238ms)
[info] downloading https://repo1.maven.org/maven2/org/glassfish/jersey/core/jersey-server/2.22.2/jersey-server-2.22.2.jar ...
[info] 	[SUCCESSFUL ] org.apache.spark#spark-launcher_2.11;2.2.1!spark-launcher_2.11.jar (86ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/spark/spark-network-common_2.11/2.2.1/spark-network-common_2.11-2.2.1.jar ...
[info] 	[SUCCESSFUL ] org.apache.httpcomponents#httpcore;4.3.3!httpcore.jar (100ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient/4.3.6/httpclient-4.3.6.jar ...
[info] 	[SUCCESSFUL ] org.slf4j#slf4j-log4j12;1.7.16!slf4j-log4j12.jar (300ms)
[info] downloading https://repo1.maven.org/maven2/com/ning/compress-lzf/1.0.3/compress-lzf-1.0.3.jar ...
[info] 	[SUCCESSFUL ] org.apache.httpcomponents#httpclient;4.3.6!httpclient.jar (513ms)
[info] downloading https://repo1.maven.org/maven2/javax/activation/activation/1.1.1/activation-1.1.1.jar ...
[info] 	[SUCCESSFUL ] com.ning#compress-lzf;1.0.3!compress-lzf.jar(bundle) (261ms)
[info] downloading https://repo1.maven.org/maven2/org/xerial/snappy/snappy-java/1.1.2.6/snappy-java-1.1.2.6.jar ...
[info] 	[SUCCESSFUL ] org.glassfish.jersey.core#jersey-server;2.22.2!jersey-server.jar (531ms)
[info] downloading https://repo1.maven.org/maven2/org/glassfish/jersey/containers/jersey-container-servlet/2.22.2/jersey-container-servlet-2.22.2.jar ...
[info] 	[SUCCESSFUL ] org.apache.spark#spark-network-common_2.11;2.2.1!spark-network-common_2.11.jar (551ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/spark/spark-network-shuffle_2.11/2.2.1/spark-network-shuffle_2.11-2.2.1.jar ...
[info] 	[SUCCESSFUL ] org.glassfish.jersey.containers#jersey-container-servlet;2.22.2!jersey-container-servlet.jar (169ms)
[info] downloading https://repo1.maven.org/maven2/org/glassfish/jersey/containers/jersey-container-servlet-core/2.22.2/jersey-container-servlet-core-2.22.2.jar ...
[info] 	[SUCCESSFUL ] javax.activation#activation;1.1.1!activation.jar (180ms)
[info] downloading https://repo1.maven.org/maven2/mx4j/mx4j/3.0.2/mx4j-3.0.2.jar ...
[info] 	[SUCCESSFUL ] org.apache.spark#spark-network-shuffle_2.11;2.2.1!spark-network-shuffle_2.11.jar (158ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/spark/spark-unsafe_2.11/2.2.1/spark-unsafe_2.11-2.2.1.jar ...
[info] 	[SUCCESSFUL ] org.xerial.snappy#snappy-java;1.1.2.6!snappy-java.jar(bundle) (225ms)
[info] 	[SUCCESSFUL ] org.glassfish.jersey.containers#jersey-container-servlet-core;2.22.2!jersey-container-servlet-core.jar (100ms)
[info] downloading https://repo1.maven.org/maven2/io/netty/netty-all/4.0.43.Final/netty-all-4.0.43.Final.jar ...
[info] 	[SUCCESSFUL ] org.apache.spark#spark-unsafe_2.11;2.2.1!spark-unsafe_2.11.jar (93ms)
[info] downloading https://repo1.maven.org/maven2/net/java/dev/jets3t/jets3t/0.9.3/jets3t-0.9.3.jar ...
[info] 	[SUCCESSFUL ] mx4j#mx4j;3.0.2!mx4j.jar (140ms)
[info] downloading https://repo1.maven.org/maven2/javax/mail/mail/1.4.7/mail-1.4.7.jar ...
[info] 	[SUCCESSFUL ] javax.mail#mail;1.4.7!mail.jar (577ms)
[info] downloading https://repo1.maven.org/maven2/org/bouncycastle/bcprov-jdk15on/1.51/bcprov-jdk15on-1.51.jar ...
[info] 	[SUCCESSFUL ] net.java.dev.jets3t#jets3t;0.9.3!jets3t.jar (632ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/curator/curator-recipes/2.6.0/curator-recipes-2.6.0.jar ...
[info] 	[SUCCESSFUL ] io.netty#netty-all;4.0.43.Final!netty-all.jar (666ms)
[info] downloading https://repo1.maven.org/maven2/io/netty/netty/3.9.9.Final/netty-3.9.9.Final.jar ...
[info] 	[SUCCESSFUL ] org.apache.curator#curator-recipes;2.6.0!curator-recipes.jar(bundle) (540ms)
[info] 	[SUCCESSFUL ] io.netty#netty;3.9.9.Final!netty.jar(bundle) (553ms)
[info] downloading https://repo1.maven.org/maven2/com/clearspring/analytics/stream/2.7.0/stream-2.7.0.jar ...
[info] 	[SUCCESSFUL ] org.bouncycastle#bcprov-jdk15on;1.51!bcprov-jdk15on.jar (622ms)
[info] downloading https://repo1.maven.org/maven2/com/jamesmurty/utils/java-xmlbuilder/1.0/java-xmlbuilder-1.0.jar ...
[info] 	[SUCCESSFUL ] com.clearspring.analytics#stream;2.7.0!stream.jar (70ms)
[info] 	[SUCCESSFUL ] com.jamesmurty.utils#java-xmlbuilder;1.0!java-xmlbuilder.jar (57ms)
[info] downloading https://repo1.maven.org/maven2/net/iharder/base64/2.3.8/base64-2.3.8.jar ...
[info] 	[SUCCESSFUL ] net.iharder#base64;2.3.8!base64.jar (64ms)
[info] downloading https://repo1.maven.org/maven2/org/json4s/json4s-core_2.11/3.2.11/json4s-core_2.11-3.2.11.jar ...
[info] 	[SUCCESSFUL ] org.json4s#json4s-core_2.11;3.2.11!json4s-core_2.11.jar (136ms)
[info] downloading https://repo1.maven.org/maven2/org/json4s/json4s-ast_2.11/3.2.11/json4s-ast_2.11-3.2.11.jar ...
[info] 	[SUCCESSFUL ] org.json4s#json4s-ast_2.11;3.2.11!json4s-ast_2.11.jar (86ms)
[info] downloading https://repo1.maven.org/maven2/com/thoughtworks/paranamer/paranamer/2.6/paranamer-2.6.jar ...
[info] 	[SUCCESSFUL ] com.thoughtworks.paranamer#paranamer;2.6!paranamer.jar (54ms)
[info] downloading https://repo1.maven.org/maven2/org/scala-lang/scalap/2.11.8/scalap-2.11.8.jar ...
[info] 	[SUCCESSFUL ] org.scala-lang#scalap;2.11.8!scalap.jar (157ms)
[info] downloading https://repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.11.8/scala-compiler-2.11.8.jar ...
[info] 	[SUCCESSFUL ] org.scala-lang#scala-compiler;2.11.8!scala-compiler.jar (2033ms)
[info] downloading https://repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.11.8/scala-reflect-2.11.8.jar ...
[info] 	[SUCCESSFUL ] org.scala-lang#scala-reflect;2.11.8!scala-reflect.jar (655ms)
[info] downloading https://repo1.maven.org/maven2/org/scala-lang/modules/scala-xml_2.11/1.0.4/scala-xml_2.11-1.0.4.jar ...
[info] 	[SUCCESSFUL ] org.scala-lang.modules#scala-xml_2.11;1.0.4!scala-xml_2.11.jar(bundle) (139ms)
[info] downloading https://repo1.maven.org/maven2/org/scala-lang/modules/scala-parser-combinators_2.11/1.0.4/scala-parser-combinators_2.11-1.0.4.jar ...
[info] 	[SUCCESSFUL ] org.scala-lang.modules#scala-parser-combinators_2.11;1.0.4!scala-parser-combinators_2.11.jar(bundle) (101ms)
[info] downloading https://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.0.1/javax.ws.rs-api-2.0.1.jar ...
[info] 	[SUCCESSFUL ] javax.ws.rs#javax.ws.rs-api;2.0.1!javax.ws.rs-api.jar (67ms)
[info] downloading https://repo1.maven.org/maven2/org/glassfish/hk2/hk2-api/2.4.0-b34/hk2-api-2.4.0-b34.jar ...
[info] 	[SUCCESSFUL ] org.glassfish.hk2#hk2-api;2.4.0-b34!hk2-api.jar (75ms)
[info] downloading https://repo1.maven.org/maven2/org/glassfish/hk2/external/javax.inject/2.4.0-b34/javax.inject-2.4.0-b34.jar ...
[info] 	[SUCCESSFUL ] org.glassfish.hk2.external#javax.inject;2.4.0-b34!javax.inject.jar (59ms)
[info] downloading https://repo1.maven.org/maven2/org/glassfish/hk2/hk2-locator/2.4.0-b34/hk2-locator-2.4.0-b34.jar ...
[info] 	[SUCCESSFUL ] org.glassfish.hk2#hk2-locator;2.4.0-b34!hk2-locator.jar (98ms)
[info] downloading https://repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.2/javax.annotation-api-1.2.jar ...
[info] 	[SUCCESSFUL ] javax.annotation#javax.annotation-api;1.2!javax.annotation-api.jar (61ms)
[info] downloading https://repo1.maven.org/maven2/org/glassfish/jersey/bundles/repackaged/jersey-guava/2.22.2/jersey-guava-2.22.2.jar ...
[info] 	[SUCCESSFUL ] org.glassfish.jersey.bundles.repackaged#jersey-guava;2.22.2!jersey-guava.jar(bundle) (173ms)
[info] downloading https://repo1.maven.org/maven2/org/glassfish/hk2/osgi-resource-locator/1.0.1/osgi-resource-locator-1.0.1.jar ...
[info] 	[SUCCESSFUL ] org.glassfish.hk2#osgi-resource-locator;1.0.1!osgi-resource-locator.jar (55ms)
[info] downloading https://repo1.maven.org/maven2/org/glassfish/hk2/hk2-utils/2.4.0-b34/hk2-utils-2.4.0-b34.jar ...
[info] 	[SUCCESSFUL ] org.glassfish.hk2#hk2-utils;2.4.0-b34!hk2-utils.jar (69ms)
[info] downloading https://repo1.maven.org/maven2/org/glassfish/hk2/external/aopalliance-repackaged/2.4.0-b34/aopalliance-repackaged-2.4.0-b34.jar ...
[info] 	[SUCCESSFUL ] org.glassfish.hk2.external#aopalliance-repackaged;2.4.0-b34!aopalliance-repackaged.jar (55ms)
[info] downloading https://repo1.maven.org/maven2/org/javassist/javassist/3.18.1-GA/javassist-3.18.1-GA.jar ...
[info] 	[SUCCESSFUL ] org.javassist#javassist;3.18.1-GA!javassist.jar(bundle) (154ms)
[info] downloading https://repo1.maven.org/maven2/org/glassfish/jersey/media/jersey-media-jaxb/2.22.2/jersey-media-jaxb-2.22.2.jar ...
[info] 	[SUCCESSFUL ] org.glassfish.jersey.media#jersey-media-jaxb;2.22.2!jersey-media-jaxb.jar (68ms)
[info] downloading https://repo1.maven.org/maven2/javax/validation/validation-api/1.1.0.Final/validation-api-1.1.0.Final.jar ...
[info] 	[SUCCESSFUL ] javax.validation#validation-api;1.1.0.Final!validation-api.jar (62ms)
[info] downloading https://repo1.maven.org/maven2/com/fasterxml/jackson/module/jackson-module-paranamer/2.6.5/jackson-module-paranamer-2.6.5.jar ...
[info] 	[SUCCESSFUL ] com.fasterxml.jackson.module#jackson-module-paranamer;2.6.5!jackson-module-paranamer.jar(bundle) (60ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.6.5/hadoop-yarn-server-nodemanager-2.6.5.jar ...
[info] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-yarn-server-nodemanager;2.6.5!hadoop-yarn-server-nodemanager.jar (138ms)
[info] downloading https://repo1.maven.org/maven2/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar ...
[info] 	[SUCCESSFUL ] org.codehaus.jettison#jettison;1.1!jettison.jar(bundle) (73ms)
[info] downloading https://repo1.maven.org/maven2/org/codehaus/janino/janino/3.0.0/janino-3.0.0.jar ...
[info] 	[SUCCESSFUL ] org.codehaus.janino#janino;3.0.0!janino.jar (148ms)
[info] downloading https://repo1.maven.org/maven2/org/codehaus/janino/commons-compiler/3.0.0/commons-compiler-3.0.0.jar ...
[info] 	[SUCCESSFUL ] org.codehaus.janino#commons-compiler;3.0.0!commons-compiler.jar (59ms)
[info] downloading https://repo1.maven.org/maven2/org/antlr/antlr4-runtime/4.5.3/antlr4-runtime-4.5.3.jar ...
[info] 	[SUCCESSFUL ] org.antlr#antlr4-runtime;4.5.3!antlr4-runtime.jar (109ms)
[info] downloading https://repo1.maven.org/maven2/commons-codec/commons-codec/1.10/commons-codec-1.10.jar ...
[info] 	[SUCCESSFUL ] commons-codec#commons-codec;1.10!commons-codec.jar (122ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-common/1.8.2/parquet-common-1.8.2.jar ...
[info] 	[SUCCESSFUL ] org.apache.parquet#parquet-common;1.8.2!parquet-common.jar (65ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-encoding/1.8.2/parquet-encoding-1.8.2.jar ...
[info] 	[SUCCESSFUL ] org.apache.parquet#parquet-encoding;1.8.2!parquet-encoding.jar (95ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-format/2.3.1/parquet-format-2.3.1.jar ...
[info] 	[SUCCESSFUL ] org.apache.parquet#parquet-format;2.3.1!parquet-format.jar (118ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-jackson/1.8.2/parquet-jackson-1.8.2.jar ...
[info] 	[SUCCESSFUL ] org.apache.parquet#parquet-jackson;1.8.2!parquet-jackson.jar (200ms)
[info] downloading https://repo1.maven.org/maven2/jline/jline/2.12.1/jline-2.12.1.jar ...
[info] 	[SUCCESSFUL ] jline#jline;2.12.1!jline.jar (78ms)
[info] Done updating.
[warn] Found version conflict(s) in library dependencies; some are suspected to be binary incompatible:
[warn] 	* io.netty:netty:3.9.9.Final is selected over {3.6.2.Final, 3.7.0.Final}
[warn] 	    +- org.apache.spark:spark-core_2.11:2.2.1             (depends on 3.7.0.Final)
[warn] 	    +- org.apache.zookeeper:zookeeper:3.4.6               (depends on 3.6.2.Final)
[warn] 	    +- org.apache.hadoop:hadoop-hdfs:2.6.5                (depends on 3.6.2.Final)
[warn] 	* commons-net:commons-net:2.2 is selected over 3.1
[warn] 	    +- org.apache.spark:spark-core_2.11:2.2.1             (depends on 3.1)
[warn] 	    +- org.apache.hadoop:hadoop-common:2.6.5              (depends on 3.1)
[warn] 	* com.google.guava:guava:11.0.2 is selected over {12.0.1, 16.0.1}
[warn] 	    +- org.apache.hadoop:hadoop-yarn-client:2.6.5         (depends on 11.0.2)
[warn] 	    +- org.apache.hadoop:hadoop-yarn-api:2.6.5            (depends on 11.0.2)
[warn] 	    +- org.apache.hadoop:hadoop-yarn-common:2.6.5         (depends on 11.0.2)
[warn] 	    +- org.apache.hadoop:hadoop-yarn-server-nodemanager:2.6.5 (depends on 11.0.2)
[warn] 	    +- org.apache.hadoop:hadoop-common:2.6.5              (depends on 11.0.2)
[warn] 	    +- org.apache.hadoop:hadoop-yarn-server-common:2.6.5  (depends on 11.0.2)
[warn] 	    +- org.apache.hadoop:hadoop-hdfs:2.6.5                (depends on 11.0.2)
[warn] 	    +- org.apache.curator:curator-framework:2.6.0         (depends on 16.0.1)
[warn] 	    +- org.apache.curator:curator-client:2.6.0            (depends on 16.0.1)
[warn] 	    +- org.apache.curator:curator-recipes:2.6.0           (depends on 16.0.1)
[warn] 	    +- org.htrace:htrace-core:3.0.4                       (depends on 12.0.1)
[warn] Run 'evicted' to see detailed eviction warnings
[info] Compiling 1 Scala source to /home/dan/spark10/target/scala-2.11/classes ...
[info] Non-compiled module 'compiler-bridge_2.11' for Scala 2.11.8. Compiling...
[info]   Compilation completed in 11.116s.
[info] Done compiling.
[info] Packaging /home/dan/spark10/target/scala-2.11/simple-project_2.11-1.0.jar ...
[info] Done packaging.
[success] Total time: 63 s, completed Jan 11, 2018 6:13:54 PM
dan@h80:~/spark10 $ 
dan@h80:~/spark10 $ 
dan@h80:~/spark10 $

Next, I ran simple shell commands:

ls -la
ls -la target/
ls -la target/scala-2.11/
~/spark/bin/spark-submit --class "SimpleApp" --master local[4] target/scala-2.11/simple-project_2.11-1.0.jar

I saw this:


dan@h80:~/spark10 $ ls -la
total 28
drwxrwxr-x  6 dan dan 4096 Jan 11 18:22 .
drwxr-xr-x 46 dan dan 4096 Jan 11 18:06 ..
-rw-rw-r--  1 dan dan  137 Jan 11 17:45 build.sbt
drwxrwxr-x  3 dan dan 4096 Jan 11 18:12 project
drwxrwxr-x  2 dan dan 4096 Jan 11 18:22 spark-warehouse
drwxrwxr-x  3 dan dan 4096 Jan 11 17:46 src
drwxrwxr-x  4 dan dan 4096 Jan 11 18:13 target
dan@h80:~/spark10 $ 
dan@h80:~/spark10 $ 
dan@h80:~/spark10 $ 



dan@h80:~/spark10 $ ls -la target/
total 16
drwxrwxr-x 4 dan dan 4096 Jan 11 18:13 .
drwxrwxr-x 6 dan dan 4096 Jan 11 18:22 ..
drwxrwxr-x 4 dan dan 4096 Jan 11 18:13 scala-2.11
drwxrwxr-x 4 dan dan 4096 Jan 11 18:12 streams
dan@h80:~/spark10 $ 
dan@h80:~/spark10 $ 
dan@h80:~/spark10 $ 



dan@h80:~/spark10 $ ls -la target/scala-2.11/
total 20
drwxrwxr-x 4 dan dan 4096 Jan 11 18:13 .
drwxrwxr-x 4 dan dan 4096 Jan 11 18:13 ..
drwxrwxr-x 2 dan dan 4096 Jan 11 18:13 classes
drwxrwxr-x 5 dan dan 4096 Jan 11 18:13 resolution-cache
-rw-rw-r-- 1 dan dan 3691 Jan 11 18:13 simple-project_2.11-1.0.jar
dan@h80:~/spark10 $ 
dan@h80:~/spark10 $ 
dan@h80:~/spark10 $ 



dan@h80:~/spark10 $ ~/spark/bin/spark-submit --class "SimpleApp" --master local[4] target/scala-2.11/simple-project_2.11-1.0.jar
Lines with a: 39, Lines with b: 44
dan@h80:~/spark10 $ 
dan@h80:~/spark10 $ 
dan@h80:~/spark10 $ 

That seemed to go well so I tried the next example listed in the quick-start:

dan@pavlap:~ $ cd ~/spark/
dan@pavlap:~/spark $
dan@pavlap:~/spark $ bin/run-example SparkPi
Pi is roughly 3.1407957039785197
dan@pavlap:~/spark $
dan@pavlap:~/spark $
dan@pavlap:~/spark $

That seemed to go well so I tried the next example listed in the quick-start:

dan@pavlap:~/spark $
dan@pavlap:~/spark $
dan@pavlap:~/spark $ bin/spark-submit examples/src/main/python/pi.py
Pi is roughly 3.136320
dan@pavlap:~/spark $
dan@pavlap:~/spark $

That seemed to go well so I tried the next example listed in the quick-start:


dan@h80:~/spark $ 
dan@h80:~/spark $ ./bin/spark-submit examples/src/main/r/dataframe.R
Loading required package: methods

Attaching package: ‘SparkR’

The following objects are masked from ‘package:stats’:

    cov, filter, lag, na.omit, predict, sd, var, window

The following objects are masked from ‘package:base’:

    as.data.frame, colnames, colnames<-, drop, intersect, rank, rbind,
    sample, subset, summary, transform, union

Warning message:
package ‘SparkR’ was built under R version 3.4.2 
Spark package found in SPARK_HOME: /home/dan/spark
Java ref type org.apache.spark.sql.SparkSession id 1 
Warning message:
In Filter(nzchar, unlist(strsplit(input, ",|\\s"))) :
  bytecode version mismatch; using eval
root
 |-- name: string (nullable = true)
 |-- age: double (nullable = true)
root
 |-- age: long (nullable = true)
 |-- name: string (nullable = true)
    name
1 Justin
dan@h80:~/spark $ 
dan@h80:~/spark $ 
dan@h80:~/spark $ 

That seemed to go well so I was happy. That was the last example in the quick-start.

Scala Questions to Study

Class04 Lab


learn4.us About Blog Contact Class01 Class02 Class03 Class04 Class05 Class06 Class07 Class08 Class09 Class10 dan101 Forum Google Hangout Vboxen