Cannot run scalding-repl

[transistor]

I cannot seem to launch the scalding REPL. I have tried in three separate environments: OS X 10.14 (Mojave), and Debian linux (Stretch) within Virtualbox, and on a bare metal machine.

From the Scalding-REPL page:

Starting up the REPL in local mode is as easy as:
./sbt "scalding-repl/run --local"

When I try this, I end up with the same error on all systems:

[warn] Executing in batch mode.
[warn]   For better performance, hit [ENTER] to switch to interactive mode, or
[warn]   consider launching sbt without any commands, or explicitly passing 'shell'
[info] Loading project definition from /root/scalding/project
[info] Resolving key references (16977 settings) ...
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[info] Set current project to scalding (in build file:/root/scalding/)
[error] Expected configuration
[error] Expected ':' (if selecting a configuration)
[error] Expected key
[error] Expected '::'
[error] Expected end of input.
[error] scalding-repl/run --local
[error]                  ^

My build steps prior to this point were to install a JRE/JDK, scala, sbt, followed by a clone and build of the Scalding repo.

For example, on Debian:

# operate as superuser
sudo su

# General update
apt-get update && apt-get upgrade

# java 
apt-get install -y  default-jredefault-jdk

# scala
apt-get install scala

# sbt: from .deb
cd /tmp
wget https://dl.bintray.com/sbt/debian/sbt-1.2.8.deb
dpkg -i sbt-1.2.8.deb
apt-get install -f

# scalding
apt-get install -y git
git clone https://github.com/twitter/scalding.git
cd scalding
sbt update
sbt assembly    # 40 - 50 minutes; some tests fail

# gradle
apt-get install -y gradle

# Errors when trying to launch scalding repl (see above)
./sbt "scalding-repl/run --local"    # fail

I looked at the readme in the scalding/scalding-repl directory. Trying to launch the repl using scald.rb also failed to work:

./scripts/scald.rb --repl --local

When trying to run the repl, the /root/scalding/scalding-repl/target/scala-2.11/scalding-repl-assembly-0.18.0-SNAPSHOT.jar is missing, you probably need to run ./sbt scalding-repl/assembly

If I try to run that command, it fails on one test, and no jar is generated:

./sbt scalding-repl/assembly
...
[info] - should enumerate matching files
[info] - should ignore directories with restricted permissions *** FAILED ***
[info]   List(/tmp/scalding-repl4084457038924675121/this_matches) did not equal List() (ReplTest.scala:213)
[info]   org.scalatest.exceptions.TestFailedException:
[info]   at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:528)
...
[info]   at sbt.ForkMain$Run$2.call(ForkMain.java:286)
[info]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[info]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[info]   at java.lang.Thread.run(Thread.java:748)
[info] ScalaTest
[info] Run completed in 2 minutes, 19 seconds.
[info] Total number of tests run: 32
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 31, failed 1, canceled 0, ignored 0, pending 0
[info] *** 1 TEST FAILED ***
[error] Failed: Total 32, Failed 1, Errors 0, Passed 31
[error] Failed tests:
[error] 	com.twitter.scalding.ReplTest
[error] (scalding-repl/test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 148 s, completed Apr 9, 2019 5:38:38 AM

find scalding-repl "*.jar"    # None found

On another issue page, I learned that I can run ./sbt scalding-repl/console, but I can't seem to do anything useful with that. It launches a scala repl (not scalding), and then begins to print various warnings:

             import com.twitter.scalding._
                                         ^
<console>:13: warning: Unused import
             import com.twitter.scalding.ReplImplicits._
                                                       ^
<console>:14: warning: Unused import
             import com.twitter.scalding.ReplImplicitContext._
                                                             ^

At this point, I can use things like TypedPipe, but the above errors are printed every time I submit a command.

At this point, all I can ask is: what am I doing wrong?

[johnynek]

There are three separate issues at play here:

1. the instructions for running with sbt have broken since an sbt update it appears. I can run with sbt declaring the main directly:

scalding-repl/runMain com.twitter.scalding.ScaldingShell --local

unfortunately, #1021 has regressed, and we see the same bug. You can just change the hadoop-client line in the scaldingRepl section to:

"org.apache.hadoop" % "hadoop-client" % hadoopVersion, 

, then the above runMain works for me.
2. a repl test does not pass on your machine (we do run tests on CI, so that is something we can dig into). On my machine, as on CI, scalding-repl/test does pass. You can uncomment the second line:

  // Uncomment if you don't want to run all the tests before building assembly
  // test in assembly := {},

so it just says: test in assembly := {} and it should not bother to run the tests before making the assembly. That should also unblock you.
3. scala's built in repl interacts badly with unused variable warnings: https://www.google.com/search?client=firefox-b-1-d&q=scala+repl+unused
That said, using the console does miss out on some implicits that the scalding-repl sets up, so this isn't probably the best way to go.

Sorry you hit all these issues. Hopefully most people don't hit issue 2, since the tests pass on their machines, but we should fix 1.