I Am Chris

A Look at How Scala Compiles to Java

Wed, 11 Nov 2009 00:43:24 -0500

Consider this contrived example, based on an example from Beginning Scala. The point of the snippet was to demonstrate the congruency between using the higher-order functions map, flatMap, foreach, and filter (see Iterable), and performing the same operations inside a for comprehension.

object App {
	
	def isEven(i: Int) = i % 2 == 0
	
	def isOdd(i: Int) = i % 2 == 1
	
	def main(args: Array[String]): Unit = {
		val n = (1 to 10).toList
		n.filter(isEven).flatMap(i => n.filter(isOdd).map(j => i * j))
	}
	
}

Save this code to a file named App.scala, or anything you want (scala doesn’t have the same file/class name restrictions as java). Assuming you chose App.scala, compile it:

clewis$ scalac App.scala

Now check out the generated class files (ls App*class). You should see the following:

App$$anonfun$main$1.class
App$$anonfun$main$2$$anonfun$apply$1.class
App$$anonfun$main$2$$anonfun$apply$2.class
App$$anonfun$main$2.class
App$.class
App.class

Why were 6 classes compiled from this one singleton definition? Let’s start with App.class and App$.class.

The Singleton Object

Scala does away with Java’s statics. Instead we get singleton objects, which are declared with a syntax similar to that used for class declarations, but using the object keyword instead. When a singleton object shares the same name as a class, it becomes that classes “companion” object. Companion objects have special privileges, including access to private instance fields on instances of the companion type. In this example, App is simply a singleton object, since we haven’t also defined a class named App.

The Scala compiler implements companion objects by generating an anonymous class inside the companion class (the class of the same name as the object). The same is done for singleton objects, but the compiler also generates the containing class. Like the Java compiler, Scala names anonymous the class as [class name]$. Because we created an object named App, we get 2 classes: App and App$.

Higher Order Functions and Their Function Arguments

In Scala, functions are objects; instances of one of the FunctionN traits. Of course this knowledge isn’t obvious by the Scala syntax, and that’s part of the beauty: it just feels right. The compiler compiles any functions down to anonymous classes inside the containing class. If you understand Java’s scoping rules for inner classes, then you should now have some understanding of how Scala implements closures.

Methods in Scala are also different from functions. Methods are functions defined as part of a class definition with the the def keyword. Functions are instances of one of the FunctionN traits, and the Scala syntax provides several different ways to express them tersely. Methods are the only primitives (non-objects) in Scala, but the compiler makes it easy to promote methods to function instances. Back to our example, notice that we define the methods isEven and isOdd in our singleton object. They are method primitives, not functions. Because the compiler recognizes that methods often want to be treated as functions, it makes provisions; one such provision is that we can pass a method to a higher-order function.

Method Promotions

In order for the compiler to promote a method to a function instance, it must create an anonymous class for that instance. In our example, we first pass the method isEven as an argument to the higher-order function filter, and so the compiler generated the class App$$anonfun$main$1, which is our promoted method. Note that we also pass the isOdd to a subsequent call to filter, a promotion for which the compiler generated the class App$$anonfun$main$2$$anonfun$apply$1.

Function Literals

We’ve covered all but 2 of the generated classes. As it turns out, there are still 2 functions we haven’t discussed in the series of transformations. Take another look:

n.filter(isEven).flatMap(i => n.filter(isOdd).map(j => i * j))

Did you see them? If you’re new to Scala, you may not spot them at first because they are using function-literal syntax. The functions flatMap and map also take functions as arguments, and so we define them literally, embedding the one passed to map inside the one passed to flatMap. For these two function literals, the compiler generated the anonymous classes App$$anonfun$main$2 and App$$anonfun$main$2$$anonfun$apply$2.

To seal in what’s happening here, fire up the Scala REPL and run this code:

List(1,2,3).map(j => j + 1)

Now run this code:

List(1,2,3).map(new Function1[Int, Int] {
	override def apply(j: Int) = j + 1
})

The two do the exact same thing, because they are exactly the same.

The literal syntax may look strange, and even non-obvious at first. I thought so too, but it didn’t take long for me to greatly appreciate and recognize it as easily as you might recognize a class definition.

A Word on the Class Names

You may have noticed a pattern in the anonymous function class names. For starters, they’re prefixed with App$$anonfun$main$ and then continue in with 1.class and 2.class. There’s also another level nested inside the first level. If you inspect each of these using javap -c [class], you’ll notice the ordering and nesting follows the order in which we use function objects (by passing them as arguments to other functions).

isEven is the first function we pass, so it is the first function for which an anonymous class is generated, yielding App$$anonfun$main$1.
The second is the literal function passed to flatMap, so App$$anonfun$main$2 is generated for it.
The third is isOdd, yielding the class App$$anonfun$main$2$$anonfun$apply$1. This function is used inside the function passed to flatMap. It’s the first function in this scope, and it’s nested, so its name reflects its nesting and order in the sequence.
Finally we arrive at the literal function passed to map>, resulting in App$$anonfun$main$2$$anonfun$apply$2. It’s the second function referenced inside the function passed to flatMap, and so its name also reflects its nesting and order.

This ordering and nesting is deliberate. In Java, an inner class has access to its parent’s scope, which includes class instance variables and local variables (if the inner class was created in a method). This is what makes closures in Scala possible. Look again at the literal function passed to map:

n.filter(isEven).flatMap(i => n.filter(isOdd).map(j => i * j))

The function multiplies a “free” variable i by its single argument j. Where did i come from? Again, this function is defined inside another literal function passed to flatMap. That function receives a single argument, which it labels i. Because the function passed to map is nested inside the one passed to flatMap, it has access to that scope.

Lift Off

Fri, 30 Oct 2009 08:54:35 -0400

I arrived at the FGM building by way of my couchsurfing host, Jenny. At the moment, Reston looks a bit like London (at least if you look straight up). Fall has painted the leaves brilliantly, a sight I’m left wanting in Florida.

Jenny graciously dropped me off at the FGM building on her way to school (she’s a school teacher). I gladly accepted arriving an hour early for a free ride, and so I just started wandering the office park. I stumbled on this office “cafe” as it were, and here I sit. I seem to be just outside a free guest wifi network, but that’s ok. For a first-timer at an unconference, or any tech conference for that matter, I feel fairly prepared.

I’ve just made my way into the FGM suite. So far we’re about 32-30 strong!

Functional Content Negotiation in Scala

Thu, 15 Oct 2009 22:44:00 -0400

Today I needed some code to assess if an HTTP request matched certain criteria, based on its declared content type. Essentially, I needed to know if the user-agent wanted XML (or JSON), and that it wanted only that type.

Requirements:

Check the Accept request header for a given content type.
Support a non-standard X-Accept header serving the same purpose as Accept (IE doesn’t let XmlHttpRequest specify the Accept header).
The header value is specifically required to be one, and only one MIME type, without an explicit “q” value.

Assuming we have an HttpServletRequest instance that only accepts application/xml:

scala> req.getHeader("Accept")
res6: String = application/xml

and that we have an object with a method implementing the match algorithm, we should see this behavior:

scala> matcher accepts(req, "application/xml")
res7: Boolean = true

scala> matcher accepts(req, "application/json")
res8: Boolean = false

The complete code:

type HttpServletRequest = {
  def getHeader(h: String): String
}

object req {
  def getHeader(h: String) = h match {
    case "Accept" => "application/xml"
    case "X-Accept" => "text/html"
    case _ => null
  }
}

trait Acceptor {

  val headers = "Accept" :: "X-Accept" :: Nil

  def accepts(request: HttpServletRequest, contentType: String) =
    headers map (request.getHeader) filter(_ != null) map {
      _ == contentType
    } reduceLeft(_ || _)
}

object matcher extends Acceptor

Not much, but let’s look at it more closely.

type HttpServletRequest = {
  def getHeader(h: String): String
}

A type alias. This is a structural type, aliased as HttpServletRequest, with a method getHeader(String): String. This makes it easy to prototype the code in the scala REPL, or compile it against javax.servlet.http.HttpServletRequest.

Next, a singleton instance of the aliased type:

object req {
  def getHeader(h: String) = h match {
    case "Accept" => "application/xml"
    case "X-Accept" => "text/html"
    case _ => null
  }
}

This conforms to our requirements above, and also behaves like a real HttpServletRequest in that it returns the dreaded null if no header by the provided name exists.

Next is the trait:

trait Acceptor {

  val headers = "Accept" :: "X-Accept" :: Nil

  def accepts(request: HttpServletRequest, contentType: String) =
    headers map (request.getHeader) filter(_ != null) map {
    _ == contentType
  } reduceLeft(_ || _)

}

Notice the headers val. This is a list of strings containing the header names to read from the request and interpret as Accept header values.

The fun part is the accepts method, which takes a servlet request (conveniently aliased for REPL play), and a content (MIME) type. This is purely functional; there are no side-affects, it is deterministic, and the algorithm is a series of four transformations occurring on one collection after another. Through the transformations, the collection is reduced to a single boolean value. Let’s start with the first line:

headers map (request.getHeader) filter(_ != null)

Starting with the list of header names to inspect in the request, we map it with request.getHeader. This creates a new list, where each element is the result of applying request.getHeader to each element in the original headers list. The original list contains 2 elements, “Accept” and “X-Accept”, and so the resulting list now looks like this: List(application/xml, text/html). Our request object responded to the query for “Accept” with “application/xml”, and “text/html” to “X-Accept.” Had either of those header names not been present, a null value would now be present in the list.

Now we filter this resulting list. Filter, like map, applies a function to each element in the collection and results in a new collection. The function receives each element and returns a boolean value. For each invocation that results in true, that element is added to the resulting list (you filter a collection using a function). We use a literal shorthand to assert that the element is not null, and so the resulting list will contain non-null strings. This filtering is simply protection against null. Our transformed list doesn’t contain any nulls, so the new list looks the same as the old one: List(application/xml, text/html).

Next we have another mapping operation, using brackets instead of parentheses.

map {
  _ == contentType
}

The use of brackets here is a style choice, and in this case is the same as writing map(_ == contentType). This part also evolved from a partial function, where the body is a case statement, matching on the single argument. The function being mapped simply checks if each element is equal to the contentType argument. Remember that == in scala, contrary to java, is an object equality test and not an identity test (eq is for identity tests). We left off at List(application/xml, text/html), and after this mapping transformation, arrive at List(true, false).

The final transformation, simple as it is, actually took me a bit to realize. The function itself had undergone a few iterations, each representing in some way, a group of header values in the request that did or didn’t match a given content type. I couldn’t seem to reach a solution that felt natural. I thought about filtering on true, and then checking that the result list wasn’t empty, but that didn’t really express the essence of the computation. Same for a find (and then matching on Option(true)). They would have worked, but they missed the point and, as all point-missers do, cluttered the algorithm.

Then it hit me:

reduceLeft(_ || _)

Exactly what I wanted. reduceLeft receives a 2 argument function, walks the contents of the collection, and returns the accumulated result. On each invocation, it passes the current and the next element as arguments, progressing from left to right, until the collection is exhausted. The result of one invocation becomes the input to the next. As a simple example, consider list summations:

scala> List(1,2,3) reduceLeft(_ + _)
res14: Int = 6

scala> List(1,2,3,4) reduceLeft(_ + _)
res15: Int = 10

Back to our exmaple, using function-literal shorthand, I OR the two arguments. This means that if either of the two are true, the resulting value is true. Reducing a list of booleans with this function results in a single boolean value. Because I’m using OR, a single true value in the list will result in a true result at the end of the reduction. We left off at a List(true, false), and after our reduction, arrive at a single true result. Perfect.

Last of all, a singleton that mixes in the Acceptor trait.

object matcher extends Acceptor

This just gives us an object with Acceptor behavior mixed in, so we can play in the REPL.

"Great minds discuss ideas; Average minds discuss events; Small minds discuss people."

Mon, 12 Oct 2009 21:59:25 -0400

“Great minds discuss ideas; Average minds discuss events; Small minds discuss people.”

- Eleanor Roosevelt

A Solution to the Set Combinations Problem

Mon, 12 Oct 2009 21:58:13 -0400

def combine[T](sets: List[List[T]]): List[List[T]] = sets match {
  case List(l1, l2) => l1.flatMap(i1 => l2.map(List(i1, _)))
  case head :: tail => combine(tail).flatMap(set => head.map(_ :: set))
  case _ => sets
}

Recursion with an Arbitrary Number of Lists

Tue, 29 Sep 2009 13:00:18 -0400

{ apple, banana }
{ milk, cheese, yogurt }
{ nuts, legumes }

— would result in —

{ apple, milk, nuts }
{ apple, milk, legumes }
{ apple, cheese, nuts }
{ apple, cheese, legumes }
{ apple, yogurt, nuts }
{ apple, yogurt, legumes }
{ banana, milk, nuts }
{ banana, milk, legumes }
{ banana, cheese, nuts }
{ banana, cheese, legumes }
{ banana, yogurt, nuts }
{ banana, yogurt, legumes }

Dependency Injection Using Language-only Constructs

Sun, 16 Aug 2009 22:29:40 -0400

I’ve been exploring dependency injection in Scala. Digesting this article by Debasish Ghosh, I was inspired to play with the code. At the time of the article’s posting it needed a few pieces to compile. You can find the inferred working source here. As an exploratory exercise, I transposed it to java. Chew on it a bit; it clarified some things for me about both the technique and the workings of Scala abstractions.

Vote for ROSA LOVES!

Thu, 16 Jul 2009 16:36:15 -0400

Vote for ROSA LOVES!

Super quick start with Lift's ProtoUser

Mon, 13 Jul 2009 21:44:00 -0400

Lift provides a trait called ProtoUser, which helps you in quickly adding base functionality to your application for user-specific data. The Lift book mentions this class very briefly, but it makes the assumption that the reader will be to intuit how to complete the example code to have a base starting point. Here’s what you must have (assuming this is in the model directory):

package app.model

class User extends ProtoUser[User] with LongKeyedMapper[User] {
def getSingleton = User
}

object User extends User with LongKeyedMetaMapper[User]

You need to mix in the LongKeyedMapper trait, and you need the User singleton.

Erin nailing the word but missing the number - by far the best...

Sun, 05 Jul 2009 21:32:57 -0400

Erin nailing the word but missing the number - by far the best sparkler art of the night!

erinlewis:

this year.

titled: “happy backwards four + the letter t”

erinlewis: these are from last year. hope to make lots more...

Sat, 04 Jul 2009 13:04:26 -0400

erinlewis:

these are from last year.

hope to make lots more tonight.

happy 4th.

http://groups.google.com/group/the-lift-book/files

Sat, 04 Jul 2009 12:58:50 -0400

http://groups.google.com/group/the-lift-book/files:

The Lift Book group provides the built source of the full book “The Definitive Guide to Lift.” Grab the full PDF here (master.pdf).

Lift Walk-through at Developerworks

Sat, 04 Jul 2009 12:53:00 -0400

Lift Walk-through at Developerworks

Lift Framework API

Sat, 04 Jul 2009 12:50:00 -0400

Lift Framework API

Dear Neighbors

Mon, 22 Jun 2009 08:37:38 -0400

I live in St. Augustine Beach. If you live in the low-numbered streets or the early letter streets, chances are we’ve seen each other. I’m writing to tell you that yesterday, Sunday, June 21, my house was broken into and robbed. Our door was locked as it always is when we are gone, but the door was kicked in. Noone was harmed (noone was home), and fortunately no major damage was done to the house. However…

MY LAPTOP WAS STOLEN

Being a software engineer, this is a painful blow. If you live anywhere near me, I ask of you 2 things.

1) Please contact me if someone you know comes in to a new MacBook Pro 17in glossy screen. Model from late 2007 (pre black monitor trim).

2) Get an alarm system and a strong door. Apparently these kinds of robberies are on the rise in our area. What a shame.

"In many “OO” languages, especially those with dot notation, the methods are owned by the..."

Sun, 21 Jun 2009 22:07:00 -0400

“In many “OO” languages, especially those with dot notation, the methods are owned by the class, and you cannot access a method without compiler promises that the object it will call that method with is of a class that owns that method. This is not object-orientation, it is just plain lunacy.”

- Erik Naggum, comp.lang.lisp

100+

Sun, 21 Jun 2009 09:29:50 -0400

It’s been a nice weekend. Great time with friends on Friday night, Saturday and Sunday at the in-laws, and lots of ~~hacking~~ relaxing interspersed throughout. This week I’ll be giving an expanded presentation on Scala, which is where most of my hack time has been spent.

Bonus: Erin is playing the piano

http://delicious.com/chris_lewis

Sun, 21 Jun 2009 09:19:56 -0400

http://delicious.com/chris_lewis:

I realized that for a while I was using tumblr in cases where delicious would have better fit. Sooo - here’s my delicious page.

Wishlist

Mon, 13 Apr 2009 09:47:53 -0400

I’ve created a book a book wishlist at Amazon. If you feel compelled to hook me up, that’d be a good place to look :-)

Parleys: The Feel Of Scala

Sat, 04 Apr 2009 19:11:32 -0400

Parleys: The Feel Of Scala