Conversations about Software Engineering

Conversations about Software Engineering (CaSE) is an interview podcast for software developers and architects about Software Engineering and related topics. We release a new episode every three weeks.

Transcript

Stefan Tilkov: Welcome, listeners, to a new episode of the CaSE Podcast, another conversation about software engineering. This is Stefan Tilkov.

Stefan Tilkov: My guest today is Daniel Westheide. Daniel is a colleague of mine at innoQ and he works as a senior consultant, and he's also well-known in the Scala community because he's a frequent conference speaker, and also the author of the blog and the book The Neophyte's Guide to Scala. Welcome, Daniel!

Stefan Tilkov: Daniel Westheide:[00:00:27.26] Hello, Stefan!

Stefan Tilkov: Do you actually prefer to be called Daniel when you speak English, or do you prefer Daniel as we would say in German?

Daniel Westheide: Well, I'm used to be called Daniel whenever I speak English, so that's fine.

Stefan Tilkov: Let's stick to Daniel then. Daniel, today we're going to talk about advanced Scala topics. We had a podcast about Scala basics recently, so we thought that this time we might add to that and talk about some of the more advanced features. And that's very interesting to me, because I very much am a Scala newbie, I've only dabbled a bit in the language, so you can be sure that I will ask a lot of dumb questions.

Daniel Westheide: I'm looking forward to that. I must say that the first Scala podcast recently, there were actually also some advanced topics already, so let's see.

Stefan Tilkov: Very true.

Daniel Westheide: If you remember, they mentioned a monad, so...

Stefan Tilkov: They mentioned the m-word, but I think we're not going to mention the m-word today, are we?

Daniel Westheide: No, we're actually not doing that at all.

Stefan Tilkov: That's kind of weird; it's what you'd expect in an advanced podcast... Anyway, let's get started. We decided to talk about a few of the features that people find appealing in Scala, and maybe talk about some of the downsides or the challenges with them. Let's start with what I consider to be one of the nicest selling points for Scala - not knowing what I'm talking about, as usual - case classes. Are they the killer feature for Java converts?

Daniel Westheide: Well, for Java people that are coming to Scala, it's often one of the nicest things that you immediately recognize as something that adds value to what they are familiar from Scala.

Stefan Tilkov: Can you briefly explain what case classes are?

Daniel Westheide: Case classes are classes where you only specify the fields that these classes have, and then a lot of code is generated for you. Actually, you only specify the construct, and then they are immediately turned into immutable public fields by the compiler, and it also generates an equals and a hashCode method, and a Scala companion object with a Factory Method called 'apply', which makes it nicer to instantiate values of this case class.

Daniel Westheide: It also generates a copy method, so this is something that you also need to handcraft in Java as a copy constructor, usually. So this copy method has named and default arguments, so when you have one value of the case class, you can immediately create a new value from it just changing a single field, for example.

Daniel Westheide: Another thing, of course, is that they can be destructured in a pattern-matching expression... Where pattern-matching is something not at all available in Java, but all the other things are typical boilerplate code things that Java developers have to write a lot.

Stefan Tilkov: That actually sounds pretty great. Is there any problem with it?

Daniel Westheide: Well, there are some problems... Maybe let's first speak about why they exist and also why they are useful. In functional programming we like to separate data from behavior, so data types are typically transparent, so their structure really becomes a public API. As I said, you can destructure case classes in a pattern-matching expression, so you really see their structure and their internals. That's great for defining algebraic data types. There are some in the standard library, like Option, which only has two possible subtypes - Some and None. Or a similar type like Either, but even List is an algebraic data type, a link list, because it only has the Nil value or the Cons value. Lisp programmers might be familiar with that, of course.

Daniel Westheide: For that, case classes are really nice. If you're coming from object oriented programming, you'll usually tend to use opaque data types instead, where the internals are encapsulated. And of course, there are good reasons for both, but this opaqueness I think is really a good choice if you want data types that enforce their invariance. If you want to do that with case classes, you suddenly have to write a lot more code, and it's also quite easy to get this wrong.

Stefan Tilkov: Sorry for interrupting you, but didn't you say that case classes always were immutable?

Daniel Westheide: The fields are...

Stefan Tilkov: Okay, the fields are immutable.

Daniel Westheide: By default. You can make them mutable, but that's an explicit choice and actually not recommended.

Stefan Tilkov: I was kind of wondering, because you mentioned invariance, and actually with an immutable object, I don't see anything that could vary.

Daniel Westheide: Well, the invariance, you usually have to assert them at construction time. That's also something you would do in a Java class. Maybe you have some class called 'area', where there's a width and a height, and you want to assert that they are not negative, or greater than zero, something like that. This would be an assertion in the constructor, in the Java world.

Stefan Tilkov: And it could be the same thing in a case class...

Daniel Westheide: In the case class you could do the same thing. You could just throw an exception in the constructor if such an invariant is violated. But in functional programming we don't like to throw exceptions, of course, because we want to maintain referential transparency. Functional error handling looks a bit different. We capture potential errors in the return type of a function.

Stefan Tilkov: Can you briefly explain those two technical computer science terms you used? The first one was algebraic data types, and the second one was referential transparency.

Daniel Westheide: There are actually two possible types of algebraic data types. One is a sum type, and the other is a product type. Basically, you can think of a sum type like an enumeration of possible types. As I said with the option type, it can either be a Some, with some value in it, or a None, representing no value. So this is just a fancy term for that.

Daniel Westheide: The product type - any case class is a product type; the algebra is defined by the combination of the fields of the case class. So this is an algebraic data type, and referential transparency is something we always aim for in functional programming, at least when we want to do purely functional programming, where there are no side effects.

Daniel Westheide: The idea is that you can always replace an expression with a result of that expression, without making a difference. This is like mathematical equations. But if you throw an exception and catch it, it actually makes a difference when you replace an expression with its result, so then this is violated. We try to avoid that by not throwing exceptions at all.

Daniel Westheide: Of course, it's not easy if you actually integrate with the Java ecosystem, where exceptions are thrown all over the place, but there are techniques for capturing those early in the integration layer. Then you would turn those into certain types that capture the possibility of an error.

Daniel Westheide: The easiest would be an Option - it would just say "There is a value" or "There's no value", or with an Either, you can have either an error or the actual return type. So this is a disjunction. In Scala and other type function languages we use types like these to capture errors. But of course, you can't do that with a constructor, right? A constructor isn't a function, so what you would need to do is to turn the constructor into a private constructor, and write your own factory method on the companion object that returns such a functional error type, and then you don't have to throw exceptions anymore.

Stefan Tilkov: Why isn’t the constructor another function?

Daniel Westheide: I guess that's a JVM technicality. In Java, the constructor is also not a function, right? It's something special.

Stefan Tilkov: Okay. I was thinking about whether there's any in an abstract way, because the function that returns an object based on the input parameters -- but what you're referring to was the technical implementation where a constructor is a different thing with the VM.

Daniel Westheide: Yes. I would certainly like it if a constructor was already just another function, but that's not the case on the JVM, of course.

Daniel Westheide: There's another problem, of course, because this generated copy method - it allows us to create a new value of our data type by changing some field; so even if we have a private constructor and a nice Factory method, it would still be possible to first create a legal value of our type, and then change it and create a new value from it that is illegal, by just setting a field to something it shouldn't be.

Daniel Westheide: There are some tricks to avoid that, but it's quite complicated and you have to write a lot of code. A nice alternative to that is to not use a case class at all for such an opaque data type. Of course, you can just handcraft everything, but there's also a nice annotation from the Scalameta project, which is called @data. With that, you can specify things that should generate for you. It can actually generate all the things that a case class would generate for you, but you can also tell it to not generate certain things, so that it leaves out a copy method for example, or it doesn't create an Apply method, or it's not possible to destructure it in pattern-matching... Whatever you want.

Stefan Tilkov: That sounds like that @data annotation would probably use Scala's built-in macro facilities to do that. Is that the right term, even in Scala?

Daniel Westheide: Scalameta itself is a macro library, so it allows you to write annotation macros, so that's how it's implemented. There's also macros in the Scala standard library, but they are a bit different and deprecated by now, as far as I know. There was a first attempt at creating a macro system for Scala, and there have been various attempts since then, and Scalameta is one of them.

Stefan Tilkov: Do you see any chance that something like this might become the standard way of doing things, or do you have to make a choice of the right library to use?

Daniel Westheide: I think there are now attempts to create a new standard Scala macro system, which is not exactly Scalameta, but all the experience from that - I think it will be influencing that. Currently, there's still a wide range of libraries that are using that old macro system, and I guess it won't completely go away so soon, but at some point everything will have to migrate to the new way of doing macros.

Stefan Tilkov: To maybe sum up the case class discussion, I gather they're only supposed to be used in very simple cases, and as soon as you notice that you're fighting against what's been generated, it's better to switch to a different method. Is that a fair summary?

Daniel Westheide: Yes. It's really nice if you want to define your own algebraic data types, which sometimes makes a lot of sense, but if you have a use case where you want some opaqueness, where you need to enforce some invariants in your data type, then they are probably not a good choice. There are some blog articles discussing this, which we can link to in the show notes, I guess.

Stefan Tilkov: Perfect. Okay, on to the next topic, then. I keep hearing about those fancy type classes, and somehow implicits seem to be associated with that, so you need to explain to me what that actually is.

Daniel Westheide: Those are things that are widely used in the Scala community, on the Scala ecosystem, but it's also something that newcomers to the language are quite scared about. The short answer to what it is - they are an alternative to polymorphism by subtyping, and they originally come from the Haskell language. But while they are a first-class language construct in Haskell, that's not the case in Scala. They're more like a pattern, and you have to use some other language features to implement that pattern.

Stefan Tilkov: Excuse me... Maybe let's talk first about subtype polymorphism - what's that?

Stefan Tilkov: Daniel Westheide:[00:15:23.29] Let's say that you have a method called 'sort', which allows you to sort a list of integers, right? That's pretty easy to implement, but at some point maybe you think it's not generic enough. You want to be able to sort any list of a type that is sortable, or orderable. If you want to do that, you would introduce a type parameter to that method, and it will no longer get a list of integers, but a list of this type parameter. Let's call it A. That's called parametric polymorphism. But now we don't know anything about A, because it could be anything, right? So we really cannot implement 'sort' anymore.

Stefan Tilkov: What we need to do is to add a constraint to our type parameter, and if we use subtype polymorphism, which is what you're usually doing in the Java world, or object oriented programming in general, then you would say "My type parameter A has to be a subtype of Comparable" in Java, for example. In Scala you could also do that; there's a trait called 'Ordered', which any class can extend, which even extends Java's Comparable. So this is how you could do it.

Stefan Tilkov: So my 'sort' method would just invoke methods of the comparable type, or sortable, or whatever the name is, and so my code will compile because the type tracker would validate that these methods are actually available in that certain type.

Daniel Westheide: Yes, exactly. And of course, this works, but now we can only use our 'sort' method with types that we can control, so that we can make them inherit from comparable, or ordered, or whatever. But we can't use it with types we don't control - some external libraries, for example.

Daniel Westheide: With type classes, we have something called ad-hoc polymorphism, which means that we actually do have the chance to make our method polymorphic for types that we do not control, and we'll get to how that works. So we still need to put a constraint on our type parameter A sort method, but this time the constraint would be that there must be some evidence to the compiler that there is an instance of the ordering type class for any concrete type A that we want to use here.

Stefan Tilkov: Okay, wait a minute. So what you're saying is I don't need to make that class implement, or what was the example...?

Daniel Westheide: Yes, we don't need to implement ordered, or extended...

Stefan Tilkov: Yes, because maybe it has been written by somebody else and I can't make them implement my library, or maybe it would introduce a cycle of dependency that I don't want to accept. So what I'm doing instead is I'm telling the compiler or I'm asserting that there is something that can convert that thing -- is that what you're saying? What is the actual assertion, what's the actual statement that I'm making?

Daniel Westheide: Well, in type class terms on a conceptual level you would say that there's evidence that there's an instance of that type class for your concrete type.

Stefan Tilkov: Okay, so I need an instance of a type-- sorry, go ahead.

Daniel Westheide: What that means in Scala - I think we'll probably get to that in a few minutes...

Stefan Tilkov: Okay, so let me ask on the conceptual level first, just to get the terminology straight, because I keep mixing it up and only half-understanding what type classes are. So an instance of a type class for that type - is that the terminology you used?

Daniel Westheide: Yes.

Stefan Tilkov: So I would have an instance of ordered for int, and I would have an instance of ordered for string.

Daniel Westheide: Yes, exactly.

Stefan Tilkov: So a type class instance is something that exists once for each type that I want to apply to, or define it for, or...?

Daniel Westheide: In Haskell, it can only exist as a single implementation for each type. In Scala it is actually possible to have different implementations, so different instances of that type class for a specific type. That's a difference that is due to the way that type classes as a pattern are implemented in Scala.

Stefan Tilkov: How is that similar or different from other languages? For example, Clojure has protocols. Is that the same thing?

Daniel Westheide: Clojure and also Elixir both have this thing called protocols, right? Actually, that's quite similar. Both solve the so-called expression problem, that we can make arbitrary types conform to some interface, or provide some behavior or capability (like ordering) without having to recompile those sources for that type, or for our interface. I'm not sure if everyone is familiar with Clojure or Elixir protocols.

Stefan Tilkov: Probably not, and I don't think we had a Clojure episode yet, but I think we should do one. Anyway... That means that for each type that I want to instantiate that type class for (is that the way to say it...?), I have to write some code that sort of mediates between the existing type and the interface that I wanna provide, correct?

Daniel Westheide: Yes, that's true.

Stefan Tilkov: How do I do that? How do I implement a type class?

Daniel Westheide: To answer that, it's probably best first to understand how a type class itself is defined, and that's actually quite simple.

Stefan Tilkov: So what you're saying is I'm not implementing it yet for a specific type, I'm just defining it first, in an abstract way?

Daniel Westheide: No, what I'm saying is someone has to say that there is an ordering type class, for example.

Stefan Tilkov: Independent of the particular type that I'm going to instantiate it for...

Daniel Westheide: Yes. Ordering is already in the standard library, but it's interesting to see what that actually looks like, in order to know how to implement it for your own type. And of course, you could also define your own type classes at some point, but that's a bit more advanced.

Daniel Westheide: A type class like ordering is just a trait with a type parameter A, and it would have one or more methods. These methods usually take one or more arguments of this type A, or they return something of that type A. It really depends on the type class, but in the case of ordering of A, you would have a method called 'compare', which takes two values of A and returns an integer. This is exactly like Java's comparator interface, and it actually even extends Java's comparator interface, which is interesting.

Daniel Westheide: A comparator in Java also allows you to implement comparison functionality for types you don't control, but the difference of course is that you have to pass that comparator to your function as an explicit argument.

Stefan Tilkov: You have to explicitly instantiate it in order to do it.

Daniel Westheide: Yes. But the shape of a type class in general is usually quite similar to this comparator interface in Java. So if you are familiar with that, you should have a good idea of what type classes in Scala look like, how they are defined.

Daniel Westheide: Now, the question you had is "How do I define instances of my type class?" What you would usually do is you implement it by implementing the ordering trait, just as you would implement a comparator in Java. But then usually, what people do is do that as an anonymous class, and assign this to a value identifier, so a val of type ordering of int, for example.

Daniel Westheide: The interesting thing about this val is it has to have the implicit keyword, which we haven't really talked about yet... Just in the beginning, when we mentioned type classes. But this implicit keyword is important because methods that want to be constrained by this type class, like our Sort method, they make use of this implicit resolution mechanism... Because the Sort method would actually get a second parameter list. So it will not only get the list of A as an argument, but it will also have an implicit parameter list, in which it expects an ordering of A.

Daniel Westheide: Of course, you can explicitly pass in your defined ordering of int here, but you don't have to do that, whereas in Java you always have to pass in your comparator as a second argument explicitly. But since our parameter is implicit and we have defined an implicit value, the compiler will try to find a matching implicit value, and it will find our implicit ordering of int, in this case.

Stefan Tilkov: So it'll just look for a val in the lexical environment, and use that.

Daniel Westheide: Yes. It can be defined locally, it can be defined in some trait we extend, or it can be in an import... So there are different ways of getting it into our scope, but at the end of the day it will hopefully find an instance. And if it doesn't, it will not even compile, of course.

Stefan Tilkov: It reminds me a bit of dependency injection, or something like the @Injet annotation a little bit.

Daniel Westheide: There are actually some people who use implicit resolution also to inject dependencies in Scala, so that's also possible. It's a different way of using implicits than for type classes, but it's also possible.

Stefan Tilkov: So probably type classes are a more specific thing, right?

Daniel Westheide: Yes, they are a specific way of using implicits. But another way of using implicits would also be for dependency injection.

Stefan Tilkov: So what will happen is then the type will be automatically converted or wrapped or decorated in that type class, or handled by that type class instance when I call methods like that... Is that how it works?

Stefan Tilkov: I don't think I've got it yet... Let's say I pass in something of type A, but this A type doesn't have my compare method or whatever it is that I need to do the sorting. Can I still do a dot-'compare' … no, I think I get it now, I will just call a free function method compare and then I will pass in the two A's.

Daniel Westheide: Actually, you wouldn't call compare on the value of A, but you have this implicit type class parameter, which can have any name. Let's call it ordering in our case.

Stefan Tilkov: And I would do ordering.compare.

Daniel Westheide: So you would call ordering.compare and pass in the two values.

Stefan Tilkov: Now I got it. Okay, makes sense. It doesn't sound as fancy as I expected it to be, actually.

Daniel Westheide: Yes, it's not fancy. If you know Comparator from Java, basically, you have almost understood type classes in Scala.

Stefan Tilkov: Okay, so you've managed to get this into my thick head, which is a success, the first one, now we actually wanted to talk about move advanced stuff, not educate me about the basics... So what are some of the problems of type classes or implicits? What problems do people have with them?

Daniel Westheide: I think because they are not first-class constructs, they were actually quite heavy on the syntax. So method signatures with type class constraints can be a bit difficult to read, especially if the constraint is actually on multiple type classes, which could also be the case. So I guess especially for newcomers to Scala, that can be a bit scary.

Daniel Westheide: Another thing is that this resolution mechanism for implicit parameters is often not well understood by people who are new to Scala. You really have to dig deep into how this works in order to understand why for example a certain implicit value was chosen over another. Because as I said, unlike in Haskell, you can define multiple ordering of ints, for example, as implicit values, and which one is chosen is totally dependent on this resolution mechanism, which looks in a specific order.

Daniel Westheide: For example, local implicit values are always chosen over imported ones, and they are chosen over mixed in ones, and stuff like that. If you don't know this resolution mechanism, you might be surprised at the behavior.

Daniel Westheide: Another thing is that some libraries, in my opinion, misuse this typeless feature a bit, because they define their own type classes, and then they provide some default instances of these type classes for certain standard types, and sometimes that default behavior is a bit unexpected, and of course, you get it automatically sometimes. Sometimes it's difficult to opt out of these default instances if they are not well designed, and that can also be frustrating for people.

Stefan Tilkov: Okay. Have we actually talked about how to define the constraint? I'm not sure we have. How do I define my Sort method to say that it expects lists of A, but also expects that implicit type class instance?

Daniel Westheide: I briefly mentioned that you have this implicit parameter list in which you would have your ordering of A as an implicit parameter. That's what it actually always boils down to - you can define it like that. But there's also a shorthand notation for type class constraints which is called context bounds. So instead of having this implicit parameter list, which is quite verbose, you could just have your type parameter; so it's called A, so we can say our type parameter A:ordering, which means there's a context bound for A called 'ordering'. But that is just syntactic sugar, and it will always compile to this implicit parameter list. But it's just a bit shorter to read, so it's actually quite popular to use these context bounds.

Stefan Tilkov: Fair enough. So you're not advocating to not use type classes, you're just saying "Be careful and don't misuse them", correct?

Daniel Westheide: Yes. They are a really powerful feature, a nice feature. I'm using them a lot. But if you define some default instances, make sure people can opt out of them. That's definitely an important point. And understand the resolution mechanism if you are new to Scala, because the whole Scala ecosystem is using type classes a lot, so you will have a hard time completely avoiding them.

Stefan Tilkov: Okay, fair enough. On to the next topic - type level programming. That sounds scary!

Daniel Westheide: Yes, it is!

Stefan Tilkov: What is type level programming?

Daniel Westheide: Scala is, of course, a functional language, but also an object oriented language... But you can also use it for logic programming, because you could say that there's actually a prolog in the Scala compiler.

Stefan Tilkov: I expected this to be scary; I'm not sure I expected it to be this scary, but it's okay, go ahead.

Daniel Westheide: If you want to do logic programming, you are doing it in the compiler, which means it's type-level programming. I guess we need to explain a little bit about prolog to understand this. In prolog, or logic programming in general, we have facts and we have rules, and then you have some induction, so you can derive new facts from existing facts and rules. So it's a very declarative way of programming. Does it make sense?

Stefan Tilkov: Yes, it makes sense. If people have never heard of prolog, it's probably a bit scary, but I don't think we can squeeze in a complete prolog tutorial.

Daniel Westheide: No, we can't.

Stefan Tilkov: So a rule-based programming, fact-based programming - that should probably be enough. Okay, so how does that translate to the compiler and Scala's type system?

Daniel Westheide: Yes, I don't want to go into too much detail on this, because it's quite difficult, but in general you could say that facts from logic programming can be represented by implicit values, because an implicit value is an evidence to the compiler that something exists or something is true... Like the ordering of int.

Stefan Tilkov: Okay, so in the prolog examples that I know, that would be something like "Paul is Laura's father" or "Paul is the father of Laura", or something like that. Is that something that you would put into a value by using the mechanism that you just talked about?

Daniel Westheide: Yes. Of course, you first need to define some types.

Stefan Tilkov: So maybe the parent or the father relationship.

Daniel Westheide: Yes, this relationship will probably have to be some type parameters... Maybe a relationship with two parameters; well, a father relationship. So you could define a fact like that.

Daniel Westheide: The actual value will never be used, of course, because we're only doing this at compile time, so it doesn't really matter what the implementation would be.

Stefan Tilkov: Okay.

Daniel Westheide: And it probably doesn't even have to be an implementation in this case. Anyway, that's facts... And now rules, represented by something I haven't mentioned yet when we talked about implicit and type classes, because I said you only can define implicit values in order to provide evidence to a method that has a type class constraint. But you can also define implicit methods, which are defined by the 'def' keyword. With those, you can basically represent rules in logical programming.

Stefan Tilkov: Wait, what's an implicit def? I think you lost me there.

Daniel Westheide: An implicit def would be evidence to the compiler that there is an instance of a type class, let's say foo, for a given type A.

Stefan Tilkov: Like the comparator or the ordering for integers, like we talked about before.

Daniel Westheide: Yes. But this evidence depends on some other implicit value. So you would have a method defined with the implicit keyword, and it also has an implicit parameter list in which it expects evidence for something else. And by that, you can say -- well, I don't have a good example for the ordering case now, but in general you can say "I can provide evidence for something that depends on some other implicit that has to be there."

Stefan Tilkov: You need to give me an example, otherwise I'll never understand this. What's a good example, outside of ordering?

Daniel Westheide: Maybe we can actually try to explain this with our father relationship thing. So we have an implicit value... Let's say we want to have evidence of a grandfather relationship. We could derive that from two facts, right? We have two implicit values which define father relationships.

Stefan Tilkov: Let's call it parent relationships to make it easier...

Daniel Westheide: Yes, that's the parent. And then we have grandparent, which would be an implicit method, and implicit def. So this one would give us evidence of a grandparent relationship, given that we already know of these two parent relationships.

Stefan Tilkov: Okay, so I can't see the syntax in my head right now, but let's just assume that you would be able to do that. I think I get the point that if somebody is somebody's parent, then whoever is their parent is the grandparent of the one they're the parent of... Because we all know how parent relationships work.

Daniel Westheide: Yes, so syntax is really not easy to explain in an audio...

Stefan Tilkov: Yes, in a podcast that's pretty hard to do, so maybe we can put that into the show notes somewhere. But the first question I have is why would you use that for -- what is an example use case for that outside of this logic programming? What conceivable use does that have in a general programming language?

Daniel Westheide: Of course, there are all kinds of fancy things people do with this that are not really practical, but a real use case would be something like type-safe indexing of HLists. An HList is a heterogeneous list, which means it's a list in which every element can have a different type, and those types are known at compile time. One implementation of this is in a library called Shapeless. Here, the length of an HList is also known at compile time.

Daniel Westheide: Type-safe indexing would mean you cannot try to access an element that is out of bounds on a specific HList. And if you do index something that is within the bounds, of course you exactly get the element with the correct type that you would expect, because everything is known at compile time. So for each index, the type would be known.

Stefan Tilkov: That makes sense. In a language that focuses on type safety, that makes a lot of sense. Okay, so maybe back to the rule-based programming thing... I get that if you can define those kinds of relationships between your types, you can use the type checker as a logic engine. Is that the point you were getting at originally?

Daniel Westheide: Exactly, yes. But of course, if you don't really have to do towers of Hanoi in the compiler... It's a nice exercise--

Stefan Tilkov: I could also do them using C++ template metaprogramming, just as great.

Daniel Westheide: Yes.

Stefan Tilkov: Okay.

Daniel Westheide: So it's interesting to know... I think it helps with understanding how implicit resolution works, if you think about implicit values and implicit defs as facts and rules, but in practice you usually wouldn't do real logic programming normally.

Stefan Tilkov: Okay, that actually makes a lot of sense. So you are not advocating for actually using the Scala type system as a prolog replacement, but rather to understand how it works by making that analogy.

Daniel Westheide: Yes, absolutely.

Stefan Tilkov: Okay, that makes a lot of sense. While we're talking about types, one of the differences that I know between programming in, say, Java, which is the statically-typed language that I know best, and a language like, for example, Clojure, is that in Clojure you're programming mostly with generic existing types, using maps and vectors and lists, as opposed to defining your own type for everything. Is that something that's at all doable in the Scala world? Is it frowned upon? Is there a use case for doing that?

Daniel Westheide: Well, usually people really define their own types for everything, so it would be quite difficult to just work with lists and maps and sets, with some keys in there. Of course, it's possible, but then there's no real benefit of using Scala at all; you could just use Clojure, because it's much more comfortable and natural to do that in Clojure.

Daniel Westheide: Of course, it means that if you have all these case classes and types, it's difficult to write code that processes data in a really generic way, which is really easy to do in Clojure. I guess that's also your experience, right?

Stefan Tilkov: Absolutely, yes.

Daniel Westheide: So there are ways around this, of course. There are ALWAYS ways around it... Because there is way of abstracting over case classes. I already mentioned this Shapeless library, and it has the capability of representing case class values in a generic form, without any boilerplate and also without using runtime reflection. To do that, it has a type called 'generic', and that is based on this HList type that we talked about before.

Daniel Westheide: Here the idea is that two case classes, for example, they both have two fields, where the first one is of type string and the second of type int. They would have exactly the same shape if you represent them as an HList. The HList would be int, string, hnil in both cases. So when you represent a case class value like that, you can do generic programming with it. And there's also something called labeled generic, which also takes into account field names. So you would also be able to extract those, together with the values.

Daniel Westheide: So you guess it's represented as an HList of tuples or something like that, which includes the name of the field and the value. And with that, it's possible to write code that is independent of specific case classes, so you don't have to repeat everything for different case classes.

Stefan Tilkov: That obviously maps the statically structured or the -- well, I'm lacking the right words here... If I understood you correctly, the structure or the case class, the type that I defined has its field turned into a list of things of a different type - the HList that you mentioned.

Daniel Westheide: Yes.

Stefan Tilkov: Maybe my question is, does that happen at runtime, or does it happen at compile time?

Daniel Westheide: The conversion would happen at runtime.

Stefan Tilkov: So at runtime it is actually represented in this other way... It's not as if I'm accessing it as if it were this other thing, it actually is turned into that other thing.

Daniel Westheide: Yes, but the important thing is that you don't have to use reflection, because at compile time it already knows how to do that. So you don't have to use reflection to look at the names of the fields, for example, and their types.

Stefan Tilkov: I see. So I make a copy of the individual values, and I have to pay the overhead of the list structure that wraps those values, but I don't have to pay for the runtime overhead. No, actually -- yes, sure, the list will have some overhead for accessing its fields, at least one extra indirection, but it will not pay for the reflection overhead.

Daniel Westheide: Yes, exactly. So if you have that, you can also automatically turn any case class value into JSON, for example, using this technique, without reflection. Most Java libraries for JSON serialization as far as I know, they're all using reflection. In Scala, of course, this is a bit frowned upon, so we like to avoid reflection.

Stefan Tilkov: What happens if things don't match? If I send some JSON to you that doesn't match your expectation?

Daniel Westheide: So this would be the other way around, of course... But this is also possible. The first thing was turning the value into JSON...

Stefan Tilkov: Sure, that's the easy part. I understand how that would work, but it only makes sense if you can do it the other way around, too. Does that form an exception, or a crash, taking down the machine?

Daniel Westheide: Yes, we don't like exceptions, so the way this is implemented is -- so the code is also generated for parsing a JSON structure and trying to turn it into a value of a certain type. If that is not possible because the structure doesn't match, the field names don't match or whatever, you would usually get a functional error type. For example, the Circe JSON library, which is using Shapeless to do all this automatic encoder/decoder derivation, it would return an Either, where you get a nice error message about what was wrong... Or you get a successful result in the Either. That's how it's usually done in the Scala world.

Stefan Tilkov: Any downsides to that?

Daniel Westheide: Yes, so if you make heavy use of all these shapeless features, it can often lead to slow compilation times, because they use quite a lot of compiler tricks and a lot of implicits, and that can slow down the compiler. It's one of the more difficult things for the compiler, to do all this implicit resolution and some of the other things Shapeless is doing there.

Stefan Tilkov: What does it actually mean slowing down compilation? Is it like half a second, or is it really noticeable, like it takes half a minute, or something like that?

Stefan Tilkov: Daniel Westheide:[00:48:50.06] Well, on big projects it can be really noticeable. If you have a huge codebase, with lots of implicits and lots of Shapeless stuff in there, you'll really notice the difference... So much so that you might be distracted and do other things, instead of just waiting for the compiler. That's always a critical point for me.

Stefan Tilkov: So you mean like three seconds, or...? Who knows, everybody has their own "Let's check Slack..." or "Let's check Twitter."

Daniel Westheide: We have this incremental compilation in Scala, so you usually only compile what has actually changed in this interactive mode of the Scala build tool, but depending on how big your project is and how entangled everything is, even one change might lead to multiple seconds of compilation, if there are a lot of implicits to check for the compiler. So don't overdo it!

Stefan Tilkov: Okay. Maybe we can talk a bit about the effects of all of those things. As you can probably guess, for me as a non-Scala person, all this sounds a little bit scary... Not as scary as I expected, actually.

Daniel Westheide: That's good to hear.

Stefan Tilkov: This is not the first time we're doing this, so maybe I'm learning a little bit on the way... But still, I do understand the value in most of those things. Do you see a downside due to the complexity of all of this? It's a lot to understand, a lot to grasp. Do Scala programmers typically understand those things?

Daniel Westheide: Well, I think it's a long process until you really understand it in-depth. With beginning Scala programmers, they usually don't have such a deep understanding quickly of, for example, how type classes and implicits work. So yes, there is quite a learning curve, it's a challenge, but in my opinion it's worth it. Of course, it depends on how long your project is. If you start learning Scala at the beginning of a project, it might not make sense to do that for a short project, but you really have to invest for it in the long-term.

Daniel Westheide: There are probably other alternatives to Java that are easier to get started with, like Kotlin. I think it's easier to be productive with it immediately, but then it also stops at some point where I think Scala can be even more helpful, if you like statically typed languages.

Stefan Tilkov: What are some resources that you would recommend to people who wanna familiarize themselves with Scala beyond the intro level?

Daniel Westheide: Good question. Well, I guess I shouldn't mention my own book...

Stefan Tilkov: Oh, you should definitely mention your own book, it's fine. We'll link it anyway. It's called The Neophyte's Guide To Scala. Anything other than that?

Daniel Westheide: What I really like - there's a weekly thing called This Week In Scala; it's a little bit like a newsletter, but it's posted as a blog post, and it always contains links to interesting blog posts, or other resources that have come up in that week. With that, you usually have plenty of resources to dig deeper into Scala.

Daniel Westheide: If you really want to learn a bit more about purely functional programming, and also even category theory, there's a nice book called Advanced Scala (I think it's called). It teaches those concepts with the Cats library. Cats is a library in the Scala ecosystem mainly for purely functional programming. I guess we can also put that into the show notes.

Stefan Tilkov: We definitely can. Good, I think that concludes it in terms of advanced topics for today. Thank you very much for your time.

Daniel Westheide: Thank you for having me.

Stefan Tilkov: And thanks to the listeners for listening. Bye-bye!

Daniel Westheide: Bye!