NOEL: Hello, welcome to the Tech Done Right podcast, Table XI's podcast about building better software, careers, companies, and communities. I'm Noel Rappin. Each episode of Tech Done Right focuses on people in tech talking about interesting problems. Today, we're talking about testing with two people that I consider experts in the field. They're going to hate me for saying that already. On our panel we have Sam Phippen. Sam is an engineer at Digital Ocean. He's British but he lives in New York now. He's also known for being Justin Searls on Ruby Conferences, as well as his work at being a member of the RSpec core team. He describes himself as having a love-hate relationship with computers, like I suppose many of us. We also have Justin Searls, who I think is occasionally known for being Sam Phippen at Ruby Conferences. Nobody knows bad code like Justin Searls. He writes bad code effortlessly and he told me to say that. Its giving him the chance to study why the industry has gotten so good at making bad software and he also co-founded Test Double, an agency focused on fixing everything that's broken about software. Hi guys. SAM: Hi. JUSTIN: Hello. NOEL: I wanted to talk to the two of you to have a conversation about testing that would go a little bit beyond how do you do TDD. Is TDD dead, on life support, zombified, anything like that? And actually talk about how we approach testing, we approach the things that we may still disagree on, the things that we find challenging. Maybe we start by talking about the kinds of things that you see as being intermediate level problems in testing, the things that happen after you get the hang of the RSpec DSL and after you figured out the basics of the TDD structure or how that works with your workflow. What was the next kind of thing that you start to have problems with? JUSTIN: Sam and I are both very deferential people who don't like to talk too much, So Sam, why don't you kick us off? NOEL: That's very rarely been my experience of either one of you but go ahead. SAM: I think we both just like respect each other a little bit too much. One of the problems I see on what our teams encounter, assuming you have to test every level that your testing framework presents to you. Like in Rails, you write your capybara feature spec and your iteration test and your routine test and your view test and your controller test and your model test. You know, you feel like for every feature, you have to put something in every single one of those buckets. I think that's not the case and I see a lot of teams are struggling with that because the frameworks sort of make it so attractive and so easy to do that. Even though in the long term, that was ultimately a lot of redundant testing, a lot of slow testing, hitting systems that you don't need on every single test. NOEL: One of things that I would say about that is you see teams that have mastered testing and really get into testing and I think the next level struggle is my test suite is now 20 minutes, 40 minutes, an hour because we've added five tests for each feature and every individual one is only a tiny little bit. But after six months or a year, they really start to add up. JUSTIN: Well, there's a silver lining to that scenario at least because it means that your business is successful enough to have warranted that slow of a test suite that lived long enough to get that big. If it is slow, then you know, Noel and I both work at agencies that would love to help you make it faster, that being Test Double and Table XI, respectively. Or actually, irrespectively. But my intermediate idea, I would like to draw an analogy just in the hopes of the audience perhaps resonating to JavaScript churn. You know, everyone's always joking about how JavaScript frameworks and build tools are changing constantly and it creates a lot of noise for the team. The team's job is to build an application. They have the secondary job, which is they have to actually build and deploy the application in an automated fashion. It's not really their core responsibility but it is a responsibility so they figure out how to do that in their spare time. But because there's 50 different options and 50 different approaches that are always changing, not only do they have to learn how under a strange condition, they have to debate what's the right way to do it under that time constraint as well. All the while being frustrated by the fact. Even once they've settled on something, it's the case in JavaScript frameworks and stuff that you can at least clearly articulate the 'why' of what you chose what you did. But I feel testing is almost analogous but it applies to every single language where we struggle with how do we test this, how do we test this, how do we test this? How do I handle my test data? How do I get this thing automated in a headless environment? Once you've answered all those 'how' questions, not only do you have all the same back and forth debate but you actually do have to bring your own answer for why did I test that, like what's the value statement? What's the return on investment for this test? Most teams just don't have the time to seriously sit and ponder that question. It's considered good enough, if not excellent to even get to the point that everything is tested. NOEL: Sure. I think that a lot of teams figure like there is no negative value to a test. SAM: Right NOEL: Sometimes I like to conceptualize this that the tests have value has sort of in two time frames, like there's a kind of test that has value while you're developing a feature to prove that the feature is working. Then there's a slightly different kind of test that has value over the infinite life cycle of the application and they're not quite the same thing. SAM: I think there's not a lot of this in the tutorial material that exists. There's not a lot of good communication around where that pain really comes from. We sort of talked about runtime earlier and that's one cause. But another one is like you have to maintain your test suite just like you have to maintain your application but your test suite is not tested. It's something that some people like to say, which means that changes to it, kind of have to be done much more carefully. I think a lot of teams who are intermediate on testing, treat the test suite as very sacred and are very wary to delete any part of it for any reason with the assumption that everything in that has value. JUSTIN: Well, if you don't know why something exists, then you certainly don't know when it's safe to delete it or you have no value system by which you can even judge. NOEL: Right, I think if the team is in any kind of legacy code state, where there is code that they don't really have the context of when it was created, then they're going to, I think necessarily be really cautious about deleting tests. SAM: Right. Absolutely. It's not to say that caution is unwarranted but a hallmark of more advanced testers is having a series of critical lenses through which to look at those things and begin to construct the story for what can and cannot go away. NOEL: We've all sort of mentioned tests having value and tests having cost and the story of why you would write a test. What kinds of things do you think of when you say, "I'm looking at this feature and I'm going to write this test because --" What are some of the things that go into that process for you? JUSTIN: Well, I think that there are generalized answers that we could give to this that are kind of almost prescriptive like in a holistic way. For instance, I could respond to your question by saying, "If you're thinking about writing a feature test or an integration test or whatever you want to call it, something that runs the system under a relatively integrated context," then the value that I hope to get from that test typically is make sure everything's plugged together right and make sure that I can tie a bow around a basic workflow such that I could walk away with confidence that some stakeholder can accomplish a basic task. What I would avoid and what I'm trying to imply, I wouldn't do in that scenario in terms of just being too costly and would result in a runtime of many times that first test, would be to avoid testing edge cases, avoid testing sad path items or extreme cases. Avoid testing like entire tables of input data versus output data because, A) it's really integrated so it's really slow and B) when those things fail, they can fail for more reasons and when they fail, the explanation you get from the test, the feedback you get from the failing test is not going to be very localized to the source of what changed to break that test. In general for the high-level stuff, I focus on just making sure everything's plugged together right and if you want to get coverage of the low-level details, which you should, make that as a local as possible to the object or function that actually implements that behavior. NOEL: Yeah, I agree with that. I think that one thing that people have trouble with in that context, certainly it's something that I kind of fight with on anything that's a moderately complicated is I have an end to end test. Then I have some object -- if it's a workflow or if it's an action or some unit of feature -- then I probably have an object or a method that is running that entire feature flow, more or less, like that integrated test is somehow calling some feature. I think where people get duplicative is I think that it's really easy to see how you test the end to end thing from outside with Capybara or whatever. I think it's easy to see how you test these individual small units to do tiny pieces of it. I think the testing that intermediate piece that is some method that is directing traffic, whether it's a controller or some sort of service object or some sort of method, testing that method without it feeling duplicative of either, the unit test or the end to end test, I think that is hard. People aren't used to thinking of that kind of control piece as a unit in and of itself and doing that correctly involves creating test doubles and isolating that off from the things that's calling in a way that I think a lot of people find not super intuitive. JUSTIN: You know what, that whole middle of the pyramid, middle of the call stack is where almost all of this pain comes from NOEL: I agree, yeah. JUSTIN: and the only way to solve it is through conscientious design. If you look at Gary Bernhardt's excellent talk, Boundaries, he introduces a meme called 'Functional Core, Imperative Shell', where all the branching in your application and the loops and all those different functional constructs and control flow and stuff get pushed as far down into pure functions that are easy to test as possible so that when you have the controller or that traffic cop in the middle, you can still have a test for it. But because it should be basically imperative without any of those loops, without any of the ifs and elses, a single pass through test is going to make sure that the thing is calling everything it needs to do without test doubles. SAM: I think one thing that you sort of glance by and didn't go quite into there, Noel is that like the things like Rails controllers, are usually very hard to sort of just in a week or two or the pain comes in that middle. It's very hard sensibly structure a test that feels, either isolated enough or integrated enough without being redundant, for those things because they're so glued into the Rails framework. NOEL: Rails was not designed with that workflow in mind. SAM: Right, absolutely. But this is also true for things like Ember, Django in Python and many frameworks beside -- NOEL: You all have this kind of control or middle objects that are tightly integrated into the framework and therefore kind of hard to feel you're meaningfully testing them. JUSTIN: But they all give you test helpers for that layer because a framework like Ember. It's giving you these 10 basic types so when they think about building test helpers, they're like, "Let's build a test helper for every single type," and Rails did the same thing and mostly frameworks did the same thing. There's this implicit behavior that people assume that that should be tested as well. NOEL: What's interesting for that is that Rails has deprecated controller test. Not for the reason I think that you or I would say that they should be deprecated controller test but they did deprecate it. SAM: That's not technically true. They deprecated, assigns and assert template but not the actual controller testing. NOEL: So they deprecated what people were doing in controller tests. SAM: Yes. JUSTIN: When I think about this like to Sam's point, if you can't construct an object easily or put something under a relatively isolated test easily, then you should use it like a real user would use it, not like you're just testing a method. This idea of symmetry between our test and our code is really, really desirable in general because it's easy to organize. But if you have a controller and you have a function inside the controller and you just want to write controller tests of exactly that function, then it's incumbent upon that test to figure out how to instantiate a controller and none of us on this call know how to instantiate a controller. In fact, that would be a great technical interview -- an hour-long pairing session with an interview candidate and say, "Just instantiate a controller in Rails," and watch them flail for 60 minutes. NOEL: That's true. That becomes an argument or part of the argument to have as little code in that layer as possible. SAM: Right JUSTIN: Because it's not real code. It's not object oriented. It's like this weird, wacky DSL that looks like classes and methods but it's not really under our control at all. NOEL: Right and you can imagine a framework like there are other frameworks where that layer is more object oriented, at least I think there are. SAM: It's worth noting that what controller testing used to do is basically expose the instant variables of the objects to a test, which is something you would never ever do under normal circumstances. I guess this is the point here is that because your code is inextricably linked to framework code, your test sort of ends up testing the framework, if you put a bunch of stuff in there as well. The framework is already tested and so much of that work is kind of redundant. NOEL: For a long time, the purpose of controller tests on most of the teams that I was on that used them was to make sure that the view was getting variables that it was expecting. I mean, in the context of a traditional Rails application, I suppose it's a useful thing to know. But it's also a relatively slow test for the amount of value it provides. SAM: Yeah, for sure. I mean, request specs which are iteration test which have now sort of replaced controller tests are faster and do more work which is like kind of ridiculous but it is also super good. JUSTIN: But for anyone listening, we did talk about tests having costs and then we talked about two of the most costly kind of tests, without really contextualizing, yes it's tied in framework but at the end of the day, if somebody is listening to this, I would tell them, "Skip controller test, even skip request specs and start with a really, really good isolated unit testing. Start with a really good smoke testing of the full stack that uses the thing just like a real user does." Only after you've gotten really good at those, then you can start even asking about those middle layers, rather than just look at some book or look at some framework's test help guide. NOEL: Yeah, I tend to agree with that. That are certainly pretty consistent with my actual practice right now in Rails. I'll write an integration test and then start pushing as much of the functionality as I can into objects, ideally, not even active record objects that I can just test. SAM: I think it's worth de-jargonizing one of the things that we're talking about, which is isolated unit tests because I think one thing that people sort of assume is that the moment they are instantiating a single Ruby object, instead of hitting the Rails stack, thats a unit test. But we're using something very specialized here, which is you're only testing one production object and all of the other pieces of data objects and behavior in your test are not from your production code. They are from some test environment. NOEL: Right so possibly, they're being stubbed with test doubles with canned data coming in some way and you're specifically have an object under test or a method under test and really the idea is that the only place that that test can fail is if something is wrong with that particular method or object and issues elsewhere in the code won't affect it. SAM: Right, exactly. When we say isolated unit test, we're exactly talking about single objects, single method and everything else, either comes from your test framework or something you've invented in your test, depending on the power of your testing framework. JUSTIN: As well as the nature of the unit, right, like for instance, you guys might talk about an isolated unit test of a method on an active record object or at least, when Noel mentioned that having canned data you were feeding it, it intimated that, whereas for me an isolated unit test, I gave up long ago trying to even approach isolating an active record object. Even from the superclass, you get the entire universe including the assumption that you're connected to a database. Not to mention all the framework gunk. When I say isolated unit test, I'm mostly thinking 80% pure functions and those are just simple, plain old Ruby objects of plain old methods. NOEL: I guess, there's two things of this reminds me of that I want to maybe touch on. One of them is something that Justin, you told me at one point, which is that you felt like when you were working in an environment like Rails, you couldn't use testing for domain discovery because Rails was basically controlling your domain discovery. I want to come back to that in a second. Because other thing that what we start talking about, if you are an intermediate developer, an intermediate tester and you've learned how RSpec works and you think you know how Rails works and now we start talking about isolated unit tests and service objects and things like that and pulling things out of active record. I think that there's a gap there of how do we help somebody to make the first steps towards structuring their applications that way, even when it means pulling away from what is quote-unquote "the Rails way". I think that starts to make people a little bit nervous and they don't quite feel like they know how to do it and I'm not sure that we do a really great job of explaining how to do it or why to do it. SAM: Yeah, I agree. I think it's sort of necessarily in following this approach that we're discussing where you have some integration tests and a bunch of highly isolated unit tests. Actually, it makes you write Rails applications that look like Rails applications that nearly nobody writes. In practical terms, I don't think I have ever encountered a project that somebody else started that was amenable for the style testing we're discussing. NOEL: In practice that leads you to have a whole layer of objects where your business logic is that are not active record objects. People call them a million different things but basically it means, you don't put logic in active record. JUSTIN: You don't subclass a framework-type. That's the simplest, probably. SAM: Right or is it the framework module? NOEL: It's hard to write Rails without having subclasses of active record pop up. You just don't put logic in there. JUSTIN: Here's how I would do that -- to that hypothetical intermediate developer who's trying to follow like what the hell we're all talking about. I think the way that I would frame it is view all of the code that's inside a file that either Rails ordained for you or that extends a Rails super-type. View all of that as configuration and view Rails as a DSL for configuring, whether it's a controller or a router or a model callback. Don't add anything that represents actual behavior of your application. That all assume has to go and live in its own plain old Ruby object and if you start carving off configuration versus behavior along those lines, then I think that gives at least people a value system for starting to understand what goes where. NOEL: What's kind of interesting about that is that other frameworks have that distinction. A lot of the Java frameworks make that distinction between business objects and data objects and Rails was created in part to kind of get away from that, from having what I think DHH would say is an extraneous level of complexity that then you come to a certain point and you realize actually that's not an extraneous level of complexity at all. It's a kind of a necessary level of complexity, or at least a necessary or a valuable separation to make in the code. SAM: I think with the theme you just said is the word necessary. It is definitely easier to just read a straight up vanilla the Rails way Rails app and understand exactly what is going on. The reason we shunt in these layers of abstraction, these isolated unit tests all of this is to make change easier and that's definitely a tradeoff. I guess the thing I'm saying is like I don't think the position of doing the Rails way is necessarily invalid. It just represents a different style. NOEL: No, it's not. The Rails way comes with a set of a pretty focused set of tradeoffs and practices that I think work really well for certain kinds of problems and certain kinds of approaches. And don't necessarily lead themselves to a really isolated test suite or perhaps keeping the long term cost of change low. JUSTIN: When I'm talking to a client for example and helping them plan like how they want to architect, even just like within Rails like a Rails app, questions of what's your level of investment in this code base. If it's low, then Rails way is almost certainly better because there's going to be a lot of Rubyists that are able to figure it out. If you're more focused on the short term time to delivery then you are focused on the long term maintainability, sustainability, changeability of the thing, then Rails way is probably better. But if you've been burned in the past and this is a rewrite because what you found was going things the easy way to just go Rails way, go vanilla and then try to get to a place where the code base was more changeable, immutable and controllable and the test were able to be run fast in isolation and all that jazz, that's a really hard transition to make and I think the reason that the three of us tend to talk that about isolated test case a lot is, A, we see a lot of really complex, convoluted rescue projects and B, to anyone who's just starting with Rails, on day one, Rails way is always going to be the easier way and that's why you have to just plan conscientiously. NOEL: I think that one of the problems with this sort of application design and this sort of test design and a lot of the things that we kind of think of as agile practices is that if you half-ass them, they will really punish you. I think one of the things that gets in the way of people getting really comfortable with testing is that they start and they want to like dip a toe in and they kind of partially do test-driven development. But then they kind of don't do it all the way or they don't do it consistently or they create isolated objects sometimes and not other times. In some other way that they don't fully commit to it and then it gets really almost harder than if you hadn't even started in the first place. SAM: Totally. I think one of the things you touched on is going from maybe having tried testing a bit to going on into full-on TDD. I know like nearly everyone I've seen who tries to do strongly academic TDD struggles with it when they start because it requires such a different style of the actual code that you write. Most people don't realize why that's happening, is the tests are forcing them to change their actual programming behavior. They actually feel like they're bad at testing because their code is changing when what's actually happening is the tests that they're writing are fine. It's just forcing them to change the way they write that code. NOEL: And they're a little bit resistant to that. SAM: Yeah because they assume the way they've been writing code before is fine or perhaps a better way to phrase it is like amenable to this style of writing tests and then it's not. There seems sort of cognitive dissonance there and I've seen nearly everyone who tries it, go through it. JUSTIN: I think, we as thought leaders -- SAM: Oh, Justin. JUSTIN: -- I think we need to be always careful of words like should and value judgments. Sam, you called out the word necessary before and the reason that we're so wishy-washy about this is that it's not necessarily that TDD or real rigorous approaches to testing make your code better. I would argue it makes it more usable because the code is used in more contexts. But that's again, another tradeoff. If you have an object that's instantiated 20 times by a test and one time in production, is it better that it's easy to instantiate? Well, it is because it's more testable but it's also still just called the one time in production. The benefit there is that if all your objects look that way, then all your objects are brain-dead simple for using by other people once they know that style. But they have to learn that style and that's unfortunately just not the norm, either. It's not that TDD is going to get you to a better design necessarily but it will get you to a design, hopefully that normalizes over time so that it's less covered in, complex patterns and convoluted means of creating objects and all the jazz that you typically see in a typical big application. NOEL: One of the ways I often say that is that TDD approach leads you to small methods that don't have side effects, that don't really relate to each other and small objects that interact in very constrained ways. The reason that that happens is because it's easier to test those things. It's easier to test things when you have a unit that you can wall off that is small. Also, almost like coincidentally turns out that the same features that make a thing testable, also makes code more amenable to a certain kind of change. SAM: Yeah, I think one of the things I observed when I'm doing, I guess what you could call for lack of a better term, academic TDD, I'm very sensitive to the number of things I have to set up in order to get my test to pass. If that goes above a certain threshold, which is usually like a very small number of collaborative objects, that's probably another object screaming to be left out here in the thing that I'm building. But that's very subtle and I think one of the things I see more intermediate testers doing is doing TDD cycles but growing these very large tests sort of one red-green cycle at a time and not quite realizing that that's not aligned with what the process is for. NOEL: I've certainly done that. You could look at the code examples in the money book and one of them definitely gets out of control with a lot more set up than I'm super comfortable with, in part because in an instructional mode, where the focus is on the code, I didn't want to get too bogged down in creating a lot of small intermediate objects that would be hard to explain. SAM: Right and that comes back to this idea that bigger procedural things are easier to understand in one reading than lots of small objects so in a didactic setting like that then having it in the money book, big examples with big tests seems actually quite sensible. JUSTIN: I definitely hear you and I think that you're right in terms of a developer with an intermediate skill in testing, tends to run into this thing, even once they get set up, once they get comfortable with the red-green refactor flow, things just tend to get bigger and bigger and bigger and you can sort of see the tests is being like the tip of the iceberg and they still arrive at this really convoluted implementation underneath of ever growing more and more private methods and maybe spin off objects and stuff, whereas that's not our practice. What I've practiced -- NOEL: Certainly not our practice at our best, yes? JUSTIN: I guess, what it is for me personally, my workflow of doing outside of test driven development that was informed by the London XP Group, of Steve Freeman and Nat Pryce -- NOEL: _Growing Object Oriented Software_ is the reference for that, right? JUSTIN: Yeah, although, I've deviated enough that Nat has asked me to stop referencing him so I call it discovery testing now but the idea is I assume that a problem needs to be broken down when I'm at the top of the call stack. My first test is just asking myself what two or three dependencies what I need to implement this thing? And use test doubles just write an isolated test of fake things and make sure the thing calls those three things and I keep repeating that sort of reductive exercise until the function that I'm describing is so simple that it would never take more than three or four tests to fully utilize. In that way, the test is actually informing. You're going to shake out like a tree of objects and functions that are really, really tiny and really focused and really well named to avoid the eventuality of, "Oh, shit. I wrote like a hundred tests that were focused entirely on implementing the behavior as fast as humanly possible and now I've got this big gnarly mess that's also difficult to refactor. But at least I have these tests," which is around 90% of people end up with TDD. NOEL: Yeah, I say this because I've certainly been there. If you don't do it the way that Justin was saying, then you wind up with a test that has a lot of set up and probably runs slowly and it's a little bit hard to figure out when it goes wrong because you just add individual pieces of set up one by one as the logic gets more complicated and that's what leads people to sort of conclude. The way in which you separate a problem into small testable units, I think is either counterintuitive or not super-natural to a lot of developers or maybe the way a lot of developers are taught. I think that makes it hard to try and test a complicated feature because I don't think people practice breaking a problem out into testable pieces. SAM: I think, there's another factor at play there. It was interesting when Justin was talking about lots of small units and having to name them and so on. One of the things that popped into my mind is this idea that the frameworks that we use kind of give us this position that those are the only places that you're allowed to put code. But the things we're talking about force you to have many, many other places to put your code within app/services directory or lib directory that's full of tons of tiny files, I think a lot of people are very hesitant to do that because structuring those things is actually pretty difficult. The moment you go outside of the framework of well-defined places to put code, you have to work out how to do that yourself and that is actually a difficult thing to do. NOEL: Because the framework was designed to solve that problem. SAM: Right but that's the trained resistance there because people love their framework so much and that's not a bad thing but it is just sort of another contributing factor to why some people -- it takes a mindset shift to get to this place. JUSTIN: Point of order, I think loving anything about software is a bad thing. NOEL: I think that there's a genuine learned reluctance to do that, especially if you've been a Rails programmer for a couple of years and you've seen projects get super messed up because they probably tried to go outside the framework in a bunch of different ways that don't really play well together and you wind up with code that is very hard to get your head around. To have us come and say, "well, all you need to do is create objects outside the framework," and people are going to raise eyebrows at that, based on their own experience. SAM: Right, exactly. I've seen applications where they have all the usual framework buckets and then a folder called app/services and everything else lives in app/services. It's just a dumping ground for any plain old Ruby object you care to name. JUSTIN: It frustrates me that perennially most difficult issue with people improving at testing is people improving at basic code organization of files and folders. NOEL: Well, methods and classes, really. JUSTIN: I mean, unless you do Smalltalk, they're the same thing. My frustration is, I think as programmers, our job first and foremost is essentially to convert business processes into a taxonomy, whether that's of behavior or of data. We're doing ourselves and our colleagues a disservice if we just assume that the best taxonomy for everything is one big gigantic flat bucket. But that's somehow where everyone or the vast majority of people end up when they're using a framework like Rails because we are a prescription for like, "Here's these three buckets for this common set of problems," but a common set of problems is about network IO and rendering HTML to a string. It's not about whatever the business needs. That's still on us to come up with our own custom buckets for every single domain that we work on. NOEL: Yeah, I think that's fair. It's also a little relevant actually that TDD really started in a Smalltalk environment since you brought it up. Have you recently had the opportunity to work in a TDD Smalltalk environment, just as a toy either of you? SAM: I think I have never actually program Smalltalk. NOEL: One of the things about TDD and Smalltalk, especially as compared to TDD in Rails, where you're building up the framework, is it is fast. It is incredibly fast. Everything's already loaded, the code's already loaded in. In some sense, if you've only done command line RSpec test development against loading the whole Rails stack, you have not had the kind of experience that Kent Beck was originally writing about, where you're in the Smalltalk editor and then you push a button and you get green or red in milliseconds. Because I wrote a toy program a while back, going back to that after having been in Ruby and Rails for so long was really interesting because it really does change the cost-benefit analysis of writing a small test and running your tests when they happen in under a second as opposed to 15 minutes. JUSTIN: If it's faster to use a REPL to answer the question of, "If I do X, will I get Y?" people will use the REPL and then as soon as the REPL session is over, it goes away forever. But if it's equally fast to just codify that in a quick test or check like it is in Smalltalk or even environments like Java where your IDE, maybe you just hit 'CTRL+R' or something that'll run the test of the file in half a second. It definitely reduces that cost. SAM: I think one thing that is worth noting though is it is possible to achieve that for your isolated tests in a Rails application. You just have to be pretty disciplined about what parts of the framework you load. That's why RSpec now separates Rails helper and spec helper and spec helper just loads only RSpec and then it's up to you to do the rest. You don't even get the active support constant loader. When you load the spec helper so you'll have manually require your isolated test file but it does also mean that if you're running like that, and you have your own set up, you can get that make a change and run all of your tests in a couple of seconds cycle. JUSTIN: In fact, if you're a MiniTest person and not an RSpec person, RSpec has the benefit that it controls the CLI. But for MiniTest users, they're stuck with Rake and Rake under Rails 4 and later, always loads your Rails application. The bottom is already quite high and only gets higher as the application starts adding more gems and becomes more complex for its initial load time. I recently actually had to write a blog post, I think I called it "Rake without Rails" about setting up a second Rake file because I was using MiniTest and I found that it is very, very difficult. You almost need a helper to not only convince you that this fast feedback loop really matters but to help you with the drudgery of the custom one-off set up because literally, everything is fighting you to write isolated tests. SAM: I think Rails is perhaps one of the least amenable environments to doing this that you could possibly hope to use. NOEL: Right, which gets back to the Rails way and the Rails way still being very successful in a lot of different environments. But obviously not designed for this one, I think, it's fair to say. One thing that I'd like to talk about is the different test tools, especially in the Ruby community. I think at one point, it was pretty common to say, "Oh, it doesn't matter what tool you used. Just test," and I think there's some truth to that. But I also think that the tool you choose is going to have some effect on the way you think about tests because of what it provides and what it doesn't provide. SAM: I think that's absolutely fair. I am in the position where nearly every single time that I give a conference talk, somebody asks me sort of as a troll, "When should I use RSpec versus when should I use MiniTest?" I've taken to try and be as overwhelmingly reasonable as possible when responding to this question. I usually say that like, "MiniTest is very strict back in what it offers you so that actually forces you to write your production code in a very specific way like it has a much less powerful mocking and stubbing framework which fundamentally changes the way you have to structure your code. NOEL: That's where I bounce on MiniTest at the moment is the stub framework. SAM: Right. The correlation to that is RSpec has an all-powerful mocking framework that can do basically anything. It's actually much more amenable to testing legacy code but it also has a bunch of constructs in it that you probably don't want to use, if you're writing new code from scratch. It's sort of like a tradeoff and picking one of those two sort of depends on your context. JUSTIN: I'll go a step further to say that it is a long trodden point of disagreement between Sam and I, on whether or not test doubles are actually useful or advisable under legacy code scenarios. Sam tends to prefer them and I tend to say don't use fake things in an environment where you have very low confidence already, instead focus that time on identifying a seam that you can test as realistically as possible. SAM: I think that's fair. NOEL: Okay. But in terms of the tool choice, how do you see the difference between the minimalist tool and the maximalist tool, Justin? JUSTIN: For me, I think that Dan North's post about how he organized his JUnit tests sort of at the beginning of 2006, the key insight that was like the granddaddy of behavior driven development or BDD was simply what if instead of just making my test subordinate to my code in terms of how I organize it instead the test is organized around what should the code be doing? So I'm prompted and I'm framing and I'm thinking it should do this when this happens. It should validate this when the phone number is wrong. Then write all your tests in that format until you run out of ideas of what the code should do and that was really the start of BDD and it first born JBehave and Cucumber and tools like RSpec. Then of course, because of RSpec's influence from the Ruby community into JavaScript, Jasmine and Mocha are born out of the same prompting, where there's this DSL that's trying to get you to describe in kind of holistic or separate terms, what the purpose of the code is so that you're divorced from the implementation details and you can just look at it in terms of what am I accomplishing with the code. All of that is fantastic and really useful framing. It comes at the cost of never ending indirection and difficulty, debugging and less useful stack traces and less useful tooling like CTags. What I've found over the last decade is that the framing and the prompting did change my behavior at the outset and then that benefit had diminishing returns but the cost never really went down. I've slowly moved back to more minimalist style testing tools. I wrote one that still supports nested cascading example group types called teeny test for Node.js but otherwise when I'm in Ruby, I tend to use MiniTest, despite the fact that I can't nest, just take advantage of the fact that tests are methods and I can use CTags and I can navigate my environment way faster when there's a failure. I can see the stack and it's an honest thing. NOEL: You would love the original SUnit Justin its very minimalist. JUSTIN: You got to start somewhere. Sam, how would you react to my journey? SAM: I think it's entirely fair. I will say that the one thing that you lose with a framework that doesn't have nesting is sort of the ability to strongly de-duplicate, your example like your tests but to some degree, we've sort of covered writing small tests anyway so that benefit is smaller, as you pointed out as you write those very small tests. I think the one thing I really like about RSpec and its style of framework is the full English string that you put in your test descriptions where it's like writing something that looks a little bit like English as a Ruby method name has always bugged me but it's more of an aesthetic choice than anything else. I don't disagree with anything that you said. It's just not my preference. JUSTIN: Yeah and I have to agree. If you have a test method string that's like longer than 20 characters or six or seven words, it's pretty ugly. The converse is it's a healthy pressure to keep them short and not have too many examples and keep your objects small. SAM: Totally. I think one other things I like to do with teams where I'm sort of trying to help to level up their testing practice is run the RSpec documentation formatter and then print it out and have them actually read it back to themselves. You'd be surprised how often you just get total nonsense with all those frames are sort of concatenated together. NOEL: Yeah, I can see that. I think that being able to nest tests is also a place where I bounce off of MiniTest but it is a feature that you can easily overuse and make things really hard, I think just in general test set up and test set up that winds up being far away from the actual test is a very powerful feature that is really easy to overuse. SAM: Right. I know for example the good folks at Thoughtbot, actually even when they are nesting duplicate out all the constructions so they are right there in the test, they call it this the Mystery Guest. I'm by no means convinced by this but I think that's comes back that they're writing smaller tests whereas if you're writing a big one, then having to scroll all the way up the file to look at what the test subject is, then you come back down and scroll half way up to find one of the pieces your data were using, can definitely get problematic quickly. JUSTIN: I think for me in terms of just style points and comprehensibility, I have always tried as hard as I can to get -- because typically lines of test code versus lines of production code ranges somewhere between 1.5 lines of test code per line of production code to three or four, and I want it to look as symmetrical as possible. My ideal would be, let's imagine a 30-line file listing of a Ruby class with a single public method on it and I would love to have a 30-line test that was exactly symmetrical to that thing. If there's a lot of branching than I understand, it's going to be two or three times longer because there's more test methods. But one of the things that I really like about the nesting approach is that you can DRY up that common set up across all the different types that you're not repeating yourself a whole bunch. But if you're diligent about it and focus on keeping things really short, what it results in is you can just keep these two listing side by side and immediately know exactly what's happening on both sides. Unfortunately, it's just not how very many people really seem to use nested test example groups in practice. SAM: Right. I think the sort of other thing is that when you working in a less powerful testing framework, for beginner and intermediate testers that could be this driving effect to write more integrated tests. Because you don't have the same capabilities of isolation, I often see people being like, "I'm just going to write a requests test for every single piece of behavior in my application." That comes back to the sort of cost benefit situation. NOEL: Okay, we're pretty much at time or near time so I will unfortunately, because I'm enjoying this, kind of stop us there but if you guys want to see one or two quick resources or tips that people can take away and start applying quickly to their day to day code, if you have something like that. Sam? SAM: I think Sandi Metz's Magic Tricks of Testing talk is probably very good. Justin has a great little repository on GitHub, which is a list of testing smells so I'm going to take that one off him so he has to tell us about something different. NOEL: That's a good one. I like that. SAM: This is me being excessively complementary towards Justin but I also think his talk on 'How to stop hating your test suite' is probably the seminal piece. If you were to just watch that over and over again, that would level you out of beginner and into intermediate pretty quickly. NOEL: Justin, what do you got? JUSTIN: Now I feel bad that I don't have two Sam Phippen citations right in front of me. If I could provide one piece of advice to anyone listening to this conversation, especially if they feel a little bit lost, especially if they feel they were being condescended, maybe particularly by me when I was chastising people for just putting everything on Rails buckets. My advice would be find some little problem, find some toy that you want to write, some little CLI or script in your spare time and instead of implementing it in a Rails app, just type 'bundle gem' and then make up a name and practice. Practice, whether it's with RSpec, practice whether it's with MiniTest and invent your own way of organizing code and tests, such that you're able to break these things down because you need a safe space to practice, if you're not already really good at that and if you're not really good at that, almost all of our advice about how to cope with testing beyond just using the out of the box Rails test helpers is going to ring hollow or seem frustrating. That would be my, I think one action item that I would give anyone who listened to this and found themselves kind of tense. NOEL: One thing, I wanted to bring up is I recently had an opportunity to go back and reread the original Kent Beck and Gamma test infected essay, it's called 'JUnit Test Infected: Programmers Love Writing Tests'. One of the things that kind of striking about that is it's not very doctrinaire about writing the tests first but it is pretty doctrinaire about writing tests and code in small pieces, whichever one comes first. That kind of resonates with something that I've thought for a long time, which is the writing code in small pieces is actually the most important part of the test. My piece of advice would be to find that essay and read it. Then also, as you're trying to test stuff, really try to focus on going back and forth between the test and the code more rapidly than you probably are doing so right now because I think that's going to be something that's going to focus you toward smaller methods and smaller tests. SAM: Totally. I agree. I think the most important point to pull out there for me is being really, really prescriptive about how you write a test on a day to day basis isn't super helpful. I changed that nearly every day. Sometimes, multiple times a day, depending on what I'm doing. But also understanding why all the prescriptive practices are prescriptive as they are is also very helpful. NOEL: Okay, and with that, let;s take us out. Thanks to Justin and Sam for being with us. This is the Tech Done Right podcast from Table XI. You can find it at TechDoneRight.io or you can download it via iTunes, hopefully by the time you hear this or whereever you get your podcasts. You can send us email at TechDoneRight@TableXI.com or you can follow us on Twitter @tech_done_right. The Tech Done Right podcast is brought to you by Table XI, a design and software development company in Chicago. We are 35 meticulous and curious minds, with a 15-year history of building websites, mobile applications and custom digital experiences. Our partners trust us to create innovative solutions to drive their business forward. Find us at TableXI.com, where you can learn more about working with us or working for us. Now, we'll be back in a couple of weeks for the next episode of Tech Done Right. Thanks Justin. Thanks Sam. SAM: Thank you. JUSTIN: Bye.