*Intro music* CHRIS: Welcome to Elixir Outlaws: the hallway track of the Elixir community. AMOS: So how’s everybody doing? ANNA: Good! CHRIS: We are all here in San Francisco ANNA: We have Ben Marks, Friend of the Show, here with us today. AMOS: I am super jealous of you guys all being together. CHRIS: Yes, Friend of the Show, official title, that we’ve started doling out - Friend of the Show, Ben Marks is here with us. Say hello, Ben. BEN: Hello. I’m having fun showing Chris around San Francisco. He can’t go anywhere without me. CHRIS: No, yeah, he’s like my official handler. AMOS: Do you have a leash for him? CHRIS: It’s a metaphorical leash, but yes. AMOS: Ah, that’s too bad. I think you would be good in one of those harness leashes that go over your shoulders and don’t allow - ANNA: Amos, how was your trip to the desert? CHRIS: How was your vision quest? You’re back! AMOS: It was fantastic. Never being alone, ever in my life, really, wasn’t something I realised I was missing, but a few days of absolute quiet with nobody really talking to you is kind of amazing. And a lot of relaxation. Although when you come back to the real world and someone talks to you in the gas station, you’re just like ‘can you shut up? The things you’re talking about are pointless. Just stop talking, please’. CHRIS: So you truly had a vision quest. Like, you’ve seen into the heart of the things, and then your material needs are no longer important. What’s money, man? Have you thought about money? AMOS: Maybe not quite to that level. I did try to read Purely Functional Data Structures while I was there - ANNA: In the desert. AMOS: In the desert. And thought it would be a great time to just blast through that book. And I did realise there’s one really nice thing about not being out in the middle of nowhere, and that is having internet so that you can look things up whenever you’re frustrated with the book. CHRIS: What were your frustrations with the book? AMOS: Really, just exercises. Some of them, I could not think of a way… I’m only on like, page 23, so I didn’t make it very far before I said ‘this might be a pointless exercise to try to read this out here’. But there is an exercise about leftist heaps - here, let me see if I can find it… Basically when inserting into a leftist heap, there’s apparently two ways to do it. One is where you rewrite the state as you go down in order to insert the new one at the bottom - so you’re rewriting the list, which makes sense. And then they said - write it so you don’t have to do that, and use an exception. I was like ‘what?? I have no idea… how that works…’ So I needed to - I needed to take a step back and look some things up, and I just couldn’t do that out there, so I read on a little further, and all I could think about was that one exercise, and I couldn’t pay any attention to the rest of the book, so I decided I needed to stop. How about you? What are you guys - I listened to the episode on the way back. It was fantastic. I mean, I disagreed with Chris about consultants, but that’s okay. I guess it depends on the consultants. ANNA: I saw that tweet. Yeah, Chris? How was day one of your new thing? CHRIS: It was… overwhelming. But, uh, there was just a lot going on, a lot to take in. But overall, it’s been very good, and everyone was very welcoming. Surprisingly welcoming. Given what I know of Ben. AMOS: Ben seems like a good guy. I don’t think I’ve ever seen him not smiling. But I also haven’t worked with him either. BEN: We had a meetup last night, and Chris unlocked the achievement of his first day at BR and then speaking at meetup held at BR that night. CHRIS: It was the real gauntlet. It was like, go through a day of orientation and getting things set up. And then also, hey, can you give a talk tonight, and then I was in too much of a stupor to say no, so I had to go up and give a talk. ANNA: It was great! CHRIS: It was fun, and, um, the crowd was - the people there were pretty engaged and had a lot of good questions, so that was a lot of fun. AMOS: What’s the topic? CHRIS: I gave my talk that I gave at LoneStar and Elixir Days. Slightly more informal. Somewhat more expletive laden. AMOS: So what have you been up to, Ben? Other than handling Chris. BEN: Well, I was handling Sonny two weeks before, so it’s been a lot of handling various people. No, but it’s nice to have Sonny and Chris join. And… Uh, what am I doing… A lot of planning for the next couple years. Over the last three years at Bleacher Report, we’re about moving over to the new system and getting everything working really well, and that’s proven to be pretty successful, and now we’re trying to figure out how we can be - like, we’ve seen some areas that, as we continue to grow, might cause problems, and now we’re looking into playing around with distributing Elixir, for - not for the entire system, but for the parts that need it so we can still deploy and it’s not an all or nothing type thing. And so, I think there’s a new project today - on the second day, that Chris will get involved with - which is tangentially related to that. But that being said, we have a lot of projects coming up, so a lot of it is planning, and other endeavours, so tracing with open sensors, so open sensors and open tracing, starting to lean towards open census, because Tristan Sloughter wrote the open census lab writing. So, um, looking forward to that. CHRIS: You wanna - for people who may not know, do you wanna talk a little bit about what open tracing and open census is? Because I think that they’re deeply fascinating and really cool. BEN: Yeah, uh, I mean I’ll just give the very basic thing - it’s just based off Zipkin is another tracing library that you can - so you can look at it - when you load chrome tools you can see how long it takes to an asset to respond. So, it takes one second, half a second, a quarter of a second, to pull up these images. And it’s sort of like that but for distributive systems, so one of the things that we noticed is when we have high traffic events like yesterday, and like, the day before, there was the NBA lottery pick, and for some reason people are very fascinated by this - I’m not a sports person - so the NBA lottery is just the lottery who gets the first number of picks. No one actually does any picking. It’s just, your team gets the first pick in the lottery, like a pre-draft lottery. And our traffic spiked like 10x and we didn’t really expect that. And things were mostly fine, but we were seeing some hotspots here and there. And the advantage of tracing and open census, which are really two competing, slightly different standards, is that you can trace, just like you can with a Chrome developer or with a Firefox developer, fully, just how that goes, so you can find out those areas that are slower or unexpected, or maybe you have, like, a dapper - is the Google white paper, is their open source - or, not their open source - that’s their white paper, their dapper paper, and ZipPin is an open source version of that. That’s the reason that Google made their paper, to show how traces happen in their system. And their system is infinitely more complex than ours. So that’s sort of - that’s what we’re doing with those things, so we can continue to improve. CHRIS: It’s super cool. You can actually start to see, when we played around with it, you can actually start to see, like, request comes in to the front door of your API, the front door passes it on to this service, this service passes it on to this service, the next services calls into postgres, and you get all of that tracing data, all in like, a single timeline. So you can see the entire request as it goes all the way through the system. It’s super cool stuff. I think it gives a ton of observability to complex systems that have a lot of different interconnections… BEN: It’s pretty lightweight as well. Not a lot of overhead. CHRIS: It’s kind of hilariously lightweight. At least, open tracing is. Open tracing is like the Dread Pirate Roberts - more of an idea, really. It’s not a specification, like, at all, in a lot of ways. It’s mostly, like ‘here’s some core fundamental ideas, now implement it in whatever protocol you want to implement it in’. It’s really - it’s pretty straightforward. BEN: That’s the distinction between open tracing and open census. Open tracing is more ‘here it is’ and open census is like ‘this is the way you build it’. CHRIS: See, I don’t know anything about open census. This is new to me. BEN: I think it’s a relatively new standard, and, I mean, the only reason I know about it is because I was talking with James and he was like, ‘oh, you should look at this’, and so… He hasn’t lead me wrong. ANNA: Yet! BEN: Yet. ANNA: I’m just kidding. James is great. BEN: No, you know, there’s always the option. AMOS: James who? I missed… James Fish? ANNA: Yep, Friend of the Show. AMOS: I wasn’t sure, because I met with another James this week - James Edward Gray. I had dinner with him on the way back from my vision quest. CHRIS: How was that? I saw that happen on Twitter. And then you were like ‘hey, you should come and try to have dinner’ and I was like, ‘I am several hours away from you’. That’d be a long drive. AMOS: Oh, I knew it. I was just giving you a hard time… It was pretty good. We talked about house building, and program building, and apparently he thinks the show is okay. Uh, it’s not bad. No, he said he’s a big fan. So, I guess, James Evergrey, also Friend of the Show. Maybe we could just name everybody in the community and just say ‘Friend of the Show’ in this episode. That’d be fantastic. CHRIS: At some point, we’ll - one of the things - one of the pieces of - this may be too inside baseball - AMOS: Wait, wait. You start working at a sports company, and now you’re gonna make sports analogies. That’s perfect. CHRIS: I know… Listen, Amos, this is just blocking and tackling. Just basic blocking and tackling. You gotta skate to where the puck is at. AMOS: So what’s your inside baseball? CHRIS: A lot of the criticisms we’ve got before - they’re like, ‘you namedrop so many people, and you don’t say who they are. You just call them by their first name, then just assume that everybody knows who you’re talking about. AMOS: Well, that works with Fred, though, right? ANNA: Well, yeah. And we have to mention Fred every time. CHRIS: Yeah, I think we’re contractually obligated. AMOS: But only his first name, because his last name is French and I’ll mess it up anyway. ANNA: Ben, is there anything - I’m curious and following up on what you were talking about - about the tracing stuff. Is there anything that you all learned with regards to your system that you’re actually able to talk about, that was interesting? BEN: I mean, this is still sort of a main proven concept, if you know what I mean. But what we were doing before when we were, uh, a service calls another service, we have methods to monitor that, so we sort of hand-stitched our own tracing, so you know, you can see - so you can pull up Datadog, and can see the time, and if we see that there’s a slow - an unexpectedly slow bit, or see that something unexpected happens, we can sort of trace that, really by comparing the requests from service to service. This would really be a nicer way to see what’s - like, in the dapper paper, the authors talk about what’s really interesting - and, again, this is Google, whose system is significantly more complex than ours - they were talking about how these services were hitting other services, and they didn’t even realise it was happening. And, that’s sort of the way that the last little bit of our legacy stuff is moving over. I mean, we’re not going to implement, so this’ll be a bit of a black box, um, but it’ll be really nice, we we go forward, to keep these things in mind, so we don’t have this situation again. We had a meeting yesterday, and we talked about the kinds of service, and it happen to be the service that we were talking about, and the designers were really like, ‘what have I gotten myself into?’ But that was like three years ago, and this is just snuffing out the last bit of fire, I guess. AMOS: So, when you’re doing the tracing, my experience in the past, is that it can have a pretty big impact. I’m guessing you’re doing it on a production system? BEN: No, not yet, so, so we have the sort of, halfway tracing thing, where we measure requests and responses, and that’s just - we use insometer to get all those metrics. As far as the open census, open tracing stuff goes, that’s just me setting up everything, like, local environment, and trying to move it to safety environment, by the end of the week. But your point - were you talking about overhead? Like, performance overhead, Amos? AMOS: Yeah, like trying to decide what to trace and when to trace so you’re not doing too much tracing and it have negative effects on your performance system. I’ve seen systems brought to their knees from bad choices in tracing, so just… how do you decide? BEN: Now, whenever I read a technical book or article, I do mark down - because I read my own handwriting, so now I… AMOS: Ben Marks, calligraphist. BEN: Ironically, my wife is dabbling in calligraphy, so… and she said ‘I’ll teach you calligraphy’. And I said, ‘if you want to have a bad time, you can teach me calligraphy.’ But, but, so, when I’m reading a technical book, I have half the screen as the book, and half the screen as notes, but let me see if I can find my notes because they talk about sampling rates. And I’m not gonna go and find a course on - even with a significantly reduced number of samples, the results seem to be pretty okay, and I mean, they’re talking about, like, nanoseconds, or spanning over - now I’m totally butchering this. ANNA: Have you seen the editing on this show? It’s gonna be highly edited. AMOS: Totally. BEN: But I wish I’d done this years ago, because now - I always forget everything. So now, instead of having to go reread the book, I can just read through notes. AMOS: I think you may have just changed the way I read books. BEN: I wish I’d done this twenty years ago, because even in, like, college or whatever, I could never read my handwriting. ANNA: No, I have the same problem. BEN: Yep, left handed. ANNA: No, I’m right handed, but I just have terrible handwriting, AMOS: I bought a palm 5 to make up for my handwriting… I might be dating myself a little bit… BEN: Here’s a question - do you take notes on books? Like, do you ever print papers and take notes on the white paper itself? AMOS: Yes, but not in books. I have this, like obsession with not writing on my books. Because I bought some used books for a while, and they would have stuff underlined or crossed out that just confused me. And then I thought, ‘what if I let someone borrow my book, and it’s got all this stuff in it and it just confuses them?’ Or, in a year, I read it again, and I think ‘that was a dumb note’. I would rather have it on a separate piece of paper. CHRIS: See, I love that. I am notorious for taking pens and just going to town on all kinds of books - even fiction and stuff like that. I’ll just write down how I feel about certain passages, and stuff like that. I love being able to go back and look at my - I think the word is marginalia. AMOS: I don’t know, I don’t know. I might have to try it sometime, but it feels - it just feels dirty to me. BEN: But it’s not searchable. Like, I would do the same thing - I’d write in books or make notes - fiction, nonfiction, whatever, but then, - I would be like, ‘I have a note in some book somewhere’ and then I’d flip through the pages, and I couldn’t read my own handwriting on the page. AMOS: Well, and that’s the same thing with, like, buying a hardcopy of a book versus a PDF. The PDF is searchable, which is really nice, but sometimes I like to just hold the book in my hand and flip back and forth - I think that’s easier than staring at it on a screen. ANNA: Especially for something like Functional Data Structures, where there’s the exercises and things, being able to flip through is helpful… I did have a question for you, Chris. I was thinking about your talk last night - for folks that are new to this idea, or are interested in this idea of distributing systems, what would you suggest - it sounded like there were some questions, and questions to people who are newer - what were your responses to some of those questions last night, as far as digging in and really wrapping your head around how to really set up these kinds of systems, and working with them. CHRIS: As a really, kind of, getting started, and, like where would you learn this kind of stuff, um… So there’s a couple of really good resources that helped me out a lot, uh, there’s the book, uh, distributive systems, uh, the third edition just came out of that. I got the ebook. And I think the ebook is free - you just have to sign up for it - but I’ve also, because this book is very important, because I wanted to support that - I bought a physical copy of it. That’s a really really good resource. It talks a lot about guarantees, it talks about some really fundamental concepts, like message guarantees, like - are you gonna be, at most once, at least once, the perils of exactly one, message delivery… it talks a lot about replication and those sorts of things. It’s a very - it’s not nearly as dense as an actual textbook - it’s very readable. Um - it’s a really good resource… Other things that I learned a lot from… I learned a lot from reading the actual Gibson blog posts. I learned a ton just by reading through those… and especially the early ones are really fun. They’re really fun to read, because he was not doing this on his job yet, so they’re a little more informal and a lot more reverent, and they’re a really really good read. You can learn a ton from those - a lot more, like, just song references, and kinda goofy stuff. Those are really good. And then the third thing is, I read a lot of - I mean, it was a combination of things. I went back and read a lot of papers - I read a lot of white papers. Which was hard to get into at first, because I was not good at reading white papers… Actually, I’ll mention this because it’s a good trick, but there’s a really nice way to read white papers - or, at least, the way that I found that works for me - which is, you read them three times. And the way you actually read it is - the first time you read it, you just read the intro and you read all of the headlines - all of the main topic points, and you read the conclusion. And that’s it. And then the second time through you read it, but you skip all the diagrams and all the math, and if you need to know more, and it’s important to know more, you go back and read it a third time, and actually read it all. But yeah, I just started reading a bunch of papers - like, I picked up the dynamo paper, and certain papers on CRDTs and that sort of stuff. And I was really fortunate because I was surrounded - hilariously, in Chattanooga, I was surrounded by these people who had a ton of experience doing distributive systems. And they all work for a company called Pylon now, but I’ve been hanging around with those folks for forever and they’ve knew all this distributive systems stuff, so I was able to glean all this knowledge from them, and ask them a lot of questions, and try to steal a bunch of their knowledge - so I think there’s something to that, too, which is - surround yourself with, you know, go to a meetup, or try to find friends who know a little bit about it. Those are the things that have been the most impactful - for me, anyway. AMOS: I didn’t realise Chattanooga had such a big programming community. CHRIS: It’s not a big community… ANNA: It punches above it’s - CHRIS: It punches way above its weight, in terms of the people who are working there, and in terms of the kind of work that they were doing - it’s been a surprise. I didn’t expect to find it in a city of that size. I mean, it’s totally - I came in and these were all people who were real big on Scheme, and now we’re doing Closure because you can no longer get away with writing Scheme for production, and you know, just that sort of mindset - these are people who have been doing functional programming for a long time, and have been doing distributive systems for a long time. And I think as they - as they hired more and more people - they did a think which I think is really really good, and a lot of companies struggle with, which is they hire people who were smarter than them. They hired people who knew more about certain topics than them. And they continued to grow in this community. ANNA: Is that what you are, Chris? Ben was looking at you, and I was like ‘oh, yeah that makes sense’. CHRIS: So… yeah. I’ve been really fortunate. But I - there’s nothing - there’s absolutely nothing better than finding a community who knows their stuff, whether that’s online or in person. And hanging around with those folks and talking about it - you’ll learn a ton just from their experience. You can’t beat hard won knowledge in that space. BEN: There’s a surprising number of connections between the Chattanooga group and the us - like, someone who works here’ll say ‘hey, these guys are coming up from Chattanooga. You wanna have dinner with them?’ And then Chris sends me a message and says ‘I heard you’re having dinner with the Pylon guys’ and I’m like, ‘how do you know this?’ And he’s like, ‘Yeah, we’re close friends.’ And we had dinner and it’s really nice. And then we went - uh, Neil, in Denver, he works for Pylon. And it’s just pretty cool to see how these different people are connected. Because Chris knew about the dinner before I did… So what desert did you visit? AMOS: Oh, I was out north of Santa Fe. I actually went and stayed at a monastery. BEN: Oh, was there like, no speaking? AMOS: No, they talk. But you can put on this little nametag and nobody will talk to you if you wear it. And I did that for one day, and the hardest part was not talking to myself, because I’m so extroverted that I would sit there and start talking to myself, then stop that. And it was good. That was a good thing to do. It made me really think about what I want to say, and so - basically it’s nothing. I don’t want to say anything because I’m afraid that I’ll be wrong. But it - I had a really great time out there. The monastery is supposed to be the most remote monastery in the western hemisphere. So you could go out and see the line of the Milky Way. And just… so many stars… And I kinda live in the middle of the country anyway, so I didn’t expect it to be much different… It was hugely different. It’s nestled in a little, I guess, valley. There’s like a cliff on both sides and a river that goes through there, and every once in a while, you’ll see somebody come through on a raft or a kayak. And other than that, it’s super quiet. It’s just really pristine. ANNA: I love that you brought Purely Functional Data Structures with you. AMOS: Yeah… Some - it didn’t do me much good. But it did look nice on my table. ANNA: Was there anything you gleaned from the first 23 pages that was cool? AMOS: Really, the same thing that James Evergrey said when I went to eat with him - is that, when Chris talked about it being a reference book, and not a bad book, that Chris must be way smarter than me. And James said the same thing, because it was pretty difficult. Although the structures themselves - I would say that I didn’t learn anything life changing. There are things that I already knew but didn’t really think about. Like if you want to change an element in a list functionally, you have to copy every element in the list up to that point. And, you know, I know that - I’ve written it, I’ve done it. It’s just not something in the forefront of my mind, that that’s what’s happening, so it was nice for it to call attention to things like that. CHRIS: Well, and later on - have you gotten to the stuff about like, Banker’s Method and Ameturize time and that sort of stuff? AMOS: No, that’s way passed page 23. CHRIS: Okay. Well, and I think we talked about this - or, we talked about this in our private conversation - in our channel - but, had you done any private ML languages, like Haskell or SML or anything like that, before trying to go through the book? AMOS: I’ve played with Haskell in the past. CHRIS: Okay. Because there’s definitely - if you’re having to both absorb the syntax, and figure out how SML works, and learn the data structures, that can kind of - that can be more challenging. AMOS: I’d done enough Haskell that looking at the syntax in here - it’s not Haskell, but what - SML? It’s not too bad. It did take me just a second to parce the first example. And then after that, it wasn’t too bad. The types signatures are pretty easy. The first example, though, what is there - X in there, like a multiplication symbol, and I thought, ‘what the Hell are they multiplying there?’ And then I realised that was just how they list the arguments together. So. Instead of a comma between arguments, you have an X. CHRIS: So, the big thing is, there probably isn’t anything super life changing in the first 23 pages, but what you’ll find out is… BEN: 24, huh? AMOS: Just rub it in. CHRIS: But page 25 is gonna blow your mind. I mean, the thing you find out is like, the thing with the lists, right? Well, that’s what you use if you want persistent immutable structures, like if you want to have an original copy of it, and not change the original copy. You have to duplicate that data. What you find out is, there are other ways to create lists, other than your typical data structures. You can use trees to implement lists, and in a functional way. And that’s the sort of stuff you start getting into deeper into the book - is, like, efficient ways to build conventional data structures, but using efficient ways in an immutable function. So. I’m sure we’ll get there. AMOS: I did bring two other books, though. ANNA: What other books? AMOS: I brought Pearls of Functional Algorithms CHRIS: That sounds like a good, light read. AMOS: Oh, I actually did get something out of the first pages of it. So, the first exercise in it is - like, if I have a list of numbers - not necessarily in order - what is it… natural numbers? I don’t know - the counting numbers. 1, 2, 3… How do I find the smallest number not on the list. Like, if you’re using it to point to data somewhere, and an efficient algorithm to do that. And they go through like, one way is to go through and order it, and find the first one, even if there’s a gap. Which, to me would take a while, and has a lot of comparisons. But what they ended up doing is using an array that is the same length of your number list. And the array is set to default values of false, and step through your list, and every time you find an integer, and put it in to that position of the array. So, like, if you find one, the first position of the array you change to true. And then you find four, and you place that in the fourth position in the array to true, and then when you get to the list, because you know the list is of length n, you know the smallest number has to be somewhere in length n, because if I have five elements, I would have 1, 3, 7, 2, 12. I know that it has to be less than five. It would either be one through five and it would be six, or somewhere in between. So then they just, go through the array and look for the first position that’s false. And so it reduces the number of comparisons that you’re making. ANNA: That’s cool. CHRIS: I like that. ANNA: What was this book again? Pearls of Functional… AMOS: Pearls of Functional Algorithm Design. This is another one that we should probably talk about at some point. We should start doing a book a month or something. And then the other one I brought was Functional Programming through Lambda Calculus. CHRIS: That sounds fancy. ANNA: How’s that? AMOS: Uh… It’s pretty good. So I’ve looked at Lambda Calculus a few times, and really, the hardest thing that I find with it, is the default Lambda Calculus can be really hard to read. And so they quickly start to allow you to define names and styff, so you can get a decent syntax. So I’m not very far into any of those, but they’re all pretty good. CHRIS: And they’re all light reading. AMOS: Oh, yeah. CHRIS: It’s nice bedtime reading, really. AMOS: That’s what I normally do - is read at night. So, since I was away from my kids and stuff, I thought I could do some reading undisturbed. But I found out that there’s a little bit of learning in my head of ‘when I read, it’s time to sleep’. So even in the middle of the day, I would read a few pages, and then have to put it down and go walk around because I’d start to feel tired. CHRIS: There’s nothing I love more than making a nice cup of tea, curling up on the couch with a blanket, and purely functional data structures book. AMOS: And reading all night. CHRIS: And reading it. I think we’re almost on our - I know we’re on hard deadlines with, like, time. Is there anything you wanted to throw in there, about like - well, just anything you want to bring up before we call it a day? BEN: I am more of a hammock person. CHRIS: I do enjoy a hammock. AMOS: I don’t have anything more. I just need to get this book read, and everybody online says the Ameturize time is the coolest thing in the book. ANNA: Oh, yeah. I need to do that, too. CHRIS: Should we put, like, a - we’ll say we’ll at least have some amount of this read by next time we record? AMOS: By next time? Holey Cow! CHRIS: Well, you wanna say, like, we’ll read - not the whole thing, but like, the first two chapters or something? AMOS: Oh, do these a little bit at a time? I was just gonna say, once a month, we should have a whole book we can talk about. But if you want to do a little bit at a time, that might be better, because there’s a lot in this book. You know more about this book than I do. I only know 23 pages. And probably, I only know twelve of those. BEN: You should do, like, a serious book like that, and then like, a murder mystery. ANNA: Well, we’ll figure it out. Anyway, glad you had a good trip! CHRIS: Yeah, welcome back. AMOS: Thank you. It was nice seeing you guys. Ben, thanks for joining us. ANNA: Thanks, Ben! CHRIS: Yeah, thanks for walking down to the office and doing this, Ben. BEN: Oh, it’s been a pleasure. You can edit me out entirely - I think it’ll be fine. AMOS: We can do that. Just have blank space. ‘Hey, Ben, how’s it going?’ CHRIS: Yeah, don’t crop it. Just cut his voice out. ANNA: Alright, y’all. Bye!