WEBVTT 1 00:00:01.150 --> 00:00:03.650 line:15% The topic this week is the functional core 2 00:00:03.650 --> 00:00:06.680 line:15% of an application versus the imperative shell. 3 00:00:06.680 --> 00:00:08.770 We're going to use my Twitter client once again 4 00:00:08.770 --> 00:00:11.870 as an example, because it is constructed in this way. 5 00:00:11.870 --> 00:00:14.770 And to start us off, let's look at the simplest 6 00:00:14.770 --> 00:00:18.480 functional class in the system, the tweet. 7 00:00:18.480 --> 00:00:21.750 We saw this class last time, it has an ID, a username 8 00:00:21.750 --> 00:00:24.260 and a text, and it has two very simple methods 9 00:00:24.260 --> 00:00:27.850 to just return values, no mutation going on. 10 00:00:27.850 --> 00:00:30.440 Moving on to something a little bit more complex, 11 00:00:30.440 --> 00:00:34.220 there is a timeline class that stores an array of tweets 12 00:00:34.220 --> 00:00:36.230 and it knows how to update itself, 13 00:00:36.230 --> 00:00:39.160 given some new tweets coming in from the API. 14 00:00:39.160 --> 00:00:40.930 But of course when I say update itself, 15 00:00:40.930 --> 00:00:43.500 I don't mean that it changes its own state, 16 00:00:43.500 --> 00:00:45.930 I mean that it constructs a new timeline 17 00:00:45.930 --> 00:00:48.980 with the new tweets plus the old tweets. 18 00:00:48.980 --> 00:00:50.900 Moving on to something With a little bit more 19 00:00:50.900 --> 00:00:54.030 complex behavior, there is a cursor class 20 00:00:54.030 --> 00:00:58.340 and its job is to remember where the cursor is on the screen 21 00:00:58.340 --> 00:01:00.730 and move it up and down and it acts a lot 22 00:01:00.730 --> 00:01:03.870 like the cursor in vim moving up and down. 23 00:01:03.870 --> 00:01:05.470 This class is quite a bit larger, 24 00:01:05.470 --> 00:01:06.820 it's almost a hundred lines. 25 00:01:06.820 --> 00:01:09.220 I think it's actually too big and its responsibilities 26 00:01:09.220 --> 00:01:11.950 are sort of muddy and I'm not happy with it, 27 00:01:11.950 --> 00:01:14.670 but I am happy with the fact that it is immutable, 28 00:01:14.670 --> 00:01:17.040 it has an array of tweets and then a selection 29 00:01:17.040 --> 00:01:20.060 in those tweets and it has methods that construct 30 00:01:20.060 --> 00:01:23.280 new cursors when we move up or move down 31 00:01:23.280 --> 00:01:26.900 in the tweet list and they do not change any state 32 00:01:26.900 --> 00:01:29.700 on this object or any other object. 33 00:01:29.700 --> 00:01:32.640 Finally as our last example of the functional core, 34 00:01:32.640 --> 00:01:34.810 there is a tweet renderer and we built 35 00:01:34.810 --> 00:01:37.840 a very simple version of this in the previous screencast. 36 00:01:37.840 --> 00:01:40.280 But this one, as you can see is quite a bit longer 37 00:01:40.280 --> 00:01:42.040 about 50 lines of code. 38 00:01:42.040 --> 00:01:44.890 It takes in an array of tweets, a selected tweet 39 00:01:44.890 --> 00:01:47.920 and a height of the screen and it ultimately ends up 40 00:01:47.920 --> 00:01:50.530 returning an array of lines of text 41 00:01:50.530 --> 00:01:53.860 that the screen class should draw. 42 00:01:53.860 --> 00:01:56.580 These are all examples of the functional core. 43 00:01:56.580 --> 00:02:00.300 We do things like storing tweets, storing timelines, 44 00:02:00.300 --> 00:02:03.610 merging timelines, moving the cursor up and down 45 00:02:03.610 --> 00:02:05.770 or at least remembering where it is 46 00:02:05.770 --> 00:02:07.740 and rendering tweets onto the screen. 47 00:02:07.740 --> 00:02:10.320 But all of those things are functional. 48 00:02:10.320 --> 00:02:12.840 The only mutation that's ever done in any of these 49 00:02:12.840 --> 00:02:16.210 is to rebind local variables like you see right here 50 00:02:16.210 --> 00:02:18.840 we have a lines variable and then we potentially 51 00:02:18.840 --> 00:02:20.210 rebind it later. 52 00:02:20.210 --> 00:02:22.260 But that's not visible from the outside. 53 00:02:22.260 --> 00:02:24.820 So it doesn't affect the fact that these objects 54 00:02:24.820 --> 00:02:28.270 are immutable from an outside perspective. 55 00:02:28.270 --> 00:02:30.490 If we have all of this functional code 56 00:02:30.490 --> 00:02:32.560 and nothing is actually being mutated then 57 00:02:32.560 --> 00:02:34.810 how do we do things like handle user input, 58 00:02:34.810 --> 00:02:38.000 talk to the network, update the local database, 59 00:02:38.000 --> 00:02:39.560 all that kind of stuff? 60 00:02:39.560 --> 00:02:41.770 Well, that's all done by the imperative show 61 00:02:41.770 --> 00:02:44.420 which is encapsulated in a single file, 62 00:02:44.420 --> 00:02:47.880 mostly at least about 160 lines long. 63 00:02:47.880 --> 00:02:52.040 And this is the file that does things like read standard in 64 00:02:52.040 --> 00:02:56.100 or handle updates from the network or update references 65 00:02:56.100 --> 00:02:57.400 to those immutable objects 66 00:02:57.400 --> 00:03:00.330 as the state of the world changes. 67 00:03:00.330 --> 00:03:02.850 Let's go back to the example of moving the cursor 68 00:03:02.850 --> 00:03:04.380 up and down on the screen 69 00:03:04.380 --> 00:03:06.660 and you can see that happening right here. 70 00:03:06.660 --> 00:03:10.400 If I hit J it moves down, if I hit K it moves up. 71 00:03:10.400 --> 00:03:12.450 But in both cases, we're just calling a method 72 00:03:12.450 --> 00:03:14.640 on the cursor that constructs a new one 73 00:03:14.640 --> 00:03:18.060 and then assigning it to this instance variable. 74 00:03:18.060 --> 00:03:20.470 Right above this method you see another example 75 00:03:20.470 --> 00:03:22.390 where we construct an array of lines 76 00:03:22.390 --> 00:03:25.140 by using the tweet renderer and then we render them 77 00:03:25.140 --> 00:03:27.900 onto the screen which is a destructive action. 78 00:03:27.900 --> 00:03:29.860 We covered this last time so I'm not going to go 79 00:03:29.860 --> 00:03:31.600 into any detail about it, 80 00:03:31.600 --> 00:03:34.470 but I will go into detail about a third example 81 00:03:34.470 --> 00:03:38.520 which involves pulling down new tweets from the Twitter API. 82 00:03:38.520 --> 00:03:41.010 As the application is starting up, this run method 83 00:03:41.010 --> 00:03:42.630 is the first one that runs. 84 00:03:42.630 --> 00:03:44.780 It does some stuff we're going to gloss over. 85 00:03:44.780 --> 00:03:46.210 But one of the things it does 86 00:03:46.210 --> 00:03:48.840 is to start the timeline stream. 87 00:03:48.840 --> 00:03:51.840 This is a background thread that pulls new tweets down 88 00:03:51.840 --> 00:03:54.510 from Twitter and feeds them into the main thread 89 00:03:54.510 --> 00:03:56.200 via a queue. 90 00:03:56.200 --> 00:03:59.470 If we jump down to the start timeline stream method 91 00:03:59.470 --> 00:04:02.040 you can see that it constructs a timeline stream 92 00:04:02.040 --> 00:04:05.300 and then it starts a thread that just runs in infinite loop 93 00:04:05.300 --> 00:04:07.070 inside of that object. 94 00:04:07.070 --> 00:04:08.920 Jumping down to the class, 95 00:04:08.920 --> 00:04:10.640 that method is sort of unsurprising, 96 00:04:10.640 --> 00:04:11.920 it has an infinite loop. 97 00:04:11.920 --> 00:04:15.090 It pulls down new tweets and constructs a new timeline 98 00:04:15.090 --> 00:04:17.810 and then it throws that timeline into the queue. 99 00:04:17.810 --> 00:04:20.120 Now, this does mean that that timeline object 100 00:04:20.120 --> 00:04:22.640 is shared between two different threads, 101 00:04:22.640 --> 00:04:27.190 but I have no fear of a race condition or of mutating it 102 00:04:27.190 --> 00:04:30.760 in the wrong way because the object is immutable. 103 00:04:30.760 --> 00:04:33.280 So there's just no danger there. 104 00:04:33.280 --> 00:04:35.210 Once we put that timeline into the queue 105 00:04:35.210 --> 00:04:37.530 the main thread is going to wake up and pull it out. 106 00:04:37.530 --> 00:04:39.570 So let's find where that happens. 107 00:04:39.570 --> 00:04:42.110 And it is a call to the select system call 108 00:04:42.110 --> 00:04:45.060 on both standard in, so we might be reading in 109 00:04:45.060 --> 00:04:47.980 input from the user or we might be reading in 110 00:04:47.980 --> 00:04:49.580 new stuff in the queue. 111 00:04:49.580 --> 00:04:52.550 This is an instance of a special queue class that I have 112 00:04:52.550 --> 00:04:54.610 that is usable in a select statement 113 00:04:54.610 --> 00:04:58.040 because it has a fake IO object inside. 114 00:04:58.040 --> 00:05:00.310 Once we find an event with this select 115 00:05:00.310 --> 00:05:03.230 if it is a timeline event, then we call handle timeline 116 00:05:03.230 --> 00:05:05.170 which is the interesting part of this. 117 00:05:05.170 --> 00:05:08.420 And scrolling down, we pull the new timeline object 118 00:05:08.420 --> 00:05:09.253 out of the queue. 119 00:05:09.253 --> 00:05:12.330 So this is a complete timeline fed to us 120 00:05:12.330 --> 00:05:14.460 by the background thread. 121 00:05:14.460 --> 00:05:16.290 We then write it to the database 122 00:05:16.290 --> 00:05:18.060 and then we update the world. 123 00:05:18.060 --> 00:05:22.220 The world is really just a wrapper object for a timeline. 124 00:05:22.220 --> 00:05:24.880 The first interesting part of this is that 125 00:05:24.880 --> 00:05:26.300 as I mentioned before, 126 00:05:26.300 --> 00:05:28.690 the timeline is constructed in the background thread, 127 00:05:28.690 --> 00:05:31.230 it's read here, but there's no risk of race 128 00:05:31.230 --> 00:05:33.030 because it's immutable. 129 00:05:33.030 --> 00:05:36.300 The second interesting part is the database update. 130 00:05:36.300 --> 00:05:38.710 Just like the standard in manipulation 131 00:05:38.710 --> 00:05:41.320 the standard out manipulation, the main thread 132 00:05:41.320 --> 00:05:43.730 is the only one who touches the database. 133 00:05:43.730 --> 00:05:46.440 So I have no fears about multiple threads 134 00:05:46.440 --> 00:05:49.030 racing on the database or racing on standard in 135 00:05:49.030 --> 00:05:50.520 or on standard out. 136 00:05:50.520 --> 00:05:54.290 All of these things are effectively synchronized for free 137 00:05:54.290 --> 00:05:57.030 because only the main thread touches them. 138 00:05:57.030 --> 00:06:00.170 So that is the imperative shell that marries 139 00:06:00.170 --> 00:06:02.890 the state of the world to the state of the program 140 00:06:02.890 --> 00:06:04.860 as well as storing that state. 141 00:06:04.860 --> 00:06:07.440 And we also saw the functional core, which is the bulk 142 00:06:07.440 --> 00:06:10.230 of the program containing all of the logic 143 00:06:10.230 --> 00:06:12.810 and the data manipulation. 144 00:06:12.810 --> 00:06:15.420 Now let's talk about the implications for testing 145 00:06:15.420 --> 00:06:18.300 because I care a lot about testing. 146 00:06:18.300 --> 00:06:20.730 First of all, all of that functional core code 147 00:06:20.730 --> 00:06:22.670 is tested in isolation. 148 00:06:22.670 --> 00:06:27.380 So if we look at the timeline class that's the simplest one, 149 00:06:27.380 --> 00:06:30.080 it has very simple tests that take a timeline, 150 00:06:30.080 --> 00:06:31.380 add a new tweet to it 151 00:06:31.380 --> 00:06:33.980 and then assert that the right tweets came out. 152 00:06:33.980 --> 00:06:37.200 This is a single line of code that reads very easily, 153 00:06:37.200 --> 00:06:40.200 whereas if this were doing mutation, then we would have to 154 00:06:40.200 --> 00:06:43.960 construct a timeline, destructively push new tweets into it 155 00:06:43.960 --> 00:06:45.960 and then assert that it's state changed. 156 00:06:45.960 --> 00:06:49.670 So that would probably be three different lines of code. 157 00:06:49.670 --> 00:06:51.460 Of course timeline is very simple, 158 00:06:51.460 --> 00:06:54.590 so let's look at something more complex like cursor. 159 00:06:54.590 --> 00:06:57.490 The cursor tests are actually very similar. 160 00:06:57.490 --> 00:06:59.420 There are a lot more lines and they're a little bit 161 00:06:59.420 --> 00:07:02.980 more dense, but if we look at this slicing behavior 162 00:07:02.980 --> 00:07:06.160 this is where we want to give the cursor a given tweet 163 00:07:06.160 --> 00:07:09.340 and return everything starting at that tweet and later. 164 00:07:09.340 --> 00:07:12.130 So this is how it renders onto the screen, for example. 165 00:07:12.130 --> 00:07:15.570 This is how it finds out which tweets are visible. 166 00:07:15.570 --> 00:07:18.380 If we have a cursor that has two tweets inside of it 167 00:07:18.380 --> 00:07:20.140 and we tell it to start at tweet one 168 00:07:20.140 --> 00:07:22.290 then we should get tweet one tweet two. 169 00:07:22.290 --> 00:07:23.810 If we tell it to start a tweet two, 170 00:07:23.810 --> 00:07:25.440 we should only get tweet two. 171 00:07:25.440 --> 00:07:27.540 And if we tell it to start at some third tweet 172 00:07:27.540 --> 00:07:29.040 that isn't actually in the list, 173 00:07:29.040 --> 00:07:31.180 then we should get an index error. 174 00:07:31.180 --> 00:07:34.010 So we have three different assertions here in this test. 175 00:07:34.010 --> 00:07:36.340 Each of them is a single line and each of them 176 00:07:36.340 --> 00:07:40.070 is very simple explicit input and output. 177 00:07:40.070 --> 00:07:43.540 Finally, let's move on to the tweet renderer class. 178 00:07:43.540 --> 00:07:47.100 Last time we TDD this and our tests were very simple. 179 00:07:47.100 --> 00:07:49.460 The actual tests are a little bit more complex 180 00:07:49.460 --> 00:07:52.540 because it has more complex behavior, like for example, 181 00:07:52.540 --> 00:07:54.220 it renders multiple tweets. 182 00:07:54.220 --> 00:07:57.010 And the thing that it renders is not plain text 183 00:07:57.010 --> 00:08:00.170 but these text objects that have color information 184 00:08:00.170 --> 00:08:01.920 encoded inside of them. 185 00:08:01.920 --> 00:08:03.480 But even with those complexities 186 00:08:03.480 --> 00:08:05.640 you can see this test is very simple. 187 00:08:05.640 --> 00:08:07.270 It constructs two tweets, 188 00:08:07.270 --> 00:08:09.240 it throws them into the tweet render 189 00:08:09.240 --> 00:08:11.670 and it asserts that it gets the right lines out. 190 00:08:11.670 --> 00:08:13.640 There's very little complexity here. 191 00:08:13.640 --> 00:08:15.910 There's no need to track down other classes 192 00:08:15.910 --> 00:08:17.950 because it's all data in data out. 193 00:08:17.950 --> 00:08:20.820 And there's no need to reason about mocking or stubbing 194 00:08:20.820 --> 00:08:23.100 because there just isn't any. 195 00:08:23.100 --> 00:08:25.510 Speaking of a lack of mocking and stubbing, 196 00:08:25.510 --> 00:08:28.220 if we search the source code for the word mock, 197 00:08:28.220 --> 00:08:29.510 it doesn't show up. 198 00:08:29.510 --> 00:08:31.760 And if we search for stub, it does show up 199 00:08:31.760 --> 00:08:35.240 in exactly one place but that's because for some reason 200 00:08:35.240 --> 00:08:38.300 I stubbed instead of using the tweet value object. 201 00:08:38.300 --> 00:08:40.980 I have the feeling that this was the very first test 202 00:08:40.980 --> 00:08:44.190 or pair of tests that I wrote for this class 203 00:08:44.190 --> 00:08:45.870 but I could easily replace those 204 00:08:45.870 --> 00:08:47.790 with an actual tweet value object 205 00:08:47.790 --> 00:08:50.370 and then this entire system would be tested 206 00:08:50.370 --> 00:08:52.780 without a single test double even though, 207 00:08:52.780 --> 00:08:55.810 almost all of it is tested in isolation. 208 00:08:55.810 --> 00:08:58.480 Of course this raises a pretty obvious question. 209 00:08:58.480 --> 00:09:00.340 If I'm testing everything in isolation 210 00:09:00.340 --> 00:09:02.410 and I'm not using mocks or stubs 211 00:09:02.410 --> 00:09:06.220 then how am I testing the imperative show around the program 212 00:09:06.220 --> 00:09:08.440 where all of these references are being updated, 213 00:09:08.440 --> 00:09:11.910 it's touching the network and standard in standard out 214 00:09:11.910 --> 00:09:13.540 all these stuff? 215 00:09:13.540 --> 00:09:15.160 Well, the answer may surprise you 216 00:09:15.160 --> 00:09:17.970 but this file is not tested at all. 217 00:09:17.970 --> 00:09:22.350 All 165 lines of this the entire outer shell of the program 218 00:09:22.350 --> 00:09:24.320 contains no tests. 219 00:09:24.320 --> 00:09:26.570 Now, there are a couple of reasons that I do this, 220 00:09:26.570 --> 00:09:29.750 the first of which is sort of technical in nature. 221 00:09:29.750 --> 00:09:32.250 There are very few conditionals in this code. 222 00:09:32.250 --> 00:09:35.350 If we start scrolling down you see a lot of flat methods, 223 00:09:35.350 --> 00:09:36.520 flat methods. 224 00:09:36.520 --> 00:09:40.190 There is a loop here, but there's very few conditionals. 225 00:09:40.190 --> 00:09:42.680 Here's one, but both of these have to happen 226 00:09:42.680 --> 00:09:44.700 for any tweets to get into the system, 227 00:09:44.700 --> 00:09:47.080 so I'm not really afraid of those breaking. 228 00:09:47.080 --> 00:09:50.010 And if I keep scrolling down, we have once again a loop 229 00:09:50.010 --> 00:09:51.790 but no conditionals. 230 00:09:51.790 --> 00:09:53.480 Scrolling down, scrolling down. 231 00:09:53.480 --> 00:09:56.670 Here's a conditional, but this is easy enough to test. 232 00:09:56.670 --> 00:09:58.850 I use all of these keys all the time. 233 00:09:58.850 --> 00:10:01.670 If this became more complex, then I would become afraid 234 00:10:01.670 --> 00:10:03.670 and I would really want to test this. 235 00:10:03.670 --> 00:10:06.940 But other than this block of code, there's nothing in here 236 00:10:06.940 --> 00:10:09.660 that I'm really afraid of going out of sync. 237 00:10:09.660 --> 00:10:11.910 The second reason that I don't test this file 238 00:10:11.910 --> 00:10:15.420 is that I tend to just cowboy code into here to either 239 00:10:15.420 --> 00:10:17.540 explore a new library that I don't know 240 00:10:17.540 --> 00:10:19.860 because I didn't know any of the Twitter libraries 241 00:10:19.860 --> 00:10:22.030 before I started this or to explore 242 00:10:22.030 --> 00:10:24.090 just how the application should work. 243 00:10:24.090 --> 00:10:25.880 I often don't know what I want, 244 00:10:25.880 --> 00:10:29.760 so I write some pretty nasty code in here, figure it out 245 00:10:29.760 --> 00:10:32.410 and once I know what it's going to do or I at least have 246 00:10:32.410 --> 00:10:35.610 a decent design then I will move the source code 247 00:10:35.610 --> 00:10:39.780 into unit tested files often by doing TDD. 248 00:10:39.780 --> 00:10:41.240 To make that a bit more concrete, 249 00:10:41.240 --> 00:10:43.720 let's look at a piece of the git history, 250 00:10:43.720 --> 00:10:47.210 you can see right here I add the tweet renderer class. 251 00:10:47.210 --> 00:10:48.710 And if we look at this commit, 252 00:10:50.320 --> 00:10:53.480 it adds a new spec file, it adds a new production code file 253 00:10:53.480 --> 00:10:55.220 but it changes nothing else. 254 00:10:55.220 --> 00:10:59.140 So this new code was not used by the actual system. 255 00:10:59.140 --> 00:11:02.030 When this code was written, the system did already have 256 00:11:02.030 --> 00:11:04.850 this behavior and I was reimplementing it. 257 00:11:04.850 --> 00:11:07.840 I was doing TDD, but I also was referring 258 00:11:07.840 --> 00:11:09.160 to the existing code. 259 00:11:09.160 --> 00:11:11.490 So they did end up being very similar 260 00:11:11.490 --> 00:11:14.110 despite having slightly different interfaces. 261 00:11:14.110 --> 00:11:16.270 Now, if we look at the history again, 262 00:11:16.270 --> 00:11:18.770 after I add the tweet renderer, there's a commit that says 263 00:11:18.770 --> 00:11:22.130 replace untested tweet drawing with tweet renderer. 264 00:11:22.130 --> 00:11:23.820 And if we look at that commit, 265 00:11:23.820 --> 00:11:27.410 it adds one little piece to the screen class, 266 00:11:27.410 --> 00:11:29.930 which is not very interesting but scrolling down, 267 00:11:29.930 --> 00:11:33.430 you see a lot of red and this is all in that main file 268 00:11:33.430 --> 00:11:35.880 in the imperative shell of the program. 269 00:11:35.880 --> 00:11:38.560 All of this code went away and was replaced 270 00:11:38.560 --> 00:11:42.440 with a unit tested tweet renderer class. 271 00:11:42.440 --> 00:11:46.150 This is the cadence of the entire project so far. 272 00:11:46.150 --> 00:11:49.860 I tend to bloat up this outer file up to something like 273 00:11:49.860 --> 00:11:52.060 250 lines in some cases. 274 00:11:52.060 --> 00:11:54.600 And then I extract behavior out, moving it 275 00:11:54.600 --> 00:11:56.930 into unit tested classes, figuring out 276 00:11:56.930 --> 00:11:58.650 how to make it more functional. 277 00:11:58.650 --> 00:12:02.320 So this is not by any means a traditional TDD flow. 278 00:12:02.320 --> 00:12:05.360 Although in most cases, when I pull those classes out 279 00:12:05.360 --> 00:12:07.670 I do do TDD there. 280 00:12:07.670 --> 00:12:10.690 We've now seen the functional core, the imperative show 281 00:12:10.690 --> 00:12:12.530 and the testing implications. 282 00:12:12.530 --> 00:12:15.000 Obviously it is not going to be easy to transition 283 00:12:15.000 --> 00:12:17.080 any given program into this style 284 00:12:17.080 --> 00:12:19.330 where you have a single functional core 285 00:12:19.330 --> 00:12:22.710 and a single thin imperative shell wrapped around it. 286 00:12:22.710 --> 00:12:24.650 I think that the more important thing is to 287 00:12:24.650 --> 00:12:26.950 think about where the mutation is happening 288 00:12:26.950 --> 00:12:29.700 and try to localize it as much as possible. 289 00:12:29.700 --> 00:12:31.620 I think that the ideal program contains 290 00:12:31.620 --> 00:12:35.810 a whole lot of immutable code in sort of functional style 291 00:12:35.810 --> 00:12:38.550 and then small pieces of mutable code 292 00:12:38.550 --> 00:12:42.050 doing imperative things, but having them be very localized 293 00:12:42.050 --> 00:12:44.200 and separate from the data 294 00:12:44.200 --> 00:12:47.900 and separate from the core behavior of the system. 295 00:12:47.900 --> 00:12:49.980 I feel like I'm about to start preaching so I think 296 00:12:49.980 --> 00:12:52.280 that's probably a good place to stop. 297 00:12:52.280 --> 00:12:54.470 That is going to be it for the functional core, 298 00:12:54.470 --> 00:12:57.610 the imperative shell and the testing implications of them. 299 00:12:57.610 --> 00:13:00.503 And I will see you in two weeks.