small projects
Sun, Jan 17, 2016I’ve really been enjoying doing some small, useful projects.
I really like figuring out good interfaces for libraries. How others will use my work to build things. So I like tools.
I’ve really been enjoying doing some small, useful projects.
I really like figuring out good interfaces for libraries. How others will use my work to build things. So I like tools.
Rambling thoughts ahead:
So, it looks like after pushing for Haskell, it didn’t work out (yet, at least). I think that it’s partly because my mindset is not currently capable of keeping up with Haskell’s abstraction levels, and also that Haskell is difficult to move from toy to “real program”, at least in my experience.
In any case, in the fall, I decided to focus less on Haskell. The stuff I cared most about was functions like map, filter, and fold, as well as structuring data.
My experience from work has been that not knowing what data is flowing through code is probably the biggest headache. (Javascript!). Followed by lack of compile-time guarantees. Lack of coverage of all control paths is also fairly painful. This is all pretty much summed up by one of my most common Javascript errors: cannot read 'foo' of undefined.
Anyways, Haskell provides solutions to all of the above in spades, but something about how it handles state in larger programs was hard for me to wrap my head around. I think that the choice of threading config/state through a large program or using a monad transformer (which I could only barely understand) were the last straws. So great for Euler Project, but hard for me personally to do other things.
Rust seems to be more accessible for my brain, while also providing many of the guarantees that I enjoyed from Haskell. Plus it has a community geared towards beginners and has great tooling for development.
Since I like challenges, I also enjoy that Rust has it’s own difficult high-level concept: memory management without garbage collection. Or, let’s say it’s memory management done mostly at compile time. That means that in code, I need to know a lot about how long variables live, and how I can borrow them. Fun!
Anyways, long story short, I think that Rust is great for me now, and I can see it carrying me a long way in my career. Itsatisfies my need to tweak, safely. It has enough conveniences that it doesn’t feel like I’m writing C. (It’s more like a verbose Python). It has my favorite high-level concepts from Haskell (data types, type inference, higher-order functions). And it has brackets! Who knew, my time in Javascript has taught me to love brackets.
I still love Haskell, though, and hope that there will be a time I’ll get back to it in the future when I’m more experienced.
So, learning Rust for real this time. Which has also corresponded with leveling up in programming in general.
Over the last 3⁄4 of a year, I started a new job mostly programming in node.js. I’d say that, besides learning node and js, the biggest thing it gave me was the ability to navigate and debug large projects. However, I didn’t really get to practice my chops on creating my own projects, or working on harder problems (or even simply trying to get more performance from my own code).
In November, I decided that I really wanted to invest more time in my coding skills. I still love work, but it would simply not give me enough practice to get to the next level. My plan was to code at a cafe every morning before work, starting at 7am. (Turns out I’m so productive at this time, now that I can actually wake up this early!).
I think that this simple habit has actually done the most for my coding skills. In the previous months, it had been hard to code even 2 hours a week, and now I can easily get 9-10 hours a week.
An additional help was getting good materials (which of course require regular practice to get the most out of!).
My Rust journey restarted, I think, when I watched one of Steve Klabnik’s talks on concurrency in rust. I watched once, got excited, then tried to do the tutorial myself, then watched again. Hooked!
Since then, I also went to a wonderful Rust Meetup taught by Niko Matsakis, which covered similar material but with more depth and some different idioms.
Realizing I needed some easy exercises to get moving, I did Code Kata and Advent of Code.
Bigger projects: I wanted to port some known projects to give me some baseline, so I tried wscat and my own slackbot.
I think the bigger projects were great for doing some struggling. But the small exercises were great for getting some of the basics down.
And along those lines, Learning Rust with Entirely Too Many Linked Lists was a great catalyst recently. After figuring out my own way of laying out programs and parsing using Advent of Code, I found out some more idiomatic ways with this tutorial. It was great to get more familiar with a library setup, as well as using pub, impl, iterators, lifetimes, and take(). I didn’t fully grock some stuff like Rc, RefCell, but it’s there when I need it! I literally typed my way through the entire tutorial, loved it all.
Overall: I just wanted to put down some of the resources that have been great for me learning recently, at a time when I feel like I’m doing really well. Lots of small exercises to ingrain some basics and as a starter for hard days, getting some feedback by going through some tutorials for some more idiomatic usage of Rust. Then pushing against some bigger projects (but not too big, and starting by porting).
Oh yeah, I also stopped reading hackernews.
My latest: just started on porting tinydb from Python. I already implemented my own LRU cache!
I’m very excited to be working as a software developer! I also found a great place to land. Good energy, good people, get things done, and very encouraging towards my development as a programmer.
I’m going to learn a lot about web application architecture in Nodejs and Python. The product also has electronic device components, so I’m excited to learn how the devices and the network connect all together with web back-end and mobile apps.
In the meantime, I’m also planning on continuing my Haskell education. It’s just a lot of fun, and provides some amazingly powerful tools for structuring programs. On my shortlist for reading: Parallel and Concurrent Programming in Haskell; Practical Vim; and Concrete Mathematics.
Even while I’ve been teaching myself to code, it’s been difficult to maintain focus and and finish projects. At work, I already feel like it’s much easier to buckle down on a program. I think programming for an external purpose will help me just keep going and practice more. And that will help keep me disciplined and motivated for my own more “fun” projects.
I really feel like I’m moving forward! I can’t wait to see where this job will take me!
(also, just to insert a piece of code in here:
Why does replicateM 2 [1, 3] give the output it does?)
Today, my friend Andrew was kind enough to take me through some of the basics of Git and Github.
For the MGA ladder, we’re going to set up some software to help manage recording and displaying of results. Hopefully, we’ll also get automatic submission of rated games to the AGA.
I’ve already written some utilities in Haskell to automate updating of the ladder site, using a markdown -> pandoc ->html -> upload to site. So what used to take me 5 minutes now only takes me less than a minute! Maybe not that much absolute time saved, in the grand scheme of things, but I savor every keystroke I save (and especially point and clicks).
The software that Andrew and I are starting will eventually be much more ambitious. Instead of working just on flat files as a data store, and manually running a program to convert results to a website, we’ll be using flask to set up a web framework with a database.
Today’s first major step was to get a workflow so that we could cooperate on the project. Github is one of the go-tos for collaboration, but it’s pretty complicated for a novice. Fortunately, Andrew has a lot of experience with Git for his sofware engineering work. I learned about forking, the difference between upstream and origin (and master), and how to use branches. We stepped through our planned workflow:
We also covered some tools that would be helpful for using Git, like some options for Git log that would enable visualization of branches. I’m also planning on installing the Hub tool.
So, a lot for my brain to take in today. Git is very powerful and flexible, and lots of thanks to Andrew for his help. I don’t think I could have figured this out in even a day on my own, and we spent just one hour.
I can see why Git experience in particular is a highly requested skill for software developers. It’s not just a “version control” (in perhaps the back-up sense), it’s a very powerful coordination software that makes very large engineering projects possible.
Feels like I’m turning a corner! I think that I’m starting to feel the possibility of putting projects together now.
I think a large part of this is simply volume of code. I finished 50 problems on Project Euler (Yay!), which involved a lot of good repetition. I especially became comfortable with the use of higher-order functions. And from the repetition, I’m beginning to see how I would want to organize bigger programs. As usual, not something that I can really understand by reading a book, but it grows organically out of the process of learning. I’m beginning to see my own preferences for how to break up functions, when to use algebraic data types.
The other part was finally getting motivated to do some real-world programming for the MGA ladder. I wrote a small program for computing ladder standings, and for updating files to the ftp server. I mucked around in the plumbing of doing IO and ftp stuff, and found that it wasn’t really that hard. Then, today I re-found some old tutorials on building an IRC bot and an interpreter (Write Yourself A Scheme). Suddenly, not so imposing! Two months ago they would have seemed really imposing!
Of course, this doesn’t mean that I’m comfortable with things like functors and applicatives yet. I understand them theoretically, but I see now that I will need to apply them in practial contexts to really get it. So, I just need to code more! Judging from how I feel after Project Euler, I don’t need to code much bigger projects. In fact, that might be counter-productive. Instead, lots of smaller projects that feel good to finish, and now more variety might be nice.
While trying to get the contents of the current directory in a small Haskell script, I inadvertantly put together some cool connections about Monads and Functors. (disclaimer: this is not a Monad tutorial. Also, beware of ugly code and circuitous reasoning).
The basic idea was to STDOUT the contents of the current directory. Pretty simple? In Haskell pseudocode, I thought:
h <- getDirectoryContents getCurrentDirectory
putStrLn h
Basically, bind the result of the IO operation (get contents of the current directory) to h, and then print h. This did not work on many levels.
Before moving on, I’m going to change the syntax to:
getDirectoryContents getCurrentDirectory >>= putStrLn
The >>= operator may look a little strange. As I was just learning more about Monads, I was excited to try this syntax, it basically binds the result on the left directly to the right side, so the right side can be applied to it. This means that the intermediate step of binding to h in the above example is skipped.
Ok, now that the preliminary is out of the way, why didn’t my function work? Let’s look at the types.
getDirectoryContents :: FilePath -> IO [FilePath]
getCurrentDirectoy :: IO FilePath
putStrLn :: String -> IO ()
FilePath is an alias for a String. That’s pretty straightforward. But what about IO? For me, it was easiest to think of it as a “container” or “computational context.” Because Haskell tries to stay pure, it keeps its Input Output computations in a separate context from “pure” functions.
This idea of a context for values has really helped me understand some of the abstractions of Haskell. For example, a functor is kind of like a container that has the property that you can transform the value inside without affecting the container. This property is expressed in the fmap function, which applies a function to a value (or values) inside of a “box”. fmap can transform integers in a list, or a string inside a Maybe, or even a string inside of IO!
IO also happens to be a Monad, another abstraction built around the idea of a “container.” This time the container encapsulates values and sequences of computations (which appears kind of like imperative programming). With my shallow understanding, I hope I’m not too wrong. In any case, in order to access a value inside a Monad like IO, you need to “reach inside,” which is the idea behind h <- getDirectoryContents "/" By binding the value that was in the IO context, you can work on that value with pure functions.
(Edit 12⁄16/2014: bind is (>>=) :: Monad m => m a -> (a -> m b) -> m b, so the function it takes should take a value but return a Monad (or IO) value. I’ve seen <- described as “slurping” or “draw from”)
Ok, now knowing the basic structure of how things fit together, I needed to actually put it together.
getDirectoryContents takes a FilePath as an argument, but getCurrentDirectory returns an IO FilePath. They don’t match! so what should I do? I just realized now that I could have used <- or >>= to bind the value inside IO FilePath. But I had learned something fancier last night, liftM! And I wanted to use it! I knew that I could “lift” the function into the Monad context, so that it could operate on a value enclosed in a Monad (like IO). So I did, and it worked great, except now there was a nested monad.
liftM getDirectoryContents getCurrentDirectory :: IO(IO [FilePath]))
When I tried to bind the above code to any function, it wouldn’t work because of the nested IO!
But this ugly version would work (you can see the nesting):
h' <- liftM getDirectoryContents getCurrentDirectory
h <- h'
putStrLn h --This line doesn't actually work but I'll explain later
So, as part of this long rambling story, I happened to be reading about functors tonight and decided to explore using fmap to do some simple CLI string manipulation, like so:
main = do
fmap reverse getLine >>= putStrLn
I decided to search and see if I could find an example I remembered of an infinite loop of input and output. Something like interact. But in my search I came across a Stack Overflow question. In the answer, I saw that the reverse bind =<< is equal to fmap followed by join. My reaction was great surprise!
By looking at types, I figured out the fmap part, but still wasn’t sure how using join would get the correct type result. So I looked it up on Hoogle, and I found out that join removes a level of monadic structure! And that was exactly what I had been looking for earlier today!
So, the problem of getting rid of the nesting was solved with join!
join (liftM getDirectoryContents getCurrentDirectory) :: IO [FilePath]
Now I could bind it to a function using the >>= operator. But notice that since the type of the argument would be IO [FilePath], I can’t just use putStrLn, which takes a String and not a [String]. What I’d like to do is map the function putStrLn to each of the values in [FilePath].
Normal map won’t work to print. I guess I’m not completely clear why, but I think that using map doesn’t actually evaluate the monad (in this case putStrLn). mapM maps and also evaluates and also returns a result. mapM_ is more appropriate for this case of just printing, where return value is not needed, since it maps and evaluates but doesn’t return.
So a long-winded and long-winding road to get to the final code:
join (liftM getDirectoryContents getCurrentDirectory) >>= mapM_ putStrLn
It’s probably more cryptic than, say, a typical Python implementation. But it has its own beauty, that comes from understanding the abstractions and the building blocks that come together to build this piece of code. It fits together kind of like Legos.
I think that after this little excursion, and my attempt to write it up, I’ll be able to read a lot more example code! And this was much more fun to me than trying to work through a monster assignment on functors and monads. Start small!
getCurrentDirectory >>= getDirectoryContents >>= mapM_ putStrln
Hmmm… a lot simpler and more elegant, right? Still, I think I learned more by doing it another way first, just because it was what I was thinking at the time. And I had a fun excursion, trying to apply liftM. Now I know liftM and join are together like >==.
Puzzling through like this is fun, but I also see that it required a decent bit of knowledge already. I guess it’s just hard (and fun) to be a beginner.
Any mistakes in understanding on my own, feel free to let me know if there’s mistakes! This is not meant to be a teaching tool, but rather a place for me to think about how Haskell works! I’m practicing my writing skills too, practice makes perfect! (or at least better?)
Exciting! I wrote my first program that’s for “real” work! It’s a small script to upload files to an ftp server that hosts the MGA ladder.
It’s kind of funny, it seems like such a simple program when I look at it now. But maybe one month ago, it might have felt very complicated in Haskell. I think that’s because it helped this time around to know more about monads and >== and liftM. The steps of the script are not hard, nor was calling the libraries (for ftp, directories, etc.). But understanding what Haskell was actually doing (I think that this would be called semantics?) finally happened.
So, a cool little program that saved me the huge annoyance of using a GUI ftp client, while also giving me some fun hands-on coding and digging in libraries, combined with furthering my understanding of Haskell abstractions like monads.
In addition, I also just realized that I can easily extend this utility for helping me convert file formats. I currently write in Markdown and convert to html using pandoc (which is written in Haskell!). I had thought that it was a pity that I couldn’t call the pandoc executable from inside the script, like I might have done with a bash script…. But I can integrate the pandoc library into my program! Even more saved keystrokes coming my way! I may even be able to use this script to apply html headers/templates so that it will be easier to add css and javascript to the basic site.
Of course, eventually the full site will be database driven (Flask, or maybe even Scotty if there’s enough documentation and I have enough guts). But for now, it’s fun working on this utility. Following close behind is another CLI utility to generate AGA ratings submission forms.
It’s the end of the fall (and almost the end of the year) and I’m still looking. I’m actually fairly optimistic, though. I hope to have some nice projects up in time for the new year, I feel like I’ll have some good chances then.
I’ve been growing my skills a lot, it just takes a lot of time. At least for me, it’s difficult to learn by just blasting away at a complex project for a couple of weeks. I’ve been doing a lot of smaller bits of code, especially Project Euler, while also investigating some larger scale stuff. I’m going to write a program to help me manage the MGA ladder, because that’s a pretty big pain point (I hate repetitive data entry tasks!).
I think in terms of learning, I’ve gotten to a point where books will actually be useful to me now. I just had to do a lot of coding in whatever my way my brain wanted to, now I have a jumping off point for understanding more interesting stuff. So I’m going to go through LYAH again, and I’ve ordered the data processing cookbook by Shuklan.
I tried to use books before, but they felt too “boring,” which I guess was my brain’s code that I couldn’t internalize the information in a useful way. I’ve read some stuff that said that writing code is most important for beginners (even if it’s not “good code”), and then reading more code is more important later. I think I was prematurely optimizing a bit, thinking about how to make my code “good” and thinking about clean code. Even in Haskell, I can do quite a bit without understanding all the higher level concepts.
I’ve also started going through algorithms and data structure with a friend from the Go club. I think that it’s going to be super helpful. I forget if I mentioned it in a previous post, but I’m realizing that I’m a very social learner. While I do like to read, most of the integration comes from talking with others. And not just listening, it has to be interactive. I think that’s why it felt easy to learn Go and martial arts: lots of social feedback.
I’ve also noticed that online classes don’t really do it for me. It’s very difficult for me to sit passively through a lecture now. So, I won’t be doing a lot of online courses on my own, although I’ll probably pick up bits and pieces here and there, and I might go through exercises with a friend.
So, I’m going to apply this newfound knowledge! Less “trying to figure things out on my own,” more looking for like-minded people for all the things I want to learn: yoga, martial arts, Go, programming, languages. (I think that at this point in my life, I’m also just less embarassed about what I don’t know or can’t do).
Since my last post, I’ve been doing a lot of investigating. I could have jumped even further down the Python rabbit hole by immediately trying to start on projects. I think two things stopped me: one, that I didn’t have a pressing need that I could address with a project, and two, that my brain was telling me it wanted to explore more what programming could mean. The result was a foray into Haskell.
Haskell is described as a lazy, functional language that has a very strong notion of purity. Great explanations of the language can be found at haskell.org.
I had messed around with Learn You a Haskell, a great introductory book, but had not really gotten a grasp on the language. I decided to work through CIS194, a class on Haskell taught by Brent Yorgey at UPenn, with all materials online! The exercises ended up being at a great level for me (similar to Think Python). I’ve currently worked through the midway point, right before things get abstract.
I think that, coming from Python, my greatest delights were:
I’ve found that for me, the syntax makes it really easy to see the structure of the program. Visual noise takes away from shape, and I’m extremely shape-oriented (as I know from playing Go).
The Algebraic Data Types (ADT) were a revelation in terms of rethinking how to organize programs. For me, they make a lot more sense than what I know about classes and OO programming. ADTs in Haskell are strict but extremely expressive, and can tell you so much about the shape of a program up-front without delving into “details” of execution.
Of course, to operate in the real world, there needs to be a certain amount of sacrifice of purity. In Haskell, this often has to do with I/O. I haven’t gotten to really grapple with this part yet, but I’m looking forward to it as I continue to learn.
Another language I looked at recently was Rust, which is a new language often compared to Golang. I’m glad I branched out to Haskell, because I learned some things I really enjoyed in programming (like ADT), and those things really popped out when looking at Rust. Rust seems to be another imperative programming language in the vein of Python or C++, but what struck me was that it had its own implementation of ADT! I wrote some toy programs in Rust, thinking that perhaps the combination of imperative with functional would fit my mindset really well and make programming easy. But Rust has another trick up its sleeve: memory-safe programming, which is actually the language’s main calling card. I only know a little, but it looks like Rust places responsibility for keeping track of when and where memory is allocated to variables, while also creating a framework so that the compiler can help you keep track.
Thinking about memory, as well as pointers and mutability explicitly in programs, is pretty complicated above just making my program run. Python basically took care of it all for me, with some cost in terms of performance, structure, and safety. Haskell takes care of those things to, but in part by limiting the programming paradigm. The more technical, close-to-the-metal stuff hurts my head a bit, and was definitely one of the reasons I found programming hard in high school and college. (glad I was reintroduced through Python!). But I’m glad to know what’s possible in a programming language.
I want to keep investigating both Rust and Haskell. I guess it’ll be a little bit of juggling as I also want to learn Python. But it does seem to me that picking up a bit of syntax or learning a new language is not too hard for very basic competency (especially C related languages like Rust or Python). I’ve done it a few times now, with these languages and also Java and Turbo Pascal in the past.
The real challenge for me is not being fluent in the “syntax” of the language, but to learn how to construct programs. Right now, I don’t know how much of this will relate to the idioms of each language (the structure of a Haskell is different in certain ways from a Python program). But I do know that learning different idioms is good. In fact, learning Haskell is one recommended way of getting better at Python (or even programming in general).
In any case, I’m going to keep plugging away and finding bigger projects, whatever they may be. Currently on my radar are a toy browser engine in Rust, some exploration of AGA ratings, some graphing for a nonlinear dynamics class, and hopefully some fun small project in Haskell. I won’t try to go too big; I know that I need to build lots of small projects first.
So does any of this have to do with a job search? After all, I’m not building my skills in a pattern that necessarily makes it obvious to future employers that I’m getting better. I’ve been told quite a few times that it’s important to have projects to show, especially for somebody who is self taught.
I guess I wanted to give myself something to aspire to. While web development in and of itself isn’t necessarily a boring job, I wanted to know a little better what it would take to tackele interesting computational problems, whether in web development or in another industry. I didn’t want to just aim for patching together frameworks.
I’m still glad I took the time to look around at the landscape and dig into some fundamentals (I implemented binary trees in Haskell!). I know it will serve me well in the future to have a broad outlook even at the beginning. I just have to keep going so I can build skills that will actually be useful to employers, while also maintaining my enthusiasm.
I’m starting a new episode in my life! About one month ago, I committed myself to learning programming and getting a job as a software developer. For at least the next five years, I want to learn some new skills and explore a new industry. After that, I’ll reassess and see where I want to go next.
So far, I’ve learned Python through Think Python by Allen Downey, which I think is an excellent book. I appreciated the thoughtful exercises, and the focus on teaching only what’s useful in the moment while building towards a bigger project. I find that too much information without implementation results in shallow learning. Perhaps for other learners, there’s a different sweet spot, but for me Think Python worked great. I’m even planning on working through his book Think Complexity.
With some Python basics under my belt, I’m investigating some other projects. The way my brain works, the computational aspect of CS doesn’t seem daunting, but the organizational part does. For example, data structures in a void make sense, but what’s the best way to organize them to do something useful? I guess this is the software engineering side, and it looks quite open-ended and complex.
My current plans for Python projects are basic inventory management from CLI and to investigate Flask. However, I’ve realized that I’m not very results driven, I’m more of a process person. Of course, getting something to work is satisfying, but I feel uncomfortable with magic and frameworks at the moment because I don’t understand what’s going on. So as I continue to explore, those projects may never get finished.
On the theme of less magic, I’ve also been investigating Golang. I’ve finally got it set up, and done some tutorials. I like the simplicity of the language, and I want to use it to build a simple webserver first. Perhaps to serve sgf files (Go game records).
There is, of course, a ton of other stuff I also find interesting: SICP, Haskell, algorithms, data mining and analyis, numerical computing and modelling. I’m all too capable of running off in many different directions. So I’m going to shelve those for now and concentrate on Python and Golang.
I’ve been networking with Williams alumni and they’ve been very kind and helpful. I may have an internship set up for the near future, and I’ll also be looking for other opportunities through more networking. I’ve found that my default of responding to Craigslist ads has not worked especially well, maybe because of my unconventional background. Although, I’m also planning on sending some cold emails.
Walther Chen, aspiring software developer!
Currently interested in:
Some stuff for the future:
Some other stuff I love doing:
I help organize the Massachusetts Go Association and its ladder tournament.
In previous lives I’ve been a:
If you’re interested in any of the above, and you’re in Cambridge MA, feel free to contact me!