Marzhill Musings

<<Prev ...

go-html-transform an html transformation and scraping library

Published On: 2013-02-26 17:05:00

http://code.google.com/p/go-html-transform is my html transformation library for go. I use it as an html templating language and scraping library. It's not your typical approach to html templating but it's an approach I've really come to enjoy. HTML templating can be grouped in roughly about 3 categories.

  1. Templating languages.
  2. HTML DSLs.
  3. Functional transforms.

go-html-transform is an example of that last one. The basic theory is that an html template is just data. No logic is in the template. All the logic is in the functions that operate on the template and any input data. Using the input data you can transfrom a template and then render the transformed AST back into html. This has a number of benefits.

  • Your template transforms are context aware.
  • Multipass templating is just another transform.
  • All your logic is expressed in real honest to goodness code not a limited templating language. In the case of go-html-transform your templating logic is actually typechecked by the go compiler.
  • It's impossible to generate bad html.
  • Your mocks are your templates.
  • You can use an html dsl in combination with this approach as well if the dsl outputs the same AST.

Example usage.

    1 package main
    2 
    3 import (
    4   "strings"
    5   "os"
    6 
    7   "code.google.com/p/go-html-transform/html/transform"
    8   "code.google.com/p/go-html-transform/h5"
    9 )
   10 
   11 func toSSL(url string) string {
   12   return strings.Replace("http:", "https:", 1)
   13 }
   14 
   15 func main() {
   16   f, err := os.Open("~/file.html")
   17   defer f.Close()
   18   if err != nil { return } // handle errors here.
   19   tree, err := transform.NewDocFromReader(f)
   20   if err != nil { return } // handle errors here.
   21   t := transform.NewTransformer(tree)
   22   t.ApplyAll(
   23     Trans(ReplaceChildren(h5.Text("foo"), "span"), // replace every span tags contents with foo
   24     // turn every link and img into an ssl link
   25     Trans(TransformAttrib("href", toSSL), "a"),
   26     Trans(TransformAttrib("src", toSSL), "img"),
   27   )
   28 
   29   t.Render(os.Stdout) // render html to stdout.
   30 }

Tags:

From Homeless to a Career

Published On: 2013-02-22 12:14:00

A number of folks have requested I write up the story of how I went from homeless with kids and a wife to a career as a software developer. The story could start from any number of places but I'll start mine as a young teenager. Our family didn't have a lot of money. I wasn't really aware of it since my parents did a good job of ensuring we always had food and a place to live. One year though they took the tax return and used it to purchase a computer. That computer opened up a whole world to me. It came with a copy of gw-basic and some games written in basic that I could play. Not only could I play them but I could see the source code when I started them up. My first ventures into programming were modifying those first games. Mostly so I could cheat. This is s a perfectly hacker like mentality. The challenge of cheating was more interesting than playing the actual game.

Before long though, I started to wonder if I could do more. I also loved science fiction at the time and I had read a book, whose title I no longer remember, that featured Conways Game of Life as a part of the plot. The rules, of which there were only 4, seemed very well suited to a program so my first truly creative endeavor with basic was programming a version of the Game of Life. It would be many years later before I discovered on the internet how popular this game was with geeky computer people like me.

Fast forward a few years and I'm on my way to college. I wasn't as you might expect going for a computer science degree though. I wanted to be a virus hunter for the CDC. I loved biology and viruses were just really cool. I got a 95% paid scholarship to Missouri Baptist College in St. Louis and away I go. While I'm there I meet a wonderful girl, the love of my life. We end up getting married and I take a fulltime job at the college which had two benefits that were really appealing. It was a full ride tuition to the school. It was fulltime income to support my new family. Things seemed like they were progressing just they way I wanted.

What I didn't know was how hard it is to attend school, work full-time, and keep a marriage going. My progression in school slowed and to top it off I began to get the feeling that working in a biology lab somewhere probably wasn't going to be what I really wanted to do with my life. Cue life crisis. Sudenly I didn't really know what I should do next. I was coasting. The whole time my hobby with computers continued to hang in the background. Eventually the college ends up laying me off. I lose the full ride, I no longer have the scholarships since I had to give those up when I took the full ride. I have a family with 4 kids and no job. It all starts crashing down. What am I supposed to do now?

Of course when you don't have a job it's hard to make the rent, buy food, and otherwise make ends meet. I go through a succession of different jobs never really holding on to any of them until finally it all culminates with my family being homeless. My wife is sticking by me this whole time never once questioning her choice to marry me. Something that to this day still amazes me. We end up having to live with family and I'm working at labor ready to make ends meet. And the whole while my hobby is still hanging in the background. There were days when I thought I was going to have to sell my computer to get food. My wife always insisted I keep it though. I think she knew this was going to be the key to getting out of the whole we were in someday.

So there I am I have a computer, an internet connection. And pretty much nothing else. What's a Guy supposed to do when he's not job searching? Of Course! Hang out on the internet. Two very influential Open Source projects became my life at this point. Blender and Planeshift. I hung out in their irc chatrooms and message boards I downloaded the code and I started hacking in earnest. I learned C and C++ on those projects and more importantly I interacted with people who not only knew how to code but did it professionally and managed a large project. Keep in mind this is still all just my hobby.

Eventually I get a full time job working for a company managing networks in school districts in Southern Illinois and Missouri. We start to pull out of the hole we were in financially and even purchase a house. A dirt cheap house that needed a lot of work but things were looking up. Then disaster strikes again. My boss at the company had never quite figured out what to do with me. He knew I had some experience as a programmer in my hobby and I knew my way around a network. He couldn't quite decide if I was going to be a part of the new programming division or just another network admin. Eventually that indecision led to my being let go. Which could have been a problem except that I had made some contacts with people through the job and was able to immediately get work as a contractor for a local company doing web development in perl.

I start updating my blog with what I learn while working and even publish a CPAN module or two. I'm pretty satisified with where I landed and figure this is how my life will be for the forseeable future. I'm not looking for a job but my blog and personal network in open source have started to get me a little attention. Not much but enough that one day a recruiter from Chicago cold calls me. I've gotten used to being a small time operator and getting refused at all my attempts to get a job in the industry so I'm assuming of course that this is a fluke and I'll never get the job. I do a phone interview, then another phone interview, then they call me in for a face to face. I start to believe that maybe I have a chance here. After the face to face I wait and they call me back saying I've got the job. Not only that but I discover the company is getting bought by Google pending approval from various governments. Long story short I get hired, the company purchase goes through, I survive the transition to Google and I find myself working at one of the top companies in my new career.

So why is this story interesting? I think it highlights a few things that are still true about the Software Development industry today.

  1. The most important contributor to your success is your desire to learn and your persistence.
  2. Programming has more free and self driven avenues for learning than any other subject matter. And the resources have only gotten better since I started.
  3. OpenSource is a merit based approach to gaining credentials and learning in this field.

So, if your passion is Software Development, there is literally nothing standing in your way except you.

One Last thing

Throughout all of the above events I had family and friends praying for me. I'm a devout christrian and I strongly believe I'm at Google now because God wants me there. Their support was a large part of my success as well.


Tags:
<<Prev ...