Marzhill Musings

... Next>>

On Inertia

Published On: 2014-05-21

This is my last week at Google.

Friday will be my last day. Which feels very strange I must admit. I'm a high inertia person. I tend to stay the course once I've picked one which is probably why I've been at Google for 6 years now. But now that is all changing so this is as a good a time as any to look back.

Google is without a doubt the best company I've worked for to date. It's a deeply engineering centric company which has many benefits. They invest heavily in engineering tooling and engineering culture. Everything from build tools to source control has had significant resources poured into them at Google and it shows.

But by far the best feature of Google as a company has nothing to do with the engineering culture. Googles best feature is it's conscience. Without a doubt Google is the most moral company I have ever worked for. It's the kind of thing that is hard to verify from the outside but from the inside it's obvious that Google has a significant conscience in it's employees and it listens to that conscience and changes course as a result.


Tags:

First Look at Polymer Elements

Published On: 2013-09-17

Introduction

For work I've been getting up to speed on the W3C's set of webcomponents standards. Which means I've been looking at Polymer. Polymer is both an experimental javascript framework and a shim library that simulates the portions of the W3C standard which are not yet implemented in browsers. Specifically I've been looking at Custom Elements and Templates since they are the more concrete portions of the standard right now.

At some point when you are exploring a new technology the docs and tutorials stop being useful to you and to really get a feel you have to build something in it. I decided to port parts of a javascript application, DynamicBible, that I had already written as a learning exercise. DynamicBible currently uses requirejs for it's javascript code as a way to manage dependencies and keep scoping sane. This made it perfect for my purposes since it allowed me to explore two types of polymer elements.

  • UI elements.
  • Utility elements that don't have a visible presence on the page.

For my purposes I ported the DynamicBible search box and a requirejs importer element. In this article I'll cover the search box element. The requirejs element will be covered a later article.

Creating your own Polymer Element

    1 <polymer-element name="my-element">
    2 <template>
    3 ...
    4 </template>
    5 <script>
    6 ...
    7 </script>
    8 </polymer-element>

The <polymer-element> is the declarative way to define your polymer element in html itself. The bare minimum to define a polymer element would be

    1 <polymer-element name="search-box">
    2 <script>
    3 Polymer('search-box');
    4 </script>
    5 </polymer-element>

This is as about as useful as the span element though and html already has a span element. We need a little more than this to be worth it. Our element needs some attributes and behavior. Polymer lets us describe the expected attributes using an attribute called, what else, attributes.

    1 <polymer-element name="search-box" attributes="id query">

As for the behavior to attach to this element, that brings us the the Polymer construction function.

    1 <polymer-element name="search-box" attributes="id query">
    2 <script>
    3 // Polymers construction function.
    4 Polymer('search-box', {
    5   // default values for the attributes
    6   id: "",
    7   query: "",
    8   // a search method as part of the search-boxes api.
    9   search: function() {
   10     submitSearch(this.value);
   11   }
   12 });
   13 </script>
   14 </polymer-element>

You can use the element the same way you would any other html element.

    1 <search-box id="searchBox"></search-box>

Now our element has a method on it that will submit a search using the value of our search-box query attribute. We could trigger this behavior right now with javascript.

    1 document.querySelector('#searchBox').query = "what is a polymer element?";
    2 document.querySelector('#searchBox').search();

It's kind of silly that we have to do that manually with javascript though. What we really want is for this element to detect changes in our query and perform the search for us.

    1 <polymer-element name="search-box" attributes="id query">
    2 <script>
    3 Polymer('search-box', {
    4   id: "",
    5   query: "",
    6   search: function() {
    7     submitSearch(this.value);
    8   },
    9   // event handlers
   10   queryChanged: function() { // called on query attribute change
   11     this.search();
   12   },
   13   created: function() { // called on element creation
   14   }
   15 });
   16 </script>
   17 </polymer-element>

Now when the element's query attribute changes a search is triggered.

Up to now our element hasn't been very visible. We need to give it an image boost. We can do this two different ways.

  1. Using a template
  2. Using element inheritance

We'll go with the template element for now. The element inheritance will come in handy later.

    1 <polymer-element name="search-box" attributes="id query">
    2 <template>
    3   <input type="text" value="{{ query }}">
    4 </template>
    5 <script>
    6 Polymer('search-box', {
    7   id: "",
    8   query: "",
    9   search: function() {
   10     submitSearch(this.value);
   11   },
   12   // event handlers
   13   queryChanged: function() { // called on query attribute change
   14     this.search();
   15   },
   16   created: function() { // called on element creation
   17   }
   18 });
   19 </script>
   20 </polymer-element>

There are a number of things going on in this template element. First we define some html that will be used whenever we render the element on a page. There is some complicated merging logic involving the shadow dom but we'll ignore that for now. Second our value attribute on the the input element has funny looking content. The {{ query }} tells polymer that the contents of the text box and the query a should be kept in sync. A change to one of them is reflected in the other. Furthermore a change to the input box will result in the queryChanged event firing and causing a search to get submitted. There are several more built in events and Polymer includes a way to create and listen to your own events. as well.

I'll cover a utility element that shims requirejs modules to make them useable in your polymer elements in a later article.

Out elements template element isn't terribly complicated and it turns out in our case is completely unnecessary. We can use extends to tell Polymer our element should inherit from the input element.

Our last tweak to the search-box element looks like this.

    1 // input already defines the attributes we need
    2 <polymer-element name="search-box" extends="input">
    3 <script>
    4 Polymer('search-box', {
    5   id: "",
    6   // query becomes the value attribute of an input element.
    7   value: "",
    8   search: function() {
    9     submitSearch(this.query);
   10   },
   11   // event handlers
   12   valueChanged: function() { // called on value attribute change
   13     this.search();
   14   },
   15   created: function() {
   16     // Initialize the type to text on element creation
   17     this.type = "text"
   18   }
   19 });
   20 </script>
   21 </polymer-element>

Tags:

Using bufio.Scanner to build a Tokenizer

Published On: 2013-05-09 00:00:00

Go's standard lib comes with a number of useful tools. Each one of them written in an idiomatic fashion illustrating how to design good api's in Go. One of the most informative and useful are the io packages. both the "io" and "bufio" packages are examples of beautiful and idiomatic Go. I've recently decided to look at using the bufio.Scanner to build a tokenizer for a very simple lisp. The tokenizer's job is to emit a stream of tokens for each element of the lisp's syntax. The syntax is very basic with only the following tokens.

  • Whitespace a series of newline, linefeed, tab or space characters
  • StartExpr: a single '('
  • EndExpr: a single ')'
  • String: anything between double or single quotes with \ escaping the next character.
  • Number: any series of numbers surrounded by whitespace
  • Word: any series of characters surrounded by whitespace and starting with a-z or A-Z

The tokenizer we are going to build should emit a stream of these tokens for us while also keeping track of where in the file these tokens occur. Go's bufio package has a useful general purpose Scanner type that splits an io.Reader into a series of byte slices for you. Our first task is to figure out how to use the Scanner to split up any reader into a stream of slices representing each of these tokens. The Scanner type has 5 methods. We'll be focusing on just 4 of them.

  • The Scan method scans the stream looking for the next byte slice to emit. It returns false if the Scan hit an error or EOF.
  • The Bytes method returns the current byte slice after Scan has run.
  • The Err method returns the last non io.EOF error encountered by Scan.
  • The Split method sets the SplitFunc to use for the next Scan.

Together these give us all the tools necessary to tokenize any io.Reader. The first step is to create a Scanner to use. 1 2 package main 3 4 import ( 5 "strings" 6 "bufio" 7 "fmt" 8 ) 9 10 func main() { 11 s := bufio.NewScanner(strings.NewReader("(foo 1 'bar')\n(baz 2 'quux')")) 12 for s.Scan() { 13 tok := s.Bytes() 14 fmt.Println("Found token ", tok) 15 } 16 }

You can play with this code here http://play.golang.org/p/fhOV_4fVOI. Out of the box the scanner splits the io.Reader into lines which is obviously not what we want. In order to let us specify our own rules for splitting bufio.Scanner uses composition. We can pass in our splitting logic in the form of a SplitFunc. A SplitFunc takes a candidate slice of data for splitting and returns how far to advance the position in the reader stream, the byte slice to return for this split, and an error if any is needed. Below you'll find the basic skeleton for our splitting logic. 1 2 package main 3 4 import ( 5 "strings" 6 "bufio" 7 "fmt" 8 ) 9 10 func consumeWord(data []byte) (int, []byte, error) { 11 // TODO 12 return 0, nil, nil 13 } 14 15 func consumeWhitespace(data []byte) (int, []byte, error) { 16 // TODO 17 return 0, nil, nil 18 } 19 20 func consumeNum(data []byte) (int, []byte, error) { 21 // TODO 22 return 0, nil, nil 23 } 24 25 func consumeString(data []byte) (int, []byte, error) { 26 // TODO 27 return 0, nil, nil 28 } 29 30 func main() { 31 s := bufio.NewScanner(strings.NewReader("(foo 1 'bar')\n(baz 2 'quux')")) 32 split := func(data []byte, atEOF bool) (advance int, token []byte, err error) { 33 // Because our grammar is simple we can switch off the first 34 // character in the reader. 35 switch data[0] { 36 case '(', ')': 37 advance, token, err = 1, data[:1], nil 38 case '"', '\'': 39 advance, token, err = consumeString(data) 40 case '0', '1', '2', '3', '4', '5', '6', '7', '8', '9': 41 advance, token, err = consumeNum(data) 42 case ' ', '\n', '\r', '\t': 43 advance, token, err = consumeWhitespace(data) 44 default: 45 advance, token, err = consumeWord(data) 46 } 47 return 48 } 49 s.Split(split) 50 for s.Scan() { 51 tok := s.Bytes() 52 fmt.Println("Found token ", tok) 53 } 54 }

The split func inspects the start of our candidate slice and determines how it should handle it. The parens case is easy so we implented it first by telling the Scanner to advance by one byte and emit a byte slice of size one containing the paren and nil for the error. We've defined names for consume functions for each of our other tokens but left them unimplemented so far. We'll start with consumeWord to give an example of how they should work. 1 2 func consumeWord(data []byte) (int, []byte, error) { 3 var accum []byte 4 for i, b := range data { 5 if b == ' ' || b == '\n' || b == '\t' || b == '\r' { 6 return i, accum, nil 7 } else { 8 accum = append(accum, b) 9 } 10 } 11 return 0, nil, nil 12 }

Tokenizing a Word token is easy. A Word starts with any character that isn't a quote, paren or Number and they continue until you hit whitespace. Our switch statement handles the quote, paren, and number cases specially and anything left over is a Word. Then we call consumeWord with the candidate slice. consumeWords job is to scan the candidate for whitespace accumulating as we go. We could if we want bypass the accumulator but this was easier to understand for now. If no whitespace is seen then we don't have a full word yet so return 0, nil, nil. If we do encounter any whitespace then return the amount to advance, not including the whitespace, as well the word we've accumulated and nil. The scanner will keep trying our split function on increasing slice sizes until it either gets an error, non-zero advance or hits the end of the stream. We code similar consume functions for the Whitespace, String, and Number tokens. 1 2 func consumeWhitespace(data []byte) (int, []byte, error) { 3 var accum []byte 4 for i, b := range data { 5 if b == ' ' || b == '\n' || b == '\t' || b == '\r' { 6 accum = append(accum, b) 7 } else { 8 return i, accum, nil 9 } 10 } 11 return 0, nil, nil 12 } 13 14 func consumeNum(data []byte) (int, []byte, error) { 15 var accum []byte 16 for i, b := range data { 17 if '0' <= b && b <= '9' { 18 accum = append(accum, b) 19 } else { 20 return i, accum, nil 21 } 22 } 23 return len(accum), accum, nil 24 } 25 26 func consumeString(data []byte) (int, []byte, error) { 27 delim := data[0] 28 skip := false 29 accum := []byte{data[0]} 30 for i, b := range data[1:] { 31 if b == delim && !skip { 32 return i + 2, accum, nil 33 } 34 skip = false 35 if b == '\\' { 36 skip = true 37 continue 38 } 39 accum = append(accum, b) 40 } 41 return 0, nil, nil 42 }

If you play with this code here http://play.golang.org/p/dksq4YjJtb. then you'll see it emit one of each of our tokens.

We still aren't done though. This Scanner while useful doesn't tell us our Token Locations. We need it to tell us the Line and column that each token starts on. To do this we'll use composition again. Only this time we'll compose two structs using Go's embedded fields. 1 2 type lineTrackingReader struct { 3 // Embedded Scanner so our LineTrackingReader can be used just like 4 // a Scanner. 5 *bufio.Scanner 6 lastL, lastCol int // Position Tracking fields. 7 l, col int 8 } 9 10 func newTrackingReader(r io.Reader) *lineTrackingReader { 11 s := bufio.NewScanner(r) 12 rdr := &lineTrackingReader{ 13 Scanner: s, 14 } 15 split := func(data []byte, atEOF bool) (advance int, token []byte, err error) { 16 // Initiliaze our position tracking fields using a split like a closure. 17 if rdr.l == 0 { 18 rdr.l = 1 19 rdr.col = 1 20 rdr.lastL = 1 21 rdr.lastCol = 1 22 } 23 switch data[0] { 24 case '(', ')': 25 advance, token, err = 1, data[:1], nil 26 case '"', '\'': // TODO(jwall): Rune data? 27 advance, token, err = consumeString(data) 28 case '0', '1', '2', '3', '4', '5', '6', '7', '8', '9': 29 advance, token, err = consumeNum(data) 30 case ' ', '\n', '\r', '\t': 31 advance, token, err = consumeWhitespace(data) 32 default: 33 advance, token, err = consumeWord(data) 34 } 35 // If we are advancing then we should see if there are any line 36 // endings here 37 if advance > 0 { 38 rdr.lastCol = rdr.col 39 rdr.lastL = rdr.l 40 for _, b := range data[:advance] { 41 if b == '\n' || atEOF { // Treat eof as a newline 42 rdr.l++ 43 rdr.col = 1 44 } 45 rdr.col++ 46 } 47 } 48 49 return 50 } 51 s.Split(split) 52 return rdr 53 } 54 55 func main() { 56 s := newTrackingReader(strings.NewReader("(foo 1 'bar')\n(baz 2 'quux')")) 57 58 for s.Scan() { 59 tok := s.Bytes() 60 fmt.Printf("Found token %q at line #%d col #%d\n", 61 string(tok), s.Line(), s.Column) 62 } 63 } 64 65 func (l *lineTrackingReader) Line() int { 66 return l.lastL 67 } 68 69 func (l *lineTrackingReader) Column() int { 70 return l.lastCol 71 }

You can play with the finished code here http://play.golang.org/p/_omf7AGefg.


Tags:
... Next>>