summaryrefslogtreecommitdiffstats
path: root/dimension/tokenize.c
Commit message (Expand)AuthorAgeFilesLines
* Tokenize identifiers.Tavian Barnes2009-10-281-6/+41
* Fix line calculation in comment tokenizer.Tavian Barnes2009-10-281-0/+1
* Split some tokenizing work out into helper functions.Tavian Barnes2009-10-281-56/+82
* Ignore comments in tokenizer.Tavian Barnes2009-10-271-4/+35
* Add line and column numbers to tokens.Tavian Barnes2009-10-271-1/+7
* Tokenize numeric values.Tavian Barnes2009-10-271-2/+59
* Support more symbols in tokenizer.Tavian Barnes2009-10-271-18/+46
* Make tokenizer a bit more robust.Tavian Barnes2009-10-271-3/+19
* New tokenizer test.Tavian Barnes2009-10-271-4/+4
* New mmap-based tokenizer.Tavian Barnes2009-10-271-13/+33
* Begin tokenizer.Tavian Barnes2009-10-261-0/+114