Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | Add line and column numbers to tokens. | Tavian Barnes | 2009-10-27 | 1 | -1/+7 |
| | |||||
* | Tokenize numeric values. | Tavian Barnes | 2009-10-27 | 1 | -2/+59 |
| | |||||
* | Support more symbols in tokenizer. | Tavian Barnes | 2009-10-27 | 1 | -18/+46 |
| | |||||
* | Make tokenizer a bit more robust. | Tavian Barnes | 2009-10-27 | 1 | -3/+19 |
| | |||||
* | New tokenizer test. | Tavian Barnes | 2009-10-27 | 1 | -4/+4 |
| | |||||
* | New mmap-based tokenizer. | Tavian Barnes | 2009-10-27 | 1 | -13/+33 |
| | |||||
* | Begin tokenizer. | Tavian Barnes | 2009-10-26 | 1 | -0/+114 |