<!-- Do not edit this file. It is automatically generated by API Documenter. -->

[Home](./index.md) &gt; [@csstools/css-tokenizer](./css-tokenizer.md)

## css-tokenizer package

Tokenize CSS following the [CSS Syntax Level 3 specification](https://drafts.csswg.org/css-syntax/#tokenization)<!-- -->.

## Remarks

The tokenizing and parsing tools provided by CSS Tools are designed to be low level and generic with strong ties to their respective specifications.

Any analysis or mutation of CSS source code should be done with the least powerful tool that can accomplish the task. For many applications it is sufficient to work with tokens. For others you might need to use [@csstools/css-parser-algorithms](https://github.com/csstools/postcss-plugins/tree/main/packages/css-parser-algorithms) or a more specific parser.

## Example

Tokenize a string of CSS into an array of tokens:

```js
import { tokenize } from '@csstools/css-tokenizer';

const myCSS = `@media only screen and (min-width: 768rem) {
	.foo {
		content: 'Some content!' !important;
	}
}
`;

const tokens = tokenize({
	css: myCSS,
});

console.log(tokens);
```

## Classes

<table><thead><tr><th>

Class


</th><th>

Description


</th></tr></thead>
<tbody><tr><td>

[ParseError](./css-tokenizer.parseerror.md)


</td><td>

The CSS Tokenizer is forgiving and will never throw on invalid input. Any errors are reported through the `onParseError` callback.


</td></tr>
<tr><td>

[ParseErrorWithToken](./css-tokenizer.parseerrorwithtoken.md)


</td><td>


</td></tr>
</tbody></table>

## Enumerations

<table><thead><tr><th>

Enumeration


</th><th>

Description


</th></tr></thead>
<tbody><tr><td>

[HashType](./css-tokenizer.hashtype.md)


</td><td>

The type of hash token


</td></tr>
<tr><td>

[NumberType](./css-tokenizer.numbertype.md)


</td><td>

The type of number token Either `integer` or `number`


</td></tr>
<tr><td>

[TokenType](./css-tokenizer.tokentype.md)


</td><td>

All possible CSS token types


</td></tr>
</tbody></table>

## Functions

<table><thead><tr><th>

Function


</th><th>

Description


</th></tr></thead>
<tbody><tr><td>

[cloneTokens(tokens)](./css-tokenizer.clonetokens.md)


</td><td>

Deep clone a list of tokens. Useful for mutations without altering the original list.


</td></tr>
<tr><td>

[isToken(x)](./css-tokenizer.istoken.md)


</td><td>

Assert that a given value has the general structure of a CSS token: 1. is an array. 2. has at least four items. 3. has a known token type. 4. has a string representation. 5. has a start position. 6. has an end position.


</td></tr>
<tr><td>

[isTokenAtKeyword(x)](./css-tokenizer.istokenatkeyword.md)


</td><td>


</td></tr>
<tr><td>

[isTokenBadString(x)](./css-tokenizer.istokenbadstring.md)


</td><td>


</td></tr>
<tr><td>

[isTokenBadURL(x)](./css-tokenizer.istokenbadurl.md)


</td><td>


</td></tr>
<tr><td>

[isTokenCDC(x)](./css-tokenizer.istokencdc.md)


</td><td>


</td></tr>
<tr><td>

[isTokenCDO(x)](./css-tokenizer.istokencdo.md)


</td><td>


</td></tr>
<tr><td>

[isTokenCloseCurly(x)](./css-tokenizer.istokenclosecurly.md)


</td><td>


</td></tr>
<tr><td>

[isTokenCloseParen(x)](./css-tokenizer.istokencloseparen.md)


</td><td>


</td></tr>
<tr><td>

[isTokenCloseSquare(x)](./css-tokenizer.istokenclosesquare.md)


</td><td>


</td></tr>
<tr><td>

[isTokenColon(x)](./css-tokenizer.istokencolon.md)


</td><td>


</td></tr>
<tr><td>

[isTokenComma(x)](./css-tokenizer.istokencomma.md)


</td><td>


</td></tr>
<tr><td>

[isTokenComment(x)](./css-tokenizer.istokencomment.md)


</td><td>


</td></tr>
<tr><td>

[isTokenDelim(x)](./css-tokenizer.istokendelim.md)


</td><td>


</td></tr>
<tr><td>

[isTokenDimension(x)](./css-tokenizer.istokendimension.md)


</td><td>


</td></tr>
<tr><td>

[isTokenEOF(x)](./css-tokenizer.istokeneof.md)


</td><td>


</td></tr>
<tr><td>

[isTokenFunction(x)](./css-tokenizer.istokenfunction.md)


</td><td>


</td></tr>
<tr><td>

[isTokenHash(x)](./css-tokenizer.istokenhash.md)


</td><td>


</td></tr>
<tr><td>

[isTokenIdent(x)](./css-tokenizer.istokenident.md)


</td><td>


</td></tr>
<tr><td>

[isTokenNumber(x)](./css-tokenizer.istokennumber.md)


</td><td>


</td></tr>
<tr><td>

[isTokenNumeric(x)](./css-tokenizer.istokennumeric.md)


</td><td>

Assert that a token is a numeric token


</td></tr>
<tr><td>

[isTokenOpenCurly(x)](./css-tokenizer.istokenopencurly.md)


</td><td>


</td></tr>
<tr><td>

[isTokenOpenParen(x)](./css-tokenizer.istokenopenparen.md)


</td><td>


</td></tr>
<tr><td>

[isTokenOpenSquare(x)](./css-tokenizer.istokenopensquare.md)


</td><td>


</td></tr>
<tr><td>

[isTokenPercentage(x)](./css-tokenizer.istokenpercentage.md)


</td><td>


</td></tr>
<tr><td>

[isTokenSemicolon(x)](./css-tokenizer.istokensemicolon.md)


</td><td>


</td></tr>
<tr><td>

[isTokenString(x)](./css-tokenizer.istokenstring.md)


</td><td>


</td></tr>
<tr><td>

[isTokenUnicodeRange(x)](./css-tokenizer.istokenunicoderange.md)


</td><td>


</td></tr>
<tr><td>

[isTokenURL(x)](./css-tokenizer.istokenurl.md)


</td><td>


</td></tr>
<tr><td>

[isTokenWhitespace(x)](./css-tokenizer.istokenwhitespace.md)


</td><td>


</td></tr>
<tr><td>

[isTokenWhiteSpaceOrComment(x)](./css-tokenizer.istokenwhitespaceorcomment.md)


</td><td>

Assert that a token is a whitespace or comment token


</td></tr>
<tr><td>

[mirrorVariant(token)](./css-tokenizer.mirrorvariant.md)


</td><td>

Get the mirror variant of a given token


</td></tr>
<tr><td>

[mirrorVariantType(type)](./css-tokenizer.mirrorvarianttype.md)


</td><td>

Get the mirror variant type of a given token type


</td></tr>
<tr><td>

[mutateIdent(ident, newValue)](./css-tokenizer.mutateident.md)


</td><td>

Set the ident value and update the string representation. This handles escaping.


</td></tr>
<tr><td>

[mutateUnit(ident, newUnit)](./css-tokenizer.mutateunit.md)


</td><td>

Set the unit and update the string representation. This handles escaping.


</td></tr>
<tr><td>

[stringify(tokens)](./css-tokenizer.stringify.md)


</td><td>

Concatenate the string representation of a list of tokens. This is not a proper serializer that will handle escaping and whitespace. It only produces valid CSS for a token list that is also valid.


</td></tr>
<tr><td>

[tokenize(input, options)](./css-tokenizer.tokenize.md)


</td><td>

Tokenize a CSS string into a list of tokens.


</td></tr>
<tr><td>

[tokenizer(input, options)](./css-tokenizer.tokenizer.md)


</td><td>

Create a tokenizer for a CSS string.


</td></tr>
</tbody></table>

## Interfaces

<table><thead><tr><th>

Interface


</th><th>

Description


</th></tr></thead>
<tbody><tr><td>

[Token](./css-tokenizer.token.md)


</td><td>

The CSS Token interface


</td></tr>
<tr><td>

[TokenAtKeyword](./css-tokenizer.tokenatkeyword.md)


</td><td>


</td></tr>
<tr><td>

[TokenBadString](./css-tokenizer.tokenbadstring.md)


</td><td>


</td></tr>
<tr><td>

[TokenBadURL](./css-tokenizer.tokenbadurl.md)


</td><td>


</td></tr>
<tr><td>

[TokenCDC](./css-tokenizer.tokencdc.md)


</td><td>


</td></tr>
<tr><td>

[TokenCDO](./css-tokenizer.tokencdo.md)


</td><td>


</td></tr>
<tr><td>

[TokenCloseCurly](./css-tokenizer.tokenclosecurly.md)


</td><td>


</td></tr>
<tr><td>

[TokenCloseParen](./css-tokenizer.tokencloseparen.md)


</td><td>


</td></tr>
<tr><td>

[TokenCloseSquare](./css-tokenizer.tokenclosesquare.md)


</td><td>


</td></tr>
<tr><td>

[TokenColon](./css-tokenizer.tokencolon.md)


</td><td>


</td></tr>
<tr><td>

[TokenComma](./css-tokenizer.tokencomma.md)


</td><td>


</td></tr>
<tr><td>

[TokenComment](./css-tokenizer.tokencomment.md)


</td><td>


</td></tr>
<tr><td>

[TokenDelim](./css-tokenizer.tokendelim.md)


</td><td>


</td></tr>
<tr><td>

[TokenDimension](./css-tokenizer.tokendimension.md)


</td><td>


</td></tr>
<tr><td>

[TokenEOF](./css-tokenizer.tokeneof.md)


</td><td>


</td></tr>
<tr><td>

[TokenFunction](./css-tokenizer.tokenfunction.md)


</td><td>


</td></tr>
<tr><td>

[TokenHash](./css-tokenizer.tokenhash.md)


</td><td>


</td></tr>
<tr><td>

[TokenIdent](./css-tokenizer.tokenident.md)


</td><td>


</td></tr>
<tr><td>

[TokenNumber](./css-tokenizer.tokennumber.md)


</td><td>


</td></tr>
<tr><td>

[TokenOpenCurly](./css-tokenizer.tokenopencurly.md)


</td><td>


</td></tr>
<tr><td>

[TokenOpenParen](./css-tokenizer.tokenopenparen.md)


</td><td>


</td></tr>
<tr><td>

[TokenOpenSquare](./css-tokenizer.tokenopensquare.md)


</td><td>


</td></tr>
<tr><td>

[TokenPercentage](./css-tokenizer.tokenpercentage.md)


</td><td>


</td></tr>
<tr><td>

[TokenSemicolon](./css-tokenizer.tokensemicolon.md)


</td><td>


</td></tr>
<tr><td>

[TokenString](./css-tokenizer.tokenstring.md)


</td><td>


</td></tr>
<tr><td>

[TokenUnicodeRange](./css-tokenizer.tokenunicoderange.md)


</td><td>


</td></tr>
<tr><td>

[TokenURL](./css-tokenizer.tokenurl.md)


</td><td>


</td></tr>
<tr><td>

[TokenWhitespace](./css-tokenizer.tokenwhitespace.md)


</td><td>


</td></tr>
</tbody></table>

## Variables

<table><thead><tr><th>

Variable


</th><th>

Description


</th></tr></thead>
<tbody><tr><td>

[ParseErrorMessage](./css-tokenizer.parseerrormessage.md)


</td><td>


</td></tr>
</tbody></table>

## Type Aliases

<table><thead><tr><th>

Type Alias


</th><th>

Description


</th></tr></thead>
<tbody><tr><td>

[CSSToken](./css-tokenizer.csstoken.md)


</td><td>

The union of all possible CSS tokens


</td></tr>
<tr><td>

[NumericToken](./css-tokenizer.numerictoken.md)


</td><td>

The union of all possible CSS tokens that represent a numeric value


</td></tr>
</tbody></table>

