Batch Files: Tokens and Delimiters (FOR Loops)

Quite often when writing a batch file, you will come across a FOR loop. It might look something like this:

FOR /f "tokens=* delims= " %%a IN (MyFile) DO ECHO %%a

I am constantly hearing people asking "What do tokens and delims mean?". Well, here you are.

Tokens and delimiters in action

Tokens basically tell the batch file where to look to set the variable (%a). Delimiters are what separate each token. It's a little difficult to explain, so here's an example.

A file, called MyFile, contains this:

Hello World! How are you doing today?

Now, let's pretend that we want is the word "World!". How will we accomplish that? Using a space as a delimiter (separator) we can grab it with the second token.

FOR /f "tokens=2 delims= " %%a IN (MyFile.txt) DO ECHO %%a

Grabbing a range

Now let's say that we want the "How are you doing today?" portion. Still using space as a delimiter, we can grab it with tokens 3-7.

FOR /f 'tokens=3-7 delims= " %%a IN (MyFile.txt) DO ECHO %%a %%b %%c %%d %%e

Notice the variables. The batch file sets the first variable defined (%a) then will move on to the next one, in this case %b.

If I wanted to, I could set all the variables, then just echo %a %c %e, resulting in "How you today?"

Variations on a range

Now, just because we've decided to become very aggressive, we're going to attempt to get the words "Hello", "How", and "doing". Using the same logic as before, we can use the first, third, and sixth token.

FOR /f 'tokens=1, 3, 6 delims= " %%a IN (MyFile.txt) DO ECHO %%a %%b %%c

Again, notice the variables (See above).

Taking everything

We can set the entire line if we want to, using an asterisk (*).

FOR /f "tokens=* delims= " %%a IN (MyFile.txt) DO ECHO %%a

The asterisk, sets every token to the variable.

Another example

A delimiter can be anything, for example, if I have a text file containing:

Hello World! /How are you today?/

I could grab just the part enclosed within the backslash using this:

FOR /f "tokens=2 delims=/" %%a IN (MyFile) DO ECHO %%a

Notice haw I used the second token. This is because the first would be what ever is in from of the first slash ("Hello World! ") is set as the first.

Try it out yourself!

Do not be discouraged if you do not understand it all at once, it can be confusing. Give it a few tries, and write a few samples to help yourself learn.

Good luck.