Scripts file formats

Submission scripts might fail for no apparent reason before the code itself starts to run. Before diving into some heavy debugging, you may want to check how your submission script has been created.

Every operating system uses a special character (or sequence of characters) to signify the end of a line of text. They cannot use standard, common characters to represent the line end because those could appear in normal text. Because of this, they use special, non-printing characters. Each operating system uses different ones:

  • Linux and Mac OS X inherited the Unix style of using LF (line feed, an ASCII control character) at the end of each line.
  • Older Macintosh systems use CR (carriage return, an ASCII control character).
  • Windows uses a pair of characters -- both a CR and an LF.


File format check

To check what a particular text file uses to indicate a new line, try hexdump, which lets you inspect the contents of a file at the byte level.

A simple submission script has been created respectively on a Linux (using vi) and Windows (using edit) environment:

#!/bin/csh
#$ -M afs_id@nd.edu
#$ -m ae
echo TEST
$ hexdump -cb  test.linux 
0000000   #   !   /   b   i   n   /   c   s   h  \n   #   $       -   M
0000000 043 041 057 142 151 156 057 143 163 150 012 043 044 040 055 115
0000010       a   f   s   _   i   d   @   n   d   .   e   d   u  \n   #
0000010 040 141 146 163 137 151 144 100 156 144 056 145 144 165 012 043
0000020   $       -   m       a   e  \n   e   c   h   o       T   E   S
0000020 044 040 055 155 040 141 145 012 145 143 150 157 040 124 105 123
0000030   T  \n  \n                                                    
0000030 124 012 012                                                    
0000033
$ hexdump -cb  test.win
0000000   #   !   /   b   i   n   /   c   s   h  \r  \n   #   $       -
0000000 043 041 057 142 151 156 057 143 163 150 015 012 043 044 040 055
0000010   M       a   f   s   _   i   d   @   n   d   .   e   d   u  \r
0000010 115 040 141 146 163 137 151 144 100 156 144 056 145 144 165 015
0000020  \n   #   $       -   m       a   e  \r  \n   e   c   h   o    
0000020 012 043 044 040 055 155 040 141 145 015 012 145 143 150 157 040
0000030   T   E   S   T  \r  \n  \r  \n                                
0000030 124 105 123 124 015 012 015 012                                
0000038

Notice that the Linux file has a \n character at the end of each line, while the Windows version uses \r and \n.

An older Macintosh file would have used a single \r character instead.


File format conversion

To convert files from Windows to Linux, you can use the dos2unix command.

 dos2unix test.win 

This will delete all extra \r and will rewrite on the same file.

 dos2unix < test.win > test.win2linux 

This will delete all extra \r and will create a new file You can get more information with dos2unix -h or man dos2unix.

To convert files from Mac to Linux, you can use the mac2unix command.


If those commands are not installed on the system, you can use the Unix command tr

  • On Windows, the CR characters will be removed with the option -d
tr -d "\015" < test.win > test.win2linux 
  • On Mac, each CR (octal 015) will be changed into LF (octal 012)
 tr "\015" "\012" < test.mac > test.mac2linux