CSV & Text files

The two workhorse functions for reading text files (a.k.a. flat files) are read_csv() and read_table(). They both use the same parsing code to intelligently convert tabular data into a DataFrame object. See thecookbook for some advanced strategies

They can take a number of arguments:

• filepath_or_buffer: Either a string path to a file, or any object with a read method (such as an open file or StringIO).
• sep or delimiter: A delimiter / separator to split fields on. read_csv is capable of inferring the delimiter automatically in some cases by “sniffing.” The separator may be specified as a regular expression; for instance you may use ‘|\s*’ to indicate a pipe plus arbitrary whitespace.
• delim_whitespace: Parse whitespace-delimited (spaces or tabs) file (much faster than using a regular expression)
• compression: decompress 'gzip' and 'bz2' formats on the fly.
• dialect: string or csv.Dialect instance to expose more ways to specify the file format
• dtype: A data type name or a dict of column name to data type. If not specified, data types will be inferred.
• header: row number to use as the column names, and the start of the data. Defaults to 0 if no names passed, otherwise None. Explicitly pass header=0 to be able to replace existing names.
• skiprows: A collection of numbers for rows in the file to skip. Can also be an integer to skip the first n rows
• index_col: column number, column name, or list of column numbers/names, to use as the index (row labels) of the resulting DataFrame. By default, it will number the rows without using any column, unless there is one more data column than there are headers, in which case the first column is taken as the index.
• names: List of column names to use as column names. To replace header existing in file, explicitly pass header=0.
• na_values: optional list of strings to recognize as NaN (missing values), either in addition to or in lieu of the default set.
• true_values: list of strings to recognize as True
• false_values: list of strings to recognize as False
• keep_default_na: whether to include the default set of missing values in addition to the ones specified in na_values
• parse_dates: if True then index will be parsed as dates (False by default). You can specify more complicated options to parse a subset of columns or a combination of columns into a single date column (list of ints or names, list of lists, or dict) [1, 2, 3] -> try parsing columns 1, 2, 3 each as a separate date column [[1, 3]] -> combine columns 1 and 3 and parse as a single date column {‘foo’ : [1, 3]} -> parse columns 1, 3 as date and call result ‘foo’
• keep_date_col: if True, then date component columns passed into parse_dates will be retained in the output (False by default).
• date_parser: function to use to parse strings into datetime objects. If parse_dates is True, it defaults to the very robust dateutil.parser. Specifying this implicitly sets parse_dates as True. You can also use functions from community supported date converters from date_converters.py
• dayfirst: if True then uses the DD/MM international/European date format (This is False by default)
• thousands: sepcifies the thousands separator. If not None, then parser will try to look for it in the output and parse relevant data to integers. Because it has to essentially scan through the data again, this causes a significant performance hit so only use if necessary.
• comment: denotes the start of a comment and ignores the rest of the line. Currently line commenting is not supported.
• nrows: Number of rows to read out of the file. Useful to only read a small portion of a large file
• iterator: If True, return a TextParser to enable reading a file into memory piece by piece
• chunksize: An number of rows to be used to “chunk” a file into pieces. Will cause an TextParser object to be returned. More on this below in the section on iterating and chunking
• skip_footer: number of lines to skip at bottom of file (default 0)
• converters: a dictionary of functions for converting values in certain columns, where keys are either integers or column labels
• encoding: a string representing the encoding to use for decoding unicode data, e.g. 'utf-8` or 'latin-1'.
• verbose: show number of NA values inserted in non-numeric columns
• squeeze: if True then output with only one column is turned into Series
• error_bad_lines: if False then any lines causing an error will be skipped bad lines

Consider a typical CSV file containing, in this case, some time series data:

In [1021]: print open('foo.csv').read()
date,A,B,C
20090101,a,1,2
20090102,b,3,4
20090103,c,4,5

The default for read_csv is to create a DataFrame with simple numbered rows:

In [1022]: pd.read_csv('foo.csv')
Out[1022]: 
       date  A  B  C
0  20090101  a  1  2
1  20090102  b  3  4
2  20090103  c  4  5

In the case of indexed data, you can pass the column number or column name you wish to use as the index:

In [1023]: pd.read_csv('foo.csv', index_col=0)
Out[1023]: 
          A  B  C
date             
20090101  a  1  2
20090102  b  3  4
20090103  c  4  5

In [1024]: pd.read_csv('foo.csv', index_col='date')
Out[1024]: 
          A  B  C
date             
20090101  a  1  2
20090102  b  3  4
20090103  c  4  5

You can also use a list of columns to create a hierarchical index:

In [1025]: pd.read_csv('foo.csv', index_col=[0, 'A'])
Out[1025]: 
            B  C
date     A      
20090101 a  1  2
20090102 b  3  4
20090103 c  4  5

The dialect keyword gives greater flexibility in specifying the file format. By default it uses the Excel dialect but you can specify either the dialect name or a csv.Dialect instance.

Suppose you had data with unenclosed quotes:

In [1026]: print data
label1,label2,label3
index1,"a,c,e
index2,b,d,f

By default, read_csv uses the Excel dialect and treats the double quote as the quote character, which causes it to fail when it finds a newline before it finds the closing double quote.

We can get around this using dialect

In [1027]: dia = csv.excel()

In [1028]: dia.quoting = csv.QUOTE_NONE

In [1029]: pd.read_csv(StringIO(data), dialect=dia)
Out[1029]: 
       label1 label2 label3
index1     "a      c      e
index2      b      d      f

All of the dialect options can be specified separately by keyword arguments:

In [1030]: data = 'a,b,c~1,2,3~4,5,6'

In [1031]: pd.read_csv(StringIO(data), lineterminator='~')
Out[1031]: 
   a  b  c
0  1  2  3
1  4  5  6

Another common dialect option is skipinitialspace, to skip any whitespace after a delimiter:

In [1032]: data = 'a, b, c\n1, 2, 3\n4, 5, 6'

In [1033]: print data
a, b, c
1, 2, 3
4, 5, 6

In [1034]: pd.read_csv(StringIO(data), skipinitialspace=True)
Out[1034]: 
   a  b  c
0  1  2  3
1  4  5  6

The parsers make every attempt to “do the right thing” and not be very fragile. Type inference is a pretty big deal. So if a column can be coerced to integer dtype without altering the contents, it will do so. Any non-numeric columns will come through as object dtype as with the rest of pandas objects.

Specifying column data types

Starting with v0.10, you can indicate the data type for the whole DataFrame or individual columns:

In [1035]: data = 'a,b,c\n1,2,3\n4,5,6\n7,8,9'

In [1036]: print data
a,b,c
1,2,3
4,5,6
7,8,9

In [1037]: df = pd.read_csv(StringIO(data), dtype=object)

In [1038]: df
Out[1038]: 
   a  b  c
0  1  2  3
1  4  5  6
2  7  8  9

In [1039]: df['a'][0]
Out[1039]: '1'

In [1040]: df = pd.read_csv(StringIO(data), dtype={'b': object, 'c': np.float64})

In [1041]: df.dtypes
Out[1041]: 
a      int64
b     object
c    float64
dtype: object

Handling column names

A file may or may not have a header row. pandas assumes the first row should be used as the column names:

In [1042]: from StringIO import StringIO

In [1043]: data = 'a,b,c\n1,2,3\n4,5,6\n7,8,9'

In [1044]: print data
a,b,c
1,2,3
4,5,6
7,8,9

In [1045]: pd.read_csv(StringIO(data))
Out[1045]: 
   a  b  c
0  1  2  3
1  4  5  6
2  7  8  9

By specifying the names argument in conjunction with header you can indicate other names to use and whether or not to throw away the header row (if any):

In [1046]: print data
a,b,c
1,2,3
4,5,6
7,8,9

In [1047]: pd.read_csv(StringIO(data), names=['foo', 'bar', 'baz'], header=0)
Out[1047]: 
   foo  bar  baz
0    1    2    3
1    4    5    6
2    7    8    9

In [1048]: pd.read_csv(StringIO(data), names=['foo', 'bar', 'baz'], header=None)
Out[1048]: 
  foo bar baz
0   a   b   c
1   1   2   3
2   4   5   6
3   7   8   9

If the header is in a row other than the first, pass the row number to header. This will skip the preceding rows:

In [1049]: data = 'skip this skip it\na,b,c\n1,2,3\n4,5,6\n7,8,9'

In [1050]: pd.read_csv(StringIO(data), header=1)
Out[1050]: 
   a  b  c
0  1  2  3
1  4  5  6
2  7  8  9

Filtering columns (usecols)

The usecols argument allows you to select any subset of the columns in a file, either using the column names or position numbers:

In [1051]: data = 'a,b,c,d\n1,2,3,foo\n4,5,6,bar\n7,8,9,baz'

In [1052]: pd.read_csv(StringIO(data))
Out[1052]: 
   a  b  c    d
0  1  2  3  foo
1  4  5  6  bar
2  7  8  9  baz

In [1053]: pd.read_csv(StringIO(data), usecols=['b', 'd'])
Out[1053]: 
   b    d
0  2  foo
1  5  bar
2  8  baz

In [1054]: pd.read_csv(StringIO(data), usecols=[0, 2, 3])
Out[1054]: 
   a  c    d
0  1  3  foo
1  4  6  bar
2  7  9  baz

Dealing with Unicode Data

The encoding argument should be used for encoded unicode data, which will result in byte strings being decoded to unicode in the result:

In [1055]: data = 'word,length\nTr\xe4umen,7\nGr\xfc\xdfe,5'

In [1056]: df = pd.read_csv(StringIO(data), encoding='latin-1')

In [1057]: df
Out[1057]: 
      word  length
0  Träumen       7
1    Grüße       5

In [1058]: df['word'][1]
Out[1058]: u'Gr\xfc\xdfe'

Some formats which encode all characters as multiple bytes, like UTF-16, won’t parse correctly at all without specifying the encoding.

Index columns and trailing delimiters

If a file has one more column of data than the number of column names, the first column will be used as the DataFrame’s row names:

In [1059]: data = 'a,b,c\n4,apple,bat,5.7\n8,orange,cow,10'

In [1060]: pd.read_csv(StringIO(data))
Out[1060]: 
        a    b     c
4   apple  bat   5.7
8  orange  cow  10.0

In [1061]: data = 'index,a,b,c\n4,apple,bat,5.7\n8,orange,cow,10'

In [1062]: pd.read_csv(StringIO(data), index_col=0)
Out[1062]: 
            a    b     c
index                   
4       apple  bat   5.7
8      orange  cow  10.0

Ordinarily, you can achieve this behavior using the index_col option.

There are some exception cases when a file has been prepared with delimiters at the end of each data line, confusing the parser. To explicitly disable the index column inference and discard the last column, pass index_col=False:

In [1063]: data = 'a,b,c\n4,apple,bat,\n8,orange,cow,'

In [1064]: print data
a,b,c
4,apple,bat,
8,orange,cow,

In [1065]: pd.read_csv(StringIO(data))
Out[1065]: 
        a    b   c
4   apple  bat NaN
8  orange  cow NaN

In [1066]: pd.read_csv(StringIO(data), index_col=False)
Out[1066]: 
   a       b    c
0  4   apple  bat
1  8  orange  cow

Specifying Date Columns

To better facilitate working with datetime data, read_csv() and read_table() uses the keyword arguments parse_dates and date_parser to allow users to specify a variety of columns and date/time formats to turn the input text data into datetime objects.

The simplest case is to just pass in parse_dates=True:

# Use a column as an index, and parse it as dates.
In [1067]: df = pd.read_csv('foo.csv', index_col=0, parse_dates=True)

In [1068]: df
Out[1068]: 
            A  B  C
date               
2009-01-01  a  1  2
2009-01-02  b  3  4
2009-01-03  c  4  5

# These are python datetime objects
In [1069]: df.index
Out[1069]: 
<class 'pandas.tseries.index.DatetimeIndex'>
[2009-01-01 00:00:00, ..., 2009-01-03 00:00:00]
Length: 3, Freq: None, Timezone: None

It is often the case that we may want to store date and time data separately, or store various date fields separately. the parse_dates keyword can be used to specify a combination of columns to parse the dates and/or times from.

You can specify a list of column lists to parse_dates, the resulting date columns will be prepended to the output (so as to not affect the existing column order) and the new column names will be the concatenation of the component column names:

In [1070]: print open('tmp.csv').read()
KORD,19990127, 19:00:00, 18:56:00, 0.8100
KORD,19990127, 20:00:00, 19:56:00, 0.0100
KORD,19990127, 21:00:00, 20:56:00, -0.5900
KORD,19990127, 21:00:00, 21:18:00, -0.9900
KORD,19990127, 22:00:00, 21:56:00, -0.5900
KORD,19990127, 23:00:00, 22:56:00, -0.5900

In [1071]: df = pd.read_csv('tmp.csv', header=None, parse_dates=[[1, 2], [1, 3]])

In [1072]: df
Out[1072]: 
                  1_2                 1_3     0     4
0 1999-01-27 19:00:00 1999-01-27 18:56:00  KORD  0.81
1 1999-01-27 20:00:00 1999-01-27 19:56:00  KORD  0.01
2 1999-01-27 21:00:00 1999-01-27 20:56:00  KORD -0.59
3 1999-01-27 21:00:00 1999-01-27 21:18:00  KORD -0.99
4 1999-01-27 22:00:00 1999-01-27 21:56:00  KORD -0.59
5 1999-01-27 23:00:00 1999-01-27 22:56:00  KORD -0.59

By default the parser removes the component date columns, but you can choose to retain them via the keep_date_col keyword:

In [1073]: df = pd.read_csv('tmp.csv', header=None, parse_dates=[[1, 2], [1, 3]],
   ......:                  keep_date_col=True)
   ......:

In [1074]: df
Out[1074]: 
                  1_2                 1_3     0         1          2  \
0 1999-01-27 19:00:00 1999-01-27 18:56:00  KORD  19990127   19:00:00   
1 1999-01-27 20:00:00 1999-01-27 19:56:00  KORD  19990127   20:00:00   
2 1999-01-27 21:00:00 1999-01-27 20:56:00  KORD  19990127   21:00:00   
3 1999-01-27 21:00:00 1999-01-27 21:18:00  KORD  19990127   21:00:00   
4 1999-01-27 22:00:00 1999-01-27 21:56:00  KORD  19990127   22:00:00   
5 1999-01-27 23:00:00 1999-01-27 22:56:00  KORD  19990127   23:00:00   
           3     4  
0   18:56:00  0.81  
1   19:56:00  0.01  
2   20:56:00 -0.59  
3   21:18:00 -0.99  
4   21:56:00 -0.59  
5   22:56:00 -0.59  

Note that if you wish to combine multiple columns into a single date column, a nested list must be used. In other words, parse_dates=[1, 2] indicates that the second and third columns should each be parsed as separate date columns while parse_dates=[[1, 2]] means the two columns should be parsed into a single column.

You can also use a dict to specify custom name columns:

In [1075]: date_spec = {'nominal': [