pyspark.pandas.to_datetime#

pyspark.pandas.to_datetime(arg, errors='raise', format=None, unit=None, infer_datetime_format=False, origin='unix')[source]#

Convert argument to datetime.

Parameters
arginteger, float, string, datetime, list, tuple, 1-d array, Series

or DataFrame/dict-like

errors{‘ignore’, ‘raise’, ‘coerce’}, default ‘raise’

  • If ‘raise’, then invalid parsing will raise an exception

  • If ‘coerce’, then invalid parsing will be set as NaT

  • If ‘ignore’, then invalid parsing will return the input

formatstring, default None

strftime to parse time, eg “%d/%m/%Y”, note that “%f” will parse all the way up to nanoseconds.

unitstring, default None

unit of the arg (D,s,ms,us,ns) denote the unit, which is an integer or float number. This will be based off the origin. Example, with unit=’ms’ and origin=’unix’ (the default), this would calculate the number of milliseconds to the unix epoch start.

infer_datetime_formatboolean, default False

If True and no format is given, attempt to infer the format of the datetime strings, and if it can be inferred, switch to a faster method of parsing them. In some cases this can increase the parsing speed by ~5-10x.

originscalar, default ‘unix’

Define the reference date. The numeric values would be parsed as number of units (defined by unit) since this reference date.

  • If ‘unix’ (or POSIX) time; origin is set to 1970-01-01.

  • If ‘julian’, unit must be ‘D’, and origin is set to beginning of Julian Calendar. Julian day number 0 is assigned to the day starting at noon on January 1, 4713 BC.

  • If Timestamp convertible, origin is set to Timestamp identified by origin.

Returns
retdatetime if parsing succeeded.

Return type depends on input:

  • list-like: DatetimeIndex

  • Series: Series of datetime64 dtype

  • scalar: Timestamp

In case when it is not possible to return designated types (e.g. when any element of input is before Timestamp.min or after Timestamp.max) return will have datetime.datetime type (or corresponding array/Series).

Examples

Assembling a datetime from multiple columns of a DataFrame. The keys can be common abbreviations like [‘year’, ‘month’, ‘day’, ‘minute’, ‘second’, ‘ms’, ‘us’, ‘ns’]) or plurals of the same

>>> df = ps.DataFrame({'year': [2015, 2016],
...                    'month': [2, 3],
...                    'day': [4, 5]})
>>> ps.to_datetime(df)
0   2015-02-04
1   2016-03-05
dtype: datetime64[ns]

If a date does not meet the timestamp limitations, passing errors=’ignore’ will return the original input instead of raising any exception.

Passing errors=’coerce’ will force an out-of-bounds date to NaT, in addition to forcing non-dates (or non-parseable dates) to NaT.

>>> ps.to_datetime('13000101', format='%Y%m%d', errors='ignore')  
datetime.datetime(1300, 1, 1, 0, 0)
>>> ps.to_datetime('13000101', format='%Y%m%d', errors='coerce')
NaT

Passing infer_datetime_format=True can often-times speedup a parsing if its not an ISO8601 format exactly, but in a regular format.

>>> s = ps.Series(['3/11/2000', '3/12/2000', '3/13/2000'] * 1000)
>>> s.head()
0    3/11/2000
1    3/12/2000
2    3/13/2000
3    3/11/2000
4    3/12/2000
dtype: object

>>> import timeit
>>> timeit.timeit(
...    lambda: repr(ps.to_datetime(s, infer_datetime_format=True)),
...    number = 1)  
0.35832712500000063

>>> timeit.timeit(
...    lambda: repr(ps.to_datetime(s, infer_datetime_format=False)),
...    number = 1)  
0.8895321660000004

Using a unix epoch time

>>> ps.to_datetime(1490195805, unit='s')
Timestamp('2017-03-22 15:16:45')
>>> ps.to_datetime(1490195805433502912, unit='ns')
Timestamp('2017-03-22 15:16:45.433502912')

Using a non-unix epoch origin

>>> ps.to_datetime([1, 2, 3], unit='D', origin=pd.Timestamp('1960-01-01'))
DatetimeIndex(['1960-01-02', '1960-01-03', '1960-01-04'], dtype='datetime64[ns]', freq=None)