Faster way to impute datetime columns

User 2568 | 5/6/2016, 1:47:44 AM

I have a datetime column, where some items are None. I want to impute these from another datetime column. Is there a better, faster way to implement this

sf['srch_ci'] = sf.apply(lambda row: row['srch_ci'] if row['srch_ci']  else row['date_time'] + ahead)

In panda think you could write something like

mask =  sf['srch_ci'] == None
sf[mask]['srch_ci']  = sf[mask]['date_time'] + ahead)

Comments

User 1189 | 5/8/2016, 2:39:52 AM

I am looking into implementing a general version of this feature... i.e. replacing a value (or a mask of values) from another column. Unfortunately, this feature is still under design and development.

In the meantime... This is a hack. But I think it should work.

Assume one column is A and the other column is B. ` import datetime

convert to integer (this is just a unix timestamp) and filling in missing values with 0

Aint = A.astype(int).fillna(0) Bint = B.astype(int).fillna(0)

Now Aint and Bint can be added together, but we only want to select the entries of Bint where A is missing

Resultint = Aint + Bint * (A == None)

convert back to datetime

Result = Resultint.astype(datetime.datetime) `

Yucheng