pandas 如何连接两个列值在一定范围内的 Dataframe ?

vyswwuz2  于 2023-05-21  发布在  其他
关注(0)|答案(9)|浏览(159)

给定两个dataframe df_1df_2,如何连接它们,使得datetime列df_1位于dataframe df_2中的startend之间:

print df_1

  timestamp              A          B
0 2016-05-14 10:54:33    0.020228   0.026572
1 2016-05-14 10:54:34    0.057780   0.175499
2 2016-05-14 10:54:35    0.098808   0.620986
3 2016-05-14 10:54:36    0.158789   1.014819
4 2016-05-14 10:54:39    0.038129   2.384590

print df_2

  start                end                  event    
0 2016-05-14 10:54:31  2016-05-14 10:54:33  E1
1 2016-05-14 10:54:34  2016-05-14 10:54:37  E2
2 2016-05-14 10:54:38  2016-05-14 10:54:42  E3

得到对应的event,其中df1.timestampdf_2.startdf2.end之间

timestamp              A          B          event
0 2016-05-14 10:54:33    0.020228   0.026572   E1
1 2016-05-14 10:54:34    0.057780   0.175499   E2
2 2016-05-14 10:54:35    0.098808   0.620986   E2
3 2016-05-14 10:54:36    0.158789   1.014819   E2
4 2016-05-14 10:54:39    0.038129   2.384590   E3
wb1gzix0

wb1gzix01#

一个简单的解决方案是从start and end设置closed = both创建interval index,然后使用get_loc获取事件,即(希望所有日期时间都在时间戳dtype中)

df_2.index = pd.IntervalIndex.from_arrays(df_2['start'],df_2['end'],closed='both')
df_1['event'] = df_1['timestamp'].apply(lambda x : df_2.iloc[df_2.index.get_loc(x)]['event'])

输出:

timestamp         A         B event
0 2016-05-14 10:54:33  0.020228  0.026572    E1
1 2016-05-14 10:54:34  0.057780  0.175499    E2
2 2016-05-14 10:54:35  0.098808  0.620986    E2
3 2016-05-14 10:54:36  0.158789  1.014819    E2
4 2016-05-14 10:54:39  0.038129  2.384590    E3
vc6uscn9

vc6uscn92#

首先使用IntervalIndex基于感兴趣的区间创建引用索引,然后使用get_indexer对包含感兴趣的离散事件的 Dataframe 进行切片。

idx = pd.IntervalIndex.from_arrays(df_2['start'], df_2['end'], closed='both')
event = df_2.iloc[idx.get_indexer(df_1.timestamp), 'event']

event
0    E1
1    E2
1    E2
1    E2
2    E3
Name: event, dtype: object

df_1['event'] = event.to_numpy()
df_1
            timestamp         A         B event
0 2016-05-14 10:54:33  0.020228  0.026572    E1
1 2016-05-14 10:54:34  0.057780  0.175499    E2
2 2016-05-14 10:54:35  0.098808  0.620986    E2
3 2016-05-14 10:54:36  0.158789  1.014819    E2
4 2016-05-14 10:54:39  0.038129  2.384590    E3

参考:A question on IntervalIndex.get_indexer.

nnvyjq4y

nnvyjq4y3#

可以使用模块pandasql

import pandasql as ps

sqlcode = '''
select df_1.timestamp
,df_1.A
,df_1.B
,df_2.event
from df_1 
inner join df_2 
on d1.timestamp between df_2.start and df2.end
'''

newdf = ps.sqldf(sqlcode,locals())
lf3rwulv

lf3rwulv4#

备选方案1

idx = pd.IntervalIndex.from_arrays(df_2['start'], df_2['end'], closed='both')
df_2.index=idx
df_1['event']=df_2.loc[df_1.timestamp,'event'].values

备选方案2

df_2['timestamp']=df_2['end']
pd.merge_asof(df_1,df_2[['timestamp','event']],on='timestamp',direction ='forward',allow_exact_matches =True)
Out[405]: 
            timestamp         A         B event
0 2016-05-14 10:54:33  0.020228  0.026572    E1
1 2016-05-14 10:54:34  0.057780  0.175499    E2
2 2016-05-14 10:54:35  0.098808  0.620986    E2
3 2016-05-14 10:54:36  0.158789  1.014819    E2
4 2016-05-14 10:54:39  0.038129  2.384590    E3
sgtfey8w

sgtfey8w5#

在此方法中,我们假设使用了TimeStamp对象。

df2  start                end                  event    
   0 2016-05-14 10:54:31  2016-05-14 10:54:33  E1
   1 2016-05-14 10:54:34  2016-05-14 10:54:37  E2
   2 2016-05-14 10:54:38  2016-05-14 10:54:42  E3

event_num = len(df2.event)

def get_event(t):    
    event_idx = ((t >= df2.start) & (t <= df2.end)).dot(np.arange(event_num))
    return df2.event[event_idx]

df1["event"] = df1.timestamp.transform(get_event)

get_event说明

对于df1中的每个时间戳,例如t0 = 2016-05-14 10:54:33
(t0 >= df2.start) & (t0 <= df2.end)将包含1个true。(参见实施例1)。然后,与np.arange(event_num)进行点积,以获得t0所属事件的索引。

示例:

  • 实施例1*
t0 >= df2.start    t0 <= df2.end     After &     np.arange(3)    
0     True                True         ->  T              0        event_idx
1    False                True         ->  F              1     ->     0
2    False                True         ->  F              2

t2 = 2016-05-14 10:54:35为例

t2 >= df2.start    t2 <= df2.end     After &     np.arange(3)    
0     True                False        ->  F              0        event_idx
1     True                True         ->  T              1     ->     1
2    False                True         ->  F              2

最后,我们使用transform将每个时间戳转换为一个事件。

v1l68za4

v1l68za46#

通过将df_1的索引设置为timestamp字段,可以使pandas索引对齐为您所用。

import pandas as pd

df_1 = pd.DataFrame(
    columns=["timestamp", "A", "B"],
    data=[
        (pd.Timestamp("2016-05-14 10:54:33"), 0.020228, 0.026572),
        (pd.Timestamp("2016-05-14 10:54:34"), 0.057780, 0.175499),
        (pd.Timestamp("2016-05-14 10:54:35"), 0.098808, 0.620986),
        (pd.Timestamp("2016-05-14 10:54:36"), 0.158789, 1.014819),
        (pd.Timestamp("2016-05-14 10:54:39"), 0.038129, 2.384590),
    ],
)
df_2 = pd.DataFrame(
    columns=["start", "end", "event"],
    data=[
        (
            pd.Timestamp("2016-05-14 10:54:31"),
            pd.Timestamp("2016-05-14 10:54:33"),
            "E1",
        ),
        (
            pd.Timestamp("2016-05-14 10:54:34"),
            pd.Timestamp("2016-05-14 10:54:37"),
            "E2",
        ),
        (
            pd.Timestamp("2016-05-14 10:54:38"),
            pd.Timestamp("2016-05-14 10:54:42"),
            "E3",
        ),
    ],
)
df_2.index = pd.IntervalIndex.from_arrays(df_2["start"], df_2["end"], closed="both")

只需将df_1["event"]设置为df_2["event"]

df_1["event"] = df_2["event"]

df_1["event"]

timestamp
2016-05-14 10:54:33    E1
2016-05-14 10:54:34    E2
2016-05-14 10:54:35    E2
2016-05-14 10:54:36    E2
2016-05-14 10:54:39    E3
Name: event, dtype: object
ugmeyewa

ugmeyewa7#

一个选项是使用pyjanitor的conditional_join:

# pip install pyjanitor
import pandas as pd
import janitor

(df_1                         
.conditional_join(
          df_2, 
          # variable arguments
          # tuple is of the form:
          # col_from_left_df, col_from_right_df, comparator
          ('timestamp', 'start', '>='), 
          ('timestamp', 'end', '<='),
          how = 'inner')
.drop(columns=['start', 'end'])
)

            timestamp         A         B event
0 2016-05-14 10:54:33  0.020228  0.026572    E1
1 2016-05-14 10:54:34  0.057780  0.175499    E2
2 2016-05-14 10:54:35  0.098808  0.620986    E2
3 2016-05-14 10:54:36  0.158789  1.014819    E2
4 2016-05-14 10:54:39  0.038129  2.384590    E3

您可以使用how参数决定连接类型=> leftrightinner

w51jfk4q

w51jfk4q8#

在解决方案by firelynx here on StackOverflow中,这表明多态性不起作用。我不得不同意FireLynx(经过广泛的测试)。然而,将多态性的思想与the numpy broadcasting solution of piRSquared结合起来,它可以工作!
唯一的问题是,最终,在引擎盖下,numpy广播实际上做了某种交叉连接,我们过滤了所有相等的元素,给出了O(n1*n2)内存和O(n1*n2)性能。也许,有人可以使这在一般意义上更有效。
我在这里发帖的原因是,firelynx的解决方案问题是作为这个问题的重复而关闭的,我倾向于不同意。因为当你有多个点属于多个区间时,这个问题和其中的答案并没有给予解决方案,而只是针对属于多个区间的一个点。我下面提出的解决方案,* 确实 * 照顾到了这些n-m关系。
基本上,为多态性创建以下两个类PointInTimeTimespan

from datetime import datetime

class PointInTime(object):
    doPrint = True
    def __init__(self, year, month, day):
        self.dt = datetime(year, month, day)

    def __eq__(self, other):
        if isinstance(other, self.__class__):
            r = (self.dt == other.dt)
            if self.doPrint:
                print(f'{self.__class__}: comparing {self} to {other} (equals) gives {r}')
            return (r)
        elif isinstance(other, Timespan):
            r = (other.start_date < self.dt < other.end_date)
            if self.doPrint:
                print(f'{self.__class__}: comparing {self} to {other} (Timespan in PointInTime) gives {r}')
            return (r)
        else:
            if self.doPrint:
                print(f'Not implemented... (PointInTime)')
            return NotImplemented

    def __repr__(self):
        return "{}-{}-{}".format(self.dt.year, self.dt.month, self.dt.day)

class Timespan(object):
    doPrint = True
    def __init__(self, start_date, end_date):
        self.start_date = start_date
        self.end_date   = end_date

    def __eq__(self, other):
        if isinstance(other, self.__class__):
            r = ((self.start_date == other.start_date) and (self.end_date == other.end_date))
            if self.doPrint:
                print(f'{self.__class__}: comparing {self} to {other} (equals) gives {r}')
            return (r)
        elif isinstance (other, PointInTime):
            r = self.start_date < other.dt < self.end_date
            if self.doPrint:
                print(f'{self.__class__}: comparing {self} to {other} (PointInTime in Timespan) gives {r}')
            return (r)
        else:
            if self.doPrint:
                print(f'Not implemented... (Timespan)')
            return NotImplemented

    def __repr__(self):
        return "{}-{}-{} -> {}-{}-{}".format(self.start_date.year, self.start_date.month, self.start_date.day, self.end_date.year, self.end_date.month, self.end_date.day)

顺便说一句,如果你不想使用==,而是使用其他运算符(比如!=,<,>,<=,>=),您可以为它们创建相应的函数(__ne____lt____gt____le____ge__)。
您可以将其与广播结合使用的方式如下所示。

import pandas as pd
import numpy as np

df1 = pd.DataFrame({"pit":[(x) for x in [PointInTime(2015,1,1), PointInTime(2015,2,2), PointInTime(2015,3,3), PointInTime(2015,4,4)]], 'vals1':[1,2,3,4]})
df2 = pd.DataFrame({"ts":[(x) for x in [Timespan(datetime(2015,2,1), datetime(2015,2,5)), Timespan(datetime(2015,2,1), datetime(2015,4,1)), Timespan(datetime(2015,2,1), datetime(2015,2,5))]], 'vals2' : ['a', 'b', 'c']})
a = df1['pit'].values
b = df2['ts'].values
i, j = np.where((a[:,None] == b))

res = pd.DataFrame(
    np.column_stack([df1.values[i], df2.values[j]]),
    columns=df1.columns.append(df2.columns)
)
print(df1)
print(df2)
print(res)

这给出了预期的输出。

<class '__main__.PointInTime'>: comparing 2015-1-1 to 2015-2-1 -> 2015-2-5 (Timespan in PointInTime) gives False
<class '__main__.PointInTime'>: comparing 2015-1-1 to 2015-2-1 -> 2015-4-1 (Timespan in PointInTime) gives False
<class '__main__.PointInTime'>: comparing 2015-1-1 to 2015-2-1 -> 2015-2-5 (Timespan in PointInTime) gives False
<class '__main__.PointInTime'>: comparing 2015-2-2 to 2015-2-1 -> 2015-2-5 (Timespan in PointInTime) gives True
<class '__main__.PointInTime'>: comparing 2015-2-2 to 2015-2-1 -> 2015-4-1 (Timespan in PointInTime) gives True
<class '__main__.PointInTime'>: comparing 2015-2-2 to 2015-2-1 -> 2015-2-5 (Timespan in PointInTime) gives True
<class '__main__.PointInTime'>: comparing 2015-3-3 to 2015-2-1 -> 2015-2-5 (Timespan in PointInTime) gives False
<class '__main__.PointInTime'>: comparing 2015-3-3 to 2015-2-1 -> 2015-4-1 (Timespan in PointInTime) gives True
<class '__main__.PointInTime'>: comparing 2015-3-3 to 2015-2-1 -> 2015-2-5 (Timespan in PointInTime) gives False
<class '__main__.PointInTime'>: comparing 2015-4-4 to 2015-2-1 -> 2015-2-5 (Timespan in PointInTime) gives False
<class '__main__.PointInTime'>: comparing 2015-4-4 to 2015-2-1 -> 2015-4-1 (Timespan in PointInTime) gives False
<class '__main__.PointInTime'>: comparing 2015-4-4 to 2015-2-1 -> 2015-2-5 (Timespan in PointInTime) gives False
        pit  vals1
0  2015-1-1      1
1  2015-2-2      2
2  2015-3-3      3
3  2015-4-4      4
                     ts vals2
0  2015-2-1 -> 2015-2-5     a
1  2015-2-1 -> 2015-4-1     b
2  2015-2-1 -> 2015-2-5     c
        pit vals1                    ts vals2
0  2015-2-2     2  2015-2-1 -> 2015-2-5     a
1  2015-2-2     2  2015-2-1 -> 2015-4-1     b
2  2015-2-2     2  2015-2-1 -> 2015-2-5     c
3  2015-3-3     3  2015-2-1 -> 2015-4-1     b

与基本的Python类型相比,拥有类的开销可能会带来额外的性能损失,但我还没有对此进行研究。
以上就是我们如何创建“内部”连接。创建“(外部)左”、“(外部)右”和“(完全)外部”连接应该很简单。

bgibtngc

bgibtngc9#

如果df_2中的时间跨度不重叠,您可以使用numpy广播将时间戳与所有时间跨度进行比较,并确定它福尔斯哪个时间跨度之间。然后使用argmax来计算要分配哪个'Event'(因为最多只能有1个不重叠的时间跨度)。
where条件用于NaN任何可能落在所有时间跨度之外的对象(因为argmax不会正确处理此问题)

import numpy as np

m = ((df_1['timestamp'].to_numpy() >= df_2['start'].to_numpy()[:, None])
      & (df_1['timestamp'].to_numpy() <= df_2['end'].to_numpy()[:, None]))

df_1['Event'] = df_2['event'].take(np.argmax(m, axis=0)).where(m.sum(axis=0) > 0)
print(df_1)
            timestamp         A         B Event
0 2016-05-14 10:54:33  0.020228  0.026572    E1
1 2016-05-14 10:54:34  0.057780  0.175499    E2
2 2016-05-14 10:54:35  0.098808  0.620986    E2
3 2016-05-14 10:54:36  0.158789  1.014819    E2
4 2016-05-14 10:54:39  0.038129  2.384590    E3

相关问题