下面的selenium, bs4 and pandas示例运行得很顺利，我必须使用Javascript执行来单击并完成show more result。

范例：

import time
from bs4 import BeautifulSoup
import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By

options = webdriver.ChromeOptions()
options.add_argument("start-maximized")

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=options)

url = 'https://www.cardmarket.com/en/Magic/Products/Singles/Exodus/Survival-of-the-Fittest'
driver.get(url)
time.sleep(5)

lst=[]
while True:

    soup=BeautifulSoup(driver.page_source,'lxml')
    for card in soup.select('[class="d-flex has-content-centered mr-1"] > a'):
        lst.append({'name': card.text})

    try:     
        driver.execute_script("arguments[0].scrollIntoView();",driver.find_element(By.XPATH,'//*[@id="loadMore"]/button/span[2]'))
        pos= driver.find_element(By.XPATH,'//*[@id="loadMore"]/button/span[2]').click()

        time.sleep(2)
    except:
        break

df=pd.DataFrame(lst)
print(df)

输出：

name
0               Lugones
1              odaJoana
2        Arcana-Trieste
3        Arcana-Trieste
4              Impavido
..                  ...
145              yoeril
146  JacobMartinNielsen
147               Artia
148               Nanau
149           magictuga

[150 rows x 1 columns]