Scrapy shell view(response) not showing all HTML

• 我们只知道Scrapy无法呈现JavaSvript
• 由于scrapy无法呈现JavaSvript，这就是scrapy shell视图（响应）无法看到JavaScript动态加载的HTML部分的原因，也正是因为这个原因，Scrapy shell才不会显示所有HTML
• Safari，chrome或任何其他浏览器将显示完整的HTML DOM，无论它是动态或静态，但只有这样，你会看到动态和静态HTML之间的区别，在任何浏览器，当你从浏览器make disable JavaScript和刷新oppend网址，然后你永远不会看到动态HTML。这就是为什么Scrapy视图（响应）不显示所有HTML。

以Pandas拉静表

import pandas as pd
df =pd.read_html('https://www.baseball-reference.com/players/b/bondsba01.shtml')[0]
print(df)

输出：

Year            Age             Tm  ...  IBB     Pos          Awards
0            1985             20        PIT-min  ...    0     NaN      PRW · CARL
1            1986             21        PIT-min  ...    0     NaN       HAW · PCL
2            1986             21            PIT  ...    2    *8/H           RoY-6
3            1987             22            PIT  ...    3  *78H/9             NaN
4            1988             23            PIT  ...   14   *7H/8             NaN
5            1989             24            PIT  ...   22    *7/H             NaN
6            1990             25            PIT  ...   15   *7/H8  AS,MVP-1,GG,SS
7            1991             26            PIT  ...   25   *7/H8     MVP-2,GG,SS
8            1992             27            PIT  ...   32    *7/H  AS,MVP-1,GG,SS
9            1993             28            SFG  ...   43    *7/H  AS,MVP-1,GG,SS
10           1994             29            SFG  ...   18    *7/H  AS,MVP-4,GG,SS
11           1995             30            SFG  ...   22    *7/H       AS,MVP-12
12           1996             31            SFG  ...   30   *7/H8  AS,MVP-5,GG,SS
13           1997             32            SFG  ...   34      *7  AS,MVP-5,GG,SS
14           1998             33            SFG  ...   29    *7/H     AS,MVP-8,GG
15           1999             34            SFG  ...    9    7/DH          MVP-24
16           2000             35            SFG  ...   22    *7/H     AS,MVP-2,SS
17           2001             36            SFG  ...   35   *7/DH     AS,MVP-1,SS
18           2002             37            SFG  ...   68   *7/DH     AS,MVP-1,SS
19           2003             38            SFG  ...   61   *7/DH     AS,MVP-1,SS
20           2004             39            SFG  ...  120   *7/HD     AS,MVP-1,SS
21           2005             40            SFG  ...    3     7/H             NaN
22           2006             41            SFG  ...   38   *7H/D             NaN
23           2007             42            SFG  ...   43   *7H/D              AS
24         22 Yrs         22 Yrs         22 Yrs  ...  688     NaN             NaN
25  162 Game Avg.  162 Game Avg.  162 Game Avg.  ...   37     NaN             NaN
26            NaN            NaN            NaN  ...  IBB     Pos          Awards
27   SFG (15 yrs)   SFG (15 yrs)   SFG (15 yrs)  ...  575     NaN             NaN
28    PIT (7 yrs)    PIT (7 yrs)    PIT (7 yrs)  ...  113     NaN             NaN

[29 rows x 30 columns]

这些表并不是真正的动态的，它们实际上只是在html注解中。
有两种方法可以实现：
1.使用BeautifulSoup提取Comments，然后解析它
1.只需删除注解标记
这将得到所有的表。现在只需要通过特定的属性或df_list中的索引位置来取出你想要的表。

import pandas as pd
import requests

response = requests.get('https://www.baseball-reference.com/players/b/bondsba01.shtml')
html = response.text.replace("<!--","").replace("-->","")

df_list = pd.read_html(html)

要指定表，请执行以下操作：

import pandas as pd
import requests

response = requests.get('https://www.baseball-reference.com/players/b/bondsba01.shtml')
html = response.text.replace("<!--","").replace("-->","")

df = pd.read_html(html, attrs={'id':'batting_postseason'})[0]

输出：

print(df)
                Year               Age                Tm  ...   IBB   WPA    cWPA
0               1990                25               PIT  ...   0.0 -0.13   -0.2%
1               1991                26               PIT  ...   0.0 -0.68  -14.7%
2               1992                27               PIT  ...   1.0  0.08    1.0%
3                NaN               NaN               NaN  ...   NaN   NaN     NaN
4               1997                32               SFG  ...   0.0  0.31    3.3%
5                NaN               NaN               NaN  ...   NaN   NaN     NaN
6               2000                35               SFG  ...   1.0 -0.10   -1.6%
7                NaN               NaN               NaN  ...   NaN   NaN     NaN
8               2002                37               SFG  ...   3.0  0.05    2.6%
9               2002                37               SFG  ...   3.0  0.59    9.0%
10              2002                37               SFG  ...   7.0  0.56   22.9%
11              2003                38               SFG  ...   6.0  0.50    5.7%
12  7 Yrs (9 Series)  7 Yrs (9 Series)  7 Yrs (9 Series)  ...  21.0  1.18   27.9%
13            4 NLDS            4 NLDS            4 NLDS  ...  10.0  0.76    9.9%
14            4 NLCS            4 NLCS            4 NLCS  ...   4.0 -0.14   -5.0%
15              1 WS              1 WS              1 WS  ...   7.0  0.56   22.9%

[16 rows x 32 columns]