xpath表达式txt = ''.join([ x.get().strip() for x in response.xpath('//p//text()')])将提取值"ABCDEF".
由碎贝壳证实：

In [1]: from scrapy.selector import Selector

In [2]: %paste
html = '''
<p>
    A
    <br>
    <br>
    B
    <a ...>
        <span >C</span>
    </a>
    D
    <a ...>
        <span >E</span>
    </a>
    F
</p>
'''

## -- End pasted text --

In [3]: res= Selector(text=html)

In [4]: res.xpath('//p//text()').getall()
Out[4]: 
['\n    A\n    ',
 '\n    ',       
 '\n    B\n    ',
 '\n        ',   
 'C',
 '\n    ',       
 '\n    D\n    ',
 '\n        ',   
 'E',
 '\n    ',
 '\n    F\n']

In [5]: txt = [ x.get().strip() for x in res.xpath('//p//text()')]

In [6]: txt
Out[6]: ['A', '', 'B', '', 'C', '', 'D', '', 'E', '', 'F']

In [7]: txt = ''.join([ x.get().strip() for x in res.xpath('//p//text()')])

In [8]: txt
Out[8]: 'ABCDEF'

xpath('/p/text()')

or 

xpath('p ::text()')

两者都应该可以。检查this answer以获得更清晰的信息。另外，如果使用python -为提取的元素创建一个列表（）以保持顺序。