Scrapy使用:not CSS选择器排除部分文本

khbbv19g  于 2022-11-09  发布在  其他
关注(0)|答案(1)|浏览(141)

我一直致力于Scrapy,试图真正理解它,但是我似乎仍然有一些概念上的障碍。我现在正在抓取一个页面,在那里我想从一个div的子div中提取所有文本,同时排除其中一个子div。在我看来,我可以使用response.css("parent-div ::text").getall()从父div中提取文本,然后删除将在子div中找到的文本(* 虽然我觉得这很难理解,尽管我有代码可以实现 *)或者我可以尝试使用:not selector伪类。但是,我很难理解如何正确地使用它们。
我在这里抓取的HTML的基本结构是这样的(实际页面是:(第10页)

<div class = "entry-content">
  <p></p>
  <p></p>
  <p></p>
  <p></p>
  <p></p>
  <p></p>
  <div id="jp-post-flair" class="sharedaddy sd-like-enabled sd-sharing-enabled">
</div>

现在,我已经设法制作了一个代码片段,它设法只返回我不想要的内容。

response.css("div.entry-content div:not([id='jp-post-flair']) ::text").getall()

以下是我目前的解决方法

chapt_text = response.css("div.entry-content ::text").getall()
text_exclude = response.css("div.entry-content div:not([id='jp-post-flair']) ::text").getall()
chapt_text = [elem for elem in chapt_text if elem not in text_exclude] # removing some of the text we don't want

我上面的变通方法 * 是为了得到我的最终结果 *,但我希望能够用途:not,这样它就更紧凑了(尽管我认为这不会那么明确,因为它有自己的缺点)。
大多数情况下,我寻找的是更多关于如何最好地使用的信息:不正确。我没有找到太多真正描述如何使用它的资源。
预期输出(预加载项和管道)(明显示例)
['文本','更多文本','等']

编辑日期:2012年9月22日

在阅读了答案之后,我意识到我不需要用途:not来获得我想要的东西。我犯了一个错误,使用::text来提取父容器(div. entry-content)的所有子文本,而不是意识到我可以提取p标记response.css("div.entry-content > p ::text").getall()的所有子文本。因此,虽然我得到了一个更简洁的工作结果,但我最终没有使用:not伪类。

bjg7j2ky

bjg7j2ky1#

我不是css选择器的Maven,也不是使用:not qsuedo状态的Maven,但是如果你的目标是简洁,还有很多其他的选择可以实现。
例如,您可以使用一个XPath表达式来获得您要查找的结果。

chapt_text = response.xpath('//div[@class="entry-content"]/p//text()').getall()

输出功率

['“The priests lie, my friend. A bargain with a devil does not pervert your meanings, or seek to twist your nature. Why would it need to, when the honest desires of men are already so wicked?”', '\n– Kayode Owusu, Warlock under Dread
Emperors Vindictive I and Nihilis', 'When I’d told Tariq that if he wanted to talk about the Saint we’d have to do it while walking, I’d meant it as a way to put him off. Considering we were in a broken ruin of a realm infested with d
evils, undead and whatever else had might have been summoned and bound, it seemed foolish to have such a conversation when we should be keeping our eyes out on our surroundings instead. How silly of me not to realize that I was dealin
g with the Grey Pilgrim: he was more than willing to take my words at face value if it got him his way, and I couldn’t even recant. Not without seeming like I was the one out to get the heroine, anyway, which would win me no favours w
ith the heroic three fifths of our party as well as quite possibly turn into a liability down the line. It was one thing if I killed the Saint of Sword in my own defence or that of Masego’s, another if just like when I’d snatched back
 Black’s body I was baiting her to better take a swing. One would be a tragedy that could be mended, in time, but the other would eat away at the foundation of the alliances I wanted to make. So, when after a few traded whispers with
the Peregrine the Saint went on ahead to scout the way through, I sighed but did not object when he fell in at my side. Quite a pair we made, the winded old man and the dusty cripple.', '“I had a conversation with your teacher, before
 his soul was cut out and sealed,” Tariq Fleet-foot suddenly said.', 'He’d meant to catch me by surprise, which made the way I was just a little too slow in keeping that surprise off my face all the more irritating. My limp faltered,
and the way I turned it into a painful longer stride wouldn’t have fooled me – much less an old hand like the Pilgrim.', '“Did you?” I blandly replied. “Interesting.”', 'Like a horse about to bold, there was now no telling where this
was headed. If he’d wanted my undivided attention, well, he godsdamned had it.', '“He is,” the Pilgrim agreed. “The qualities that steer him could be considered virtues, in a certain light. Had he chosen to serve the cause of Above in
stead of Below he would have made a great champion.”', 'My lips quirked, though it was mockery and not amusement that moved them. All I could think of was green eyes burning with something mad, in a little room in Marchford, and that
implacable anger that was at the heart of him. Amadeus of the Green Stretch, carrying the banner of the Heavens? No, it would go against every grain of who he was – he was capable of doing great good, he truly was, in that Tariq had g
rasped him exact. But his disdain for Good was set in the marrow of his bones, and there would be no changing that without changing every other part of him.', '“I expect if you told him as much it was not well-received,” I said.', '“I
 believe he made his finest effort to wound me with words alone,” the old man said, sounding unmoved.', 'I threw an assessing glance at the Grey Pilgrim, finding his tone just a little too blithe. His face was the same, so tranquil I
could not help but wonder if it was forced. I’d known Black to twist or break people with but a few calculated sentences, and though the Peregrine would be made of sterner stuff than these he would also have a graveyard’s worth of ske
letons in his closet. On the other hand, Black had cultivated his reputation – his legends – into as much of a weapon as the rest of him. It was always hard to discern what he could and could not do, which had always been the way the
man liked it.', '“Yet his insights, though harshly delivered, have allowed me to shed different light on things I once believed myself to fully understand,” the Pilgrim continued. “In the east, I believe a distinction is drawn between
 ', 'Name', ' and ', 'Role', '.”', '“The Book of All Things does to begin with, if you read into certain parts,” I pointed out.', 'For a beat I sought the exact passage, one of the few I’d actually learned by rote.', '“To every soul,
great and small, purpose will be tendered,” I quoted. “Through crucible of choice are lives shaped, and one’s mark on Creation defined.”', 'The passage went on to say some pointed things about villainy being a twisting of that tendere
d purpose, and so Evil as well as evil, but I’d always taken the Book with a grain of salt. It was a beloved and well-worn story in Callow that some ancient Count of Denier had used that very passage to argue that it was in fact impio
us not pay taxes promptly and in full. Once words were put to ink, anybody could put them to use and those particular words were so old none could say who’d first written them – more than simply the purposes, I suspected that the word
s themselves had shifted over the centuries. They couldn’t ', 'not', ' have, after all, considering no one in those days had spoken Lower Miezan before said empire came to Calernia and the Callowan manuscripts of the Book were in that
 language. No translation could be perfect, my expanding repertoire of spoke and written languages had made painfully clear. The Grey Pilgrim’s glance at me was openly amused, which was when I was forced to acknowledge I’d just quoted
 scripture at a man who rubbed elbows with angels. Ah. Awkward.', '“As you say, Queen Catherine,” he said. “I must commend whoever it was that saw to your religious education.”', 'I wondered how he’d take if I told him I’d drifted thr
ough most sermons at the House and only begun studying the Book with any seriousness at the prompting of the wicked servant of the Hellgods better known as the Black Knight. Or, for all that matter, that the only person I’d comprehens
ively discussed theology with in the last few years was Masego, a man whose main interest in the matter was the practicalities of deicide. ', 'In all fairness', ', I thought', ', that’s turned out surprisingly pertinent to our lives.'
, '“In Levant, we speak of it simply as Bestowal,” Tariq said. “A gift from Above or a curse from Below. What is done with these is our choice, and the strength of the mark left on Creation is but the illustration of the character of
they who were bestowed. One who cultivated customs leading to greatness will leave great legacy behind, deeds worthy of recording. One who allowed mortal failings to remain paramount will be but a line in the ledgers of the Blood, soo
n forgot.”', '“I’d noticed,” I slowly said, “that your nobles – your Blood – seem particularly set in their ways.”', '“We seek to emulate admirable people, Queen Catherine, but those people are long gone,” the Pilgrim sadly said. “And
 their wars, their foes, their disasters are no longer our own. In being inflexible of virtue we have made virtue of inflexibility, often to our detriment. It is a way of thinking, you see, that exalts great deeds done in the name of
the Heavens without giving though to their aftermath. Their consequences. At our finest – and make no mistake, for all its flaws the Dominion has rendered great and righteous service for no rewards at all – my people are an assembly o
f heroes, Bestowed or not. At our worst, we seek glory heedlessly and recklessly kill over matters of honour.”', 'Which, while a fascinating look into the Dominion from a man who knew it like few others could and likely ever would, ha
d little bearing on the Saint of Swords or even Black that I could see.', '“I had thought myself, through the nature events that shaped me, freed of these fetters so common to my people,” the Grey Pilgrim quietly said. “I was, it has
become clear, terribly wrong in this.”', 'After the first surprise he’d sprung on me I’d grown careful to mask my thoughts, but hearing the old man that was arguably the most accomplish hero of our age – and likely a century or two be
fore that – bluntly admit he’d made a grave mistake almost put another stutter to my steps. There was regret in the way the Peregrine had spoken, but mostly it was an honest admission of error. And that was, I thought, why even when h
e sought to end me it was difficult to hate the man. Because even when he dipped into hypocrisy, even when he dug in his heels long past the point he should, the Grey Pilgrim was trying to do good. And when he failed in that, he looke
d the truth of it in the eye and owned it.', '“I do not regret for a moment my service of the Heavens, Black Queen,” the old man honestly said, “but my blindness to the consequences of it is on my head. In doing merciful work I have s
own the seeds of reprisal far and wide and though ', 'never once', ' will I bend my head to Evil for fear of contest, more should have been done to prepare Calernia for the storm.”', 'It sounded, I thought, like he was blaming himself
 for the Dead King’s stirring. Which seemed backward to me, considering I was fairly sure it was Malicia who’d first opened the gates for his intervention in Creation. Oh, I’d sought to make a bargain as well after receiving envoy fro
m Neshamah but she’d been wearing a body in Keter long before I arrived. If my suspicions were correct and the Dead King avoided intervention save at the invitation of another Evil – to place, in a way, the burden of opposition to Goo
d on another – then it was the Tower’s hand and not any hero’s that was at work. ', 'On the other hand, would he have moved if he’d not seen opportunity?', ' I wondered. I doubted an invitation was all it took to secure the aid of the
 Dead King. Perhaps the Grey Pilgrim was right, and in some eldritch way his works had paved the grounds for the King of Death’s coming. But even so, fuck the idea that the old man was ', 'responsible', ' for the slaughter that ensued
. I’d stood on the opposite end of the field from the Pilgrim more than once, but I could only praise the vast majority of what he’d done over his many decades of holding a Name.', '“You’ve been a helping hand,” I replied. “Sometimes
I question the soundness of the causes you’ve helped, but not your intent.”', '“That is kind of you,” Tariq said, bowing his head. “And you are not wrong to say I was hand, and mayhaps on occasion a finger on the scale. I was offered
chances, you see, to intervene when there was still contest to be had. When the balance had yet to swing.”', 'He paused.', '“Laurence de Montfort was sent forth, for near as many years as I, when there was absolutely nothing left to s
ave,” he gravely said.', 'And there we were at last, I thought. The song and dance to convince me to stay my hand if a moment came where she turned on me. That the Pilgrim had pressed so hard for this conversation to happen in the fir
st place told me everything I needed to know about the odds of it happening.', '“So she’s seen the deep end,” I said, unimpressed.', '“No, Queen Catherine, she has ', 'swum', ' in it,” the old man sadly said. “When we first spoke in C
allow, years ago, you told me you were tired of killing children because they were on the wrong side. Asked me if I was. And I am, Black Queen, Heavens forgive me but I am. Yet mine was still the lighter of the burdens, for even Laure
nce’s victories have only ever come in the wake of disaster.”', 'My brows furrowed. If I was following his meaning correctly, he was implying that while his role had been snuffing out disasters before they could fully form while the S
aint of Swords had been… well, cutting of limbs when the rot took.', '“You see her now, after a life of holding back the darkness, and find only bitterness and distrust,” Tariq said. “I do not expect these to endear her to you, Your M
ajesty, or even for cordiality to be attained. But I ask that you see her bared fangs for what they are: the scars left behind by a lifetime spent facing down the horrors of Calernia so no one else would have to.”', 'His voice wasn’t
pleading, not exactly, though knowing what I knew about the Peregrine if he thought that tossing aside his pride would save the Saint’s life he would discard it without a second thought. In that sense he was remarkably similar to my o
wn teacher, seeing little worth in personal dignity when it stood in the way of results. But though shy of a plea, there was no denying that a suit was being made.', '“I know better than most what it costs someone to tread through rui
n,” I acknowledged. “And many of mine were of my own making. But that must be owned, Pilgrim. It does not abnegate responsibility – ', 'especially', ' not in the powerful.”', '“Those ties got both ways,” the old man said. “There is no
t a soul on Calernia, Black Queen, that has not benefitted from the toil that clouded Laurence de Montfort. Sword in hand, she has danced with death for the sake of others a hundred times. From the windswept plains of the Chain of Hun
ger to the silent deeps of the Brocelian Forest: she has drowned plagues that would have killed dozens of thousands in the blood of hundreds, slain beloved heroes who sunk into madness and slaughter, sent scuttling back into the dark
all manners of old gods whose hungers grew wicked – though not before they had their taste.”', 'His blue eyes grew hard as steel, when he met mine.', '“All this she has endured, and endured for so long that Creation itself tempered he
r into something beyond breaking,” the old man said. “I have known souls sworn to Endurance that would weep at having lived half her life – and for this she has asked no reward, no riches nor titles nor honours. Not a single thing, fo
r above all things Laurence de Montfort believes that strength must be put to righteous purpose.”', 'The Grey Pilgrim let out a long breath.', '“She is not kind,” he admitted, “for Creation has burned kindness out of her. She is not f
orgiving, for there are graves sown across many lands that taught her to cast forgiveness aside. She is not witty or brilliant or fascinating, those traits that so often make the worst of us seem forgivable. She is rough and brusque,
mistrusting, and there will never be a day where she does not see you as a seed of the Enemy.”', 'The Peregrine, old and bent as he was, held himself with the presence of ruler when he so wished. This was not one of those times, for h
e did not try to tower over me or browbeat into acquiescence. He was asking, as an equal or something close to it.', '“And still,” he said, voice growing rough with feeling, “I ask you to see you for what she is: a woman who saw evil
preying on the world and took up the sword in its defence. Selflessly, without once grudging what such service would wreak upon her soul.”', 'And I could see, through the grief in his voice, that there truly was a tragedy there. Becau
se he might be a decent actor, I thought, and perhaps a liar of some skill if there was cause for it, but he had not taken to it the way some of the people I knew had. The tremor in his voice was genuine, coming from someone who’d nev
er learned to fake it so perfectly they’d blurred even to themselves the difference between truth and lies.', '“It may be,” the Grey Pilgrim said, “that for the harrowing life she has led Laurence will be given place of honour at the
feet of the Gods when death finally takes her. That for greater service greater accolade will be rendered unto her. But that is the debt of the Gods Above, Black Queen, and that realm known only to the just is beyond our mortal unders
tanding.”', 'His fingers twisted into a symbol I did not recognize, though he did not even seem to notice their movement.', '“Those are not the Gods to which you keep, regardless, and so I do not ask you to keep to their ways or their
 dues,” Tariq said. “I speak to you instead as one of the living. We who still tread Creation, who have benefited from her shattering labours. We who owe better than a shallow grave to this woman. Not for what she might still do, thou
gh few are better suited for war on the Hidden Horror, or for the expedience of earthly alliances. We owe it for what she has ', 'already done', '.”', 'It was, I thought, a touching speech. Well spoken and from the heart. It might jus
t be, too, that every word he had spoken was true. That for all that I’d thrown my castigations in the face of these heroes when the Tenth Crusade came baying at my door for their temerity in coming to offer their ', 'salvation', ' mo
re than two decades too late, I’d still lived in the shadow of their protection. That these two old killers had borne the weight of half this continent on their back and these days had nothing but scars and bared swords to show for it
. It would have felt right, to follow the course of that thread to the conclusion that what had shaped Laurence de Montfort excused who she’d become. ', 'And yet.', '“You ask me, in essence,” I said, “to extend the courtesy of a staye
d hand because what has sharpened her to a fault was beyond her control.”', '“No,” the Pilgrim said, “you mistake me. She made the choice to-“', '“I understand you perfectly,” I said. “Just the same as your Blood, her character has le
d her to this place and this strife. That character is good, and so you ask me to excuse her.”', '“How carelessly you reduce a life of doing good to a single sentence,” he said.', '“It does weigh on the scales, what you say she did,”
I admitted. “But I have to ask, Pilgrim: this courtesy you ask of me, will you extend it in turn?”', 'The old man blinked in surprise.', '“I too have my bevy of broken souls,” I said. “And oh, they’re a vicious lot. No denying that. S
avage from their days in the wild, but they’re learning. One step at a time.”', 'I thought of the Doom, of the same woman who’d let her madness drench the world in blood whispering of the sacrifice she’d made and the woman it’d made h
er into.', '“Some are beyond redemption,” I admitted. “Others…”', 'Half the world, turned into a prop for the glory of the other half', ', spoken in a burning whisper. A sardonic smile beneath pale green eyes. And a knife into his rib
s, after the Folly, that I could not regret.', '“Have declared their own war on despair, and mutilated themselves in pursuit of victory,” I continued. “I’ve gathered them to me, by fate or happenstance, and they’re my responsibility.
Even the one high up in the palace, whose grief has sent into a dark not even his eyes can see through.\xa0 So I ask you again: when the time comes, and they are to be judged, will you return the courtesy you ask of me?”', 'Blue eyes
in a tanned face assessed me, wondering. He did not reply.', '“I thought so,” I replied. “Then were are allies in convenience, Pilgrim, and you earn no courtesy from me. If she bares her blade at Hierophant or myself, I will snuff her
 out.”', '“I had thought,” the old man said, “that agreement could be reached.”', '“You didn’t offer an agreement,” I calmly replied. “You asked for a concession.”', '“Then a barter,” Tariq said, “though we are both lessened for it.”'
, 'And it shamed me just a little when he said. That it’d come to this, but also the entire span – every intrigue I’d woven through and around the Tyrant, every trick I had yet to ply. And this man, I reminded myself, had mere hours b
een trying to leash me with the threat of death through a pattern of three. Not even a day had passed since we’d been at war, and still the disappointment in his gaze stung just a bit. ', 'I’ve disappointed people I love', ', I though
t, meeting his gaze. ', 'And that did not stay my hand. Neither will this.', '“You are in need of an eight crown,” the Pilgrim said. “To cast down yours now would endanger your efforts, for war is ill-time for succession. Kairos Theod
osian will fight you over his to his dying breath, for there is nothing he loves half as much in this world as the legacy he embodies and stripping him of right to rule would rob him of this.”', 'I inclined my head to the side in sile
nt concession.', '“I was once Tariq Isbili, of the Grey Pilgrim’s Blood, Honoured Son under the Seljun of Levant,” the old man said, and his voice rang with quiet authority. “Though stricken from the ledgers I have raised rulers of Le
vant and I have cast them down. My word has been taken for law, and my honour for the honour of the Dominion. If I took the Tattered Throne, the bloodlines would rally to my banner and acclaim me Seljun by right. That crown I promise
you, for the life of Laurence de Montfort.”', 'My fingers clenched, then unclenched.', '“If she kills Masego, I will murder her without hesitation,” I told him, meaning every word.', 'He grimaced, but he must have understood that ther
e was no concession in his power that would possibly make me effectively concede the right to the Saint to kill one of my dearest friends without consequence.', '“If she does not kill the Hierophant,” he said.', '“Then we have a barga
in,” I said.', 'We shook on it, amongst the ruins of what had once been a great city. It was not long after that the Saint returned, the Rogue Sorcerer looking harried and bloody as he leant against her. The Tyrant of Helike, he annou
nced, had betrayed us.', 'Finally', ', I thought.']

相关问题