admin管理员组

文章数量:1313313

I have a script that uses the requests module to check a webpage for news updates. It worked up until a week ago, but now I am having a weird error. I am able to access the base url (www.example) using requests, but when I go to www.example/news/123456 it is appearing to have an error 500. This is what it says:

Internal Server Error
Sorry, There were some technical issues while processing your request.

However, I do not have this problem with selenium. Does anyone know what may be causing this error which is only happening with requests?

Here is my code:

import requests

url = 'www.example/news/123456/'
#Headers that I got from the (working) selenium request
headers = {'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/90.0.4430.212 Safari/537.36'}
response = requests.get(url, headers=headers)

webpage_content = response.text
soup = BeautifulSoup(webpage_content, 'html.parser')
raw_lines = soup.find_all()

lines = '\n'.join([line.get_text() for line in raw_lines])
lines = lines.splitlines()

for line in lines:
    print(line)

本文标签：

版权声明：本文标题：python - Server error when trying to access webpage with Requests, but no error with Selenium - Stack Overflow 内容由网友自发贡献，该文观点仅代表作者本人， 转载请联系作者并注明出处：http://www.betaflare.com/web/1741942835a2406242.html， 本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌抄袭侵权/违法违规的内容，一经查实，本站将立刻删除。