admin管理员组

文章数量:1316662

I'd like to implement custom Scrapy HTTP cache. I know how to set HTTPCACHE_STORAGE to my class and what methods to implement, but the problem is that the storage I want to use is asynchronous, while the HTTPCACHE_STORAGE protocol is expected to be synchronous. Is there any way I can do this?

This isn't inside spider and for the methods to work, they must be simple def retrieve_response(...) and so on. But inside that method I need to call await .... Since there's already one event loop running in Scrapy, I cannot run another one. How do I do this?

I use TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor".

本文标签：

版权声明：本文标题：python asyncio - How to extend Scrapy with custom HTTP cache which needs to perform asynchronous calls - Stack Overflow 内容由网友自发贡献，该文观点仅代表作者本人， 转载请联系作者并注明出处：http://www.betaflare.com/web/1742001326a2411069.html， 本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌抄袭侵权/违法违规的内容，一经查实，本站将立刻删除。