python - Fetching multiple urls with aiohttp from a loopgenerator - Stack Overflow

admin管理员组

文章数量:1332872

All examples I saw for fetching multiple urls with aiohttp suggest to to the following:

async def fetch(session, url):
    async with session.get(url, ssl=ssl.SSLContext()) as response:
        return await response.json()


async def fetch_all(urls, loop):
    async with aiohttp.ClientSession(loop=loop) as session:
        results = await asyncio.gather(*[fetch(session, url) for url in urls], return_exceptions=True)
        return results


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    urls = url_list
    htmls = loop.run_until_complete(fetch_all(urls, loop))
    print(htmls)

()

In practice, however, I typically have a generator (can be also async) returning domain objects from db, one attribute of which is url, but I also need access to other attributes later in the loop:

async for domain_obj in generator:
    url = domain_obj.url
    response = xxx # need to fetch single url here in async manner
    # do something with response

Of course I can batch collect domain_objs in a list, and fetch all of them like in example, but this doesn't feel right.

All examples I saw for fetching multiple urls with aiohttp suggest to to the following:

async def fetch(session, url):
    async with session.get(url, ssl=ssl.SSLContext()) as response:
        return await response.json()


async def fetch_all(urls, loop):
    async with aiohttp.ClientSession(loop=loop) as session:
        results = await asyncio.gather(*[fetch(session, url) for url in urls], return_exceptions=True)
        return results


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    urls = url_list
    htmls = loop.run_until_complete(fetch_all(urls, loop))
    print(htmls)

(https://stackoverflow/a/51728016/294103)

In practice, however, I typically have a generator (can be also async) returning domain objects from db, one attribute of which is url, but I also need access to other attributes later in the loop:

async for domain_obj in generator:
    url = domain_obj.url
    response = xxx # need to fetch single url here in async manner
    # do something with response

Of course I can batch collect domain_objs in a list, and fetch all of them like in example, but this doesn't feel right.

• python
• python-asyncio
• aiohttp

Share Improve this question Follow asked Nov 21, 2024 at 6:26 dragoondragoon 5,74455 gold badges3939 silver badges5656 bronze badges 2

• 1 As an aside, do not use the deprecated asyncio.get_event_loop method. – Booboo Commented Nov 21, 2024 at 13:51
• yes, thank you, was copied from another question. – dragoon Commented Nov 21, 2024 at 18:04

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 1

My first thought is that you probably want to use TaskGroup.

Something like:

    async with aiohttp.ClientSession() as session:
        tasks = []
        async with asyncio.TaskGroup() as tg:
            async for domain in generator:
                tasks.append(tg.create_task(fetch(session, domain)))
    return [t.result() for t in tasks]

See the linked reference for details on exception handling etc.

This will allow the tasks to start executing while waiting for more results from the generator, rather than needing to exhaust the generator before any tasks are started.

本文标签： pythonFetching multiple urls with aiohttp from a loopgeneratorStack Overflow