Writing

Python Asynchronous Programming #3: Network Requests

This tutorial was written in 2019. Python and library APIs may need changes to run these examples today.

The previous post introduced asyncio. This post compares ways of overlapping network waits while fetching dictionary pages.

The dictionary URLs and timings below are from the 2019 original. The service and results may have changed.

Sequential requests

Using requests, each call waits for its response before the next starts:


import requests
import time
from bs4 import BeautifulSoup

def get_text_from_url(url):
    print(f'Send request to ... {url}')
    res = requests.get(url, headers={'user-agent': 'Mozilla/5.0'})
    print(f'Get response from ... {url}')
    text = BeautifulSoup(res.text, 'html.parser').text
    return text

if __name__ == "__main__":
    start = time.time()

    base_url = 'https://www.macmillandictionary.com/us/dictionary/american/{keyword}'
    keywords = ['hi', 'apple', 'banana', 'call', 'feel',
                'hello', 'bye', 'like', 'love', 'environmental',
                'buzz', 'ambition', 'determine']

    urls = [base_url.format(keyword=keyword) for keyword in keywords]
    for url in urls:
        text = get_text_from_url(url)
        print(text[:100].strip())
    end = time.time()
    print(f'time taken: {end-start}')
>>
Send request to ... https://www.macmillandictionary.com/us/dictionary/american/hi
Get response from ... https://www.macmillandictionary.com/us/dictionary/american/hi
hi (interjection) American English definition and synonyms | Macmillan Dictionary
Send request to ... https://www.macmillandictionary.com/us/dictionary/american/apple
Get response from ... https://www.macmillandictionary.com/us/dictionary/american/apple
apple (noun) American English definition and synonyms | Macmillan Dictionary
Send request to ... https://www.macmillandictionary.com/us/dictionary/american/banana
Get response from ... https://www.macmillandictionary.com/us/dictionary/american/banana
...
Send request to ... https://www.macmillandictionary.com/us/dictionary/american/determine
Get response from ... https://www.macmillandictionary.com/us/dictionary/american/determine
determine (verb) American English definition and synonyms | Macmillan Dictionary
time taken: 12.112096309661865

The original run took about 12 seconds.

asyncio with requests and an executor

requests is blocking, so run_in_executor runs each request in a thread. functools.partial supplies keyword arguments, and gather waits for the scheduled tasks:

import requests
import time
import asyncio
from functools import partial
from bs4 import BeautifulSoup

async def get_text_from_url(url):
    print(f'Send request to ... {url}')
    loop = asyncio.get_event_loop()


    request = partial(requests.get, url, headers={
        'user-agent': 'Mozilla/5.0'})


    res = await loop.run_in_executor(None, request)
    print(f'Get response from ... {url}')
    text = BeautifulSoup(res.text, 'html.parser').text
    print(text[:100].strip())

async def main():
    base_url = 'https://www.macmillandictionary.com/us/dictionary/american/{keyword}'
    keywords = ['hi', 'apple', 'banana', 'call', 'feel',
                'hello', 'bye', 'like', 'love', 'environmental',
                'buzz', 'ambition', 'determine']


    futures = [asyncio.ensure_future(get_text_from_url(
        base_url.format(keyword=keyword))) for keyword in keywords]

    await asyncio.gather(*futures)

if __name__ == "__main__":

    start = time.time()
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())
    end = time.time()
    print(f'time taken: {end-start}')
>>
Send request to ... https://www.macmillandictionary.com/us/dictionary/american/hi
Send request to ... https://www.macmillandictionary.com/us/dictionary/american/apple
Send request to ... https://www.macmillandictionary.com/us/dictionary/american/banana
Send request to ... https://www.macmillandictionary.com/us/dictionary/american/call

...

Get response from ... https://www.macmillandictionary.com/us/dictionary/american/ambition
ambition (noun) American English definition and synonyms | Macmillan Dictionary
Get response from ... https://www.macmillandictionary.com/us/dictionary/american/apple
apple (noun) American English definition and synonyms | Macmillan Dictionary
Get response from ... https://www.macmillandictionary.com/us/dictionary/american/love
love (verb) American English definition and synonyms | Macmillan Dictionary
Get response from ... https://www.macmillandictionary.com/us/dictionary/american/determine
determine (verb) American English definition and synonyms | Macmillan Dictionary
...

time taken: 1.4359536170959473

The original run took about 1.43 seconds. Requests were started before earlier responses arrived, and completion order differed from request order. The improvement is specific to that run and environment.

asyncio with aiohttp

aiohttp offers asynchronous HTTP calls, so the code can await responses without assigning each request to a thread:

import time
import asyncio

import aiohttp
from bs4 import BeautifulSoup

async def get_text_from_url(url):
    print(f'Send request to ... {url}')

    async with aiohttp.ClientSession() as sess:
        async with sess.get(url, headers={'user-agent': 'Mozilla/5.0'}) as res:
            text = await res.text()

    print(f'Get response from ... {url}')
    text = BeautifulSoup(text, 'html.parser').text
    print(text[:100].strip())

async def main():
    base_url = 'https://www.macmillandictionary.com/us/dictionary/american/{keyword}'
    keywords = ['hi', 'apple', 'banana', 'call', 'feel',
                'hello', 'bye', 'like', 'love', 'environmental',
                'buzz', 'ambition', 'determine']


    futures = [asyncio.ensure_future(get_text_from_url(
        base_url.format(keyword=keyword))) for keyword in keywords]

    await asyncio.gather(*futures)

if __name__ == "__main__":

    start = time.time()
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())
    end = time.time()
    print(f'time taken: {end-start}')
>>
Send request to ... https://www.macmillandictionary.com/us/dictionary/american/hi
Send request to ... https://www.macmillandictionary.com/us/dictionary/american/apple
Send request to ... https://www.macmillandictionary.com/us/dictionary/american/banana
Send request to ... https://www.macmillandictionary.com/us/dictionary/american/call
...

Get response from ... https://www.macmillandictionary.com/us/dictionary/american/hello
hello (interjection) American English definition and synonyms | Macmillan Dictionary
Get response from ... https://www.macmillandictionary.com/us/dictionary/american/environmental
environmental (adjective) American English definition and synonyms | Macmillan Dictionary
Get response from ... https://www.macmillandictionary.com/us/dictionary/american/apple
apple (noun) American English definition and synonyms | Macmillan Dictionary
...

time taken: 1.4322197437286377

The original example was too small to reveal a substantial difference between the executor and aiohttp approaches. When crawling, limit concurrency and respect the target service; rapid requests can overload it or lead to blocking.

Asyncio ecosystem

Awesome Asyncio lists libraries built around asyncio. The 2019 article also mentioned Sanic and Vibora as asynchronous web frameworks and linked to Vibora's historical benchmark. The benchmark image from the original post is no longer accessible; it should not be used to infer current performance.

That concludes this series on asynchronous Python.