[Proposal] Support Python asyncio by migrating to httpx from requests #27

Activity

1. adarshdigievo

   added

   improvement

   on May 31, 2026
2. jvmvik commented on Jun 1, 2026

   

   jvmvik

   on Jun 1, 2026

   Contributor

   More actions

   I agree with the motivation. I have already created a branch with 0.2.0 based on aiohttp which is based on asyncio.

   

   But I stopped in my tracks for 2 reasons related to customer concerns.

   • python 2.x is no longer supported.
   • dependencies: new dependencies which breaks enterprise stacks.
   • synchronization with https://serpapi.com/integrations/python

   Historically we have done a bad job synchronizing the web documentation update, and the plugin release.
   The plugin doc is pull automatically in the website, but the "Code to integrate" need manual update.

   

   solutions:

   1. drop python2 compatibility. (need to check market share left of python 2)
   2. add the new library under python3 entry, and keep python2

   I'd like 2 because it does break existing enterprise stack, and allow them to migrate slowly. but maybe this argument doesn't stand anymore in the age of AI.

   what do you think ?
3. adarshdigievo commented on Jun 3, 2026

   

   adarshdigievo

   on Jun 3, 2026

   MemberAuthor

   More actions

   Hi @jvmvik,

   Thanks for sharing the details. Please see my comments below:

   • Breaking changes/updates needed for integrations page: My idea is to keep the existing sync API as-is. We will not break any existing code. Existing users will not require any code changes.

   Even after the httpx migration, the below sync usage will work:

   import os
   import serpapi
   
   def send():
     client = serpapi.Client(api_key=os.getenv("SERPAPI_KEY"))
     results = client.search({
       "engine": "google",
       "q": "coffee"
     })
   
     print(results)

   ---

   For Async, we can introduce a new client (serpapi.AsyncClient)- this was not mentioned correctly in my previous comment:

   import os
   import serpapi
   
   async def send():
     client = serpapi.AsyncClient(api_key=os.getenv("SERPAPI_KEY"))
     results = await client.search({
       "engine": "google",
       "q": "coffee"
     })
   
     print(results)

   This is the advantage I see with httpx: it supports both sync and async network requests and has a design that is almost compatible with our current requests library. By migrating to httpx, we can maintain backward compatibility with our user-facing methods and support async as well.

   Because of the backward compatibility, we can keep our integrations page as-is and can mention async support in our library docs.

   • Regarding the performance benefits, I agree that aiohttp will be more performant. But as you rightly noted, a full migration will be costly - we need to add a new package so that users of our current sync client will not see any breaking changes. Or we need to have sync calls from our package using requests under the hood (keeping existing behavior) and implement new async methods that use aiohttp, which will increase the maintenance burden.

   Before starting work on this, I will run a benchmark using our real API calls to measure actual performance.

   If we are moving forward with httpx, we can implement support for both sync and async calls with a single dependency with minimal changes to existing code. This is the unique benefit of httpx.

   # sync
   r = httpx.get(''https://www.example.com/')

   # async
   async with httpx.AsyncClient() as client:
        r = await client.get('https://www.example.com/')

   Also, the community is very active around httpx - Pydantic is maintaining their fork of httpx, and I expect to see performance improvements going forward.

   • Python 2 compatibility: I don't think this should be a concern for us. Python 2 reached end of life in 2020, and most of our dependencies do not support it. The current version of requests supports only Python 3.9+. Our pyproject.toml currently lists the required Python version as >=3.6, so Python2 support is safe to ignore. I also verified this using a Python 2.7 Docker image: pip install serpapi fails on Python 2.

   • Dependency breakages: As a part of this change, we will have our dependency changed from requests to httpx. Due to backward-compatible design, if a customer integration gets our updated client, their existing code will work without any major updates. Also, since we will be bumping the version number for the release, pinned package install will have the previous version using requests installed.
     I guess internal dependency migrations like this are common (saw that OpenAI Python SDK also made a similar migration to httpx from requests, ref)
4. jvmvik commented on Jun 8, 2026

   

   jvmvik

   on Jun 8, 2026

   Contributor

   More actions

   Convince me httpx is the right choice. Benchmarks between httpx and aiohttp would likely be too close to show
   meaningful differences. Most developers avoid async programming because it adds substantial complexity for minimal
   returns in real-world applications—the only real benefit is high performance, and Python isn't a high-performance
   language anyway.
5. jvmvik commented on Jun 8, 2026

   

   jvmvik

   on Jun 8, 2026

   Contributor

   More actions

   about the package manager, what are you planning to use ? for marketing reason we should document both uv and pip3

   👍React with 👍1Reacted by Adarsh Divakaran
6. adarshdigievo commented on Jun 15, 2026

   

   adarshdigievo

   on Jun 15, 2026

   MemberAuthor

   More actions

   @jvmvik Please see https://serpapi.slack.com/archives/C09AS452KMZ/p1781258667885529 - added a few benchmark results and analysis on our internal discussion thread.
7. adarshdigievo

   marked Add async support (e.g. AsyncClient) for use in async frameworks #35 as a duplicate of this issue on Jul 10, 2026