aiohttp with charset-normalizer


  • aiohttp.TCPConnector(limit=16)
  • alpine linux
  • nginx 1.21
  • python 3.9
  • aiohttp dev-master
  • chardet 4.0.0 (aiohttp-chardet)
  • charset-normalizer 2.0.6 (aiohttp-next)

Prepare the environment

docker-compose build aiohttp-next server
docker-compose build --no-cache aiohttp-chardet

docker-compose up -d server

Collect results

docker-compose up aiohttp-chardet
docker-compose up aiohttp-next

Memory sampling (to be run in parallel of aiohttp-* services)

docker stats

RAW performance

This come with no surprise as it was already proven previously.

  • using charset-normalizer
    • 37.7s with 467 files –> 81ms
  • using chardet
    • 189.3s with 467 files –> 405ms

5 times faster while keeping 88% (410 / 464 files) full backward compatible results.

RAM footprint

Following issue it could be a good idea to showcase that aspect.
Using docker stats for the sampling.

Bellow, the peak usage for each flavor.

  • using charset-normalizer
    • 38MiB
  • using chardet
    • 67MiB

1.7 times less memory consumption. For a 5 times faster guess output (in avg).


Keep in mind that the actual delays may vary depending on your CPU capabilities.
The factor between chardet/charset-normalizer flavors should remain intact.