python - Twitter(X) Filtered Stream (v2 API) - "TooManyConnections" - Stack Overflow

admin管理员组

文章数量:1346681

I have an intermittent issue when streaming the Twitter Filtered Stream endpoint.

I would say once a day, at different times I am hit with the response -

{
  "title": "ConnectionException",
  "detail": "This stream is currently at the maximum allowed connection limit.",
  "connection_issue": "TooManyConnections",
  "type": ";
}

I use the endpoint to stream filtered Tweets based on my published rules, and I believe (according to Twitters somewhat unreliable documentation) that this endpoint is not rate limited based on how many Tweets you download, rather how many times you reconnect to the stream.

Therein lies my puzzlement. The script connecting is the only one I have, i.e. no other scripts in dev or staging. Below is how I connect -

while True:
    try:
        response = requests.get(url, headers=headers, stream=True, timeout=(10, None))

        if response.status_code == 200:
            # Reset retry count on successful connection
            retry_count = 0

            for line in response.iter_lines():
                if line:
                    try:
                        tweet_data = json.loads(line.decode('utf-8'))
                        insert_tweet(tweet_data)
                    except json.JSONDecodeError as e:
                        logging.error(f"JSON Decode Error: {e}")

        elif response.status_code == # Here I handle other response code accordingly.

If I do hit an issue in the response, I use an exponential backoff strategy (jitter) so as to not smash the reconnects -

def backoff_with_jitter(retry_count):
    base_backoff = min(INITIAL_BACKOFF * (2 ** retry_count), MAX_BACKOFF)
    jitter = random.uniform(0, base_backoff)
    time.sleep(jitter)

I also log every time there is a reconnection, and this happens rarely. My question is why am I getting the above error response when, 1. only this script is connecting and 2. the reconnections are sparse?

In my research I did come across this thread on the official forum and this SO question, unfortunately it looks like I am not the only one to experience this. Note: I am using python which I believe handles garbage collection i.e. closes the response before reconnect, and I am on the PRO license for the api access.

本文标签： pythonTwitter(X) Filtered Stream (v2 API)quotTooManyConnectionsquotStack Overflow