From 004b8e54b5c43f88ae880870f0fa08b9e5975b2e Mon Sep 17 00:00:00 2001 From: "Kaicheng(Kevin) Yang" Date: Sun, 7 May 2023 23:15:20 -0400 Subject: [PATCH 01/12] Add announcement regarding Twitter API change --- README.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 085e62b..0808aa2 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,14 @@ Behind the scenes, this uses the Botometer's HTTP endpoint, available via RapidAPI usage/account related questions should be posted on RapidAPI discussion. -## [Change Note] +## [Change Note/Announcement] + +### May, 2023 + +Twitter is in the process of suspending free API access for third-party developers. Once the Botometer users lose access to Twitter API V1.1, they can no longer obtain fresh data needed for Botometer Pro API. For the time being, we will keep the Botometer Pro API endpoints alive so that users can analyze data collected before. Users wishing for a refund for the latest billing cycle of the Botometer Pro API can send their requests to RapidAPI (support@rapidapi.com) directly. + +We are actively working on new Botometer Pro API endpoints compatible with Twitter API V2.0 data. The goal is to allow users who can obtain data from Twitter (through purchase or other approaches) to continue using Botometer API for analysis. Please stay tuned for the updates. + ### September, 2021 From 37eb07b2ba8b7bac5a3736fc59c4030265995c91 Mon Sep 17 00:00:00 2001 From: yang3kc Date: Mon, 17 Jun 2024 10:24:53 -0400 Subject: [PATCH 02/12] remove old readme --- README.md | 274 +++++------------------------------------------------- 1 file changed, 25 insertions(+), 249 deletions(-) diff --git a/README.md b/README.md index 0808aa2..07469b5 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ -# Botometer Python API +# Botometer X Python API -A Python API for [Botometer by OSoMe](https://osome.iu.edu). +A Python API for [Botometer X by OSoMe](https://osome.iu.edu). Previously known as `botornot-python`. Behind the scenes, this uses the Botometer's HTTP endpoint, available via @@ -10,304 +10,85 @@ RapidAPI usage/account related questions should be posted on RapidAPI discussion ## [Change Note/Announcement] -### May, 2023 +### June, 2024 -Twitter is in the process of suspending free API access for third-party developers. Once the Botometer users lose access to Twitter API V1.1, they can no longer obtain fresh data needed for Botometer Pro API. For the time being, we will keep the Botometer Pro API endpoints alive so that users can analyze data collected before. Users wishing for a refund for the latest billing cycle of the Botometer Pro API can send their requests to RapidAPI (support@rapidapi.com) directly. +We are releasing a new API endpoint for Botometer X. -We are actively working on new Botometer Pro API endpoints compatible with Twitter API V2.0 data. The goal is to allow users who can obtain data from Twitter (through purchase or other approaches) to continue using Botometer API for analysis. Please stay tuned for the updates. +Unlike the original Botometer that fetched data from Twitter and calculated bot scores on the fly, Botometer X is in archival mode and relies on pre-calculated scores based on historical data collected before June 2023. +The API endpoint allows users to fetch scores in bulk using a list of user ids or screen names, without the need of a Twitter/X's developer account. +For details of Botometer X, please refer to the [FQA](https://botometer.osome.iu.edu/faq). -### September, 2021 - -We have a minor update for botometer-python. - -Recently Tweepy had a major upgrade (v3.10.0 -> v4.0.0) that is not back compatible. -Upgrading your local Tweepy may break botometer-python and throw "No module named 'tweepy.error'" error. -To fix it, simply downgrade Tweepy to previous versions such as v3.10.0. - -We upgraded botometer-python to v1.6.1 to specify the acceptable versions of Tweepy. -There is nothing you need to do if you already have the correct version of Tweepy. - -### September, 2020 - -We have a major update for Botometer: - -1. Botometer has been upgraded to V4, and you can use the `/4/check_account` endpiont to access it. -2. The response of `/4/check_account` is reorganized. -3. A new endpoint for BotometerLite is added. It allows checking accounts in bulk. - -You can see the full [announcement](https://cnets.indiana.edu/blog/2020/09/01/botometer-v4/) for details. - -Due to the update, please upgrade `botometer-python` in your local environment to the newest version. -You may also need to modify your code to adapt to the new response from the API. -For more information, check out the documentation below. -If you want to try the new BotometerLite API, checkout the documentation below. - -### May, 2020 - -We have made some changes to our API, please read the [announcement](https://twitter.com/Botometer/status/1250557098708144131) for details. Due to the API change, the old `botometer-python` package might stop to work. Please upgrade it in your local environment to the newest version. ## Help -> You probably want to have a look at [Troubleshooting & FAQ](https://github.com/IUNetSci/botometer-python/wiki/Troubleshooting-&-FAQ) in the wiki. Please feel free to suggest and/or contribute improvements to that page. +> You probably want to have a look at [Troubleshooting & FAQ](https://github.com/osome-iu/botometer-python/wiki/Troubleshooting-&-FAQ) in the wiki. Please feel free to suggest and/or contribute improvements to that page. ## Prior to Utilizing Botometer -To begin using Botometer, you must follow the steps below before running any code: +To begin using Botometer X, you must follow the steps below before running any code: 1. Create a free [RapidAPI](https://rapidapi.com/) account. 2. Subscribe to [Botometer Pro](https://rapidapi.com/OSoMe/api/botometer-pro) on RapidApi by selecting a plan. > There is a completely free version (which does not require any credit card information) for testing purposes. -3. Create a Twitter application via https://developer.twitter.com/ - > Botometer utilizes the access credentials provided by Twitter for the application. -4. Ensure Botometer Pro's dependencies are already installed. +3. Ensure Botometer Pro's dependencies are already installed. > See the [Dependencies](#dependencies) section for details. -**Note:** These steps are necessary to access credentials and download other packages which are needed for Botometer to work properly. Please see [RapidAPI and Twitter Access Details](#access) below for more details on this topic. - ## Quickstart -From your command shell, run +From your command shell, run ``` pip install botometer ``` -### Botometer-V4 - -To access the Botometer-V4 API, enter something like this in a Python shell or script: - -```python -import botometer - -rapidapi_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" -twitter_app_auth = { - 'consumer_key': 'xxxxxxxx', - 'consumer_secret': 'xxxxxxxxxx', - 'access_token': 'xxxxxxxxx', - 'access_token_secret': 'xxxxxxxxxxx', - } -bom = botometer.Botometer(wait_on_ratelimit=True, - rapidapi_key=rapidapi_key, - **twitter_app_auth) - -# Check a single account by screen name -result = bom.check_account('@clayadavis') - -# Check a single account by id -result = bom.check_account(1548959833) - -# Check a sequence of accounts -accounts = ['@clayadavis', '@onurvarol', '@jabawack'] -for screen_name, result in bom.check_accounts_in(accounts): - # Do stuff with `screen_name` and `result` -``` - -Result: -```json -{ - "cap": { - "english": 0.8018818614025648, - "universal": 0.5557322218336633 - }, - "display_scores": { - "english": { - "astroturf": 0.0, - "fake_follower": 4.1, - "financial": 1.5, - "other": 4.7, - "overall": 4.7, - "self_declared": 3.2, - "spammer": 2.8 - }, - "universal": { - "astroturf": 0.3, - "fake_follower": 3.2, - "financial": 1.6, - "other": 3.8, - "overall": 3.8, - "self_declared": 3.7, - "spammer": 2.3 - } - }, - "raw_scores": { - "english": { - "astroturf": 0.0, - "fake_follower": 0.81, - "financial": 0.3, - "other": 0.94, - "overall": 0.94, - "self_declared": 0.63, - "spammer": 0.57 - }, - "universal": { - "astroturf": 0.06, - "fake_follower": 0.64, - "financial": 0.3133333333333333, - "other": 0.76, - "overall": 0.76, - "self_declared": 0.74, - "spammer": 0.47 - } - }, - "user": { - "majority_lang": "en", - "user_data": { - "id_str": "11330", - "screen_name": "test_screen_name" - } - } -} -``` - -Meanings of the elements in the response: - -* **user**: Twitter user object (from the user) plus the language inferred from majority of tweets -* **raw scores**: bot score in the [0,1] range, both using English (all features) and Universal (language-independent) features; in each case we have the overall score and the sub-scores for each bot class (see below for subclass names and definitions) -* **display scores**: same as raw scores, but in the [0,5] range -* **cap**: conditional probability that accounts with a score **equal to or greater than this** are automated; based on inferred language - -Meanings of the bot type scores: - -* `fake_follower`: bots purchased to increase follower counts -* `self_declared`: bots from botwiki.org -* `astroturf`: manually labeled political bots and accounts involved in follow trains that systematically delete content -* `spammer`: accounts labeled as spambots from several datasets -* `financial
`: bots that post using cashtags -* `other`: miscellaneous other bots obtained from manual annotation, user feedback, etc. - -For more information on the response object, consult the [API Overview](https://rapidapi.com/OSoMe/api/botometer-pro/details) on RapidAPI. - -### BotometerLite - -In September, 2020, the BotometerLite endpoint was added. It leverages a lightweighted model and allows detecting likely bots in bulk. -Before accessing it, please make sure you have subscribed to the ULTRA plan on RapidAPI. -Unlike Botometer-V4, BotometerLite just needs the user profile information and the timestamp of when the information was collected to perform bot detection. -There are two modes for BotometerLite: non-Twitter mode and Twitter mode. +### Botometer X -If you have already collected at least one tweet for each account you want to check, you can use the non-Twitter mode. -In this mode, you only need a RapidAPI key. +To access the Botometer X endpoint, enter something like this in a Python shell or script: ```python import botometer rapidapi_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" -blt = botometer.BotometerLite(rapidapi_key=rapidapi_key) -# Prepare a list of tweets from the users that you want to perform bot detection on. -# The list should contain no more than 100 tweets. -tweet_list = [tweet1, tweet2, ...] +bomx = botometer.BotometerX(rapidapi_key=rapidapi_key) -blt_scores = blt.check_accounts_from_tweets(tweet_list) -``` +# Check accounts by screen names -Result: +# Check accounts by ids -```json -[ - {"botscore": 0.65, "tweet_id": "1234", "user_id": 1111}, - {"botscore": 0.29, "tweet_id": "12345", "user_id": 2222} -] +# Check accounts by both screen names and ids ``` -Note that the tweet_id is also included in case multiple tweets from the same user are passed to the API. - -If you only have a set of user_ids or screen_names, you will have to use the Twitter mode. -In addition to the RapidAPI key, this mode also requires a valid Twitter APP key. -The package would first query the Twitter user lookup API to fetch the user profiles, then pass the data to the Botometer Pro API -for the bot scores. - -```python -import botometer - -rapidapi_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" -twitter_app_auth = { - 'consumer_key': 'xxxxxxxx', - 'consumer_secret': 'xxxxxxxxxx', - 'access_token': 'xxxxxxxxx', - 'access_token_secret': 'xxxxxxxxxxx', - } - -blt_twitter = botometer.BotometerLite(rapidapi_key=rapidapi_key, **twitter_app_auth) - -# Prepare a list of screen_names you want to check. -# The list should contain no more than 100 screen_names; please remove the @ -screen_name_list = ['yang3kc', 'onurvarol', 'clayadavis'] -blt_scores = blt_twitter.check_accounts_from_screen_names(screen_name_list) - -# Prepare a list of user_ids you want to check. -# The list should contain no more than 100 user_ids. -user_id_list = [1133069780917850112, 77436536, 1548959833] -blt_scores = blt_twitter.check_accounts_from_user_ids(user_id_list) -``` Result: ```json -[ - {"botscore": 0.17, "tweet_id": null, "user_id": 1133069780917850112}, - {"botscore": 0.2, "tweet_id": null, "user_id": 77436536}, - {"botscore": 0.16, "tweet_id": null, "user_id": 1548959833} -] +{} ``` -The tweet_id is set to null in this mode. +Meanings of the elements in the response: -Note that in the non-Twitter mode, the returned scores reflect the status of the accounts when the tweets were collected. -In the Twitter mode, on the other hand, the scores reflect the status of the accounts when you run the code, just like the Botometer-V4 endpoint. +For more information on the response object, consult the [API Overview](https://rapidapi.com/OSoMe/api/botometer-pro/details) on RapidAPI. + + +## Installation instructions -## Install instructions This package is on PyPI so you can install it with pip: ``` $ pip install botometer ``` - ## Dependencies ### Python dependencies * [requests](http://docs.python-requests.org/en/latest/) -* [tweepy](https://github.com/tweepy/tweepy) Both of these dependencies are available via `pip`, so you can install both at once with pip install requests tweepy - -## RapidAPI and Twitter Access Details - -### RapidAPI key - -Our API is served via [RapidAPI](//rapidapi.com). You must sign up -for a free account in order to obtain a RapidAPI secret key. The easiest way to -get your secret key is to visit -[our API endpoint page](https://rapidapi.com/OSoMe/api/botometer-pro/endpoints) -and look in the endpoint's header parametsrs for the "X-RapidAPI-Key" as shown below: - -![Screenshot of RapidAPI header parameters](/docs/rapidapi_key.png) - -### Twitter app -In order to access Twitter's API, one needs to have/create a [Twitter app](https://apps.twitter.com/). -Once you've created an app, the authentication info can be found in the "Keys and Access Tokens" tab of the app's properties: -![Screenshot of app "Keys and Access Tokens"](/docs/twitter_app_keys.png) - -## Authentication -By default, Botometer uses **user authentication** when interacting with Twitter's API as it is the least restrictive and the ratelimit matches with Botometer's **Pro** plan: 180 requests per 15-minute window. -One can instead use Twitter's **application authentication** in order to take advantage of the higher ratelimit that matches our **Ultra** plan: 450 requests per 15-minute window. Do note the differences between user and app-only authentication found under the header "Twitter API Authentication Model" in [Twitter's docs on authentication](https://developer.twitter.com/en/docs/basics/authentication/overview/oauth). - -To use app-only auth, just omit the `access_token` and `access_token_secret` in the `Botometer` constructor. - -```python -import botometer - -rapidapi_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" # now it's called rapidapi key -twitter_app_auth = { - 'consumer_key': 'xxxxxxxx', - 'consumer_secret': 'xxxxxxxxxx' - } -bom = botometer.Botometer(wait_on_ratelimit=True, - rapidapi_key=rapidapi_key, - **twitter_app_auth) -``` ## References +- **Botometer X:** The scores are calculated using the BotometerLite model. + - ***Botometer v4:*** Mohsen Sayyadiharikandeh, Onur Varol, Kai-Cheng Yang, Alessandro Flammini, Filippo Menczer. "Detection of Novel Social Bots by Ensembles of Specialized Classifiers." [DOI](https://doi.org/10.1145/3340531.3412698), [ArXiv](https://arxiv.org/abs/2006.06867) - ***BotometerLite:*** Yang, K.; Varol, O.; Hui, P.; and Menczer, F. "Scalable and Generalizable Social Bot Detection through Data Selection." AAAI (2020). [DOI](http://doi.org/10.1609/aaai.v34i01.5460), [ArXiv](https://arxiv.org/abs/1911.09179) @@ -321,8 +102,3 @@ bom = botometer.Botometer(wait_on_ratelimit=True, - Varol O., Davis C., Menczer, F., Flammini, A. "Feature Engineering for Social Bot Detection", Feature Engineering for Machine Learning and Data Analytics [Google Books](https://books.google.com/books?id=661SDwAAQBAJ&lpg=PA311&dq=info%3AsM983rg_yb8J%3Ascholar.google.com&lr&pg=PA311#v=onepage&q&f=false) - Ferrara, Emilio, Onur Varol, Clayton Davis, Filippo Menczer, and Alessandro Flammini. "The rise of social bots." Communications of the ACM 59, no. 7 (2016): 96-104. [DOI](https://doi.org/10.1145/2818717), [ArXiv](https://arxiv.org/abs/1407.5225) - - -```python - -``` From 846305b3a3eaf4cf304d3d92ba62ef41502415ac Mon Sep 17 00:00:00 2001 From: yang3kc Date: Mon, 17 Jun 2024 21:28:31 -0400 Subject: [PATCH 03/12] remove old classes --- botometer/__init__.py | 289 ------------------------------------------ 1 file changed, 289 deletions(-) diff --git a/botometer/__init__.py b/botometer/__init__.py index 488f813..e69de29 100644 --- a/botometer/__init__.py +++ b/botometer/__init__.py @@ -1,289 +0,0 @@ -from __future__ import print_function -import time - -import requests -from requests import ConnectionError, HTTPError, Timeout -import tweepy -from tweepy.error import RateLimitError, TweepError -import datetime - - -class NoTimelineError(ValueError): - def __init__(self, sn, *args, **kwargs): - msg = "user '%s' has no tweets in timeline" % sn - super(NoTimelineError, self).__init__(msg, *args, **kwargs) - - -class Botometer(object): - _TWITTER_RL_MSG = 'Rate limit exceeded for Twitter API method' - - def __init__(self, - consumer_key, consumer_secret, - access_token=None, access_token_secret=None, - rapidapi_key=None, - tweepy_kwargs={}, - **kwargs): - self.consumer_key = consumer_key - self.consumer_secret = consumer_secret - self.access_token_key = self.access_token = access_token - self.access_token_secret = access_token_secret - self.wait_on_ratelimit = kwargs.get('wait_on_ratelimit', False) - - self.rapidapi_key = rapidapi_key or kwargs.get('mashape_key') - - if self.access_token_key is None or self.access_token_secret is None: - auth = tweepy.AppAuthHandler( - self.consumer_key, self.consumer_secret) - else: - auth = tweepy.OAuthHandler( - self.consumer_key, self.consumer_secret) - auth.set_access_token( - self.access_token_key, self.access_token_secret) - - self.twitter_api = tweepy.API( - auth, - parser=tweepy.parsers.JSONParser(), - wait_on_rate_limit=self.wait_on_ratelimit, - **tweepy_kwargs) - - self.api_url = kwargs.get('botometer_api_url', - 'https://botometer-pro.p.rapidapi.com') - self.api_version = kwargs.get('botometer_api_version', 4) - - @classmethod - def create_from(cls, instance, **kwargs): - my_kwargs = vars(instance) - my_kwargs.update(kwargs) - return cls(**my_kwargs) - - - def _add_rapidapi_header(self, kwargs): - if self.rapidapi_key: - kwargs.setdefault('headers', {}).update({ - 'x-rapidapi-key': self.rapidapi_key - }) - return kwargs - - def _bom_get(self, *args, **kwargs): - self._add_rapidapi_header(kwargs) - return requests.get(*args, **kwargs) - - def _bom_post(self, *args, **kwargs): - self._add_rapidapi_header(kwargs) - return requests.post(*args, **kwargs) - - def _get_twitter_data(self, user, full_user_object=False): - try: - user_timeline = self.twitter_api.user_timeline( - user, - include_rts=True, - count=200, - ) - - except RateLimitError as e: - e.args = (self._TWITTER_RL_MSG, 'statuses/user_timeline') - raise e - - if user_timeline: - user_data = user_timeline[0]['user'] - else: - user_data = self.twitter_api.get_user(user) - screen_name = '@' + user_data['screen_name'] - - try: - search = self.twitter_api.search(screen_name, count=100) - except RateLimitError as e: - e.args = (self._TWITTER_RL_MSG, 'search/tweets') - raise e - - payload = { - 'mentions': search['statuses'], - 'timeline': user_timeline, - 'user': user_data, - } - - if not full_user_object: - payload['user'] = { - 'id_str': user_data['id_str'], - 'screen_name': user_data['screen_name'], - } - - return payload - - - #################### - ## Public methods ## - #################### - - def bom_api_path(self, method=''): - return '/'.join([ - self.api_url.rstrip('/'), - str(self.api_version), - method, - ]) - - - def check_account(self, user, full_user_object=False): - payload = self._get_twitter_data(user, - full_user_object=full_user_object) - if not payload['timeline']: - raise NoTimelineError(payload['user']) - - url = self.bom_api_path('check_account') - bom_resp = self._bom_post(url, json=payload) - bom_resp.raise_for_status() - classification = bom_resp.json() - - return classification - - - def check_accounts_in(self, accounts, full_user_object=False, - on_error=None, **kwargs): - - sub_instance = self.create_from(self, wait_on_ratelimit=True, - botometer_api_url=self.api_url) - max_retries = kwargs.get('retries', 3) - - for account in accounts: - for num_retries in range(max_retries + 1): - result = None - try: - result = sub_instance.check_account( - account, full_user_object=full_user_object) - except (TweepError, NoTimelineError) as e: - err_msg = '{}: {}'.format( - type(e).__name__, - getattr(e, 'msg', '') or getattr(e, 'reason', ''), - ) - result = {'error': err_msg} - except (ConnectionError, HTTPError, Timeout) as e: - if num_retries >= max_retries: - raise - else: - time.sleep(2 ** num_retries) - except Exception as e: - if num_retries >= max_retries: - if on_error: - on_error(account, e) - else: - raise - - if result is not None: - yield account, result - break - - -class BotometerLite(Botometer): - """ - Class to interact with the BotometerLite API. - The present class has two modes: the Twitter mode and the non-Twitter mode. - - Non-Twitter mode: - No Twitter APP key is required in this mode, i.e., only rapidapi_key is required. - The users provide a list of no more than 100 tweets that they want to perform - bot detection on. - Via the self.check_accounts_from_tweets method, the present class - queries the RapidAPI to fetch the botscores. - - Twitter mode: - A valid Tiwtter APP key is required in this mode, i.e., consumer_key, consumer_secret, - and rapidapi_key are required. - The users provide a list of no more than 100 user_ids or screen_names. - Via the self.check_accounts_from_user_ids or check_accounts_from_screen_names method, - the the present class first queries the Twitter user lookup API to fetch the user profiles, - then queries the RapidAPI to fetch the botscore. - - Lists of tweets/user_ids/screen_names with more than 100 elements would be truncated to 100. - The users are responsible to handle the exceptions. - """ - TWEETS_PER_REQUEST = 100 - def __init__( - self, - rapidapi_key=None, - consumer_key=None, - consumer_secret=None, - access_token=None, - access_token_secret=None, - **kwargs - ): - self.twitter_mode = False - lite_api_url = kwargs.get( - 'botometerlite_api_url', - 'https://botometer-pro.p.rapidapi.com' - ) - if consumer_key is None or consumer_secret is None: - # No Twitter mode: the users provide the tweets - self.api_url = lite_api_url - else: - # Twitter mode: use Twitter lookup to fetch user profile - self.twitter_mode = True - super(BotometerLite, self).__init__( - consumer_key, - consumer_secret, - access_token=access_token, - access_token_secret=access_token_secret, - botometer_api_url=lite_api_url, - wait_on_ratelimit=kwargs.get('wait_on_ratelimit', False) - ) - - self.api_version = kwargs.get('botometerlite_api_version', 'litev1') - self.rapidapi_key = rapidapi_key or kwargs.get('mashape_key') - - def _get_utc_now(self): - now_time = datetime.datetime.now(datetime.timezone.utc) - return datetime.datetime.strftime( - now_time, '%a %b %d %H:%M:%S %z %Y' - ) - - def _get_twitter_data(self, **kwargs): - try: - user_objs = self.twitter_api.lookup_users(**kwargs) - except RateLimitError as e: - e.args = (self._TWITTER_RL_MSG, 'users/lookup') - raise e - return user_objs - - def _check_accounts_twitter_mode(self, query, **kwargs): - assert self.twitter_mode, "Twitter app key missing" - - user_objs = self._get_twitter_data(**query) - now_str = self._get_utc_now() - - dummy_tweets = [] - for user_obj in user_objs: - dummy_tweets.append({ - 'created_at': now_str, - 'user': user_obj - }) - return self.check_accounts_from_tweets(dummy_tweets) - - def check_accounts_from_user_ids(self, user_ids, **kwargs): - """ - Check accounts based on a list of user_ids. - The list should contain no more than 100 elements. - A valid Twitter APP key is required. - """ - query = {"user_ids": list(user_ids)[:self.TWEETS_PER_REQUEST]} - return self._check_accounts_twitter_mode(query, **kwargs) - - def check_accounts_from_screen_names(self, screen_names, **kwargs): - """ - Check accounts based on a list of screen_names (please remove the @). - The list should contain no more than 100 elements. - A valid Twitter APP key is required. - """ - query = {"screen_names": list(screen_names)[:self.TWEETS_PER_REQUEST]} - return self._check_accounts_twitter_mode(query, **kwargs) - - def check_accounts_from_tweets(self, tweets): - """ - Check accounts based on a list of tweets. - The list should contain no more than 100 elements. - No Twitter APP key is required. - """ - url = self.bom_api_path('check_accounts_in_bulk') - bom_resp = self._bom_post(url, json=tweets) - bom_resp.raise_for_status() - classification = bom_resp.json() - - return classification From 2fd47609cebff03946a09a2ccbab558a5c802d3e Mon Sep 17 00:00:00 2001 From: yang3kc Date: Mon, 24 Jun 2024 14:01:46 -0400 Subject: [PATCH 04/12] update setup.py --- setup.py | 25 +++++++++++-------------- 1 file changed, 11 insertions(+), 14 deletions(-) diff --git a/setup.py b/setup.py index a6064fb..204ad99 100644 --- a/setup.py +++ b/setup.py @@ -1,16 +1,13 @@ from setuptools import setup -setup(name='botometer', - version='1.6.1', - description='Check Twitter accounts for bot behavior', - url='https://github.com/IUNetSci/botometer-python', - download_url='https://github.com/IUNetSci/botometer-python/archive/1.0.zip', - author='Clayton A Davis, Kai-Cheng Yang', - author_email='claydavi@indiana.edu,yangkc@iu.edu', - license='MIT', - packages=['botometer'], - install_requires=[ - 'requests', - 'tweepy >= 3.5.0,<4', - ], - ) +setup( + name="botometer", + version="2.0.0", + description="Check Twitter accounts for bot behavior", + url="https://github.com/osome-iu/botometer-python", + author="Kai-Cheng Yang", + author_email="yang3kc@gmail.com", + license="MIT", + packages=["botometer"], + install_requires=["requests"], +) From f813d9e1812dce10c543e25307cdf548e6229166 Mon Sep 17 00:00:00 2001 From: yang3kc Date: Mon, 24 Jun 2024 14:01:56 -0400 Subject: [PATCH 05/12] add code for BotometerX --- botometer/__init__.py | 98 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+) diff --git a/botometer/__init__.py b/botometer/__init__.py index e69de29..6eb84c1 100644 --- a/botometer/__init__.py +++ b/botometer/__init__.py @@ -0,0 +1,98 @@ +import requests + + +class BotometerBase(object): + def __init__(self, rapidapi_key, **kwargs): + self.rapidapi_key = rapidapi_key + + self.api_url = kwargs.get( + "botometer_api_url", "https://botometer-pro.p.rapidapi.com" + ) + + def _add_rapidapi_header(self, kwargs): + if self.rapidapi_key: + kwargs.setdefault("headers", {}).update( + {"x-rapidapi-key": self.rapidapi_key} + ) + + return kwargs + + def _bom_get(self, *args, **kwargs): + self._add_rapidapi_header(kwargs) + return requests.get(*args, **kwargs) + + def _bom_post(self, *args, **kwargs): + self._add_rapidapi_header(kwargs) + return requests.post(*args, **kwargs) + + #################### + ## Public methods ## + #################### + + def bom_api_path(self, method=""): + return "/".join( + [ + self.api_url.rstrip("/"), + str(self.api_version), + method, + ] + ) + + +class BotometerX(BotometerBase): + """ + Class to interact with the Botometer X API endpoint. + + Lists of user_ids and/or screen_names with more than 100 elements would be truncated to 100. + Users are responsible to handle the exceptions. + """ + + TWEETS_PER_REQUEST = 100 + + def __init__(self, rapidapi_key, **kwargs): + super(BotometerX, self).__init__(rapidapi_key, **kwargs) + + self.api_version = "botometer-x" + + def _is_list_of_type(self, list_to_check, type_to_check): + if isinstance(list_to_check, list): + return all(isinstance(item, type_to_check) for item in list_to_check) + return False + + def get_botscores_in_batch(self, user_ids=None, usernames=None): + """ + Get botscores based on a list of user_ids and/or screen_names. + There should be no more than 100 accounts in the query. + """ + # Assign default values if not provided + user_ids = [] if user_ids is None else user_ids + usernames = [] if usernames is None else usernames + + if not self._is_list_of_type(user_ids, int): + raise ValueError("user_ids must be a list of integers") + + if not self._is_list_of_type(usernames, str): + raise ValueError("usernames must be a list of strings") + + if len(user_ids) == 0 and len(usernames) == 0: + raise ValueError("Must provide either user_ids or usernames") + + N_BOTSCORES_PER_QUERY = 100 + + # Will only query the first N_BOTS_PER_QUERY items + if len(user_ids) > N_BOTSCORES_PER_QUERY: + # We have enough user ids, so we will query the first N_BOTS_PER_QUERY and ignore the usernames + user_ids = user_ids[:N_BOTSCORES_PER_QUERY] + usernames = [] + else: + # We will query all the user ids plus N_BOTS_PER_QUERY - len(user_ids) usernames + usernames = usernames[: N_BOTSCORES_PER_QUERY - len(user_ids)] + payload = { + "user_ids": user_ids, + "usernames": usernames, + } + + url = self.bom_api_path("get_botscores_in_batch") + bom_resp = self._bom_post(url, json=payload) + bom_resp.raise_for_status() + return bom_resp.json() From 1c07bd21d00efa86fd2b51580807eead906a6a83 Mon Sep 17 00:00:00 2001 From: yang3kc Date: Mon, 24 Jun 2024 14:18:08 -0400 Subject: [PATCH 06/12] update readme with example code and output --- README.md | 42 +++++++++++++++++++++++++++++++----------- 1 file changed, 31 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index 07469b5..4c89f25 100644 --- a/README.md +++ b/README.md @@ -48,23 +48,46 @@ import botometer rapidapi_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" bomx = botometer.BotometerX(rapidapi_key=rapidapi_key) +``` + -# Check accounts by screen names +```python +# Check accounts by usernames, note that @ is optional +bomx.get_botscores_in_batch(usernames=['@OSoMe_IU', 'botometer']) # Check accounts by ids +bomx.get_botscores_in_batch(user_ids=[2451308594, 187521608]) -# Check accounts by both screen names and ids +# Check accounts by both usernames and ids +bomx.get_botscores_in_batch(usernames=['@OSoMe_IU'], user_ids=[2451308594]) ``` +The queries will return results like those below: -Result: ```json -{} +[ + { + 'bot_score': 0.09, + 'timestamp': 'Sat, 27 May 2023 23:57:16 GMT', + 'user_id': '2451308594', + 'username': 'Botometer' + }, + { + 'bot_score': 0.21, + 'timestamp': 'Thu, 25 May 2023 22:54:53 GMT', + 'user_id': '187521608', + 'username': 'OSoMe_IU' + } +] ``` +The response will be a list of JSON objects. +Meanings of the elements in the object: +- `bot_score`: The bot score, a float number between 0 and 1 +- `timestamp`: The time when the bot score was calculated +- `user_id`: ID of the account +- `username`: Username of the account -Meanings of the elements in the response: - -For more information on the response object, consult the [API Overview](https://rapidapi.com/OSoMe/api/botometer-pro/details) on RapidAPI. +For more information on the API, consult the [API Overview](https://rapidapi.com/OSoMe/api/botometer-pro/details) on RapidAPI. ## Installation instructions @@ -80,10 +103,7 @@ $ pip install botometer ### Python dependencies * [requests](http://docs.python-requests.org/en/latest/) -Both of these dependencies are available via `pip`, so you can install both at once with - - pip install requests tweepy - +The dependency should be installed automatically with pip. ## References From 5fe7e89c051d953698623a364c93ab5623153ecd Mon Sep 17 00:00:00 2001 From: yang3kc Date: Mon, 24 Jun 2024 14:27:11 -0400 Subject: [PATCH 07/12] fix JSON format --- README.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 4c89f25..2a24998 100644 --- a/README.md +++ b/README.md @@ -67,16 +67,16 @@ The queries will return results like those below: ```json [ { - 'bot_score': 0.09, - 'timestamp': 'Sat, 27 May 2023 23:57:16 GMT', - 'user_id': '2451308594', - 'username': 'Botometer' + "bot_score": 0.09, + "timestamp": "Sat, 27 May 2023 23:57:16 GMT", + "user_id": "2451308594", + "username": "Botometer" }, { - 'bot_score': 0.21, - 'timestamp': 'Thu, 25 May 2023 22:54:53 GMT', - 'user_id': '187521608', - 'username': 'OSoMe_IU' + "bot_score": 0.21, + "timestamp": "Thu, 25 May 2023 22:54:53 GMT", + "user_id": "187521608", + "username": "OSoMe_IU" } ] ``` From 7c921764af53b9332ace68c9f0d474094711242b Mon Sep 17 00:00:00 2001 From: yang3kc Date: Mon, 24 Jun 2024 18:43:37 -0400 Subject: [PATCH 08/12] Add support for list of string ids --- botometer/__init__.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/botometer/__init__.py b/botometer/__init__.py index 6eb84c1..095589d 100644 --- a/botometer/__init__.py +++ b/botometer/__init__.py @@ -68,8 +68,10 @@ def get_botscores_in_batch(self, user_ids=None, usernames=None): user_ids = [] if user_ids is None else user_ids usernames = [] if usernames is None else usernames - if not self._is_list_of_type(user_ids, int): - raise ValueError("user_ids must be a list of integers") + if not self._is_list_of_type(user_ids, int) and not self._is_list_of_type( + user_ids, str + ): + raise ValueError("user_ids must be a list of integers or strings") if not self._is_list_of_type(usernames, str): raise ValueError("usernames must be a list of strings") From 8419d878e7078f1fdb8c2bc1bc8c2f4dcb4dbc02 Mon Sep 17 00:00:00 2001 From: yang3kc Date: Mon, 24 Jun 2024 18:43:43 -0400 Subject: [PATCH 09/12] bump version --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 204ad99..a49ab1b 100644 --- a/setup.py +++ b/setup.py @@ -2,7 +2,7 @@ setup( name="botometer", - version="2.0.0", + version="2.0.1", description="Check Twitter accounts for bot behavior", url="https://github.com/osome-iu/botometer-python", author="Kai-Cheng Yang", From d9b01431924ee5f81ad56ed4effd81ab750ee88c Mon Sep 17 00:00:00 2001 From: "Kaicheng(Kevin) Yang" Date: Tue, 25 Jun 2024 10:56:26 -0400 Subject: [PATCH 10/12] add version badge --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 2a24998..d53d144 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,5 @@ +[![PyPI version](https://badge.fury.io/py/botometer.svg)](https://badge.fury.io/py/botometer) + # Botometer X Python API A Python API for [Botometer X by OSoMe](https://osome.iu.edu). From 9cbb46810caed74d0a7d7a78fbb75d01d4d7f806 Mon Sep 17 00:00:00 2001 From: "Kaicheng(Kevin) Yang" Date: Wed, 26 Jun 2024 09:32:42 -0400 Subject: [PATCH 11/12] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d53d144..9359c0f 100644 --- a/README.md +++ b/README.md @@ -84,7 +84,7 @@ The queries will return results like those below: ``` The response will be a list of JSON objects. Meanings of the elements in the object: -- `bot_score`: The bot score, a float number between 0 and 1 +- `bot_score`: The bot score, a float number between 0 and 1 (note that we rescale the score to 1 to 5 on the website) - `timestamp`: The time when the bot score was calculated - `user_id`: ID of the account - `username`: Username of the account From e25972b01b99cd68b1fba4a8063f54a4657bd044 Mon Sep 17 00:00:00 2001 From: "Kaicheng(Kevin) Yang" Date: Wed, 26 Jun 2024 09:33:08 -0400 Subject: [PATCH 12/12] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 9359c0f..0daff98 100644 --- a/README.md +++ b/README.md @@ -84,7 +84,7 @@ The queries will return results like those below: ``` The response will be a list of JSON objects. Meanings of the elements in the object: -- `bot_score`: The bot score, a float number between 0 and 1 (note that we rescale the score to 1 to 5 on the website) +- `bot_score`: The bot score, a float number between 0 and 1 (note that we rescale the score to 1 to 5 on the Botometer X website) - `timestamp`: The time when the bot score was calculated - `user_id`: ID of the account - `username`: Username of the account