Now that we have access to the API, we are building a request for the endpoint we use and the parameters we want to pass.

The function defined above contains two pieces:

A. search_url:

What is the link to the endpoint we want to reach.

There are many different endpoints in the Twitter API, here is a list of endpoints that are currently available at the time of this writing:

Some of the early access endpoints that exist at the time of writing this article

Here you will also find a complete list and more information about each endpoint link.

Because this article is aimed at academic researchers who may be trying to take advantage of Twitter’s new product, we use full archive search endpoint.

B. Query parameters:

Parameters provided by the endpoint that allow us to customize the request to be sent.

Each endpoint has different parameters that we can pass on to it, and of course Twitter has an API reference for each in its documentation!

For example, the endpoint search endpoint used in this article can be found in the list of query parameters here. API reference page below “Query Parameters” section.

Example image of query parameters from a screenshot of the entire archive endpoint API documentation

We can break down the above query parameters into a function that we wrote into three parts:

Simple breakdown of parameters from this endpoint

1. The first 4 parameters are the ones we control

'query':        keyword,
'start_time': start_date,
'end_time': end_date,
'max_results': max_results,

2. The next 4 parameters are basically us who tell the endpoint to return more data that is optional and not returned by default.

'expansions':   'author_id,in_reply_to_user_id,geo.place_id',
'tweet.fields': 'id,text,author_id,in_reply_to_user_id,geo,conversation_id,created_at,lang,public_metrics,referenced_tweets,reply_settings,source',
'user.fields': 'id,name,username,created_at,description,public_metrics,verified',
'place.fields': 'full_name,id,country,country_code,geo,name,place_type',

3. The last parameter ‘next_token’ gives the next ‘page’ results. The value used in the parameter is taken directly from the response provided by the API, if there are more results per request than the upper limit, we will talk more about this in this article.

'next_token': {}

Now that we know what the create_url function does, a few important notes:

In the case of a full archive search endpoint, the * query * parameter is the only parameter ** required ** to make a request. Always check the documentation for the endpoint you are using to confirm which parameters MUST exist to avoid problems.

inquiry the parameter is where you put the keywords to search.

Queries can be as simple as searching for tweets that contain the word “xbox”, or as complex as “(xbox europe) OR (xbox usa)”, which returns tweets that contain the words xbox AND europe or xbox AND usa .

The * query * can also be customized using * search operators *. There are so many options to help you narrow down your search results. Hopefully we will discuss operators in more depth in another article. So far, you will find a complete list of survey construction operators here.

Example of a simple query with an operator: “Xbox lang: en”

end of time and start time the format that Twitter uses for timestamps

VVVV-KK-STHTHH: mm: ssZ (ISO 8601 / RFC 3339)

So be sure to convert the dates you want to this format. If you’re not sure how, this is nice timestamp converter it will definitely help.

The number of search results returned by your request is currently limited to 10 to 500.

Now you might ask, how do I get over 500 results then? There * next_token * and paging will come into play!

The answer is simple: If there are more results for the query, Twitter will return a unique one next_token which you can use in your next request and it will give you new results.

If you want to retrieve all the Tweets for your query, just continue using the requests with the new * next_token * service that you receive each time until next_token there is, which means you have downloaded all Tweets!

I hope you are not too confused! But don’t worry, once we perform all the functions we just created, it’s obvious!


Please enter your comment!
Please enter your name here