twilio export message logs to csv via python

twilio list all messages
twilio exceptions apiexception upgrade required

I'm trying to retrieve a month of message logs from twilio using python. The code below works fine when retrieving a day or two but is extremely slow when expanding the date range. Is there a better way to do this?

import pandas as pd
from datetime import date
from twilio.rest import Client

# Your Account Sid and Auth Token from twilio.com/user/account
account_sid = "####"
auth_token = "####"
client = Client(account_sid, auth_token)


messages = client.messages.list(
    date_sent_after=date(2017, 3, 1),
    date_sent_before=date(2017, 3, 2)
)


d = []
for message in messages:
   d.append((message.from_, message.to, message.body, message.status, 
   message.date_sent, message.price))

print(d)

df = pd.DataFrame(d, columns=('From', 'To', 'Body', 'Status', 'Sent Date', 'Price'))


print(df.dtypes)

df.to_csv('export.csv', index=False, encoding='utf-8')

Twilio's "export to CSV" option on their logs page simply won't be able to export 100s of thousands of results. In order to do this you will need your own script that paginates the results from Twilio's API and generates one CSV.

Here's how I recommend doing that:

import csv
import requests

account_sid = ""
auth_pass = ""

results_per_page = 1000
num_pages = 500
start_date = "2010-04-01"

base_url = "https://api.twilio.com"
initial_page = "/" + start_date + "/Accounts/" + account_sid + "/Messages.json?PageSize=" + str(results_per_page)

response = requests.get(base_url + initial_page, auth=(account_sid, auth_pass)).json()
next_page = response['next_page_uri']

page = 0
data = response['messages']

# print response['previous_page_uri']
# print response

if not next_page == None:
  while page < num_pages:
    response = requests.get(base_url + next_page, auth=(account_sid, auth_pass)).json()

    next_page = response['next_page_uri']

    print "On page: " + str(page)

    data = data + response['messages']
    page += 1

    if next_page == None:
      print "No more pages"
      break

csv_file = open('export.csv', 'w')
csvwriter = csv.writer(csv_file)

count = 0

for message in data:

  if count == 0:
    header = message.keys()
    csvwriter.writerow(header)
    count += 1

  #only include outbound sms because inbound ascii/emojis throws csv writer errors
  if message.values()[2] == "outbound-api":
    csvwriter.writerow(message.values())

csv_file.close()

print "Done!"

You will need to input account_sid and auth_pass with your credentials. Check Twilio API Explorer if unsure what these are. Create a twilio.py file with this code in it and then to run the script in mac terminal: python twilio.py

One known issue that I ran across: if inbound messages contain emojis or strange characters the csv writing throws an error. Fixed this by filtering to only write outbound messages.

If you'd like to improve this script for others here's a link to the github page: https://github.com/benjitastic/twilio-export-messages

Exporting SMS and Call Logs – Twilio Support, First, let's make sure you are in the right place. Are you looking for raw call and sms logs? https://api.twilio.com/2010-04-01/Accounts/AC12345/Calls.csv? To see all your filtering options for exporting logs, see our Calls list documentation and our Messages list documentation. Getting lots of data If you are going to be downloading lots of data from the Twilio API, we highly suggest that you use one of the official Twilio Helper Libraries to do this.

Twilio developer evangelist here.

I have a little trick for you here. You can actually get CSVs straight from Twilio. All you need to do is construct the URL for your data and then add .csv to the end.

Edit

To construct the URL for your messages you should check out the Messages resource.

The list is at the URL:

https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Messages

and to turn it to a CSV, just add .csv

https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Messages.csv

You will need to request the URL with your account SID and auth token as authentication and you can add any query parameters to the end of the URL that you like too.

To do date ranges, like your example of after 01/03/2017 and before 02/03/2017 your URL would look like this:

https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Messages.csv?DateSent>=2017-03-01&DateSent<=2017-03-02

Check out the example of listing messages with curl to see this in more detail.

twilio export message logs to csv via python, twilio export message logs to csv via python - python. Here's how I recommend doing that: import csv import requests account_sid = "" auth_pass  Follow the instructions on screen to download a .csv file of your Twilio message logs. Have a 🍺 (optional) Notes. This is meant to be a quick and easy (no bells and whistles) way for you to download your Twilio message logs. Please help yourself to the code in this repo if you're looking to build your own solution. Find a bug, or want a

If you came here looking on how to pull call logs (instead of message logs), here's an example. You'll notice this is very close to the answers above (and was in fact derived from them).

import csv, json
import requests
account_sid = ''
auth_token = ''

results_per_page = 1000
num_pages = 1000
start_date = '2010-04-01'
base_url = 'https://api.twilio.com'
initial_page = '/' + start_date + '/Accounts/' + account_sid + '/Calls.json?PageSize=' + str(results_per_page)
response = requests.get(base_url + initial_page, auth=(account_sid, auth_token)).json()
next_page = response['next_page_uri']

page = 0
data = response['calls']

# print response['previous_page_uri']
# print response
while page < num_pages and next_page is not None:
    print('On page: ' + str(page))
    response = requests.get(base_url + next_page, auth=(account_sid, auth_token)).json()
    next_page = response['next_page_uri']
    data = data + response['calls']
    page += 1

csv_file = open('twilio-call-logs.csv', 'w')
csvwriter = csv.writer(csv_file)

count = 0
for message in data:
    if count == 0:
        header = message.keys()
        csvwriter.writerow(header)
        count += 1

    csvwriter.writerow(message.values())

csv_file.close()
print('Done!')

[Twilio] Dump a Message log from a specified date till now into a CSV, [Twilio] Dump a Message log from a specified date till now into a CSV. message_list.py. #!/usr/bin/env python with open('log.csv', 'w', newline='\n') as csvfile:. The first thing we need for the above code to work is a Twilio account. Sign up for your free trial account here. We also need an SMS-enabled phone number. You can search for and buy one in the Twilio console. Sending an SMS using Twilio is as simple as making an HTTP POST request to the /Messages resource in the Twilio API.

sendgrid/sendgrid-python: The Official Twilio SendGrid Led , Quickly get started with Docker. This library allows you to quickly and easily use the SendGrid Web API v3 via Python. View all of README.md  Send an SMS message in Python via the REST API. To send an outgoing SMS message from your Twilio account you’ll need to make an HTTP POST to Twilio's Message resource. Twilio's Python library helps you to create a new instance of the Message resource, specifying the To, From, and Body parameters of your message.

Exporting raw data via the API, The file will either be in JSON or CSV format, depending on how you construct the call. To automate this process with a Python Script, see more here. Historical​  export twilio_account_sid = 'your_account_sid' export twilio_auth_token = 'your_auth_token' If you had trouble setting your environment variables or are running Windows, check out this blog post . Now enter this in your terminal to install the Twilio Python library and run your code (from the same directory the file is saved in):

Twilio Node, The most important part of the configuration is the Account SID and Auth Token for the Twilio account that should be used to send the SMS messages. In the  In order to receive incoming messages, without sending an auto-response, the Twilio app should respond with a simple empty Response - <Response></Response> . Read on for more details and instructions to get your Twilio phone number configured for this use case. Use a TwiML Bin to Receive SMS and MMS Messages without Responding.

Comments
  • I was able to tweak this to use it for pulling bulk call logs (it only required a few minor changes). Thank you so much for this answer! It was a lifesaver -- I don't know why you don't have more upvotes. Just knowing I can set PageSize= was very important (I coudn't find most of this in any documentation)
  • Can you give a little more clarity about how to construct this url? I'm not sure how to include a date range. Thanks.
  • Updated my answer with more information about the URL and parameters.
  • this is helpful, but you didn't mention PageSize= which is pretty critical since anyone willing to write code to pull data probably needs more than 50 results at a time....