Tutorial: how to retrieve large datasets with the Messages API and node.js
Messages API intro and overview #
The Messages API is a REST API that allows you to get details about any outbound or inbound messages that were processed through your Postmark server. The Messages API also supports search functionality through the use of query parameters. However, each search can only return up to 10,000 messages. If you need to retrieve message details for more than 10,000 messages, you will need to use search parameters to narrow down the results.
This tutorial gives you two examples for breaking down larger sets of messages into chunks of less than 10,000 using node.js. The results will be exported as a CSV file that can be used for further searching and auditing. After completing this tutorial, you can use the provided code to write your own script to export outbound or inbound email message data from Postmark using the Messages API.
Important consideration: The Messages API is useful for retrieving data about messages that have already been sent. If you know that you will always want details about every single message that processes through your Postmark server (for auditing purposes for example), we highly recommend using webhooks. Webhooks also offer the advantage of providing the data in real time, rather than relying on querying the API after the fact to check for new messages.
Tutorial prerequisites #
Before getting started, you will need the following:
- A Postmark account and Server
To retrieve data from the Messages API, you will also need to process outbound or inbound messages through your Server. This can also be done in Sandbox mode if you don’t already have existing data.
Create the node.js app and install dependencies #
The first step is to create a new node.js app and install the necessary dependencies. This tutorial only requires two dependencies: axios for making HTTP calls, and csv to save the resulting data as a .csv file.
In the terminal of your choice, navigate to the directory where you will create the app, or create a new folder and navigate to it.
$ mkdir postmark-tutorial && cd postmark-tutorialOnce you have navigated to your project directory, use the following command to create a new node.js app:
$ npm initThis command will initiate the creation of a new node.js app. There will be a short series of questions prompted from this command. Once you have completed the short series of prompts, a new package.json file will be created in your project directory.
Once your package.json file is created, install the two dependencies:
$ npm install --save axios csvThis command will install the axios and csv dependencies and add them to your package.json file.
Retrieving messages by tag #
Now that the package.json file has been created, you are ready to write some code! In the same directory as your package.json file, create a new file called index.js and open it in your code editor of choice.
In index.js, add the following lines of code to import the dependencies that you installed and the built-in fs dependency, which will allow you to save the CSV file to your filesystem:
const axios = require("axios").default;
const { stringify } = require("csv/sync");
const fs = require("fs");The next step is to apply some default headers that will be used on all HTTP calls made by axios. These headers are required by Postmark for the API call to be valid and authenticated.
axios.defaults.headers["X-Postmark-Server-Token"] = "YOUR-POSTMARK-SERVER-KEY";
axios.defaults.headers.common["Accept"] = "application/json";Next, we will add a function called getMessageDataForTag() to make HTTP calls to the Messages API, searching by tag:
async function getMessageDataForTag(startOffset = 0, startCount = 500, tag) {
try {
const url = `https://api.postmarkapp.com/messages/outbound?count=${startCount}&offset=${startOffset}&tag=${tag}`;
const res = await axios.get(url);
return res.data;
}
catch (err) {
console.log(err)
}
}The getMessageDataForTag() function has three parameters. The startOffset and startCount parameters are used for pagination and required by the Messages API endpoint. The tag parameter allows you to specify the tag to be searched for.
The getMessageDataForTag() function retrieves 500 records at a time. However, in many cases more than 500 records will exist. Add the following function to your code, which will paginate through the results, starting from offset 0.
async function getAllDataForTag(tag) {
var allData = [];
const result = await getMessageDataForTag(0, 500, tag).then(async function (res) {
console.log(res.TotalCount + " total records for tag.");
if (res.TotalCount >= 500 && res.Messages.length > 0) {
let requests = 1;
let moreRecords = true;
allData.push(...res.Messages);
while (moreRecords) {
let offset = requests * 500;
let response = await getMessageDataForTag(offset, 500, tag).then(function (res) {
requests++;
if (res.Messages.length < 500) {
moreRecords = false;
}
allData.push(...res.Messages);
})
}
}
else {
allData.push(...res.Messages);
}
});
return allData;
}The getAllDataForTag() function will check the TotalCount property returned by the Messages API, and if additional records exist, it will continue to paginate through the records using the offset property. Each record is added to an array called allData, which the function returns when pagination is completed.
The getAllDataForTag() function will return up to 10,000 messages per tag. If you know that you will need to retrieve message data after sending and want to use a similar function, you can rotate tags in batches of 10,000 messages to make retrieving data by tag more manageable.
Now that you have a function to retrieve data from the Messages API, the next step is to save the results as a CSV file. The following function will use the csv dependency that we imported to create a CSV file from an input array.
const createCsv = function (data, csvName) {
var csv = stringify(data, { header: true });
fs.writeFile(csvName, csv, 'utf-8', (err) => console.log(err));
}To try out the createCsv() function, we can call the getAllDataForTag() function and pass the resulting array into the createCsv() function. This should create a new CSV file in the same directory called dataOutputForTag.csv.
getAllDataForTag("your-tag-name").then(function (result) {
createCsv(result, "dataOutputForTag");
});Retrieving messages by date range #
If you have not tagged your messages in advance, or if you have batches of tags larger than 10,000 messages, then you may need to use a different search parameter on the Messages API to remain under the 10,000 message search result limit.
In this case, you can use the todate and fromdate parameters to filter your search into a smaller date range. Using these parameters, you can write a function that will iterate over a series of days, retrieving the message data for each day.
async function getMessageDataForDay(startOffset = 0, startCount = 500, todate, fromdate) {
try {
const url = `https://api.postmarkapp.com/messages/outbound?count=${startCount}&offset=${startOffset}&todate=${todate}&fromdate=${fromdate}`;
const res = await axios.get(url);
return res.data;
}
catch (err) {
console.log(err)
}
}Similarly to retrieving data by tag, the first step is to create a function that will call the Messages API with the proper search parameters. Much like the previous function, the getMessageDataForDay() function requires a startOffset and startCount parameter. In addition, this function requires a todate and fromdate parameter, which will be used to search the Messages API.
Now that you have a function that can get message data for a single day, the next step is to write a function that will iterate over a number of days and accumulate the data into an array, which can be passed into the existing createCsv() function.
async function getAllDataByDays(daysToRetrieve) {
var allData = [];
for (let day = 0; day <= daysToRetrieve; day++) {
console.log("Retreiving day " + day);
const end = new Date();
const start = new Date();
end.setDate(end.getDate() - day);
start.setDate(start.getDate() - (day + 1));
console.log("End: " + end.toISOString());
console.log("Start: " + start.toISOString());
const result = await getMessageDataForDay(0, 500, end.toISOString(), start.toISOString()).then(async function (res) {
console.log(res.TotalCount + " total records for day.");
if (res.TotalCount >= 500 && res.Messages.length > 0) {
let requests = 1;
let moreRecords = true;
allData.push(...res.Messages);
while (moreRecords) {
let offset = requests * 500;
let response = await getMessageDataForDay(offset, 500, end.toISOString(), start.toISOString()).then(function (res) {
requests++;
if (res.Messages.length < 500) {
moreRecords = false;
}
allData.push(...res.Messages);
})
}
}
else {
allData.push(...res.Messages);
}
});
}
return allData;
}The getAllDataByDays() function will check the TotalCount property returned by the Messages API, and if additional records exist, it will continue to paginate through the records using the offset property. Each record is added to an array called allData, which the function returns when pagination is completed.
The getAllDataByDays() function will iterate over the specified number of days, and retrieve the message data for each day. However, if there are more than 10,000 messages in a day, this function will not be able to retrieve them. You can work around this by breaking your calls into even smaller time ranges, such as 1 hour instead of 1 day. For the purposes of this tutorial, we will stick with iterating by a full day at a time.
You can use this function similarly to how the getAllDataForTag() function was used:
getAllDataByDays(14).then(function (result) {
createCsv(result, "dataOutputForDateRange.csv");
});If you've made it this far, you should have two different ways to retrieve data from the Messages API - by tag or by date range. If you've followed along, your index.js file should look like this:
const axios = require("axios").default;
const { stringify } = require('csv/sync');
const fs = require("fs");
axios.defaults.headers["X-Postmark-Server-Token"] = "YOUR-POSTMARK-SERVER-KEY";
axios.defaults.headers.common["Accept"] = "application/json";
async function getMessageDataForTag(startOffset = 0, startCount = 500, tag) {
try {
const url = `https://api.postmarkapp.com/messages/outbound?count=${startCount}&offset=${startOffset}&tag=${tag}`;
const res = await axios.get(url);
return res.data;
}
catch (err) {
console.log(err)
}
}
async function getMessageDataForDay(startOffset = 0, startCount = 500, todate, fromdate) {
try {
const url = `https://api.postmarkapp.com/messages/outbound?count=${startCount}&offset=${startOffset}&todate=${todate}&fromdate=${fromdate}`;
const res = await axios.get(url);
return res.data;
}
catch (err) {
console.log(err)
}
}
async function getAllDataForTag(tag) {
var allData = [];
const result = await getMessageDataForTag(0, 500, tag).then(async function (res) {
console.log(res.TotalCount + " total records for tag.");
if (res.TotalCount >= 500 && res.Messages.length > 0) {
let requests = 1;
let moreRecords = true;
allData.push(...res.Messages);
while (moreRecords) {
let offset = requests * 500;
let response = await getMessageDataForTag(offset, 500, tag).then(function (res) {
requests++;
if (res.Messages.length < 500) {
moreRecords = false;
}
allData.push(...res.Messages);
})
}
}
else {
allData.push(...res.Messages);
}
});
return allData;
}
async function getAllDataByDays(daysToRetrieve) {
var allData = [];
for (let day = 0; day <= daysToRetrieve; day++) {
console.log("Retreiving day " + day);
const end = new Date();
const start = new Date();
end.setDate(end.getDate() - day);
start.setDate(start.getDate() - (day + 1));
console.log("End: " + end.toISOString());
console.log("Start: " + start.toISOString());
const result = await getMessageDataForDay(0, 500, end.toISOString(), start.toISOString()).then(async function (res) {
console.log(res.TotalCount + " total records for day.");
if (res.TotalCount >= 500 && res.Messages.length > 0) {
let requests = 1;
let moreRecords = true;
allData.push(...res.Messages);
while (moreRecords) {
let offset = requests * 500;
let response = await getMessageDataForDay(offset, 500, end.toISOString(), start.toISOString()).then(function (res) {
requests++;
if (res.Messages.length < 500) {
moreRecords = false;
}
allData.push(...res.Messages);
})
}
}
else {
allData.push(...res.Messages);
}
});
}
return allData;
}
const createCsv = function (data, csvName) {
var csv = stringify(data, { header: true });
fs.writeFile(csvName, csv, 'utf-8', (err) => console.log(err));
}
getAllDataForTag("your-tag-name").then(function (result) {
createCsv(result, "dataOutPutForTag.csv");
});
getAllDataByDays(14).then(function (result) {
createCsv(result, "dataOutputForDateRange.csv");
});You can execute your app using the following terminal command:
$ node . index.jsNext Steps #
If you’ve completed this tutorial, you should have enough knowledge to get started writing your own node.js code to retrieve message data from the Messages API. Here are some recommendations for next steps:
Use environment variables instead of hard-coding the API key
Modify the code to support other valid search parameters such as subject or metadata_ value
Modify the Date-based code to support wider or narrower time ranges
Update the code to save the results of each API call to a database rather than exporting a CSV
Use the official Postmark node.js library to retrieve data
