API test automation using Postman simplified: Part 1
QA Engineer - 06 March 2020 -
QA Engineer - 06 March 2020 -
Every application you build today relies on APIs. This means it’s crucial to thoroughly verify APIs before rolling out your product to the client or end-users. Although multiple tools for automating API testing are available and known to QAs, we have to decide on the tool that best suites our project requirements and can scale and run without much maintenance/upgrade while creating a test suite. And once the tool is finalized, we have to design an end to end flow with the tool which is easy to use, and we can get most benefits out of automation efforts.
I am writing this two blog series to talk about end to end flow of API automation with Postman from deciding on the tool to implementing the suite till integration with CI tool and report generation .thus the content of these blogs is totally based on our experience and learning while setting it up in our project.
In the first blog of the series, we’ll talk about the phases till implementation while in the next blog we’ll discuss integration with Jenkins, monitoring etc.
Thus I have structured this blog into following phases:
Before moving on to talk on these topics in detail , For those who are new to postman, in brief, Postman is one of the most renowned tools for testing the APIs and is most commonly used by developers and testers. It allows for repeatable, reliable tests that can be automated and used in a variety of environments like Dev, Staging, and Production etc. It presents you with a friendly GUI for constructing requests and reading responses, easy for anyone to get started without any prior knowledge of any scripting language since Postman also has a feature called ‘Snippets’. By default, they are present in Java script but by using it you can generate code snippets in a variety of languages and frameworks such as Java, Python, C, CURL and many others.
Let’s now move to each phase one by one wherein I’ll talk about our project specific criteria and examples in detail.
A few months back when we came up with a plan to automate API tests of our application, first question in mind was, what tool to use which will best suit our requirement?
Mostly the team was briefly familiar with Postman, JMETER, rest assured and FitNesse tools/frameworks for API automation. The main criteria for selecting the tool was to have an open source option which helps to quickly get started with the test automation task, is easy to use, gives a nice and detailed reporting and is easy to integrate with CI tool.
We could quickly create a POC of complete end to end flow using postman and a similar POC for comparison purpose on JMETER. However, Postman came out as a better option in terms of reporting and user-friendliness since it does not require much of scripting knowledge and hence anyone in the team can pitch in anytime and contribute in the automation effort.
Now since we had liked the tool and wanted to go ahead with the same to build a complete automation suite, next set of questions on our mind was related to limitations and scalability of Postman free version,
This was important to be evaluated first and foremost before starting the actual automation effort as we wanted to avoid any unnecessary rework. Thus we started finding answers to our questions. While we could find few answers through the web searches, for some of the clarifications we had to reach out to postman customer support to be double sure on the availability and limitations of the tool.
As a gist, it is important to know that if you are using postman free version then:
This helped us to understand whether free version suffices our requirements or not. Since we were not planning to use postman APIs or monitoring services, we were good to go ahead with postman free version.
Creating an automation suite with postman requires understanding of following building blocks (Refer Figure 1)
Figure 1: Basic Building Blocks of Postman
Using the base framework that we created during POC we were able to extend it to add multiple request and multiple tests around each request’s response to having a full-fledged automation suite ready with us and running daily.
In our case our first problem statement to be achieved in API automation was a set of APIs for a reporting module.
Since report contains dynamic data and generating a set of test data is also very tough due to multiple environmental factors, it was not possible for us to apply fixed assertions to validate data accuracy. That’s why we had to come up with other ways to test that don’t exactly match the correctness of the data but still are thorough enough to check the validity of the data and report actual failures.
Thus while doing this exercise what we followed and that really turned out quite beneficial for us was that before starting to write the tests in the tool itself, it is very important to clearly list down everything in detail as to what exactly we want to assert.
For simple APIs with static responses, this list might be pretty straightforward to define. But In our example, it required a good amount of brainstorming to come up with list of assertions which can actually check the validity of the response without knowing the data value itself.
So we thoroughly studied the API responses, came up with our PASS/FAIL criteria, listed down each assertion in own words in our plan and then went ahead with converting them into actual postman assertions. For ex:
|-> Response Code 200 OK|
|-> Schema Validation|
|-> Not Null check for applicable values|
|-> Exact value check for a set of values|
|-> Match Request start/end time with response start/end time|
|-> Range validation for a set of values (between 0-1)|
|-> Data Validation Logic: Detailed logic in terms of response objects/data with if/else criteria for defined PASS/FAIL cases (details removed)|
As we see in above list, we have a number of positive and negative tests covered. While we have such assertions in place in Postman, we can’t say it will work when such response gets generated in actual until we test it thoroughly. If we are testing postman collection of APIs with actual environment response, we might not get each type of failed responses.
And thus to test it we need a way to mock the API request and response which is very similar to the actual response but has some values modified to invalid values to test whether out script and assertion catch them as failure or not. This is possible in postman through mock servers. You can add a mock server for a new or existing collection by navigating to New->Mock server in postman (Refer to Figure 2)
A Postman mock server lets you mock a server response, allowing a team to develop or write tests against a service that is not yet complete or is unstable so that instead of hitting the actual endpoint URL, request is made to specified request path given in mock server and accordingly mocked test responses are returned and we can see how our script behaves for such requests and responses. Thus during actual execution of live endpoints if similar scenarios occur, we already know how our script is going to handle them.
Figure 2: Adding mocked request/response using Mock Server in Postman
Now once we have our test suite ready with required cases added and tested, we are good to start scheduling it to run daily for our test environments so that it checks the API health and reports failures.
In our next blog we will discuss these phases in detail. Stay tuned.