Question

ScrollID in Postman and Python

  • 22 March 2023
  • 6 replies
  • 73 views

Badge

Hello fellow API Callers.

I had a great conversation with @link_black  today, getting started with the PX API.

Postman and ScrollIDs: I am running a User report in Postman and of course hit the 10K limit. What’s the best way to iterate through scroll IDs to get the complete record set in one file using Postman?

Python and ScrollIDs: I’ve replicated the Postman query in Python. Has anyone built a nice script to iterate through scroll IDs with Python?

Thanks,

JC

 


6 replies

Userlevel 3

@Jhaydon - I’m in the process of putting a script together to gather survey responses from multiple surveys.  I plan to build in the handling of the scrollID for pagination.  Once I’m done and have it working, I’ll post some snippets here.

Userlevel 3

@Jhaydon - Hoping this will help you with whatever you’re writing up from a Python perspective:
 

############################################
# FUNCTION TO GET API CALL WITH SCROLLING.
# Reference API documentation:
# https://gainsightpx.docs.apiary.io/#reference/surveyresponse
############################################

def get_surveyresponsesapi(engagementId, scrollId):

if scrollId == "":
# Set the request url for the first page of results
url = BASE_URL + "/v1/survey/responses?filter=engagementId=="+ engagementId + "&pageSize=" + str(API_PAGE_SIZE) + "&sort=-date"

else:
# Set the request url for the subsequent pages of results
url = BASE_URL + "/v1/survey/responses?filter=engagementId=="+ engagementId + "&pageSize=" + str(API_PAGE_SIZE) + "&sort=-date" + "&scrollId=" + scrollId

payload={}
headers = {
'Accept': 'application/json',
API_KEY_NAME: API_KEY_VALUE
}

# Make the API call
return requests.request("GET", url, headers=headers, data=payload)


############################################
# SETTING UP TO PARSE THE API CALL(S)
############################################

# create a counter for the number of survey responses returned
totalResponseCount = 1
currentresponsecount = 0
currentScrollId = ""

# Does currentresponsecount < totalResponseCount? If so, we need to scroll through the results
# Good Reference: https://jonathansoma.com/lede/foundations-2018/classes/apis/multiple-pages-of-data-from-apis/


##############################################################################
# LOOP WHILE CURRENT RESPONSE COUNT IS LESS THAN TOTAL COUNT RETURNED BY API
##############################################################################

while currentresponsecount < totalResponseCount:

# Make the API call
surveyResponses = get_surveyresponsesapi(engagement, currentScrollId)

if surveyResponses.status_code == 200:
#Successful API call

# Parse the response as JSON
responseData = surveyResponses.json()

# Extract the list of surveys from the JSON response
responses = responseData['results']

# Increase the totalResponseCount by the API's count of total responses
# Note this may update if new responses are received while we parse each page.
totalResponseCount = responseData['totalHits']

# count the number of responses returned in this page of API responses
currentresponsecount = currentresponsecount + len(responses)

# set the scrollId for the next page of results
currentScrollId = responseData['scrollId']

######################################
# DO THE WORK THEN LOOP
######################################


else:
Print("Error: " + surveyResponses.status_code)

 

Userlevel 7
Badge +4

Looking forward to it @rterakedis 🙌🏼

Userlevel 3

@revathimenon  - I posted a response last week with some python code snippets, but it looks like it had to get reviewed by a moderator.   Can you please review and post it?   Thank you!

Userlevel 7
Badge +4

@rterakedis - the code must have alerted the system for spam check.

Approved it now, thanks for sharing! 

Userlevel 3

If you’re looking for something more complete, I put a sanitized/slimmed down version that also supports “last x days” filtering in GitHub:  https://gist.github.com/rterakedis/437614bb6de6d5e74b9128de0de20c1f

Reply