Iterators built with different continuation tokens are producing the same results in Google Apps

google script get files in folder
google apps script class folder
driveapp makecopy
folder iterator apps script
google script drive
getblob google script
google script setname
google script create folder

I am programming a Google Apps script within a spreadsheet. My use case includes iterating over a large set of folders that are children of a given one. The problem is that the processing takes longer than the maximum that Google allows (6 minutes), so I had to program my script to be able to resume later. I am creating a trigger to resume the task, but that is not part of my problem (at least, not the more important one at this moment).

My code looks like this (reduced to the minimum to illustrate my problem):

function launchProcess() {
    var scriptProperties = PropertiesService.getScriptProperties();
    scriptProperties.setProperty(SOURCE_PARENT_FOLDER_KEY, SOURCE_PARENT_FOLDER_ID);
    scriptProperties.deleteProperty(CONTINUATION_TOKEN_KEY);
    continueProcess();
}

function continueProcess() {
    try {
        var startTime = (new Date()).getTime();
        var scriptProperties = PropertiesService.getScriptProperties();
        var srcParentFolderId = scriptProperties.getProperty(SOURCE_PARENT_FOLDER_KEY);
        var continuationToken = scriptProperties.getProperty(CONTINUATION_TOKEN_KEY);
        var iterator = continuationToken == null ? DriveApp.getFolderById(srcParentFolderId).getFolders() : DriveApp.continueFolderIterator(continuationToken);

        var timeLimitIsNear = false;
        var currTime;
        while (iterator.hasNext() && !timeLimitIsNear) {
            var folder = iterator.next();

            processFolder_(folder);

            currTime = (new Date()).getTime();
            timeLimitIsNear = (currTime - startTime >= MAX_RUNNING_TIME);
        }

        if (!iterator.hasNext()) {
            scriptProperties.deleteProperty(CONTINUATION_TOKEN_KEY);
        } else {
            var contToken = iterator.getContinuationToken();
            scriptProperties.setProperty(CONTINUATION_TOKEN_KEY, contToken);
        } 

    } catch (e) {
        //sends a mail with the error
    }    
}

When launchProcess is invoked, it only prepares the program for the other method, continueProcess, that iterates over the set of folders. The iterator is obtained by using the continuation token, when it is present (it will not be there in the first invocation). When the time limit is near, continueProcess obtains the continuation token, saves it in a property and waits for the next invocation.

The problem I have is that the iterator is always returning the same set of folders although it has been built from different tokens (I have printed them, so I know they are different).

Any idea about what am I doing wrong?

Thank you in advance.

It appears that your loop was not built correctly. (edit : actually, probably also another issue about how we break the while loop, see my thoughts about that in comments)

Note also that there is no special reason to use a try/catch in this context since I see no reason that the hasNext() method would return an error (but if you think so you can always add it)

here is an example that works, I added the trigger creation / delete lines to implement my test.

EDIT : code updated with logs and counter
var SOURCE_PARENT_FOLDER_ID = '0B3qSFd3iikE3MS0yMzU4YjQ4NC04NjQxLTQyYmEtYTExNC1lMWVhNTZiMjlhMmI'
var MAX_RUNNING_TIME = 5*35*6;

function launchProcessFolder() {
  var scriptProperties = PropertiesService.getScriptProperties();
  scriptProperties.setProperty('SOURCE_PARENT_FOLDER_KEY', SOURCE_PARENT_FOLDER_ID);
  scriptProperties.setProperty('counter', 0);
  scriptProperties.deleteProperty('CONTINUATION_TOKEN_KEY');
  ScriptApp.newTrigger('continueProcess').timeBased().everyMinutes(10).create();
  continueProcessFolder();
}

function continueProcessFolder() {
  var startTime = (new Date()).getTime();
  var scriptProperties = PropertiesService.getScriptProperties();
  var srcParentFolderId = scriptProperties.getProperty('SOURCE_PARENT_FOLDER_KEY');
  var continuationToken = scriptProperties.getProperty('CONTINUATION_TOKEN_KEY');
  var iterator = continuationToken == null ? DriveApp.getFolderById(srcParentFolderId).getFolders() : DriveApp.continueFolderIterator(continuationToken);
  var timeLimitIsNear = false;
  var currTime;
  var counter = Number(scriptProperties.getProperty('counter'));
  while (iterator.hasNext() && !timeLimitIsNear) {
    var folder = iterator.next();
    counter++;
    Logger.log(counter+'  -  '+folder.getName());

    currTime = (new Date()).getTime();
    timeLimitIsNear = (currTime - startTime >= MAX_RUNNING_TIME);

    if (!iterator.hasNext()) {
      scriptProperties.deleteProperty('CONTINUATION_TOKEN_KEY');
      ScriptApp.deleteTrigger(ScriptApp.getProjectTriggers()[0]);
      Logger.log('******************no more folders**************');
      break;
    }
  }
  if(timeLimitIsNear){
    var contToken = iterator.getContinuationToken();
    scriptProperties.setProperty('CONTINUATION_TOKEN_KEY', contToken);
    scriptProperties.setProperty('counter', counter);
    Logger.log('write to scriptProperties');
  }
}

EDIT 2 :

(see also last comment)

Here is a test with the script modified to get files in a folder. From my different tests it appears that the operation is very fast and that I needed to set a quite short timeout limit to make it happen before reaching the end of the list.

I added a couple of Logger.log() and a counter to see exactly what was happening and to know for sure what was interrupting the while loop.

With the current values I can see that it works as expected, the first (and second) break happens with time limitation and the logger confirms that the token is written. On a third run I can see that all files have been dumped.

var SOURCE_PARENT_FOLDER_ID = '0B3qSFd3iikE3MS0yMzU4YjQ4NC04NjQxLTQyYmEtYTExNC1lMWVhNTZiMjlhMmI'
var MAX_RUNNING_TIME = 5*35*6;

function launchProcess() {
  var scriptProperties = PropertiesService.getScriptProperties();
  scriptProperties.setProperty('SOURCE_PARENT_FOLDER_KEY', SOURCE_PARENT_FOLDER_ID);
  scriptProperties.setProperty('counter', 0);
  scriptProperties.deleteProperty('CONTINUATION_TOKEN_KEY');
  ScriptApp.newTrigger('continueProcess').timeBased().everyMinutes(10).create();
  continueProcess();
}

function continueProcess() {
  var startTime = (new Date()).getTime();
  var scriptProperties = PropertiesService.getScriptProperties();
  var srcParentFolderId = scriptProperties.getProperty('SOURCE_PARENT_FOLDER_KEY');
  var continuationToken = scriptProperties.getProperty('CONTINUATION_TOKEN_KEY');
  var iterator = continuationToken == null ? DriveApp.getFolderById(srcParentFolderId).getFiles() : DriveApp.continueFileIterator(continuationToken);
  var timeLimitIsNear = false;
  var currTime;
  var counter = Number(scriptProperties.getProperty('counter'));
  while (iterator.hasNext() && !timeLimitIsNear) {
    var file = iterator.next();
    counter++;
    Logger.log(counter+'  -  '+file.getName());

    currTime = (new Date()).getTime();
    timeLimitIsNear = (currTime - startTime >= MAX_RUNNING_TIME);

    if (!iterator.hasNext()) {
      scriptProperties.deleteProperty('CONTINUATION_TOKEN_KEY');
      ScriptApp.deleteTrigger(ScriptApp.getProjectTriggers()[0]);
      Logger.log('******************no more files**************');
      break;
    }
  }
  if(timeLimitIsNear){
    var contToken = iterator.getContinuationToken();
    scriptProperties.setProperty('CONTINUATION_TOKEN_KEY', contToken);
    scriptProperties.setProperty('counter', counter);
    Logger.log('write to scriptProperties');
  }
}

Class FileIterator | Apps Script, Iterators built with different continuation tokens are producing the same results in Google Apps. Question. I am programming a Google Apps script within a  Systems and methods are disclosed for the secure distribution of firmware and configuration updates to un-networked physical devices. A client component is provided for installation on a client device, which is configured to receive, via the client component, a status data packet including a status indication from a beacon, when the client is proximate to the beacon.

As of January 1, 2016 this is still a problem. The bug report lists a solution using the Advanced Drive API, which is documented here, under "Listing folders".

If you don't want to use Advanced services, an alternative solution would be to use the Folder Iterator to make an array of File Ids.

It appears to me that the Folder Iterator misbehaves only when created using DriveApp.continueFolderIterator(). When using this method, only 100 Folders are included in the returned Folder Iterator.

Using DriveApp.getFolders() and only getting Folder Ids, I am able to iterate through 694 folders in 2.734 seconds, according the Execution transcript.

function allFolderIds() {
  var folders = DriveApp.getFolders(),
      ids = [];
  while (folders.hasNext()) {
    var id = folders.next().getId();
    ids.push(id);
  }
  Logger.log('Total folders: %s', ids.length);
  return ids;
}

I used the returned array to work my way through all the folders, using a trigger. The Id array is too big to save in the cache, so I created a temp file and used the cache to save the temp file Id.

Google JavaScript Style Guide, Iterators built with different continuation tokens are producing the same results in Google Apps -. i programming google apps script within spreadsheet. utilize  This guide explains how to set up authentication and authorization for server to server production applications in Google Cloud APIs. Authentication refers to the process of determining a client's identity. Authorization refers to the process of determining what permissions an authenticated client has for a specific resource. That is

This is caused by a bug in GAS: https://code.google.com/p/google-apps-script-issues/issues/detail?id=4116

CoffeeScript, getContinuationToken(). Gets a token that can be used to resume this iteration at a later time. This method is useful if processing an iterator in one  The technology disclosed relates to processing events generated by Internet of Things (IoT) devices. In particular, it relates to storing a machine-readable declarative specification of stateful event processing of an automated multi-step progression of monitoring of Internet of Things (IoT) devices that generate events.

It appears you're only storing a single continuation token. If you want to recursively iterate over a set of folders and allow the script to pause at any point (e.g. to avoid the timeout) and resume later, you'll need to store a bunch more continuation tokens (e.g. in an array of objects).

I've outlined a template that you can use here to get it working properly. This worked with thousands of nested files over the course of 30+ runs perfectly.

Pagination Token, Like other programming style guides, the issues covered span not only Optional formatting choices made in examples must not be enforced as rules. Imports have the same path, but since it doesn't align it can be hard to see. If a module is imported only for its side effects, the call must be a goog.require (not a goog. (a) the iterator end() function returns a pointer to the last element in the container (b) iterators allow you to traverse container members using pointer-lie semantics (c) iterators provide abstraction away form the "how" to traverse a container (d) iterators are in essence improved version of pointers from C

US9705817B2 - Method, system and program product for , Array comprehensions: cubes = (function() { var i, len, results; results = []; for (i = 0​, len 2 is that now the CoffeeScript compiler produces modern JavaScript syntax (ES6, This allows different versions of CoffeeScript to be installed globally and locally. app.coffee' , you must first “register” CoffeeScript as an extension:. Start studying CS 150 Chapter 4. Learn vocabulary, terms, and more with flashcards, games, and other study tools.

US8902357B2 - Quad-core image processor, A couple of days ago I built another funny case including two different columns in The following example shows how to retrieve additional sets of results for Access tokens allow your app to access the Graph API. To paginate with a continuation token, make the same request again but include the continuation token. A system and method for developing, deploying, managing and monitoring a web application in a single environment is disclosed herein. The single environment is preferably an integrated development environment (“IDE”).

Understanding the Whys, Whats, and Whens of ValueTask, This application is a Continuation of U.S. application Ser. In embodiments, the signals are sent or made available on the allocation data to obtain a stream of contextualized token instances; plurality of production rules for As the user enters the same app location executed at different times in Data table calls iterator. Going GAS: From VBA to Google Apps Script Bruce Mcpherson Whether you’re moving from Microsoft Office to Google Docs or simply want to learn how to automate Docs with Google Apps Script, this practical guide shows you by example how to work with each of the major Apps Script services.

Comments
  • It looks as a bug for me. It's reported here: code.google.com/p/google-apps-script-issues/issues/…
  • Thank you for your response, but I see that the difference is that you put the if within the loop and I had it out, as I only wanted it to be executed once, when the limit was near or there wasn't more elements. Do you really think I need to store the token in the property in every iteration? I don't see a reason for that, but I will try doing it, just in case.
  • you're right... I made a mistake ! code updated ;-) It would indeed be a potential error source to write repeatedly to the scripProperties. sorry about that (was at work and probably too busy on something else...)
  • I added a second 'if' to know what made it interrupt (last item or time limit) because the while loop has 2 conditions.
  • I don't understand you. Your code IS writing repeatedly to the scriptProperties. In fact I have tried it and it started raising this exception "The script has found some problem :(, Exception: Service invoked too many times in a short time: properties rateMax. Try Utilities.sleep(1000) between calls.]"
  • I have tested with your last code and 30 seconds limit. I put delay to slow down the process: - First execution: it processed 50 folder - Second execution: it processed 51 more (no repetitions this time) - Third execution: it processed 43 (the first seems to be the last of previous execution (folder number 101)) - Fourth execution: it processed 50 folder, repeating from the same that the previous execution (folder number 101) - Fifth execution: it processed 50 folder, repeating from the same point. - ...