Getting Started with Cloud Backup

Requirements: Cloud Backup

Cloud Backup provides a suite of intuitive, easy-to-use mail backup components. Quickly and securely back up mail using popular cloud services such as Microsoft Office 365 (Outlook) and Gmail. Messages are stored in standard .eml (RFC 822) format which can be opened by any mail program including Outlook, Thunderbird, Apple Mail, and more.

Introduction
Authentication
Starting the Backup

Initialization Phase
Backup Phase
Sync Phase

Ending the Backup

Introduction

To use the component, first authenticate to the service by configuring the OAuth property and calling the Authorize method. Next, set the DataFolder property to a local path and call the StartBackup method. Messages will be grouped by month and stored in subfolders of DataFolder. For instance, all messages from October 2023 would be stored in the folder 2023-10. The following sections describe the behavior and configuration options for the component in more detail.

Authentication

Communication with Gmail and Office 365 APIs requires authentication via OAuth 2.0. This can be performed by using the component's OAuth settings. After configuring the OAuth settings appropriately, calling the Authorize method will retrieve the required access token used to access the relevant protected resources. This token will be used automatically by the component in subsequent requests. Please see the following for a simple example:


component.OAuth.ClientProfile = OAuthClientProfiles.cocpApplication;
component.OAuth.GrantType = OAuthGrantTypes.cogtAuthorizationCode;
component.OAuth.ClientId = CLIENT_ID;
component.OAuth.ClientSecret = CLIENT_SECRET;

component.Authorize();

The component will request a refresh token during authorization by default. The refresh token allows the component to automatically obtain a new access token if the current access token is about to expire. Whether to automatically obtain a new access token is controlled by the OAuthAutomaticRefresh setting and is true by default.

For an in-depth look at various configurations of the OAuth settings, or performing OAuth authentication using a separate process, please refer to the help documentation.

Starting the Backup

Before calling the StartBackup method set the DataFolder property to a path on disk where the component will save all messages. Within this folder, messages will be saved to a subfolder with the year and month the message was created. For example, all messages from October 2023 would be stored in the folder \DataFolder\2023-10\.

DataFolder may be set to the location of a previous backup. During the backup, the component will only retrieve messages that have not already been backed up.

After setting DataFolder, the Filter, StartDate, and EndDate properties can be set to select a subset of messages for backup. These properties will be utilized when retrieving a list of message IDs during the Initialization Phase.

Additionally, the MaxConnections property can be set to control the number of simultaneous connections used during the process. Generally, more connections yield better performance, but excessive connections can impair performance. Please see the help documentation for further details.

Lastly, call StartBackup to begin the backup process. Note that this method is non-blocking, and will return before the backup completes. For example:


  bool backupComplete = false;

  component.OnEndBackup += (o, e) => {
    backupComplete = true;
  }

  // Assuming successful authentication
  component.Authorize();
  
  component.DataFolder = "C:\\BackupDirectory\\";
  component.StartDate = "2023/07/01";                // Only backup messages created after July 01, 2023
  component.EndDate = "2023/07/31";                  // Only backup messages created before July 31, 2023
  component.MaxConnections = 5;                      // 5 simultaneous connections
  
  component.StartBackup();

  while (!backupComplete) {
    component.DoEvents();
  }

At any point during the backup, the AbortBackup method can be called to stop the backup process. Once the backup is complete, the EndBackup event will fire. The following events may fire during the backup and provide information about the progress of the backup:

BeforeMessageBackup
MessageBackup
AfterMessageBackup
MessageError
MessageDelete
EndBackup
Log

The backup process is separated into the phases listed below:

Initialization Phase: The component will authenticate to the server and obtain a list of messages to be considered for backup.
Backup Phase: The component retrieves the content of each message that will be backed up. Messages that have been previously backed up will be automatically skipped by default.
Sync Phase: If the SyncDeletes property is true any messages that have been deleted remotely will be deleted locally. The default value of SyncDeletes is false and no messages will be deleted locally.

The sections below provide a detailed description of each phase.

Initialization Phase

After calling StartBackup the component will first retrieve a list of all message IDs to backup. This operation is performed before saving any messages on disk.

Note that multiple requests may be made in this phase as results from the server may be paged. This process may take some time depending on the number of messages to be backed up.

Information regarding this process and the number of retrieved message IDs will be available in the Log event. Once this phase is completed, the MessageCount property will contain the total number of messages the component will attempt to back up.

Backup Phase

Once all message IDs have been retrieved, the component will start the Backup Phase. During this phase, the component will fetch and save the message content for each message ID retrieved during the Initialization Phase. Assuming no errors are encountered during the process, for each message the BeforeMessageBackup, MessageBackup, and AfterMessageBackup events will fire in that order. Please see a description and possible applications of each event below.

BeforeMessageBackup

This event will fire before the component fetches the content of each message. Before the message's content is fetched, this event can be used to:

Query various message properties.
Modify the on-disk location this message will be saved to.
Skip backing up this message.

MessageBackup

This event will fire after a message's content is retrieved successfully and indicates the message is about to be saved locally. Before the message is saved locally, this event can be used to:

Query various message properties.
Query the message content.
Skip backing up this message.

AfterMessageBackup

This event will fire after the message is saved locally and indicates the message was successfully backed up. If this event does not fire for a specific message, it could mean:

The message was skipped by the user or component.
An error was encountered fetching the message content.
An error was encountered saving the message locally.

If an error is encountered for any reason during the Backup Phase, the MessageError event will fire, containing information related to the error itself. By default, the component will attempt to retry the operation MaxRetryCount times, and subsequent requests will be made every RetryInterval seconds assuming the error continues to occur.

For more control over the retry logic in the event of a message error, MessageError can be handled accordingly. Please see the help documentation for more details.

Additionally, if DataFolder points to an existing backup, the component will first check to see if the message is present locally before retrieving the message.

When using the GmailBackup component, if the message is already present, the component will skip backing up this message. Otherwise, the message will be backed up as expected.

However, when using the Office365Backup component, if the message is already present, the component will compare the last modified date of the local and remote messages. If the last modified date differs between the two locations, the local message content will be updated. Otherwise, the component will skip backing up this message.

Note for both components, the mentioned behavior can be observed and/or overridden within BeforeMessageBackup.

Sync Phase

Once the Backup Phase is complete, the optional Sync Phase may begin. By default, this phase will not occur but can be enabled by setting the SyncDeletes property to true.

During this phase, the component will check every message in the local backup to determine if it has been deleted remotely. If the component finds that a message no longer exists remotely, the MessageDelete event will fire, and the message will be removed locally. For example:


bool backupComplete = false;

component.OnEndBackup += (o, e) => {
  backupComplete = true;
};

component.OnMessageDelete += (o, e) => {
  Console.WriteLine("Message does not exist remotely: " + e.Id);
  Console.WriteLine("Deleting local file: " + e.BackupFile);
};

component.SyncDeletes = true;

component.DataFolder = "C:\\BackupDirectory\\";
component.StartBackup();

while (!backupComplete) {
  component.DoEvents();
}

Once this phase is finished, the EndBackup event will fire, indicating the backup process is complete.

Ending the Backup

Once all phases are complete, EndBackup will fire, indicating the backup is complete. At any point during the backup process, the AbortBackup method can be called, terminating the ongoing process. In this case, the existing local backup state will be preserved up until the point this method was called, and EndBackup will fire as expected.

For a detailed example of the complete process, please see below:


bool backupComplete = false;
int messagesBackedUp = 0;

component.OnEndBackup += (o, e) => {
  backupComplete = true;
};

component.OnBeforeMessageBackup += (o, e) => {
  if (e.Skip) {
    e.Skip = false; // Force message to be backed up
  }
};

component.OnAfterMessageBackup += (o, e) => {
  messagesBackedUp++;
};

component.OnMessageError += (o, e) => {
  if (e.Retry) {
    Console.WriteLine("Error backing up message, retrying: " + e.ErrorCode + ": " + e.ErrorMessage);
  } else {
    Console.WriteLine("Error backing up message, skipping: " + e.ErrorCode + ": " + e.ErrorMessage);
  }
};

component.onMessageDelete += (o, e) => {
  Console.WriteLine("Message does not exist remotely: " + e.Id);
  Console.WriteLine("Deleting local file: " + e.BackupFile);
};

component.MaxConnections = 10;
component.DataFolder = "C:\\BackupDirectory\\";
component.StartDate = "2023/07/01";
component.EndDate = "2023/07/31";
component.SyncDeletes = true;

component.StartBackup();

while (!backupComplete) {
  component.DoEvents();
}

Console.WriteLine("Backup Complete!");
Console.WriteLine("Messages retrieved: " + messagesBackedUp);
Console.WriteLine("Messages skipped: " + component.MessageCount - messagesBackedUp);

We appreciate your feedback. If you have any questions, comments, or suggestions about this article please contact our support team at support@nsoftware.com.