General Discussion

Tech Talk: Implementation – Microsoft Azure Blob Storage Integration

  • 1.  Tech Talk: Implementation – Microsoft Azure Blob Storage Integration

    Posted 06-20-2024 16:20
    Edited by Ian Chamberlain 06-20-2024 17:47

    Azure Blob Storage is Microsoft's cloud-based object storage solution, similar to Amazon's S3 (Simple Storage Service). Both services offer scalable and secure storage for large amounts of unstructured data, such as text or binary data. They provide similar features, including high availability, durability, and multiple redundancy options to ensure data protection. Users can store and retrieve any amount of data at any time, benefiting from ease of access through various APIs and tools, making both services ideal for a wide range of applications including backup, archival, and big data analytics. 

    This is in line with the last implementation post about making a connection to Microsoft to leverage their Graph API. This time, we are going to be writing TrueContext Submission JSON files into a Microsoft Storage account so that the data can be processed and sent to any other source in Azure. 

    Sending submission data to Azure Blob Storage allows you to both keep a record of all the data that has been captured and is also a gateway to getting this data into your other Azure services using a Data Pipeline. 

    We will start by creating or using an existing Storage Account in Azure and then a new (or an existing) container inside the Storage Account. 

    Creating the connection in TrueContext 

    Following similar steps to create the App Registration in Azure as found here you can assign different permissions to all the app to have access to Azure Blob Storage. The level of permissions needed will be up to the business, but the service will need the ability to write to the Storage Account.  

    1. In TrueContext, create an HTTP Connection and we will use the OAuth 2 method. The settings that will be input will be as follows: 

      1. Base Connection URL: https://account.blob.core.windows.net (replace account with the storage account name; this is the name that you define when creating the Storage Account) 

      2. Access Token URL: https://login.microsoftonline.com/TENENTID/oauth2/v2.0/token (replace TENENTID with the ID of your tenant, this can be found in the App Registration)

      3. Client ID: found in the app registration

      4. Client Secret: Created in the app registration

      5. Scopes: https://storage.azure.com/.default (this is important for the Azure Blob Storage API calls to work) 

    1. Next you will click on the Connect button in the connection and should be presented with a Connection Established. If you receive an error, please review your app registration permissions and the values of the Client ID and Secret that were input. 

    This connection can now be used to make API calls to Azure Blob Storage to send your JSON/PDF or any output for storage and processing. 

     

    Creating a destination to Azure Blob Storage 

    Now we will create the destination in TrueContext to write submissions in the form of a JSON into the container of choice. 

    1. Navigate to the FormSpace that has permissions to use the Connection that we just created. Preferably, test this in a development FormSpace first. 

    1. Create a new HTTP Data Destination 

    1. Give it a name and ensure that it is using the connection that we created. 

    1. On the Request Configuration tab, use the following settings: 

      1. HTTP Method: PUT (POST will return an error)

      2. URL Suffix: /containerName/filename.json

        1. Replace containerName with the container that you would like to write into. 

        2. Replace filename with the naming convention that you would like. A common one would be %r.json to use the submission identifier.

        3. If you would like to build a folder structure in the Container, you can specify a path and the folders will be created automatically. /containerName/folder1/folder2/filename.json 

      3. Request Headers:

        1. Key: Content-type Value: application/octet-stream

        2. Key: x-ms-blob-type Value: BlockBlob 

    1. Click on Update to finalize the data destination 

    Testing this out will be as simple as adding this destination to any form, attaching the standard JSON (V2) document type and submitting the form. 

    Once the form is submitted, the json file should land in the Container/Folder that was specified. 

    Now that you are sending data in Azure Blob Storage, you can leverage Azure Data Factory to parse, process, and manipulate the data just the way that you need to send it off to other services like Azure Cosmo DB, SQL Server, and many others. 

    #Microsoft #TechTalkImplementation



    ------------------------------
    Ian Chamberlain
    Solutions Architect
    TrueContext
    ------------------------------



Reminder: Content posted to our Community is public content.  Please be careful not to post Intellectual Property that you do not have permission to share.  For more information please refer to our Terms Of Use