General Discussion

Hi TrueContext Community,

I have a form deployed which uses the Teamwork feature, and we have many incomplete forms available for users to claim.

There are a few different versions of the form out there, and I'm wondering if there are any best practices out there for "upgrading" existing incomplete forms to the newest form version?

My workaround to accomplish form updates is:

1. Add a dispatch destination to the old version of the form which triggers when the user is me. The dispatch creates a updated version of the form with all the answers carried over and it goes to me.
2. Claim or assign the old form on my device, and then Transfer it to trigger the dispatch destination.
3. Open the updated form and Transfer it to put it back in the incomplete pool.
4. Go onto the portal and delete the old form.

This is pretty time-consuming to do for hundreds of forms... don't ask me how I know...

Any better ideas out there??

Cheers,

Mark

Hi Mark

That sounds...... very painful :(

What you are running into is that a submission can only ever be tied to one version of a form, this is intentional. If we allowed a form to change what it contains and how it works midway through it being used, it could cause data loss/missing data, work needing to be redone, and just general confusion. While some small simple changes may not cause issues, we do have to anticipate worst cases.

I can't think off the top of my head a better way to do this, but here are a few things to consider (some of these you obviously know, but I am putting them here to help others as well).

• A submission in our system is a single data record, from start (dispatch/forms tab) to finish (complete status).
• A submission can have multiple versions, but the data record ID doesn't change when new versions are added.
  • This could be a simple as someone opening a new form and submitting as complete (one submission version), or 
  • It could be a dispatched form going through multiple Teamwork incomplete submissions, (each teamwork incomplete submission is a submission version), or 
  • Some mix of all of this (various number of submission versions). 
• When you use a dispatch data destination you are creating a new submission, with a new data record ID. There is a record of which previous submission created it, but it can make it a bit more complicated if you want to consider/treat two different submissions to be a single submission in how you implement your workflows and store in your systems of record.
• Teamwork is a good way to keep the same data record ID for a submission that has multiple steps/contributors (i.e. form submission versions). This was a key part of the Teamwork feature and we intend to continue building our system to keep to a single submission where it makes sense to do so.
• When a new submission is created, either from dispatching or someone opening a blank form from the app, it creates the submission based on the latest form definition their device has (note this may not even be the latest form version if they are offline, if this is a concern please refer to: Refresh on Form Open).

So knowing all of that, there isn't much wiggle room and you've come up with a clever solution, but as you noted, it isn't very scalable.

What is it about your scenario that makes you need to often update the form?

• Do your submissions cover a long period of time (so a greater chance of a form change happening while it is in use?)
• Is there a need to change your forms quite often?
  
  • Is your process still maturing? or is it just the nature of the process that it needs to continuously change?

These answers may help up come up with something to help.

------------------------------
Natalie Tallon
Product Manager
TrueContext
ntallon@truecontext.com

Original Message:
Sent: 07-02-2024 17:00
From: Mark Warwick
Subject: Upgrading a Form Version

Hi TrueContext Community,

I have a form deployed which uses the Teamwork feature, and we have many incomplete forms available for users to claim.

There are a few different versions of the form out there, and I'm wondering if there are any best practices out there for "upgrading" existing incomplete forms to the newest form version?

My workaround to accomplish form updates is:

1. Add a dispatch destination to the old version of the form which triggers when the user is me. The dispatch creates a updated version of the form with all the answers carried over and it goes to me.
2. Claim or assign the old form on my device, and then Transfer it to trigger the dispatch destination.
3. Open the updated form and Transfer it to put it back in the incomplete pool.
4. Go onto the portal and delete the old form.

This is pretty time-consuming to do for hundreds of forms... don't ask me how I know...

Any better ideas out there??

Cheers,

Mark

------------------------------
Mark Warwick
Gas Engineer
PG&E
------------------------------

Hi Natalie,

Thanks for your thoughtful reply. The part about creating the submission based on the latest form definition on the device is definitely new to me, so that is really good to know.

We are intentionally making use of the Teamwork feature to keep a single "submission" with consistent data record. Another reason I use dispatches and teamwork is to maintain any photos/attachments/repeat section entries without having to figure out how to import/export those from data sources.

You guessed the general reasons for needing to make form updates - many of these "work orders" stay open for a long time, and the process has been maturing over the past couple years. For the most part, the older form versions are fine and we can just complete those, but I did make an error early on that required me to "upgrade" a large number of early forms. I have a read-only "end date" field which is automatically set to 1 year in the future. Unfortunately, I had also selected "in the future" range validation for that field. Once the submissions were over 1 year old, the "end date" was in the past, and still read-only, meaning the form could never be submitted.

Right now our process starts with a "Request" form, which dispatches to an "Active" form (Teamwork enabled), which, when completed, dispatches to a supervisor "Review" form. The reason for this is to keep have different rules in place for each form. Users have to complete all the required info before they can "Send" a request form. Then, in the Active phase, most fields are unlocked and the form can be transferred around until the work is complete. Finally the Review form has most fields locked for data integrity (i.e. approver can't edit tech's work).

I was thinking about trying to merge the Active and Review forms so that the tech just "transfers" the form to the reviewer, and it's easier to for the reviewer to transfer the form back if something needs to be fixed. If I do this, I'll have one less form to maintain, but I'll lose the "data integrity" rules. I do have some ideas for that, but first just wanted to see if there would be a clean way to transition existing incomplete forms.

------------------------------
Mark Warwick
Gas Engineer
PG&E

Original Message:
Sent: 07-03-2024 11:21
From: Natalie Tallon
Subject: Upgrading a Form Version

Hi Mark

That sounds...... very painful :(

What you are running into is that a submission can only ever be tied to one version of a form, this is intentional. If we allowed a form to change what it contains and how it works midway through it being used, it could cause data loss/missing data, work needing to be redone, and just general confusion. While some small simple changes may not cause issues, we do have to anticipate worst cases.

I can't think off the top of my head a better way to do this, but here are a few things to consider (some of these you obviously know, but I am putting them here to help others as well).

• A submission in our system is a single data record, from start (dispatch/forms tab) to finish (complete status).
• A submission can have multiple versions, but the data record ID doesn't change when new versions are added.
  • This could be a simple as someone opening a new form and submitting as complete (one submission version), or 
  • It could be a dispatched form going through multiple Teamwork incomplete submissions, (each teamwork incomplete submission is a submission version), or 
  • Some mix of all of this (various number of submission versions). 
• When you use a dispatch data destination you are creating a new submission, with a new data record ID. There is a record of which previous submission created it, but it can make it a bit more complicated if you want to consider/treat two different submissions to be a single submission in how you implement your workflows and store in your systems of record.
• Teamwork is a good way to keep the same data record ID for a submission that has multiple steps/contributors (i.e. form submission versions). This was a key part of the Teamwork feature and we intend to continue building our system to keep to a single submission where it makes sense to do so.
• When a new submission is created, either from dispatching or someone opening a blank form from the app, it creates the submission based on the latest form definition their device has (note this may not even be the latest form version if they are offline, if this is a concern please refer to: Refresh on Form Open).

So knowing all of that, there isn't much wiggle room and you've come up with a clever solution, but as you noted, it isn't very scalable.

What is it about your scenario that makes you need to often update the form?

• Do your submissions cover a long period of time (so a greater chance of a form change happening while it is in use?)
• Is there a need to change your forms quite often?
  
  • Is your process still maturing? or is it just the nature of the process that it needs to continuously change?

These answers may help up come up with something to help.

------------------------------
Natalie Tallon
Product Manager
TrueContext
ntallon@truecontext.com

Original Message:
Sent: 07-02-2024 17:00
From: Mark Warwick
Subject: Upgrading a Form Version

Hi TrueContext Community,

I have a form deployed which uses the Teamwork feature, and we have many incomplete forms available for users to claim.

There are a few different versions of the form out there, and I'm wondering if there are any best practices out there for "upgrading" existing incomplete forms to the newest form version?

My workaround to accomplish form updates is:

1. Add a dispatch destination to the old version of the form which triggers when the user is me. The dispatch creates a updated version of the form with all the answers carried over and it goes to me.
2. Claim or assign the old form on my device, and then Transfer it to trigger the dispatch destination.
3. Open the updated form and Transfer it to put it back in the incomplete pool.
4. Go onto the portal and delete the old form.

This is pretty time-consuming to do for hundreds of forms... don't ask me how I know...

Any better ideas out there??

Cheers,

Mark

------------------------------
Mark Warwick
Gas Engineer
PG&E
------------------------------

Hi Mark,

I have a similar use case where I have data records that are re-used on a monthly/quarterly/annual basis. I've dealt with similar issues in the past as well. One thing you should keep in mind if planning to keep running records for a long time is TC only guarantees your submission data for 365 days (last I checked.) In practice I don't think any of my submissions have been deleted in 6 years, but I think the standard policy is still 365 days.

Anyway, I think based on your first post you already know that you can add new data destinations to existing versions of forms, and when you do, it will re-execute any changed destinations automatically which would generate new records on the latest form version. Unfortunately there's no easy way (that I know of) to unassign a form in the Dispatched state - the API only lets you unassign forms in the Reassigned state.

Depending on what kind of off-platform resources you have available to you, and how far out of the box you're willing to stray, one thing you could look at is the Dispatch API. We use it to do something similar - redispatch data from previous data records. It essentially works like this for us:

1. User completes the form and submits it.
2. Data record (JSON) is saved on our back end.
3. Next user returns to job site to complete the monthly work; they select the original data to dispatch (via a separate form or an in-house app - lots of ways to do this)
4. My back end API does some processing to convert the original data record into a Dispatch record and dispatches it to that user.
5. Cycle restarts from Step 1.

As I said, you have to go pretty far outside the box, but it does give you a lot of power and flexibility. For example, I have a similar review process, but it only triggers on a small % of submissions (due to volume) and is random, so my back-end will randomly choose submissions to mark for review, send the final document to a reviewer, and then the reviewer approves or rejects it. Rejected submissions are automatically sent for edit back to the submitter via the API.

For my use case, my back-end API provides a lot of utility when it comes to dispatching as well - I can re-dispatch a record with all data intact, or I can dispatch it with only chosen data (e.g. don't carry forward test data that a user must re-enter every time), and I can even configure it to convert one form into another (e.g. change questionA in sectionC to questionB in sectionD).

Because you're dispatching existing form data, when you're making changes to your forms, you need to make sure that your old data will be compatible with the new form structure, so renaming/deleting/moving questions can break existing data unless you have processes in place to handle that. The original reason I developed the form conversion feature of my API was to facilitate "updating" old form data to the latest version when it gets dispatched.

I've gotta run but am happy to elaborate on our process further if needed!

Hi Natalie,

Thanks for your thoughtful reply. The part about creating the submission based on the latest form definition on the device is definitely new to me, so that is really good to know.

We are intentionally making use of the Teamwork feature to keep a single "submission" with consistent data record. Another reason I use dispatches and teamwork is to maintain any photos/attachments/repeat section entries without having to figure out how to import/export those from data sources.

You guessed the general reasons for needing to make form updates - many of these "work orders" stay open for a long time, and the process has been maturing over the past couple years. For the most part, the older form versions are fine and we can just complete those, but I did make an error early on that required me to "upgrade" a large number of early forms. I have a read-only "end date" field which is automatically set to 1 year in the future. Unfortunately, I had also selected "in the future" range validation for that field. Once the submissions were over 1 year old, the "end date" was in the past, and still read-only, meaning the form could never be submitted.

Right now our process starts with a "Request" form, which dispatches to an "Active" form (Teamwork enabled), which, when completed, dispatches to a supervisor "Review" form. The reason for this is to keep have different rules in place for each form. Users have to complete all the required info before they can "Send" a request form. Then, in the Active phase, most fields are unlocked and the form can be transferred around until the work is complete. Finally the Review form has most fields locked for data integrity (i.e. approver can't edit tech's work).

I was thinking about trying to merge the Active and Review forms so that the tech just "transfers" the form to the reviewer, and it's easier to for the reviewer to transfer the form back if something needs to be fixed. If I do this, I'll have one less form to maintain, but I'll lose the "data integrity" rules. I do have some ideas for that, but first just wanted to see if there would be a clean way to transition existing incomplete forms.

------------------------------
Mark Warwick
Gas Engineer
PG&E

Original Message:
Sent: 07-03-2024 11:21
From: Natalie Tallon
Subject: Upgrading a Form Version

Hi Mark

That sounds...... very painful :(

What you are running into is that a submission can only ever be tied to one version of a form, this is intentional. If we allowed a form to change what it contains and how it works midway through it being used, it could cause data loss/missing data, work needing to be redone, and just general confusion. While some small simple changes may not cause issues, we do have to anticipate worst cases.

I can't think off the top of my head a better way to do this, but here are a few things to consider (some of these you obviously know, but I am putting them here to help others as well).

• A submission in our system is a single data record, from start (dispatch/forms tab) to finish (complete status).
• A submission can have multiple versions, but the data record ID doesn't change when new versions are added.
  • This could be a simple as someone opening a new form and submitting as complete (one submission version), or 
  • It could be a dispatched form going through multiple Teamwork incomplete submissions, (each teamwork incomplete submission is a submission version), or 
  • Some mix of all of this (various number of submission versions). 
• When you use a dispatch data destination you are creating a new submission, with a new data record ID. There is a record of which previous submission created it, but it can make it a bit more complicated if you want to consider/treat two different submissions to be a single submission in how you implement your workflows and store in your systems of record.
• Teamwork is a good way to keep the same data record ID for a submission that has multiple steps/contributors (i.e. form submission versions). This was a key part of the Teamwork feature and we intend to continue building our system to keep to a single submission where it makes sense to do so.
• When a new submission is created, either from dispatching or someone opening a blank form from the app, it creates the submission based on the latest form definition their device has (note this may not even be the latest form version if they are offline, if this is a concern please refer to: Refresh on Form Open).

So knowing all of that, there isn't much wiggle room and you've come up with a clever solution, but as you noted, it isn't very scalable.

What is it about your scenario that makes you need to often update the form?

• Do your submissions cover a long period of time (so a greater chance of a form change happening while it is in use?)
• Is there a need to change your forms quite often?
  
  • Is your process still maturing? or is it just the nature of the process that it needs to continuously change?

These answers may help up come up with something to help.

------------------------------
Natalie Tallon
Product Manager
TrueContext
ntallon@truecontext.com

Original Message:
Sent: 07-02-2024 17:00
From: Mark Warwick
Subject: Upgrading a Form Version

Hi TrueContext Community,

I have a form deployed which uses the Teamwork feature, and we have many incomplete forms available for users to claim.

There are a few different versions of the form out there, and I'm wondering if there are any best practices out there for "upgrading" existing incomplete forms to the newest form version?

My workaround to accomplish form updates is:

1. Add a dispatch destination to the old version of the form which triggers when the user is me. The dispatch creates a updated version of the form with all the answers carried over and it goes to me.
2. Claim or assign the old form on my device, and then Transfer it to trigger the dispatch destination.
3. Open the updated form and Transfer it to put it back in the incomplete pool.
4. Go onto the portal and delete the old form.

This is pretty time-consuming to do for hundreds of forms... don't ask me how I know...

Any better ideas out there??

Cheers,

Mark

------------------------------
Mark Warwick
Gas Engineer
PG&E
------------------------------

Hi Calvin,

That sounds like a solid process. The extent of what I could do for a "backend system" would be using Microsoft Power Automate, but I think what you're describing would be possible. It's the photos and attachments that trip me up - I believe they can be dispatched via the API but it's tricky.

The 365 day storage limitation is definitely something for me to consider. In that case it seems it would be wise to re-dispatch or otherwise refresh the forms on an annual basis to make sure they are staying current. The other reason I have found for this is that the default app search only returns results from the past year. To get anything older than a year you have to explicitly set the dates in the advanced search parameters.

Cheers,
Mark

Hi Mark,

I have a similar use case where I have data records that are re-used on a monthly/quarterly/annual basis. I've dealt with similar issues in the past as well. One thing you should keep in mind if planning to keep running records for a long time is TC only guarantees your submission data for 365 days (last I checked.) In practice I don't think any of my submissions have been deleted in 6 years, but I think the standard policy is still 365 days.

Anyway, I think based on your first post you already know that you can add new data destinations to existing versions of forms, and when you do, it will re-execute any changed destinations automatically which would generate new records on the latest form version. Unfortunately there's no easy way (that I know of) to unassign a form in the Dispatched state - the API only lets you unassign forms in the Reassigned state.

Depending on what kind of off-platform resources you have available to you, and how far out of the box you're willing to stray, one thing you could look at is the Dispatch API. We use it to do something similar - redispatch data from previous data records. It essentially works like this for us:

1. User completes the form and submits it.
2. Data record (JSON) is saved on our back end.
3. Next user returns to job site to complete the monthly work; they select the original data to dispatch (via a separate form or an in-house app - lots of ways to do this)
4. My back end API does some processing to convert the original data record into a Dispatch record and dispatches it to that user.
5. Cycle restarts from Step 1.

As I said, you have to go pretty far outside the box, but it does give you a lot of power and flexibility. For example, I have a similar review process, but it only triggers on a small % of submissions (due to volume) and is random, so my back-end will randomly choose submissions to mark for review, send the final document to a reviewer, and then the reviewer approves or rejects it. Rejected submissions are automatically sent for edit back to the submitter via the API.

For my use case, my back-end API provides a lot of utility when it comes to dispatching as well - I can re-dispatch a record with all data intact, or I can dispatch it with only chosen data (e.g. don't carry forward test data that a user must re-enter every time), and I can even configure it to convert one form into another (e.g. change questionA in sectionC to questionB in sectionD).

Because you're dispatching existing form data, when you're making changes to your forms, you need to make sure that your old data will be compatible with the new form structure, so renaming/deleting/moving questions can break existing data unless you have processes in place to handle that. The original reason I developed the form conversion feature of my API was to facilitate "updating" old form data to the latest version when it gets dispatched.

I've gotta run but am happy to elaborate on our process further if needed!

Hi Natalie,

Thanks for your thoughtful reply. The part about creating the submission based on the latest form definition on the device is definitely new to me, so that is really good to know.

We are intentionally making use of the Teamwork feature to keep a single "submission" with consistent data record. Another reason I use dispatches and teamwork is to maintain any photos/attachments/repeat section entries without having to figure out how to import/export those from data sources.

You guessed the general reasons for needing to make form updates - many of these "work orders" stay open for a long time, and the process has been maturing over the past couple years. For the most part, the older form versions are fine and we can just complete those, but I did make an error early on that required me to "upgrade" a large number of early forms. I have a read-only "end date" field which is automatically set to 1 year in the future. Unfortunately, I had also selected "in the future" range validation for that field. Once the submissions were over 1 year old, the "end date" was in the past, and still read-only, meaning the form could never be submitted.

Right now our process starts with a "Request" form, which dispatches to an "Active" form (Teamwork enabled), which, when completed, dispatches to a supervisor "Review" form. The reason for this is to keep have different rules in place for each form. Users have to complete all the required info before they can "Send" a request form. Then, in the Active phase, most fields are unlocked and the form can be transferred around until the work is complete. Finally the Review form has most fields locked for data integrity (i.e. approver can't edit tech's work).

I was thinking about trying to merge the Active and Review forms so that the tech just "transfers" the form to the reviewer, and it's easier to for the reviewer to transfer the form back if something needs to be fixed. If I do this, I'll have one less form to maintain, but I'll lose the "data integrity" rules. I do have some ideas for that, but first just wanted to see if there would be a clean way to transition existing incomplete forms.

------------------------------
Mark Warwick
Gas Engineer
PG&E

Original Message:
Sent: 07-03-2024 11:21
From: Natalie Tallon
Subject: Upgrading a Form Version

Hi Mark

That sounds...... very painful :(

What you are running into is that a submission can only ever be tied to one version of a form, this is intentional. If we allowed a form to change what it contains and how it works midway through it being used, it could cause data loss/missing data, work needing to be redone, and just general confusion. While some small simple changes may not cause issues, we do have to anticipate worst cases.

I can't think off the top of my head a better way to do this, but here are a few things to consider (some of these you obviously know, but I am putting them here to help others as well).

• A submission in our system is a single data record, from start (dispatch/forms tab) to finish (complete status).
• A submission can have multiple versions, but the data record ID doesn't change when new versions are added.
  • This could be a simple as someone opening a new form and submitting as complete (one submission version), or 
  • It could be a dispatched form going through multiple Teamwork incomplete submissions, (each teamwork incomplete submission is a submission version), or 
  • Some mix of all of this (various number of submission versions). 
• When you use a dispatch data destination you are creating a new submission, with a new data record ID. There is a record of which previous submission created it, but it can make it a bit more complicated if you want to consider/treat two different submissions to be a single submission in how you implement your workflows and store in your systems of record.
• Teamwork is a good way to keep the same data record ID for a submission that has multiple steps/contributors (i.e. form submission versions). This was a key part of the Teamwork feature and we intend to continue building our system to keep to a single submission where it makes sense to do so.
• When a new submission is created, either from dispatching or someone opening a blank form from the app, it creates the submission based on the latest form definition their device has (note this may not even be the latest form version if they are offline, if this is a concern please refer to: Refresh on Form Open).

So knowing all of that, there isn't much wiggle room and you've come up with a clever solution, but as you noted, it isn't very scalable.

What is it about your scenario that makes you need to often update the form?

• Do your submissions cover a long period of time (so a greater chance of a form change happening while it is in use?)
• Is there a need to change your forms quite often?
  
  • Is your process still maturing? or is it just the nature of the process that it needs to continuously change?

These answers may help up come up with something to help.

------------------------------
Natalie Tallon
Product Manager
TrueContext
ntallon@truecontext.com

Original Message:
Sent: 07-02-2024 17:00
From: Mark Warwick
Subject: Upgrading a Form Version

Hi TrueContext Community,

I have a form deployed which uses the Teamwork feature, and we have many incomplete forms available for users to claim.

There are a few different versions of the form out there, and I'm wondering if there are any best practices out there for "upgrading" existing incomplete forms to the newest form version?

My workaround to accomplish form updates is:

1. Add a dispatch destination to the old version of the form which triggers when the user is me. The dispatch creates a updated version of the form with all the answers carried over and it goes to me.
2. Claim or assign the old form on my device, and then Transfer it to trigger the dispatch destination.
3. Open the updated form and Transfer it to put it back in the incomplete pool.
4. Go onto the portal and delete the old form.

This is pretty time-consuming to do for hundreds of forms... don't ask me how I know...

Any better ideas out there??

Cheers,

Mark

------------------------------
Mark Warwick
Gas Engineer
PG&E
------------------------------

Hey Mark and Calvin,

Just wanted to make one clarifying point on the conversation. Form submission metadata remains accessible in the TrueContext production cloud for at least 365 days from the date of form submission, or until the data is deleted. You can see our Customer Data Lifecycle article on our documentation page to review the policies in more detail.

Hi Calvin,

That sounds like a solid process. The extent of what I could do for a "backend system" would be using Microsoft Power Automate, but I think what you're describing would be possible. It's the photos and attachments that trip me up - I believe they can be dispatched via the API but it's tricky.

The 365 day storage limitation is definitely something for me to consider. In that case it seems it would be wise to re-dispatch or otherwise refresh the forms on an annual basis to make sure they are staying current. The other reason I have found for this is that the default app search only returns results from the past year. To get anything older than a year you have to explicitly set the dates in the advanced search parameters.

Cheers,
Mark

Hi Mark,

I have a similar use case where I have data records that are re-used on a monthly/quarterly/annual basis. I've dealt with similar issues in the past as well. One thing you should keep in mind if planning to keep running records for a long time is TC only guarantees your submission data for 365 days (last I checked.) In practice I don't think any of my submissions have been deleted in 6 years, but I think the standard policy is still 365 days.

Anyway, I think based on your first post you already know that you can add new data destinations to existing versions of forms, and when you do, it will re-execute any changed destinations automatically which would generate new records on the latest form version. Unfortunately there's no easy way (that I know of) to unassign a form in the Dispatched state - the API only lets you unassign forms in the Reassigned state.

Depending on what kind of off-platform resources you have available to you, and how far out of the box you're willing to stray, one thing you could look at is the Dispatch API. We use it to do something similar - redispatch data from previous data records. It essentially works like this for us:

1. User completes the form and submits it.
2. Data record (JSON) is saved on our back end.
3. Next user returns to job site to complete the monthly work; they select the original data to dispatch (via a separate form or an in-house app - lots of ways to do this)
4. My back end API does some processing to convert the original data record into a Dispatch record and dispatches it to that user.
5. Cycle restarts from Step 1.

As I said, you have to go pretty far outside the box, but it does give you a lot of power and flexibility. For example, I have a similar review process, but it only triggers on a small % of submissions (due to volume) and is random, so my back-end will randomly choose submissions to mark for review, send the final document to a reviewer, and then the reviewer approves or rejects it. Rejected submissions are automatically sent for edit back to the submitter via the API.

For my use case, my back-end API provides a lot of utility when it comes to dispatching as well - I can re-dispatch a record with all data intact, or I can dispatch it with only chosen data (e.g. don't carry forward test data that a user must re-enter every time), and I can even configure it to convert one form into another (e.g. change questionA in sectionC to questionB in sectionD).

Because you're dispatching existing form data, when you're making changes to your forms, you need to make sure that your old data will be compatible with the new form structure, so renaming/deleting/moving questions can break existing data unless you have processes in place to handle that. The original reason I developed the form conversion feature of my API was to facilitate "updating" old form data to the latest version when it gets dispatched.

I've gotta run but am happy to elaborate on our process further if needed!

Hi Natalie,

Thanks for your thoughtful reply. The part about creating the submission based on the latest form definition on the device is definitely new to me, so that is really good to know.

We are intentionally making use of the Teamwork feature to keep a single "submission" with consistent data record. Another reason I use dispatches and teamwork is to maintain any photos/attachments/repeat section entries without having to figure out how to import/export those from data sources.

You guessed the general reasons for needing to make form updates - many of these "work orders" stay open for a long time, and the process has been maturing over the past couple years. For the most part, the older form versions are fine and we can just complete those, but I did make an error early on that required me to "upgrade" a large number of early forms. I have a read-only "end date" field which is automatically set to 1 year in the future. Unfortunately, I had also selected "in the future" range validation for that field. Once the submissions were over 1 year old, the "end date" was in the past, and still read-only, meaning the form could never be submitted.

Right now our process starts with a "Request" form, which dispatches to an "Active" form (Teamwork enabled), which, when completed, dispatches to a supervisor "Review" form. The reason for this is to keep have different rules in place for each form. Users have to complete all the required info before they can "Send" a request form. Then, in the Active phase, most fields are unlocked and the form can be transferred around until the work is complete. Finally the Review form has most fields locked for data integrity (i.e. approver can't edit tech's work).

I was thinking about trying to merge the Active and Review forms so that the tech just "transfers" the form to the reviewer, and it's easier to for the reviewer to transfer the form back if something needs to be fixed. If I do this, I'll have one less form to maintain, but I'll lose the "data integrity" rules. I do have some ideas for that, but first just wanted to see if there would be a clean way to transition existing incomplete forms.

------------------------------
Mark Warwick
Gas Engineer
PG&E

Original Message:
Sent: 07-03-2024 11:21
From: Natalie Tallon
Subject: Upgrading a Form Version

Hi Mark

That sounds...... very painful :(

What you are running into is that a submission can only ever be tied to one version of a form, this is intentional. If we allowed a form to change what it contains and how it works midway through it being used, it could cause data loss/missing data, work needing to be redone, and just general confusion. While some small simple changes may not cause issues, we do have to anticipate worst cases.

I can't think off the top of my head a better way to do this, but here are a few things to consider (some of these you obviously know, but I am putting them here to help others as well).

• A submission in our system is a single data record, from start (dispatch/forms tab) to finish (complete status).
• A submission can have multiple versions, but the data record ID doesn't change when new versions are added.
  • This could be a simple as someone opening a new form and submitting as complete (one submission version), or 
  • It could be a dispatched form going through multiple Teamwork incomplete submissions, (each teamwork incomplete submission is a submission version), or 
  • Some mix of all of this (various number of submission versions). 
• When you use a dispatch data destination you are creating a new submission, with a new data record ID. There is a record of which previous submission created it, but it can make it a bit more complicated if you want to consider/treat two different submissions to be a single submission in how you implement your workflows and store in your systems of record.
• Teamwork is a good way to keep the same data record ID for a submission that has multiple steps/contributors (i.e. form submission versions). This was a key part of the Teamwork feature and we intend to continue building our system to keep to a single submission where it makes sense to do so.
• When a new submission is created, either from dispatching or someone opening a blank form from the app, it creates the submission based on the latest form definition their device has (note this may not even be the latest form version if they are offline, if this is a concern please refer to: Refresh on Form Open).

So knowing all of that, there isn't much wiggle room and you've come up with a clever solution, but as you noted, it isn't very scalable.

What is it about your scenario that makes you need to often update the form?

• Do your submissions cover a long period of time (so a greater chance of a form change happening while it is in use?)
• Is there a need to change your forms quite often?
  
  • Is your process still maturing? or is it just the nature of the process that it needs to continuously change?

These answers may help up come up with something to help.

------------------------------
Natalie Tallon
Product Manager
TrueContext
ntallon@truecontext.com

Original Message:
Sent: 07-02-2024 17:00
From: Mark Warwick
Subject: Upgrading a Form Version

Hi TrueContext Community,

I have a form deployed which uses the Teamwork feature, and we have many incomplete forms available for users to claim.

There are a few different versions of the form out there, and I'm wondering if there are any best practices out there for "upgrading" existing incomplete forms to the newest form version?

My workaround to accomplish form updates is:

1. Add a dispatch destination to the old version of the form which triggers when the user is me. The dispatch creates a updated version of the form with all the answers carried over and it goes to me.
2. Claim or assign the old form on my device, and then Transfer it to trigger the dispatch destination.
3. Open the updated form and Transfer it to put it back in the incomplete pool.
4. Go onto the portal and delete the old form.

This is pretty time-consuming to do for hundreds of forms... don't ask me how I know...

Any better ideas out there??

Cheers,

Mark

------------------------------
Mark Warwick
Gas Engineer
PG&E
------------------------------

Hi David,

Quick question on this - for a Teamwork enabled form, am I correct to assume that each time an incomplete form is transferred, this 365 minimum restarts? Are incomplete forms handled any differently?

Hey Mark and Calvin,

Just wanted to make one clarifying point on the conversation. Form submission metadata remains accessible in the TrueContext production cloud for at least 365 days from the date of form submission, or until the data is deleted. You can see our Customer Data Lifecycle article on our documentation page to review the policies in more detail.

Hi Calvin,

That sounds like a solid process. The extent of what I could do for a "backend system" would be using Microsoft Power Automate, but I think what you're describing would be possible. It's the photos and attachments that trip me up - I believe they can be dispatched via the API but it's tricky.

The 365 day storage limitation is definitely something for me to consider. In that case it seems it would be wise to re-dispatch or otherwise refresh the forms on an annual basis to make sure they are staying current. The other reason I have found for this is that the default app search only returns results from the past year. To get anything older than a year you have to explicitly set the dates in the advanced search parameters.

Cheers,
Mark

Hi Mark,

I have a similar use case where I have data records that are re-used on a monthly/quarterly/annual basis. I've dealt with similar issues in the past as well. One thing you should keep in mind if planning to keep running records for a long time is TC only guarantees your submission data for 365 days (last I checked.) In practice I don't think any of my submissions have been deleted in 6 years, but I think the standard policy is still 365 days.

Anyway, I think based on your first post you already know that you can add new data destinations to existing versions of forms, and when you do, it will re-execute any changed destinations automatically which would generate new records on the latest form version. Unfortunately there's no easy way (that I know of) to unassign a form in the Dispatched state - the API only lets you unassign forms in the Reassigned state.

Depending on what kind of off-platform resources you have available to you, and how far out of the box you're willing to stray, one thing you could look at is the Dispatch API. We use it to do something similar - redispatch data from previous data records. It essentially works like this for us:

1. User completes the form and submits it.
2. Data record (JSON) is saved on our back end.
3. Next user returns to job site to complete the monthly work; they select the original data to dispatch (via a separate form or an in-house app - lots of ways to do this)
4. My back end API does some processing to convert the original data record into a Dispatch record and dispatches it to that user.
5. Cycle restarts from Step 1.

As I said, you have to go pretty far outside the box, but it does give you a lot of power and flexibility. For example, I have a similar review process, but it only triggers on a small % of submissions (due to volume) and is random, so my back-end will randomly choose submissions to mark for review, send the final document to a reviewer, and then the reviewer approves or rejects it. Rejected submissions are automatically sent for edit back to the submitter via the API.

For my use case, my back-end API provides a lot of utility when it comes to dispatching as well - I can re-dispatch a record with all data intact, or I can dispatch it with only chosen data (e.g. don't carry forward test data that a user must re-enter every time), and I can even configure it to convert one form into another (e.g. change questionA in sectionC to questionB in sectionD).

Because you're dispatching existing form data, when you're making changes to your forms, you need to make sure that your old data will be compatible with the new form structure, so renaming/deleting/moving questions can break existing data unless you have processes in place to handle that. The original reason I developed the form conversion feature of my API was to facilitate "updating" old form data to the latest version when it gets dispatched.

I've gotta run but am happy to elaborate on our process further if needed!

Hi Natalie,

Thanks for your thoughtful reply. The part about creating the submission based on the latest form definition on the device is definitely new to me, so that is really good to know.

We are intentionally making use of the Teamwork feature to keep a single "submission" with consistent data record. Another reason I use dispatches and teamwork is to maintain any photos/attachments/repeat section entries without having to figure out how to import/export those from data sources.

You guessed the general reasons for needing to make form updates - many of these "work orders" stay open for a long time, and the process has been maturing over the past couple years. For the most part, the older form versions are fine and we can just complete those, but I did make an error early on that required me to "upgrade" a large number of early forms. I have a read-only "end date" field which is automatically set to 1 year in the future. Unfortunately, I had also selected "in the future" range validation for that field. Once the submissions were over 1 year old, the "end date" was in the past, and still read-only, meaning the form could never be submitted.

Right now our process starts with a "Request" form, which dispatches to an "Active" form (Teamwork enabled), which, when completed, dispatches to a supervisor "Review" form. The reason for this is to keep have different rules in place for each form. Users have to complete all the required info before they can "Send" a request form. Then, in the Active phase, most fields are unlocked and the form can be transferred around until the work is complete. Finally the Review form has most fields locked for data integrity (i.e. approver can't edit tech's work).

I was thinking about trying to merge the Active and Review forms so that the tech just "transfers" the form to the reviewer, and it's easier to for the reviewer to transfer the form back if something needs to be fixed. If I do this, I'll have one less form to maintain, but I'll lose the "data integrity" rules. I do have some ideas for that, but first just wanted to see if there would be a clean way to transition existing incomplete forms.

------------------------------
Mark Warwick
Gas Engineer
PG&E

Original Message:
Sent: 07-03-2024 11:21
From: Natalie Tallon
Subject: Upgrading a Form Version

Hi Mark

That sounds...... very painful :(

What you are running into is that a submission can only ever be tied to one version of a form, this is intentional. If we allowed a form to change what it contains and how it works midway through it being used, it could cause data loss/missing data, work needing to be redone, and just general confusion. While some small simple changes may not cause issues, we do have to anticipate worst cases.

I can't think off the top of my head a better way to do this, but here are a few things to consider (some of these you obviously know, but I am putting them here to help others as well).

• A submission in our system is a single data record, from start (dispatch/forms tab) to finish (complete status).
• A submission can have multiple versions, but the data record ID doesn't change when new versions are added.
  • This could be a simple as someone opening a new form and submitting as complete (one submission version), or 
  • It could be a dispatched form going through multiple Teamwork incomplete submissions, (each teamwork incomplete submission is a submission version), or 
  • Some mix of all of this (various number of submission versions). 
• When you use a dispatch data destination you are creating a new submission, with a new data record ID. There is a record of which previous submission created it, but it can make it a bit more complicated if you want to consider/treat two different submissions to be a single submission in how you implement your workflows and store in your systems of record.
• Teamwork is a good way to keep the same data record ID for a submission that has multiple steps/contributors (i.e. form submission versions). This was a key part of the Teamwork feature and we intend to continue building our system to keep to a single submission where it makes sense to do so.
• When a new submission is created, either from dispatching or someone opening a blank form from the app, it creates the submission based on the latest form definition their device has (note this may not even be the latest form version if they are offline, if this is a concern please refer to: Refresh on Form Open).

So knowing all of that, there isn't much wiggle room and you've come up with a clever solution, but as you noted, it isn't very scalable.

What is it about your scenario that makes you need to often update the form?

• Do your submissions cover a long period of time (so a greater chance of a form change happening while it is in use?)
• Is there a need to change your forms quite often?
  
  • Is your process still maturing? or is it just the nature of the process that it needs to continuously change?

These answers may help up come up with something to help.

------------------------------
Natalie Tallon
Product Manager
TrueContext
ntallon@truecontext.com

Original Message:
Sent: 07-02-2024 17:00
From: Mark Warwick
Subject: Upgrading a Form Version

Hi TrueContext Community,

I have a form deployed which uses the Teamwork feature, and we have many incomplete forms available for users to claim.

There are a few different versions of the form out there, and I'm wondering if there are any best practices out there for "upgrading" existing incomplete forms to the newest form version?

My workaround to accomplish form updates is:

1. Add a dispatch destination to the old version of the form which triggers when the user is me. The dispatch creates a updated version of the form with all the answers carried over and it goes to me.
2. Claim or assign the old form on my device, and then Transfer it to trigger the dispatch destination.
3. Open the updated form and Transfer it to put it back in the incomplete pool.
4. Go onto the portal and delete the old form.

This is pretty time-consuming to do for hundreds of forms... don't ask me how I know...

Any better ideas out there??

Cheers,

Mark

------------------------------
Mark Warwick
Gas Engineer
PG&E
------------------------------

Hi Calvin,

That sounds like a solid process. The extent of what I could do for a "backend system" would be using Microsoft Power Automate, but I think what you're describing would be possible. It's the photos and attachments that trip me up - I believe they can be dispatched via the API but it's tricky.

The 365 day storage limitation is definitely something for me to consider. In that case it seems it would be wise to re-dispatch or otherwise refresh the forms on an annual basis to make sure they are staying current. The other reason I have found for this is that the default app search only returns results from the past year. To get anything older than a year you have to explicitly set the dates in the advanced search parameters.

Cheers,
Mark

Hi Mark,

I have a similar use case where I have data records that are re-used on a monthly/quarterly/annual basis. I've dealt with similar issues in the past as well. One thing you should keep in mind if planning to keep running records for a long time is TC only guarantees your submission data for 365 days (last I checked.) In practice I don't think any of my submissions have been deleted in 6 years, but I think the standard policy is still 365 days.

Anyway, I think based on your first post you already know that you can add new data destinations to existing versions of forms, and when you do, it will re-execute any changed destinations automatically which would generate new records on the latest form version. Unfortunately there's no easy way (that I know of) to unassign a form in the Dispatched state - the API only lets you unassign forms in the Reassigned state.

Depending on what kind of off-platform resources you have available to you, and how far out of the box you're willing to stray, one thing you could look at is the Dispatch API. We use it to do something similar - redispatch data from previous data records. It essentially works like this for us:

1. User completes the form and submits it.
2. Data record (JSON) is saved on our back end.
3. Next user returns to job site to complete the monthly work; they select the original data to dispatch (via a separate form or an in-house app - lots of ways to do this)
4. My back end API does some processing to convert the original data record into a Dispatch record and dispatches it to that user.
5. Cycle restarts from Step 1.

As I said, you have to go pretty far outside the box, but it does give you a lot of power and flexibility. For example, I have a similar review process, but it only triggers on a small % of submissions (due to volume) and is random, so my back-end will randomly choose submissions to mark for review, send the final document to a reviewer, and then the reviewer approves or rejects it. Rejected submissions are automatically sent for edit back to the submitter via the API.

For my use case, my back-end API provides a lot of utility when it comes to dispatching as well - I can re-dispatch a record with all data intact, or I can dispatch it with only chosen data (e.g. don't carry forward test data that a user must re-enter every time), and I can even configure it to convert one form into another (e.g. change questionA in sectionC to questionB in sectionD).

Because you're dispatching existing form data, when you're making changes to your forms, you need to make sure that your old data will be compatible with the new form structure, so renaming/deleting/moving questions can break existing data unless you have processes in place to handle that. The original reason I developed the form conversion feature of my API was to facilitate "updating" old form data to the latest version when it gets dispatched.

I've gotta run but am happy to elaborate on our process further if needed!

Hi Natalie,

Thanks for your thoughtful reply. The part about creating the submission based on the latest form definition on the device is definitely new to me, so that is really good to know.

We are intentionally making use of the Teamwork feature to keep a single "submission" with consistent data record. Another reason I use dispatches and teamwork is to maintain any photos/attachments/repeat section entries without having to figure out how to import/export those from data sources.

You guessed the general reasons for needing to make form updates - many of these "work orders" stay open for a long time, and the process has been maturing over the past couple years. For the most part, the older form versions are fine and we can just complete those, but I did make an error early on that required me to "upgrade" a large number of early forms. I have a read-only "end date" field which is automatically set to 1 year in the future. Unfortunately, I had also selected "in the future" range validation for that field. Once the submissions were over 1 year old, the "end date" was in the past, and still read-only, meaning the form could never be submitted.

Right now our process starts with a "Request" form, which dispatches to an "Active" form (Teamwork enabled), which, when completed, dispatches to a supervisor "Review" form. The reason for this is to keep have different rules in place for each form. Users have to complete all the required info before they can "Send" a request form. Then, in the Active phase, most fields are unlocked and the form can be transferred around until the work is complete. Finally the Review form has most fields locked for data integrity (i.e. approver can't edit tech's work).

I was thinking about trying to merge the Active and Review forms so that the tech just "transfers" the form to the reviewer, and it's easier to for the reviewer to transfer the form back if something needs to be fixed. If I do this, I'll have one less form to maintain, but I'll lose the "data integrity" rules. I do have some ideas for that, but first just wanted to see if there would be a clean way to transition existing incomplete forms.

------------------------------
Mark Warwick
Gas Engineer
PG&E

Original Message:
Sent: 07-03-2024 11:21
From: Natalie Tallon
Subject: Upgrading a Form Version

Hi Mark

That sounds...... very painful :(

What you are running into is that a submission can only ever be tied to one version of a form, this is intentional. If we allowed a form to change what it contains and how it works midway through it being used, it could cause data loss/missing data, work needing to be redone, and just general confusion. While some small simple changes may not cause issues, we do have to anticipate worst cases.

I can't think off the top of my head a better way to do this, but here are a few things to consider (some of these you obviously know, but I am putting them here to help others as well).

• A submission in our system is a single data record, from start (dispatch/forms tab) to finish (complete status).
• A submission can have multiple versions, but the data record ID doesn't change when new versions are added.
  • This could be a simple as someone opening a new form and submitting as complete (one submission version), or 
  • It could be a dispatched form going through multiple Teamwork incomplete submissions, (each teamwork incomplete submission is a submission version), or 
  • Some mix of all of this (various number of submission versions). 
• When you use a dispatch data destination you are creating a new submission, with a new data record ID. There is a record of which previous submission created it, but it can make it a bit more complicated if you want to consider/treat two different submissions to be a single submission in how you implement your workflows and store in your systems of record.
• Teamwork is a good way to keep the same data record ID for a submission that has multiple steps/contributors (i.e. form submission versions). This was a key part of the Teamwork feature and we intend to continue building our system to keep to a single submission where it makes sense to do so.
• When a new submission is created, either from dispatching or someone opening a blank form from the app, it creates the submission based on the latest form definition their device has (note this may not even be the latest form version if they are offline, if this is a concern please refer to: Refresh on Form Open).

So knowing all of that, there isn't much wiggle room and you've come up with a clever solution, but as you noted, it isn't very scalable.

What is it about your scenario that makes you need to often update the form?

• Do your submissions cover a long period of time (so a greater chance of a form change happening while it is in use?)
• Is there a need to change your forms quite often?
  
  • Is your process still maturing? or is it just the nature of the process that it needs to continuously change?

These answers may help up come up with something to help.

------------------------------
Natalie Tallon
Product Manager
TrueContext
ntallon@truecontext.com

Original Message:
Sent: 07-02-2024 17:00
From: Mark Warwick
Subject: Upgrading a Form Version

Hi TrueContext Community,

I have a form deployed which uses the Teamwork feature, and we have many incomplete forms available for users to claim.

There are a few different versions of the form out there, and I'm wondering if there are any best practices out there for "upgrading" existing incomplete forms to the newest form version?

My workaround to accomplish form updates is:

1. Add a dispatch destination to the old version of the form which triggers when the user is me. The dispatch creates a updated version of the form with all the answers carried over and it goes to me.
2. Claim or assign the old form on my device, and then Transfer it to trigger the dispatch destination.
3. Open the updated form and Transfer it to put it back in the incomplete pool.
4. Go onto the portal and delete the old form.

This is pretty time-consuming to do for hundreds of forms... don't ask me how I know...

Any better ideas out there??

Cheers,

Mark

------------------------------
Mark Warwick
Gas Engineer
PG&E
------------------------------