In this blog series, you will learn how to apply a "State Pattern" based on Azure Queue Storage to your Flow solution architecture. This allows you to design Flows in a more sophisticated way and overcome general challenges, you were not able to resolve before reading this. Hope, that's reason to read..
How does a "State Machine" look like?
State Patterns are basically here to resolve mathematical problems. So they don't "look" that much, but I tried to visualize the high level solution architecture in color, so you can get an idea of the whole thing and come back, when reading the details later on.
When building more complex Flows you will face some of the infamous challenges like the 30 days run duration limitation, challenges in error handling and retry policies, limitations in nested actions and max action count, but the most painful is the non-existence of a State Machine mechanism.
How can we build enterprise-level processes with a service, which is there to handle processes, which implicates state, but doesn’t provide the essentials?
In this blog article series, you will learn how to apply a "State Pattern" based on Azure Queue Storage to your Flow solution architecture. We are going to cover the creation process of required Storage Queues, Controller and Worker Flows and establish Flow to Flow communication to transfer state as a JSON message object.
Azure Storage Queues are providing us with enterprise software capabilities at our fingertips for nearly no money. The impact on solution stability of this robust, highly scalable and easy to use lego brick, is insane. These connectors belong to the set of Standard Connectors and don't require a premium plan for each user.
All of this allows you to design Flows in a more sophisticated way and overcome general challenges. Come to this session to learn more about how this elegant software pattern can be applied to Flow.
Content Index
If you are familiar with the concept and the above schema makes sense for you, then jump directly into the details and learn more about the single "lego bricks" making out the solution.
> Implementation Guide (WiP)
If you would like to learn more about the state pattern and develop a deeper understanding, please read on here. I create some great intro content, so we can all profit and build better Flows with less headache.
Introduction
First, I want to give my credits to Serge Luca aka Dr. Flow, who inspired my with the brilliant idea of a State Machine Pattern in Flow based on the HTTP trigger.
To Christian Flückiger, my co-worker and brain behind the details and last but not least my buddy Jussi "Juicy" Mori for dropping me into
situations causing me to use Flow heavily from head to toe.
Let's start with some intro content about the "State Pattern" topic in general and develop an understanding about our business usecase, the tech involved, challenges and benefits. It's important to get a fundamental understanding of the theory, only then you can suceed in implementation later on and have some fun.
To warm up, let me introduce you to some basic concepts and questions around the topic a little later in this article:
What are the top benefits of the State Pattern?
How does such a "State Machine" looks like?
Why do you need a State Machine in Flow?
What's a business usecase for a State Machine?
Definition of the term "State Machine"
Common challenges in Flow and my point of view
What's the benefit of Azure Queue Storage?
What are the benefits for Flow Solution Design?
Why do I write this series?
Why do I need a State Machine in Flow?
When building more complex Flows you will face some of the infamous challenges like the 30 days run duration limitation, challenges in error handling and retry policies, limitations in nested actions and max action count, but the most painful is the non-existence of a State Machine mechanism.
How can we build enterprise-level processes with a service, which is there to handle processes, which implicates state, but doesn’t provide the essentials?
In this blog article series, you will learn how to apply a "State Pattern" based on Azure Queue Storage to your Flow solution architecture. We are going to cover the creation process of required Storage Queues, Controller and Worker Flows and establish Flow to Flow communication to transfer state as a JSON message object.
All of this allows you to design Flows in a more sophisticated way and overcome general challenges. Come to this session to learn more about how this elegant software pattern can be applied to Flow.
What's a business usecase for a State Machine?
Some of the requirements of an enterprise level document approval process, as it could be similar in many enterprises.
The Flow insiders know, that this list contains a lot of challenges and the main challenge is to connect all the individual ingridients to one final pie:
Dynamic initialization form
Parallel and serial approval task flow
Dynamic number of approvers
Approval due date
Automated reminder e-mail based on approval date
Approval escalation/timeout when no response
Centralized and "immutable" approval log
SharePoint integrated via Flow launch panel
Advanced error handling with notification to initiator
Definition of term "State Machine"
I found this definition is a good start for the further reading of this blog series:
"A state machine is any device storing the status of something at a given time. The status changes based on inputs, providing the resulting output for the implemented changes. A finite state machine has finite internal memory. Input symbols are read in a sequence producing an output feature in the form of a user interface.
State machines are represented using state diagrams. The output of a state machine is a function of the input and the current state. State machines play a significant role in areas such as electrical engineering, linguistics, computer science, philosophy, biology, mathematics, and logic. They are best used in the modeling of application behavior, software engineering, design of hardware digital systems, network protocols, compilers, and the study of computation and languages."
> accoding to https://www.techopedia.com/definition/16447/state-machine
Common "Challenges" in Flow
Some of the common issues most of us faced somewhen:
Limitation of Flow run duration / 30 days
Until January 2019, the maximum run duration for a Flow is 30 days. After this amount of time, a running Flow gets canceled.
On a first sight, this might look troubling. But but from a design an operations point of view, it makes sense. It's better to store state outside of Flow and use timer or any other scheduled trigger to initialize a dedicated process for a dedicated action with a defined timeframe.
If nothing happens within a defined timeframe, it's mostly a reason to react. So, keep Flows small and dedicated and run them always with a defined timeframe for the next step.
Limitation of approval run duration / 30 days
The same limitation applies to to approval run duration, as a approval is running as an activity within the Flow. When the Flow get canceled, the Flow Approval Task gets canceled.
Limitation of actions per Flow / 250 actions
If u reached this limit, welcome to the chaos and please read this article carefully - it's your relief. ;-)
Limitation of nested actions per Flow / 8 levels
Can hit hard, especially, when using scopes, switches one maybe some nested conditions or loops.
Designabilty and Readability in Flow Designer for larger Flows
When buliding bigger Flows, we Concepts for Logging and Analytics
Logging and Analytics exist, butJust you need to implement it by yourself and oob there are different sources of information to get an global picture of all #Flows doing their thing or debug an error.
We often see users struggeling with trying to debug JSON response messages from actions, then bugging IT with the issue and operational efficiency gain trough "modern" is lost. Missing State Machine
That was and kind of still is a painful one. Especially as Flow's purpose is workflows/processes/connecting systems and that heavily involves the requirement for stateful processes.
Anyways the good news is, that Business Process Flows are able to handle state. They kind of ported the Business Process Designer from Dynamics to "Modern" and used this to get state capabilities. The story is now obviously evolving, as things are movering closer to ERP/CRM or simply said "Core Systems" in the middle.
What's the benefit of Azure Queue Storage?
Queues are providing us with enterprise software capabilities at our fingertips for nearly no money. The impact on solution stability of this robust, highly scalable and easy to use lego brick, is insane. Azure Storage Queues are a very cost effective solution. These connectors belong to the set of Standard Connectors and don't require a premium plan for each user. As for example the HTTP connectors will require starting from spring 2019. This makes a huge difference in price, as 1GB of queue storage costs about $0.07 and max message size is 64k.
Top 5 overall Benefits
Beside overcoming general pitfalls, this state pattern based on Azure Storage Queues, provides you with a lot of beauty. I picked my personal top 5 benefits:
Much better readability/designability of your Flows, thanks to splitting of controller and worker logic. This is huge, then just buying a bigger screen shouldn't be the answer to efficient Flow design and limits exist for a reason.
Elimination of duplicate logic and repeating patterns within a Flow, because of trying to build all in one giant. Microsoft will come up with "state" in Business Process Flows, which will allow much more flexible design options, but at least until then, this here is a great solution.
Very easy to co-create with others. As we agree on clear design patterns, development work can be done in parallel by different dev's in different solution areas. This results in lower time to market of the solution and more rapid implementation cycles.
Azure Queue Storage is very cost effective solution. These connectors belong to the set of Standard Connectors and don't require a premium plan for each user. As for example the HTTP connectors will require starting from spring 2019. This makes a huge difference in price, as 1GB of queue storage costs about $0.07 and max message size is 64k.
It's an easy to implement, generic, scalable, robust and meeting enterprise software design standards. JSON used as a data exchange format for the state object, is very flexible and can basically contain any data, even binary.
This is a pattern, use it only, when it makes sense and adopt it to your needs. I'm here to help, just contact me on Twitter.
Why do I write this article series?
In the last 18 months, my beloved Jussi Mori and I as co-speaker did quite a tour and spoke at more than 15+ SharePoint Saturdays (SPS Events) in Europe and in the USA and a variety of User Groups, the AI Bootcamp and ThriveConf.
Our session with the title “The rise of the Citizen Developer - How to build intelligent business applications in Office 365” was heavily based on Flow and aimed to showcase the power and variety of the toolbelt.
Basically, we wanted to involve as many systems as possible in a funny end-to-end customer scenario. Driven by Flow as the “glue” of digital storytelling with real tools and scenarios. Think we ended up with 11, or smth. 😉
We covered deep Flow integration of cognitive services with features like mood detection, face/person detection, sentiment detection, keyword detection, translation. We demoed deep integration in Teams with webhooks, bots, rich message cards,etc.
PowerApps were used to take photos for mood detection, stored in SharePoint and to track delivery status of products and location tracking. The session involved interactions between services like PowerApps, Teams, Azure Functions, Microsoft Graph, Approvals, Planner, Dynamics 365, Modern SharePoint, Twilio and Etherum Blockchain.
We asked the audience if they were familiar with Flow. Mostly we heard “YES”, we heard of flow and we have an understanding of what it’s there fore. When we asked about use-cases, people gave answers like moving e-mails around mailboxes, managing files on OneDrive or creating Tasks in Planner. So, mostly we heard scenarios from the personal productivity sector.
Our session was declared as a level 300 because it needs a high-level understanding of modern and serverless cloud architecture and current web technologies. So, people were impressed, by what’s possible but couldn’t really imagine how it’s done. Mostly attendees came up with questions about “well known” limitations and how to overcome these.