Deprecated Actions
As the connector ecosystem matures some actions within connectors and perhaps even whole connectors are being deprecated. To avoid unexpected failures it’s useful to know when a connector or action is deprecated so that one can take action, such as migrating to an updated or alternate connector. This first post in a series of blog posts explains how to detect connectors or actions that have been deprecated using an Azure Durable Function and a Power Automate Custom Connector. We’ll extend this in later blog posts to make it more usable.
Scenario
Microsoft’s connector reference lists the Microsoft and non-Microsoft published connectors. The documentation for each connector states whether an action is deprecated. However we want to find a computer-friendly list of connectors and actions so that we can use it to check our environments for usages of deprecated actions.
Although the documentation explains how to mark an operation as deprecated, and most connectors are available on the PowerPlatformConnectors github, there are significant gaps:
- Microsoft connectors are not on github, e.g. ExcelOnlineBusiness, Azure Blob Storage, Microsoft Dataverse.
- Some fully deprecated connectors do not exist on github, e.g. ElasticOCR
For the above reasons the only reliable source of the deprecation status is the connector reference.
Scraping
Our goal is to make available the status of any actions on connectors we’re using via Power Automate. Step one is to scrape the Microsoft Docs website.
We could potentially do this via Power Automate Desktop, as suggested in my initial twitter question by Carsten Groth. However, it’s also possible via Azure Functions and our own custom connector. We’re going to use the durable functions extension to create a connector that will accept an array of connector names and return the deprecation status for each of the actions (aka operations) in the connectors.
We’re going to use the following components:
- Azure Functions to provide the serverless compute resource, within which we’ll use:
- Durable Functions to implement the fan-out/fan-in pattern allowing multiple connectors to be scraped in parallel.
- HtmlAgilityPack to load and parse the Microsoft docs connector reference pages.
- XPath to navigate the HTML DOM and locate the pertinent parts we want to extract.
- Custom Connectors to make our azure function available within the Power Platform, particularly Power Automate Cloud Flows.
- Power Automate Cloud Flows for a basic example of how use our new custom connector.
Models
It’s useful to start of with some simple models to describe the inputs to and outputs from function, these are easily serialised to JSON objects usable within the Power Platform.
Request
As above, our request is a simple list of connectors. We’ll use the unique name for each connector as found in the URL of each connectors docs page, e.g. excelonlinebusiness or commondataserviceforapps
public class ScrapeConnectorsRequest
{
public String[] SelectedConnectors { get; set; }
}
This converts to a simple JSON object, e.g.
{
"SelectedConnectors": ["excelonlinebusiness", "commondataserviceforapps"]
}
Response
For each connector requested we want to return information about the connector and a list of the actions for each connector.
public class ConnectorInfo
{
public string UniqueName { get; set; }
public string DocumentationUrl { get; set; }
public List<ActionInfo> Actions { get; set; }
}
public class ActionInfo
{
public string OperationId { get; set; }
public string Name { get; set; }
public string Description { get; set; }
public string Anchor { get; set; }
public bool IsDeprecated { get; set; }
}
Main Orchestrator
We’re using a durable function with the ‘Fan out/fan in’ pattern to:
- Download the Connector Reference and read the available connectors, i.e.
- Read the page using HtmlAgilityPack
- Locate the connectors table using XPath
- Store the hyperlinks to the detailed page for each of the connectors required
- For each connector, we ‘Fan-Out’ and
- Download the connector details page
- Find the actions in the connector
- Detect the deprecated status for each action
- ‘Fan-In’ to combine the results from all the required connectors
- Output a sorted collection of connectors and their actions in a JSON format
Our main orchestrator looks like the following
[FunctionName("ScrapeConnectorsOrchestration")]
public static async Task<List<ConnectorInfo>> RunOrchestrator(
[OrchestrationTrigger] IDurableOrchestrationContext context, ILogger log)
{
// Scrape the connectors from the main page
var connectors = await context.CallActivityAsync<List<ConnectorInfo>>(nameof(ScrapeConnectorsOrchestration_ScrapeConnectors), null);
// Get the request parameters passed into the orchestrator
var request = context.GetInput<ScrapeConnectorsRequest>();
if (request.SelectedConnectors != null)
{
// Filter connectors found to just the selectedConnectors required
connectors = connectors.Where(c => request.SelectedConnectors.Contains(c.UniqueName)).ToList();
}
// Create a task for each connector to get the actions from their own docs page
var tasks = new List<Task<ConnectorInfo>>();
foreach (var connector in connectors)
{
tasks.Add(context.CallActivityAsync<ConnectorInfo>(nameof(ScrapeConnectorsOrchestration_GetConnectorInfo), connector));
}
// Wait for all the tasks to finish (Fan-In)
await Task.WhenAll(tasks);
// Put the results together and return them
return tasks
.Select(t => t.Result)
.OrderBy(r => r.UniqueName)
.ToList();
}
Action Functions
The action functions are where the real work gets done. The orchestrator can be configured to retry an action function to increase resilience, but we’re not going to cover that here.
We have two action functions:
1. ScrapeConnectors
This function downloads the connector-reference from the MS docs website and returns a list of ‘ConnectorInfo’ objects. If Microsoft were to change the structure of the Power Automate page then the XPath expression xpath id('list-of-connectors')/following-sibling::table/tr/td/a may need to be updated.
public static List<ConnectorInfo> ScrapeConnectorsOrchestration_ScrapeConnectors([ActivityTrigger] string name, ILogger log)
{
var url = @"https://docs.microsoft.com/en-us/connectors/connector-reference/";
HtmlWeb web = new HtmlWeb();
var htmlDoc = web.Load(url);
var connectorNodes = htmlDoc.DocumentNode.SelectNodes("id('list-of-connectors')/following-sibling::table/tr/td/a");
// We could etxract the
// icon
// availability in Logic Apps, Power automate and Power Apps
// whether connector is Preview
// whether connector is Premium
// but don't do that. We can probably extract from in the docs for each connector
var relativeUrls = connectorNodes.Select(x => x.Attributes["href"].Value).ToList();
log.LogInformation($"Found {relativeUrls.Count()} connectors");
var connectorInfo = new List<ConnectorInfo>();
foreach (var relativeUrl in relativeUrls)
{
var match = sConnectorUniqueName.Match(relativeUrl);
if (!match.Success)
{
log.LogInformation($"Could not determine uniqueName for relativeUrl {relativeUrl}");
continue;
}
var uniqueName = match.Groups[1].Value;
// Note that there's a more direct URL at https://docs.microsoft.com/en-us/connectors/<uniqueName>/
// but we'll continue to use the URL scrapped from the main connector reference which redirects
var documentationUrl = $"{url}{relativeUrl}";
// Populate basic information for the connector, we expand on this when the individual connector is scraped
var c = new ConnectorInfo
{
UniqueName = uniqueName,
DocumentationUrl = documentationUrl,
Actions = new List<ActionInfo>(),
};
connectorInfo.Add(c);
}
return connectorInfo;
}
2. GetConnectorInfo
The second action function will take a single ‘ConnectorInfo’ object that contains the URL of the detailed connector documentation. It will then download the documentation page, then extract the actions and the information about the actions.
// Get information about a single connector
[FunctionName(nameof(ScrapeConnectorsOrchestration_GetConnectorInfo))]
public async static Task<ConnectorInfo> ScrapeConnectorsOrchestration_GetConnectorInfo([ActivityTrigger] ConnectorInfo connector, ILogger log)
{
log.LogInformation($"Getting connector documentation for {connector.UniqueName}.");
HtmlWeb web = new HtmlWeb();
var htmlDoc = await web.LoadFromWebAsync(connector.DocumentationUrl);
// Get the list of anchors from the initial table, from this we can get the:
// description
// if the action is deprecated
// But we can't get the OperationId of the action
var actionNodes = htmlDoc.DocumentNode.SelectNodes("id('actions')/following-sibling::table/tr");
log.LogInformation($"Found {actionNodes?.Count() ?? 0} actions on connector {connector.UniqueName}");
// Some connectors don't have any actions - abort early.
if (actionNodes == null)
{
return connector;
}
// We have to convert to a List (not IEnumerable) because we're going to edit the contents of the list
var actions = actionNodes.Select(x => new ActionInfo
{
Anchor = x.SelectSingleNode("td/a").GetAttributeValue("href", ""),
Name = x.SelectSingleNode("td/a").InnerText,
Description = x.SelectSingleNode("td/a/../following-sibling::td").InnerText.Trim(),
}).ToList();
// For all of the actions found find the operationId, and flag if it's deprecated
foreach (var action in actions)
{
var anchorText = action.Anchor.Substring(1);
var xpath = "id(\"" + anchorText + "\")/following-sibling::div/dl/dd";
var operationId = htmlDoc.DocumentNode.SelectSingleNode(xpath)?.InnerText?.Trim() ?? "";
action.OperationId = operationId;
var lowerName = action.Name.ToLower();
action.IsDeprecated = lowerName.Contains("[deprecated]") || lowerName.Contains("(deprecated)");
}
// Sort the actions within the connector
connector.Actions = actions.OrderBy(a => a.OperationId).ToList();
return connector;
}
Calling Durable Functions
Durable functions can be triggered via multiple means, and for this example we’re going to trigger it via a ’normal’ HTTP request function.
Many examples show returning the initial response from this function using return starter.CreateCheckStatusResponse(req, instanceId); as shown below. We’ll come back to this in a later blog post.
[FunctionName(nameof(ScrapeConnectors))]
public static async Task<IActionResult> ScrapeConnectors(
[HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequest req,
[DurableClient] IDurableOrchestrationClient starter,
ILogger log)
{
// Extract the request object from the request body
string requestBody = new StreamReader(req.Body).ReadToEnd();
var request = JsonConvert.DeserializeObject<ScrapeConnectorsRequest>(requestBody);
log.LogInformation($"{nameof(ScrapeConnectors)} triggered at {DateTime.UtcNow:O}. " +
$"SelectedConnectors=${String.Join(",", request.SelectedConnectors)}");
// Start the orchestrator with this request
string instanceId = await starter.StartNewAsync("ScrapeConnectorsOrchestration", request);
log.LogInformation($"Started orchestration with ID = '{instanceId}'.");
return starter.CreateCheckStatusResponse(req, instanceId);
}
Testing locally
We can test the Azure function is working locally, as shown below. We’re using the REST Client in VS Code for it’s simplicity.
We can see that durable functions implement the Asynchronous Request-Reply pattern because our initial request returns an HTTP 202 ‘Accepted’ status with a ‘Location’ header informing where to check after a delay.
If we manually check on the URL, it will either return again with status 202 ‘Accepted’, or once the durable function has finished processing successfully will return 200 ‘OK’, with our output, as shown below:
Now that we’ve tested the function works locally we can publish to the cloud as per any Azure function.
Calling our Azure Function from Power Automate
Firstly, we create a custom connector. We’re going to do this manually for now, but we’ll improve on this in the next blog.
Create a custom connector
We create a connector as normal, by providing a connector name, scheme, host and base URL.
Set the security on the connector
For this example we’re using the function key method, which is good for testing.
Add action definition to call the connector
The critical setup is the creation of the action that we’re going to make available. We can create the request using the URL of the azure function and the request that we used when testing locally.
Test the connector
We can now create a connection and test the connector. Note that when testing, the response is 202 Accepted. The header contains a ’location’ and ‘Reply-After’ property. The body of the response is not actually used by Power Automate.
Test in Power Automate
We now have a connector, so we can create a Cloud Flow. For an example this cloud flow is manually triggered and we’ve hardcoded it to look for just the Excel Online (Business) and 365 Training connectors.
When the durable function completes, it returns a JSON object with properties for input, output and runtimeStatus. We can see that the ‘AddRow’ operation in the Excel Online (Business) connector has been deprecated because the IsDeprecated property is true.
{
"name": "ScrapeConnectorsOrchestration",
"instanceId": "6aed49620eab4d79a22db94825fe277b",
"runtimeStatus": "Completed",
"input": {
"SelectedConnectors": [
"excelonlinebusiness",
"365training"
]
},
"output": [
{
"UniqueName": "365training",
"DocumentationUrl": "https://docs.microsoft.com/en-us/connectors/connector-reference/../365training/",
"Actions": [
{
"OperationId": "AddIdeaVote",
"Name": "Add Idea Vote",
"Description": "Add your vote to an idea to show your support.",
"Anchor": "#add-idea-vote",
"IsDeprecated": false
},
... snip ...
]
},
{
"UniqueName": "excelonlinebusiness",
"DocumentationUrl": "https://docs.microsoft.com/en-us/connectors/connector-reference/../excelonlinebusiness/",
"Actions": [
{
"OperationId": "AddRow",
"Name": "Add a row into a table [DEPRECATED]",
"Description": "This action has been deprecated. Please use Add a row into a table instead.\nAdd a new row into the Excel table.",
"Anchor": "#add-a-row-into-a-table-[deprecated]",
"IsDeprecated": true
},
... snip ...
}
],
"createdTime": "2021-09-07T15:34:27Z",
"lastUpdatedTime": "2021-09-07T15:34:28Z"
}
We’ll see in a later blog how to handle these better, but for the moment, we must check the ‘runtimeStatus’ and get the results from the ‘output’ property.
Testing
We can test the flow and see the results. Power Automate handles the 202 Accepted status and retries, eventually outputting a result, normally just after 20 seconds. We have checked the durable function executes successfully and accessed the output property. We can now use that object how we deem fit to monitor connectors that we’re using.
Summary
- We were able to use an azure function to scrape the MS docs web page for the connectors that we were interested in.
- We are having to scrape the website which is quite brittle since changes in layout of the MS docs website may break the azure function.
- We were able to parse the results of the durable function but we had to use a ‘Condition’ and ‘Parse JSON’ action.
Any comments, corrections or suggestions are gratefully received on my socials: twitter, LinkedIn
Evaluation and Potential Improvements
We will improve the function in a future blog post to resolve the following issues:
- We had to check the status of the durable function using the ‘runtimeStatus’ property, and could not use the normal Power Automate Cloud Flow error processing should the durable function error.
- The structure of the inputs had to be manually added to the custom connector, and the output had to be parsed using a ‘Parse JSON’ action.
- The Azure Function used the
starter.CreateCheckStatusResponse(req, instanceId);method which returns URLs for sendEventPostUri, terminatePostUri and purgeHistoryDeleteUri that we should not be exposing.
{
"id": "07e93a36b0d14571a9670a9b4381393c",
"statusQueryGetUri": "https://funcdeprecatedactionsblog.azurewebsites.net/runtime/webhooks/durabletask/instances/07e93a36b0d14571a9670a9b4381393c?taskHub=funcdeprecatedactionsblog&connection=Storage&code=cXxPem...snip...Jw==",
"sendEventPostUri": "https://funcdeprecatedactionsblog.azurewebsites.net/runtime/webhooks/durabletask/instances/07e93a36b0d14571a9670a9b4381393c/raiseEvent/{eventName}?taskHub=funcdeprecatedactionsblog&connection=Storage&code=cXxPem...snip...Jw==",
"terminatePostUri": "https://funcdeprecatedactionsblog.azurewebsites.net/runtime/webhooks/durabletask/instances/07e93a36b0d14571a9670a9b4381393c/terminate?reason={text}&taskHub=funcdeprecatedactionsblog&connection=Storage&code=cXxPem...snip...Jw==",
"purgeHistoryDeleteUri": "https://funcdeprecatedactionsblog.azurewebsites.net/runtime/webhooks/durabletask/instances/07e93a36b0d14571a9670a9b4381393c?taskHub=funcdeprecatedactionsblog&connection=Storage&code=cXxPem...snip...Jw=="
}
In the next blogs we’ll see how it’s possible to:
- Use a custom status function to allow Power Automate to return the result more easily.
- Use Azure Functions OpenAPI Extensions to decorate the Azure Function inputs and outputs.
Source Code
You can get the source code and solutions used on my github in the blog1 branch. Import the ‘DeprecatedConnectors’ solution before the ‘DeprecatedConnectorsFlow’ solution due to a known limitation that custom connectors must be installed first.
