Functional consultants or administrators who have been using Dynamics CRM / Dynamics 365 Customer Engagement (D365CE) for any considerable length of time will likely have built up a solid collection of FetchXML queries, that are usable for a variety of different scenarios. Such privileged individuals are in the fortunate position of being able to leverage them in the following ways:

In other words, you have a range of useful queries that can potentially meet any needs within the application from a reporting standpoint. This is all well and good if you find yourself working solely within CRM/D365CE all the time, but when you start to bring in separate tools, such as Power BI, there can be some difficulty in migrating these across straightforwardly. Typically, you may find yourself staring down the barrel of a complicated and costly redevelopment exercise, where you have to invest a lot of time within Power Query to replicate your existing FetchXML queries as efficiently as possible; this puts potentially a lot of hard work and investment made into FetchXML query development down the drain almost immediately.

Fortunately, there is a way in which we can leverage our FetchXML queries using Power BI. I did a post on this very subject a few years ago, where I talked through an example from start to finish. The main limitations with this were, however, 1) the inability to return more than 5000 records at a time, given that paging was not correctly incorporated and 2) the fact that you had to manually define code for every query that you wished to utilise, which would take a lot of time to do and increase the risk of human error occurring.

As usual in these situations, the wonderful CRM/D365CE community has delivered a solution to address the first issue raised above. The Power Query (M) Builder tool is a handy plugin within the XrmToolBox that allows you to generate M query code snippets that you can use within Power BI Desktop. Most importantly, the tool incorporates a solution from Keith Mescha and the former Sonoma Partners Power BI Accelerator to get around the paging issue and allow you to return unlimited data from the application. You can find out more about the tool by checking out Ulrik “The CRM Chart Guy” Carlsson’s blog post dedicated to this very subject.

The tool is undoubtedly great, but if you have numerous FetchXML queries in a raw format that you wish to process within Power BI, it could take you some time to get these moved across into Power BI – particularly given that the tool does not currently support the ability to “bring your own” FetchXML queries. By using the example code provided by the tool, and carrying out some further work to address the second concern, it is possible to use the following M query function that will allow you to compartmentalise all of the above functionality in an easy to call Power Query function. Simply open a new blank query within Power Query and copy & paste the below into the window:

/*
    Generate FetchXML Query Results M Function
    Required Parameters:
        crmURL = The URL of your CRM/D365CE instance e.g. https://mycrm.crm11.dynamics.com
        entityName = The OData entity name that you are querying.
        query = The FetchXML query to execute. This should NOT include the top level <fetch> node, but only all subsequent nodes with double quotes escaped e.g. <entity name=""incident""><all-attributes /></entity>
    Credits: Big thanks to the Power Query Builder tool (https://crmchartguy.com/power-query-builder/) and Keith Mescha/Sonoma Partners Power BI Accelerator
             for figuring out the paging issue. Portions of the auto-generated code from the above tool is utilised within this function.
*/



let
    Func = (crmURL as text,entityName as text,query as text) =>
let
    FullURL = Text.Combine({crmURL, "/api/data/v9.1/", entityName},""),
    QueryAll = (z as text, x as number) =>
    let
                Source = Json.Document(Web.Contents(FullURL,
                        [
                            Headers=
                                [
                               #"Prefer"="odata.include-annotations=Microsoft.Dynamics.CRM.fetchxmlpagingcookie"
                                ],
                            Query=
                                [
                                fetchXml="<fetch distinct=""True"" page=""" & Text.From(x) & """ paging-cookie=""" & z & """>" & query & "</fetch>"
                                ]
                        ]
                    )
                ),
    Paging = try Xml.Document(Source[#"@Microsoft.Dynamics.CRM.fetchxmlpagingcookie"]) otherwise null,
    Retrieve = if Paging <> null 
                then List.Combine({Source[value],@QueryAll(Text.Replace(Text.Replace(Text.Replace(Uri.Parts("http://a.b?d=" & Uri.Parts("http://a.b?d=" & Paging{0}[Attributes]{1}[Value])[Query][d])[Query][d], ">", "&gt;"), "<", "&lt;"), """", "&quot;"), x + 1)})
                else Source[value]
    in 
        Retrieve,
    GenerateEmptyTable = (query as text) =>
    let
            XML = Xml.Document(query),
            #"Expanded Value" = Table.ExpandTableColumn(XML, "Value", {"Name", "Namespace", "Value", "Attributes"}, {"Value.Name", "Value.Namespace", "Value.Value", "Value.Attributes"}),
            #"Expanded Value.Value" = Table.ExpandTableColumn(#"Expanded Value", "Value.Value", {"Name", "Namespace", "Value", "Attributes"}, {"Value.Value.Name", "Value.Value.Namespace", "Value.Value.Value", "Value.Value.Attributes"}),
            #"Expanded Value.Attributes" = Table.ExpandTableColumn(#"Expanded Value.Value", "Value.Attributes", {"Name", "Namespace", "Value"}, {"Value.Attributes.Name", "Value.Attributes.Namespace", "Value.Attributes.Value"}),
            #"Filtered Rows" = Table.SelectRows(#"Expanded Value.Attributes", each ([Value.Attributes.Name] = "name")),
            #"Removed Columns" = Table.RemoveColumns(#"Filtered Rows",{"Name", "Namespace", "Value.Name", "Value.Namespace", "Value.Value.Name", "Value.Value.Namespace", "Value.Value.Value", "Value.Value.Attributes", "Value.Attributes.Name", "Value.Attributes.Namespace", "Attributes"}),
            #"Transposed Table" = Table.Transpose(#"Removed Columns"),
            #"Promote Headers" = Table.PromoteHeaders(#"Transposed Table", [PromoteAllScalars=true]),
            #"Added Custom" = Table.AddColumn(#"Promote Headers", "@odata.etag", each ""),
            #"Reordered Columns" = Table.ReorderColumns(#"Added Custom", List.Sort(Table.ColumnNames(#"Added Custom"), Order.Ascending))
    in
        #"Reordered Columns",
    List = QueryAll("",1),
    Table = if List.IsEmpty(List)
        then GenerateEmptyTable(query)
        else #"D365CEData",
    #"D365CEData" = Table.FromList(List, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
    Expand = Table.ExpandRecordColumn( #"D365CEData", "Column1", Record.FieldNames(Table.Column(#"D365CEData", "Column1"){0})),
    D365CE = Table.ReorderColumns(Expand, List.Sort(Table.ColumnNames(Expand), Order.Ascending)),
    Results = if List.IsEmpty(List)
        then Table
        else D365CE
    in
        Results
    in 
        Func

When saved, Power BI will then generate a function that should resemble the below screenshot:

From here, you can then populate each of the required parameters as follows:

And then you are good to go! As an example, the following FetchXML query:

  <entity name=""incident"">
    <attribute name=""title"" />
    <attribute name=""ticketnumber"" />
    <attribute name=""createdon"" />
    <attribute name=""incidentid"" />
    <attribute name=""caseorigincode"" />
    <attribute name=""casetypecode"" />
    <order descending=""false"" attribute=""title"" />
    <filter type=""and"">
      <condition attribute=""createdon"" operator=""this-year"" />
      <condition attribute=""casetypecode"" operator=""in"">
        <value>2</value>
        <value>1</value>
      </condition>
    </filter>
  </entity>
  

Would return results similar to the below via the above function:

One limitation with this function, at present, is that I haven’t yet found a way to ensure formatted values return correctly, even when there are no results available. I’ll report back if I figure out a way to do this šŸ™‚ A huge thanks to Keith, Ulrik and Sonoma Partners for kindly supplying the paging code snippet into the community and in helping me to build out the above function.

For the past 13 weeks on the blog, I have delivered a series of posts concerning Microsoft Exam 70-778, specifically focused towards providing a set of detailed revision notes that cover the broad array of Power BI features assessed as part of the exam. To round things off, today’s blog will bridge together everything I have discussed thus far in the series; with the hopeĀ being that this post can be a single reference point for those who have not been following the series to date.

Microsoft Exam 70-778 Overview

The exam, with its full title Analyzing and Visualizing Data with Microsoft Power BI, is targeted towards Business Intelligence (BI) and data professionals who are looking to validate their skills in working with Power BI. The exam is a necessary component, alongside Exam 70-779: Analyzing and Visualizing Data with Microsoft Excel, in attaining the Microsoft Certified Solutions Associate (MCSA) certification in BI Reporting. Successful candidates can then (optionally) pass an additional “elective” exam to gain the Microsoft Certified Solutions Expert (MCSE) certification in Data Management and Analytics.

Skills Measured in the Exam

The skills measured are outlined below, alongside links to the relevant posts from the series and the list of essential points to remember:

Consuming and Transforming Data By Using Power BI Desktop

Connect to data sources.
Skills Measured

May include: Connect to databases, files, folders; import from Excel; connect to SQL Azure, Big Data, SQL Server Analysis Services (SSAS)

Revision Notes

Exam 70-778 Revision Notes: Importing from Data Sources

Key Takeaways
  • Power BI supports a broad range of database systems, flat file, folder, application and custom data sources. While it is impossible to memorise each data source, you should at least broadly familiarise yourself with the different types at our disposal.
  • A crucial decision for many data sources relates to the choice of either Importing a data source in its entirety or in taking advantage of DirectQuery functionality instead (if available). Both routes have their own defined set of benefits and disadvantages. DirectQuery is worth consideration if there is a need to keep data regularly refreshed and you have no requirement to work with multiple data sources as part of your solution.
  • Live Connection is a specific data connectivity option available for SQL Server Analysis Services. It behaves similarly to DirectQuery.
  • It is possible to import an existing Excel BI solution into Power BI with minimal effort, alongside the ability to import standard worksheet data in the same manner as other flat file types.
Perform transformations
Skills Measured

May include: Design and implement basic and advanced transformations; apply business rules; change data format to support visualization

Revision Notes

Exam 70-778 Revision Notes: Performing Data Transformations

Key Takeaways
  • The Power Query M formula language is used to perform transformations to data once loaded into Power BI. Although it is possible to do this via code, Power BI allows us to define all of our required data changes from within the interface, without the need to write a single line of code.
  • Each data source connected to represents itself as a Query within Power BI. There are many options at your disposal when working with Queries, such as renaming, merging, duplication and the ability to disable or reference as part of other Queries.
  • There are wide-range of column transformations that can be applied, which are too numerous to mention. The Transform tab provides the best means of seeing what is available, with options ranging from formatting through to grouping and pivoting/unpivoting.
  • New columns are addable via the Add Column tab. You can choose to base new columns on calculations, conditional logic, other column values or as a defined list of ascending numbers, which may be useful for indexing purposes.
  • It is possible to merge or append queries together to suit your specific requirements. Merging involves the horizontal combination of Queries, whereas appending represents a vertical combination.
  • Parameters can be used to help optimise any complex filtering requirements.
  • Where possible, Power Query will attempt to use the most optimal query for your data source, based on the transformation steps you define. This action is known as Query Folding and, in most cases, SQL-derived data sources will support this option by default.
Cleanse data
Skills Measured

May include: Manage incomplete data; meet data quality requirements

Revision Notes

Exam 70-778 Revision Notes: Cleansing Data

Key Takeaways
  • Data can be filtered directly within Power Query, using Excel-like functionality to assist you in only returning the most relevant data in your queries. The data type of each field plays a particularly important part of this, as only specific filter options will be at your disposal if, for example, you are working with numeric data.
  • From a data quality perspective, you typically will need to handle column values that contain one of two possible value types:
    • Errors: This will usually occur as a result of a calculated column field not working correctly. The best solution will always be to address any issues with your calculated column, such as by using a conditional statement to return a default value.
    • Blanks/NULLs: A common symptom when working with SQL derived data sources, your real problems with blank values start to appear when you attempt to implement DAX custom columns/Measures outside of the Power Query Editor. It is, therefore, recommended that these are dealt with via a Replace action, depending on your fields data types. For example, a number field with blank/NULL values should be replaced with 0.
  • The Remove Rows option(s) can act as a quick way of getting rid of any Error or Blank/NULL rows and can also be utilised further to remove duplicates or a range of rows. In most cases, you will have similar options available to you with Keep Rows instead.
  • There are a variety of formatting options available to us when working with text/string data types. These range from fixing capitalisation issues in data, through to removing whitespace/non-printable character sets and even the ability to prepend/append a new value.

Modeling and Visualizing Data

Create and optimize data models.
Skills Measured

May include: Manage relationships; optimize models for reporting; manually type in data; use Power Query

Revision Notes

Exam 70-778 Revision Notes: Create and Optimise Data Models

Key Takeaways
  • Relationships form the cornerstone of ensuring the long-term viability and scalability of a large data model. Assuming you are working with well-built out, existing data sources, Power BI will automatically detect and create Relationships for you. In situations where more granular control is required, these Relationships can be specified manually if needed. It is worth keeping in mind the following important features of Relationships:
    • They support one-to-one (1:N), one-to-many (1:N) and many-to-one (N:1) cardinality, with many-to-many (N:N) currently in preview.
    • Filter directions can be specified either one way or bi-directionally.
    • Only one relationship can be active on a table at any given time.
  • It is possible to sort columns using more highly tailored custom logic via the Sort By Column feature. The most common requirement for this generally involves the sorting of Month Names in date order but can be extended to cover other scenarios if required. To implement, you should ensure that your data has a numbered column to indicate the preferred sort order.
  • Moving outside of the Power Query Editor presents us with more flexibility when it comes to formatting data to suit particular styling or locale requirements. While the majority of this functionality provides date/time and currency formatting options, for the most part, it is also possible to categorise data based on Location, the type of URL it is or on whether or not it represents a Barcode value; these options can assist Power BI when rendering certain types of visualizations.
  • There may be ad-hoc requirements to add manually defined data into Power BI – for example, a list of values that need linking to a Slicer control. The Enter Data button is the “no-code” route to achieving this and supports the ability to copy & paste data from external sources. For more advanced scenarios, you also have at your disposal a range of M code functionality to create Lists, Records and Tables, which can be extended further as required.
Create calculated columns, calculated tables, and measures
Skills Measured

May include: Create DAX formulas for calculated columns, calculated tables, and measures; Use What If parameters

Revision Notes

Exam 70-778 Revision Notes: Using DAX for Calculated Columns

Key Takeaways
  • DAX is the primary formula language when working with datasets outside of Power Query. It includes, to date, more than 200 different types of functions that can assist in all sorts of data modelling.
  • An important concept to grasp within DAX is context and, specifically, row context (formulas that calculate a result for each row in a dataset) and filter context (formulas that automatically apply any filtering carried out at report level).
  • The sheer amount of DAX functions available makes it impossible to master and remember all of them, particularly when it comes to the exam. Your learning should, therefore, focus on learning the general syntax of DAX and the general types of functions available (aggregation, date/time etc.)
  • There are three principal means of utilising DAX with Power BI:
    • As Measures: These typically present a scalar value of some description, often an aggregation or a result of a complex formula. Using them in association with a Card visualization type is recommended, but this is not a strict requirement.
    • As Calculated Columns: Similar to the options available within Power Query, Calculated Columns provide a dynamic and versatile means of adding new columns onto your datasets. Compared with the options available within Power Query and the complexity of the M language, DAX Calculated Columns might represent a more straightforward means of adding custom columns onto your datasets.
    • As Calculated Tables: A powerful feature, mainly when used in conjunction with Calculated Columns, you have the ability here to create entirely new datasets within the model. These will typically derive from any existing datasets you have brought in from Power Query, but you also have functionality here to create Date tables, sequence numbers and manually defined datasets as well.
  • What-if Parameters provide of means of testing DAX formulas, as well as allowing report users to perform predictive adjustments that can affect multiple visualizations on a report.
Measure performance by using KPIs, gauges and cards.
Skills Measured

May include: calculate the actual; calculate the target; calculate actual to target; configure values for gauges; use the format settings to manually set values

Revision Notes

Exam 70-778 Revision Notes: Utilising KPIs with Gauge Visualisations

Key Takeaways
  • There are two principle visualization types available within Power BI to help track actual-to-target progress – KPIs and Gauges.
  • KPIs provide a more visually unique means of a binary success/fail determination when tracking towards a target. It is also possible to use KPI’s to track variance over time via the Trend axis. The Indicator will typically be the result of some form of aggregation or Measure.
  • Gauges provide a less visually distinctive, but non-binary, mechanism of viewing progress towards a target. Gauges support more potential field well values when compared with KPIs, nearly all of which are optional in some way. You can also manually define some of these values, for situations where your data model does not contain the required information.
  • All visualizations within Power BI are modifiable from a display or formatting perspective. The same basic options will generally be supported – such as changing a font type or background colour – with more specific configuration properties available per unique visualization type. For example, a KPI visualization can be customised to hide the background Trend Axis entirely. All of these options are designed to give developers greater control over the look and feel of their reports and to mirror them as closely as possible to any potential branding requirement.
  • When building out a solution designed to monitor progress within Power BI, the steps involved will typically be more in-depth than merely creating a new visualization. In most cases, there will be a requirement to bring together a lot of the other skills that have been discussed previously within this series – such as creating DAX formulas, modifying data within Power Query or bringing together different data sources into a single model. It is essential, therefore, not to underestimate the amount of time and effort involved in creating a practical solution that takes advantage of KPIs or Gauges.
Create hierarchies
Skills Measured

May include: Create date hierarchies; create hierarchies based on business needs; add columns to tables to support desired hierarchy

Revision Notes

Exam 70-778 Revision Notes: Creating Hierarchies

Key Takeaways
  • Hierarchies within Power BI provide a means of logically categorising data into an order of preference or precedence, providing greater flexibility to Power BI report users when they interact with visualizations.
  • Date Hierarchies are created and managed automatically by Power BI for each Date or Date/Time field defined within your model. These automatically create fields that contain the Year, Quarter, Month & Day values from the respective date fields. These fields can then be utilised as part of a Table visualization or within a DAX formula. Date Hierarchies can also be completely disabled if required.
  • Custom (or User-Defined) Hierarchies need to be created manually and provide additional customisation options when compared to Date Hierarchies, such as the number of fields they contain, the order and its name. A Custom Hierarchy will typically make use of one of several Parent/Child DAX functions, such as PATH or PATHITEM.
  • Including a hierarchy as part of a chart visualization, such as a Pie chart or Donut chart, opens up other drill-down capabilities around your data. Indicated by the additional arrow icons included at the top of the visualization, they provide the means for users to interrogate data points that interest them the most straightforwardly.
Create and format interactive visualizations.
Skills Measured

May include: Select a visualization type; configure page layout and formatting; configure interactions between visual; configure duplicate pages; handle categories that have no data; configure default summarization and data category of columns; position, align, and sort visuals; enable and integrate R visuals; format measures; Use bookmarks and themes for reports

Revision Notes

Exam 70-778 Revision Notes: Create and Format Interactive Visualizations

Key Takeaways
  • Power BI delivers, out of the box, a range of different visualizations that cater towards most (if not all) reporting requirements. Should you find yourself in need of additional visualizations, then Microsoft AppSource is your go-to destination for finding visualizations developed by others. If you have experience working with either Node.js or R, then these can be used to build bespoke visualizationsĀ also.
  • When first developing a report, you should be able to match a requirement for a specific visualization type, to ensure that you are delivering a solution that is both meaningful and useful. From an exam perspective, this becomes a more critical consideration, and you should be prepared to suggest the most optimal visualization to use when given a specific scenario.
  • After adding visualization’s to your report, you have additional options available to customise them further. For example, you can specify a preferred sorting order for your data, override any summarizations used and move/align your visual on the report page.
  • By default, visualizations in Power BI are designed to change automatically, based on how users interact with the report. All of these options are controllable via the Edit interactions button, allowing you to specify your preferred cross-filtering and cross-highlighting conditions.
  • There is a range of report page customisation options available to developers. It is possible to resize a page to any possible height/width, allowing you to optimise your report for specific devices.Ā Also, you can modify the colour of a page (or its wallpaper) or add an image instead. Pages can also be renamed, reordered or duplicated.
  • Measures can be formatted in the same way as calculated columns, meaning you can specify a data type or, for numerics, modify the number of decimal places.
  • Bookmarks allow developers to set up “checkpoints” within a report, based on how a report page has been filtered. These can then be used to automatically navigate the user through a report, applying these filtering steps automatically. This feature can help transform your report into an interactive story.
  • Visualizations will automatically inherit their various colour properties from the currently selected report theme. Although these can be modified granularly, the fastest and most consistent way of making these changes en masse is to change the Theme. Power BI includes some Themes out of the box, but you also have the option of building your own using a custom JSON file; this can then be imported into different reports, providing a portable means of enforcing a particular branding requirement.
Manage custom reporting solutions
Skills Measured
  • May include: Configure and access Microsoft Power BI Embedded; enable developers to create and edit reports through custom applications; enable developers to embed reports in applications; use the Power BI API to push data into a Power BI dataset; enable developers to create custom visuals
Revision Notes

Exam 70-778 Revision Notes: Managing Custom Reporting Solutions

Key Takeaways
  • Power BI Embedded is an Azure hosted offering that allows you add Power BI Report content into bespoke applications. This deployment option can be incredibly useful if you wish to make available your Power BI solution to users outside of your organisation or if you have an existing, bespoke application system that can benefit from utilising Power BI content. An Azure subscription is required to begin working with Power BI Embedded and you are billed based on node size, not individual user licenses. All Power BI content requires publishing to the online service before its contents become available for Power BI Embedded to access. Report developers will, therefore, need granting a Power BI Professional license to carry out these activities.
  • The Power BI API grants access to developers to perform automation or administrative actions programmatically against the Power BI Online service. Utilising a REST API, developers can determine the optimal programming language of choice to interact with the API, allowing them to streamline the deployment of Reports or Dashboards to the Power BI service or leverage additional functionality when utilising Power BI Embedded. The API can also cater to specific data load requirements, although more complex needs in this area would require addressing via alternate means (SSIS, Azure Data Factory etc.)
  • Developers can add their own bespoke visualizations to a Power BI Report by either developing them using Node.js or using the R language. The first of these options facilitate a more streamlined deployment mechanism and allows developers to add their visualizations to AppSource, whereas the second option may be more useful for complex visualization types with an analytical or statistical function.

Configure Dashboards, Reports and Apps in the Power BI Service

Access on-premises data
Skills Measured

May include: Connect to a data source by using a data gateway;publish reports to the Power BI service from Power BI Desktop; edit Power BI Service reports by using Power BI desktop

Revision Notes

Exam 70-778 Revision Notes: Report Publishing, On-Premise Gateway & Creating Dashboards

Key Takeaways
  • The Power BI On-Premise Gateway provides a streamlined route to working with non-cloud data sources within Power BI, Microsoft Flow and PowerApps. As a lightweight and easy-to-configure client application, it supports a wide variety of data sources, making them accessible as if they were in the cloud. Once set up, corresponding Data Sources are then made available for configuration and for leveraging as part of any Power BI Dataset.
  • Reports can be published into Power BI Online, meaning that they become accessible online and to a broader group of users, without requiring access to Power BI Desktop. Reports need deploying into a Workspace, which can be created manually or derived from an Office 365 Group. Each Report contains a corresponding Dataset, where all queries defined within Power BI Desktop exist.
  • Reports that already exist on Power BI Online can be updated by just publishing a new version of the Report from Power BI Desktop. It is also possible to modify Reports from directly within the browser and by downloading a copy of the .pbix Report file as well, which can then be altered and re-published.
Configure a dashboard
Skills Measured

May include: Add text and images; filter dashboards; dashboard settings; customize the URL and title; enable natural language queries

Revision Notes

Exam 70-778 Revision Notes: Report Publishing, On-Premise Gateway & Creating Dashboards

Key Takeaways
  • Dashboards provide a means of grouping together various content as tiles, designed for at-a-glance analysis and optimal end-user experience.
  • The list of content that can be pinned to a Dashboard includes:
    • Visualizations
    • Web content
    • Images
    • Text boxes
    • Videos
    • Custom streaming data
  • Pinned content can be re-arranged on Dashboard via drag and drop functionality. It is also possible to resize tiles to any given height/width.
  • Within the settings of a Dashboard, it is possible to enable/disable features such as natural language queries (Q&A’s) and Notes.
  • Some features of a Dashboard are only available if you have a Power BI Professional subscription, such as sharing and email subscriptions.

 

Publish and embed reports
Skills Measured

May include: Publish to web; publish to Microsoft SharePoint; publish reports to a Power BI Report Server

Revision Notes

Exam 70-778 Revision Notes: Publish and Embed Reports

Key Takeaways
  • The Publish to web option allows for non-licensed, external users to view a Power BI Report in its entirety. A URL and IFrame embed code can be generated for this at any time within the portal and then dropped into virtually any website. Although you will lose some functionality when deploying a Report out in this manner, you can expect that users will be able to perform most types of interactions with visualizations, Report pages and other components, as if they were accessing the Report through Power BI Online. In some cases, you may be unable to use the Publish to web option if your Report uses certain kinds of features, such as R Visuals or row-level security. You must also take into account any privacy or data protection concerns, as Reports deployed out in this manner will be publically accessible; where this is an issue, the Embed option is available as a secure alternative.
  • There are three steps involved if you wish to add a Report to SharePoint. First, you must generate the unique SharePoint embed URL within Power BI. Secondly, you then need to add on the dedicated control for this feature on your target SharePoint page and configure the relevant display options. Finally, you then need to ensure that all SharePoint users have been granted access to the Report, either at a Workspace level (recommended option) or by having the Report shared with them. By implication, in this scenario, all SharePoint users would have to have at least a Power BI Professional license to take full advantage of this functionality.
  • Publishing a Report to Power BI Report Server is mostly the same as if you were to do the same with the online version of the product. Instead of selecting a Workspace to add the Report to you, specify the name of the Report Server folder where the Report will reside. From a development standpoint, the dedicated Power BI Desktop for Power BI Report Server must be used and may differ in functionality from the “normal” version of the tool. There is also no option to edit a report from within Power BI Report Server like you can through the online version.
Configure security for dashboards, reports and apps.
Skills Measured

May include: Create a security group by using the Admin Portal; configure access to dashboards and app workspaces; configure the export and sharing setting of the tenant; configure Row-Level Security

Revision Notes

Exam 70-778 Revision Notes: Securing Power BI Dashboards, Reports and Apps

Key Takeaways
  • Workspaces act as a container for the various components that form a Power BI Reporting solution. Within a Workspace, you will find all of the Dashboards, Reports, Workbooks and Datasets that developers have published content to. Each User has a Workspace created for them in Power BI when they first access the service. Additional Workspaces can be added through Office 365 Groups or by installing a Power BI App from AppSource. Dashboards and Reports created within your a Users Workspace are shareable to other Users, provided that your account has a Power BI Professional license assigned to it.
  • To help manage permissions to Dashboards/Reports in a more efficient manner, Administrators can create Security Groups on the same Office 365 Tenant where Power BI Online resides. These can contain multiple groups of Users, allowing administrators to minimise the amount of effort involved in managing Dashboard/Report access. Most crucially, this will also enable Users that do not have an Exchange Online mailbox to access Dashboards/Reports when they are shared out in this manner.
  • Administrators have a whole host of options available to them within the Tenant settings area of the Admin Portal. These include, but are not limited to:
    • Export and Sharing Settings
    • Enable/Disable Content Sharing
    • Enable/Disable Publish To Web
    • Enable/Disable Export Reports as PowerPoint Presentations
    • Enable/Disable Print Dashboards and Reports
    • Content Pack and App Settings
    • Integration Settings
    • Custom Visuals Settings
    • R Visuals Settings
    • Audit and Usage Settings
    • Dashboard Settings
    • Developer Settings
  • All of these settings can be enabled for a specific security group, the entire organisation (excepting specific security groups) or allowed for particular security groups, excluding all others in the organisation.
  • Row-Level Security (RLS) allows report developers to restrict data, based on Roles. Row-level DAX evaluation formulas are used to achieve this, which filters the data that is returned, depending on a TRUE/FALSE logic test. To utilise the feature, you must define both the Roles and DAX formulas for each query within your data model. Then, after deploying your Report to Power BI Online, you then assign Users or Security Groups to the Role(s) created within Power BI Desktop. It is possible to view the effect of a Role at any time, within Power BI Desktop or Online, via the View As Role functionality. With the wide-array of DAX formulas available, including specific ones that return the details for the current user accessing a Report, it is possible to define very granular filtering within a Power BI report, to suit particular security or access models.
Configure apps and apps workspaces.
Skills Measured

May include: Create and configure an app workspace; publish an app; update a published app; package dashboards and reports as apps

Revision Notes

Exam 70-778 Revision Notes: Working with Apps and App Workspaces

Key Takeaways
  • Workspaces act as a container for the various components that form a Power BI Reporting solution. Within a Workspace, you will find all of the Dashboards, Reports, Workbooks and Datasets that developers have published content to. Each User has a Workspace created for them in Power BI when they first access the service. It is also possible to create additional Workspaces, either through the Power BI Online interface or by creating an Office 365 Group. A new experience for creating Workspaces is currently in preview which, once released, would negate the need for each Workspace to have an associated Office 365 Group.
  • When creating a Workspace, you can define various settings such as the type of access each user has (read-only or ability to modify its content), its members and whether it requires assignment to a Power BI Premium node. It is not possible to change the access type for a Workspace after creation, but you can freely change its name or modify its membership at any time.
  • The contents of a Workspace can be published as an App, enabling you to expose your solution to a broader audience within or outside your organisation. Once published, users navigate to the Power BI AppSource store for their tenant, which lists all Apps available for installation. Once installed, they will then become visible from within the Apps area of the application. You can update content within an App at any time by republishing its corresponding Workspace. It is also possible to define individual properties within an App, such as its description, access rights and landing page. To install and use Apps, the user in question must have a Power BI Professional license.

Additional Preperation Resources

The official Microsoft exam reference book is a helpful learning aid for this exam, particularly given that it includes numerous exercises that you can work through to familiarise yourself with different Power BI functionality. There is also an online course available on the edX website which, despite not covering the whole exam syllabus, does provide a useful visual aid and includes a lot of the features you are expected to know for the exam. Finally, nothing beats actually working with the product itself and trying out the various features yourself. Power BI Desktop is a free download and, with access to one of the sample databases provided by Microsoft, you can very quickly provision an environment on your own home computer to enable you to experience everything that Power BI has to offer.

Exams are always a nightmarish experience, both when preparing for them and when you are sat there in the test centre. I hope that this post, and this whole series, proves to be useful in helping with your exam preparation and getting you ready to pass the exam with flying colours šŸ™‚

The New Year is almost upon us, meaning its time to put in place some resolutions for the year ahead. I can think of no better commitment then to learn more about Power BI in 2019, which is hopefully the reason why you are reading this right now šŸ™‚ . Welcome to the eighth post in my series concerning Microsoft Exam 70-778, where I hope to provide a learning/revision tool for anyone who is taking the exam or looking to increase their Power BI expertise. Last week, we investigated how to manage Key Performance Indicator (KPI) reporting using Power BI. We now move into another topic area that is also tied closely to visualizations – Create hierarchies. The related skills for this exam area are:

Create date hierarchies; create hierarchies based on business needs; add columns to tables to support desired hierarchy

In the sections that follow, I will provide an overview of the two different types of Hierarchies configurable within Power BI, before then providing a step-by-step guide on how to create one. As with previous posts in this series, this subject does bring together some other skill areas, such as importing data with Power Query and using DAX calculated columns. Knowledge of these topics will hold you in good stead when starting with hierarchies.

What is a Hierarchy?

Hierarchies form the cornerstone of a broad aspect of human existence, which makes it natural they are a common theme when it comes to Business Intelligence (BI) solutions. A hierarchy is definable as a logical, often visual, representation of an order of precedence. Hierarchies, in strict Power BI terms, do not differ much from this general definition; they allow developers to order data by preference, priority or anything in between. When configured and used in isolation, they offer minimal benefit to Power BI report consumers. They start to become really valuable when utilised alongside visualizations, providing an additional interaction point for key data points and allowing report users to tailor a visualization to suit their needs. Some potential usage scenarios for hierarchical data may include:

  • Providing top-level and subcategory classifications for product records.
  • Modelling a business reporting line hierarchy, from CEO down to Developer.
  • Splitting sale date by quarter, year or even month.

Hierarchies come in two flavours within Power BI – Date Hierarchies and Custom (or User Defined) Hierarchies. The next two sections delve deeper into the inner workings of each.

Date Hierarchies

Date Hierarchies are best described as a logical breakdown of the constituent components of a date value, based on four different levels:

  • Year
  • Quarter
  • Month
  • Day

For each field that contains a Date Hierarchy, the appropriate value for each of the above is exposed out for utilisation across Power BI Desktop. So, for example, the underlying hierarchy values for the 3rd March 2017 would be:

  • Year: 2017
  • Quarter: Qtr 1
  • Month: March
  • Day: 3

All columns with a data type of Date or Date/Time will have a Date Hierarchy created for them automatically. The hierarchy will then become accessible for use in the following areas of Power BI Desktop:

  • When building out formulas using DAX. For example, the screenshot below shows how it is possible to access the hierarchy field values listed above from the Date/Time field LastEditedWhen:
  • When working with a Table visualization. Adding a Date or Date/Time column into the Values well will automatically create a table containing all of the hierarchy fields. An example of how this looks for the LastEditedWhen field is seen in the image below:

It is possible to disable the automatic creation of Date Hierarchies within the File -> Options area of Power BI Desktop, by toggling the Auto Date/Time option. Take care when toggling this though, as doing so will remove any existing Date Hierarchies within your model:

This fact is confirmed when returning to the Table created using the LastEditedWhen field, which automatically reverts to the base column value when theĀ Auto Date/Time option is disabled:

Custom Hierarchies

For more bespoke requirements, a Custom Hierarchy affords the same kind of functionality discussed in the previous section. They can be created from the right-hand Fields pane by selecting a field within a Power BI table and choosing the New hierarchy option:

Once established, you can then:

  • Rename the hierarchy. By default, it will be named using the convention <Field Name> Hierarchy.
  • Include additional fields by dragging and dropping them into the hierarchy.
  • Reorder the hierarchy, by dragging and dropping the fields into your preferred order, from top to bottom.
  • Delete the hierarchy through right-clicking it and selecting the Delete option. All fields included in the Hierarchy will remain as part of the underlying query.

A Custom Hierarchy typically relies on a self-relationship that a query might have on itself. An excellent example of this, already alluded to, is an Employee table, where an individuals Manager is trackable via a ManagerID column, that resolves back to an EmployeeID record. In these cases, there are several DAX functions available that will assist in getting custom columns set up to support your hierarchy, described collectively as Parent and Child Functions. Usage of these functions is essential when building out Hierarchies involving parent/child relationships.

TheĀ exercise at the end of this post will go through the detailed steps involved in creating a Custom Hierarchy.

Using Hierarchies with Visualizations

A good reason to include hierarchies as part of your Power Bi Reports is the options they unlock from a data drill-down perspective. We’ve seen already the behaviour Date Hierarchies adopt when using the Table visualization, which offers nothing in this respect. Other visualization types support, by comparison, far richer options for “homing in” on a particular piece of data, epitomised by the following buttons that appear at the top of the supported visualization:

In order, from left to right, these let you:

  • Drill up your data one level in the hierarchy.
  • Enable the Click to turn on Drill Down feature, allowing you to click on any part of a visual to drill-down to the next level of the hierarchy.
  • Go to the next level in the hierarchy.
  • Expand all down one level in the hierarchy. This behaves differently to the Click to turn on Drill Down option mentioned already, by grouping together the previous hierarchy level each time you drill-down.

An example of how all this works as part of a Pie chart visualization is viewable in the sequence below:

As should hopefully be clear, hierarchies are very much the icing on the cake in building an engaging and wholly interactive reporting solution within Power BI.

Example: Creating a Hierarchy and adding it to a Visualization

The steps that follow provide instructions on how to build out a product categorisation hierarchy, using sample data, and then how to apply this to a Donut chart visualization. All that is required to work through these steps is access to the Power BI Desktop application:

  1. In Power BI Desktop, click onĀ Edit Queries to open the Power Query Editor. Navigate to the New Source button and select the Blank Query option:
  2. Select the newly created Query1 in the Queries pane and click the Advanced Editor button. Copy and paste the following M code into the window and press Done:
    1. let
          Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("nZNNbsMgEIWvgrxqJRb438mybZZRonrpZoGScYNiYwvsqu11epOerBjstHKwE3WBhJj3vjdoIMsc18HO4/MabXjBOCzRijcgasEkoDtJy7qA+5sUgVrfXwEOCXF2OHO8kWsrqhykZBWnxRR5SmPYHo6jRLP9kS9tKD9QcZji2uqGGeE4cTUz6D0piDe2v7i85ThSS1tDtVm974+Uv0Kf+Vc3Uzojus0mz7sEPwptafPVgROPlC1rJinj2sBIfnVPIE9NVVsIlsrgX1yq8kqgNTtI9gnopSXEiyZe0X99iZ5mgheumaZL7LC0pEWBHlqpBiGlenBQsra80sF1k4kPfM+Ed38qPVIB24rxxjL32WKkYS4h2Pf623SfKT191KD7OXdiAd8iMgEhDn2F3/0A", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type text) meta [Serialized.Text = true]) in type table [ProductID = _t, Name = _t, #"Product ID" = _t, Parent = _t, TotalSales = _t]),
          #"Changed Type" = Table.TransformColumnTypes(Source,{{"ProductID", Int64.Type}, {"Name", type text}, {"Product ID", type text}, {"Parent", Int64.Type}, {"TotalSales", Currency.Type}}),
          #"Replaced Value" = Table.ReplaceValue(#"Changed Type",null,0,Replacer.ReplaceValue,{"TotalSales"})
      in
          #"Replaced Value"
      
  3. This will then load up a table object that resembles the below, which derives from the sample Product data from Dynamics CRM/365 Customer Engagement:
  4. Right-click the Query1 query and use theĀ Rename option to change its name to Products. Select the Close & Apply button to exit the Power Query Editor.
  5. Open the Data pane, select the Products query and use the following DAX formulas to create New Columns for the table:
    • ProductHierarchy = PATH(Products[ProductID], Products[Parent])
    • Product Category = LOOKUPVALUE(Products[Name], Products[ProductID], PATHITEM(Products[ProductHierarchy], 1, INTEGER) )
    • Product Grouping = LOOKUPVALUE(Products[Name], Products[ProductID], PATHITEM(Products[ProductHierarchy], 2, INTEGER) )
    • Base Product = LOOKUPVALUE(Products[Name], Products[ProductID], PATHITEM(Products[ProductHierarchy], 3, INTEGER) )
  6. The first field creates the required hierarchy list in a delimited format; the remaining columns then list out the three levels of the hierarchy as separate column values. The data in your table should resemble the below if done correctly:
  7. Right-click the Product Category field in the Fields pane and select New Hierarchy, to create a new hierarchy as indicated below:
  8. Modify the newly created hierarchy so that:
    • It is named Product Hierarchy
    • It contains, in addition to the Product Category field, the Product Grouping and Base Product fields.
    • The order of the newly added fields reflect the following:
      • Product Category
      • Product Grouping
      • Base Product
  9. If done correctly, the Hierarchy should resemble the following:
  10. Click on the Report tab and add an empty Donut chart visualization to the report. Populate the Field well values as shown below:
  11. The Donut chart should update accordingly, and with the appropriate drill-down options available at the top of the visualization. Notice also that hovering over each section of the Donut chart displays an aggregated total for that level of the hierarchy:

Key Takeaways

  • Hierarchies within Power BI provide a means of logically categorising data into an order of preference or precedence, providing greater flexibility to Power BI report users when they interact with visualizations.
  • Date Hierarchies are created and managed automatically by Power BI for each Date or Date/Time field defined within your model. These automatically create fields that contain the Year, Quarter, Month & Day values from the respective date fields. These fields can then be utilised as part of a Table visualization or within a DAX formula. Date Hierarchies can also be completely disabled if required.
  • Custom (or User-Defined) Hierarchies need to be created manually and provide additional customisation options when compared to Date Hierarchies, such as the number of fields they contain, the order and its name. A Custom Hierarchy will typically make use of one of several Parent/Child DAX functions, such as PATH or PATHITEM.
  • Including a hierarchy as part of a chart visualization, such as a Pie chart or Donut chart, opens up other drill-down capabilities around your data. Indicated by the additional arrow icons included at the top of the visualization, they provide the means for users to interrogate data points that interest them the most straightforwardly.

The past couple of posts have already started to introduce some of the visualization options available within Power BI. In next week’s post, we will take a much closer look at all of the options available within the application to display data, as well as look at features such as Bookmarks and the various report page customisation tasks available to developers when building out a report.

Another week, another Power BI post šŸ™‚ This is the fifth post in my series focusing on Microsoft Exam 70-778, where I aim to provide a detailed overview and description of the skills needed to tackle the exam successfully. Last week’s post rounded off the opening theme, Consuming and Transforming Data By Using Power BI Desktop, as we focused on the options available to us to help tidy up data within the Power Query Editor. We now start to move outside of this area of the application with the Modelling and Visualising Data theme and, specifically, the following skill area:

Manage relationships; optimize models for reporting; manually type in data; use Power Query

As before, to follow on as part of any examples, make sure you have downloaded the WideWorldImporters sample database and hooked up Power BI Desktop to the tables described in my first post in this series.

Relationships

It will be a rare occurrence if you find yourself regularly working with a single table object within Power BI. The required data for most reporting situations will generally reside in different locations – whether within separate tables in a single SQL database or across different instances or applications. In most cases, there will be some field – or key – that will link records together and act as the bedrock when defining Power BI Relationships.

For those who are coming from a SQL or Dynamics 365 Customer Engagement background, I would expect that the concept of Relationships will present little difficulty in grasping. But, for those who are coming in cold, it is worth considering the following question – Why might you want to look at defining Relationships in the first place? The reasons will generally be specific to your business need, but can be generalised as follows:

  • Database normalization creates a degree of complexity that can be difficult to unweave from a reporting standpoint. Customer records may exist in one table, Address details in another, their orders in a third…you get the picture. Consequently, there is a need to bring this all together into a single/simplified view when developing reporting solutions; Relationships help to meet this objective.
  • The great benefit of using Power BI is its ability to provide a consistent, contextual filtering experience across Reports. So, for example, when adjusting the range on a Date slicer, all other visualisations are automatically refreshed to only show data from the updated period. This wizardry is all achieved by Relationships and the cross-filtering rules you define for them.
  • Power BI data models can grow in complexity over time. The defining of Relationships simplifies this to a degree and allows you to view all of your queries in a clean, visual manner.

We can see an example of how a Power BI Relationship looks in the picture below:

We can see above two one-to-many (1:N) Relationship between the Sales.Customers and Sales.Invoices tables from the WideWorldImporters database. The matching key, in this case, is the CustomerID field.

Relationships are created and managed in one of two ways:

  • Automatically when bringing data into Power BI Desktop the first time. Power BI will automatically detect and build any Relationships within a SQL Server-derived data source, based on any PRIMARY KEY/FOREIGN KEY CONSTRAINTs that exist already. The Select Related Tables option can be used to intelligently determine which tables can be brought in on this basis.
  • By going to the Manage Relationships button on the Home tab within the main Power BI window (NOT the Power Query Editor). This option will allow you to manage and create new Relationships based on your requirements. An Autodetect… option is also available, which behaves similarly to the data source auto-detection discussed previously. The example below shows the WideWorldImporters Relationships referenced earlier:

The walkthrough exercise at the end of this post will go through the process of creating a Relationship from start to finish, but it is useful at this stage to dive into the more detailed concepts involving Relationships:

  • The following types of Relationships (or Cardinality) can be specified:
    • One to Many (1:N) / Many to One (N:1)
    • One to One (1:1)
    • Many to Many (N:N) – This feature is currently in Preview and caution is advised when using it.
  • Each Relationship requires that a Cross filter direction is specified, which will determine how filters cascade out to related tables. These are configurable as either Single (in the direction of your choosing – left to right or right to left) or Both. 1:N Relationships must always use Both.
  • Only one Relationship is classifiable as Active between two tables at any given time. Despite this limitation, it is possible to define multiple Relationships, if required. It is essential to understand with this that only Active Relationships are utilisable when performing DAX column/measure calculation, which may lead to unintended results. You can get around this by either changing the Inactive Relationship to Active or by taking advantage the USERELATIONSHIP function to force DAX to pick your preferred Relationship.
  • The Assume Referential integrity option is only available when you are using DirectQuery and, if enabled, underlying queries will attempt to use INNER JOINs as opposed to OUTER JOINs in any query folding. While this can speed things up, it may also lead to missing or incomplete results.

Formatting Columns

We saw in last week’s post some of the more functional data formatting options that the Power Query Editor provides. I say the word functional in the sense of meaning formatting that serves a specific, data quality need, as opposed to a styling requirement. Moving outside of the Power Query Editor and we can discover a whole host of additional styling formatting options on the Modelling tab:

Let’s break all this down into a bit more detail:

Formatting

  • Data Type: Here, it is possible to override the data type for a particular column value. The options available are:
    • Decimal Number
    • Fixed decimal number
    • Whole Number
    • Date/Time
    • Date
    • Time
    • Text
    • True/False
    • Binary
  • Format: Depending on the Data Type value selected, the list of choices here will change accordingly. In most cases, such as forĀ Text values, you will only have a single selection available, but your options for Date/Time and Currency values are considerably richer. For example:
    • Date/Time values can be formatted in numerous different ways, both from a styling and locale perspective. This includes formatting dates in the good ol’ British way, meaning that the ISO 8601 date value 2018-01-01 can appear as 01-01-2018, 1 January 2018 or even Monday, 1 January 2018.
    • Currency values are modifiable to any global currency sign or shortcode. So, the value 50 is displayable as both Ā£50 and 50 GBP for British Pound currency figures.
  • $ Button: This is, in effect, an alternate way of changing the Currency value on a field, as described previously.
  • % Button: Converts a value to a percentage. For example, the Whole Number value of 75 would appear as 75%.
  • ‘ Button: Adds a thousand separator to a number value. Therefore, 100000 becomes 100,000.
  • .0/.00 Button: Adjusts the number of decimal places at the end of a numeric, up to a maximum limit of 15.

Properties

  • Data Category: This option, designed primarily for String values, allows you to indicate the type of data the field represents, with a specific focus towards mapping capabilities. For example, it is possible to mark a field as State or Province, to ensure that Power BI does not make an incorrect assumption when attempting to plot an address to a map. Some additional options here include:
    • Latitude/Longitude to, again, ensure accurate plotting to a map visual.
    • Classifying a field as containing either a Web URL or Image URL.
    • Categorising a field as Barcode value.
  • Default Summarization: When building out a report and adding fields to visualizations, Power BI assumes the preferred aggregation value to use. This will be pretty spot on in most cases – for example, Currency fields will default to a Sum aggregation. However, it is possible to override this option to use any of the following, easily guessable, alternatives:
    • Don’t summarize.
    • Sum
    • Average
    • Minimum
    • Maximum
    • Count
    • Count (Distinct)

Sort By Column

There may be a requirement to perform sorting of column values, based on some predefined logic as opposed to merely alphabetical or numeric order. The best example to provide here would be for Month Names. Instead of sorting in alphabetical order, like so:

April

August

December

February

etc.

We would want to sort by logical Month order i.e.:

January

February

March

April

etc.

Another scenario could be column sorting based on High, Medium and Low categorisation; all records with a value of High would need to appear first and the Medium second, as opposed to Low if sorted alphabetically.

Sort By Column gives us the flexibility to achieve these requirements, but it is crucial first to ensure that a proper sorting column resides within your query. Going into the Power Query Editor and modifying theĀ Sales CustomerTransactions table enables us to add on a Month Number column for the TransactionDate field by going to the Add Column tab and selecting Date -> Month -> Month option:

Coming out of the Power Query Editor using the Close & Apply option, go into the Sales CustomerTransactions table within theĀ Data tab and select theĀ Sort By Column option, selecting theĀ MonthNumberĀ field as the Sort by Column:

We should now see when adding a simple Table visualization to a report, that the Month values derived from TransactionDate appear in correct sort order:

Through this example, you can hopefully see how straightforward it is to accommodate more unique sorting requirements with minimal effort.

Manual Data Entry

Although Power BI assumes, for the most part, no requirement to define any manual data as part of your reports, there may be situations where this need arises. For example, on the WideWorldImporters Sales.CustomerTransactions table that has been built out through this series, you may want to define a manual list of all FullName values that can then be linked up to a slicer for filtering purposes. There are two ways this requirement can be met, with both options having their benefits and potential preference, based on how much you like to code šŸ™‚ :

  • The Enter Data button on the Power Bi Desktop Home tab allows you to create a new Table object based on manual data entry. The table can have as many columns as you like and, what’s nifty is that it supports full copy + paste functionality from other applications. It is a VERY convenient feature to bear in mind:
  • Using PowerQuery, it is also possible to define custom Tables populated with data. We saw an example of how to achieve this in last weeks post when creating some example data for cleansing, but there are some further options available to us:
    • Simple lists of data can appear within brackets. For example, the following M code:
      • let
            Source = { "This", "Is", "A", "Test"}
        in
            Source
    • Would generate the following data within Power Query:
    • Records with multiple rows/fields can also be generated using the example code below:
      • Table.FromRecords({[Did = "This", You = "is", Know = "Testing"]})
    • Which would appear as follows:
    • Finally, as shown last week, it is possible to create a Table object, using the example code below:
      • #table(
            {"Forename", "Surname", "Description"},
            {
                {"JANE","smith","   this describes the record"},
                {"alan", "JOHNSON", "    record description detected    "},
                {"   MARK", "CORRIGAN    ","another description"},
                {"JANE","smith","   this describes the record"}
            }
        )

The tools available here help to satisfy many potential ad-hoc requirements when using Power BI Desktop and when used in conjunction with other features described in this post and earlier in the series, start to become pretty powerful.

Example: Creating Relationships

In this very straightforward example, we will see how it is possible to create a Relationship manually from within Power BI Desktop, using the WideWorldImporters sample database as our connection source:

  1. It will be necessary to disable to automatic creation of Relationships when importing data for this exercise. This is done by going to the Options -> Data Load screen and ensuring that the appropriate option has been unticked:

  1. Follow the instructions discussed in this post to connect to the WideWorldImporters database but, on this occasion, ensure that the following tables only are selected for import:

  1. Once the data loads into Power Bi, navigate to the Relationships tab and you should see the two table objects indicated below (you may need to resize them accordingly for all fields to display):
  2. Because of the setting change carried out in step 1, the underlying Relationship between both of these tables in SQL has not been detected and added. To fix this, click on the Manage Relationships button on the Home tab and then the New button to open the Create relationship window, as indicated below:

  1. Define the following settings for the new Relationship, as described and illustrated below:
    • In the first drop-down box, select Sales Orders and verify that the OrderID column highlights itself.
    • In the second drop-down box, select Sales OrderLines and verify that the OrderID column highlights itself.
    • For Cardinality, ensure One to many (1:*) is selected.
    • For Cross filter direction, ensure Both is selected.
    • Ensure Make this relationship active is ticked.

  1. Press OK and then Done to create the Relationship. The Relationship window should refresh accordingly to indicate that a new Relationship exists between the two tables:

Key Takeaways

  • Relationships form the cornerstone of ensuring the long-term viability and scalability of a large data model. Assuming you are working with well-built out, existing data sources, Power BI will automatically detect and create Relationships for you. In situations where more granular control is required, these Relationships can be specified manually if needed. It is worth keeping in mind the following important features of Relationships:
    • They support one-to-one (1:N), one-to-many (1:N) and many-to-one (N:1) cardinality, with many-to-many (N:N) currently in preview.
    • Filter directions can be specified either one way or bi-directionally.
    • Only one relationship can be active on a table at any given time.
  • It is possible to sort columns using more highly tailored custom logic via the Sort By Column feature. The most common requirement for this generally involves the sorting of Month Names in date order but can be extended to cover other scenarios if required. To implement, you should ensure that your data has a numbered column to indicate the preferred sort order.
  • Moving outside of the Power Query Editor presents us with more flexibility when it comes to formatting data to suit particular styling or locale requirements. While the majority of this functionality provides date/time and currency formatting options, for the most part, it is also possible to categorise data based on Location, the type of URL it is or on whether or not it represents a Barcode value; these options can assist Power BI when rendering certain types of visualizations.
  • There may be ad-hoc requirements to add manually defined data into Power BI – for example, a list of values that need linking to a Slicer control. The Enter Data button is the “no-code” route to achieving this and supports the ability to copy & paste data from external sources. For more advanced scenarios, you also have at your disposal a range of M code functionality to create Lists, Records and Tables, which can be extended further as required.

Hopefully, by now, you are starting to get a good feel for how Power BI works and also the expected focus areas for the exam. Next weeks post is going to be a biggie, as we jump head first into DAX formulas and how they can be used for calculated columns and Measures. We’ll also introduce the concept of What-if Parameters and how they work in practice.

Welcome to the third post in my series on Microsoft Exam 70-778, where I aim to provide a detailed overview and description of the skills needed to tackle the exam successfully. We saw in the previous post how we could use the Power Query Editor to perform a multitude of different transformations against data sources; this will now be taken further, as we start to look at how to ensure optimal quality within our Power BI data models. The relevant skills for this area as follows:

Manage incomplete data; meet data quality requirements

To follow on as part of the examples below, make sure you have downloaded the WideWorldImporters sample database and hooked up Power BI Desktop to the tables described in my first post in this series. With all this done, it is crucial first to grapple a vital concept relating to the management of incomplete data, namely, the ability to…

Filter Data

If you are consuming an entire SQL tables worth of data within the Power Query Editor, the size of your model can grow over time. In this scenario, it may be necessary to apply column-level filters to your data directly within Power Query, as opposed to merely dumping a cartload of irrelevant data in front of your users. Fortunately, the whole experience of filtering data should be a cakewalk, given its similarity with Microsoft Excel filters, as we can see below when attempting to filter the FullName field on theĀ WideWorldImporters Sales.CustomerTransactions table:

This particular feature does have some differences compared with Excel, though:

  • For large datasets, only a subset of all total, distinct record values are returned. This fact is indicated above via the List may be incomplete warning sign. Clicking the Load more option will do exactly that, but may take some time to process.
  • The range of filters available will differ depending on the data type of the column. For example, String data types will have filters such as Begins With...,Ā  Does Not Begin With…, whereas Dates are filterable based on Year, Quarter, Month etc.
  • The Remove Empty option will do just that – remove any columns that have a blank or NULL value.
  • As discussed on the Performing Data Transformations post, when you start combining filters with Parameters, it is possible to transform particularly unwieldy filtering solutions into more simplistic variants.

Column Errors

When it comes to adding custom columns as part of your Power Query transformations, there is always the potential for these to error, due to a misconfigured calculation or some other kind of unforeseen issue. When this occurs, the corresponding column value is flagged accordingly within the Power Query Editor and can be inspected further to determine the root cause of the issue. To demonstrate this, add on the following custom column using the following formula onto the Sales.CustomerTransactions table

[AmountExcludingTax] * [Error]

The value for each row in the new column should resemble the following:

When this issue occurs, it may be most prudent to first try and address the issue with your underlying calculation, ideally fixing it so that no error occurs at all. Where this is not possible, you can then look towards removing any rows that contain such a value. We will see how this can be done in the Adding/Removing Rows section later in this post.

Blank Data

On the flip side of field value errors is blank data. In most cases, when working with SQL data sources, this will rear its head when there are NULL values in your database. For example, take a look at below at the CreditLimit field on the WideWorldImporters Sales.Customers table:

When these fields are fed through from Power Query and relied upon as part of DAX custom column/Measure creation, you may start to get some unintended consequences. For example, after filtering the same table above only to retain rows where the CreditLimit equals null, attempting to create a Measure that totals up all CreditLimit values results in the following when displayed as a Card visualisation:

If you, therefore, have a desire to perform additional aggregations or custom calculations on fields that contain blank/null values, then you should take the appropriate steps to either a) remove all rows that contain one of these two values or b) perform a Replace action on the column to ensure a proper, default value appears instead. For the CreditLimit field, this can be as simple as replacing all null values with 0:

Adding/Removing Rows

Often our data sources are not pristine clean from a data perspective – duplicate rows may be common, it could be that rows exist with completely blank or null values or your incoming data file could be a complete mess from a column header perspective. With this problem in mind, the Power Query Editor provides us with the functionality to keep or remove rows based on several different conditions:

The options granted here should be reasonably self-explanatory, but the list below contains some additional guidance if you need it:

  • Keep/Remove Top Rows: Keeps or removes the top number of rows, in ascending order, based on the amount you specify.
  • Keep/Remove Bottom Rows: Keeps or removes the bottom number of rows, in descending order, based on the number you specify.
  • Keep Range of Rows: Keeps the number of rows specified based on the starting row number. For example, for a 50-row table, if a First row value of 1 and a Number of rows value of 10 is selected, then the first ten rows will be retained.
  • Keep/Remove Duplicates: Based on the currently selected column(s), keeps or removes all rows with duplicate values.
  • Keep/Remove Errors: Based on the currently selected column(s), keeps or removes all rows that have an Error value.
  • Remove Blank Rows: Removes any row that has a blank or NULL value.

Formatting Column Data

Data from a live, production system, such as Dynamics 365 Customer Engagement, can sometimes be a complete mess from a readability perspective; incorrect casing and invalid characters are typically commonplace in this situation. Fortunately, there are a range of options at our disposal with the Power Query Editor, on the Transform tab:

Most of these are self-explanatory, with the exception of the Trim and Clean options:

  • Trim removes any leading/trailing whitespace characters from a string value.
  • Clean detects and removes any non-printable characters from a string value.

Although not technically a data cleansing options, there are some clear usage scenarios for the Add Prefix & Add Suffix options, such as creating unique reference code for each column value, based on the unique record number value (e.g. ABCD-1, ABCD-2 etc.).

Formatting options for other column types are not available from within Power Query. So if, for example, you wished to format all date values in the format YYYY-MM-DD, you would have to move outside of the Power Query Editor to achieve this. The steps involved to accomplish this will be a topic for a future post.

Example: Cleansing Data

Having reviewed each of the possible cleansing options at our disposal, let’s now take a look at an example of how to cleanse a troublesome dataset:

  1. Within the Power Query Editor, on the Home tab, select the New Source -> Blank Query option. This will create a new Query in the left-hand pane called Query1.
  2. Select Query1 and press F2 to allow you to rename it to CleanseExample.
  3. Right-click the CleanseExample query and select the Advanced Editor option:
  4. Within the Advanced Editor window, copy & paste the following code into the window:
    • #table(
          {"Forename", "Surname", "Description"},
          {
              {"JANE","smith","   this describes the record"},
              {"alan", "JOHNSON", "    record description detected    "},
              {"   MARK", "CORRIGAN    ","another description"},
              {"JANE","smith","   this describes the record"}
          }
      )
  5. It should resemble the below if done correctly:
  6. When you are ready, press the Done button. PowerQuery will then create a table object using the code specified, populating it with records, as indicated below:
  7. There are three key issues with this data that need resolving:
    • The inconsistent word casing on the Forename/Surname.
    • Whitespacing on the Description and ForeName fields.
    • Duplicate records.
  8. These issues are fixable by taking the following action:
    • For the casing issue, CTRL + left click to select the Forename & Surname fields, go to the Transform tab and select Format -> Capitalize Each Word. Your data will then be modified to resemble the below:
    • For the whitespace issue, select the Forename & Description fields and, on the Transform tab, select Format -> Trim:
    • Finally, to remove the duplicate record for Jane Smith, highlight the Forename & Surname fields, navigate to the Home tab and select Remove Rows -> Remove Duplicates. This will then leave us with three records, as illustrated below:
  9. As a final (optional) step, we can also look to clean up the Description field values by applying the Capitalize Each Word formatting option:

Et voilĆ ! We now have a tidy and clean table, ready for consumption within Power BI šŸ™‚

Key Takeaways

  • Data can be filtered directly within Power Query, using Excel-like functionality to assist you in only returning the most relevant data in your queries. The data type of each field plays a particularly important part of this, as only specific filter options will be at your disposal if, for example, you are working with numeric data.
  • From a data quality perspective, you typically will need to handle column values that contain one of two possible value types:
    • Errors: This will usually occur as a result of a calculated column field not working correctly. The best solution will always be to address any issues with your calculated column, such as by using a conditional statement to return a default value.
    • Blanks/NULLs: A common symptom when working with SQL derived data sources, your real problems with blank values start to appear when you attempt to implement DAX custom columns/Measures outside of the Power Query Editor. It is, therefore, recommended that these are dealt with via a Replace action, depending on your fields data types. For example, a number field with blank/NULL values should be replaced with 0.
  • The Remove Rows option(s) can act as a quick way of getting rid of any Error or Blank/NULL rows and can also be utilised further to remove duplicates or a range of rows. In most cases, you will have similar options available to you with Keep Rows instead.
  • There are a variety of formatting options available to us when working with text/string data types. These range from fixing capitalisation issues in data, through to removing whitespace/non-printable character sets and even the ability to prepend/append a new value.

Data cleansing is a reasonably short subject area in the grander scheme of things, but the steps covered represent key stages towards building out a competent and trustworthy reporting solution. The next post in the series will discuss the options available to us in building out more complex and bespoke data models.