When it comes to handling large datasets in a formal, structured and highly scalable manner, nothing beats SQL Server. Having worked with the product for almost six years, I always look forward to the opportunity of putting together some SQL queries or to build out a new database. I think of it as a nice little treat, a reward amongst the midst of other, sometimes tedious, tasks that I have to deal with on a weekly basis. I rank knowledge of SQL Server pretty highly if you are aiming to become a serious Dynamics CRM/365 for Customer Engagement professional and I credit my experience with SQL as one of the things that helped to streamline my journey into Dynamics.
It may be, however, that others are not as keen at the prospect of working with SQL databases, particularly when it is unable to accommodate some of the alternative data storage mechanisms that are commonplace with application developers. A good example of this is JSON (JavaScript Object Notation), a format that is used widely today as a portable and “easy to read” mechanism of transferring data. For someone who is more used to working with SQL, getting your head around JSON can be a bit of a challenge at first and – traditionally – was not something that Microsoft factored into the design of SQL Server. A lot has changed with Microsoft – to the extent that services such as Microsoft Flow and Azure now use JSON extensively – and with SQL Server, as a whole host of related functions were added to SQL Server 2014 to provide straightforward conversions into JSON. The FOR JSON PATH clause is the primary gateway into this world and is a function which I have slowly, but surely, started to get my head around. What I wanted to do in this week’s blog post was provide a “crash course” on how to use this nifty piece of functionality, hopefully with the aim of giving you everything you need to start utilising it in your environment(s).
Before we begin…
To best illustrate how the clause works in practice, it is necessary to create an appropriate set of inter-related tables within SQL Server, that will be used for all examples that follow. Here’s one I (rather unimaginatively) made earlier:
The code snippets to create them can be found below:
CREATE TABLE dbo.[ParentTable]
(
ParentTableID INT IDENTITY(1,1) PRIMARY KEY NOT NULL,
Field1 BIT NOT NULL,
Field2 CHAR(10) NULL,
Field3 VARCHAR(MAX) NULL
)
GO
CREATE TABLE dbo.[ChildTable1]
(
ChildTable1ID INT IDENTITY(1,1) PRIMARY KEY NOT NULL,
ParentTableID INT FOREIGN KEY REFERENCES dbo.[ParentTable](ParentTableID) NULL,
Field1 BIT NOT NULL,
Field2 CHAR(10) NULL,
Field3 VARCHAR(MAX) NULL
)
CREATE TABLE dbo.[ChildTable2]
(
ChildTable2ID INT IDENTITY(1,1) PRIMARY KEY NOT NULL,
ParentTableID INT FOREIGN KEY REFERENCES dbo.[ParentTable](ParentTableID) NULL,
Field1 BIT NOT NULL,
Field2 CHAR(10) NULL,
Field3 VARCHAR(MAX) NULL
)
GO
CREATE TABLE dbo.[GrandchildTable]
(
GrandchildTableID INT IDENTITY(1,1) PRIMARY KEY NOT NULL,
ChildTable2ID INT FOREIGN KEY REFERENCES dbo.[ChildTable2](ChildTable2ID) NULL,
Field1 BIT NOT NULL,
Field2 CHAR(10) NULL,
Field3 VARCHAR(MAX) NULL
)
GO
The table structures are incredibly basic, but note, in particular, the FOREIGN KEY relationships from the 2 Child Tables to the Parent and also the additional parent/child relationship between the GrandchildTable and ChildTable2. You will also need to look at inserting some test data into the tables to properly follow through the rest of this post.
With our environment prepped, let’s take a look at the different ways we can convert our dataset into JSON format, with minimal effort involved.
Example 1: FOR JSON AUTO
If we were to look at doing a straightforward SELECT * query on all our tables, our query and expected output might look something like this:
SELECT *
FROM dbo.ParentTable AS PT
INNER JOIN dbo.ChildTable1 AS CT1
ON PT.ParentTableID = CT1.ParentTableID
INNER JOIN dbo.ChildTable2 AS CT2
ON PT.ParentTableID = CT2.ParentTableID
INNER JOIN dbo.GrandchildTable AS GT
ON CT2.ChildTable2ID = GT.ChildTable2ID
Our main issue with this query is that, because of how T-SQL works, the 2 ParentTable records are returned for every child and grandchild record. For a client application, this can be somewhat cumbersome to handle. FOR JSON AUTO can be straightforwardly added to the end of the above query to convert the query output accordingly:
SELECT *
FROM dbo.ParentTable AS PT
INNER JOIN dbo.ChildTable1 AS CT1
ON PT.ParentTableID = CT1.ParentTableID
INNER JOIN dbo.ChildTable2 AS CT2
ON PT.ParentTableID = CT2.ParentTableID
INNER JOIN dbo.GrandchildTable AS GT
ON CT2.ChildTable2ID = GT.ChildTable2ID
FOR JSON AUTO
//Example output of the first 25 lines below:
[{
"ParentTableID": 1,
"Field1": true,
"Field2": "Test ",
"Field3": "This is a test record",
"CT1": [{
"ChildTable1ID": 1,
"ParentTableID": 1,
"Field1": true,
"Field2": "Test ",
"Field3": "This is a test record",
"CT2": [{
"ChildTable2ID": 1,
"ParentTableID": 1,
"Field1": false,
"Field2": "Test ",
"Field3": "This is a test record",
"GT": [{
"GrandchildTableID": 1,
"ChildTable2ID": 1,
"Field1": false,
"Field2": "Test ",
"Field3": "This is a test record"
}]
...
This output provides a much more sensible structure, with no record duplication and proper nesting of child records.
Example 2: FOR JSON PATH, ROOT
With some modifications to the above query, it is also possible to specify names for each root element. This can be tailored depending on your specific requirements. For example, let’s say we had to provide the following root element names for each of the example tables:
- dbo.ParentTable -> Parent
- dbo.ChildTable1 -> FirstChildTable
- dbo.ChildTable2 -> SecondChildTable
The following query would achieve these requirements, in addition to adding a master root element name of MyTestSQLJSONObject:
SELECT PT.ParentTableID AS [Parent.ParentTableID], PT.Field1 AS [Parent.Field1], PT.Field2 AS [Parent.Field2], PT.Field3 AS [Parent.Field3],
ChildTable1ID AS [FirstChildTable.ChildTable1ID], CT1.Field1 AS [FirstChildTable.Field1], CT1.Field2 AS [FirstChildTable.Field2], CT1.Field3 AS [FirstChildTable.Field3],
CT2.ChildTable2ID AS [SecondChildTable.ChildTable1ID], CT2.Field1 AS [SecondChildTable.Field1], CT2.Field2 AS [SecondChildTable.Field2], CT2.Field3 AS [SecondChildTable.Field3],
GT.GrandchildTableID AS [GrandchildTable.GrandchildTableID], GT.Field1 AS [GrandchildTable.Field1], CT2.Field2 AS [GrandchildTable.Field2], CT2.Field3 AS [GrandchildTable.Field3]
FROM dbo.ParentTable AS PT
INNER JOIN dbo.ChildTable1 AS CT1
ON PT.ParentTableID = CT1.ParentTableID
INNER JOIN dbo.ChildTable2 AS CT2
ON PT.ParentTableID = CT2.ParentTableID
INNER JOIN dbo.GrandchildTable AS GT
ON CT2.ChildTable2ID = GT.ChildTable2ID
FOR JSON PATH, ROOT('MyTestSQLJSONObject')
//Example of first 25 lines below:
{
"MyTestSQLJSONObject": [{
"Parent": {
"ParentTableID": 1,
"Field1": true,
"Field2": "Test ",
"Field3": "This is a test record"
},
"FirstChildTable": {
"ChildTable1ID": 1,
"Field1": true,
"Field2": "Test ",
"Field3": "This is a test record"
},
"SecondChildTable": {
"ChildTable1ID": 1,
"Field1": false,
"Field2": "Test ",
"Field3": "This is a test record"
},
"GrandchildTable": {
"GrandchildTableID": 1,
"Field1": false,
"Field2": "Test ",
"Field3": "This is a test record"
...
Example 3: NULL Field Values
One thing worth bearing in mind when working with the FOR JSON clause is how NULL field values behave. Take a look at the following example query output from dbo.ParentTable:
When attempting to query this single record using the FOR JSON AUTO clause, we get the following output:
//Example output below. Notice that no field name/value is returned for Field2 now
[{
"ParentTableID": 1,
"Field1": true,
"Field3": "This is a test record"
}]
If you have a requirement always to return a value for every NULL field, then you can use the INCLUDE_NULL_VALUES option to get around this:
SELECT *
FROM dbo.ParentTable AS PT
WHERE PT.ParentTableID = 1
FOR JSON AUTO, INCLUDE_NULL_VALUES
//Notice now that Field2 returns as expected
[{
"ParentTableID": 1,
"Field1": true,
"Field2": null,
"Field3": "This is a test record"
}]
Even with this option specified, there may still be issues with outputting this field with a value of null. In these scenarios, on a T-SQL side, you would generally use the ISNULL function to replace NULL values with an empty string. Further, because the field type in this example is a CHAR(10) data type, there are ten characters of whitespace that need removing from the output string. The following query will fix both of these problems:
SELECT ParentTableID, Field1, LTRIM(ISNULL(Field2, '')) AS Field2, Field3
FROM dbo.ParentTable AS PT
WHERE PT.ParentTableID = 1
FOR JSON AUTO
Example 4: Using sub-queries to return child records as JSON objects
In most cases involving parent and child records, returning all the data as JSON may not be required. Instead, it may be necessary to return the fields from the parent record only, and all child records as a single JSON object field on the parent record. Using Subqueries, we can accommodate this scenario as follows
SELECT PT.ParentTableID, PT.Field1, PT.Field2, PT.Field3,
(
SELECT *
FROM dbo.ChildTable1 AS CT1
WHERE CT1.ParentTableID = PT.ParentTableID
FOR JSON AUTO
) AS ChildTable1,
(
SELECT *
FROM dbo.ChildTable2 AS CT2
INNER JOIN dbo.GrandchildTable AS GT
ON CT2.ChildTable2ID = GT.ChildTable2ID
WHERE CT2.ParentTableID = PT.ParentTableID
FOR JSON AUTO
) AS ChildTable2
FROM dbo.ParentTable AS PT
Example 5: Storing FOR JSON Query Output in Parameters
In most scenarios, you will generally provide functions or Stored Procedures for developers to interface with when connecting to the database. It is in this situation where the ability to store the output of any query – including those that use the FOR JSON clause – within a parameter becomes very useful. The following snippet will store the output of a FOR JSON query into a parameter called @JSON, which is then retrievable at any time via a SELECT query:
DECLARE @JSON NVARCHAR(MAX) = (SELECT * FROM dbo.ChildTable2 AS CT2 INNER JOIN dbo.GrandchildTable AS GT ON CT2.ChildTable2ID = GT.ChildTable2ID FOR JSON AUTO)
SELECT @JSON
Wrapping Up: A Few Things to Bear in Mind
- The FOR JSON clause is not compatible with Common Table Expressions (CTE’s).
- When storing the output of a FOR JSON query in a parameter, you have no native capability to query the inner contents of the JSON object via a SELECT…WHERE query.
- Because there is no way of specifying the name of the single column that returns as part of a FOR JSON query, you cannot create a view that uses this clause.
I hope that this post has given you all the information and guidance needed to start working with FOR JSON clauses in your queries. The ability to straightforwardly convert SQL tabular data into a format that most developers would readily scream for is a convenient feature to have at our disposal and is indicative of how far the SQL Server product has moved forward in recent years. Hopefully, this feature will save you some hassle in the future and will mean that you can make friends with your developer colleagues 🙂
