Merge origin/dev-v7-deploy into dev-v8-zbwip

README.md

@@ -1,8 +1,15 @@
 Umbraco CMS
 ===========
-Umbraco is a free open source Content Management System built on the ASP.NET platform.
+The friendliest, most flexible and fastest growing ASP.NET CMS used by more than 350,000 websites worldwide: [https://umbraco.com](https://umbraco.com)
+
+[![ScreenShot](vimeo.png)](https://vimeo.com/172382998/)
+
+## Umbraco CMS ##
+Umbraco is a free open source Content Management System built on the ASP.NET platform. Our mission is to help you deliver delightful digital experiences by making Umbraco friendly, simpler and social.
+
 
 ## Building Umbraco from source ##
+
 The easiest way to get started is to run `build/build.bat` which will build both the backoffice (also known as "Belle") and the Umbraco core. You can then easily start debugging from Visual Studio, or if you need to debug Belle you can run `grunt vs` in `src\Umbraco.Web.UI.Client`.
  
 If you're interested in making changes to Belle without running Visual Studio make sure to read the [Belle ReadMe file](src/Umbraco.Web.UI.Client/README.md). 
@@ -13,30 +20,33 @@ Note that you can always [download a nightly build](http://nightly.umbraco.org/?
 
 [![ScreenShot](http://umbraco.com/images/whatisumbraco.png)](https://umbraco.tv/videos/umbraco-v7/content-editor/basics/introduction/cms-explanation/)
 
-## Umbraco - the simple, flexible and friendly ASP.NET CMS ##
+## Umbraco - The Friendly CMS ## 
 
-**More than 350,000 sites trust Umbraco** 
+For the first time on the Microsoft platform, there is a free user and developer friendly CMS that makes it quick and easy to create websites - or a breeze to build complex web applications. Umbraco has award-winning integration capabilities and supports ASP.NET MVC or Web Forms, including User and Custom Controls, out of the box. 
 
-For the first time on the Microsoft platform, there is a free user and developer friendly CMS that makes it quick and easy to create websites - or a breeze to build complex web applications. Umbraco has award-winning integration capabilities and supports ASP.NET MVC or Web Forms, including User and Custom Controls, out of the box. It's a developer's dream and your users will love it too. 
+Umbraco is not only loved by developers, but is a content editors dream. Enjoy intuitive editing tools, media management, responsive views and approval workflows to send your content live.
 
-Used by more than 350,000 active websites including [http://daviscup.com](http://daviscup.com), [http://heinz.com](http://heinz.com), [http://peugeot.com](http://peugeot.com), [http://www.hersheys.com/](http://www.hersheys.com/) and **The Official ASP.NET and IIS.NET website from Microsoft** ([http://asp.net](http://asp.net) / [http://iis.net](http://iis.net)), you can be sure that the technology is proven, stable and scales.  
+Used by more than 350,000 active websites including Carlsberg, Segway, Amazon and Heinz and **The Official ASP.NET and IIS.NET website from Microsoft** ([https://asp.net](https://asp.net) / [https://iis.net](https://iis.net)), you can be sure that the technology is proven, stable and scales. Backed by the team at Umbraco HQ, and supported by a dedicated community of over 200,000 craftspeople globally, you can trust that Umbraco is a safe choice and is here to stay. 
 
-To view more examples, please visit [http://umbraco.com/why-umbraco/#caseStudies](http://umbraco.com/why-umbraco/#caseStudies)
+To view more examples, please visit [https://umbraco.com/why-umbraco/#caseStudies](https://umbraco.com/why-umbraco/#caseStudies)
+
+## Why Open Source? ## 
+As an Open Source platform, Umbraco is more than just a CMS. We are transparent with our roadmap for future versions, our incremental sprint planning notes are publicly accessible and community contributions and packages are available for all to use.
 
 ## Downloading ##
 
-The downloadable Umbraco releases live at [http://our.umbraco.org/download](http://our.umbraco.org/download).
+The downloadable Umbraco releases live at [https://our.umbraco.org/download](https://our.umbraco.org/download).
 
 ## Forums ##
 
-We have a forum running on [http://our.umbraco.org](http://our.umbraco.org). The discussions group on [Google Groups](https://groups.google.com/forum/#!forum/umbraco-dev) is for discussions on developing the core, and not on Umbraco-implementations or extensions in general. For those topics, please use [http://our.umbraco.org](http://our.umbraco.org).
+Peer-to-peer support is available 24/7 at the community forum on [https://our.umbraco.org](https://our.umbraco.org). 
 
 ## Contribute to Umbraco ##
 
-If you want to contribute back to Umbraco you should check out our [guide to contributing](http://our.umbraco.org/contribute).
+Umbraco is contribution focused and community driven. If you want to contribute back to Umbraco please check out our [guide to contributing](https://our.umbraco.org/contribute).
 
 ## Found a bug? ##
 
-Another way you can contribute to Umbraco is by providing issue reports. For information on how to submit an issue report refer to our [online guide for reporting issues](http://our.umbraco.org/contribute/report-an-issue-or-request-a-feature).
+Another way you can contribute to Umbraco is by providing issue reports. For information on how to submit an issue report refer to our [online guide for reporting issues](https://our.umbraco.org/contribute/report-an-issue-or-request-a-feature).
 
-To view existing issues, please visit [http://issues.umbraco.org](http://issues.umbraco.org).
+To view existing issues, please visit [http://issues.umbraco.org](http://issues.umbraco.org).

src/Umbraco.Core/ICompletable.cs

@@ -0,0 +1,9 @@
+using System;
+
+namespace Umbraco.Core
+{
+    public interface ICompletable : IDisposable
+    {
+        void Complete();
+    }
+}

src/Umbraco.Core/IO/FileSystems.cs

@@ -195,7 +195,7 @@ namespace Umbraco.Core.IO
         //    _shadowEnabled = true;
         //}
 
-        public ShadowFileSystemsScope Shadow(Guid id)
+        public ICompletable Shadow(Guid id)
         {
             var typed = _wrappers.ToArray();
             var wrappers = new ShadowWrapper[typed.Length + 7];

src/Umbraco.Core/IO/ShadowFileSystem.cs

@@ -5,7 +5,7 @@ using System.Linq;
 
 namespace Umbraco.Core.IO
 {
-    public class ShadowFileSystem : IFileSystem
+    internal class ShadowFileSystem : IFileSystem
     {
         private readonly IFileSystem _fs;
         private readonly IFileSystem _sfs;

src/Umbraco.Core/IO/ShadowFileSystemsScope.cs

@@ -5,7 +5,7 @@ using Umbraco.Core.Logging;
 
 namespace Umbraco.Core.IO
 {
-    public class ShadowFileSystemsScope : IDisposable
+    internal class ShadowFileSystemsScope : ICompletable
     {
         // note: taking a reference to the _manager instead of using manager.Current
         // to avoid using Current everywhere but really, we support only 1 scope at

src/Umbraco.Core/IO/ShadowWrapper.cs

@@ -5,7 +5,7 @@ using System.Linq;
 
 namespace Umbraco.Core.IO
 {
-    public class ShadowWrapper : IFileSystem
+    internal class ShadowWrapper : IFileSystem
     {
         private readonly IFileSystem _innerFileSystem;
         private readonly string _shadowPath;

src/Umbraco.Core/MainDom.cs

@@ -95,7 +95,7 @@ namespace Umbraco.Core
         /// <param name="release">An action to execute before the AppDomain releases the main domain status.</param>
         /// <param name="weight">An optional weight (lower goes first).</param>
         /// <returns>A value indicating whether it was possible to register.</returns>
-        /// <remarks>If registering is successful, then the <paramref name="install"/> action 
+        /// <remarks>If registering is successful, then the <paramref name="install"/> action
         /// is guaranteed to execute before the AppDomain releases the main domain status.</remarks>
         public bool Register(Action install, Action release, int weight = 100)
         {
@@ -125,7 +125,7 @@ namespace Umbraco.Core
 
             try
             {
-                _logger.Debug<MainDom>("Stopping.");
+                _logger.Info<MainDom>("Stopping.");
                 foreach (var callback in _callbacks.OrderBy(x => x.Key).Select(x => x.Value))
                 {
                     try
@@ -146,7 +146,7 @@ namespace Umbraco.Core
                 // in any case...
                 _isMainDom = false;
                 _asyncLocker.Dispose();
-                _logger.Debug<MainDom>("Released.");
+                _logger.Info<MainDom>("Released.");
             }
         }
 
@@ -159,11 +159,11 @@ namespace Umbraco.Core
                 // the handler is not installed so that would be the hosting environment
                 if (_signaled)
                 {
-                    _logger.Debug<MainDom>("Cannot acquire (signaled).");
+                    _logger.Info<MainDom>("Cannot acquire (signaled).");
                     return false;
                 }
 
-                _logger.Debug<MainDom>("Acquiring.");
+                _logger.Info<MainDom>("Acquiring.");
 
                 // signal other instances that we want the lock, then wait one the lock,
                 // which may timeout, and this is accepted - see comments below
@@ -190,7 +190,7 @@ namespace Umbraco.Core
 
                 HostingEnvironment.RegisterObject(this);
 
-                _logger.Debug<MainDom>("Acquired.");
+                _logger.Info<MainDom>("Acquired.");
                 return true;
             }
         }

src/Umbraco.Core/Media/UploadAutoFillProperties.cs

@@ -138,7 +138,7 @@ namespace Umbraco.Core.Media
                     using (var filestream = _mediaFileSystem.OpenFile(filepath))
                     {
                         var extension = (Path.GetExtension(filepath) ?? "").TrimStart('.');
-                        var size = _mediaFileSystem.IsImageFile(extension) ? (Size?)_mediaFileSystem.GetDimensions(filestream) : null;
+                        var size = _mediaFileSystem.IsImageFile(extension) ? (Size?) _mediaFileSystem.GetDimensions(filestream) : null;
                         SetProperties(content, autoFillConfig, size, filestream.Length, extension);
                     }
                 }

src/Umbraco.Core/Models/PublicAccessEntry.cs

@@ -107,10 +107,7 @@ namespace Umbraco.Core.Models
 
         public void ClearRules()
         {
-            for (var i = _ruleCollection.Count - 1; i >= 0; i--)
-            {
-                RemoveRule(_ruleCollection[i]);
-            }
+            _ruleCollection.Clear();
         }
 
         [DataMember]

src/Umbraco.Core/Packaging/PackageBinaryInspector.cs

@@ -85,7 +85,7 @@ namespace Umbraco.Core.Packaging
         /// <summary>
         /// Performs the assembly scanning
         /// </summary>
-        /// <typeparam name="T"></typeparam>        
+        /// <typeparam name="T"></typeparam>
         /// <param name="assemblies"></param>
         /// <param name="errorReport"></param>
         /// <returns></returns>
@@ -114,7 +114,7 @@ namespace Umbraco.Core.Packaging
         /// <summary>
         /// Performs the assembly scanning
         /// </summary>
-        /// <typeparam name="T"></typeparam>        
+        /// <typeparam name="T"></typeparam>
         /// <param name="dllPath"></param>
         /// <param name="errorReport"></param>
         /// <returns></returns>
@@ -161,7 +161,7 @@ namespace Umbraco.Core.Packaging
 
             //get the list of assembly names to compare below
             var loadedNames = loaded.Select(x => x.GetName().Name).ToArray();
-            
+
             //Then load each referenced assembly into the context
             foreach (var a in loaded)
             {
@@ -177,7 +177,7 @@ namespace Umbraco.Core.Packaging
                     }
                     catch (FileNotFoundException)
                     {
-                        //if an exception occurs it means that a referenced assembly could not be found                        
+                        //if an exception occurs it means that a referenced assembly could not be found
                         errors.Add(
                             string.Concat("This package references the assembly '",
                                           assemblyName.Name,
@@ -186,7 +186,7 @@ namespace Umbraco.Core.Packaging
                     }
                     catch (Exception ex)
                     {
-                        //if an exception occurs it means that a referenced assembly could not be found                        
+                        //if an exception occurs it means that a referenced assembly could not be found
                         errors.Add(
                             string.Concat("This package could not be verified for compatibility. An error occurred while loading a referenced assembly '",
                                           assemblyName.Name,
@@ -204,7 +204,7 @@ namespace Umbraco.Core.Packaging
             {
                 //now we need to see if they contain any type 'T'
                 var reflectedAssembly = a;
-                
+
                 try
                 {
                     var found = reflectedAssembly.GetExportedTypes()
@@ -217,8 +217,8 @@ namespace Umbraco.Core.Packaging
                 }
                 catch (Exception ex)
                 {
-                    //This is a hack that nobody can seem to get around, I've read everything and it seems that 
-                    // this is quite a common thing when loading types into reflection only load context, so 
+                    //This is a hack that nobody can seem to get around, I've read everything and it seems that
+                    // this is quite a common thing when loading types into reflection only load context, so
                     // we're just going to ignore this specific one for now
                     var typeLoadEx = ex as TypeLoadException;
                     if (typeLoadEx != null)
@@ -239,7 +239,7 @@ namespace Umbraco.Core.Packaging
                         Current.Logger.Error<PackageBinaryInspector>("An error occurred scanning package assemblies", ex);
                     }
                 }
-                
+
             }
 
             errorReport = errors.ToArray();
@@ -259,7 +259,7 @@ namespace Umbraco.Core.Packaging
 
             var contractType = contractAssemblyLoadFrom.GetExportedTypes()
                 .FirstOrDefault(x => x.FullName == typeof(T).FullName && x.Assembly.FullName == typeof(T).Assembly.FullName);
-            
+
             if (contractType == null)
             {
                 throw new InvalidOperationException("Could not find type " + typeof(T) + " in the LoadFrom assemblies");

src/Umbraco.Core/Persistence/NPocoDatabaseExtensions.cs

@@ -3,9 +3,11 @@ using System.Collections.Generic;
 using System.Data;
 using System.Data.Common;
 using System.Data.SqlClient;
+using System.Data.SqlServerCe;
 using System.Linq;
 using System.Text.RegularExpressions;
 using NPoco;
+using StackExchange.Profiling.Data;
 using Umbraco.Core.Persistence.SqlSyntax;
 
 namespace Umbraco.Core.Persistence
@@ -158,137 +160,270 @@ namespace Umbraco.Core.Persistence
 
         // todo: review NPoco native InsertBulk to replace the code below
 
-        public static void BulkInsertRecordsWithTransaction<T>(this IDatabase db, ISqlSyntaxProvider sqlSyntax, IEnumerable<T> records)
+        /// <summary>
+        /// Bulk-inserts records within a transaction.
+        /// </summary>
+        /// <typeparam name="T">The type of the records.</typeparam>
+        /// <param name="database">The database.</param>
+        /// <param name="records">The records.</param>
+        /// <param name="useNativeBulkInsert">Whether to use native bulk insert when available.</param>
+        public static void BulkInsertRecordsWithTransaction<T>(this Database database, IEnumerable<T> records, bool useNativeBulkInsert = true)
         {
             var recordsA = records.ToArray();
             if (recordsA.Length == 0)
                 return;
 
             // no need to "try...catch", if the transaction is not completed it will rollback!
-            using (var tr = db.GetTransaction())
+            using (var tr = database.GetTransaction())
             {
-                db.BulkInsertRecords(sqlSyntax, recordsA);
+                database.BulkInsertRecords(recordsA, useNativeBulkInsert);
                 tr.Complete();
             }
         }
 
         /// <summary>
-        /// Performs the bulk insertion in the context of a current transaction with an optional parameter to complete the transaction
-        /// when finished
+        /// Bulk-inserts records.
         /// </summary>
-        /// <typeparam name="T"></typeparam>
-        /// <param name="db"></param>
-        /// <param name="sqlSyntax"></param>
-        /// <param name="records"></param>
-        public static void BulkInsertRecords<T>(this IDatabase db, ISqlSyntaxProvider sqlSyntax, IEnumerable<T> records)
+        /// <typeparam name="T">The type of the records.</typeparam>
+        /// <param name="database">The database.</param>
+        /// <param name="records">The records.</param>
+        /// <param name="useNativeBulkInsert">Whether to use native bulk insert when available.</param>
+        /// <returns>The number of records that were inserted.</returns>
+        public static int BulkInsertRecords<T>(this Database database, IEnumerable<T> records, bool useNativeBulkInsert = true)
         {
             var recordsA = records.ToArray();
-            if (recordsA.Length == 0)
-                return;
+            if (recordsA.Length == 0) return 0;
 
-            // if it is sql ce or it is a sql server version less than 2008, we need to do individual inserts.
-            var sqlServerSyntax = sqlSyntax as SqlServerSyntaxProvider;
-            if ((sqlServerSyntax != null && (int) sqlServerSyntax.ServerVersion.ProductVersionName < (int) SqlServerSyntaxProvider.VersionName.V2008)
-                || sqlSyntax is SqlCeSyntaxProvider)
-            {
-                // SqlCe doesn't support bulk insert statements!
-                foreach (var poco in recordsA)
-                    db.Insert(poco);
-            }
-            else
-            {
-                string[] sqlStatements;
-                var cmds = db.GenerateBulkInsertCommand(recordsA, db.Connection, out sqlStatements);
-                for (var i = 0; i < sqlStatements.Length; i++)
-                {
-                    using (var cmd = cmds[i])
-                    {
-                        cmd.CommandText = sqlStatements[i];
-                        cmd.ExecuteNonQuery();
-                    }
-                }
-            }
-        }
+            var pocoData = database.PocoDataFactory.ForType(typeof(T));
+            if (pocoData == null) throw new InvalidOperationException("Could not find PocoData for " + typeof(T));
 
-        private static bool IncludeColumn(PocoData pocoData, string columnKey, PocoColumn column)
-        {
-            // exclude result columns,
-            // exclude primary key column if auto-increment
-            return column.ResultColumn == false && (pocoData.TableInfo.AutoIncrement == false || columnKey != pocoData.TableInfo.PrimaryKey);
+            if (database.DatabaseType.IsSqlCe())
+            {
+                return useNativeBulkInsert
+                    ? BulkInsertRecordsSqlCe(database, pocoData, recordsA)
+                    : BulkInsertRecordsWithCommands(database, recordsA);
+            }
+
+            if (database.DatabaseType.IsSqlServer())
+            {
+                return useNativeBulkInsert && database.DatabaseType.IsSqlServer2008OrLater()
+                    ? BulkInsertRecordsSqlServer(database, pocoData, recordsA)
+                    : BulkInsertRecordsWithCommands(database, recordsA);
+            }
+
+            if (database.DatabaseType.IsMySql())
+                return BulkInsertRecordsWithCommands(database, recordsA);
+
+            throw new NotSupportedException();
         }
 
         /// <summary>
-        /// Creates a bulk insert command
+        /// Bulk-insert records using commands.
         /// </summary>
-        /// <typeparam name="T"></typeparam>
-        /// <param name="db"></param>
-        /// <param name="records"></param>
-        /// <param name="connection"></param>
-        /// <param name="sql"></param>
-        /// <returns>Sql commands with populated command parameters required to execute the sql statement</returns>
-        /// <remarks>
-        /// The limits for number of parameters are 2100 (in sql server, I think there's many more allowed in mysql). So
-        /// we need to detect that many params and split somehow.
-        /// For some reason the 2100 limit is not actually allowed even though the exception from sql server mentions 2100 as a max, perhaps it is 2099
-        /// that is max. I've reduced it to 2000 anyways.
-        /// </remarks>
-        internal static IDbCommand[] GenerateBulkInsertCommand<T>(
-            this IDatabase db,
-            T[] records,
-            DbConnection connection,
-            out string[] sql)
+        /// <typeparam name="T">The type of the records.</typeparam>
+        /// <param name="database">The database.</param>
+        /// <param name="records">The records.</param>
+        /// <returns>The number of records that were inserted.</returns>
+        private static int BulkInsertRecordsWithCommands<T>(Database database, T[] records)
         {
-            var pocoData = db.PocoDataFactory.ForType(typeof(T));
+            foreach (var command in database.GenerateBulkInsertCommands(records))
+                command.ExecuteNonQuery();
+
+            return records.Length; // what else?
+        }
+
+        /// <summary>
+        /// Creates bulk-insert commands.
+        /// </summary>
+        /// <typeparam name="T">The type of the records.</typeparam>
+        /// <param name="database">The database.</param>
+        /// <param name="records">The records.</param>
+        /// <returns>The sql commands to execute.</returns>
+        internal static IDbCommand[] GenerateBulkInsertCommands<T>(this Database database, T[] records)
+        {
+            var pocoData = database.PocoDataFactory.ForType(typeof(T));
 
             // get columns to include, = number of parameters per row
-            var columns = pocoData.Columns.Where(c => IncludeColumn(pocoData, c.Key, c.Value)).ToArray();
-            var paramsPerRow = columns.Length;
+            var columns = pocoData.Columns.Where(c => IncludeColumn(pocoData, c)).ToArray();
+            var paramsPerRecord = columns.Length;
 
             // format columns to sql
-            var tableName = db.DatabaseType.EscapeTableName(pocoData.TableInfo.TableName);
-            var columnNames = string.Join(", ", columns.Select(c => tableName + "." + db.DatabaseType.EscapeSqlIdentifier(c.Key)));
+            var tableName = database.DatabaseType.EscapeTableName(pocoData.TableInfo.TableName);
+            var columnNames = string.Join(", ", columns.Select(c => tableName + "." + database.DatabaseType.EscapeSqlIdentifier(c.Key)));
 
-            // example calc:
-            // given: we have 4168 items in the collection, each item contains 8 command parameters (values to be inserted)
-            // 2100 / 8 = 262.5
-            // Math.Floor(2100 / 8) = 262 items per trans
-            // 4168 / 262 = 15.908... = there will be 16 trans in total
+            // example:
+            // assume 4168 records, each record containing 8 fields, ie 8 command parameters
+            // max 2100 parameter per command
+            // Math.Floor(2100 / 8) = 262 record per command
+            // 4168 / 262 = 15.908... = there will be 16 command in total
+            // (if we have disabled db parameters, then all records will be included, in only one command)
+            var recordsPerCommand = paramsPerRecord == 0 ? int.MaxValue : Convert.ToInt32(Math.Floor(2000.00 / paramsPerRecord));
+            var commandsCount = Convert.ToInt32(Math.Ceiling((double)records.Length / recordsPerCommand));
 
-            // if we have disabled db parameters, then all items will be included, in only one transaction
-            var rowsPerCommand = paramsPerRow == 0 ? int.MaxValue : Convert.ToInt32(Math.Floor(2000.00 / paramsPerRow));
-            var commandsCount = Convert.ToInt32(Math.Ceiling((double) records.Length / rowsPerCommand));
-
-            sql = new string[commandsCount];
             var commands = new IDbCommand[commandsCount];
-
+            var recordsIndex = 0;
+            var recordsLeftToInsert = records.Length;
+            var prefix = database.DatabaseType.GetParameterPrefix(database.ConnectionString);
             for (var commandIndex = 0; commandIndex < commandsCount; commandIndex++)
             {
-                var itemsForTrans = records
-                    .Skip(commandIndex * rowsPerCommand)
-                    .Take(rowsPerCommand);
-
-                var cmd = db.CreateCommand(connection, CommandType.Text,  "");
-                var prefix = db.DatabaseType.GetParameterPrefix(cmd.Connection.ConnectionString);
-                var pocoValues = new List<string>();
-                var index = 0;
-                foreach (var poco in itemsForTrans)
+                var command = database.CreateCommand(database.Connection, CommandType.Text, string.Empty);
+                var parameterIndex = 0;
+                var commandRecords = Math.Min(recordsPerCommand, recordsLeftToInsert);
+                var recordsValues = new string[commandRecords];
+                for (var commandRecordIndex = 0; commandRecordIndex < commandRecords; commandRecordIndex++, recordsIndex++, recordsLeftToInsert--)
                 {
-                    var values = new List<string>();
-                    foreach (var column in columns)
+                    var record = records[recordsIndex];
+                    var recordValues = new string[columns.Length];
+                    for (var columnIndex = 0; columnIndex < columns.Length; columnIndex++)
                     {
-                        db.AddParameter(cmd, column.Value.GetValue(poco));
-                        values.Add(prefix + index++);
+                        database.AddParameter(command, columns[columnIndex].Value.GetValue(record));
+                        recordValues[columnIndex] = prefix + parameterIndex++;
                     }
-                    pocoValues.Add("(" + string.Join(",", values.ToArray()) + ")");
+                    recordsValues[commandRecordIndex] = "(" + string.Join(",", recordValues) + ")";
                 }
 
-                sql[commandIndex] = $"INSERT INTO {tableName} ({columnNames}) VALUES {string.Join(", ", pocoValues)}";
-                commands[commandIndex] = cmd;
+                command.CommandText = $"INSERT INTO {tableName} ({columnNames}) VALUES {string.Join(", ", recordsValues)}";
+                commands[commandIndex] = command;
             }
 
             return commands;
         }
 
+        /// <summary>
+        /// Determines whether a column should be part of a bulk-insert.
+        /// </summary>
+        /// <param name="pocoData">The PocoData object corresponding to the record's type.</param>
+        /// <param name="column">The column.</param>
+        /// <returns>A value indicating whether the column should be part of the bulk-insert.</returns>
+        /// <remarks>Columns that are primary keys and auto-incremental, or result columns, are excluded from bulk-inserts.</remarks>
+        private static bool IncludeColumn(PocoData pocoData, KeyValuePair<string, PocoColumn> column)
+        {
+            return column.Value.ResultColumn == false
+                   && (pocoData.TableInfo.AutoIncrement == false || column.Key != pocoData.TableInfo.PrimaryKey);
+        }
+
+        /// <summary>
+        /// Bulk-insert records using SqlCE TableDirect method.
+        /// </summary>
+        /// <typeparam name="T">The type of the records.</typeparam>
+        /// <param name="database">The database.</param>
+        /// <param name="pocoData">The PocoData object corresponding to the record's type.</param>
+        /// <param name="records">The records.</param>
+        /// <returns>The number of records that were inserted.</returns>
+        internal static int BulkInsertRecordsSqlCe<T>(Database database, PocoData pocoData, IEnumerable<T> records)
+        {
+            var columns = pocoData.Columns.ToArray();
+
+            // create command against the original database.Connection
+            using (var command = database.CreateCommand(database.Connection, CommandType.TableDirect, string.Empty))
+            {
+                command.CommandText = pocoData.TableInfo.TableName;
+                // fixme - not supporting transactions?
+                //cmd.Transaction = GetTypedTransaction<SqlCeTransaction>(db.Connection.);
+
+                // fixme - why the double 'using' here?
+                var count = 0;
+                using (var tCommand = GetTypedCommand<SqlCeCommand>(command)) // execute on the real command
+                {
+                    // seems to cause problems, I think this is primarily used for retrieval, not inserting.
+                    // see: https://msdn.microsoft.com/en-us/library/system.data.sqlserverce.sqlcecommand.indexname%28v=vs.100%29.aspx?f=255&MSPPError=-2147217396
+                    //sqlCeCommand.IndexName = pd.TableInfo.PrimaryKey;
+
+                    using (var resultSet = tCommand.ExecuteResultSet(ResultSetOptions.Updatable))
+                    {
+                        var updatableRecord = resultSet.CreateRecord();
+                        foreach (var record in records)
+                        {
+                            for (var i = 0; i < columns.Length; i++)
+                            {
+                                // skip the index if this shouldn't be included (i.e. PK)
+                                if (IncludeColumn(pocoData, columns[i]))
+                                {
+                                    var val = columns[i].Value.GetValue(record);
+                                    updatableRecord.SetValue(i, val);
+                                }
+                            }
+                            resultSet.Insert(updatableRecord);
+                            count++;
+                        }
+                    }
+                }
+
+                return count;
+            }
+        }
+
+        /// <summary>
+        /// Bulk-insert records using SqlServer BulkCopy method.
+        /// </summary>
+        /// <typeparam name="T">The type of the records.</typeparam>
+        /// <param name="database">The database.</param>
+        /// <param name="pocoData">The PocoData object corresponding to the record's type.</param>
+        /// <param name="records">The records.</param>
+        /// <returns>The number of records that were inserted.</returns>
+        internal static int BulkInsertRecordsSqlServer<T>(Database database, PocoData pocoData, IEnumerable<T> records)
+        {
+            // create command against the original database.Connection
+            using (var command = database.CreateCommand(database.Connection, CommandType.Text, string.Empty))
+            {
+                // use typed connection and transactionf or SqlBulkCopy
+                var tConnection = GetTypedConnection<SqlConnection>(database.Connection);
+                var tTransaction = GetTypedTransaction<SqlTransaction>(command.Transaction);
+                var tableName = pocoData.TableInfo.TableName;
+
+                var umbracoDatabase = database as UmbracoDatabase;
+                if (umbracoDatabase == null) throw new NotSupportedException("Database must be UmbracoDatabase.");
+                var syntax = umbracoDatabase.SqlSyntax as SqlServerSyntaxProvider;
+                if (syntax == null) throw new NotSupportedException("SqlSyntax must be SqlServerSyntaxProvider.");
+
+                using (var copy = new SqlBulkCopy(tConnection, SqlBulkCopyOptions.Default, tTransaction) { BulkCopyTimeout = 10000, DestinationTableName = tableName })
+                using (var bulkReader = new PocoDataDataReader<T, SqlServerSyntaxProvider>(records, pocoData, syntax))
+                {
+                    copy.WriteToServer(bulkReader);
+                    return bulkReader.RecordsAffected;
+                }
+            }
+        }
+
+        /// <summary>
+        /// Returns the underlying connection as a typed connection - this is used to unwrap the profiled mini profiler stuff
+        /// </summary>
+        /// <typeparam name="TConnection"></typeparam>
+        /// <param name="connection"></param>
+        /// <returns></returns>
+        private static TConnection GetTypedConnection<TConnection>(IDbConnection connection)
+            where TConnection : class, IDbConnection
+        {
+            var profiled = connection as ProfiledDbConnection;
+            return profiled == null ? connection as TConnection : profiled.InnerConnection as TConnection;
+        }
+
+        /// <summary>
+        /// Returns the underlying transaction as a typed transaction - this is used to unwrap the profiled mini profiler stuff
+        /// </summary>
+        /// <typeparam name="TTransaction"></typeparam>
+        /// <param name="transaction"></param>
+        /// <returns></returns>
+        private static TTransaction GetTypedTransaction<TTransaction>(IDbTransaction transaction)
+            where TTransaction : class, IDbTransaction
+        {
+            var profiled = transaction as ProfiledDbTransaction;
+            return profiled == null ? transaction as TTransaction : profiled.WrappedTransaction as TTransaction;
+        }
+
+        /// <summary>
+        /// Returns the underlying command as a typed command - this is used to unwrap the profiled mini profiler stuff
+        /// </summary>
+        /// <typeparam name="TCommand"></typeparam>
+        /// <param name="command"></param>
+        /// <returns></returns>
+        private static TCommand GetTypedCommand<TCommand>(IDbCommand command)
+            where TCommand : class, IDbCommand
+        {
+            var profiled = command as ProfiledDbCommand;
+            return profiled == null ? command as TCommand : profiled.InternalCommand as TCommand;
+        }
+
         public static void TruncateTable(this IDatabase db, ISqlSyntaxProvider sqlSyntax, string tableName)
         {
             var sql = new Sql(string.Format(

src/Umbraco.Core/Persistence/NPocoDatabaseTypeExtensions.cs

@@ -17,6 +17,11 @@ namespace Umbraco.Core.Persistence
             return databaseType is NPoco.DatabaseTypes.SqlServerDatabaseType;
         }
 
+        public static bool IsSqlServer2008OrLater(this DatabaseType databaseType)
+        {
+            return databaseType is NPoco.DatabaseTypes.SqlServer2008DatabaseType;
+        }
+
         public static bool IsSqlCe(this DatabaseType databaseType)
         {
             return databaseType is NPoco.DatabaseTypes.SqlServerCEDatabaseType;

src/Umbraco.Core/Persistence/PocoDataDataReader.cs

@@ -0,0 +1,139 @@
+using System;
+using System.Collections.Generic;
+using System.Data;
+using System.Linq;
+using NPoco;
+using Umbraco.Core.Persistence.DatabaseAnnotations;
+using Umbraco.Core.Persistence.DatabaseModelDefinitions;
+using Umbraco.Core.Persistence.SqlSyntax;
+
+namespace Umbraco.Core.Persistence
+{
+    /// <summary>
+    /// A data reader used for reading collections of PocoData entity types
+    /// </summary>
+    /// <remarks>
+    /// We are using a custom data reader so that tons of memory is not consumed when rebuilding this table, previously
+    /// we'd generate SQL insert statements, but we'd have to put all of the XML structures into memory first. Alternatively
+    /// we can use .net's DataTable, but this also requires putting everything into memory. By using a DataReader we don't have to 
+    /// store every content item and it's XML structure in memory to get it into the DB, we can stream it into the db with this 
+    /// reader.
+    /// </remarks>
+    internal class PocoDataDataReader<T, TSyntax> : BulkDataReader 
+        where TSyntax : ISqlSyntaxProvider
+    {        
+        private readonly MicrosoftSqlSyntaxProviderBase<TSyntax> _sqlSyntaxProvider;
+        private readonly TableDefinition _tableDefinition;
+        private readonly PocoColumn[] _readerColumns;
+        private readonly IEnumerator<T> _enumerator;
+        private readonly ColumnDefinition[] _columnDefinitions;
+        private int _recordsAffected = -1;
+
+        public PocoDataDataReader(
+            IEnumerable<T> dataSource,             
+            PocoData pd,
+            MicrosoftSqlSyntaxProviderBase<TSyntax> sqlSyntaxProvider)
+        {
+            if (dataSource == null) throw new ArgumentNullException(nameof(dataSource));            
+            if (sqlSyntaxProvider == null) throw new ArgumentNullException(nameof(sqlSyntaxProvider));
+
+            _tableDefinition = DefinitionFactory.GetTableDefinition(pd.Type, sqlSyntaxProvider);
+            if (_tableDefinition == null) throw new InvalidOperationException("No table definition found for type " + pd.Type);
+            
+            _readerColumns = pd.Columns.Select(x => x.Value).ToArray();
+            _sqlSyntaxProvider = sqlSyntaxProvider;
+            _enumerator = dataSource.GetEnumerator();
+            _columnDefinitions = _tableDefinition.Columns.ToArray();
+            
+        }
+
+        protected override string SchemaName => _tableDefinition.SchemaName;
+
+        protected override string TableName => _tableDefinition.Name;
+
+        public override int RecordsAffected => _recordsAffected <= 0 ? -1 : _recordsAffected;
+
+        /// <summary>
+        /// This will automatically add the schema rows based on the Poco table definition and the columns passed in
+        /// </summary>
+        protected override void AddSchemaTableRows()
+        {
+            //var colNames = _readerColumns.Select(x => x.ColumnName).ToArray();
+            //foreach (var col in _columnDefinitions.Where(x => colNames.Contains(x.Name, StringComparer.OrdinalIgnoreCase)))
+            foreach (var col in _columnDefinitions)
+            {
+                SqlDbType sqlDbType;
+                if (col.HasSpecialDbType)
+                {
+                    //get the SqlDbType from the 'special type'
+                    switch (col.DbType)
+                    {
+                        case SpecialDbTypes.NTEXT:
+                            sqlDbType = SqlDbType.NText;
+                            break;
+                        case SpecialDbTypes.NCHAR:
+                            sqlDbType = SqlDbType.NChar;
+                            break;
+                        default:
+                            throw new ArgumentOutOfRangeException();
+                    }
+                }
+                else if (col.Type.HasValue)
+                {
+                    //get the SqlDbType from the DbType
+                    sqlDbType = _sqlSyntaxProvider.GetSqlDbType(col.Type.Value);
+                }
+                else
+                {
+                    //get the SqlDbType from the clr type
+                    sqlDbType = _sqlSyntaxProvider.GetSqlDbType(col.PropertyType);
+                }
+                
+                AddSchemaTableRow(
+                    col.Name, 
+                    col.Size > 0 ? (int?)col.Size : null,
+                    col.Precision > 0  ? (short?)col.Precision : null, 
+                    null, col.IsUnique, col.IsIdentity, col.IsNullable, sqlDbType, 
+                    null, null, null, null, null);                
+            }
+        }
+
+        /// <summary>
+        /// Get the value from the column index for the current object
+        /// </summary>
+        /// <param name="i"></param>
+        /// <returns></returns>
+        public override object GetValue(int i)
+        {
+            return _enumerator.Current == null ? null : _readerColumns[i].GetValue(_enumerator.Current);
+        }
+
+        /// <summary>
+        /// Advance the cursor
+        /// </summary>
+        /// <returns></returns>
+        public override bool Read()
+        {
+            var result = _enumerator.MoveNext();
+            if (result)
+            {
+                if (_recordsAffected == -1)
+                    _recordsAffected = 0;
+                _recordsAffected++;
+            }
+            return result;
+        }
+
+        /// <summary>
+        /// Ensure the enumerator is disposed
+        /// </summary>
+        /// <param name="disposing"></param>
+        protected override void Dispose(bool disposing)
+        {
+            base.Dispose(disposing);
+
+            if (disposing)
+                _enumerator.Dispose();
+        }
+    }
+}

src/Umbraco.Core/Persistence/Repositories/PermissionRepository.cs

@@ -145,7 +145,7 @@ namespace Umbraco.Core.Persistence.Repositories
                 }
             }
 
-            _unitOfWork.Database.BulkInsertRecords(SqlSyntax, toInsert);
+            db.BulkInsertRecords(toInsert);
 
             //Raise the event
             AssignedPermissions.RaiseEvent(
@@ -177,7 +177,7 @@ namespace Umbraco.Core.Persistence.Repositories
                 UserId = userId
             }).ToArray();
 
-            _unitOfWork.Database.BulkInsertRecords(SqlSyntax, actions);
+            db.BulkInsertRecords(actions);
 
             //Raise the event
             AssignedPermissions.RaiseEvent(
@@ -209,7 +209,7 @@ namespace Umbraco.Core.Persistence.Repositories
                 UserId = id
             }).ToArray();
 
-            _unitOfWork.Database.BulkInsertRecords(SqlSyntax, actions);
+            db.BulkInsertRecords(actions);
 
             //Raise the event
             AssignedPermissions.RaiseEvent(
@@ -237,7 +237,7 @@ namespace Umbraco.Core.Persistence.Repositories
                 UserId = p.UserId
             }).ToArray();
 
-            _unitOfWork.Database.BulkInsertRecords(SqlSyntax, actions);
+            db.BulkInsertRecords(actions);
 
             //Raise the event
             AssignedPermissions.RaiseEvent(

src/Umbraco.Core/Persistence/SqlSyntax/MicrosoftSqlSyntaxProviderBase.cs

@@ -1,4 +1,6 @@
 using System;
+using System.Data;
+using System.Linq;
 using Umbraco.Core.Persistence.Querying;
 
 namespace Umbraco.Core.Persistence.SqlSyntax
@@ -133,5 +135,111 @@ namespace Umbraco.Core.Persistence.SqlSyntax
                     throw new ArgumentOutOfRangeException("columnType");
             }
         }
+
+        /// <summary>
+        /// This uses a the DbTypeMap created and custom mapping to resolve the SqlDbType
+        /// </summary>
+        /// <param name="clrType"></param>
+        /// <returns></returns>
+        public virtual SqlDbType GetSqlDbType(Type clrType)
+        {
+            var dbType = DbTypeMap.ColumnDbTypeMap.First(x => x.Key == clrType).Value;
+            return GetSqlDbType(dbType);
+        }
+
+        /// <summary>
+        /// Returns the mapped SqlDbType for the DbType specified
+        /// </summary>
+        /// <param name="dbType"></param>
+        /// <returns></returns>
+        public virtual SqlDbType GetSqlDbType(DbType dbType)
+        {
+            var sqlDbType = SqlDbType.NVarChar;
+
+            //SEE: https://msdn.microsoft.com/en-us/library/cc716729(v=vs.110).aspx
+            // and https://msdn.microsoft.com/en-us/library/yy6y35y8%28v=vs.110%29.aspx?f=255&MSPPError=-2147217396
+            switch (dbType)
+            {
+                case DbType.AnsiString:
+                    sqlDbType = SqlDbType.VarChar;
+                    break;
+                case DbType.Binary:
+                    sqlDbType = SqlDbType.VarBinary;
+                    break;
+                case DbType.Byte:
+                    sqlDbType = SqlDbType.TinyInt;
+                    break;
+                case DbType.Boolean:
+                    sqlDbType = SqlDbType.Bit;
+                    break;
+                case DbType.Currency:
+                    sqlDbType = SqlDbType.Money;
+                    break;
+                case DbType.Date:
+                    sqlDbType = SqlDbType.Date;
+                    break;
+                case DbType.DateTime:
+                    sqlDbType = SqlDbType.DateTime;
+                    break;
+                case DbType.Decimal:
+                    sqlDbType = SqlDbType.Decimal;
+                    break;
+                case DbType.Double:
+                    sqlDbType = SqlDbType.Float;
+                    break;
+                case DbType.Guid:
+                    sqlDbType = SqlDbType.UniqueIdentifier;
+                    break;
+                case DbType.Int16:
+                    sqlDbType = SqlDbType.SmallInt;
+                    break;
+                case DbType.Int32:
+                    sqlDbType = SqlDbType.Int;
+                    break;
+                case DbType.Int64:
+                    sqlDbType = SqlDbType.BigInt;
+                    break;
+                case DbType.Object:
+                    sqlDbType = SqlDbType.Variant;
+                    break;
+                case DbType.SByte:
+                    throw new NotSupportedException("Inferring a SqlDbType from SByte is not supported.");
+                case DbType.Single:
+                    sqlDbType = SqlDbType.Real;
+                    break;
+                case DbType.String:
+                    sqlDbType = SqlDbType.NVarChar;
+                    break;
+                case DbType.Time:
+                    sqlDbType = SqlDbType.Time;
+                    break;
+                case DbType.UInt16:
+                    throw new NotSupportedException("Inferring a SqlDbType from UInt16 is not supported.");
+                case DbType.UInt32:
+                    throw new NotSupportedException("Inferring a SqlDbType from UInt32 is not supported.");
+                case DbType.UInt64:
+                    throw new NotSupportedException("Inferring a SqlDbType from UInt64 is not supported.");
+                case DbType.VarNumeric:
+                    throw new NotSupportedException("Inferring a VarNumeric from UInt64 is not supported.");
+                case DbType.AnsiStringFixedLength:
+                    sqlDbType = SqlDbType.Char;
+                    break;
+                case DbType.StringFixedLength:
+                    sqlDbType = SqlDbType.NChar;
+                    break;
+                case DbType.Xml:
+                    sqlDbType = SqlDbType.Xml;
+                    break;
+                case DbType.DateTime2:
+                    sqlDbType = SqlDbType.DateTime2;
+                    break;
+                case DbType.DateTimeOffset:
+                    sqlDbType = SqlDbType.DateTimeOffset;
+                    break;
+                default:
+                    throw new ArgumentOutOfRangeException();
+            }
+            return sqlDbType;
+        }
     }
 }

src/Umbraco.Core/Properties/AssemblyInfo.cs

@@ -32,6 +32,7 @@ using System.Security.Permissions;
 
 [assembly: InternalsVisibleTo("umbraco.editorControls")]
 [assembly: InternalsVisibleTo("Umbraco.Tests")]
+[assembly: InternalsVisibleTo("Umbraco.Tests.Benchmarks")]
 [assembly: InternalsVisibleTo("Umbraco.Core")]
 [assembly: InternalsVisibleTo("Umbraco.Web")]
 [assembly: InternalsVisibleTo("Umbraco.Web.UI")]

src/Umbraco.Core/Sync/DatabaseServerMessenger.cs

@@ -30,7 +30,6 @@ namespace Umbraco.Core.Sync
     public class DatabaseServerMessenger : ServerMessengerBase
     {
         private readonly IRuntimeState _runtime;
-        private readonly DatabaseServerMessengerOptions _options;
         private readonly ManualResetEvent _syncIdle;
         private readonly object _locko = new object();
         private readonly ProfilingLogger _profilingLogger;
@@ -41,6 +40,8 @@ namespace Umbraco.Core.Sync
         private bool _syncing;
         private bool _released;
 
+        protected DatabaseServerMessengerOptions Options { get; private set; }
+
         public DatabaseServerMessenger(
             IRuntimeState runtime, DatabaseContext dbContext, ILogger logger, ProfilingLogger proflog,
             bool distributedEnabled, DatabaseServerMessengerOptions options)
@@ -55,7 +56,7 @@ namespace Umbraco.Core.Sync
             Logger = logger;
             _runtime = runtime;
             _profilingLogger = proflog;
-            _options = options;
+            Options = options;
             _lastPruned = _lastSync = DateTime.UtcNow;
             _syncIdle = new ManualResetEvent(true);
         }
@@ -128,7 +129,17 @@ namespace Umbraco.Core.Sync
                     {
                         _released = true; // no more syncs
                     }
-                    _syncIdle.WaitOne(); // wait for pending sync
+
+                    // wait a max of 5 seconds and then return, so that we don't block
+                    // the entire MainDom callbacks chain and prevent the AppDomain from
+                    // properly releasing MainDom - a timeout here means that one refresher
+                    // is taking too much time processing, however when it's done we will
+                    // not update lastId and stop everything
+                    var idle =_syncIdle.WaitOne(5000);
+                    if (idle == false)
+                    {
+                        Logger.Warn<DatabaseServerMessenger>("The wait lock timed out, application is shutting down. The current instruction batch will be re-processed.");
+                    }
                 },
                 weight);
 
@@ -167,14 +178,16 @@ namespace Umbraco.Core.Sync
                 else
                 {
                     //check for how many instructions there are to process
+                    //TODO: In 7.6 we need to store the count of instructions per row since this is not affective because there can be far more than one (if not thousands)
+                    // of instructions in a single row.
                     var count = Database.ExecuteScalar<int>("SELECT COUNT(*) FROM umbracoCacheInstruction WHERE id > @lastId", new {lastId = _lastId});
-                    if (count > _options.MaxProcessingInstructionCount)
+                    if (count > Options.MaxProcessingInstructionCount)
                     {
                         //too many instructions, proceed to cold boot
                         Logger.Warn<DatabaseServerMessenger>("The instruction count ({0}) exceeds the specified MaxProcessingInstructionCount ({1})."
                             + " The server will skip existing instructions, rebuild its caches and indexes entirely, adjust its last synced Id"
                             + " to the latest found in the database and maintain cache updates based on that Id.",
-                            () => count, () => _options.MaxProcessingInstructionCount);
+                            () => count, () => Options.MaxProcessingInstructionCount);
 
                         coldboot = true;
                     }
@@ -192,8 +205,8 @@ namespace Umbraco.Core.Sync
                         SaveLastSynced(maxId);
 
                     // execute initializing callbacks
-                    if (_options.InitializingCallbacks != null)
-                        foreach (var callback in _options.InitializingCallbacks)
+                    if (Options.InitializingCallbacks != null)
+                        foreach (var callback in Options.InitializingCallbacks)
                             callback();
                 }
 
@@ -211,12 +224,14 @@ namespace Umbraco.Core.Sync
                 if (_syncing)
                     return;
 
+                //Don't continue if we are released
                 if (_released)
                     return;
 
-                if ((DateTime.UtcNow - _lastSync).TotalSeconds <= _options.ThrottleSeconds)
+                if ((DateTime.UtcNow - _lastSync).TotalSeconds <= Options.ThrottleSeconds)
                     return;
 
+                //Set our flag and the lock to be in it's original state (i.e. it can be awaited)
                 _syncing = true;
                 _syncIdle.Reset();
                 _lastSync = DateTime.UtcNow;
@@ -228,7 +243,8 @@ namespace Umbraco.Core.Sync
                 {
                     ProcessDatabaseInstructions();
 
-                    if ((DateTime.UtcNow - _lastPruned).TotalSeconds <= _options.PruneThrottleSeconds)
+                    //Check for pruning throttling
+                    if ((_released || (DateTime.UtcNow - _lastPruned).TotalSeconds <= Options.PruneThrottleSeconds))
                         return;
 
                     _lastPruned = _lastSync;
@@ -244,7 +260,12 @@ namespace Umbraco.Core.Sync
             }
             finally
             {
-                _syncing = false;
+                lock (_locko)
+                {
+                    //We must reset our flag and signal any waiting locks
+                    _syncing = false;
+                }
+
                 _syncIdle.Set();
             }
         }
@@ -268,13 +289,17 @@ namespace Umbraco.Core.Sync
             //
             // FIXME not true if we're running on a background thread, assuming we can?
 
+
             var sql = Sql().SelectAll()
                 .From<CacheInstructionDto>()
                 .Where<CacheInstructionDto>(dto => dto.Id > _lastId)
                 .OrderBy<CacheInstructionDto>(dto => dto.Id);
 
-            var dtos = Database.Fetch<CacheInstructionDto>(sql);
-            if (dtos.Count <= 0) return;
+            //only retrieve the top 100 (just in case there's tons)
+            // even though MaxProcessingInstructionCount is by default 1000 we still don't want to process that many
+            // rows in one request thread since each row can contain a ton of instructions (until 7.5.5 in which case
+            // a row can only contain MaxProcessingInstructionCount)
+            var topSql = sql.SelectTop(100);
 
             // only process instructions coming from a remote server, and ignore instructions coming from
             // the local server as they've already been processed. We should NOT assume that the sequence of
@@ -282,8 +307,22 @@ namespace Umbraco.Core.Sync
             var localIdentity = LocalIdentity;
 
             var lastId = 0;
-            foreach (var dto in dtos)
+
+            //tracks which ones have already been processed to avoid duplicates
+            var processed = new HashSet<RefreshInstruction>();
+
+            //It would have been nice to do this in a Query instead of Fetch using a data reader to save
+            // some memory however we cannot do thta because inside of this loop the cache refreshers are also
+            // performing some lookups which cannot be done with an active reader open
+            foreach (var dto in Database.Fetch<CacheInstructionDto>(topSql))
             {
+                //If this flag gets set it means we're shutting down! In this case, we need to exit asap and cannot
+                // continue processing anything otherwise we'll hold up the app domain shutdown
+                if (_released)
+                {
+                    break;
+                }
+
                 if (dto.OriginIdentity == localIdentity)
                 {
                     // just skip that local one but update lastId nevertheless
@@ -304,27 +343,69 @@ namespace Umbraco.Core.Sync
                     continue;
                 }
 
-                // execute remote instructions & update lastId
-                try
-                {
-                    NotifyRefreshers(jsonA);
-                    lastId = dto.Id;
-                }
-                catch (Exception ex)
-                {
-                    Logger.Error<DatabaseServerMessenger>(
-                        $"DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions ({dto.Id}: \"{dto.Instructions}\"). Instruction is being skipped/ignored", ex);
+                var instructionBatch = GetAllInstructions(jsonA);
 
-                    //we cannot throw here because this invalid instruction will just keep getting processed over and over and errors
-                    // will be thrown over and over. The only thing we can do is ignore and move on.
-                    lastId = dto.Id;
+                //process as per-normal
+                var success = ProcessDatabaseInstructions(instructionBatch, dto, processed, ref lastId);
+
+                //if they couldn't be all processed (i.e. we're shutting down) then exit
+                if (success == false)
+                {
+                    Logger.Info<DatabaseServerMessenger>("The current batch of instructions was not processed, app is shutting down");
+                    break;
                 }
+
             }
 
             if (lastId > 0)
                 SaveLastSynced(lastId);
         }
 
+        /// <summary>
+        /// Processes the instruction batch and checks for errors
+        /// </summary>
+        /// <param name="instructionBatch"></param>
+        /// <param name="dto"></param>
+        /// <param name="processed">
+        /// Tracks which instructions have already been processed to avoid duplicates
+        /// </param>
+        /// <param name="lastId"></param>
+        /// <returns>
+        /// returns true if all instructions in the batch were processed, otherwise false if they could not be due to the app being shut down
+        /// </returns>
+        private bool ProcessDatabaseInstructions(IReadOnlyCollection<RefreshInstruction> instructionBatch, CacheInstructionDto dto, HashSet<RefreshInstruction> processed, ref int lastId)
+        {
+            // execute remote instructions & update lastId
+            try
+            {
+                var result = NotifyRefreshers(instructionBatch, processed);
+                if (result)
+                {
+                    //if all instructions we're processed, set the last id
+                    lastId = dto.Id;
+                }
+                return result;
+            }
+            //catch (ThreadAbortException ex)
+            //{
+            //    //This will occur if the instructions processing is taking too long since this is occuring on a request thread.
+            //    // Or possibly if IIS terminates the appdomain. In any case, we should deal with this differently perhaps...
+            //}
+            catch (Exception ex)
+            {
+                    Logger.Error<DatabaseServerMessenger>(
+                        $"DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions ({dto.Id}: \"{dto.Instructions}\"). Instruction is being skipped/ignored", ex);
+
+                //we cannot throw here because this invalid instruction will just keep getting processed over and over and errors
+                // will be thrown over and over. The only thing we can do is ignore and move on.
+                lastId = dto.Id;
+                return false;
+            }
+
+            ////if this is returned it will not be saved
+            //return -1;
+        }
+
         /// <summary>
         /// Remove old instructions from the database
         /// </summary>
@@ -335,7 +416,7 @@ namespace Umbraco.Core.Sync
         /// </remarks>
         private void PruneOldInstructions()
         {
-            var pruneDate = DateTime.UtcNow.AddDays(-_options.DaysToRetainInstructions);
+            var pruneDate = DateTime.UtcNow.AddDays(-Options.DaysToRetainInstructions);
 
             // using 2 queries is faster than convoluted joins
 
@@ -472,8 +553,14 @@ namespace Umbraco.Core.Sync
             return jsonRefresher;
         }
 
-        private static void NotifyRefreshers(IEnumerable<JToken> jsonArray)
+        /// <summary>
+        /// Parses out the individual instructions to be processed
+        /// </summary>
+        /// <param name="jsonArray"></param>
+        /// <returns></returns>
+        private static List<RefreshInstruction> GetAllInstructions(IEnumerable<JToken> jsonArray)
         {
+            var result = new List<RefreshInstruction>();
             foreach (var jsonItem in jsonArray)
             {
                 // could be a JObject in which case we can convert to a RefreshInstruction,
@@ -482,35 +569,64 @@ namespace Umbraco.Core.Sync
                 if (jsonObj != null)
                 {
                     var instruction = jsonObj.ToObject<RefreshInstruction>();
-                    switch (instruction.RefreshType)
-                    {
-                        case RefreshMethodType.RefreshAll:
-                            RefreshAll(instruction.RefresherId);
-                            break;
-                        case RefreshMethodType.RefreshByGuid:
-                            RefreshByGuid(instruction.RefresherId, instruction.GuidId);
-                            break;
-                        case RefreshMethodType.RefreshById:
-                            RefreshById(instruction.RefresherId, instruction.IntId);
-                            break;
-                        case RefreshMethodType.RefreshByIds:
-                            RefreshByIds(instruction.RefresherId, instruction.JsonIds);
-                            break;
-                        case RefreshMethodType.RefreshByJson:
-                            RefreshByJson(instruction.RefresherId, instruction.JsonPayload);
-                            break;
-                        case RefreshMethodType.RemoveById:
-                            RemoveById(instruction.RefresherId, instruction.IntId);
-                            break;
-                    }
-
+                    result.Add(instruction);
                 }
                 else
                 {
-                    var jsonInnerArray = (JArray) jsonItem;
-                    NotifyRefreshers(jsonInnerArray); // recurse
+                    var jsonInnerArray = (JArray)jsonItem;
+                    result.AddRange(GetAllInstructions(jsonInnerArray)); // recurse
                 }
             }
+            return result;
+        }
+
+        /// <summary>
+        /// executes the instructions against the cache refresher instances
+        /// </summary>
+        /// <param name="instructions"></param>
+        /// <param name="processed"></param>
+        /// <returns>
+        /// Returns true if all instructions were processed, otherwise false if the processing was interupted (i.e. app shutdown)
+        /// </returns>
+        private bool NotifyRefreshers(IEnumerable<RefreshInstruction> instructions, HashSet<RefreshInstruction> processed)
+        {
+            foreach (var instruction in instructions)
+            {
+                //Check if the app is shutting down, we need to exit if this happens.
+                if (_released)
+                {
+                    return false;
+                }
+
+                //this has already been processed
+                if (processed.Contains(instruction))
+                    continue;
+
+                switch (instruction.RefreshType)
+                {
+                    case RefreshMethodType.RefreshAll:
+                        RefreshAll(instruction.RefresherId);
+                        break;
+                    case RefreshMethodType.RefreshByGuid:
+                        RefreshByGuid(instruction.RefresherId, instruction.GuidId);
+                        break;
+                    case RefreshMethodType.RefreshById:
+                        RefreshById(instruction.RefresherId, instruction.IntId);
+                        break;
+                    case RefreshMethodType.RefreshByIds:
+                        RefreshByIds(instruction.RefresherId, instruction.JsonIds);
+                        break;
+                    case RefreshMethodType.RefreshByJson:
+                        RefreshByJson(instruction.RefresherId, instruction.JsonPayload);
+                        break;
+                    case RefreshMethodType.RemoveById:
+                        RemoveById(instruction.RefresherId, instruction.IntId);
+                        break;
+                }
+
+                processed.Add(instruction);
+            }
+            return true;
         }
 
         private static void RefreshAll(Guid uniqueIdentifier)

src/Umbraco.Core/Umbraco.Core.csproj

@@ -288,7 +288,6 @@
     <Compile Include="Logging\WebProfilerComponent.cs" />
     <Compile Include="Macros\XsltExtensionCollection.cs" />
     <Compile Include="Macros\XsltExtensionCollectionBuilder.cs" />
-    <Compile Include="Media\ImageExtensions.cs" />
     <Compile Include="Models\DictionaryItemExtensions.cs" />
     <Compile Include="Models\PublishedContent\IPublishedFragment.cs" />
     <Compile Include="Models\PublishedContent\NoopPublishedContentModelFactory.cs" />
@@ -306,6 +305,7 @@
     <Compile Include="Persistence\ThreadStaticUmbracoDatabaseAccessor.cs" />
     <Compile Include="Plugins\HideFromTypeFinderAttribute.cs" />
     <Compile Include="HttpContextExtensions.cs" />
+    <Compile Include="ICompletable.cs" />
     <Compile Include="IDisposeOnRequestEnd.cs" />
     <Compile Include="IO\ShadowFileSystem.cs" />
     <Compile Include="IO\ShadowFileSystemsScope.cs" />
@@ -324,6 +324,7 @@
     <Compile Include="Logging\ParallelForwardingAppender.cs" />
     <Compile Include="Mandate.cs" />
     <Compile Include="Manifest\GridEditorConverter.cs" />
+    <Compile Include="Media\ImageExtensions.cs" />
     <Compile Include="Media\UploadAutoFillProperties.cs" />
     <Compile Include="Models\AuditItem.cs" />
     <Compile Include="Models\AuditType.cs" />
@@ -384,6 +385,7 @@
     <Compile Include="Models\UmbracoDomain.cs" />
     <Compile Include="Models\DoNotCloneAttribute.cs" />
     <Compile Include="Models\IDomain.cs" />
+    <Compile Include="Persistence\BulkDataReader.cs" />
     <Compile Include="SafeCallContext.cs" />
     <Compile Include="Persistence\DatabaseNodeLockExtensions.cs" />
     <Compile Include="Persistence\Factories\ExternalLoginFactory.cs" />
@@ -439,6 +441,7 @@
     <Compile Include="Media\Exif\TIFFStrip.cs" />
     <Compile Include="Media\Exif\Utility.cs" />
     <Compile Include="Persistence\Migrations\Upgrades\TargetVersionSevenThreeOne\UpdateUserLanguagesToIsoCode.cs" />
+    <Compile Include="Persistence\PocoDataDataReader.cs" />
     <Compile Include="Persistence\RecordPersistenceType.cs" />
     <Compile Include="Persistence\Repositories\AuditRepository.cs" />
     <Compile Include="Persistence\Repositories\EntityContainerRepository.cs" />

src/Umbraco.Tests.Benchmarks/App.config

@@ -0,0 +1,48 @@
+<?xml version="1.0" encoding="utf-8"?>
+<configuration>
+    <startup> 
+        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6.1"/>
+    </startup>
+  <runtime>
+    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
+
+      <dependentAssembly>
+        <assemblyIdentity name="System.Data.SqlServerCe" publicKeyToken="89845dcd8080cc91"/>
+        <bindingRedirect oldVersion="0.0.0.0-4.0.0.1" newVersion="4.0.0.1"/>
+      </dependentAssembly>
+      
+      <dependentAssembly>
+        <assemblyIdentity name="Microsoft.Owin" publicKeyToken="31bf3856ad364e35" culture="neutral"/>
+        <bindingRedirect oldVersion="0.0.0.0-3.0.1.0" newVersion="3.0.1.0"/>
+      </dependentAssembly>
+      <dependentAssembly>
+        <assemblyIdentity name="Microsoft.Owin.Security.OAuth" publicKeyToken="31bf3856ad364e35" culture="neutral"/>
+        <bindingRedirect oldVersion="0.0.0.0-3.0.1.0" newVersion="3.0.1.0"/>
+      </dependentAssembly>
+      <dependentAssembly>
+        <assemblyIdentity name="Microsoft.Owin.Security" publicKeyToken="31bf3856ad364e35" culture="neutral"/>
+        <bindingRedirect oldVersion="0.0.0.0-3.0.1.0" newVersion="3.0.1.0"/>
+      </dependentAssembly>
+      <dependentAssembly>
+        <assemblyIdentity name="Microsoft.Owin.Security.Cookies" publicKeyToken="31bf3856ad364e35" culture="neutral"/>
+        <bindingRedirect oldVersion="0.0.0.0-3.0.1.0" newVersion="3.0.1.0"/>
+      </dependentAssembly>
+      <dependentAssembly>
+        <assemblyIdentity name="System.Web.Mvc" publicKeyToken="31bf3856ad364e35" culture="neutral"/>
+        <bindingRedirect oldVersion="0.0.0.0-5.2.3.0" newVersion="5.2.3.0"/>
+      </dependentAssembly>
+      <dependentAssembly>
+        <assemblyIdentity name="System.Web.WebPages" publicKeyToken="31bf3856ad364e35" culture="neutral"/>
+        <bindingRedirect oldVersion="0.0.0.0-3.0.0.0" newVersion="3.0.0.0"/>
+      </dependentAssembly>
+      <dependentAssembly>
+        <assemblyIdentity name="System.Web.Http" publicKeyToken="31bf3856ad364e35" culture="neutral"/>
+        <bindingRedirect oldVersion="0.0.0.0-5.2.3.0" newVersion="5.2.3.0"/>
+      </dependentAssembly>
+      <dependentAssembly>
+        <assemblyIdentity name="System.Net.Http.Formatting" publicKeyToken="31bf3856ad364e35" culture="neutral"/>
+        <bindingRedirect oldVersion="0.0.0.0-5.2.3.0" newVersion="5.2.3.0"/>
+      </dependentAssembly>
+    </assemblyBinding>
+  </runtime>
+</configuration>

src/Umbraco.Tests.Benchmarks/BulkInsertBenchmarks.cs

@@ -0,0 +1,218 @@
+using System;
+using System.Collections.Generic;
+using System.Data.SqlServerCe;
+using System.IO;
+using System.Linq;
+using BenchmarkDotNet.Attributes;
+using BenchmarkDotNet.Configs;
+using BenchmarkDotNet.Diagnostics.Windows;
+using BenchmarkDotNet.Jobs;
+using Umbraco.Core;
+using Umbraco.Core.Logging;
+using Umbraco.Core.Models.Rdbms;
+using Umbraco.Core.Persistence;
+using Umbraco.Core.Persistence.Mappers;
+using Umbraco.Core.Persistence.Migrations.Initial;
+using Umbraco.Core.Persistence.SqlSyntax;
+using Umbraco.Tests.TestHelpers;
+using ILogger = Umbraco.Core.Logging.ILogger;
+
+namespace Umbraco.Tests.Benchmarks
+{
+    [Config(typeof(Config))]
+    public class BulkInsertBenchmarks
+    {
+        private class Config : ManualConfig
+        {
+            public Config()
+            {
+                Add(new MemoryDiagnoser());
+                //Add(ExecutionValidator.FailOnError);
+
+                //The 'quick and dirty' settings, so it runs a little quicker
+                // see benchmarkdotnet FAQ
+                Add(Job.Default
+                    .WithLaunchCount(1) // benchmark process will be launched only once
+                    .WithIterationTime(100) // 100ms per iteration
+                    .WithWarmupCount(3) // 3 warmup iteration
+                    .WithTargetCount(3)); // 3 target iteration                
+            }
+        }
+
+        private static byte[] _initDbBytes;
+
+        // fixme - should run on LocalDb same as NPoco tests!
+
+        private UmbracoDatabase GetSqlServerDatabase(ILogger logger)
+        {
+            IDatabaseFactory f = null;
+            var l = new Lazy<IDatabaseFactory>(() => f);
+            f = new DefaultDatabaseFactory(
+                "server=.\\SQLExpress;database=YOURDB;user id=YOURUSER;password=YOURPASS",
+                Constants.DatabaseProviders.SqlServer,
+                new [] { new SqlServerSyntaxProvider(l) },
+                logger,
+                new ThreadStaticUmbracoDatabaseAccessor(), 
+                new MapperCollection(Enumerable.Empty<BaseMapper>()));
+            return f.GetDatabase();
+        }
+
+        private UmbracoDatabase GetSqlCeDatabase(string cstr, ILogger logger)
+        {
+            var f = new DefaultDatabaseFactory(
+                cstr,
+                Constants.DatabaseProviders.SqlCe,
+                new[] { new SqlCeSyntaxProvider() },
+                logger,
+                new ThreadStaticUmbracoDatabaseAccessor(),
+                new MapperCollection(Enumerable.Empty<BaseMapper>()));
+            return f.GetDatabase();
+        }
+
+        [Setup]
+        public void Setup()
+        {
+            var logger = new DebugDiagnosticsLogger();
+            var path = TestHelper.CurrentAssemblyDirectory;
+
+            SetupSqlCe(path, logger);
+            SetupSqlServer(logger);
+
+            
+        }        
+
+        private void SetupSqlServer(ILogger logger)
+        {
+            //create the db
+            _dbSqlServer = GetSqlServerDatabase(logger);
+
+            //drop the table
+            // note: DROP TABLE IF EXISTS is SQL 2016+
+            _dbSqlServer.Execute("IF OBJECT_ID('dbo.umbracoServer', 'U') IS NOT NULL DROP TABLE [umbracoServer]");
+
+            //re-create it
+            _dbSqlServer.Execute(@"CREATE TABLE [umbracoServer](
+	[id] [int] IDENTITY(1,1) NOT NULL,
+	[address] [nvarchar](500) NOT NULL,
+	[computerName] [nvarchar](255) NOT NULL,
+	[registeredDate] [datetime] NOT NULL CONSTRAINT [DF_umbracoServer_registeredDate]  DEFAULT (getdate()),
+	[lastNotifiedDate] [datetime] NOT NULL,
+	[isActive] [bit] NOT NULL,
+	[isMaster] [bit] NOT NULL,
+ CONSTRAINT [PK_umbracoServer] PRIMARY KEY CLUSTERED 
+(
+	[id] ASC
+)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
+)");
+        }
+
+        private void SetupSqlCe(string path, ILogger logger)
+        {
+            var dbName = string.Concat("Umb", Guid.NewGuid(), ".sdf");
+            AppDomain.CurrentDomain.SetData("DataDirectory", path);
+            var sqlCeConnectionString = $"Datasource=|DataDirectory|\\{dbName};Flush Interval=1;";
+            
+            _dbFile = Path.Combine(path, dbName);
+
+            //only create the db one time
+            if (_initDbBytes == null)
+            {
+                using (var engine = new SqlCeEngine(sqlCeConnectionString))
+                {
+                    engine.CreateDatabase();
+                }
+
+                //use the db  to create the initial schema so we can reuse in each bench
+                using (_dbSqlCe = GetSqlCeDatabase(sqlCeConnectionString, logger))
+                {
+                    var creation = new DatabaseSchemaCreation(_dbSqlCe, logger);
+                    creation.InitializeDatabaseSchema();
+                }
+                _initDbBytes = File.ReadAllBytes(_dbFile);            
+            }
+            else
+            {
+                File.WriteAllBytes(_dbFile, _initDbBytes);
+            }            
+
+            //create the db
+            _dbSqlCe = GetSqlCeDatabase(sqlCeConnectionString, logger);
+        }
+
+        private List<ServerRegistrationDto> GetData()
+        {
+            var data = new List<ServerRegistrationDto>();
+            for (var i = 0; i < 1000; i++)
+            {
+                data.Add(new ServerRegistrationDto
+                {
+                    ServerAddress = "address" + Guid.NewGuid(),
+                    ServerIdentity = "computer" + Guid.NewGuid(),
+                    DateRegistered = DateTime.Now,
+                    IsActive = true,
+                    DateAccessed = DateTime.Now
+                });
+            }
+            return data;
+        }
+
+        [Cleanup]
+        public void Cleanup()
+        {
+            _dbSqlCe.Dispose();
+            _dbSqlServer.Dispose();
+            File.Delete(_dbFile);
+        }
+
+        private string _dbFile;
+        private UmbracoDatabase _dbSqlCe;
+        private UmbracoDatabase _dbSqlServer;
+
+        /// <summary>
+        /// Tests updating the existing XML way
+        /// </summary>
+        [Benchmark(Baseline = true)]
+        public void SqlCeOneByOne()
+        {
+            using (var tr = _dbSqlCe.GetTransaction())
+            {
+                _dbSqlCe.BulkInsertRecords(GetData(), false);
+                tr.Complete();
+            }
+        }
+
+        /// <summary>
+        /// Tests updating with only the object graph
+        /// </summary>
+        [Benchmark]
+        public void SqlCeTableDirect()
+        {
+            using (var tr = _dbSqlCe.GetTransaction())
+            {
+                _dbSqlCe.BulkInsertRecords(GetData());
+                tr.Complete();
+            }
+        }
+
+        [Benchmark]
+        public void SqlServerBulkInsertStatements()
+        {
+            using (var tr = _dbSqlServer.GetTransaction())
+            {
+                _dbSqlServer.BulkInsertRecords(GetData(), false);
+                tr.Complete();
+            }
+        }
+
+        [Benchmark]
+        public void SqlServerBulkCopy()
+        {
+            using (var tr = _dbSqlServer.GetTransaction())
+            {
+                _dbSqlServer.BulkInsertRecords(GetData());
+                tr.Complete();
+            }
+        }
+
+    }
+}

src/Umbraco.Tests.Benchmarks/Program.cs

@@ -0,0 +1,35 @@
+using System;
+using System.Reflection;
+using BenchmarkDotNet.Running;
+
+namespace Umbraco.Tests.Benchmarks
+{
+    internal class Program
+    {
+        public static void Main(string[] args)
+        {
+            if (args.Length == 0)
+            {
+                var summary = BenchmarkRunner.Run<BulkInsertBenchmarks>();
+                Console.ReadLine();
+            }
+            else if (args.Length == 1)
+            {
+                var type = Assembly.GetExecutingAssembly().GetType("Umbraco.Tests.Benchmarks." +args[0]);
+                if (type == null)
+                {
+                    Console.WriteLine("Unknown benchmark.");
+                }
+                else
+                {
+                    var summary = BenchmarkRunner.Run(type);
+                    Console.ReadLine();
+                }
+            }
+            else
+            {
+                Console.WriteLine("?");
+            }
+        }
+    }
+}

src/Umbraco.Tests.Benchmarks/Properties/AssemblyInfo.cs

@@ -0,0 +1,36 @@
+using System.Reflection;
+using System.Runtime.CompilerServices;
+using System.Runtime.InteropServices;
+
+// General Information about an assembly is controlled through the following 
+// set of attributes. Change these attribute values to modify the information
+// associated with an assembly.
+[assembly: AssemblyTitle("Umbraco.Tests.Benchmarks")]
+[assembly: AssemblyDescription("")]
+[assembly: AssemblyConfiguration("")]
+[assembly: AssemblyCompany("")]
+[assembly: AssemblyProduct("Umbraco.Tests.Benchmarks")]
+[assembly: AssemblyCopyright("Copyright ©  2016")]
+[assembly: AssemblyTrademark("")]
+[assembly: AssemblyCulture("")]
+
+// Setting ComVisible to false makes the types in this assembly not visible 
+// to COM components.  If you need to access a type in this assembly from 
+// COM, set the ComVisible attribute to true on that type.
+[assembly: ComVisible(false)]
+
+// The following GUID is for the ID of the typelib if this project is exposed to COM
+[assembly: Guid("86deb346-089f-4106-89c8-d852b9cf2a33")]
+
+// Version information for an assembly consists of the following four values:
+//
+//      Major Version
+//      Minor Version 
+//      Build Number
+//      Revision
+//
+// You can specify all the values or you can default the Build and Revision Numbers 
+// by using the '*' as shown below:
+// [assembly: AssemblyVersion("1.0.*")]
+[assembly: AssemblyVersion("1.0.0.0")]
+[assembly: AssemblyFileVersion("1.0.0.0")]

src/Umbraco.Tests.Benchmarks/Umbraco.Tests.Benchmarks.csproj

@@ -0,0 +1,153 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="14.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+  <Import Project="$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props" Condition="Exists('$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props')" />
+  <PropertyGroup>
+    <Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
+    <Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
+    <ProjectGuid>{86DEB346-089F-4106-89C8-D852B9CF2A33}</ProjectGuid>
+    <OutputType>Exe</OutputType>
+    <AppDesignerFolder>Properties</AppDesignerFolder>
+    <RootNamespace>Umbraco.Tests.Benchmarks</RootNamespace>
+    <AssemblyName>Umbraco.Tests.Benchmarks</AssemblyName>
+    <TargetFrameworkVersion>v4.6.1</TargetFrameworkVersion>
+    <FileAlignment>512</FileAlignment>
+    <NuGetPackageImportStamp>
+    </NuGetPackageImportStamp>
+    <TargetFrameworkProfile />
+  </PropertyGroup>
+  <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
+    <PlatformTarget>AnyCPU</PlatformTarget>
+    <DebugSymbols>true</DebugSymbols>
+    <DebugType>full</DebugType>
+    <Optimize>false</Optimize>
+    <OutputPath>bin\Debug\</OutputPath>
+    <DefineConstants>DEBUG;TRACE</DefineConstants>
+    <ErrorReport>prompt</ErrorReport>
+    <WarningLevel>4</WarningLevel>
+  </PropertyGroup>
+  <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
+    <PlatformTarget>AnyCPU</PlatformTarget>
+    <DebugType>pdbonly</DebugType>
+    <Optimize>true</Optimize>
+    <OutputPath>bin\Release\</OutputPath>
+    <DefineConstants>TRACE</DefineConstants>
+    <ErrorReport>prompt</ErrorReport>
+    <WarningLevel>4</WarningLevel>
+  </PropertyGroup>
+  <PropertyGroup>
+    <RunPostBuildEvent>Always</RunPostBuildEvent>
+  </PropertyGroup>
+  <ItemGroup>
+    <Reference Include="BenchmarkDotNet, Version=0.9.9.0, Culture=neutral, PublicKeyToken=aa0ca2f9092cefc4, processorArchitecture=MSIL">
+      <HintPath>..\packages\BenchmarkDotNet.0.9.9\lib\net45\BenchmarkDotNet.dll</HintPath>
+      <Private>True</Private>
+    </Reference>
+    <Reference Include="BenchmarkDotNet.Core, Version=0.9.9.0, Culture=neutral, PublicKeyToken=aa0ca2f9092cefc4, processorArchitecture=MSIL">
+      <HintPath>..\packages\BenchmarkDotNet.Core.0.9.9\lib\net45\BenchmarkDotNet.Core.dll</HintPath>
+      <Private>True</Private>
+    </Reference>
+    <Reference Include="BenchmarkDotNet.Diagnostics.Windows, Version=0.9.9.0, Culture=neutral, PublicKeyToken=aa0ca2f9092cefc4, processorArchitecture=MSIL">
+      <HintPath>..\packages\BenchmarkDotNet.Diagnostics.Windows.0.9.9\lib\net45\BenchmarkDotNet.Diagnostics.Windows.dll</HintPath>
+      <Private>True</Private>
+    </Reference>
+    <Reference Include="BenchmarkDotNet.Toolchains.Roslyn, Version=0.9.9.0, Culture=neutral, PublicKeyToken=aa0ca2f9092cefc4, processorArchitecture=MSIL">
+      <HintPath>..\packages\BenchmarkDotNet.Toolchains.Roslyn.0.9.9\lib\net45\BenchmarkDotNet.Toolchains.Roslyn.dll</HintPath>
+      <Private>True</Private>
+    </Reference>
+    <Reference Include="Microsoft.CodeAnalysis, Version=1.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
+      <HintPath>..\packages\Microsoft.CodeAnalysis.Common.1.3.2\lib\net45\Microsoft.CodeAnalysis.dll</HintPath>
+      <Private>True</Private>
+    </Reference>
+    <Reference Include="Microsoft.CodeAnalysis.CSharp, Version=1.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
+      <HintPath>..\packages\Microsoft.CodeAnalysis.CSharp.1.3.2\lib\net45\Microsoft.CodeAnalysis.CSharp.dll</HintPath>
+      <Private>True</Private>
+    </Reference>
+    <Reference Include="Microsoft.Diagnostics.Tracing.TraceEvent, Version=1.0.41.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL">
+      <HintPath>..\packages\Microsoft.Diagnostics.Tracing.TraceEvent.1.0.41\lib\net40\Microsoft.Diagnostics.Tracing.TraceEvent.dll</HintPath>
+      <Private>True</Private>
+    </Reference>
+    <Reference Include="NPoco, Version=3.4.5.0, Culture=neutral, processorArchitecture=MSIL">
+      <HintPath>..\packages\NPoco.3.4.5\lib\net45\NPoco.dll</HintPath>
+      <Private>True</Private>
+    </Reference>
+    <Reference Include="System" />
+    <Reference Include="System.Collections.Immutable, Version=1.1.37.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL">
+      <HintPath>..\packages\System.Collections.Immutable.1.1.37\lib\dotnet\System.Collections.Immutable.dll</HintPath>
+      <Private>True</Private>
+    </Reference>
+    <Reference Include="System.Configuration" />
+    <Reference Include="System.Core" />
+    <Reference Include="System.Data.SqlServerCe, Version=4.0.0.1, Culture=neutral, PublicKeyToken=89845dcd8080cc91, processorArchitecture=MSIL">
+      <HintPath>..\packages\SqlServerCE.4.0.0.1\lib\System.Data.SqlServerCe.dll</HintPath>
+      <Private>True</Private>
+    </Reference>
+    <Reference Include="System.Data.SqlServerCe.Entity, Version=4.0.0.1, Culture=neutral, PublicKeyToken=89845dcd8080cc91, processorArchitecture=MSIL">
+      <HintPath>..\packages\SqlServerCE.4.0.0.1\lib\System.Data.SqlServerCe.Entity.dll</HintPath>
+      <Private>True</Private>
+    </Reference>
+    <Reference Include="System.Management" />
+    <Reference Include="System.Reflection.Metadata, Version=1.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL">
+      <HintPath>..\packages\System.Reflection.Metadata.1.2.0\lib\portable-net45+win8\System.Reflection.Metadata.dll</HintPath>
+      <Private>True</Private>
+    </Reference>
+    <Reference Include="System.Runtime.Caching" />
+    <Reference Include="System.Runtime.Serialization" />
+    <Reference Include="System.Threading.Tasks.Extensions, Version=4.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51, processorArchitecture=MSIL">
+      <HintPath>..\packages\System.Threading.Tasks.Extensions.4.0.0\lib\portable-net45+win8+wp8+wpa81\System.Threading.Tasks.Extensions.dll</HintPath>
+      <Private>True</Private>
+    </Reference>
+    <Reference Include="System.Xml.Linq" />
+    <Reference Include="System.Data.DataSetExtensions" />
+    <Reference Include="Microsoft.CSharp" />
+    <Reference Include="System.Data" />
+    <Reference Include="System.Net.Http" />
+    <Reference Include="System.Xml" />
+  </ItemGroup>
+  <ItemGroup>
+    <Compile Include="BulkInsertBenchmarks.cs" />
+    <Compile Include="Program.cs" />
+    <Compile Include="Properties\AssemblyInfo.cs" />
+    <Compile Include="XmlBenchmarks.cs" />
+  </ItemGroup>
+  <ItemGroup>
+    <None Include="App.config" />
+    <None Include="packages.config" />
+  </ItemGroup>
+  <ItemGroup>
+    <Analyzer Include="..\packages\Microsoft.CodeAnalysis.Analyzers.1.1.0\analyzers\dotnet\cs\Microsoft.CodeAnalysis.Analyzers.dll" />
+    <Analyzer Include="..\packages\Microsoft.CodeAnalysis.Analyzers.1.1.0\analyzers\dotnet\cs\Microsoft.CodeAnalysis.CSharp.Analyzers.dll" />
+  </ItemGroup>
+  <ItemGroup>
+    <ProjectReference Include="..\Umbraco.Core\Umbraco.Core.csproj">
+      <Project>{31785bc3-256c-4613-b2f5-a1b0bdded8c1}</Project>
+      <Name>Umbraco.Core</Name>
+    </ProjectReference>
+    <ProjectReference Include="..\Umbraco.Tests\Umbraco.Tests.csproj">
+      <Project>{5d3b8245-ada6-453f-a008-50ed04bfe770}</Project>
+      <Name>Umbraco.Tests</Name>
+    </ProjectReference>
+  </ItemGroup>
+  <Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />
+  <Import Project="..\packages\Microsoft.Diagnostics.Tracing.TraceEvent.1.0.41\build\Microsoft.Diagnostics.Tracing.TraceEvent.targets" Condition="Exists('..\packages\Microsoft.Diagnostics.Tracing.TraceEvent.1.0.41\build\Microsoft.Diagnostics.Tracing.TraceEvent.targets')" />
+  <Target Name="EnsureNuGetPackageBuildImports" BeforeTargets="PrepareForBuild">
+    <PropertyGroup>
+      <ErrorText>This project references NuGet package(s) that are missing on this computer. Use NuGet Package Restore to download them.  For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}.</ErrorText>
+    </PropertyGroup>
+    <Error Condition="!Exists('..\packages\Microsoft.Diagnostics.Tracing.TraceEvent.1.0.41\build\Microsoft.Diagnostics.Tracing.TraceEvent.targets')" Text="$([System.String]::Format('$(ErrorText)', '..\packages\Microsoft.Diagnostics.Tracing.TraceEvent.1.0.41\build\Microsoft.Diagnostics.Tracing.TraceEvent.targets'))" />
+  </Target>
+  <PropertyGroup>
+    <PostBuildEvent>
+    </PostBuildEvent>
+  </PropertyGroup>
+  <PropertyGroup>
+    <PreBuildEvent>xcopy "$(ProjectDir)"..\packages\SqlServerCE.4.0.0.1\amd64\*.* "$(TargetDir)amd64\" /Y /F /E /I /C /D
+xcopy "$(ProjectDir)"..\packages\SqlServerCE.4.0.0.1\x86\*.* "$(TargetDir)x86\" /Y /F /E /I /C /D</PreBuildEvent>
+  </PropertyGroup>
+  <!-- To modify your build process, add your task inside one of the targets below and uncomment it. 
+       Other similar extension points exist, see Microsoft.Common.targets.
+  <Target Name="BeforeBuild">
+  </Target>
+  <Target Name="AfterBuild">
+  </Target>
+  -->
+</Project>

src/Umbraco.Tests.Benchmarks/Umbraco.Tests.Benchmarks.csproj.DotSettings

@@ -0,0 +1,2 @@
+<wpf:ResourceDictionary xml:space="preserve" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:s="clr-namespace:System;assembly=mscorlib" xmlns:ss="urn:shemas-jetbrains-com:settings-storage-xaml" xmlns:wpf="http://schemas.microsoft.com/winfx/2006/xaml/presentation">
+	<s:String x:Key="/Default/CodeInspection/CSharpLanguageProject/LanguageLevel/@EntryValue">CSharp60</s:String></wpf:ResourceDictionary>

src/Umbraco.Tests.Benchmarks/XmlBenchmarks.cs

@@ -0,0 +1,123 @@
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Text;
+using System.Threading.Tasks;
+using System.Xml;
+using BenchmarkDotNet.Attributes;
+using BenchmarkDotNet.Configs;
+using BenchmarkDotNet.Diagnostics.Windows;
+using BenchmarkDotNet.Jobs;
+
+namespace Umbraco.Tests.Benchmarks
+{
+    [Config(typeof(Config))]
+    public class XmlBenchmarks
+    {
+        private class Config : ManualConfig
+        {
+            public Config()
+            {
+                Add(new MemoryDiagnoser());
+                //Add(ExecutionValidator.FailOnError);
+
+                //The 'quick and dirty' settings, so it runs a little quicker
+                // see benchmarkdotnet FAQ
+                Add(Job.Default
+                    .WithLaunchCount(1) // benchmark process will be launched only once
+                    .WithIterationTime(100) // 100ms per iteration
+                    .WithWarmupCount(3) // 3 warmup iteration
+                    .WithTargetCount(3)); // 3 target iteration                
+            }
+        }
+
+        [Setup]
+        public void Setup()
+        {
+            var templateId = 0;
+            var xmlText = @"<?xml version=""1.0"" encoding=""utf-8""?>
+<!DOCTYPE root[ 
+<!ELEMENT Home ANY>
+<!ATTLIST Home id ID #REQUIRED>
+<!ELEMENT CustomDocument ANY>
+<!ATTLIST CustomDocument id ID #REQUIRED>
+]>
+<root id=""-1"">
+	<Home id=""1046"" parentID=""-1"" level=""1"" writerID=""0"" creatorID=""0"" nodeType=""1044"" template=""" + templateId + @""" sortOrder=""1"" createDate=""2012-06-12T14:13:17"" updateDate=""2012-07-20T18:50:43"" nodeName=""Home"" urlName=""home"" writerName=""admin"" creatorName=""admin"" path=""-1,1046"" isDoc="""">
+		<content><![CDATA[]]></content>
+		<umbracoUrlAlias><![CDATA[this/is/my/alias, anotheralias]]></umbracoUrlAlias>
+		<umbracoNaviHide>1</umbracoNaviHide>
+		<Home id=""1173"" parentID=""1046"" level=""2"" writerID=""0"" creatorID=""0"" nodeType=""1044"" template=""" + templateId + @""" sortOrder=""2"" createDate=""2012-07-20T18:06:45"" updateDate=""2012-07-20T19:07:31"" nodeName=""Sub1"" urlName=""sub1"" writerName=""admin"" creatorName=""admin"" path=""-1,1046,1173"" isDoc="""">
+			<content><![CDATA[<div>This is some content</div>]]></content>
+			<umbracoUrlAlias><![CDATA[page2/alias, 2ndpagealias]]></umbracoUrlAlias>			
+			<Home id=""1174"" parentID=""1173"" level=""3"" writerID=""0"" creatorID=""0"" nodeType=""1044"" template=""" + templateId + @""" sortOrder=""2"" createDate=""2012-07-20T18:07:54"" updateDate=""2012-07-20T19:10:27"" nodeName=""Sub2"" urlName=""sub2"" writerName=""admin"" creatorName=""admin"" path=""-1,1046,1173,1174"" isDoc="""">
+				<content><![CDATA[]]></content>
+				<umbracoUrlAlias><![CDATA[only/one/alias]]></umbracoUrlAlias>
+				<creatorName><![CDATA[Custom data with same property name as the member name]]></creatorName>
+			</Home>
+			<Home id=""1176"" parentID=""1173"" level=""3"" writerID=""0"" creatorID=""0"" nodeType=""1044"" template=""" + templateId + @""" sortOrder=""3"" createDate=""2012-07-20T18:08:08"" updateDate=""2012-07-20T19:10:52"" nodeName=""Sub 3"" urlName=""sub-3"" writerName=""admin"" creatorName=""admin"" path=""-1,1046,1173,1176"" isDoc="""">
+				<content><![CDATA[]]></content>
+			</Home>
+			<CustomDocument id=""1177"" parentID=""1173"" level=""3"" writerID=""0"" creatorID=""0"" nodeType=""1234"" template=""" + templateId + @""" sortOrder=""4"" createDate=""2012-07-16T15:26:59"" updateDate=""2012-07-18T14:23:35"" nodeName=""custom sub 1"" urlName=""custom-sub-1"" writerName=""admin"" creatorName=""admin"" path=""-1,1046,1173,1177"" isDoc="""" />
+			<CustomDocument id=""1178"" parentID=""1173"" level=""3"" writerID=""0"" creatorID=""0"" nodeType=""1234"" template=""" + templateId + @""" sortOrder=""4"" createDate=""2012-07-16T15:26:59"" updateDate=""2012-07-16T14:23:35"" nodeName=""custom sub 2"" urlName=""custom-sub-2"" writerName=""admin"" creatorName=""admin"" path=""-1,1046,1173,1178"" isDoc="""" />
+		</Home>
+		<Home id=""1175"" parentID=""1046"" level=""2"" writerID=""0"" creatorID=""0"" nodeType=""1044"" template=""" + templateId + @""" sortOrder=""3"" createDate=""2012-07-20T18:08:01"" updateDate=""2012-07-20T18:49:32"" nodeName=""Sub 2"" urlName=""sub-2"" writerName=""admin"" creatorName=""admin"" path=""-1,1046,1175"" isDoc=""""><content><![CDATA[]]></content>
+		</Home>
+	</Home>
+	<CustomDocument id=""1172"" parentID=""-1"" level=""1"" writerID=""0"" creatorID=""0"" nodeType=""1234"" template=""" + templateId + @""" sortOrder=""2"" createDate=""2012-07-16T15:26:59"" updateDate=""2012-07-18T14:23:35"" nodeName=""Test"" urlName=""test-page"" writerName=""admin"" creatorName=""admin"" path=""-1,1172"" isDoc="""" />
+</root>";
+            _xml = new XmlDocument();
+            _xml.LoadXml(xmlText);
+        }
+
+        [Cleanup]
+        public void Cleanup()
+        {
+            _xml = null;
+        }
+
+        private XmlDocument _xml;
+
+        [Benchmark]
+        public void XmlWithXPath()
+        {
+            var xpath = "/root/* [@isDoc and @urlName='home']//* [@isDoc and @urlName='sub1']//* [@isDoc and @urlName='sub2']";
+            var elt = _xml.SelectSingleNode(xpath);
+            if (elt == null) Console.WriteLine("ERR");
+        }
+
+        [Benchmark]
+        public void XmlWithNavigation()
+        {
+            var elt = _xml.DocumentElement;
+            var id = NavigateElementRoute(elt, new[] {"home", "sub1", "sub2"});
+            if (id <= 0) Console.WriteLine("ERR");
+        }
+
+        private const bool UseLegacySchema = false;
+
+        private int NavigateElementRoute(XmlElement elt, string[] urlParts)
+        {
+            var found = true;
+            var i = 0;
+            while (found && i < urlParts.Length)
+            {
+                found = false;
+                foreach (XmlElement child in elt.ChildNodes)
+                {
+                    var noNode = UseLegacySchema
+                        ? child.Name != "node"
+                        : child.GetAttributeNode("isDoc") == null;
+                    if (noNode) continue;
+                    if (child.GetAttribute("urlName") != urlParts[i]) continue;
+
+                    found = true;
+                    elt = child;
+                    break;
+                }
+                i++;
+            }
+            return found ? int.Parse(elt.GetAttribute("id")) : -1;
+        }
+    }
+}

src/Umbraco.Tests.Benchmarks/packages.config

@@ -0,0 +1,26 @@
+<?xml version="1.0" encoding="utf-8"?>
+<packages>
+  <package id="BenchmarkDotNet" version="0.9.9" targetFramework="net45" />
+  <package id="BenchmarkDotNet.Core" version="0.9.9" targetFramework="net45" />
+  <package id="BenchmarkDotNet.Diagnostics.Windows" version="0.9.9" targetFramework="net45" />
+  <package id="BenchmarkDotNet.Toolchains.Roslyn" version="0.9.9" targetFramework="net45" />
+  <package id="Microsoft.CodeAnalysis.Analyzers" version="1.1.0" targetFramework="net45" />
+  <package id="Microsoft.CodeAnalysis.Common" version="1.3.2" targetFramework="net45" />
+  <package id="Microsoft.CodeAnalysis.CSharp" version="1.3.2" targetFramework="net45" />
+  <package id="Microsoft.Diagnostics.Tracing.TraceEvent" version="1.0.41" targetFramework="net45" />
+  <package id="Microsoft.SqlServer.Compact" version="4.0.8854.1" targetFramework="net45" />
+  <package id="NPoco" version="3.4.5" targetFramework="net45" />
+  <package id="SqlServerCE" version="4.0.0.1" targetFramework="net45" />
+  <package id="System.Collections" version="4.0.0" targetFramework="net45" />
+  <package id="System.Collections.Immutable" version="1.1.37" targetFramework="net45" />
+  <package id="System.Diagnostics.Debug" version="4.0.0" targetFramework="net45" />
+  <package id="System.Globalization" version="4.0.0" targetFramework="net45" />
+  <package id="System.Linq" version="4.0.0" targetFramework="net45" />
+  <package id="System.Reflection.Metadata" version="1.2.0" targetFramework="net45" />
+  <package id="System.Resources.ResourceManager" version="4.0.0" targetFramework="net45" />
+  <package id="System.Runtime" version="4.0.0" targetFramework="net45" />
+  <package id="System.Runtime.Extensions" version="4.0.0" targetFramework="net45" />
+  <package id="System.Threading" version="4.0.0" targetFramework="net45" />
+  <package id="System.Threading.Tasks" version="4.0.0" targetFramework="net45" />
+  <package id="System.Threading.Tasks.Extensions" version="4.0.0" targetFramework="net45" />
+</packages>

src/Umbraco.Tests/IO/ShadowFileSystemTests.cs

@@ -15,6 +15,11 @@ namespace Umbraco.Tests.IO
     [TestFixture]
     public class ShadowFileSystemTests
     {
+        // tested:
+        // only 1 instance of this class is created
+        // SetUp and TearDown run before/after each test
+        // SetUp does not start before the previous TearDown returns
+
         [SetUp]
         public void SetUp()
         {

src/Umbraco.Tests/Persistence/NPocoExtensionsTest.cs

@@ -1,9 +1,13 @@
 using System;
 using System.Collections.Generic;
+using System.Linq;
 using System.Text.RegularExpressions;
 using NUnit.Framework;
+using Umbraco.Core;
+using Umbraco.Core.Logging;
 using Umbraco.Core.Models.Rdbms;
 using Umbraco.Core.Persistence;
+using Umbraco.Core.Persistence.SqlSyntax;
 using Umbraco.Tests.TestHelpers;
 
 namespace Umbraco.Tests.Persistence
@@ -15,7 +19,7 @@ namespace Umbraco.Tests.Persistence
     public class NPocoExtensionsTest : TestWithDatabaseBase
     {
         [Test]
-        public void Can_Bulk_Insert()
+        public void Can_Bulk_Insert_One_By_One()
         {
             // Arrange
             var db = DatabaseContext.Database;
@@ -24,25 +28,178 @@ namespace Umbraco.Tests.Persistence
             for (var i = 0; i < 1000; i++)
             {
                 servers.Add(new ServerRegistrationDto
-                    {
-                        ServerAddress = "address" + i,
-                        ServerIdentity = "computer" + i,
-                        DateRegistered = DateTime.Now,
-                        IsActive = true,
-                        DateAccessed = DateTime.Now
-                    });
+                {
+                    ServerAddress = "address" + i,
+                    ServerIdentity = "computer" + i,
+                    DateRegistered = DateTime.Now,
+                    IsActive = true,
+                    DateAccessed = DateTime.Now
+                });
             }
 
             // Act
             using (ProfilingLogger.TraceDuration<NPocoExtensionsTest>("starting insert", "finished insert"))
             {
-                db.BulkInsertRecordsWithTransaction(SqlSyntax, servers);
+                using (var tr = db.GetTransaction())
+                {
+                    db.BulkInsertRecords(servers, false);
+                    tr.Complete();
+                }
             }
 
             // Assert
             Assert.That(db.ExecuteScalar<int>("SELECT COUNT(*) FROM umbracoServer"), Is.EqualTo(1000));
         }
 
+        [Test]
+        public void Can_Bulk_Insert_One_By_One_Transaction_Rollback()
+        {
+            // Arrange
+            var db = DatabaseContext.Database;
+
+            var servers = new List<ServerRegistrationDto>();
+            for (var i = 0; i < 1000; i++)
+            {
+                servers.Add(new ServerRegistrationDto
+                {
+                    ServerAddress = "address" + i,
+                    ServerIdentity = "computer" + i,
+                    DateRegistered = DateTime.Now,
+                    IsActive = true,
+                    DateAccessed = DateTime.Now
+                });
+            }
+
+            // Act
+            using (ProfilingLogger.TraceDuration<NPocoExtensionsTest>("starting insert", "finished insert"))
+            {
+                using (var tr = db.GetTransaction())
+                {
+                    db.BulkInsertRecords(servers, false);
+                    //don't call complete here - the trans will be rolled back
+                }
+            }
+
+            // Assert
+            Assert.That(db.ExecuteScalar<int>("SELECT COUNT(*) FROM umbracoServer"), Is.EqualTo(0));
+        }
+
+
+        [NUnit.Framework.Ignore("Ignored because you need to configure your own SQL Server to test thsi with")]
+        [Test]
+        public void Can_Bulk_Insert_Native_Sql_Server_Bulk_Inserts()
+        {
+            // create the db
+            // prob not what we want, this is not a real database, but hey, the test is ignored anyways
+            // we'll fix this when we have proper testing infrastructure
+            var dbSqlServer = TestObjects.GetUmbracoSqlServerDatabase(new DebugDiagnosticsLogger());
+
+            //drop the table
+            dbSqlServer.Execute("DROP TABLE [umbracoServer]");
+
+            //re-create it
+            dbSqlServer.Execute(@"CREATE TABLE [umbracoServer](
+	[id] [int] IDENTITY(1,1) NOT NULL,
+	[address] [nvarchar](500) NOT NULL,
+	[computerName] [nvarchar](255) NOT NULL,
+	[registeredDate] [datetime] NOT NULL CONSTRAINT [DF_umbracoServer_registeredDate]  DEFAULT (getdate()),
+	[lastNotifiedDate] [datetime] NOT NULL,
+	[isActive] [bit] NOT NULL,
+	[isMaster] [bit] NOT NULL,
+ CONSTRAINT [PK_umbracoServer] PRIMARY KEY CLUSTERED 
+(
+	[id] ASC
+)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
+)");
+            var data = new List<ServerRegistrationDto>();
+            for (var i = 0; i < 1000; i++)
+            {
+                data.Add(new ServerRegistrationDto
+                {
+                    ServerAddress = "address" + i,
+                    ServerIdentity = "computer" + i,
+                    DateRegistered = DateTime.Now,
+                    IsActive = true,
+                    DateAccessed = DateTime.Now
+                });
+            }
+
+            using (var tr = dbSqlServer.GetTransaction())
+            {
+                dbSqlServer.BulkInsertRecords(data);
+                tr.Complete();
+            }
+
+            // Assert
+            Assert.That(dbSqlServer.ExecuteScalar<int>("SELECT COUNT(*) FROM umbracoServer"), Is.EqualTo(1000));
+        }
+
+        [Test]
+        public void Can_Bulk_Insert_Native_Sql_Bulk_Inserts()
+        {
+            // Arrange
+            var db = DatabaseContext.Database;
+
+            var servers = new List<ServerRegistrationDto>();
+            for (var i = 0; i < 1000; i++)
+            {
+                servers.Add(new ServerRegistrationDto
+                {
+                    ServerAddress = "address" + i,
+                    ServerIdentity = "computer" + i,
+                    DateRegistered = DateTime.Now,
+                    IsActive = true,
+                    DateAccessed = DateTime.Now
+                });
+            }
+
+            // Act
+            using (ProfilingLogger.TraceDuration<NPocoExtensionsTest>("starting insert", "finished insert"))
+            {
+                using (var tr = db.GetTransaction())
+                {
+                    db.BulkInsertRecords(servers);
+                    tr.Complete();
+                }
+            }
+
+            // Assert
+            Assert.That(db.ExecuteScalar<int>("SELECT COUNT(*) FROM umbracoServer"), Is.EqualTo(1000));
+        }
+
+        [Test]
+        public void Can_Bulk_Insert_Native_Sql_Bulk_Inserts_Transaction_Rollback()
+        {
+            // Arrange
+            var db = DatabaseContext.Database;
+
+            var servers = new List<ServerRegistrationDto>();
+            for (var i = 0; i < 1000; i++)
+            {
+                servers.Add(new ServerRegistrationDto
+                {
+                    ServerAddress = "address" + i,
+                    ServerIdentity = "computer" + i,
+                    DateRegistered = DateTime.Now,
+                    IsActive = true,
+                    DateAccessed = DateTime.Now
+                });
+            }
+
+            // Act
+            using (ProfilingLogger.TraceDuration<NPocoExtensionsTest>("starting insert", "finished insert"))
+            {
+                using (var tr = db.GetTransaction())
+                {
+                    db.BulkInsertRecords(servers);
+                    //don't call complete here - the trans will be rolled back
+                }
+            }
+
+            // Assert
+            Assert.That(db.ExecuteScalar<int>("SELECT COUNT(*) FROM umbracoServer"), Is.EqualTo(0));
+        }
+
         [Test]
         public void Generate_Bulk_Import_Sql()
         {
@@ -53,23 +210,22 @@ namespace Umbraco.Tests.Persistence
             for (var i = 0; i < 2; i++)
             {
                 servers.Add(new ServerRegistrationDto
-                    {
-                        ServerAddress = "address" + i,
-                        ServerIdentity = "computer" + i,
-                        DateRegistered = DateTime.Now,
-                        IsActive = true,
-                        DateAccessed = DateTime.Now
-                    });
+                {
+                    ServerAddress = "address" + i,
+                    ServerIdentity = "computer" + i,
+                    DateRegistered = DateTime.Now,
+                    IsActive = true,
+                    DateAccessed = DateTime.Now
+                });
             }
             db.OpenSharedConnection();
 
             // Act
-            string[] sql;
-            db.GenerateBulkInsertCommand(servers.ToArray(), db.Connection, out sql);
+            var commands = db.GenerateBulkInsertCommands(servers.ToArray());
             db.CloseSharedConnection();
 
             // Assert
-            Assert.That(sql[0],
+            Assert.That(commands[0].CommandText,
                         Is.EqualTo("INSERT INTO [umbracoServer] ([umbracoServer].[address], [umbracoServer].[computerName], [umbracoServer].[registeredDate], [umbracoServer].[lastNotifiedDate], [umbracoServer].[isActive], [umbracoServer].[isMaster]) VALUES (@0,@1,@2,@3,@4,@5), (@6,@7,@8,@9,@10,@11)"));
         }
 
@@ -96,13 +252,12 @@ namespace Umbraco.Tests.Persistence
             db.OpenSharedConnection();
 
             // Act
-            string[] sql;
-            db.GenerateBulkInsertCommand(servers.ToArray(), db.Connection, out sql);
+            var commands = db.GenerateBulkInsertCommands(servers.ToArray());
             db.CloseSharedConnection();
 
             // Assert
-            Assert.That(sql.Length, Is.EqualTo(5));
-            foreach (var s in sql)
+            Assert.That(commands.Length, Is.EqualTo(5));
+            foreach (var s in commands.Select(x => x.CommandText))
             {
                 Assert.LessOrEqual(Regex.Matches(s, "@\\d+").Count, 2000);
             }

src/Umbraco.Tests/Persistence/PetaPocoCachesTest.cs

@@ -1,7 +1,11 @@
-using System;
+using System;
+using System.Collections.Generic;
+using System.Diagnostics;
 using System.Linq;
+using System.Threading;
 using NUnit.Framework;
 using Umbraco.Core.Models;
+using Umbraco.Core.Persistence;
 using Umbraco.Core.Services;
 using Umbraco.Tests.Services;
 using Umbraco.Tests.TestHelpers;
@@ -15,6 +19,8 @@ namespace Umbraco.Tests.Persistence
     [TestFixture, Ignore]
     public class PetaPocoCachesTest : BaseServiceTest
     {
+
+#if DEBUG
         /// <summary>
         /// This tests the peta poco caches
         /// </summary>
@@ -189,5 +195,6 @@ namespace Umbraco.Tests.Persistence
 
             contentService.MoveToRecycleBin(content1);
         }
+#endif
     }
 }

src/Umbraco.Tests/Persistence/Repositories/ContentTypeRepositoryTest.cs

@@ -521,7 +521,7 @@ namespace Umbraco.Tests.Persistence.Repositories
             {
                 var repository = CreateRepository(unitOfWork);
                 var contentType = repository.Get(NodeDto.NodeIdSeed + 1);
-                var child1 = MockedContentTypes.CreateSimpleContentType("aabc", "aabc", contentType, randomizeAliases: true);
+                var child1 = MockedContentTypes.CreateSimpleContentType("abc", "abc", contentType, randomizeAliases: true);
                 repository.AddOrUpdate(child1);
                 var child3 = MockedContentTypes.CreateSimpleContentType("zyx", "zyx", contentType, randomizeAliases: true);
                 repository.AddOrUpdate(child3);
@@ -535,7 +535,7 @@ namespace Umbraco.Tests.Persistence.Repositories
                 // Assert
                 Assert.That(contentTypes.Count(), Is.EqualTo(3));
                 Assert.AreEqual("a123", contentTypes.ElementAt(0).Name);
-                Assert.AreEqual("aabc", contentTypes.ElementAt(1).Name);
+                Assert.AreEqual("abc", contentTypes.ElementAt(1).Name);
                 Assert.AreEqual("zyx", contentTypes.ElementAt(2).Name);
             }
 

src/Umbraco.Tests/Services/PerformanceTests.cs

@@ -200,7 +200,7 @@ namespace Umbraco.Tests.Services
 
                     //now we insert each record for the ones we've deleted like we do in the content service.
                     var xmlItems = nodes.Select(node => new ContentXmlDto { NodeId = node.NodeId, Xml = UpdatedXmlStructure }).ToList();
-                    DatabaseContext.Database.BulkInsertRecordsWithTransaction(SqlSyntax, xmlItems);
+                    DatabaseContext.Database.BulkInsertRecordsWithTransaction(xmlItems);
                 }
             }
 
@@ -221,7 +221,7 @@ namespace Umbraco.Tests.Services
                                                     INNER JOIN cmsContent ON cmsContentXml.nodeId = cmsContent.nodeId)");
 
 
-                        DatabaseContext.Database.BulkInsertRecords(SqlSyntax, xmlItems);
+                        DatabaseContext.Database.BulkInsertRecords(xmlItems);
 
                         tr.Complete();
                     }
@@ -289,7 +289,7 @@ namespace Umbraco.Tests.Services
                     Path = ""
                 });
             }
-            DatabaseContext.Database.BulkInsertRecordsWithTransaction(SqlSyntax, nodes);
+            DatabaseContext.Database.BulkInsertRecordsWithTransaction(nodes);
 
             //re-get the nodes with ids
             var sql = DatabaseContext.Database.Sql();
@@ -299,11 +299,11 @@ namespace Umbraco.Tests.Services
             //create the cmsContent data, each with a new content type id (so we can query on it later if needed)
             var contentTypeId = 0;
             var cmsContentItems = nodes.Select(node => new ContentDto { NodeId = node.NodeId, ContentTypeId = contentTypeId++ }).ToList();
-            DatabaseContext.Database.BulkInsertRecordsWithTransaction(SqlSyntax, cmsContentItems);
+            DatabaseContext.Database.BulkInsertRecordsWithTransaction(cmsContentItems);
 
             //create the xml data
             var xmlItems = nodes.Select(node => new ContentXmlDto { NodeId = node.NodeId, Xml = TestXmlStructure }).ToList();
-            DatabaseContext.Database.BulkInsertRecordsWithTransaction(SqlSyntax, xmlItems);
+            DatabaseContext.Database.BulkInsertRecordsWithTransaction(xmlItems);
 
             return nodes;
         }

src/Umbraco.Tests/TestHelpers/TestHelper.cs

@@ -205,16 +205,16 @@ namespace Umbraco.Tests.TestHelpers
 
         public static void DeleteDirectory(string path)
         {
-            if (Directory.Exists(path) == false) return;
-
             Try(() =>
             {
+                if (Directory.Exists(path) == false) return;
                 foreach (var file in Directory.EnumerateFiles(path, "*", SearchOption.AllDirectories))
                     File.Delete(file);
             });
 
             Try(() =>
             {
+                if (Directory.Exists(path) == false) return;
                 Directory.Delete(path, true);
             });
         }

src/Umbraco.Tests/Umbraco.Tests.csproj

@@ -236,6 +236,7 @@
     <Compile Include="Integration\ContentEventsTests.cs" />
     <Compile Include="Migrations\MigrationIssuesTests.cs" />
     <Compile Include="NUnitTests.cs" />
+    <Compile Include="Persistence\BulkDataReaderTests.cs" />
     <Compile Include="Persistence\Migrations\PostMigrationTests.cs" />
     <Compile Include="Persistence\NPocoExpressionsTests.cs" />
     <Compile Include="Persistence\UnitOfWorkTests.cs" />

src/Umbraco.Web.UI.Client/src/less/canvas-designer.less

@@ -21,7 +21,6 @@ body {
     font-size: 14px;
     line-height: 20px;
     color: #343434;
-    background-color: #F5F5F5;
     -webkit-transition: all 0.2s ease-in-out;
     -moz-transition: all 0.2s ease-in-out;
     transition: all 0.2s ease-in-out;

src/Umbraco.Web.UI.Client/src/less/main.less

@@ -157,7 +157,7 @@ h5.-black {
 }
 
 .controls-row {
-  padding-top: 5px;
+  padding-bottom: 5px;
   margin-left: 240px;
 }
 

src/Umbraco.Web.UI.Client/src/less/property-editors.less

@@ -232,11 +232,11 @@ ul.color-picker li a  {
   max-height: none !important;
 }
 
-.umb-sortable-thumbnails .icon-holder {
+.umb-sortable-thumbnails .umb-icon-holder {
     text-align: center;
 }
 
-.umb-sortable-thumbnails .icon-holder .icon{
+.umb-sortable-thumbnails .umb-icon-holder .icon{
     font-size: 40px;
     line-height: 50px;
     color: @gray;

src/Umbraco.Web.UI.Client/src/views/packager/views/install-local.controller.js

@@ -34,6 +34,9 @@
                 file: file
             }).progress(function (evt) {
 
+                // set view state to uploading
+                vm.state = 'uploading';
+
                 // calculate progress in percentage
                 var progressPercentage = parseInt(100.0 * evt.loaded / evt.total, 10);
 

src/Umbraco.Web.UI.Client/src/views/packager/views/install-local.html

@@ -37,10 +37,8 @@
                             - <localize key="media_orClickHereToUpload">or click here to choose files</localize>
                         </div>
                         
-                        <div class="umb-info-local-item text-error" ng-if="vm.zipFile.uploadStatus === 'error'">
-                            {{vm.zipFile.serverErrorMessage}}
-                        </div>
                     </div>
+                    
                 </div>
 
 
@@ -53,6 +51,41 @@
         </div>
     </div>
 
+    <div ng-if="vm.state === 'uploading'">
+
+        <umb-editor-sub-header>
+            <umb-editor-sub-header-content-left>
+                <a class="umb-package-details__back-link" href="" ng-click="vm.state = 'upload'">&larr; Upload another package</a>
+            </umb-editor-sub-header-content-left>
+        </umb-editor-sub-header>
+
+        <div class="umb-packages-view-wrapper">
+
+            <div class="flex items-center justify-center">
+                <div class="umb-info-local-items">
+                    <div class="umb-package-icon">
+                        <i ng-if="!vm.localPackage.iconUrl" class="icon-box"></i>
+                    </div>
+                    <div class="umb-package-info">
+
+                        <h4 class="umb-info-local-item"><strong>Uploading package</strong></h4>
+
+                        <umb-progress-bar
+                            percentage="{{vm.zipFile.uploadProgress}}">
+                        </umb-progress-bar>
+
+                        <div class="umb-info-local-item text-error mt3" ng-if="vm.zipFile.uploadStatus === 'error'">
+                            {{ vm.zipFile.serverErrorMessage }}
+                        </div>
+                        
+                    </div>
+                </div>
+            </div>
+
+        </div>
+
+    </div>
+
     <div ng-if="vm.state === 'packageDetails'">
 
         <umb-editor-sub-header>

src/Umbraco.Web.UI.Client/src/views/propertyeditors/mediapicker/mediapicker.html

@@ -11,7 +11,7 @@
 			<img ng-if="image.extension === 'svg'" ng-src="{{image.file}}" alt="" />
 
 			<!-- FILE -->
-			<span class="icon-holder" ng-hide="image.thumbnail || image.metaData.umbracoExtension.Value === 'svg' || image.extension === 'svg'">
+			<span class="umb-icon-holder" ng-hide="image.thumbnail || image.metaData.umbracoExtension.Value === 'svg' || image.extension === 'svg'">
 				<i class="icon {{image.icon}} large"></i>
 				<small>{{image.name}}</small>
 			</span>

src/Umbraco.Web.UI/config/Dashboard.config

@@ -104,16 +104,6 @@
       </control>
     </tab>
   </section>
-  <section alias="UmbracoModelsBuilder">
-    <areas>
-      <area>developer</area>
-    </areas>
-    <tab caption="Models Builder">
-      <control>
-        /App_Plugins/ModelsBuilder/modelsbuilder.htm
-      </control>
-    </tab>
-  </section>
   <section alias="RedirectUrlManagement">
     <areas>
       <area>content</area>
@@ -124,6 +114,16 @@
       </control>
     </tab>
   </section>
+  <section alias="UmbracoModelsBuilder">
+    <areas>
+      <area>developer</area>
+    </areas>
+    <tab caption="Models Builder">
+      <control>
+        /App_Plugins/ModelsBuilder/modelsbuilder.htm
+      </control>
+    </tab>
+  </section>
   <section alias="Deploy">
     <areas>
       <area>content</area>

src/Umbraco.Web.UI/umbraco/config/lang/da.xml

@@ -304,8 +304,16 @@
     <key alias="selectEditor">Vælg editor</key>
   </area>
   <area alias="dictionaryItem">
-    <key alias="description">Rediger de forskellige sprogversioner for ordbogselementet '%0%' herunder. Du tilføjer flere sprog under 'sprog' i menuen til venstre </key>
+    <key alias="description"><![CDATA[
+      Rediger de forskellige sprogversioner for ordbogselementet '%0%' herunder.<br />Du tilføjer flere sprog under 'sprog' i menuen til venstre </key>
+      ]]></key>
     <key alias="displayName">Kulturnavn</key>
+    <key alias="changeKey">Rediger navnet på ordbogselementet.</key>
+    <key alias="changeKeyError">
+      <![CDATA[
+      Navnet '%0%' eksisterer allerede.
+   ]]>
+    </key>
   </area>
   <area alias="placeholders">
     <key alias="username">Indtast dit brugernavn</key>

src/Umbraco.Web.UI/umbraco/config/lang/en.xml

@@ -144,7 +144,7 @@
     <key alias="memberrole">Role</key>
     <key alias="membertype">Member Type</key>
     <key alias="noDate">No date chosen</key>
-    <key alias="nodeName">Page Title</key>
+    <key alias="nodeName">Link title</key>
     <key alias="otherElements">Properties</key>
     <key alias="parentNotPublished">This document is published but is not visible because the parent '%0%' is unpublished</key>
     <key alias="parentNotPublishedAnomaly">This document is published but is not in the cache</key>
@@ -328,6 +328,12 @@
       Edit the different language versions for the dictionary item '<em>%0%</em>' below<br/>You can add additional languages under the 'languages' in the menu on the left
    ]]></key>
     <key alias="displayName">Culture Name</key>
+    <key alias="changeKey">Edit the key of the dictionary item.</key>
+    <key alias="changeKeyError">
+      <![CDATA[
+      The key '%0%' already exists.
+   ]]>
+    </key>
   </area>
   <area alias="placeholders">
     <key alias="username">Enter your username</key>

src/Umbraco.Web.UI/umbraco/config/lang/en_us.xml

@@ -145,7 +145,7 @@
     <key alias="memberrole">Role</key>
     <key alias="membertype">Member Type</key>
     <key alias="noDate">No date chosen</key>
-    <key alias="nodeName">Page Title</key>
+    <key alias="nodeName">Link title</key>
     <key alias="otherElements">Properties</key>
     <key alias="parentNotPublished">This document is published but is not visible because the parent '%0%' is unpublished</key>
     <key alias="parentNotPublishedAnomaly">This document is published but is not in the cache</key>
@@ -329,6 +329,12 @@
       Edit the different language versions for the dictionary item '<em>%0%</em>' below<br/>You can add additional languages under the 'languages' in the menu on the left
    ]]></key>
     <key alias="displayName">Culture Name</key>
+    <key alias="changeKey">Edit the key of the dictionary item.</key>
+    <key alias="changeKeyError">
+      <![CDATA[
+      The key '%0%' already exists.
+   ]]>
+    </key>
   </area>
   <area alias="placeholders">
     <key alias="username">Enter your username</key>

src/Umbraco.Web.UI/umbraco/create/User.ascx

@@ -22,7 +22,7 @@
         ControlToValidate="Email" ValidateEmptyText="false" OnServerValidate="EmailExistsCheck"></asp:CustomValidator>
     <asp:CustomValidator runat="server" ID="EmailValidator" OnServerValidate="EmailValidator_OnServerValidate"
         ControlToValidate="Email"
-        ErrorMessage="Invalid email address"
+        ErrorMessage='<%#ui.Text("errorHandling", "errorRegExpWithoutTab", "E-mail", CurrentUser) %>'
         Display="None" />
     <br />
     <asp:TextBox ID="Email" runat="server" Width="350px" CssClass="bigInput"></asp:TextBox>

src/Umbraco.Web/BatchedDatabaseServerMessenger.cs

@@ -86,11 +86,16 @@ namespace Umbraco.Web
 
             var instructions = batch.SelectMany(x => x.Instructions).ToArray();
             batch.Clear();
-            if (instructions.Length == 0) return;
-            WriteInstructions(instructions);
+
+            //Write the instructions but only create JSON blobs with a max instruction count equal to MaxProcessingInstructionCount
+            foreach (var instructionsBatch in instructions.InGroupsOf(Options.MaxProcessingInstructionCount))
+            {
+                WriteInstructions(instructionsBatch);
+            }
+            
         }
 
-        private void WriteInstructions(RefreshInstruction[] instructions)
+        private void WriteInstructions(IEnumerable<RefreshInstruction> instructions)
         {
             var dto = new CacheInstructionDto
             {
@@ -138,9 +143,18 @@ namespace Umbraco.Web
 
             // batch if we can, else write to DB immediately
             if (batch == null)
-                WriteInstructions(instructions.ToArray());
+            {
+                //only write the json blob with a maximum count of the MaxProcessingInstructionCount
+                foreach (var maxBatch in instructions.InGroupsOf(Options.MaxProcessingInstructionCount))
+                {
+                    WriteInstructions(maxBatch);
+                }
+            }
             else
+            {
                 batch.Add(new RefreshInstructionEnvelope(servers, refresher, instructions));
+            }
+                
         }        
     }
 }

src/Umbraco.Web/HealthCheck/Checks/Config/TrySkipIisCustomErrorsCheck.cs

@@ -27,7 +27,8 @@ namespace Umbraco.Web.HealthCheck.Checks.Config
         {
             get
             {
-                var recommendedValue = _serverVersion >= new Version("7.5.0")
+                // beware! 7.5 and 7.5.0 are not the same thing!
+                var recommendedValue = _serverVersion >= new Version("7.5")
                     ? bool.TrueString.ToLower()
                     : bool.FalseString.ToLower();
                 return new List<AcceptableConfiguration> { new AcceptableConfiguration { IsRecommended =  true, Value = recommendedValue } };
@@ -39,7 +40,7 @@ namespace Umbraco.Web.HealthCheck.Checks.Config
             get
             {
                 return TextService.Localize("healthcheck/trySkipIisCustomErrorsCheckSuccessMessage",
-                    new[] { CurrentValue, Values.First(v => v.IsRecommended).Value, _serverVersion.ToString() });
+                    new[] { Values.First(v => v.IsRecommended).Value, _serverVersion.ToString() });
             }
         }
 

src/Umbraco.Web/HtmlStringUtilities.cs

@@ -215,6 +215,13 @@ namespace Umbraco.Web
 
                                     if (!lengthReached && currentTextLength >= length)
                                     {
+                                        // if the last character added was the first of a two character unicode pair, add the second character
+                                        if (Char.IsHighSurrogate((char)ic))
+                                        {
+                                            var lowSurrogate = tr.Read();
+                                            outputtw.Write((char)lowSurrogate);
+                                        }
+
                                         // Reached truncate limit.
                                         if (addElipsis)
                                         {

src/Umbraco.Web/Models/Mapping/DataTypeModelMapper.cs

@@ -68,7 +68,7 @@ namespace Umbraco.Web.Models.Mapping
                 });
 
             config.CreateMap<IDataTypeDefinition, DataTypeDisplay>()
-                .ForMember(display => display.AvailableEditors, expression => expression.ResolveUsing<AvailablePropertyEditorsResolver>())
+                .ForMember(display => display.AvailableEditors, expression => expression.ResolveUsing(new AvailablePropertyEditorsResolver()))
                 .ForMember(display => display.PreValues, expression => expression.ResolveUsing(new PreValueDisplayResolver(lazyDataTypeService)))
                 .ForMember(display => display.SelectedEditor, expression => expression.MapFrom(
                     definition => definition.PropertyEditorAlias.IsNullOrWhiteSpace() ? null : definition.PropertyEditorAlias))
@@ -104,7 +104,7 @@ namespace Umbraco.Web.Models.Mapping
                 .ForMember(definition => definition.Key, expression => expression.Ignore())
                 .ForMember(definition => definition.Path, expression => expression.Ignore())
                 .ForMember(definition => definition.PropertyEditorAlias, expression => expression.MapFrom(save => save.SelectedEditor))
-                .ForMember(definition => definition.DatabaseType, expression => expression.ResolveUsing<DatabaseTypeResolver>())
+                .ForMember(definition => definition.DatabaseType, expression => expression.ResolveUsing(new DatabaseTypeResolver()))
                 .ForMember(x => x.CreatorId, expression => expression.Ignore())
                 .ForMember(x => x.Level, expression => expression.Ignore())
                 .ForMember(x => x.SortOrder, expression => expression.Ignore())

src/Umbraco.Web/Models/Mapping/MemberModelMapper.cs

@@ -89,7 +89,7 @@ namespace Umbraco.Web.Models.Mapping
                 .ForMember(display => display.Tabs,
                     expression => expression.ResolveUsing(new MemberTabsAndPropertiesResolver(_textService)))
                 .ForMember(display => display.MemberProviderFieldMapping,
-                    expression => expression.ResolveUsing<MemberProviderFieldMappingResolver>())
+                    expression => expression.ResolveUsing(new MemberProviderFieldMappingResolver()))
                 .ForMember(display => display.MembershipScenario,
                     expression => expression.ResolveUsing(new MembershipScenarioMappingResolver(new Lazy<IMemberTypeService>(() => _memberTypeService))))
                 .ForMember(display => display.Notifications, expression => expression.Ignore())
@@ -170,7 +170,7 @@ namespace Umbraco.Web.Models.Mapping
                 .ForMember(x => x.Alias, expression => expression.Ignore())
                 .ForMember(member => member.HasPublishedVersion, expression => expression.Ignore())
                 //do no map the custom member properties (currently anyways, they were never there in 6.x)
-                .ForMember(dto => dto.Properties, expression => expression.ResolveUsing<MemberDtoPropertiesValueResolver>());
+                .ForMember(dto => dto.Properties, expression => expression.ResolveUsing(new MemberDtoPropertiesValueResolver()));
 
             //FROM IMemberGroup TO MemberGroupDisplay
             config.CreateMap<IMemberGroup, MemberGroupDisplay>()

src/Umbraco.Web/PropertyEditors/FileUploadPropertyValueEditor.cs

@@ -63,10 +63,9 @@ namespace Umbraco.Web.PropertyEditors
                 return currentValue;
 
             // get the current file paths
-            var fs = _mediaFileSystem;
             var currentPaths = currentValue.ToString()
                 .Split(new[] { ',' }, StringSplitOptions.RemoveEmptyEntries)
-                .Select(x => fs.GetRelativePath(x)) // get the fs-relative path
+                .Select(x => _mediaFileSystem.GetRelativePath(x)) // get the fs-relative path
                 .ToArray();
 
             // if clearing, remove these files and return
@@ -110,9 +109,9 @@ namespace Umbraco.Web.PropertyEditors
 
                 using (var filestream = File.OpenRead(file.TempFilePath))
                 {
-                    fs.AddFile(filepath, filestream, true); // must overwrite!
+                    _mediaFileSystem.AddFile(filepath, filestream, true); // must overwrite!
 
-                    var ext = fs.GetExtension(filepath);
+                    var ext = _mediaFileSystem.GetExtension(filepath);
                     if (_mediaFileSystem.IsImageFile(ext))
                     {
                         var preValues = editorValue.PreValues.FormatAsDictionary();
@@ -137,7 +136,7 @@ namespace Umbraco.Web.PropertyEditors
                 _mediaFileSystem.DeleteFile(pathToRemove, true);
 
 
-            return string.Join(",", newPaths.Select(x => fs.GetUrl(x)));
+            return string.Join(",", newPaths.Select(x => _mediaFileSystem.GetUrl(x)));
         }
     }
 }

src/Umbraco.Web/PublishedCache/NuCache/FacadeService.cs

@@ -1242,7 +1242,7 @@ WHERE cmsContentNu.nodeId IN (
                 }
                 items.AddRange(guids.Select(x => GetDto(repository.GetByVersion(x), true)));
 
-                db.BulkInsertRecords(db.SqlSyntax, items);
+                db.BulkInsertRecords(items);
                 processed += items.Count;
             } while (processed < total);
         }
@@ -1302,7 +1302,7 @@ WHERE cmsContentNu.nodeId IN (
             {
                 var descendants = repository.GetPagedResultsByQuery(query, pageIndex++, groupSize, out total, "Path", Direction.Ascending, true);
                 var items = descendants.Select(m => GetDto(m, true)).ToArray();
-                db.BulkInsertRecords(db.SqlSyntax, items);
+                db.BulkInsertRecords(items);
                 processed += items.Length;
             } while (processed < total);
         }
@@ -1362,7 +1362,7 @@ WHERE cmsContentNu.nodeId IN (
             {
                 var descendants = repository.GetPagedResultsByQuery(query, pageIndex++, groupSize, out total, "Path", Direction.Ascending, true);
                 var items = descendants.Select(m => GetDto(m, true)).ToArray();
-                db.BulkInsertRecords(db.SqlSyntax, items);
+                db.BulkInsertRecords(items);
                 processed += items.Length;
             } while (processed < total);
         }

src/Umbraco.Web/PublishedCache/XmlPublishedCache/PublishedContentCache.cs

@@ -11,6 +11,8 @@ using Umbraco.Core.Xml;
 using Umbraco.Web.Routing;
 using System.Linq;
 using Umbraco.Core.Cache;
+using Umbraco.Core.Services;
+using Task = System.Threading.Tasks.Task;
 
 namespace Umbraco.Web.PublishedCache.XmlPublishedCache
 {
@@ -132,6 +134,13 @@ namespace Umbraco.Web.PublishedCache.XmlPublishedCache
             // - non-colliding, adds one complete "by route" lookup, only on the first time a url is computed (then it's cached anyways)
             // - colliding, adds one "by route" lookup, the first time the url is computed, then one dictionary looked each time it is computed again
             // assuming no collisions, the impact is one complete "by route" lookup the first time each url is computed
+            //
+            // U4-9121 - this lookup is too expensive when computing a large amount of urls on a front-end (eg menu)
+            // ... thinking about moving the lookup out of the path into its own async task, so we are not reporting errors
+            //     in the back-office anymore, but at least we are not polluting the cache
+            // instead, refactored DeterminedIdByRoute to stop using XPath, with a 16x improvement according to benchmarks
+            // will it be enough?
+
             var loopId = preview ? 0 : (_routesCache?.GetNodeId(route) ?? 0); // might be cached already in case of collision
             if (loopId == 0)
             {
@@ -168,30 +177,98 @@ namespace Umbraco.Web.PublishedCache.XmlPublishedCache
             var pos = route.IndexOf('/');
             var path = pos == 0 ? route : route.Substring(pos);
             var startNodeId = pos == 0 ? 0 : int.Parse(route.Substring(0, pos));
-            IEnumerable<XPathVariable> vars;
-
-            var xpath = CreateXpathQuery(startNodeId, path, hideTopLevelNode, out vars);
 
             //check if we can find the node in our xml cache
-            var content = GetSingleByXPath(preview, xpath, vars?.ToArray());
+            var id = NavigateRoute(preview, startNodeId, path, hideTopLevelNode);
+            if (id > 0) return GetById(preview, id);
 
             // if hideTopLevelNodePath is true then for url /foo we looked for /*/foo
             // but maybe that was the url of a non-default top-level node, so we also
             // have to look for /foo (see note in ApplyHideTopLevelNodeFromPath).
-            if (content == null && hideTopLevelNode && path.Length > 1 && path.IndexOf('/', 1) < 0)
+            if (hideTopLevelNode && path.Length > 1 && path.IndexOf('/', 1) < 0)
             {
-                xpath = CreateXpathQuery(startNodeId, path, false, out vars);
-                content = GetSingleByXPath(preview, xpath, vars?.ToArray());
+                var id2 = NavigateRoute(preview, startNodeId, path, false);
+                if (id2 > 0) return GetById(preview, id2);
             }
 
-            return content;
+            return null;
+        }
+
+        private int NavigateRoute(bool preview, int startNodeId, string path, bool hideTopLevelNode)
+        {
+            var xml = GetXml(preview);
+            XmlElement elt;
+
+            // empty path
+            if (path == string.Empty || path == "/")
+            {
+                if (startNodeId > 0)
+                {
+                    elt = xml.GetElementById(startNodeId.ToString(CultureInfo.InvariantCulture));
+                    return elt == null ? -1 : startNodeId;
+                }
+
+                elt = null;
+                var min = int.MaxValue;
+                foreach (XmlElement e in xml.DocumentElement.ChildNodes)
+                {
+                    var sortOrder = int.Parse(e.GetAttribute("sortOrder"));
+                    if (sortOrder < min)
+                    {
+                        min = sortOrder;
+                        elt = e;
+                    }
+                }
+                return elt == null ? -1 : int.Parse(elt.GetAttribute("id"));
+            }
+
+            // non-empty path
+            elt = startNodeId <= 0
+                ? xml.DocumentElement
+                : xml.GetElementById(startNodeId.ToString(CultureInfo.InvariantCulture));
+            if (elt == null) return -1;
+
+            var urlParts = path.Split(SlashChar, StringSplitOptions.RemoveEmptyEntries);
+
+            if (hideTopLevelNode && startNodeId <= 0)
+            {
+                foreach (XmlElement e in elt.ChildNodes)
+                {
+                    var id = NavigateElementRoute(e, urlParts);
+                    if (id > 0) return id;
+                }
+                return -1;
+            }
+
+            return NavigateElementRoute(elt, urlParts);
+        }
+
+        private int NavigateElementRoute(XmlElement elt, string[] urlParts)
+        {
+            var found = true;
+            var i = 0;
+            while (found && i < urlParts.Length)
+            {
+                found = false;
+                foreach (XmlElement child in elt.ChildNodes)
+                {
+                    var noNode = child.GetAttributeNode("isDoc") == null;
+                    if (noNode) continue;
+                    if (child.GetAttribute("urlName") != urlParts[i]) continue;
+
+                    found = true;
+                    elt = child;
+                    break;
+                }
+                i++;
+            }
+            return found ? int.Parse(elt.GetAttribute("id")) : -1;
         }
 
         string DetermineRouteById(bool preview, int contentId)
         {
             var node = GetById(preview, contentId);
-            if (node == null)
-                return null;
+            if (node == null) return null;
 
             // walk up from that node until we hit a node with a domain,
             // or we reach the content root, collecting urls in the way
@@ -253,10 +330,6 @@ namespace Umbraco.Web.PublishedCache.XmlPublishedCache
         {
             public const string Root = "/root";
             public const string RootDocuments = "/root/* [@isDoc]";
-            public const string DescendantDocumentById = "//* [@isDoc and @id={0}]";
-            public const string ChildDocumentByUrlName = "/* [@isDoc and @urlName='{0}']";
-            public const string ChildDocumentByUrlNameVar = "/* [@isDoc and @urlName=${0}]";
-            public const string RootDocumentWithLowestSortOrder = "/root/* [@isDoc and not(@sortOrder > ../* [@isDoc]/@sortOrder)][1]";
         }
 
         #endregion
@@ -436,82 +509,6 @@ namespace Umbraco.Web.PublishedCache.XmlPublishedCache
 
         static readonly char[] SlashChar = { '/' };
 
-        protected string CreateXpathQuery(int startNodeId, string path, bool hideTopLevelNodeFromPath, out IEnumerable<XPathVariable> vars)
-        {
-            string xpath;
-            vars = null;
-
-            if (path == string.Empty || path == "/")
-            {
-                // if url is empty
-                if (startNodeId > 0)
-                {
-					// if in a domain then use the root node of the domain
-					xpath = string.Format(XPathStrings.Root + XPathStrings.DescendantDocumentById, startNodeId);
-                }
-                else
-                {
-                    // if not in a domain - what is the default page?
-                    // let's say it is the first one in the tree, if any -- order by sortOrder
-
-					// but!
-					// umbraco does not consistently guarantee that sortOrder starts with 0
-					// so the one that we want is the one with the smallest sortOrder
-					// read http://stackoverflow.com/questions/1128745/how-can-i-use-xpath-to-find-the-minimum-value-of-an-attribute-in-a-set-of-elemen
-
-					// so that one does not work, because min(@sortOrder) maybe 1
-					// xpath = "/root/*[@isDoc and @sortOrder='0']";
-
-					// and we can't use min() because that's XPath 2.0
-					// that one works
-					xpath = XPathStrings.RootDocumentWithLowestSortOrder;
-                }
-            }
-            else
-            {
-                // if url is not empty, then use it to try lookup a matching page
-                var urlParts = path.Split(SlashChar, StringSplitOptions.RemoveEmptyEntries);
-                var xpathBuilder = new StringBuilder();
-                var partsIndex = 0;
-                List<XPathVariable> varsList = null;
-
-                if (startNodeId == 0)
-                {
-                    // if hiding, first node is not in the url
-                    xpathBuilder.Append(hideTopLevelNodeFromPath ? XPathStrings.RootDocuments : XPathStrings.Root);
-                }
-                else
-                {
-					xpathBuilder.AppendFormat(XPathStrings.Root + XPathStrings.DescendantDocumentById, startNodeId);
-					// always "hide top level" when there's a domain
-                }
-
-                while (partsIndex < urlParts.Length)
-                {
-                    var part = urlParts[partsIndex++];
-                    if (part.Contains('\'') || part.Contains('"'))
-                    {
-                        // use vars, escaping gets ugly pretty quickly
-                        varsList = varsList ?? new List<XPathVariable>();
-                        var varName = $"var{partsIndex}";
-                        varsList.Add(new XPathVariable(varName, part));
-                        xpathBuilder.AppendFormat(XPathStrings.ChildDocumentByUrlNameVar, varName);
-                    }
-                    else
-                    {
-                        xpathBuilder.AppendFormat(XPathStrings.ChildDocumentByUrlName, part);
-
-                    }
-                }
-
-                xpath = xpathBuilder.ToString();
-                if (varsList != null)
-                    vars = varsList.ToArray();
-            }
-
-            return xpath;
-        }
-
         #endregion
 
         #region Content types

src/Umbraco.Web/PublishedCache/XmlPublishedCache/PublishedMediaCache.cs

@@ -305,7 +305,7 @@ namespace Umbraco.Web.PublishedCache.XmlPublishedCache
             // that has been deleted, hence is not in the Examine index anymore (for a good reason). try to get
             // the media from the service, first
             var media = _mediaService.GetById(id);
-            if (media == null) return null; // not found, ok
+            if (media == null || media.Trashed) return null; // not found, ok
 
             // so, the media was not found in Examine's index *yet* it exists, which probably indicates that
             // the index is corrupted. Or not up-to-date. Log a warning, but only once, and only if seeing the

src/Umbraco.Web/PublishedCache/XmlPublishedCache/XmlStore.cs

@@ -1825,7 +1825,7 @@ WHERE cmsContentXml.nodeId IN (
                 // because we already have the condition on the content being published
                 var descendants = repository.GetPagedResultsByQuery(query, pageIndex++, groupSize, out total, "Path", Direction.Ascending, true, newest: false);
                 var items = descendants.Select(c => new ContentXmlDto { NodeId = c.Id, Xml = _xmlContentSerializer(c).ToDataString() }).ToArray();
-                db.BulkInsertRecords(db.SqlSyntax, items);
+                db.BulkInsertRecords(items);
                 processed += items.Length;
             } while (processed < total);
         }
@@ -1901,7 +1901,7 @@ WHERE cmsPreviewXml.nodeId IN (
                     NodeId = c.Id,
                     Xml = _xmlContentSerializer(c).ToDataString()
                 }).ToArray();
-                db.BulkInsertRecords(db.SqlSyntax, items);
+                db.BulkInsertRecords(items);
                 processed += items.Length;
             } while (processed < total);
         }
@@ -1971,7 +1971,7 @@ WHERE cmsContentXml.nodeId IN (
             {
                 var descendants = repository.GetPagedResultsByQuery(query, pageIndex++, groupSize, out total, "Path", Direction.Ascending, true);
                 var items = descendants.Select(m => new ContentXmlDto { NodeId = m.Id, Xml = _xmlMediaSerializer(m).ToDataString() }).ToArray();
-                db.BulkInsertRecords(db.SqlSyntax, items);
+                db.BulkInsertRecords(items);
                 processed += items.Length;
             } while (processed < total);
         }
@@ -2041,7 +2041,7 @@ WHERE cmsContentXml.nodeId IN (
             {
                 var descendants = repository.GetPagedResultsByQuery(query, pageIndex++, groupSize, out total, "Path", Direction.Ascending, true);
                 var items = descendants.Select(m => new ContentXmlDto { NodeId = m.Id, Xml = _xmlMemberSerializer(m).ToDataString() }).ToArray();
-                db.BulkInsertRecords(db.SqlSyntax, items);
+                db.BulkInsertRecords(items);
                 processed += items.Length;
             } while (processed < total);
         }

src/Umbraco.Web/Search/LuceneIndexerExtensions.cs

@@ -5,6 +5,8 @@ using Examine.LuceneEngine.Providers;
 using Examine.Providers;
 using Lucene.Net.Index;
 using Lucene.Net.Search;
+using Lucene.Net.Store;
+using Umbraco.Core.Logging;
 
 namespace Umbraco.Web.Search
 {
@@ -21,12 +23,20 @@ namespace Umbraco.Web.Search
         /// <returns></returns>
         public static int GetIndexDocumentCount(this LuceneIndexer indexer)
         {
-            var searcher = indexer.GetSearcher().GetSearcher() as IndexSearcher;
-            if (searcher == null) return 0;
-            using (searcher)
-            using (var reader = searcher.IndexReader)
+            try
             {
-                return reader.NumDocs();
+                var searcher = indexer.GetSearcher().GetSearcher() as IndexSearcher;
+                if (searcher == null) return 0;
+                using (searcher)
+                using (var reader = searcher.IndexReader)
+                {
+                    return reader.NumDocs();
+                }
+            }
+            catch (AlreadyClosedException)
+            {
+                Current.Logger.Warn(typeof(ExamineExtensions), "Cannot get GetIndexDocumentCount, the writer is already closed");
+                return 0;
             }
         }
 
@@ -37,12 +47,22 @@ namespace Umbraco.Web.Search
         /// <returns></returns>
         public static int GetIndexFieldCount(this LuceneIndexer indexer)
         {
-            var searcher = indexer.GetSearcher().GetSearcher() as IndexSearcher;
-            if (searcher == null) return 0;
-            using (searcher)
-            using (var reader = searcher.IndexReader)
+            //TODO: check for closing! and AlreadyClosedException
+
+            try
             {
-                return reader.GetFieldNames(IndexReader.FieldOption.ALL).Count;
+                var searcher = indexer.GetSearcher().GetSearcher() as IndexSearcher;
+                if (searcher == null) return 0;
+                using (searcher)
+                using (var reader = searcher.IndexReader)
+                {
+                    return reader.GetFieldNames(IndexReader.FieldOption.ALL).Count;
+                }
+            }
+            catch (AlreadyClosedException)
+            {
+                Current.Logger.Warn(typeof(ExamineExtensions), "Cannot get GetIndexFieldCount, the writer is already closed");
+                return 0;
             }
         }
 
@@ -53,12 +73,20 @@ namespace Umbraco.Web.Search
         /// <returns></returns>
         public static bool IsIndexOptimized(this LuceneIndexer indexer)
         {
-            var searcher = indexer.GetSearcher().GetSearcher() as IndexSearcher;
-            if (searcher == null) return true;
-            using (searcher)
-            using (var reader = searcher.IndexReader)
+            try
             {
-                return reader.IsOptimized();
+                var searcher = indexer.GetSearcher().GetSearcher() as IndexSearcher;
+                if (searcher == null) return true;
+                using (searcher)
+                using (var reader = searcher.IndexReader)
+                {
+                    return reader.IsOptimized();
+                }
+            }
+            catch (AlreadyClosedException)
+            {
+                Current.Logger.Warn(typeof(ExamineExtensions), "Cannot get IsIndexOptimized, the writer is already closed");
+                return false;
             }
         }
 
@@ -83,12 +111,20 @@ namespace Umbraco.Web.Search
         /// <returns></returns>
         public static int GetDeletedDocumentsCount(this LuceneIndexer indexer)
         {
-            var searcher = indexer.GetSearcher().GetSearcher() as IndexSearcher;
-            if (searcher == null) return 0;
-            using (searcher)
-            using (var reader = searcher.IndexReader)
+            try
             {
-                return reader.NumDeletedDocs;
+                var searcher = indexer.GetSearcher().GetSearcher() as IndexSearcher;
+                if (searcher == null) return 0;
+                using (searcher)
+                using (var reader = searcher.IndexReader)
+                {
+                    return reader.NumDeletedDocs;
+                }
+            }
+            catch (AlreadyClosedException)
+            {
+                Current.Logger.Warn(typeof(ExamineExtensions), "Cannot get GetDeletedDocumentsCount, the writer is already closed");
+                return 0;
             }
         }
     }

src/Umbraco.Web/umbraco.presentation/umbraco/settings/EditDictionaryItem.aspx.cs

@@ -5,48 +5,72 @@ using System.Web.UI.WebControls;
 using umbraco.cms.presentation.Trees;
 using Umbraco.Core;
 using Umbraco.Core.Models;
+using Umbraco.Core.Models.Membership;
 using Umbraco.Core.Services;
 using Umbraco.Web;
 using Umbraco.Web.UI;
 
 namespace umbraco.settings
 {
-	/// <summary>
-	/// Summary description for EditDictionaryItem.
-	/// </summary>
+    /// <summary>
+    /// Summary description for EditDictionaryItem.
+    /// </summary>
     [WebformsPageTreeAuthorize(Constants.Trees.Dictionary)]
 	public partial class EditDictionaryItem : Umbraco.Web.UI.Pages.UmbracoEnsuredPage
-	{
-	    
-		protected LiteralControl keyTxt = new LiteralControl();
-		protected uicontrols.TabView tbv = new uicontrols.TabView();
-		private System.Collections.ArrayList languageFields = new System.Collections.ArrayList();
+    {
+        protected LiteralControl keyTxt = new LiteralControl();
+        protected uicontrols.TabView tbv = new uicontrols.TabView();
+        private System.Collections.ArrayList languageFields = new System.Collections.ArrayList();
         private IDictionaryItem currentItem;
+        protected TextBox boxChangeKey;
+        protected Label labelChangeKey;
+        protected Literal txt;
 
-		protected void Page_Load(object sender, System.EventArgs e)
-		{
-		    currentItem = Services.LocalizationService.GetDictionaryItemById(int.Parse(Request.QueryString["id"]));                
+        protected void Page_Load(object sender, System.EventArgs e)
+        {
+		    currentItem = Services.LocalizationService.GetDictionaryItemById(int.Parse(Request.QueryString["id"]));
 
-			// Put user code to initialize the page here
-			Panel1.hasMenu = true;
+            // Put user code to initialize the page here
+            Panel1.hasMenu = true;
 			Panel1.Text = Services.TextService.Localize("editdictionary") + ": " + currentItem.ItemKey;
-			
-            uicontrols.Pane p = new uicontrols.Pane();
 
-			var save = Panel1.Menu.NewButton();
+            var save = Panel1.Menu.NewButton();
             save.Text = Services.TextService.Localize("save");
             save.Click += save_Click;
 			save.ToolTip = Services.TextService.Localize("save");
             save.ID = "save";
             save.ButtonType = uicontrols.MenuButtonType.Primary;
 
-            Literal txt = new Literal();
+            uicontrols.Pane p = new uicontrols.Pane();
+
+            boxChangeKey = new TextBox
+            {
+                ID = "changeKey-" + currentItem.Id,
+                CssClass = "umbEditorTextField",
+                Text = currentItem.ItemKey
+            };
+
+            labelChangeKey = new Label
+            {
+                ID = "changeKeyLabel",
+                CssClass = "text-error"
+            };
+
+            p.addProperty(new Literal
+            {
+                Text = "<p>" + Services.TextService.Localize("dictionaryItem/changeKey") + "</p>"
+            });
+            p.addProperty(boxChangeKey);
+            p.addProperty(labelChangeKey);
+
+
+            txt = new Literal();
             txt.Text = "<p>" + Services.TextService.Localize("dictionaryItem/description", new[] { currentItem.ItemKey }) + "</p><br/>";
             p.addProperty(txt);
-			
-			foreach (cms.businesslogic.language.Language l in cms.businesslogic.language.Language.getAll)
-			{
-              
+
+            foreach (cms.businesslogic.language.Language l in cms.businesslogic.language.Language.getAll)
+            {
+
                 TextBox languageBox = new TextBox();
                 languageBox.TextMode = TextBoxMode.MultiLine;
                 languageBox.ID = l.id.ToString();
@@ -60,68 +84,92 @@ namespace umbraco.settings
                 languageFields.Add(languageBox);
                 p.addProperty(l.FriendlyName, languageBox);
 
-			}
+            }
 
-			if (!IsPostBack)
-			{
-			    var path = BuildPath(currentItem);
-				ClientTools
+
+            if (!IsPostBack)
+            {
+                var path = BuildPath(currentItem);
+                ClientTools
 					.SetActiveTreeType(Constants.Trees.Dictionary)
                     .SyncTree(path, false);
-			}
-
+            }
 
             Panel1.Controls.Add(p);
-		}
+        }
 
 	    private string BuildPath(IDictionaryItem current)
-	    {
+        {
 	        var parentPath = current.ParentId.HasValue == false ? "" : BuildPath(current) + ",";
 	        return parentPath + current.Id;
-	    }
+        }
 
         void save_Click(object sender, EventArgs e)
         {
-            foreach (TextBox t in languageFields) 
+            labelChangeKey.Text = ""; // reset error text
+            var newKey = boxChangeKey.Text;
+            var save = true;
+            if (string.IsNullOrWhiteSpace(newKey) == false && newKey != currentItem.ItemKey)
             {
-                //check for null but allow empty string!
-                // http://issues.umbraco.org/issue/U4-1931
-				if (t.Text != null)
-				{
-				    Services.LocalizationService.AddOrUpdateDictionaryValue(
-				        currentItem,
-				        Services.LocalizationService.GetLanguageById(int.Parse(t.ID)),
-				        t.Text);
-
-                    Services.LocalizationService.Save(currentItem);
-
+                if (Services.LocalizationService.DictionaryItemExists(newKey))
+                {
+                    // reject
+                    labelChangeKey.Text = Services.TextService.Localize("dictionaryItem/changeKeyError", newKey);
+                    boxChangeKey.Text = currentItem.ItemKey; // reset key
+                    save = false;
                 }
-			}
-            ClientTools.ShowSpeechBubble(SpeechBubbleIcon.Save, Services.TextService.Localize("speechBubbles/dictionaryItemSaved"), "");	
-		}
-		#region Web Form Designer generated code
-		override protected void OnInit(EventArgs e)
-		{
-			//
-			// CODEGEN: This call is required by the ASP.NET Web Form Designer.
-			//
-            /*
-			tbv.ID="tabview1";
-			tbv.Width = 400;
-			tbv.Height = 200;
-		*/
+                else
+                {
+                    // update key
+                    currentItem.ItemKey = newKey;
 
-			InitializeComponent();
-			base.OnInit(e);
-		}
-		
-		/// <summary>
-		/// Required method for Designer support - do not modify
-		/// the contents of this method with the code editor.
-		/// </summary>
-		private void InitializeComponent()
-		{    
-		}
-		#endregion
+                    // update title
+                    Panel1.title.InnerHtml = Services.TextService.Localize("editdictionary") + ": " + newKey;
+
+                    // sync the content tree
+                    var path = BuildPath(currentItem);
+                    ClientTools.SyncTree(path, true);
+                }
+            }
+
+            if (save)
+            {
+                foreach (TextBox t in languageFields)
+                {
+                    //check for null but allow empty string!
+                    // http://issues.umbraco.org/issue/U4-1931
+                    if (t.Text != null)
+                    {
+                        Services.LocalizationService.AddOrUpdateDictionaryValue(
+                            currentItem,
+                            Services.LocalizationService.GetLanguageById(int.Parse(t.ID)),
+                            t.Text);
+                    }
+                }
+
+                Services.LocalizationService.Save(currentItem);
+                ClientTools.ShowSpeechBubble(SpeechBubbleIcon.Save, Services.TextService.Localize("speechBubbles/dictionaryItemSaved"), "");
+            }
+
+            txt.Text = "<br/><p>" + Services.TextService.Localize("dictionaryItem/description", currentItem.ItemKey) + "</p><br/>";
+        }
+
+        #region Web Form Designer generated code
+
+        override protected void OnInit(EventArgs e)
+        {
+            InitializeComponent();
+            base.OnInit(e);
+        }
+
+        /// <summary>
+        /// Required method for Designer support - do not modify
+        /// the contents of this method with the code editor.
+        /// </summary>
+        private void InitializeComponent()
+        {
+        }
+
+        #endregion
 	}
 }

src/Umbraco.Web/umbraco.presentation/umbraco/users/EditUser.aspx.cs

@@ -22,6 +22,9 @@ using Umbraco.Core.Models;
 using Umbraco.Core.Models.Membership;
 using Umbraco.Web.UI;
 using Umbraco.Web.UI.Pages;
+using System.Text.RegularExpressions;
+using System.Text;
+using Umbraco.Core.Security;
 
 namespace umbraco.cms.presentation.user
 {
@@ -35,11 +38,16 @@ namespace umbraco.cms.presentation.user
             CurrentApp = Constants.Applications.Users.ToString();
         }
         protected HtmlTable macroProperties;
-        protected TextBox uname = new TextBox();
-        protected TextBox lname = new TextBox();
+        protected TextBox uname = new TextBox() { ID = "uname" };
+        protected RequiredFieldValidator unameValidator = new RequiredFieldValidator();
+        protected TextBox lname = new TextBox() { ID = "lname" };
+        protected RequiredFieldValidator lnameValidator = new RequiredFieldValidator();
+        protected CustomValidator lnameCustomValidator = new CustomValidator();
         protected PlaceHolder passw = new PlaceHolder();
         protected CheckBoxList lapps = new CheckBoxList();
-        protected TextBox email = new TextBox();
+        protected TextBox email = new TextBox() { ID = "email" };
+        protected RequiredFieldValidator emailValidator = new RequiredFieldValidator();
+        protected CustomValidator emailCustomValidator = new CustomValidator();
         protected DropDownList userType = new DropDownList();
         protected DropDownList userLanguage = new DropDownList();
         protected CheckBox NoConsole = new CheckBox();
@@ -140,8 +148,7 @@ namespace umbraco.cms.presentation.user
                 contentPicker.Value = "-1";
 
             content.Controls.Add(contentPicker);
-
-
+            
             // Add password changer
             var passwordChanger = (passwordChanger)LoadControl(SystemDirectories.Umbraco + "/controls/passwordChanger.ascx");
             passwordChanger.MembershipProviderName = UmbracoConfig.For.UmbracoSettings().Providers.DefaultBackOfficeUserProvider;
@@ -165,10 +172,20 @@ namespace umbraco.cms.presentation.user
             passw.Controls.Add(passwordChanger);
             passw.Controls.Add(validatorContainer);
 
-            pp.addProperty(Services.TextService.Localize("user/username"), uname);
-            pp.addProperty(Services.TextService.Localize("user/loginname"), lname);
+            var validationSummary = new ValidationSummary
+            {
+                ID = "validationSummary",
+                DisplayMode = ValidationSummaryDisplayMode.BulletList,
+                CssClass = "error"
+            };
+
+            pp.addProperty(validationSummary);
+
+            pp.addProperty(Services.TextService.Localize("user/username"), uname, unameValidator);
+            pp.addProperty(Services.TextService.Localize("user/loginname"), lname, lnameValidator, lnameCustomValidator);
             pp.addProperty(Services.TextService.Localize("user/password"), passw);
-            pp.addProperty(Services.TextService.Localize("email"), email);
+
+            pp.addProperty(Services.TextService.Localize("general/email"), email, emailValidator, emailCustomValidator);
             pp.addProperty(Services.TextService.Localize("user/usertype"), userType);
             pp.addProperty(Services.TextService.Localize("user/language"), userLanguage);
 
@@ -205,12 +222,56 @@ namespace umbraco.cms.presentation.user
             save.Text = Services.TextService.Localize("save");
             save.ButtonType = MenuButtonType.Primary;
 
-            sectionValidator.ServerValidate += new ServerValidateEventHandler(sectionValidator_ServerValidate);
+            sectionValidator.ServerValidate += SectionValidator_OnServerValidate;
             sectionValidator.ControlToValidate = lapps.ID;
             sectionValidator.ErrorMessage = Services.TextService.Localize("errorHandling/errorMandatoryWithoutTab", new[] { Services.TextService.Localize("user/modules") });
             sectionValidator.CssClass = "error";
             sectionValidator.Style.Add("color", "red");
 
+            unameValidator.ControlToValidate = uname.ID;
+            unameValidator.Display = ValidatorDisplay.Dynamic;
+            unameValidator.ErrorMessage = Services.TextService.Localize("defaultdialogs/requiredField");
+            unameValidator.CssClass = "error";
+            unameValidator.Style.Add("color", "red");
+            unameValidator.Style.Add("margin-left", "5px");
+            unameValidator.Style.Add("line-height", "28px");
+
+            lnameValidator.ControlToValidate = lname.ID;
+            lnameValidator.Display = ValidatorDisplay.Dynamic;
+            lnameValidator.ErrorMessage = Services.TextService.Localize("defaultdialogs/requiredField");
+            lnameValidator.CssClass = "error";
+            lnameValidator.Style.Add("color", "red");
+            lnameValidator.Style.Add("margin-left", "5px");
+            lnameValidator.Style.Add("line-height", "28px");
+
+            lnameCustomValidator.ServerValidate += LnameCustomValidator_OnServerValidate;
+            lnameCustomValidator.Display = ValidatorDisplay.Dynamic;
+            lnameCustomValidator.ControlToValidate = lname.ID;
+            var localizedLname = Services.TextService.Localize("user/loginname");
+            lnameCustomValidator.ErrorMessage = Services.TextService.Localize("errorHandling/errorExistsWithoutTab", localizedLname);
+            lnameCustomValidator.CssClass = "error";
+            lnameCustomValidator.Style.Add("color", "red");
+            lnameCustomValidator.Style.Add("margin-left", "5px");
+            lnameCustomValidator.Style.Add("line-height", "28px");
+
+            emailValidator.ControlToValidate = email.ID;
+            emailValidator.Display = ValidatorDisplay.Dynamic;
+            emailValidator.ErrorMessage = Services.TextService.Localize("defaultdialogs/requiredField");
+            emailValidator.CssClass = "error";
+            emailValidator.Style.Add("color", "red");
+            emailValidator.Style.Add("margin-left", "5px");
+            emailValidator.Style.Add("line-height", "28px");
+
+            emailCustomValidator.ServerValidate += EmailCustomValidator_OnServerValidate;
+            emailCustomValidator.Display = ValidatorDisplay.Dynamic;
+            emailCustomValidator.ControlToValidate = email.ID;
+            var localizedEmail = Services.TextService.Localize("general/email");
+            emailCustomValidator.ErrorMessage = Services.TextService.Localize("errorHandling/errorRegExpWithoutTab", localizedEmail);
+            emailCustomValidator.CssClass = "error";
+            emailCustomValidator.Style.Add("color", "red");
+            emailCustomValidator.Style.Add("margin-left", "5px");
+            emailCustomValidator.Style.Add("line-height", "28px");
+
             SetupForm();
 
             ClientTools
@@ -218,8 +279,18 @@ namespace umbraco.cms.presentation.user
                 .SyncTree(UID.ToString(), false);
         }
 
+        private void LnameCustomValidator_OnServerValidate(object source, ServerValidateEventArgs args)
+        {
+            var usersWithLoginName = Services.UserService.GetByUsername(lname.Text);
+            args.IsValid = usersWithLoginName == null || usersWithLoginName.Id == u.Id;
+        }
 
-        void sectionValidator_ServerValidate(object source, ServerValidateEventArgs args)
+        private void EmailCustomValidator_OnServerValidate(object source, ServerValidateEventArgs args)
+        {
+            args.IsValid = MembershipProviderBase.IsEmailValid(email.Text.Trim());
+        }
+
+        private void SectionValidator_OnServerValidate(object source, ServerValidateEventArgs args)
         {
             args.IsValid = false || lapps.SelectedIndex >= 0;
         }
@@ -392,7 +463,9 @@ namespace umbraco.cms.presentation.user
             }
             else
             {
-                ClientTools.ShowSpeechBubble(SpeechBubbleIcon.Error, Services.TextService.Localize("speechBubbles/editUserError"), "");
+                ClientTools.ShowSpeechBubble(SpeechBubbleIcon.Error, 
+                    Services.TextService.Localize("speechBubbles/validationFailedHeader"), 
+                    Services.TextService.Localize("speechBubbles/validationFailedMessage"));
             }
         }
 

src/umbraco.controls/pane.cs

@@ -36,6 +36,22 @@ namespace umbraco.uicontrols
             set { m_title = value; }
         }
 
+        public void addProperty(string Caption, Control C, params BaseValidator[] validators)
+        {
+
+            PropertyPanel pp = new PropertyPanel();
+            pp.Controls.Add(C);
+
+            foreach (var validator in validators)
+            {
+                validator.Display = ValidatorDisplay.Dynamic;
+                pp.Controls.Add(validator);
+            }
+            pp.Text = Caption;
+
+            this.Controls.Add(pp);
+        }
+
         public void addProperty(string Caption, Control C)
         {
 

src/umbraco.sln

@@ -105,6 +105,8 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "build", "build", "{5B03EF4E
 		..\build\NuSpecs\build\UmbracoCms.targets = ..\build\NuSpecs\build\UmbracoCms.targets
 	EndProjectSection
 EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Umbraco.Tests.Benchmarks", "Umbraco.Tests.Benchmarks\Umbraco.Tests.Benchmarks.csproj", "{86DEB346-089F-4106-89C8-D852B9CF2A33}"
+EndProject
 Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Umbraco.Compat7", "Umbraco.Compat7\Umbraco.Compat7.csproj", "{185E098F-5706-4B97-B404-EB974F05F633}"
 EndProject
 Global
@@ -151,6 +153,10 @@ Global
 		{07FBC26B-2927-4A22-8D96-D644C667FECC}.Debug|Any CPU.Build.0 = Debug|Any CPU
 		{07FBC26B-2927-4A22-8D96-D644C667FECC}.Release|Any CPU.ActiveCfg = Release|Any CPU
 		{07FBC26B-2927-4A22-8D96-D644C667FECC}.Release|Any CPU.Build.0 = Release|Any CPU
+		{86DEB346-089F-4106-89C8-D852B9CF2A33}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{86DEB346-089F-4106-89C8-D852B9CF2A33}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{86DEB346-089F-4106-89C8-D852B9CF2A33}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{86DEB346-089F-4106-89C8-D852B9CF2A33}.Release|Any CPU.Build.0 = Release|Any CPU
 		{185E098F-5706-4B97-B404-EB974F05F633}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
 		{185E098F-5706-4B97-B404-EB974F05F633}.Debug|Any CPU.Build.0 = Debug|Any CPU
 		{185E098F-5706-4B97-B404-EB974F05F633}.Release|Any CPU.ActiveCfg = Release|Any CPU
@@ -164,5 +170,6 @@ Global
 		{5D3B8245-ADA6-453F-A008-50ED04BFE770} = {B5BD12C1-A454-435E-8A46-FF4A364C0382}
 		{E3F9F378-AFE1-40A5-90BD-82833375DBFE} = {227C3B55-80E5-4E7E-A802-BE16C5128B9D}
 		{5B03EF4E-E0AC-4905-861B-8C3EC1A0D458} = {227C3B55-80E5-4E7E-A802-BE16C5128B9D}
+		{86DEB346-089F-4106-89C8-D852B9CF2A33} = {B5BD12C1-A454-435E-8A46-FF4A364C0382}
 	EndGlobalSection
 EndGlobal

src/umbraco.sln.DotSettings

@@ -16,5 +16,5 @@
 	<s:String x:Key="/Default/PatternsAndTemplates/StructuralSearch/Pattern/=2DA32DA040A7D74599ABE288C7224CF0/Comment/@EntryValue">Disposable construction</s:String>
 	<s:String x:Key="/Default/PatternsAndTemplates/StructuralSearch/Pattern/=2DA32DA040A7D74599ABE288C7224CF0/Severity/@EntryValue">HINT</s:String>
 	<s:Boolean x:Key="/Default/PatternsAndTemplates/StructuralSearch/Pattern/=37A0B37A0ABAA34AA5CB32A93653C4FE/@KeyIndexDefined">False</s:Boolean>
-	<s:String x:Key="/Default/CodeInspection/CSharpLanguageProject/LanguageLevel/@EntryValue">CSharp50</s:String></wpf:ResourceDictionary>
+	<s:String x:Key="/Default/CodeInspection/CSharpLanguageProject/LanguageLevel/@EntryValue">CSharp60</s:String></wpf:ResourceDictionary>
 </wpf:ResourceDictionary>