Merge pull request #1843 from umbraco/temp-U4-9691

Fixes the Examine mgmt dashboard to look better and to detect if an i…
Merge pull request #1839 from umbraco/temp-uaasscrum-827
2017-04-03 16:28:36 +10:00 · 2017-04-03 16:19:27 +10:00 · 2017-04-03 16:03:13 +10:00 · 2017-04-03 15:53:30 +10:00 · 2017-03-30 10:31:51 +02:00 · 2017-03-30 09:32:22 +02:00
514 changed files with 22576 additions and 9581 deletions
@@ -141,3 +141,4 @@ build/docs.zip
 build/ui-docs.zip
 build/csharp-docs.zip
 build/msbuild.log
+.vs/
@@ -1,42 +1,52 @@
 Umbraco CMS
 ===========
-Umbraco is a free open source Content Management System built on the ASP.NET platform.
+The friendliest, most flexible and fastest growing ASP.NET CMS used by more than 390,000 websites worldwide: [https://umbraco.com](https://umbraco.com)
+
+[![ScreenShot](vimeo.png)](https://vimeo.com/172382998/)
+
+## Umbraco CMS ##
+Umbraco is a free open source Content Management System built on the ASP.NET platform. Our mission is to help you deliver delightful digital experiences by making Umbraco friendly, simpler and social.
+

 ## Building Umbraco from source ##
+
 The easiest way to get started is to run `build/build.bat` which will build both the backoffice (also known as "Belle") and the Umbraco core. You can then easily start debugging from Visual Studio, or if you need to debug Belle you can run `grunt vs` in `src\Umbraco.Web.UI.Client`.
 
 If you're interested in making changes to Belle without running Visual Studio make sure to read the [Belle ReadMe file](src/Umbraco.Web.UI.Client/README.md). 

 Note that you can always [download a nightly build](http://nightly.umbraco.org/?container=umbraco-750) so you don't have to build the code yourself.

-## Watch a introduction video ##
+## Watch an introduction video ##

 [![ScreenShot](http://umbraco.com/images/whatisumbraco.png)](https://umbraco.tv/videos/umbraco-v7/content-editor/basics/introduction/cms-explanation/)

-## Umbraco - the simple, flexible and friendly ASP.NET CMS ##
+## Umbraco - The Friendly CMS ##

-**More than 350,000 sites trust Umbraco** 
+For the first time on the Microsoft platform, there is a free user and developer friendly CMS that makes it quick and easy to create websites - or a breeze to build complex web applications. Umbraco has award-winning integration capabilities and supports ASP.NET MVC or Web Forms, including User and Custom Controls, out of the box. 

-For the first time on the Microsoft platform, there is a free user and developer friendly CMS that makes it quick and easy to create websites - or a breeze to build complex web applications. Umbraco has award-winning integration capabilities and supports ASP.NET MVC or Web Forms, including User and Custom Controls, out of the box. It's a developer's dream and your users will love it too. 
+Umbraco is not only loved by developers, but is a content editors dream. Enjoy intuitive editing tools, media management, responsive views and approval workflows to send your content live.

-Used by more than 350,000 active websites including [http://daviscup.com](http://daviscup.com), [http://heinz.com](http://heinz.com), [http://peugeot.com](http://peugeot.com), [http://www.hersheys.com/](http://www.hersheys.com/) and **The Official ASP.NET and IIS.NET website from Microsoft** ([http://asp.net](http://asp.net) / [http://iis.net](http://iis.net)), you can be sure that the technology is proven, stable and scales.  
+Used by more than 350,000 active websites including Carlsberg, Segway, Amazon and Heinz and **The Official ASP.NET and IIS.NET website from Microsoft** ([https://asp.net](https://asp.net) / [https://iis.net](https://iis.net)), you can be sure that the technology is proven, stable and scales. Backed by the team at Umbraco HQ, and supported by a dedicated community of over 200,000 craftspeople globally, you can trust that Umbraco is a safe choice and is here to stay. 

-To view more examples, please visit [http://umbraco.com/why-umbraco/#caseStudies](http://umbraco.com/why-umbraco/#caseStudies)
+To view more examples, please visit [https://umbraco.com/why-umbraco/#caseStudies](https://umbraco.com/why-umbraco/#caseStudies)
+
+## Why Open Source? ##
+As an Open Source platform, Umbraco is more than just a CMS. We are transparent with our roadmap for future versions, our incremental sprint planning notes are publicly accessible and community contributions and packages are available for all to use.

 ## Downloading ##

-The downloadable Umbraco releases live at [http://our.umbraco.org/download](http://our.umbraco.org/download).
+The downloadable Umbraco releases live at [https://our.umbraco.org/download](https://our.umbraco.org/download).

 ## Forums ##

-We have a forum running on [http://our.umbraco.org](http://our.umbraco.org). The discussions group on [Google Groups](https://groups.google.com/forum/#!forum/umbraco-dev) is for discussions on developing the core, and not on Umbraco-implementations or extensions in general. For those topics, please use [http://our.umbraco.org](http://our.umbraco.org).
+Peer-to-peer support is available 24/7 at the community forum on [https://our.umbraco.org](https://our.umbraco.org). 

 ## Contribute to Umbraco ##

-If you want to contribute back to Umbraco you should check out our [guide to contributing](http://our.umbraco.org/contribute).
+Umbraco is contribution focused and community driven. If you want to contribute back to Umbraco please check out our [guide to contributing](https://our.umbraco.org/contribute).

 ## Found a bug? ##

-Another way you can contribute to Umbraco is by providing issue reports. For information on how to submit an issue report refer to our [online guide for reporting issues](http://our.umbraco.org/contribute/report-an-issue-or-request-a-feature).
+Another way you can contribute to Umbraco is by providing issue reports. For information on how to submit an issue report refer to our [online guide for reporting issues](https://our.umbraco.org/contribute/report-an-issue-or-request-a-feature).

 To view existing issues, please visit [http://issues.umbraco.org](http://issues.umbraco.org).
@@ -27,18 +27,18 @@
      <dependency id="SharpZipLib" version="[0.86.0, 1.0.0)" />
      <dependency id="MySql.Data" version="[6.9.8, 7.0.0)" />
      <dependency id="xmlrpcnet" version="[2.5.0, 3.0.0)" />
-      <dependency id="ClientDependency" version="[1.9.1, 2.0.0)" />
-      <dependency id="ClientDependency-Mvc5" version="[1.8.0, 2.0.0)" />
+      <dependency id="ClientDependency" version="[1.9.2, 2.0.0)" />
+      <dependency id="ClientDependency-Mvc5" version="[1.8.0.0, 2.0.0)" />
 	  <!-- AutoMapper can not be updated due to: https://github.com/AutoMapper/AutoMapper/issues/373#issuecomment-127644405 -->
-      <dependency id="AutoMapper" version="[3.0.0, 3.1.0)" />
+      <dependency id="AutoMapper" version="[3.3.1, 4.0.0)" />
      <dependency id="Newtonsoft.Json" version="[6.0.8, 10.0.0)" />
-      <dependency id="Examine" version="[0.1.69, 1.0.0)" />
-      <dependency id="ImageProcessor" version="[2.4.4, 3.0.0)" />
-      <dependency id="ImageProcessor.Web" version="[4.6.4, 5.0.0)" />
+      <dependency id="Examine" version="[0.1.82, 1.0.0)" />
+      <dependency id="ImageProcessor" version="[2.5.2, 3.0.0)" />
+      <dependency id="ImageProcessor.Web" version="[4.8.2, 5.0.0)" />
      <dependency id="semver" version="[1.1.2, 2.0.0)" />
      <dependency id="UrlRewritingNet" version="[2.0.7, 3.0.0)" />	  
 	  <!-- Markdown can not be updated due to: https://github.com/hey-red/markdownsharp/issues/71#issuecomment-233585487 -->
-      <dependency id="Markdown" version="[1.14.4, 2.0.0)" />	  
+      <dependency id="Markdown" version="[1.14.7, 2.0.0)" />	  
    </dependencies>
  </metadata>
  <files>
@@ -103,4 +103,4 @@
    <file src="..\_BuildOutput\bin\UmbracoExamine.pdb" target="lib" />
    <file src="..\..\src\UmbracoExamine\**\*.cs" exclude="..\..\src\**\TemporaryGeneratedFile*.cs" target="src\UmbracoExamine" />
  </files>
-</package>
+</package>
@@ -17,7 +17,8 @@
    <dependencies>
      <dependency id="UmbracoCms.Core" version="[$version$]" />
      <dependency id="Newtonsoft.Json" version="[6.0.8, 10.0.0)" />
-      <dependency id="Umbraco.ModelsBuilder" version="[3.0.4, 4.0.0)" />
+      <dependency id="Umbraco.ModelsBuilder" version="[3.0.6, 4.0.0)" />
+      <dependency id="ImageProcessor.Web.Config" version="[2.3.0, 3.0.0)" />
    </dependencies>
  </metadata>
  <files>
@@ -43,6 +44,8 @@
    <file src="tools\trees.config.install.xdt" target="Content\config\trees.config.install.xdt" />
    <file src="tools\umbracoSettings.config.install.xdt" target="Content\config\umbracoSettings.config.install.xdt" />
    <file src="tools\Views.Web.config.install.xdt" target="Views\Web.config.install.xdt" />
+    <file src="tools\processing.config.install.xdt" target="Content\Config\imageprocessor\processing.config.install.xdt" />
+    <file src="tools\cache.config.install.xdt" target="Content\Config\imageprocessor\cache.config.install.xdt" />
    <file src="build\**" target="build" />
  </files>
 </package>
@@ -41,16 +41,6 @@
        views/dashboard/developer/examinemanagement.html
      </control>
    </tab>
-    <tab caption="Health Check" xdt:Transform="InsertIfMissing" xdt:Locator="Match(caption)">
-      <control>
-        views/dashboard/developer/healthcheck.html
-      </control>
-    </tab>
-    <tab caption="Redirect URL Management" xdt:Transform="InsertIfMissing" xdt:Locator="Match(caption)">
-      <control>
-        views/dashboard/developer/redirecturls.html
-      </control>
-    </tab>	
  </section>
  
  <section alias="StartupMediaDashboardSection" xdt:Locator="Match(alias)" xdt:Transform="InsertIfMissing">
@@ -80,4 +70,26 @@
 	<tab caption="Change Password" xdt:Locator="Match(caption)" xdt:Transform="Remove" />
 	<tab caption="Last Edits" xdt:Locator="Match(caption)" xdt:Transform="Remove" />
  </section>
+
+  <section alias="RedirectUrlManagement" xdt:Locator="Match(alias)" xdt:Transform="InsertIfMissing">
+    <areas>
+      <area>content</area>
+    </areas>
+    <tab caption="Redirect URL Management">
+      <control>
+        views/dashboard/developer/redirecturls.html
+      </control>
+    </tab>
+  </section>
+
+  <section alias="UmbracoHealthCheck" xdt:Locator="Match(alias)" xdt:Transform="InsertIfMissing">
+    <areas>
+      <area>developer</area>
+    </areas>
+    <tab caption="Health Check">
+      <control>
+        views/dashboard/developer/healthcheck.html
+      </control>
+    </tab>
+  </section>
 </dashBoard>
@@ -319,6 +319,7 @@
  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <dependentAssembly xdt:Locator="Condition(./_defaultNamespace:assemblyIdentity/@name='HtmlAgilityPack')" xdt:Transform="Remove" />
+      <dependentAssembly xdt:Locator="Condition(./_defaultNamespace:assemblyIdentity/@name='AutoMapper')" xdt:Transform="Remove" />
      <dependentAssembly xdt:Locator="Condition(./_defaultNamespace:assemblyIdentity/@name='System.Net.Http')" xdt:Transform="Remove" />
      <dependentAssembly xdt:Locator="Condition(./_defaultNamespace:assemblyIdentity/@name='Newtonsoft.Json')" xdt:Transform="Remove" />
      <dependentAssembly xdt:Locator="Condition(./_defaultNamespace:assemblyIdentity/@name='System.Web.Mvc')" xdt:Transform="Remove" />
@@ -334,10 +335,14 @@

  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
-      <dependentAssembly>
+      <dependentAssembly xdt:Transform="Insert">
        <assemblyIdentity name="HtmlAgilityPack" publicKeyToken="bd319b19eaf3b43a" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-1.4.9.0" newVersion="1.4.9.0" />
      </dependentAssembly>
+      <dependentAssembly xdt:Transform="Insert">
+        <assemblyIdentity name="AutoMapper" publicKeyToken="be96cd2c38ef1005" culture="neutral" />
+        <bindingRedirect oldVersion="0.0.0.0-3.3.1.0" newVersion="3.3.1.0" />
+      </dependentAssembly>
      <dependentAssembly xdt:Transform="Insert">
        <assemblyIdentity name="System.Net.Http" publicKeyToken="b03f5f7f11d50a3a" culture="neutral"/>
        <bindingRedirect oldVersion="0.0.0.0-4.0.0.0" newVersion="4.0.0.0"/>
@@ -0,0 +1,6 @@
+<?xml version="1.0"?>
+<caching xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
+  <caches>
+    <cache name="DiskCache" trimCache="false" xdt:Transform="SetAttributes(trimCache)" xdt:Locator="Match(name)"  />
+  </caches>
+</caching>
@@ -71,6 +71,7 @@ if ($project) {
 		if(Test-Path $umbracoBinFolder\ImageProcessor.dll) { Remove-Item $umbracoBinFolder\ImageProcessor.dll -Force -Confirm:$false }
 		if(Test-Path $umbracoBinFolder\ImageProcessor.Web.dll) { Remove-Item $umbracoBinFolder\ImageProcessor.Web.dll -Force -Confirm:$false }
 		if(Test-Path $umbracoBinFolder\Lucene.Net.dll) { Remove-Item $umbracoBinFolder\Lucene.Net.dll -Force -Confirm:$false }
+		if(Test-Path $umbracoBinFolder\Microsoft.IO.RecyclableMemoryStream.dll) { Remove-Item $umbracoBinFolder\Microsoft.IO.RecyclableMemoryStream.dll -Force -Confirm:$false }
 		if(Test-Path $umbracoBinFolder\Microsoft.AspNet.Identity.Core.dll) { Remove-Item $umbracoBinFolder\Microsoft.AspNet.Identity.Core.dll -Force -Confirm:$false }
 		if(Test-Path $umbracoBinFolder\Microsoft.AspNet.Identity.Owin.dll) { Remove-Item $umbracoBinFolder\Microsoft.AspNet.Identity.Owin.dll -Force -Confirm:$false }
 		if(Test-Path $umbracoBinFolder\Microsoft.CodeAnalysis.CSharp.dll) { Remove-Item $umbracoBinFolder\Microsoft.CodeAnalysis.CSharp.dll -Force -Confirm:$false }
@@ -0,0 +1 @@
+<processing xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform" xdt:Transform="SetAttributes(preserveExifMetaData)" preserveExifMetaData="true" />
@@ -1,2 +1,2 @@
 # Usage: on line 2 put the release version, on line 3 put the version comment (example: beta)
-7.5.3
+7.5.12
@@ -49,13 +49,9 @@
    <Reference Include="System.Core" />
    <Reference Include="System.Data.SqlServerCe, Version=4.0.0.1, Culture=neutral, PublicKeyToken=89845dcd8080cc91, processorArchitecture=MSIL">
      <HintPath>..\packages\SqlServerCE.4.0.0.1\lib\System.Data.SqlServerCe.dll</HintPath>
-      <SpecificVersion>False</SpecificVersion>
-      <Private>False</Private>
    </Reference>
    <Reference Include="System.Data.SqlServerCe.Entity, Version=4.0.0.1, Culture=neutral, PublicKeyToken=89845dcd8080cc91, processorArchitecture=MSIL">
      <HintPath>..\packages\SqlServerCE.4.0.0.1\lib\System.Data.SqlServerCe.Entity.dll</HintPath>
-      <SpecificVersion>False</SpecificVersion>
-      <Private>False</Private>
    </Reference>
    <Reference Include="System.Xml.Linq" />
    <Reference Include="System.Data.DataSetExtensions" />
@@ -21,15 +21,15 @@ namespace SqlCE4Umbraco
    public class SqlCEInstaller : DefaultInstallerUtility<SqlCEHelper>
    {
        #region Private Constants
-       
+
        /// <summary>The latest database version this installer supports.</summary>
        private const DatabaseVersion LatestVersionSupported = DatabaseVersion.Version4_8;

        /// <summary>The specifications to determine the database version.</summary>
        private static readonly VersionSpecs[] m_VersionSpecs = new VersionSpecs[] {
-					new VersionSpecs("SELECT CONSTRAINT_NAME FROM INFORMATION_SCHEMA.REFERENTIAL_CONSTRAINTS LEFT OUTER JOIN umbracoApp ON appAlias = appAlias WHERE CONSTRAINT_NAME = 'FK_umbracoUser2app_umbracoApp'", 0, DatabaseVersion.Version4_8), 
-					new VersionSpecs("SELECT id FROM umbracoNode WHERE id = -21", 1, DatabaseVersion.Version4_1),        
-					new VersionSpecs("SELECT action FROM umbracoAppTree",DatabaseVersion.Version4),
+                    new VersionSpecs("SELECT CONSTRAINT_NAME FROM INFORMATION_SCHEMA.REFERENTIAL_CONSTRAINTS LEFT OUTER JOIN umbracoApp ON appAlias = appAlias WHERE CONSTRAINT_NAME = 'FK_umbracoUser2app_umbracoApp'", 0, DatabaseVersion.Version4_8),
+                    new VersionSpecs("SELECT id FROM umbracoNode WHERE id = -21", 1, DatabaseVersion.Version4_1),
+                    new VersionSpecs("SELECT action FROM umbracoAppTree",DatabaseVersion.Version4),
                    new VersionSpecs("SELECT description FROM cmsContentType",DatabaseVersion.Version3),
                    new VersionSpecs("SELECT id FROM sysobjects",DatabaseVersion.None) };

@@ -43,8 +43,9 @@ namespace SqlCE4Umbraco
        public override bool CanConnect
        {
            get
-            {                
-                SqlHelper.CreateEmptyDatabase();
+            {
+                using (var sqlHelper = SqlHelper)
+                    sqlHelper.CreateEmptyDatabase();
                return base.CanConnect;
            }
        }
@@ -93,22 +94,22 @@ namespace SqlCE4Umbraco

        #region DefaultInstaller Members       

-		/// <summary>
-		/// Returns the sql to do a full install
-		/// </summary>
-		protected override string FullInstallSql
-		{
-			get { return string.Empty; }
-		}
+        /// <summary>
+        /// Returns the sql to do a full install
+        /// </summary>
+        protected override string FullInstallSql
+        {
+            get { return string.Empty; }
+        }


-		/// <summary>
-		/// Returns the sql to do an upgrade
-		/// </summary>
-		protected override string UpgradeSql
-		{
-			get { return string.Empty; }
-		}
+        /// <summary>
+        /// Returns the sql to do an upgrade
+        /// </summary>
+        protected override string UpgradeSql
+        {
+            get { return string.Empty; }
+        }

        // We need to override this as the default way of detection a db connection checks for systables that doesn't exist
        // in a CE db
@@ -123,8 +124,9 @@ namespace SqlCE4Umbraco
            // verify connection
            try
            {
-                if (SqlCeApplicationBlock.VerifyConnection(base.SqlHelper.ConnectionString))
-                    return DatabaseVersion.None;
+                using (var sqlHelper = SqlHelper)
+                    if (SqlCeApplicationBlock.VerifyConnection(sqlHelper.ConnectionString))
+                        return DatabaseVersion.None;
            }
            catch (Exception e)
            {
@@ -37,13 +37,15 @@ namespace SqlCE4Umbraco
            ITable table = null;

            // get name in correct casing
-            name = SqlHelper.ExecuteScalar<string>("SELECT name FROM sys.tables WHERE name=@name",
-                                                          SqlHelper.CreateParameter("name", name));
+            using (var sqlHelper = SqlHelper)
+                name = sqlHelper.ExecuteScalar<string>("SELECT name FROM sys.tables WHERE name=@name",
+                                                          sqlHelper.CreateParameter("name", name));
            if (name != null)
            {
                table = new DefaultTable(name);

-                using (IRecordsReader reader = SqlHelper.ExecuteReader(
+                using (var sqlHelper = SqlHelper)
+                using (IRecordsReader reader = sqlHelper.ExecuteReader(
                        @"SELECT c.name AS Name, st.name AS DataType, c.max_length, c.is_nullable, c.is_identity
                          FROM sys.tables AS t
                            JOIN sys.columns AS c ON t.object_id = c.object_id
@@ -51,7 +53,7 @@ namespace SqlCE4Umbraco
                            JOIN sys.types AS ty ON ty.user_type_id = c.user_type_id
                            JOIN sys.types st ON ty.system_type_id = st.user_type_id
                          WHERE t.name = @name
-                          ORDER BY c.column_id", SqlHelper.CreateParameter("name", name)))
+                          ORDER BY c.column_id", sqlHelper.CreateParameter("name", name)))
                {
                    while (reader.Read())
                    {
@@ -112,7 +114,8 @@ namespace SqlCE4Umbraco

            // create query
            StringBuilder createTableQuery = new StringBuilder();
-            createTableQuery.AppendFormat("CREATE TABLE [{0}] (", SqlHelper.EscapeString(table.Name));
+            using (var sqlHelper = SqlHelper)
+                createTableQuery.AppendFormat("CREATE TABLE [{0}] (", sqlHelper.EscapeString(table.Name));

            // add fields
            while (hasNext)
@@ -136,7 +139,8 @@ namespace SqlCE4Umbraco
            // execute query
            try
            {
-                SqlHelper.ExecuteNonQuery(createTableQuery.ToString());
+                using (var sqlHelper = SqlHelper)
+                    sqlHelper.ExecuteNonQuery(createTableQuery.ToString());
            }
            catch (Exception executeException)
            {
@@ -154,13 +158,15 @@ namespace SqlCE4Umbraco
            Debug.Assert(table != null && field != null);

            StringBuilder addColumnQuery = new StringBuilder();
-            addColumnQuery.AppendFormat("ALTER TABLE [{0}] ADD [{1}] {2}",
-                                        SqlHelper.EscapeString(table.Name),
-                                        SqlHelper.EscapeString(field.Name),
-                                        SqlHelper.EscapeString(GetDatabaseType(field)));
+            using (var sqlHelper = SqlHelper)
+                addColumnQuery.AppendFormat("ALTER TABLE [{0}] ADD [{1}] {2}",
+                                        sqlHelper.EscapeString(table.Name),
+                                        sqlHelper.EscapeString(field.Name),
+                                        sqlHelper.EscapeString(GetDatabaseType(field)));
            try
            {
-                SqlHelper.ExecuteNonQuery(addColumnQuery.ToString());
+                using (var sqlHelper = SqlHelper)
+                    sqlHelper.ExecuteNonQuery(addColumnQuery.ToString());
            }
            catch (Exception executeException)
            {
@@ -2,7 +2,7 @@
 using System.Resources;

 [assembly: AssemblyCompany("Umbraco")]
-[assembly: AssemblyCopyright("Copyright © Umbraco 2016")]
+[assembly: AssemblyCopyright("Copyright © Umbraco 2017")]
 [assembly: AssemblyTrademark("")]
 [assembly: AssemblyCulture("")]

@@ -11,5 +11,5 @@ using System.Resources;

 [assembly: AssemblyVersion("1.0.*")]

-[assembly: AssemblyFileVersion("7.5.3")]
-[assembly: AssemblyInformationalVersion("7.5.3")]
+[assembly: AssemblyFileVersion("7.5.12")]
+[assembly: AssemblyInformationalVersion("7.5.12")]
@@ -64,7 +64,8 @@ namespace Umbraco.Core.Cache
        [UmbracoWillObsolete("This cache key is only used for legacy business logic caching, remove in v8")]
        public const string ContentTypePropertiesCacheKey = "ContentType_PropertyTypes_Content:";
        
-        [UmbracoWillObsolete("This cache key is only used for legacy business logic caching, remove in v8")]
+        [Obsolete("No longer used and will be removed in v8")]
+        [EditorBrowsable(EditorBrowsableState.Never)]
        public const string PropertyTypeCacheKey = "UmbracoPropertyTypeCache";

        [Obsolete("This is no longer used and will be removed from the codebase in the future")]
@@ -26,6 +26,9 @@ namespace Umbraco.Core.Cache

        public DeepCloneRuntimeCacheProvider(IRuntimeCacheProvider innerProvider)
        {
+            if (innerProvider.GetType() == typeof(DeepCloneRuntimeCacheProvider))
+                throw new InvalidOperationException("A " + typeof(DeepCloneRuntimeCacheProvider) + " cannot wrap another instance of " + typeof(DeepCloneRuntimeCacheProvider));
+
            InnerProvider = innerProvider;
        }

@@ -105,9 +108,11 @@ namespace Umbraco.Core.Cache
                var value = result.Value; // force evaluation now - this may throw if cacheItem throws, and then nothing goes into cache
                if (value == null) return null; // do not store null values (backward compat)

+                //Clone/reset to go into the cache
                return CheckCloneableAndTracksChanges(value);
            }, timeout, isSliding, priority, removedCallback, dependentFiles);

+            //Clone/reset to go out of the cache
            return CheckCloneableAndTracksChanges(cached);
        }

@@ -0,0 +1,9 @@
+namespace Umbraco.Core.Configuration
+{
+    internal enum ContentXmlStorage
+    {
+        Default,
+        AspNetTemp,
+        EnvironmentTemp
+    }
+}
@@ -0,0 +1,27 @@
+using System;
+
+namespace Umbraco.Core.Configuration
+{
+    internal static class CoreDebugExtensions
+    {
+        private static CoreDebug _coreDebug;
+
+        public static CoreDebug CoreDebug(this UmbracoConfig config)
+        {
+            return _coreDebug ?? (_coreDebug = new CoreDebug());
+        }
+    }
+
+    internal class CoreDebug
+    {
+        public CoreDebug()
+        {
+            var appSettings = System.Configuration.ConfigurationManager.AppSettings;
+            DumpOnTimeoutThreadAbort = string.Equals("true", appSettings["Umbraco.CoreDebug.DumpOnTimeoutThreadAbort"], StringComparison.OrdinalIgnoreCase);
+        }
+
+        // when true, the Logger creates a minidump of w3wp in ~/App_Data/MiniDump whenever it logs
+        // an error due to a ThreadAbortException that is due to a timeout.
+        public bool DumpOnTimeoutThreadAbort { get; private set; }
+    }
+}
@@ -520,12 +520,25 @@ namespace Umbraco.Core.Configuration
        }

        internal static bool ContentCacheXmlStoredInCodeGen
+        {
+            get { return ContentCacheXmlStorageLocation == ContentXmlStorage.AspNetTemp; }
+        }
+
+        internal static ContentXmlStorage ContentCacheXmlStorageLocation
        {
            get
            {
-                //defaults to false
-                return ConfigurationManager.AppSettings.ContainsKey("umbracoContentXMLUseLocalTemp") 
-                    && bool.Parse(ConfigurationManager.AppSettings["umbracoContentXMLUseLocalTemp"]); //default to false
+                if (ConfigurationManager.AppSettings.ContainsKey("umbracoContentXMLStorage"))
+                {
+                    return Enum<ContentXmlStorage>.Parse(ConfigurationManager.AppSettings["umbracoContentXMLStorage"]);
+                }
+                if (ConfigurationManager.AppSettings.ContainsKey("umbracoContentXMLUseLocalTemp"))
+                {
+                    return bool.Parse(ConfigurationManager.AppSettings["umbracoContentXMLUseLocalTemp"]) 
+                        ? ContentXmlStorage.AspNetTemp 
+                        : ContentXmlStorage.Default;
+                }
+                return ContentXmlStorage.Default;
            }
        }

@@ -4,8 +4,7 @@ using System.Configuration;

 namespace Umbraco.Core.Configuration.UmbracoSettings
 {
-
-    internal class ContentElement : ConfigurationElement, IContentSection
+    internal class ContentElement : UmbracoConfigurationElement, IContentSection
    {
        [ConfigurationProperty("imaging")]
        internal ContentImagingElement Imaging
@@ -22,25 +21,13 @@ namespace Umbraco.Core.Configuration.UmbracoSettings
        [ConfigurationProperty("ResolveUrlsFromTextString")]
        internal InnerTextConfigurationElement<bool> ResolveUrlsFromTextString
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                       (InnerTextConfigurationElement<bool>)this["ResolveUrlsFromTextString"],
-                        //set the default
-                       false);
-            }
+            get { return GetOptionalTextElement<bool>("ResolveUrlsFromTextString", false); }
        }

        [ConfigurationProperty("UploadAllowDirectories")]
        internal InnerTextConfigurationElement<bool> UploadAllowDirectories
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                       (InnerTextConfigurationElement<bool>)this["UploadAllowDirectories"],
-                    //set the default
-                       true);
-            }
+            get { return GetOptionalTextElement("UploadAllowDirectories", true); }
        }

        public IEnumerable<IContentErrorPage> Error404Collection
@@ -63,121 +50,61 @@ namespace Umbraco.Core.Configuration.UmbracoSettings
        [ConfigurationProperty("ensureUniqueNaming")]
        internal InnerTextConfigurationElement<bool> EnsureUniqueNaming
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                      (InnerTextConfigurationElement<bool>)this["ensureUniqueNaming"],
-                    //set the default
-                      true);
-            }
+            get { return GetOptionalTextElement("ensureUniqueNaming", true); }
        }

        [ConfigurationProperty("TidyEditorContent")]
        internal InnerTextConfigurationElement<bool> TidyEditorContent
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                      (InnerTextConfigurationElement<bool>)this["TidyEditorContent"],
-                    //set the default
-                      false);
-            }
+            get { return GetOptionalTextElement("TidyEditorContent", false); }
        }

        [ConfigurationProperty("TidyCharEncoding")]
        internal InnerTextConfigurationElement<string> TidyCharEncoding
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<string>(
-                          (InnerTextConfigurationElement<string>)this["TidyCharEncoding"],
-                            //set the default
-                          "UTF8");
-            }
+            get { return GetOptionalTextElement("TidyCharEncoding", "UTF8"); }
        }

        [ConfigurationProperty("XmlCacheEnabled")]
        internal InnerTextConfigurationElement<bool> XmlCacheEnabled
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                          (InnerTextConfigurationElement<bool>)this["XmlCacheEnabled"],
-                    //set the default
-                          true);
-            }
+            get { return GetOptionalTextElement("XmlCacheEnabled", true); }
        }

        [ConfigurationProperty("ContinouslyUpdateXmlDiskCache")]
        internal InnerTextConfigurationElement<bool> ContinouslyUpdateXmlDiskCache
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                          (InnerTextConfigurationElement<bool>)this["ContinouslyUpdateXmlDiskCache"],
-                    //set the default
-                          true);                
-            }
+            get { return GetOptionalTextElement("ContinouslyUpdateXmlDiskCache", true); }
        }

        [ConfigurationProperty("XmlContentCheckForDiskChanges")]
        internal InnerTextConfigurationElement<bool> XmlContentCheckForDiskChanges
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                          (InnerTextConfigurationElement<bool>)this["XmlContentCheckForDiskChanges"],
-                    //set the default
-                          false); 
-            }
+            get { return GetOptionalTextElement("XmlContentCheckForDiskChanges", false); }
        }

        [ConfigurationProperty("EnableSplashWhileLoading")]
        internal InnerTextConfigurationElement<bool> EnableSplashWhileLoading
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                          (InnerTextConfigurationElement<bool>)this["EnableSplashWhileLoading"],
-                    //set the default
-                          false); 
-            }
+            get { return GetOptionalTextElement("EnableSplashWhileLoading", false); }
        }

        [ConfigurationProperty("PropertyContextHelpOption")]
        internal InnerTextConfigurationElement<string> PropertyContextHelpOption
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<string>(
-                          (InnerTextConfigurationElement<string>)this["PropertyContextHelpOption"],
-                    //set the default
-                          "text"); 
-            }
+            get { return GetOptionalTextElement("PropertyContextHelpOption", "text"); }
        }

        [ConfigurationProperty("UseLegacyXmlSchema")]
        internal InnerTextConfigurationElement<bool> UseLegacyXmlSchema
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                          (InnerTextConfigurationElement<bool>)this["UseLegacyXmlSchema"],
-                    //set the default
-                          false); 
-            }
+            get { return GetOptionalTextElement("UseLegacyXmlSchema", false); }
        }
        
        [ConfigurationProperty("ForceSafeAliases")]
        internal InnerTextConfigurationElement<bool> ForceSafeAliases
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                          (InnerTextConfigurationElement<bool>)this["ForceSafeAliases"],
-                    //set the default
-                          true); 
-            }
+            get { return GetOptionalTextElement("ForceSafeAliases", true); }
        }

        [ConfigurationProperty("PreviewBadge")]
@@ -185,123 +112,63 @@ namespace Umbraco.Core.Configuration.UmbracoSettings
        {
            get
            {
-                return new OptionalInnerTextConfigurationElement<string>(
-                          (InnerTextConfigurationElement<string>)this["PreviewBadge"],
-                    //set the default
-                          @"<a id=""umbracoPreviewBadge"" style=""position: absolute; top: 0; right: 0; border: 0; width: 149px; height: 149px; background: url('{1}/preview/previewModeBadge.png') no-repeat;"" href=""{0}/endPreview.aspx?redir={2}""><span style=""display:none;"">In Preview Mode - click to end</span></a>"); 
+                return GetOptionalTextElement("PreviewBadge", @"<a id=""umbracoPreviewBadge"" style=""position: absolute; top: 0; right: 0; border: 0; width: 149px; height: 149px; background: url('{1}/preview/previewModeBadge.png') no-repeat;"" href=""{0}/endPreview.aspx?redir={2}""><span style=""display:none;"">In Preview Mode - click to end</span></a>");                
            }
        }

        [ConfigurationProperty("UmbracoLibraryCacheDuration")]
        internal InnerTextConfigurationElement<int> UmbracoLibraryCacheDuration
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<int>(
-                          (InnerTextConfigurationElement<int>)this["UmbracoLibraryCacheDuration"],
-                    //set the default
-                          1800); 
-                
-            }
+            get { return GetOptionalTextElement("UmbracoLibraryCacheDuration", 1800); }
        }

        [ConfigurationProperty("MacroErrors")]
        internal InnerTextConfigurationElement<MacroErrorBehaviour> MacroErrors
        {
-            get
-            {
-
-                return new OptionalInnerTextConfigurationElement<MacroErrorBehaviour>(
-                          (InnerTextConfigurationElement<MacroErrorBehaviour>)this["MacroErrors"],
-                    //set the default
-                          MacroErrorBehaviour.Inline); 
-            }
+            get { return GetOptionalTextElement("MacroErrors", MacroErrorBehaviour.Inline); }
        }

        [Obsolete("This is here so that if this config element exists we won't get a YSOD, it is not used whatsoever and will be removed in future versions")]
        [ConfigurationProperty("DocumentTypeIconList")]
        internal InnerTextConfigurationElement<IconPickerBehaviour> DocumentTypeIconList
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<IconPickerBehaviour>(
-                          (InnerTextConfigurationElement<IconPickerBehaviour>)this["DocumentTypeIconList"],
-                    //set the default
-                          IconPickerBehaviour.HideFileDuplicates);
-            }
+            get { return GetOptionalTextElement("DocumentTypeIconList", IconPickerBehaviour.HideFileDuplicates); }
        }

        [ConfigurationProperty("disallowedUploadFiles")]
        internal CommaDelimitedConfigurationElement DisallowedUploadFiles
        {
-            get
-            {
-                return new OptionalCommaDelimitedConfigurationElement(
-                       (CommaDelimitedConfigurationElement)this["disallowedUploadFiles"],
-                    //set the default
-                       new[] { "ashx", "aspx", "ascx", "config", "cshtml", "vbhtml", "asmx", "air", "axd" });
-
-            }
+            get { return GetOptionalDelimitedElement("disallowedUploadFiles", new[] {"ashx", "aspx", "ascx", "config", "cshtml", "vbhtml", "asmx", "air", "axd"}); }
        }

        [ConfigurationProperty("cloneXmlContent")]
        internal InnerTextConfigurationElement<bool> CloneXmlContent
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                    (InnerTextConfigurationElement<bool>)this["cloneXmlContent"],
-                    //set the default
-                    true);
-            }
+            get { return GetOptionalTextElement("cloneXmlContent", true); }
        }

        [ConfigurationProperty("GlobalPreviewStorageEnabled")]
        internal InnerTextConfigurationElement<bool> GlobalPreviewStorageEnabled
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                    (InnerTextConfigurationElement<bool>)this["GlobalPreviewStorageEnabled"],
-                    //set the default
-                    false);
-            }
+            get { return GetOptionalTextElement("GlobalPreviewStorageEnabled", false); }
        }

        [ConfigurationProperty("defaultDocumentTypeProperty")]
        internal InnerTextConfigurationElement<string> DefaultDocumentTypeProperty
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<string>(
-                    (InnerTextConfigurationElement<string>)this["defaultDocumentTypeProperty"],
-                    //set the default
-                    "Textstring");
-            }
+            get { return GetOptionalTextElement("defaultDocumentTypeProperty", "Textstring"); }
        }

        [ConfigurationProperty("EnableInheritedDocumentTypes")]
        internal InnerTextConfigurationElement<bool> EnableInheritedDocumentTypes
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                    (InnerTextConfigurationElement<bool>) this["EnableInheritedDocumentTypes"],
-                    //set the default
-                    true);
-            }
+            get { return GetOptionalTextElement("EnableInheritedDocumentTypes", true); }
        }

        [ConfigurationProperty("EnableInheritedMediaTypes")]
        internal InnerTextConfigurationElement<bool> EnableInheritedMediaTypes
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                    (InnerTextConfigurationElement<bool>)this["EnableInheritedMediaTypes"],
-                    //set the default
-                    true);
-            }
+            get { return GetOptionalTextElement("EnableInheritedMediaTypes", true); }
        }

        string IContentSection.NotificationEmailAddress
@@ -3,42 +3,24 @@ using System.Configuration;

 namespace Umbraco.Core.Configuration.UmbracoSettings
 {
-    internal class ContentScriptEditorElement : ConfigurationElement
+    internal class ContentScriptEditorElement : UmbracoConfigurationElement
    {
        [ConfigurationProperty("scriptFolderPath")]
        internal InnerTextConfigurationElement<string> ScriptFolderPath
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<string>(
-                       (InnerTextConfigurationElement<string>)this["scriptFolderPath"],
-                    //set the default
-                       "/scripts");
-            }
+            get { return GetOptionalTextElement("scriptFolderPath", "/scripts"); }
        }

        [ConfigurationProperty("scriptFileTypes")]
        internal OptionalCommaDelimitedConfigurationElement ScriptFileTypes
        {
-            get
-            {
-                return new OptionalCommaDelimitedConfigurationElement(
-                       (OptionalCommaDelimitedConfigurationElement)this["scriptFileTypes"],
-                    //set the default
-                       new[] { "js", "xml" });                
-            }
+            get { return GetOptionalDelimitedElement("scriptFileTypes", new[] {"js", "xml"}); }
        }

        [ConfigurationProperty("scriptDisableEditor")]
        internal InnerTextConfigurationElement<bool> ScriptEditorDisable
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                    (InnerTextConfigurationElement<bool>) this["scriptDisableEditor"],
-                    //set the default
-                    false);
-            }
+            get { return GetOptionalTextElement("scriptDisableEditor", false); }
        }

    }
@@ -2,7 +2,7 @@

 namespace Umbraco.Core.Configuration.UmbracoSettings
 {
-    internal class ImagingAutoFillUploadFieldElement : ConfigurationElement, IImagingAutoFillUploadField
+    internal class ImagingAutoFillUploadFieldElement : UmbracoConfigurationElement, IImagingAutoFillUploadField
    {
        /// <summary>
        /// Allow setting internally so we can create a default
@@ -17,49 +17,25 @@ namespace Umbraco.Core.Configuration.UmbracoSettings
        [ConfigurationProperty("widthFieldAlias")]
        internal InnerTextConfigurationElement<string> WidthFieldAlias
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<string>(
-                       (InnerTextConfigurationElement<string>)this["widthFieldAlias"],
-                    //set the default
-                       "umbracoWidth");
-            }
+            get { return GetOptionalTextElement("widthFieldAlias", "umbracoWidth"); }
        }

        [ConfigurationProperty("heightFieldAlias")]
        internal InnerTextConfigurationElement<string> HeightFieldAlias
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<string>(
-                       (InnerTextConfigurationElement<string>)this["heightFieldAlias"],
-                    //set the default
-                       "umbracoHeight");
-            }
+            get { return GetOptionalTextElement("heightFieldAlias", "umbracoHeight"); }
        }

        [ConfigurationProperty("lengthFieldAlias")]
        internal InnerTextConfigurationElement<string> LengthFieldAlias
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<string>(
-                       (InnerTextConfigurationElement<string>)this["lengthFieldAlias"],
-                    //set the default
-                       "umbracoBytes");
-            }
+            get { return GetOptionalTextElement("lengthFieldAlias", "umbracoBytes"); }
        }

        [ConfigurationProperty("extensionFieldAlias")]
        internal InnerTextConfigurationElement<string> ExtensionFieldAlias
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<string>(
-                       (InnerTextConfigurationElement<string>)this["extensionFieldAlias"],
-                    //set the default
-                       "umbracoExtension");
-            }
+            get { return GetOptionalTextElement("extensionFieldAlias", "umbracoExtension"); }
        }

        string IImagingAutoFillUploadField.Alias
@@ -3,67 +3,37 @@ using System.Configuration;

 namespace Umbraco.Core.Configuration.UmbracoSettings
 {
-    internal class LoggingElement : ConfigurationElement, ILoggingSection
+    internal class LoggingElement : UmbracoConfigurationElement, ILoggingSection
    {
        
        [ConfigurationProperty("autoCleanLogs")]
        internal InnerTextConfigurationElement<bool> AutoCleanLogs
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                    (InnerTextConfigurationElement<bool>)this["autoCleanLogs"],
-                    //set the default
-                    false);            
-            }
+            get { return GetOptionalTextElement("autoCleanLogs", false); }
        }

        [ConfigurationProperty("enableLogging")]
        internal InnerTextConfigurationElement<bool> EnableLogging
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                    (InnerTextConfigurationElement<bool>)this["enableLogging"],
-                    //set the default
-                    true);            
-            }
+            get { return GetOptionalTextElement("enableLogging", true); }
        }

        [ConfigurationProperty("enableAsyncLogging")]
        internal InnerTextConfigurationElement<bool> EnableAsyncLogging
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                    (InnerTextConfigurationElement<bool>)this["enableAsyncLogging"],
-                    //set the default
-                    true);           
-            }
+            get { return GetOptionalTextElement("enableAsyncLogging", true); }
        }

        [ConfigurationProperty("cleaningMiliseconds")]
        internal InnerTextConfigurationElement<int> CleaningMiliseconds
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<int>(
-                    (InnerTextConfigurationElement<int>)this["cleaningMiliseconds"],
-                    //set the default
-                    -1);                      
-            }
+            get { return GetOptionalTextElement("cleaningMiliseconds", -1); }
        }

        [ConfigurationProperty("maxLogAge")]
        internal InnerTextConfigurationElement<int> MaxLogAge
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<int>(
-                    (InnerTextConfigurationElement<int>)this["maxLogAge"],
-                    //set the default
-                    -1);
-            }
+            get { return GetOptionalTextElement("maxLogAge", -1); }
        }

        [ConfigurationCollection(typeof(DisabledLogTypesCollection), AddItemName = "logTypeAlias")]
@@ -2,7 +2,7 @@

 namespace Umbraco.Core.Configuration.UmbracoSettings
 {
-    internal class NotificationsElement : ConfigurationElement
+    internal class NotificationsElement : UmbracoConfigurationElement
    {
        [ConfigurationProperty("email")]
        internal InnerTextConfigurationElement<string> NotificationEmailAddress
@@ -13,13 +13,7 @@ namespace Umbraco.Core.Configuration.UmbracoSettings
        [ConfigurationProperty("disableHtmlEmail")]
        internal InnerTextConfigurationElement<bool> DisableHtmlEmail
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                    (InnerTextConfigurationElement<bool>) this["disableHtmlEmail"], 
-                    //set the default
-                    false);
-            }
+            get { return GetOptionalTextElement("disableHtmlEmail", false); }
        }

    }
@@ -5,30 +5,18 @@ using System.Collections.Generic;

 namespace Umbraco.Core.Configuration.UmbracoSettings
 {
-    internal class RequestHandlerElement : ConfigurationElement, IRequestHandlerSection
+    internal class RequestHandlerElement : UmbracoConfigurationElement, IRequestHandlerSection
    {
        [ConfigurationProperty("useDomainPrefixes")]
        public InnerTextConfigurationElement<bool> UseDomainPrefixes
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                    (InnerTextConfigurationElement<bool>)this["useDomainPrefixes"],
-                    //set the default
-                    false);  
-            }
+            get { return GetOptionalTextElement("useDomainPrefixes", false); }
        }

        [ConfigurationProperty("addTrailingSlash")]
        public InnerTextConfigurationElement<bool> AddTrailingSlash
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                    (InnerTextConfigurationElement<bool>)this["addTrailingSlash"],
-                    //set the default
-                    true);  
-            }
+            get { return GetOptionalTextElement("addTrailingSlash", true); }
        }

        private UrlReplacingElement _defaultUrlReplacing;
@@ -2,66 +2,36 @@

 namespace Umbraco.Core.Configuration.UmbracoSettings
 {
-    internal class SecurityElement : ConfigurationElement, ISecuritySection
+    internal class SecurityElement : UmbracoConfigurationElement, ISecuritySection
    {
        [ConfigurationProperty("keepUserLoggedIn")]
        internal InnerTextConfigurationElement<bool> KeepUserLoggedIn
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                    (InnerTextConfigurationElement<bool>)this["keepUserLoggedIn"],
-                    //set the default
-                    true);   
-            }
+            get { return GetOptionalTextElement("keepUserLoggedIn", true); }
        }

        [ConfigurationProperty("hideDisabledUsersInBackoffice")]
        internal InnerTextConfigurationElement<bool> HideDisabledUsersInBackoffice
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                    (InnerTextConfigurationElement<bool>)this["hideDisabledUsersInBackoffice"],
-                    //set the default
-                    false);                          
-            }
+            get { return GetOptionalTextElement("hideDisabledUsersInBackoffice", false); }
        }

        [ConfigurationProperty("allowPasswordReset")]
        internal InnerTextConfigurationElement<bool> AllowPasswordReset
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                    (InnerTextConfigurationElement<bool>)this["allowPasswordReset"],
-                    //set the default
-                    true);
-            }
+            get { return GetOptionalTextElement("allowPasswordReset", true); }
        }

        [ConfigurationProperty("authCookieName")]
        internal InnerTextConfigurationElement<string> AuthCookieName
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<string>(
-                    (InnerTextConfigurationElement<string>)this["authCookieName"],
-                    //set the default
-                    Constants.Web.AuthCookieName);                
-            }
+            get { return GetOptionalTextElement("authCookieName", Constants.Web.AuthCookieName); }
        }

        [ConfigurationProperty("authCookieDomain")]
        internal InnerTextConfigurationElement<string> AuthCookieDomain
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<string>(
-                    (InnerTextConfigurationElement<string>)this["authCookieDomain"],
-                    //set the default
-                    null);                    
-            }
+            get { return GetOptionalTextElement<string>("authCookieDomain", null); }
        }

        bool ISecuritySection.KeepUserLoggedIn
@@ -3,55 +3,31 @@ using System.Configuration;

 namespace Umbraco.Core.Configuration.UmbracoSettings
 {
-    internal class TemplatesElement : ConfigurationElement, ITemplatesSection
+    internal class TemplatesElement : UmbracoConfigurationElement, ITemplatesSection
    {
        [ConfigurationProperty("useAspNetMasterPages")]
        internal InnerTextConfigurationElement<bool> UseAspNetMasterPages
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                    (InnerTextConfigurationElement<bool>)this["useAspNetMasterPages"],
-                    //set the default
-                    true);
-            }
+            get { return GetOptionalTextElement("useAspNetMasterPages", true); }
        }

        [ConfigurationProperty("enableSkinSupport")]
        internal InnerTextConfigurationElement<bool> EnableSkinSupport
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                    (InnerTextConfigurationElement<bool>)this["enableSkinSupport"],
-                    //set the default
-                    true);
-            }
+            get { return GetOptionalTextElement("enableSkinSupport", true); }
        }

        [ConfigurationProperty("defaultRenderingEngine", IsRequired = true)]
        internal InnerTextConfigurationElement<RenderingEngine> DefaultRenderingEngine
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<RenderingEngine>(
-                    (InnerTextConfigurationElement<RenderingEngine>)this["defaultRenderingEngine"],
-                    //set the default
-                    RenderingEngine.Mvc);
-            }
+            get { return GetOptionalTextElement("defaultRenderingEngine", RenderingEngine.Mvc); }
        }

        [Obsolete("This has no affect and will be removed in future versions")]
        [ConfigurationProperty("enableTemplateFolders")]
        internal InnerTextConfigurationElement<bool> EnableTemplateFolders
        {
-            get
-            {
-                return new OptionalInnerTextConfigurationElement<bool>(
-                    (InnerTextConfigurationElement<bool>)this["enableTemplateFolders"],
-                    //set the default
-                    false);
-            }
+            get { return GetOptionalTextElement("enableTemplateFolders", false); }
        }

        bool ITemplatesSection.UseAspNetMasterPages
@@ -0,0 +1,36 @@
+using System.Collections.Concurrent;
+using System.Configuration;
+
+namespace Umbraco.Core.Configuration.UmbracoSettings
+{
+    /// <summary>
+    /// Base class with shared helper methods
+    /// </summary>
+    internal class UmbracoConfigurationElement : ConfigurationElement
+    {
+        /// <summary>
+        /// Used so the RawElement types are not re-created every time they are accessed
+        /// </summary>
+        private readonly ConcurrentDictionary<string, RawXmlConfigurationElement> _rawElements = new ConcurrentDictionary<string, RawXmlConfigurationElement>();
+
+        protected OptionalInnerTextConfigurationElement<T> GetOptionalTextElement<T>(string name, T defaultVal)
+        {
+            return (OptionalInnerTextConfigurationElement<T>) _rawElements.GetOrAdd(
+                name,
+                s => new OptionalInnerTextConfigurationElement<T>(
+                    (InnerTextConfigurationElement<T>) this[s],
+                    //set the default
+                    defaultVal));
+        }
+
+        protected OptionalCommaDelimitedConfigurationElement GetOptionalDelimitedElement(string name, string[] defaultVal)
+        {
+            return (OptionalCommaDelimitedConfigurationElement) _rawElements.GetOrAdd(
+                name,
+                s => new OptionalCommaDelimitedConfigurationElement(
+                    (CommaDelimitedConfigurationElement) this[name],
+                    //set the default
+                    defaultVal));
+        }
+    }
+}
@@ -6,7 +6,7 @@ namespace Umbraco.Core.Configuration
 {
    public class UmbracoVersion
    {
-        private static readonly Version Version = new Version("7.5.3");
+        private static readonly Version Version = new Version("7.5.12");

        /// <summary>
        /// Gets the current version of Umbraco.
@@ -122,6 +122,11 @@ namespace Umbraco.Core
 				/// MediaType alias for an image.
 				/// </summary>
 				public const string Image = "Image";
+
+                /// <summary>
+                /// MediaType alias indicating allowing auto-selection.
+                /// </summary>
+			    public const string AutoSelect = "umbracoAutoSelect";
 			}
            
 		    /// <summary>
@@ -69,6 +69,11 @@ namespace Umbraco.Core
            /// </summary>
            public const string Document = "C66BA18E-EAF3-4CFF-8A22-41B16D66A972";

+            /// <summary>
+            /// Guid for a Document object.
+            /// </summary>
+            public static readonly Guid DocumentGuid = new Guid(Document);
+
            /// <summary>
            /// Guid for a Document Type object.
            /// </summary>
@@ -1,5 +1,6 @@
 using System;
 using System.Collections.Generic;
+using System.Configuration;
 using System.IO;
 using System.Linq;
 using System.Threading;
@@ -30,12 +31,13 @@ using Umbraco.Core.Manifest;
 using Umbraco.Core.Services;
 using Umbraco.Core.Sync;
 using Umbraco.Core.Strings;
+using IntegerValidator = Umbraco.Core.PropertyEditors.IntegerValidator;
 using MigrationsVersionFourNineZero = Umbraco.Core.Persistence.Migrations.Upgrades.TargetVersionFourNineZero;

 namespace Umbraco.Core
 {
    /// <summary>
-    /// A bootstrapper for the Umbraco application which initializes all objects for the Core of the application 
+    /// A bootstrapper for the Umbraco application which initializes all objects for the Core of the application
    /// </summary>
    /// <remarks>
    /// This does not provide any startup functionality relating to web objects
@@ -191,14 +193,14 @@ namespace Umbraco.Core
        protected virtual CacheHelper CreateApplicationCache()
        {
            var cacheHelper = new CacheHelper(
-                //we need to have the dep clone runtime cache provider to ensure 
+                //we need to have the dep clone runtime cache provider to ensure
                //all entities are cached properly (cloned in and cloned out)
                new DeepCloneRuntimeCacheProvider(new ObjectCacheRuntimeCacheProvider()),
                new StaticCacheProvider(),
                //we have no request based cache when not running in web-based context
                new NullCacheProvider(),
                new IsolatedRuntimeCache(type =>
-                    //we need to have the dep clone runtime cache provider to ensure 
+                    //we need to have the dep clone runtime cache provider to ensure
                    //all entities are cached properly (cloned in and cloned out)
                    new DeepCloneRuntimeCacheProvider(new ObjectCacheRuntimeCacheProvider())));

@@ -251,18 +253,18 @@ namespace Umbraco.Core
        }

        /// <summary>
-        /// Special method to initialize the ApplicationEventsResolver and any modifications required for it such 
+        /// Special method to initialize the ApplicationEventsResolver and any modifications required for it such
        /// as adding custom types to the resolver.
        /// </summary>
        protected virtual void InitializeApplicationEventsResolver()
        {
            //find and initialize the application startup handlers, we need to initialize this resolver here because
-            //it is a special resolver where they need to be instantiated first before any other resolvers in order to bind to 
+            //it is a special resolver where they need to be instantiated first before any other resolvers in order to bind to
            //events and to call their events during bootup.
            //ApplicationStartupHandler.RegisterHandlers();
            //... and set the special flag to let us resolve before frozen resolution
            ApplicationEventsResolver.Current = new ApplicationEventsResolver(
-                ServiceProvider, 
+                ServiceProvider,
                ProfilingLogger.Logger,
                PluginManager.ResolveApplicationStartupHandlers())
            {
@@ -282,7 +284,7 @@ namespace Umbraco.Core
        }

        /// <summary>
-        /// Fires after initialization and calls the callback to allow for customizations to occur & 
+        /// Fires after initialization and calls the callback to allow for customizations to occur &
        /// Ensure that the OnApplicationStarting methods of the IApplicationEvents are called
        /// </summary>
        /// <param name="afterStartup"></param>
@@ -334,7 +336,7 @@ namespace Umbraco.Core
        {
            if (_isComplete)
                throw new InvalidOperationException("The boot manager has already been completed");
-            
+
            FreezeResolution();

            //Here we need to make sure the db can be connected to
@@ -366,7 +368,7 @@ namespace Umbraco.Core
                            ProfilingLogger.Logger.Error<CoreBootManager>("An error occurred running OnApplicationStarted for handler " + x.GetType(), ex);
                            throw;
                        }
-                    }); 
+                    });
            }

            //Now, startup all of our legacy startup handler
@@ -413,7 +415,7 @@ namespace Umbraco.Core

            if (currentTry == 5)
            {
-                throw new UmbracoStartupFailedException("Umbraco cannot start. A connection string is configured but the Umbraco cannot connect to the database.");
+                throw new UmbracoStartupFailedException("Umbraco cannot start. A connection string is configured but Umbraco cannot connect to the database.");
            }
        }

@@ -455,6 +457,10 @@ namespace Umbraco.Core
            {
                ServerRegistrarResolver.Current = new ServerRegistrarResolver(new ConfigServerRegistrar());
            }
+            else if ("true".InvariantEquals(ConfigurationManager.AppSettings["umbracoDisableElectionForSingleServer"]))
+            {
+                ServerRegistrarResolver.Current = new ServerRegistrarResolver(new SingleServerRegistrar());
+            }
            else
            {
                ServerRegistrarResolver.Current = new ServerRegistrarResolver(
@@ -462,7 +468,6 @@ namespace Umbraco.Core
                        new Lazy<IServerRegistrationService>(() => ApplicationContext.Services.ServerRegistrationService),
                        new DatabaseServerRegistrarOptions()));
            }
-            

            //by default we'll use the database server messenger with default options (no callbacks),
            // this will be overridden in the web startup
@@ -473,7 +478,7 @@ namespace Umbraco.Core
                ServiceProvider, ProfilingLogger.Logger,
                () => PluginManager.ResolveAssignedMapperTypes());

-           
+
            //RepositoryResolver.Current = new RepositoryResolver(
            //    new RepositoryFactory(ApplicationCache));

@@ -34,6 +34,12 @@ namespace Umbraco.Core
        private string _connectionString;
        private string _providerName;
        private DatabaseSchemaResult _result;
+        private DateTime? _connectionLastChecked = null;
+
+        /// <summary>
+        /// The number of minutes to throttle the checks to CanConnect
+        /// </summary>
+        private const int ConnectionCheckMinutes = 1;

        [Obsolete("Use the constructor specifying all dependencies instead")]
        public DatabaseContext(IDatabaseFactory factory)
@@ -110,13 +116,28 @@ namespace Umbraco.Core
        {
            get
            {
-                if (IsDatabaseConfigured == false) return false;
-                var canConnect = DbConnectionExtensions.IsConnectionAvailable(ConnectionString, DatabaseProvider);
-                LogHelper.Info<DatabaseContext>("CanConnect = " + canConnect);
-                return canConnect;
+                if (IsDatabaseConfigured == false)
+                    return false;
+
+                //Don't check again if the timeout period hasn't elapsed
+                //this ensures we don't keep checking the connection too many times in a row like during startup.
+                //Do check if the _connectionLastChecked is null which means we're just initializing or it could 
+                //not connect last time it was checked.
+                if ((_connectionLastChecked.HasValue && (DateTime.Now - _connectionLastChecked.Value).TotalMinutes > ConnectionCheckMinutes)
+                    || _connectionLastChecked.HasValue == false)
+                {
+                    var canConnect = DbConnectionExtensions.IsConnectionAvailable(ConnectionString, DatabaseProvider);
+                    LogHelper.Info<DatabaseContext>("CanConnect = " + canConnect);
+
+                    _connectionLastChecked = canConnect == false ? null : (DateTime?) DateTime.Now;
+                    return canConnect;
+                }
+
+                return _connectionLastChecked.HasValue;
            }
        }

+
        /// <summary>
        /// Gets the configured umbraco db connection string.
        /// </summary>
@@ -174,7 +195,7 @@ namespace Umbraco.Core
        /// </summary>
        public void ConfigureEmbeddedDatabaseConnection()
        {
-            const string providerName = "System.Data.SqlServerCe.4.0";
+            const string providerName = Constants.DatabaseProviders.SqlCe;

            var connectionString = GetEmbeddedDatabaseConnectionString();
            SaveConnectionString(connectionString, providerName);
@@ -182,15 +203,10 @@ namespace Umbraco.Core
            var path = Path.Combine(GlobalSettings.FullpathToRoot, "App_Data", "Umbraco.sdf");
            if (File.Exists(path) == false)
            {
-                var engine = new SqlCeEngine(connectionString);
-                engine.CreateDatabase();
-
-                // SD: Pretty sure this should be in a using clause but i don't want to cause unknown side-effects here
-                // since it's been like this for quite some time
-                //using (var engine = new SqlCeEngine(connectionString))
-                //{
-                //    engine.CreateDatabase();    
-                //}
+                using (var engine = new SqlCeEngine(connectionString))
+                {
+                    engine.CreateDatabase();
+                }
            }

            Initialize(providerName);
@@ -0,0 +1,135 @@
+using System;
+using System.Diagnostics;
+using System.IO;
+using System.Runtime.InteropServices;
+using Umbraco.Core.IO;
+
+namespace Umbraco.Core.Diagnostics
+{
+    // taken from https://blogs.msdn.microsoft.com/dondu/2010/10/24/writing-minidumps-in-c/
+    // and https://blogs.msdn.microsoft.com/dondu/2010/10/31/writing-minidumps-from-exceptions-in-c/
+    // which itself got it from http://blog.kalmbach-software.de/2008/12/13/writing-minidumps-in-c/
+
+    internal static class MiniDump
+    {
+        private static readonly object LockO = new object();
+
+        [Flags]
+        public enum Option : uint
+        {
+            // From dbghelp.h:
+            Normal = 0x00000000,
+            WithDataSegs = 0x00000001,
+            WithFullMemory = 0x00000002,
+            WithHandleData = 0x00000004,
+            FilterMemory = 0x00000008,
+            ScanMemory = 0x00000010,
+            WithUnloadedModules = 0x00000020,
+            WithIndirectlyReferencedMemory = 0x00000040,
+            FilterModulePaths = 0x00000080,
+            WithProcessThreadData = 0x00000100,
+            WithPrivateReadWriteMemory = 0x00000200,
+            WithoutOptionalData = 0x00000400,
+            WithFullMemoryInfo = 0x00000800,
+            WithThreadInfo = 0x00001000,
+            WithCodeSegs = 0x00002000,
+            WithoutAuxiliaryState = 0x00004000,
+            WithFullAuxiliaryState = 0x00008000,
+            WithPrivateWriteCopyMemory = 0x00010000,
+            IgnoreInaccessibleMemory = 0x00020000,
+            ValidTypeFlags = 0x0003ffff,
+        }
+
+        //typedef struct _MINIDUMP_EXCEPTION_INFORMATION {
+        //    DWORD ThreadId;
+        //    PEXCEPTION_POINTERS ExceptionPointers;
+        //    BOOL ClientPointers;
+        //} MINIDUMP_EXCEPTION_INFORMATION, *PMINIDUMP_EXCEPTION_INFORMATION;
+        [StructLayout(LayoutKind.Sequential, Pack = 4)]  // Pack=4 is important! So it works also for x64!
+        public struct MiniDumpExceptionInformation
+        {
+            public uint ThreadId;
+            public IntPtr ExceptionPointers;
+            [MarshalAs(UnmanagedType.Bool)]
+            public bool ClientPointers;
+        }
+
+        //BOOL
+        //WINAPI
+        //MiniDumpWriteDump(
+        //    __in HANDLE hProcess,
+        //    __in DWORD ProcessId,
+        //    __in HANDLE hFile,
+        //    __in MINIDUMP_TYPE DumpType,
+        //    __in_opt PMINIDUMP_EXCEPTION_INFORMATION ExceptionParam,
+        //    __in_opt PMINIDUMP_USER_STREAM_INFORMATION UserStreamParam,
+        //    __in_opt PMINIDUMP_CALLBACK_INFORMATION CallbackParam
+        //    );
+
+        // Overload requiring MiniDumpExceptionInformation
+        [DllImport("dbghelp.dll", EntryPoint = "MiniDumpWriteDump", CallingConvention = CallingConvention.StdCall, CharSet = CharSet.Unicode, ExactSpelling = true, SetLastError = true)]
+        private static extern bool MiniDumpWriteDump(IntPtr hProcess, uint processId, SafeHandle hFile, uint dumpType, ref MiniDumpExceptionInformation expParam, IntPtr userStreamParam, IntPtr callbackParam);
+
+        // Overload supporting MiniDumpExceptionInformation == NULL
+        [DllImport("dbghelp.dll", EntryPoint = "MiniDumpWriteDump", CallingConvention = CallingConvention.StdCall, CharSet = CharSet.Unicode, ExactSpelling = true, SetLastError = true)]
+        private static extern bool MiniDumpWriteDump(IntPtr hProcess, uint processId, SafeHandle hFile, uint dumpType, IntPtr expParam, IntPtr userStreamParam, IntPtr callbackParam);
+
+        [DllImport("kernel32.dll", EntryPoint = "GetCurrentThreadId", ExactSpelling = true)]
+        private static extern uint GetCurrentThreadId();
+
+        private static bool Write(SafeHandle fileHandle, Option options, bool withException = false)
+        {
+            var currentProcess = Process.GetCurrentProcess();
+            var currentProcessHandle = currentProcess.Handle;
+            var currentProcessId = (uint)currentProcess.Id;
+
+            MiniDumpExceptionInformation exp;
+
+            exp.ThreadId = GetCurrentThreadId();
+            exp.ClientPointers = false;
+            exp.ExceptionPointers = IntPtr.Zero;
+
+            if (withException)
+                exp.ExceptionPointers = Marshal.GetExceptionPointers();
+
+            var bRet = exp.ExceptionPointers == IntPtr.Zero
+                ? MiniDumpWriteDump(currentProcessHandle, currentProcessId, fileHandle, (uint) options, IntPtr.Zero, IntPtr.Zero, IntPtr.Zero)
+                : MiniDumpWriteDump(currentProcessHandle, currentProcessId, fileHandle, (uint) options, ref exp, IntPtr.Zero, IntPtr.Zero);
+
+            return bRet;
+        }
+
+        public static bool Dump(Option options = Option.WithFullMemory, bool withException = false)
+        {
+            lock (LockO)
+            {
+                // work around "stack trace is not available while minidump debugging",
+                // by making sure a local var (that we can inspect) contains the stack trace.
+                // getting the call stack before it is unwound would require a special exception
+                // filter everywhere in our code = not!
+                var stacktrace = withException ? Environment.StackTrace : string.Empty;
+
+                var filepath = IOHelper.MapPath("~/App_Data/MiniDump");
+                if (Directory.Exists(filepath) == false)
+                    Directory.CreateDirectory(filepath);
+
+                var filename = Path.Combine(filepath, string.Format("{0:yyyyMMddTHHmmss}.{1}.dmp", DateTime.UtcNow, Guid.NewGuid().ToString("N").Substring(0, 4)));
+                using (var stream = new FileStream(filename, FileMode.Create, FileAccess.ReadWrite, FileShare.Write))
+                {
+                    return Write(stream.SafeFileHandle, options, withException);
+                }
+            }
+        }
+
+        public static bool OkToDump()
+        {
+            lock (LockO)
+            {
+                var filepath = IOHelper.MapPath("~/App_Data/MiniDump");
+                if (Directory.Exists(filepath) == false) return true;
+                var count = Directory.GetFiles(filepath, "*.dmp").Length;
+                return count < 8;
+            }
+        }
+    }
+}
@@ -128,8 +128,14 @@ namespace Umbraco.Core.Events
            get { return TargetSemVersion.GetVersion(); }
        }

+        /// <summary>
+        /// Gets the origin version of the migration, i.e. the one that is currently installed.
+        /// </summary>
        public SemVersion ConfiguredSemVersion { get; private set; }

+        /// <summary>
+        /// Gets the target version of the migration.
+        /// </summary>
        public SemVersion TargetSemVersion { get; private set; }

        public string ProductName { get; private set; }
@@ -41,11 +41,7 @@ namespace Umbraco.Core.IO
        {
            using (var file = fs.OpenFile(path))
            {
-                using (var sr = new StreamReader(file))
-                {
-                    var str = sr.ReadToEnd();
-                    return str.Length;
-                }
+                return file.Length;
            }
        }

@@ -1,5 +1,6 @@
 using System;
 using System.Collections.Generic;
+using System.ComponentModel;
 using System.Configuration;
 using System.IO;
 using System.Linq;
@@ -72,15 +73,28 @@ namespace Umbraco.Core.IO
        {
            get
            {
-                if (GlobalSettings.ContentCacheXmlStoredInCodeGen && SystemUtilities.GetCurrentTrustLevel() == AspNetHostingPermissionLevel.Unrestricted)
-                {
-                    return Path.Combine(HttpRuntime.CodegenDir, @"UmbracoData\umbraco.config");
+                switch (GlobalSettings.ContentCacheXmlStorageLocation)
+                {                    
+                    case ContentXmlStorage.AspNetTemp:
+                        return Path.Combine(HttpRuntime.CodegenDir, @"UmbracoData\umbraco.config");
+                    case ContentXmlStorage.EnvironmentTemp:
+                        var appDomainHash = HttpRuntime.AppDomainAppId.ToSHA1();
+                        var cachePath = Path.Combine(Environment.ExpandEnvironmentVariables("%temp%"), "UmbracoXml",
+                            //include the appdomain hash is just a safety check, for example if a website is moved from worker A to worker B and then back
+                            // to worker A again, in theory the %temp%  folder should already be empty but we really want to make sure that its not
+                            // utilizing an old path
+                            appDomainHash);
+                        return Path.Combine(cachePath, "umbraco.config");
+                    case ContentXmlStorage.Default:
+                        return IOHelper.ReturnPath("umbracoContentXML", "~/App_Data/umbraco.config");
+                    default:
+                        throw new ArgumentOutOfRangeException();
                }
-                return IOHelper.ReturnPath("umbracoContentXML", "~/App_Data/umbraco.config");
            }
        }

        [Obsolete("Use GlobalSettings.ContentCacheXmlStoredInCodeGen instead")]
+        [EditorBrowsable(EditorBrowsableState.Never)]
        internal static bool ContentCacheXmlStoredInCodeGen
        {
            get { return GlobalSettings.ContentCacheXmlStoredInCodeGen; }
@@ -86,7 +86,7 @@ namespace Umbraco.Core.IO
        private void Initialize()
        {
            Filename = _fs.GetFileName(Path);
-            Extension = _fs.GetExtension(Path) != null
+            Extension = string.IsNullOrEmpty(_fs.GetExtension(Path)) == false
                ? _fs.GetExtension(Path).Substring(1).ToLowerInvariant()
                : "";
            Url = _fs.GetUrl(Path);
@@ -1,9 +1,9 @@
-namespace Umbraco.Core.Logging
-{
-    using System;
-    using System.Runtime.CompilerServices;
+using System;
+using System.Runtime.CompilerServices;
+using ImageProcessor.Common.Exceptions;

-    using ImageProcessor.Common.Exceptions;
+namespace Umbraco.Core.Logging
+{   

    /// <summary>
    /// A logger for explicitly logging ImageProcessor exceptions.
@@ -2,10 +2,13 @@
 using System.Diagnostics;
 using System.IO;
 using System.Linq;
+using System.Reflection;
 using System.Threading;
 using System.Web;
 using log4net;
 using log4net.Config;
+using Umbraco.Core.Configuration;
+using Umbraco.Core.Diagnostics;

 namespace Umbraco.Core.Logging
 {
@@ -56,18 +59,72 @@ namespace Umbraco.Core.Logging
 		internal ILog LoggerFor(object getTypeFromInstance)
 		{
 			if (getTypeFromInstance == null) throw new ArgumentNullException("getTypeFromInstance");
-			
+
 			return LogManager.GetLogger(getTypeFromInstance.GetType());
 		}
-        
+
 		public void Error(Type callingType, string message, Exception exception)
 		{
-			var logger = LogManager.GetLogger(callingType);
-			if (logger != null)
-				logger.Error((message), exception);
+            var logger = LogManager.GetLogger(callingType);
+		    if (logger == null) return;
+
+		    var dump = false;
+
+		    if (IsTimeoutThreadAbortException(exception))
+		    {
+		        message += "\r\nThe thread has been aborted, because the request has timed out.";
+
+                // dump if configured, or if stacktrace contains Monitor.ReliableEnter
+		        dump = UmbracoConfig.For.CoreDebug().DumpOnTimeoutThreadAbort || IsMonitorEnterThreadAbortException(exception);
+
+                // dump if it is ok to dump (might have a cap on number of dump...)
+		        dump &= MiniDump.OkToDump();
+		    }
+
+            if (dump)
+		    {
+                try
+                {
+                    var dumped = MiniDump.Dump(withException: true);
+                    message += dumped
+                        ? "\r\nA minidump was created in App_Data/MiniDump"
+                        : "\r\nFailed to create a minidump";
+                }
+                catch (Exception e)
+                {
+                    message += string.Format("\r\nFailed to create a minidump ({0}: {1})", e.GetType().FullName, e.Message);
+                }
+            }
+
+            logger.Error(message, exception);
 		}

-		public void Warn(Type callingType, string message, params Func<object>[] formatItems)
+        private static bool IsMonitorEnterThreadAbortException(Exception exception)
+        {
+            var abort = exception as ThreadAbortException;
+            if (abort == null) return false;
+
+            var stacktrace = abort.StackTrace;
+            return stacktrace.Contains("System.Threading.Monitor.ReliableEnter");
+        }
+
+        private static bool IsTimeoutThreadAbortException(Exception exception)
+        {
+            var abort = exception as ThreadAbortException;
+            if (abort == null) return false;
+
+            if (abort.ExceptionState == null) return false;
+
+            var stateType = abort.ExceptionState.GetType();
+            if (stateType.FullName != "System.Web.HttpApplication+CancelModuleException") return false;
+
+            var timeoutField = stateType.GetField("_timeout", BindingFlags.Instance | BindingFlags.NonPublic);
+            if (timeoutField == null) return false;
+
+            return (bool) timeoutField.GetValue(abort.ExceptionState);
+        }
+
+        public void Warn(Type callingType, string message, params Func<object>[] formatItems)
 		{
 			var logger = LogManager.GetLogger(callingType);
 			if (logger == null || logger.IsWarnEnabled == false) return;
@@ -82,7 +139,7 @@ namespace Umbraco.Core.Logging
 			if (showHttpTrace && HttpContext.Current != null)
 			{
 				HttpContext.Current.Trace.Warn(callingType.Name, string.Format(message, formatItems.Select(x => x.Invoke()).ToArray()));
-			}	
+			}

 			var logger = LogManager.GetLogger(callingType);
 			if (logger == null || logger.IsWarnEnabled == false) return;
@@ -99,7 +156,7 @@ namespace Umbraco.Core.Logging
            var logger = LogManager.GetLogger(callingType);
            if (logger == null || logger.IsWarnEnabled == false) return;
            var executedParams = formatItems.Select(x => x.Invoke()).ToArray();
-            logger.WarnFormat((message) + ". Exception: " + e, executedParams);		
+            logger.WarnFormat((message) + ". Exception: " + e, executedParams);
 		}

 		/// <summary>
@@ -1,18 +1,21 @@
 using System;
 using System.Collections.Generic;
-using System.Diagnostics;
-using System.IO.MemoryMappedFiles;
-using System.Text;
 using System.Threading;
-using System.Threading.Tasks;
 using System.Web.Hosting;
 using Umbraco.Core.Logging;
-using Umbraco.Core.ObjectResolution;

 namespace Umbraco.Core
 {
-    // represents the main domain
-    class MainDom : IRegisteredObject
+    /// <summary>
+    /// Represents the main AppDomain running for a given application.
+    /// </summary>
+    /// <remarks>
+    /// <para>There can be only one "main" AppDomain running for a given application at a time.</para>
+    /// <para>When an AppDomain starts, it tries to acquire the main domain status.</para>
+    /// <para>When an AppDomain stops (eg the application is restarting) it should release the main domain status.</para>
+    /// <para>It is possible to register against the MainDom and be notified when it is released.</para>
+    /// </remarks>
+    internal class MainDom : IRegisteredObject
    {
        #region Vars

@@ -25,7 +28,7 @@ namespace Umbraco.Core
        private readonly AsyncLock _asyncLock;
        private IDisposable _asyncLocker;

-        // event wait handle used to notify current main domain that it should 
+        // event wait handle used to notify current main domain that it should
        // release the lock because a new domain wants to be the main domain
        private readonly EventWaitHandle _signal;

@@ -34,16 +37,26 @@ namespace Umbraco.Core
        private volatile bool _signaled; // we have been signaled

        // actions to run before releasing the main domain
-        private readonly SortedList<int, Action> _callbacks = new SortedList<int, Action>();
+        private readonly SortedList<int, Action> _callbacks = new SortedList<int, Action>(new WeightComparer());

        private const int LockTimeoutMilliseconds = 90000; // (1.5 * 60 * 1000) == 1 min 30 seconds

+        private class WeightComparer : IComparer<int>
+        {
+            public int Compare(int x, int y)
+            {
+                var result = x.CompareTo(y);
+                // return "equal" as "greater than"
+                return result == 0 ? 1 : result;
+            }
+        }
+
        #endregion

        #region Ctor

        // initializes a new instance of MainDom
-        public MainDom(ILogger logger)
+        internal MainDom(ILogger logger)
        {
            _logger = logger;

@@ -52,22 +65,47 @@ namespace Umbraco.Core
            if (HostingEnvironment.ApplicationID != null)
                appId = HostingEnvironment.ApplicationID.ReplaceNonAlphanumericChars(string.Empty);

-            var lockName = "UMBRACO-" + appId + "-MAINDOM-LCK";
+            // combining with the physical path because if running on eg IIS Express,
+            // two sites could have the same appId even though they are different.
+            //
+            // now what could still collide is... two sites, running in two different processes
+            // and having the same appId, and running on the same app physical path
+            //
+            // we *cannot* use the process ID here because when an AppPool restarts it is
+            // a new process for the same application path
+
+            var appPath = HostingEnvironment.ApplicationPhysicalPath;
+            var hash = (appId + ":::" + appPath).ToSHA1();
+
+            var lockName = "UMBRACO-" + hash + "-MAINDOM-LCK";
            _asyncLock = new AsyncLock(lockName);

-            var eventName = "UMBRACO-" + appId + "-MAINDOM-EVT";
+            var eventName = "UMBRACO-" + hash + "-MAINDOM-EVT";
            _signal = new EventWaitHandle(false, EventResetMode.AutoReset, eventName);
        }

        #endregion

-        // register a main domain consumer
+        /// <summary>
+        /// Registers a resource that requires the current AppDomain to be the main domain to function.
+        /// </summary>
+        /// <param name="release">An action to execute before the AppDomain releases the main domain status.</param>
+        /// <param name="weight">An optional weight (lower goes first).</param>
+        /// <returns>A value indicating whether it was possible to register.</returns>
        public bool Register(Action release, int weight = 100)
        {
            return Register(null, release, weight);
        }

-        // register a main domain consumer
+        /// <summary>
+        /// Registers a resource that requires the current AppDomain to be the main domain to function.
+        /// </summary>
+        /// <param name="install">An action to execute when registering.</param>
+        /// <param name="release">An action to execute before the AppDomain releases the main domain status.</param>
+        /// <param name="weight">An optional weight (lower goes first).</param>
+        /// <returns>A value indicating whether it was possible to register.</returns>
+        /// <remarks>If registering is successful, then the <paramref name="install"/> action
+        /// is guaranteed to execute before the AppDomain releases the main domain status.</remarks>
        public bool Register(Action install, Action release, int weight = 100)
        {
            lock (_locko)
@@ -97,7 +135,7 @@ namespace Umbraco.Core

            try
            {
-                _logger.Debug<MainDom>("Stopping...");
+                _logger.Info<MainDom>("Stopping...");
                foreach (var callback in _callbacks.Values)
                {
                    try
@@ -109,7 +147,7 @@ namespace Umbraco.Core
                        _logger.Error<MainDom>("Error while running callback, remaining callbacks will not run.", e);
                        throw;
                    }
-                    
+
                }
                _logger.Debug<MainDom>("Stopped.");
            }
@@ -118,12 +156,12 @@ namespace Umbraco.Core
                // in any case...
                _isMainDom = false;
                _asyncLocker.Dispose();
-                _logger.Debug<MainDom>("Released MainDom.");
+                _logger.Info<MainDom>("Released MainDom.");
            }
        }

        // acquires the main domain
-        public bool Acquire()
+        internal bool Acquire()
        {
            lock (_locko) // we don't want the hosting environment to interfere by signaling
            {
@@ -131,11 +169,11 @@ namespace Umbraco.Core
                // the handler is not installed so that would be the hosting environment
                if (_signaled)
                {
-                    _logger.Debug<MainDom>("Cannot acquire MainDom (signaled).");
+                    _logger.Info<MainDom>("Cannot acquire MainDom (signaled).");
                    return false;
                }

-                _logger.Debug<MainDom>("Acquiring MainDom...");
+                _logger.Info<MainDom>("Acquiring MainDom...");

                // signal other instances that we want the lock, then wait one the lock,
                // which may timeout, and this is accepted - see comments below
@@ -162,7 +200,7 @@ namespace Umbraco.Core

                HostingEnvironment.RegisterObject(this);

-                _logger.Debug<MainDom>("Acquired MainDom.");
+                _logger.Info<MainDom>("Acquired MainDom.");
                return true;
            }
        }
@@ -174,7 +212,7 @@ namespace Umbraco.Core
        }

        // IRegisteredObject
-        public void Stop(bool immediate)
+        void IRegisteredObject.Stop(bool immediate)
        {
            try
            {
@@ -274,6 +274,9 @@ namespace Umbraco.Core.Models
        /// </summary>
        public bool HasPublishedVersion { get { return PublishedVersionGuid != default(Guid); } }

+        [IgnoreDataMember]
+        internal DateTime PublishedDate { get; set; }
+
        /// <summary>
        /// Changes the Trashed state of the content object
        /// </summary>
@@ -404,8 +404,7 @@ namespace Umbraco.Core.Models
            return content.Properties
                          .Where(property => propertyGroup.PropertyTypes
                                                          .Select(propertyType => propertyType.Id)
-                                                          .Contains(property.PropertyTypeId))
-                          .OrderBy(x => x.PropertyType.SortOrder);
+                                                          .Contains(property.PropertyTypeId));
        }

        /// <summary>
@@ -326,7 +326,7 @@ namespace Umbraco.Core.Models
            }
        }

-            /// <summary>
+        /// <summary>
        /// A boolean flag indicating if a property type has been removed from this instance.
        /// </summary>
        /// <remarks>
@@ -4,7 +4,7 @@ namespace Umbraco.Core.Models
 {
    /// <summary>
    /// Defines a ContentType, which Media is based on
-    /// </summary
+    /// </summary>
    public interface IMediaType : IContentTypeComposition
    {

@@ -13,6 +13,7 @@ namespace Umbraco.Core.Models.Identity
        public override void ConfigureMappings(IConfiguration config, ApplicationContext applicationContext)
        {
            config.CreateMap<IUser, BackOfficeIdentityUser>()
+                .ForMember(user => user.LastLoginDateUtc, expression => expression.MapFrom(user => user.LastLoginDate.ToUniversalTime()))
                .ForMember(user => user.Email, expression => expression.MapFrom(user => user.Email))
                .ForMember(user => user.Id, expression => expression.MapFrom(user => user.Id))
                .ForMember(user => user.LockoutEndDateUtc, expression => expression.MapFrom(user => user.IsLockedOut ? DateTime.MaxValue.ToUniversalTime() : (DateTime?)null))                
@@ -24,12 +24,17 @@ namespace Umbraco.Core.Models.Identity
        /// 
        /// </summary>
        public IdentityUser()
-        {
+        {            
            this.Claims = new List<TClaim>();
            this.Roles = new List<TRole>();
            this.Logins = new List<TLogin>();
        }

+        /// <summary>
+        /// Last login date
+        /// </summary>
+        public virtual DateTime? LastLoginDateUtc { get; set; }
+
        /// <summary>
        /// Email
        /// 
@@ -69,6 +69,7 @@ namespace Umbraco.Core.Models
            OnCollectionChanged(new NotifyCollectionChangedEventArgs(NotifyCollectionChangedAction.Reset));
        }

+        //TODO: Instead of 'new' this should explicitly implement one of the collection interfaces members
        internal new void Add(PropertyType item)
        {
            using (new WriteLock(_addLocker))
@@ -25,7 +25,7 @@ namespace Umbraco.Core.Models
            NoAccessNodeId = noAccessNode.Id;
            _protectedNodeId = protectedNode.Id;

-            _ruleCollection = new ObservableCollection<PublicAccessRule>(ruleCollection);            
+            _ruleCollection = new ObservableCollection<PublicAccessRule>(ruleCollection);
            _ruleCollection.CollectionChanged += _ruleCollection_CollectionChanged;
        }

@@ -81,7 +81,7 @@ namespace Umbraco.Core.Models
        internal IEnumerable<Guid> RemovedRules
        {
            get { return _removedRules; }
-        } 
+        }

        public IEnumerable<PublicAccessRule> Rules
        {
@@ -107,10 +107,13 @@ namespace Umbraco.Core.Models

        public void ClearRules()
        {
-            foreach (var rule in _ruleCollection)
-            {
-                RemoveRule(rule);
-            }
+            _ruleCollection.Clear();
+        }
+
+
+        internal void ClearRemovedRules()
+        {
+            _removedRules.Clear();
        }

        [DataMember]
@@ -126,7 +129,7 @@ namespace Umbraco.Core.Models
            get { return _noAccessNodeId; }
            set { SetPropertyValueAndDetectChanges(value, ref _noAccessNodeId, Ps.Value.NoAccessNodeIdSelector); }
        }
-       
+
        [DataMember]
        public int ProtectedNodeId
        {
@@ -179,7 +179,7 @@ namespace Umbraco.Core.Models.PublishedContent
            }

            if (contentType == null)
-                throw new Exception(string.Format("ContentTypeService failed to find a {0} type with alias \"{1}\".",
+                throw new InvalidOperationException(string.Format("ContentTypeService failed to find a {0} type with alias \"{1}\".",
                    itemType.ToString().ToLower(), alias));

            return new PublishedContentType(contentType);
@@ -106,12 +106,13 @@ namespace Umbraco.Core.Models.PublishedContent
        /// <param name="propertyTypeAlias">The property type alias.</param>
        /// <param name="dataTypeDefinitionId">The datatype definition identifier.</param>
        /// <param name="propertyEditorAlias">The property editor alias.</param>
+        /// <param name="initConverters">Generally used only for testing, in production this will always be true</param>
        /// <remarks>
        /// <para>The new published property type does not belong to a published content type.</para>
        /// <para>The values of <paramref name="dataTypeDefinitionId"/> and <paramref name="propertyEditorAlias"/> are
        /// assumed to be valid and consistent.</para>
        /// </remarks>
-        internal PublishedPropertyType(string propertyTypeAlias, int dataTypeDefinitionId, string propertyEditorAlias)
+        internal PublishedPropertyType(string propertyTypeAlias, int dataTypeDefinitionId, string propertyEditorAlias, bool initConverters = true)
        {
            // ContentType 
            // - in unit tests, to be set by PublishedContentType when creating it
@@ -122,7 +123,8 @@ namespace Umbraco.Core.Models.PublishedContent
            DataTypeId = dataTypeDefinitionId;
            PropertyEditorAlias = propertyEditorAlias;

-            InitializeConverters();
+            if (initConverters)
+                InitializeConverters();
        }

        #endregion
@@ -19,5 +19,8 @@ namespace Umbraco.Core.Models.Rdbms

        [Column("newest")]
        public bool Newest { get; set; }
+
+        [Column("updateDate")]
+        public DateTime VersionDate { get; set; }
    }
 }
@@ -11,7 +11,6 @@ namespace Umbraco.Core.Models.Rdbms
    {
        [Column("id")]
        [PrimaryKeyColumn]
-        [Index(IndexTypes.NonClustered, Name = "IX_cmsPropertyData")]
        public int Id { get; set; }

        [Column("contentNodeId")]
@@ -1,15 +1,117 @@
 using System;
 using System.Collections.Generic;
+using System.IO;
 using System.Linq;
 using System.Text;
 using System.Threading.Tasks;
 using System.Web.UI.WebControls;
+using Umbraco.Core.Logging;
 using Umbraco.Core.Models.EntityBase;
+using Umbraco.Core.Models.Rdbms;

 namespace Umbraco.Core.Models
 {
    internal static class UmbracoEntityExtensions
    {
+        /// <summary>
+        /// Does a quick check on the entity's set path to ensure that it's valid and consistent
+        /// </summary>
+        /// <param name="entity"></param>
+        /// <returns></returns>
+        public static void ValidatePathWithException(this NodeDto entity)
+        {
+            //don't validate if it's empty and it has no id
+            if (entity.NodeId == default(int) && entity.Path.IsNullOrWhiteSpace())
+                return;
+
+            if (entity.Path.IsNullOrWhiteSpace())
+                throw new InvalidDataException(string.Format("The content item {0} has an empty path: {1} with parentID: {2}", entity.NodeId, entity.Path, entity.ParentId));
+
+            var pathParts = entity.Path.Split(new[] { ',' }, StringSplitOptions.RemoveEmptyEntries);
+            if (pathParts.Length < 2)
+            {
+                //a path cannot be less than 2 parts, at a minimum it must be root (-1) and it's own id
+                throw new InvalidDataException(string.Format("The content item {0} has an invalid path: {1} with parentID: {2}", entity.NodeId, entity.Path, entity.ParentId));
+            }
+
+            if (entity.ParentId != default(int) && pathParts[pathParts.Length - 2] != entity.ParentId.ToInvariantString())
+            {
+                //the 2nd last id in the path must be it's parent id
+                throw new InvalidDataException(string.Format("The content item {0} has an invalid path: {1} with parentID: {2}", entity.NodeId, entity.Path, entity.ParentId));
+            }
+        }
+
+        /// <summary>
+        /// Does a quick check on the entity's set path to ensure that it's valid and consistent
+        /// </summary>
+        /// <param name="entity"></param>
+        /// <returns></returns>
+        public static bool ValidatePath(this IUmbracoEntity entity)
+        {
+            //don't validate if it's empty and it has no id
+            if (entity.HasIdentity == false && entity.Path.IsNullOrWhiteSpace())
+                return true;
+
+            if (entity.Path.IsNullOrWhiteSpace())
+                return false;
+
+            var pathParts = entity.Path.Split(new[] { ',' }, StringSplitOptions.RemoveEmptyEntries);
+            if (pathParts.Length < 2)
+            {
+                //a path cannot be less than 2 parts, at a minimum it must be root (-1) and it's own id
+                return false;
+            }
+
+            if (entity.ParentId != default(int) && pathParts[pathParts.Length - 2] != entity.ParentId.ToInvariantString())
+            {
+                //the 2nd last id in the path must be it's parent id
+                return false;
+            }
+
+            return true;
+        }
+
+        /// <summary>
+        /// This will validate the entity's path and if it's invalid it will fix it, if fixing is required it will recursively 
+        /// check and fix all ancestors if required.
+        /// </summary>
+        /// <param name="entity"></param>
+        /// <param name="logger"></param>
+        /// <param name="getParent">A callback specified to retrieve the parent entity of the entity</param>
+        /// <param name="update">A callback specified to update a fixed entity</param>
+        public static void EnsureValidPath<T>(this T entity, 
+            ILogger logger, 
+            Func<T, T> getParent,
+            Action<T> update)
+            where T: IUmbracoEntity
+        {
+            if (entity.HasIdentity == false)
+                throw new InvalidOperationException("Could not ensure the entity path, the entity has not been assigned an identity");
+
+            if (entity.ValidatePath() == false)
+            {
+                logger.Warn(typeof(UmbracoEntityExtensions), string.Format("The content item {0} has an invalid path: {1} with parentID: {2}", entity.Id, entity.Path, entity.ParentId));
+                if (entity.ParentId == -1)
+                {
+                    entity.Path = string.Concat("-1,", entity.Id);
+                    //path changed, update it
+                    update(entity);
+                }
+                else
+                {
+                    var parent = getParent(entity);
+                    if (parent == null)
+                        throw new NullReferenceException("Could not ensure path for entity " + entity.Id + " could not resolve it's parent " + entity.ParentId);
+
+                    //the parent must also be valid!
+                    parent.EnsureValidPath(logger, getParent, update);
+
+                    entity.Path = string.Concat(parent.Path, ",", entity.Id);
+                    //path changed, update it
+                    update(entity);
+                }
+            }
+        }

        public static bool HasChildren(this IUmbracoEntity entity)
        {
@@ -7,15 +7,20 @@ using umbraco.interfaces;

 namespace Umbraco.Core.ObjectResolution
 {
-	/// <summary>
+    /// <summary>
 	/// A resolver to return all IApplicationEvents objects
 	/// </summary>
 	/// <remarks>
-	/// This is disposable because after the app has started it should be disposed to release any memory being occupied by instances.
+	/// <para>This is disposable because after the app has started it should be disposed to release any memory being occupied by instances.</para>
+	/// <para>Ordering handlers: handlers are ordered by (ascending) weight. By default, handlers from the Umbraco.* or Concorde.*
+	/// assemblies have a -100 weight whereas any other handler has a weight of +100. A custom weight can be assigned to a handler
+	/// by marking the class with the WeightAttribute. For example, the CacheRefresherEventHandler is marked with [Weight(int.MinValue)]
+	/// because its events need to run before anything else. Positive weights are considered "user-space" while negative weights are
+	/// "core". Finally, users can register a filter to process the list (after it has been ordered) and re-order it, or remove handlers.</para>
+	/// <para>BEWARE! handlers order is an important thing, and removing handlers or reordering handlers can have unexpected consequences.</para>
 	/// </remarks>
-    internal  sealed class ApplicationEventsResolver : ManyObjectsResolverBase<ApplicationEventsResolver, IApplicationEventHandler>, IDisposable
+    public sealed class ApplicationEventsResolver : ManyObjectsResolverBase<ApplicationEventsResolver, IApplicationEventHandler>, IDisposable
 	{
-
 	    private readonly LegacyStartupHandlerResolver _legacyResolver;

 	    /// <summary>
@@ -23,18 +28,18 @@ namespace Umbraco.Core.ObjectResolution
 	    /// </summary>
 	    /// <param name="logger"></param>
 	    /// <param name="applicationEventHandlers"></param>
-	    /// <param name="serviceProvider"></param>		
+	    /// <param name="serviceProvider"></param>
 	    internal ApplicationEventsResolver(IServiceProvider serviceProvider, ILogger logger, IEnumerable<Type> applicationEventHandlers)
            : base(serviceProvider, logger, applicationEventHandlers)
-		{
+	    {
            //create the legacy resolver and only include the legacy types
-	        _legacyResolver = new LegacyStartupHandlerResolver(
+            _legacyResolver = new LegacyStartupHandlerResolver(
                serviceProvider, logger,
-	            applicationEventHandlers.Where(x => !TypeHelper.IsTypeAssignableFrom<IApplicationEventHandler>(x)));
+	            applicationEventHandlers.Where(x => TypeHelper.IsTypeAssignableFrom<IApplicationEventHandler>(x) == false));
 		}

        /// <summary>
-        /// Override in order to only return types of IApplicationEventHandler and above, 
+        /// Override in order to only return types of IApplicationEventHandler and above,
        /// do not include the legacy types of IApplicationStartupHandler
        /// </summary>
        protected override IEnumerable<Type> InstanceTypes
@@ -42,14 +47,60 @@ namespace Umbraco.Core.ObjectResolution
            get { return base.InstanceTypes.Where(TypeHelper.IsTypeAssignableFrom<IApplicationEventHandler>); }
        }

-	    /// <summary>
-		/// Gets the <see cref="IApplicationEventHandler"/> implementations.
-		/// </summary>
-		public IEnumerable<IApplicationEventHandler> ApplicationEventHandlers
+	    private List<IApplicationEventHandler> _orderedAndFiltered;
+
+        /// <summary>
+        /// Gets the <see cref="IApplicationEventHandler"/> implementations.
+        /// </summary>
+        public IEnumerable<IApplicationEventHandler> ApplicationEventHandlers
 		{
-            get { return Values; }
+	        get
+	        {
+	            if (_orderedAndFiltered == null)
+	            {
+                    _orderedAndFiltered = GetSortedValues().ToList();
+                    if (FilterCollection != null)
+                        FilterCollection(_orderedAndFiltered);
+                }
+	            return _orderedAndFiltered;
+	        }
 		}

+        /// <summary>
+        /// Gets or sets a delegate to filter the event handler list (EXPERT!).
+        /// </summary>
+        /// <remarks>
+        /// <para>This can be set on startup in the pre-boot process in either a custom boot manager or global.asax (UmbracoApplication).</para>
+        /// <para>Allows custom logic to execute in order to filter and/or re-order the event handlers prior to executing.</para>
+        /// <para>To be used by custom boot sequences where the boot loader needs to remove some handlers, or raise their priority.</para>
+        /// <para>Filtering the event handler collection can have ugly consequences. Use with care.</para>
+        /// </remarks>
+        public Action<IList<IApplicationEventHandler>> FilterCollection
+	    {
+	        get { return _filterCollection; }
+	        set
+	        {
+                if (_orderedAndFiltered != null)
+                    throw new InvalidOperationException("Cannot set the FilterCollection delegate once the ApplicationEventHandlers are resolved");
+                if (_filterCollection != null)
+                    throw new InvalidOperationException("Cannot set the FilterCollection delegate once it's already been specified");
+
+                _filterCollection = value;
+	        }
+	    }
+
+	    protected override int GetObjectWeight(object o)
+	    {
+            var type = o.GetType();
+            var attr = type.GetCustomAttribute<WeightAttribute>(true);
+            if (attr != null) return attr.Weight;
+            var name = type.Assembly.FullName;
+
+            // we should really attribute all our Core handlers, so this is temp
+            var core = name.InvariantStartsWith("Umbraco.") || name.InvariantStartsWith("Concorde.");
+            return core ? -DefaultPluginWeight : DefaultPluginWeight;
+        }
+
        /// <summary>
        /// Create instances of all of the legacy startup handlers
        /// </summary>
@@ -62,11 +113,11 @@ namespace Umbraco.Core.ObjectResolution
 		protected override bool SupportsClear
 		{
            get { return false; }
-		}		
+		}

 		protected override bool SupportsInsert
 		{
-			get { return false; }			
+			get { return false; }
 		}

 	    private class LegacyStartupHandlerResolver : ManyObjectsResolverBase<ApplicationEventsResolver, IApplicationStartupHandler>, IDisposable
@@ -90,8 +141,9 @@ namespace Umbraco.Core.ObjectResolution

 	    private bool _disposed;
 		private readonly ReaderWriterLockSlim _disposalLocker = new ReaderWriterLockSlim();
+	    private Action<IList<IApplicationEventHandler>> _filterCollection;

-		/// <summary>
+	    /// <summary>
 		/// Gets a value indicating whether this instance is disposed.
 		/// </summary>
 		/// <value>
@@ -146,7 +198,8 @@ namespace Umbraco.Core.ObjectResolution
 	    {
            _legacyResolver.Dispose();
            ResetCollections();
+            _orderedAndFiltered.Clear();
+	        _orderedAndFiltered = null;
 	    }
-        
-	}
+    }
 }
@@ -17,7 +17,7 @@ namespace Umbraco.Core.ObjectResolution
    /// for when there is some processing overhead (i.e. Type finding in assemblies) to return the Types used to instantiate the instances. 
    /// In some these cases we don't want to have to type-find during application startup, only when we need to resolve the instances.</para>
    /// <para>Important notes about this resolver: it does not support Insert or Remove and therefore does not support any ordering unless 
-    /// the types are marked with the WeightedPluginAttribute.</para>
+    /// the types are marked with the WeightAttribute.</para>
    /// </remarks>
    public abstract class LazyManyObjectsResolverBase<TResolver, TResolved> : ManyObjectsResolverBase<TResolver, TResolved>
        where TResolved : class
@@ -14,18 +14,19 @@ namespace Umbraco.Core.ObjectResolution
 	/// <typeparam name="TResolver">The type of the concrete resolver class.</typeparam>
 	/// <typeparam name="TResolved">The type of the resolved objects.</typeparam>
 	public abstract class ManyObjectsResolverBase<TResolver, TResolved> : ResolverBase<TResolver>
-		where TResolved : class
+        where TResolved : class
        where TResolver : ResolverBase
-	{
-		private Lazy<IEnumerable<TResolved>> _applicationInstances;
-		private readonly ReaderWriterLockSlim _lock = new ReaderWriterLockSlim();
-		private readonly string _httpContextKey;
-		private readonly List<Type> _instanceTypes = new List<Type>();
-	    private IEnumerable<TResolved> _sortedValues;
+    {
+        private Lazy<IEnumerable<TResolved>> _applicationInstances;
+        private readonly ReaderWriterLockSlim _lock = new ReaderWriterLockSlim();
+        private readonly string _httpContextKey;
+        private readonly List<Type> _instanceTypes = new List<Type>();
+        private IEnumerable<TResolved> _sortedValues;
+        private readonly Func<HttpContextBase> _httpContextGetter;

-		private int _defaultPluginWeight = 10;
+        private int _defaultPluginWeight = 100;

-		#region Constructors
+        #region Constructors

        /// <summary>
        /// Initializes a new instance of the <see cref="ManyObjectsResolverBase{TResolver, TResolved}"/> class with an empty list of objects,
@@ -42,12 +43,7 @@ namespace Umbraco.Core.ObjectResolution
            if (logger == null) throw new ArgumentNullException("logger");
            CanResolveBeforeFrozen = false;
            if (scope == ObjectLifetimeScope.HttpRequest)
-            {
-                if (HttpContext.Current == null)
-                    throw new InvalidOperationException("Use alternative constructor accepting a HttpContextBase object in order to set the lifetime scope to HttpRequest when HttpContext.Current is null");
-
-                CurrentHttpContext = new HttpContextWrapper(HttpContext.Current);
-            }
+                _httpContextGetter = () => new HttpContextWrapper(HttpContext.Current);

            ServiceProvider = serviceProvider;
            Logger = logger;
@@ -61,11 +57,11 @@ namespace Umbraco.Core.ObjectResolution

        [EditorBrowsable(EditorBrowsableState.Never)]
        [Obsolete("Use ctor specifying IServiceProvider instead")]
-		protected ManyObjectsResolverBase(ObjectLifetimeScope scope = ObjectLifetimeScope.Application)
+        protected ManyObjectsResolverBase(ObjectLifetimeScope scope = ObjectLifetimeScope.Application)
            : this(new ActivatorServiceProvider(), LoggerResolver.Current.Logger, scope)
-		{
-			
-		}
+        {
+
+        }

        /// <summary>
        /// Initializes a new instance of the <see cref="ManyObjectsResolverBase{TResolver, TResolved}"/> class with an empty list of objects,
@@ -76,19 +72,19 @@ namespace Umbraco.Core.ObjectResolution
        /// <param name="httpContext">The HttpContextBase corresponding to the HttpRequest.</param>
        /// <exception cref="ArgumentNullException"><paramref name="httpContext"/> is <c>null</c>.</exception>
        protected ManyObjectsResolverBase(IServiceProvider serviceProvider, ILogger logger, HttpContextBase httpContext)
-		{
+        {
            if (serviceProvider == null) throw new ArgumentNullException("serviceProvider");
            if (httpContext == null) throw new ArgumentNullException("httpContext");
            CanResolveBeforeFrozen = false;
            Logger = logger;
-			LifetimeScope = ObjectLifetimeScope.HttpRequest;
-			_httpContextKey = GetType().FullName;
+            LifetimeScope = ObjectLifetimeScope.HttpRequest;
+            _httpContextKey = GetType().FullName;
            ServiceProvider = serviceProvider;
-            CurrentHttpContext = httpContext;
-			_instanceTypes = new List<Type>();
+            _httpContextGetter = () => httpContext;
+            _instanceTypes = new List<Type>();

            InitializeAppInstances();
-		}
+        }

        [EditorBrowsable(EditorBrowsableState.Never)]
        [Obsolete("Use ctor specifying IServiceProvider instead")]
@@ -110,57 +106,66 @@ namespace Umbraco.Core.ObjectResolution
        /// <exception cref="InvalidOperationException"><paramref name="scope"/> is per HttpRequest but the current HttpContext is null.</exception>
        protected ManyObjectsResolverBase(IServiceProvider serviceProvider, ILogger logger, IEnumerable<Type> value, ObjectLifetimeScope scope = ObjectLifetimeScope.Application)
            : this(serviceProvider, logger, scope)
-		{
-			_instanceTypes = value.ToList();
-		}
+        {
+            _instanceTypes = value.ToList();
+        }

        [EditorBrowsable(EditorBrowsableState.Never)]
        [Obsolete("Use ctor specifying IServiceProvider instead")]
        protected ManyObjectsResolverBase(IEnumerable<Type> value, ObjectLifetimeScope scope = ObjectLifetimeScope.Application)
            : this(new ActivatorServiceProvider(), LoggerResolver.Current.Logger, value, scope)
        {
-            
+
        }

-		/// <summary>
-		/// Initializes a new instance of the <see cref="ManyObjectsResolverBase{TResolver, TResolved}"/> class with an initial list of objects,
-		/// with creation of objects based on an HttpRequest lifetime scope.
-		/// </summary>
-		/// <param name="httpContext">The HttpContextBase corresponding to the HttpRequest.</param>
-		/// <param name="value">The list of object types.</param>
-		/// <exception cref="ArgumentNullException"><paramref name="httpContext"/> is <c>null</c>.</exception>
+        /// <summary>
+        /// Initializes a new instance of the <see cref="ManyObjectsResolverBase{TResolver, TResolved}"/> class with an initial list of objects,
+        /// with creation of objects based on an HttpRequest lifetime scope.
+        /// </summary>
+        /// <param name="httpContext">The HttpContextBase corresponding to the HttpRequest.</param>
+        /// <param name="value">The list of object types.</param>
+        /// <exception cref="ArgumentNullException"><paramref name="httpContext"/> is <c>null</c>.</exception>
        [Obsolete("Use ctor specifying IServiceProvider instead")]
        protected ManyObjectsResolverBase(HttpContextBase httpContext, IEnumerable<Type> value)
            : this(new ActivatorServiceProvider(), LoggerResolver.Current.Logger, httpContext)
-		{
-			_instanceTypes = value.ToList();
-		} 
-		#endregion
+        {
+            _instanceTypes = value.ToList();
+        }
+        #endregion

        private void InitializeAppInstances()
        {
            _applicationInstances = new Lazy<IEnumerable<TResolved>>(() => CreateInstances().ToArray());
        }

-		/// <summary>
-		/// Gets or sets a value indicating whether the resolver can resolve objects before resolution is frozen.
-		/// </summary>
-		/// <remarks>This is false by default and is used for some special internal resolvers.</remarks>
-		internal bool CanResolveBeforeFrozen { get; set; }
+        /// <summary>
+        /// Gets or sets a value indicating whether the resolver can resolve objects before resolution is frozen.
+        /// </summary>
+        /// <remarks>This is false by default and is used for some special internal resolvers.</remarks>
+        internal bool CanResolveBeforeFrozen { get; set; }

-		/// <summary>
-		/// Gets the list of types to create instances from.
-		/// </summary>
-		protected virtual IEnumerable<Type> InstanceTypes
-		{
-			get { return _instanceTypes; }
-		}
+        /// <summary>
+        /// Gets the list of types to create instances from.
+        /// </summary>
+        protected virtual IEnumerable<Type> InstanceTypes
+        {
+            get { return _instanceTypes; }
+        }

-		/// <summary>
-		/// Gets or sets the <see cref="HttpContextBase"/> used to initialize this object, if any.
-		/// </summary>
-		/// <remarks>If not null, then <c>LifetimeScope</c> will be <c>ObjectLifetimeScope.HttpRequest</c>.</remarks>
-		protected HttpContextBase CurrentHttpContext { get; private set; }
+        /// <summary>
+        /// Gets or sets the <see cref="HttpContextBase"/> used to initialize this object, if any.
+        /// </summary>
+        /// <remarks>If not null, then <c>LifetimeScope</c> will be <c>ObjectLifetimeScope.HttpRequest</c>.</remarks>
+        protected HttpContextBase CurrentHttpContext
+        {
+            get
+            {
+                var context = _httpContextGetter == null ? null : _httpContextGetter();
+                if (context == null)
+                    throw new InvalidOperationException("Cannot use this resolver with lifetime 'HttpRequest' when there is no current HttpContext. Either use the ctor accepting an HttpContextBase, or use the resolver from within a request exclusively.");
+                return context;
+            }
+        }

        /// <summary>
        /// Returns the service provider used to instantiate objects
@@ -174,16 +179,16 @@ namespace Umbraco.Core.ObjectResolution
 		/// </summary>
 		protected ObjectLifetimeScope LifetimeScope { get; private set; }

-		/// <summary>
-		/// Gets the resolved object instances, sorted by weight.
-		/// </summary>
-		/// <returns>The sorted resolved object instances.</returns>
-		/// <remarks>
-		/// <para>The order is based upon the <c>WeightedPluginAttribute</c> and <c>DefaultPluginWeight</c>.</para>
-		/// <para>Weights are sorted ascendingly (lowest weights come first).</para>
-		/// </remarks>
-		protected IEnumerable<TResolved> GetSortedValues()
-		{
+        /// <summary>
+        /// Gets the resolved object instances, sorted by weight.
+        /// </summary>
+        /// <returns>The sorted resolved object instances.</returns>
+        /// <remarks>
+        /// <para>The order is based upon the <c>WeightAttribute</c> and <c>DefaultPluginWeight</c>.</para>
+        /// <para>Weights are sorted ascendingly (lowest weights come first).</para>
+        /// </remarks>
+        protected IEnumerable<TResolved> GetSortedValues()
+        {
            if (_sortedValues == null)
            {
                var values = Values.ToList();
@@ -191,18 +196,18 @@ namespace Umbraco.Core.ObjectResolution
                _sortedValues = values;
            }
            return _sortedValues;
-		}
+        }

-		/// <summary>
-		/// Gets or sets the default type weight.
-		/// </summary>
-		/// <remarks>Determines the weight of types that do not have a <c>WeightedPluginAttribute</c> set on 
-		/// them, when calling <c>GetSortedValues</c>.</remarks>
-		protected virtual int DefaultPluginWeight
-		{
-			get { return _defaultPluginWeight; }
-			set { _defaultPluginWeight = value; }
-		}
+        /// <summary>
+        /// Gets or sets the default type weight.
+        /// </summary>
+        /// <remarks>Determines the weight of types that do not have a <c>WeightAttribute</c> set on
+        /// them, when calling <c>GetSortedValues</c>.</remarks>
+        protected virtual int DefaultPluginWeight
+        {
+            get { return _defaultPluginWeight; }
+            set { _defaultPluginWeight = value; }
+        }

        /// <summary>
        /// Returns the weight of an object for user with GetSortedValues
@@ -210,22 +215,22 @@ namespace Umbraco.Core.ObjectResolution
        /// <param name="o"></param>
        /// <returns></returns>
 		protected virtual int GetObjectWeight(object o)
-		{
-			var type = o.GetType();
-			var attr = type.GetCustomAttribute<WeightedPluginAttribute>(true);
-			return attr == null ? DefaultPluginWeight : attr.Weight;
-		}
+        {
+            var type = o.GetType();
+            var attr = type.GetCustomAttribute<WeightAttribute>(true);
+            return attr == null ? DefaultPluginWeight : attr.Weight;
+        }

-		/// <summary>
-		/// Gets the resolved object instances.
-		/// </summary>
-		/// <exception cref="InvalidOperationException"><c>CanResolveBeforeFrozen</c> is false, and resolution is not frozen.</exception>
-		protected IEnumerable<TResolved> Values
-		{
-			get
-			{
-			    using (Resolution.Reader(CanResolveBeforeFrozen))
-			    {
+        /// <summary>
+        /// Gets the resolved object instances.
+        /// </summary>
+        /// <exception cref="InvalidOperationException"><c>CanResolveBeforeFrozen</c> is false, and resolution is not frozen.</exception>
+        protected IEnumerable<TResolved> Values
+        {
+            get
+            {
+                using (Resolution.Reader(CanResolveBeforeFrozen))
+                {
                    // note: we apply .ToArray() to the output of CreateInstance() because that is an IEnumerable that
                    // comes from the PluginManager we want to be _sure_ that it's not a Linq of some sort, but the
                    // instances have actually been instanciated when we return.
@@ -247,7 +252,7 @@ namespace Umbraco.Core.ObjectResolution
                                CurrentHttpContext.Items[_httpContextKey] = instances;
                            }
                            return (TResolved[])CurrentHttpContext.Items[_httpContextKey];
-                            
+
                        case ObjectLifetimeScope.Application:

                            return _applicationInstances.Value;
@@ -258,132 +263,132 @@ namespace Umbraco.Core.ObjectResolution
                            return CreateInstances().ToArray();
                    }
                }
-			}
-		}
+            }
+        }

-		/// <summary>
-		/// Creates the object instances for the types contained in the types collection.
-		/// </summary>
-		/// <returns>A list of objects of type <typeparamref name="TResolved"/>.</returns>
-		protected virtual IEnumerable<TResolved> CreateInstances()
-		{
-			return ServiceProvider.CreateInstances<TResolved>(InstanceTypes, Logger);
-		}
+        /// <summary>
+        /// Creates the object instances for the types contained in the types collection.
+        /// </summary>
+        /// <returns>A list of objects of type <typeparamref name="TResolved"/>.</returns>
+        protected virtual IEnumerable<TResolved> CreateInstances()
+        {
+            return ServiceProvider.CreateInstances<TResolved>(InstanceTypes, Logger);
+        }

-		#region Types collection manipulation
+        #region Types collection manipulation

-		/// <summary>
-		/// Removes a type.
-		/// </summary>
-		/// <param name="value">The type to remove.</param>
-		/// <exception cref="InvalidOperationException">the resolver does not support removing types, or 
-		/// the type is not a valid type for the resolver.</exception>
-		public virtual void RemoveType(Type value)
-		{
-			EnsureSupportsRemove();
+        /// <summary>
+        /// Removes a type.
+        /// </summary>
+        /// <param name="value">The type to remove.</param>
+        /// <exception cref="InvalidOperationException">the resolver does not support removing types, or
+        /// the type is not a valid type for the resolver.</exception>
+        public virtual void RemoveType(Type value)
+        {
+            EnsureSupportsRemove();

-			using (Resolution.Configuration)
-			using (var l = new UpgradeableReadLock(_lock))
-			{
-				EnsureCorrectType(value);
+            using (Resolution.Configuration)
+            using (var l = new UpgradeableReadLock(_lock))
+            {
+                EnsureCorrectType(value);

-				l.UpgradeToWriteLock();
-				_instanceTypes.Remove(value);
-			}
-		}
+                l.UpgradeToWriteLock();
+                _instanceTypes.Remove(value);
+            }
+        }

-		/// <summary>
-		/// Removes a type.
-		/// </summary>
-		/// <typeparam name="T">The type to remove.</typeparam>
-		/// <exception cref="InvalidOperationException">the resolver does not support removing types, or 
-		/// the type is not a valid type for the resolver.</exception>
-		public void RemoveType<T>()
+        /// <summary>
+        /// Removes a type.
+        /// </summary>
+        /// <typeparam name="T">The type to remove.</typeparam>
+        /// <exception cref="InvalidOperationException">the resolver does not support removing types, or
+        /// the type is not a valid type for the resolver.</exception>
+        public void RemoveType<T>()
            where T : TResolved
-		{
-			RemoveType(typeof(T));
-		}
+        {
+            RemoveType(typeof(T));
+        }

-		/// <summary>
-		/// Adds types.
-		/// </summary>
-		/// <param name="types">The types to add.</param>
-		/// <remarks>The types are appended at the end of the list.</remarks>
-		/// <exception cref="InvalidOperationException">the resolver does not support adding types, or 
-		/// a type is not a valid type for the resolver, or a type is already in the collection of types.</exception>
-		protected void AddTypes(IEnumerable<Type> types)
-		{
-			EnsureSupportsAdd();
+        /// <summary>
+        /// Adds types.
+        /// </summary>
+        /// <param name="types">The types to add.</param>
+        /// <remarks>The types are appended at the end of the list.</remarks>
+        /// <exception cref="InvalidOperationException">the resolver does not support adding types, or
+        /// a type is not a valid type for the resolver, or a type is already in the collection of types.</exception>
+        protected void AddTypes(IEnumerable<Type> types)
+        {
+            EnsureSupportsAdd();

-			using (Resolution.Configuration)
-			using (new WriteLock(_lock))
-			{
-				foreach(var t in types)
-				{
-					EnsureCorrectType(t);
+            using (Resolution.Configuration)
+            using (new WriteLock(_lock))
+            {
+                foreach (var t in types)
+                {
+                    EnsureCorrectType(t);
                    if (_instanceTypes.Contains(t))
-					{
-						throw new InvalidOperationException(string.Format(
-							"Type {0} is already in the collection of types.", t.FullName));
-					}
-					_instanceTypes.Add(t);	
-				}				
-			}
-		}
+                    {
+                        throw new InvalidOperationException(string.Format(
+                            "Type {0} is already in the collection of types.", t.FullName));
+                    }
+                    _instanceTypes.Add(t);
+                }
+            }
+        }

-		/// <summary>
-		/// Adds a type.
-		/// </summary>
-		/// <param name="value">The type to add.</param>
-		/// <remarks>The type is appended at the end of the list.</remarks>
-		/// <exception cref="InvalidOperationException">the resolver does not support adding types, or 
-		/// the type is not a valid type for the resolver, or the type is already in the collection of types.</exception>
-		public virtual void AddType(Type value)
-		{
-			EnsureSupportsAdd();
+        /// <summary>
+        /// Adds a type.
+        /// </summary>
+        /// <param name="value">The type to add.</param>
+        /// <remarks>The type is appended at the end of the list.</remarks>
+        /// <exception cref="InvalidOperationException">the resolver does not support adding types, or
+        /// the type is not a valid type for the resolver, or the type is already in the collection of types.</exception>
+        public virtual void AddType(Type value)
+        {
+            EnsureSupportsAdd();

-			using (Resolution.Configuration)
-			using (var l = new UpgradeableReadLock(_lock))
-			{
-				EnsureCorrectType(value);
+            using (Resolution.Configuration)
+            using (var l = new UpgradeableReadLock(_lock))
+            {
+                EnsureCorrectType(value);
                if (_instanceTypes.Contains(value))
-				{
-					throw new InvalidOperationException(string.Format(
-						"Type {0} is already in the collection of types.", value.FullName));
-				}
+                {
+                    throw new InvalidOperationException(string.Format(
+                        "Type {0} is already in the collection of types.", value.FullName));
+                }

-				l.UpgradeToWriteLock();
-				_instanceTypes.Add(value);
-			}
-		}
+                l.UpgradeToWriteLock();
+                _instanceTypes.Add(value);
+            }
+        }

-		/// <summary>
-		/// Adds a type.
-		/// </summary>
-		/// <typeparam name="T">The type to add.</typeparam>
-		/// <remarks>The type is appended at the end of the list.</remarks>
-		/// <exception cref="InvalidOperationException">the resolver does not support adding types, or 
-		/// the type is not a valid type for the resolver, or the type is already in the collection of types.</exception>
-		public void AddType<T>()
+        /// <summary>
+        /// Adds a type.
+        /// </summary>
+        /// <typeparam name="T">The type to add.</typeparam>
+        /// <remarks>The type is appended at the end of the list.</remarks>
+        /// <exception cref="InvalidOperationException">the resolver does not support adding types, or
+        /// the type is not a valid type for the resolver, or the type is already in the collection of types.</exception>
+        public void AddType<T>()
            where T : TResolved
-		{
-			AddType(typeof(T));
-		}
+        {
+            AddType(typeof(T));
+        }

-		/// <summary>
-		/// Clears the list of types
-		/// </summary>
-		/// <exception cref="InvalidOperationException">the resolver does not support clearing types.</exception>
-		public virtual void Clear()
-		{
-			EnsureSupportsClear();
+        /// <summary>
+        /// Clears the list of types
+        /// </summary>
+        /// <exception cref="InvalidOperationException">the resolver does not support clearing types.</exception>
+        public virtual void Clear()
+        {
+            EnsureSupportsClear();

-			using (Resolution.Configuration)
-			using (new WriteLock(_lock))
-			{
-				_instanceTypes.Clear();
-			}
-		}
+            using (Resolution.Configuration)
+            using (new WriteLock(_lock))
+            {
+                _instanceTypes.Clear();
+            }
+        }

        /// <summary>
        /// WARNING! Do not use this unless you know what you are doing, clear all types registered and instances
@@ -399,38 +404,38 @@ namespace Umbraco.Core.ObjectResolution
            }
        }

-		/// <summary>
-		/// Inserts a type at the specified index.
-		/// </summary>
-		/// <param name="index">The zero-based index at which the type should be inserted.</param>
-		/// <param name="value">The type to insert.</param>
-		/// <exception cref="InvalidOperationException">the resolver does not support inserting types, or 
-		/// the type is not a valid type for the resolver, or the type is already in the collection of types.</exception>
-		/// <exception cref="ArgumentOutOfRangeException"><paramref name="index"/> is out of range.</exception>
-		public virtual void InsertType(int index, Type value)
-		{			
-			EnsureSupportsInsert();
+        /// <summary>
+        /// Inserts a type at the specified index.
+        /// </summary>
+        /// <param name="index">The zero-based index at which the type should be inserted.</param>
+        /// <param name="value">The type to insert.</param>
+        /// <exception cref="InvalidOperationException">the resolver does not support inserting types, or
+        /// the type is not a valid type for the resolver, or the type is already in the collection of types.</exception>
+        /// <exception cref="ArgumentOutOfRangeException"><paramref name="index"/> is out of range.</exception>
+        public virtual void InsertType(int index, Type value)
+        {
+            EnsureSupportsInsert();

-			using (Resolution.Configuration)
-			using (var l = new UpgradeableReadLock(_lock))
-			{
-				EnsureCorrectType(value);
+            using (Resolution.Configuration)
+            using (var l = new UpgradeableReadLock(_lock))
+            {
+                EnsureCorrectType(value);
                if (_instanceTypes.Contains(value))
-				{
-					throw new InvalidOperationException(string.Format(
-						"Type {0} is already in the collection of types.", value.FullName));
-				}
+                {
+                    throw new InvalidOperationException(string.Format(
+                        "Type {0} is already in the collection of types.", value.FullName));
+                }

-				l.UpgradeToWriteLock();
-				_instanceTypes.Insert(index, value);
-			}
-		}
+                l.UpgradeToWriteLock();
+                _instanceTypes.Insert(index, value);
+            }
+        }

        /// <summary>
        /// Inserts a type at the beginning of the list.
        /// </summary>
        /// <param name="value">The type to insert.</param>
-        /// <exception cref="InvalidOperationException">the resolver does not support inserting types, or 
+        /// <exception cref="InvalidOperationException">the resolver does not support inserting types, or
        /// the type is not a valid type for the resolver, or the type is already in the collection of types.</exception>
        public virtual void InsertType(Type value)
        {
@@ -445,9 +450,9 @@ namespace Umbraco.Core.ObjectResolution
 		/// <exception cref="ArgumentOutOfRangeException"><paramref name="index"/> is out of range.</exception>
 		public void InsertType<T>(int index)
            where T : TResolved
-		{
-			InsertType(index, typeof(T));
-		}
+        {
+            InsertType(index, typeof(T));
+        }

        /// <summary>
        /// Inserts a type at the beginning of the list.
@@ -458,68 +463,68 @@ namespace Umbraco.Core.ObjectResolution
        {
            InsertType(0, typeof(T));
        }
-        
-        /// <summary>
-		/// Inserts a type before a specified, already existing type.
-		/// </summary>
-		/// <param name="existingType">The existing type before which to insert.</param>
-		/// <param name="value">The type to insert.</param>
-		/// <exception cref="InvalidOperationException">the resolver does not support inserting types, or 
-		/// one of the types is not a valid type for the resolver, or the existing type is not in the collection,
-		/// or the new type is already in the collection of types.</exception>
-		public virtual void InsertTypeBefore(Type existingType, Type value)
-		{
-			EnsureSupportsInsert();

-			using (Resolution.Configuration)
-			using (var l = new UpgradeableReadLock(_lock))
-			{
-				EnsureCorrectType(existingType);
-				EnsureCorrectType(value);
+        /// <summary>
+        /// Inserts a type before a specified, already existing type.
+        /// </summary>
+        /// <param name="existingType">The existing type before which to insert.</param>
+        /// <param name="value">The type to insert.</param>
+        /// <exception cref="InvalidOperationException">the resolver does not support inserting types, or
+        /// one of the types is not a valid type for the resolver, or the existing type is not in the collection,
+        /// or the new type is already in the collection of types.</exception>
+        public virtual void InsertTypeBefore(Type existingType, Type value)
+        {
+            EnsureSupportsInsert();
+
+            using (Resolution.Configuration)
+            using (var l = new UpgradeableReadLock(_lock))
+            {
+                EnsureCorrectType(existingType);
+                EnsureCorrectType(value);
                if (_instanceTypes.Contains(existingType) == false)
-				{
-					throw new InvalidOperationException(string.Format(
-						"Type {0} is not in the collection of types.", existingType.FullName));
-				}
+                {
+                    throw new InvalidOperationException(string.Format(
+                        "Type {0} is not in the collection of types.", existingType.FullName));
+                }
                if (_instanceTypes.Contains(value))
-				{
-					throw new InvalidOperationException(string.Format(
-						"Type {0} is already in the collection of types.", value.FullName));
-				}
+                {
+                    throw new InvalidOperationException(string.Format(
+                        "Type {0} is already in the collection of types.", value.FullName));
+                }
                int index = _instanceTypes.IndexOf(existingType);

-				l.UpgradeToWriteLock();
-				_instanceTypes.Insert(index, value);
-			}
-		}
+                l.UpgradeToWriteLock();
+                _instanceTypes.Insert(index, value);
+            }
+        }

-		/// <summary>
-		/// Inserts a type before a specified, already existing type.
-		/// </summary>
-		/// <typeparam name="TExisting">The existing type before which to insert.</typeparam>
-		/// <typeparam name="T">The type to insert.</typeparam>
-		/// <exception cref="InvalidOperationException">the resolver does not support inserting types, or 
-		/// one of the types is not a valid type for the resolver, or the existing type is not in the collection,
-		/// or the new type is already in the collection of types.</exception>
-		public void InsertTypeBefore<TExisting, T>()
+        /// <summary>
+        /// Inserts a type before a specified, already existing type.
+        /// </summary>
+        /// <typeparam name="TExisting">The existing type before which to insert.</typeparam>
+        /// <typeparam name="T">The type to insert.</typeparam>
+        /// <exception cref="InvalidOperationException">the resolver does not support inserting types, or
+        /// one of the types is not a valid type for the resolver, or the existing type is not in the collection,
+        /// or the new type is already in the collection of types.</exception>
+        public void InsertTypeBefore<TExisting, T>()
            where TExisting : TResolved
            where T : TResolved
-		{
-			InsertTypeBefore(typeof(TExisting), typeof(T));
-		}
+        {
+            InsertTypeBefore(typeof(TExisting), typeof(T));
+        }

-		/// <summary>
-		/// Returns a value indicating whether the specified type is already in the collection of types.
-		/// </summary>
-		/// <param name="value">The type to look for.</param>
-		/// <returns>A value indicating whether the type is already in the collection of types.</returns>
-		public virtual bool ContainsType(Type value)
-		{
-			using (new ReadLock(_lock))
-			{
-				return _instanceTypes.Contains(value);
-			}
-		}
+        /// <summary>
+        /// Returns a value indicating whether the specified type is already in the collection of types.
+        /// </summary>
+        /// <param name="value">The type to look for.</param>
+        /// <returns>A value indicating whether the type is already in the collection of types.</returns>
+        public virtual bool ContainsType(Type value)
+        {
+            using (new ReadLock(_lock))
+            {
+                return _instanceTypes.Contains(value);
+            }
+        }

        /// <summary>
        /// Gets the types in the collection of types.
@@ -536,27 +541,27 @@ namespace Umbraco.Core.ObjectResolution
            return types;
        }

-		/// <summary>
-		/// Returns a value indicating whether the specified type is already in the collection of types.
-		/// </summary>
-		/// <typeparam name="T">The type to look for.</typeparam>
-		/// <returns>A value indicating whether the type is already in the collection of types.</returns>
-		public bool ContainsType<T>()
+        /// <summary>
+        /// Returns a value indicating whether the specified type is already in the collection of types.
+        /// </summary>
+        /// <typeparam name="T">The type to look for.</typeparam>
+        /// <returns>A value indicating whether the type is already in the collection of types.</returns>
+        public bool ContainsType<T>()
            where T : TResolved
-		{
-			return ContainsType(typeof(T));
-		}
+        {
+            return ContainsType(typeof(T));
+        }

-		#endregion
+        #endregion

-		/// <summary>
-		/// Returns a WriteLock to use when modifying collections
-		/// </summary>
-		/// <returns></returns>
-		protected WriteLock GetWriteLock()
-		{
-			return new WriteLock(_lock);
-		}
+        /// <summary>
+        /// Returns a WriteLock to use when modifying collections
+        /// </summary>
+        /// <returns></returns>
+        protected WriteLock GetWriteLock()
+        {
+            return new WriteLock(_lock);
+        }

        #region Type utilities

@@ -581,70 +586,71 @@ namespace Umbraco.Core.ObjectResolution
        /// </summary>
        /// <exception cref="InvalidOperationException">The resolver does not support removing types.</exception>
        protected void EnsureSupportsRemove()
-		{
-			if (SupportsRemove == false)
+        {
+            if (SupportsRemove == false)
                throw new InvalidOperationException("This resolver does not support removing types");
-		}
+        }

        /// <summary>
        /// Ensures that the resolver supports clearing types.
        /// </summary>
        /// <exception cref="InvalidOperationException">The resolver does not support clearing types.</exception>
-        protected void EnsureSupportsClear()		{
-			if (SupportsClear == false)
+        protected void EnsureSupportsClear()
+        {
+            if (SupportsClear == false)
                throw new InvalidOperationException("This resolver does not support clearing types");
-		}
+        }

        /// <summary>
        /// Ensures that the resolver supports adding types.
        /// </summary>
        /// <exception cref="InvalidOperationException">The resolver does not support adding types.</exception>
        protected void EnsureSupportsAdd()
-		{
-			if (SupportsAdd == false)
+        {
+            if (SupportsAdd == false)
                throw new InvalidOperationException("This resolver does not support adding new types");
-		}
+        }

        /// <summary>
        /// Ensures that the resolver supports inserting types.
        /// </summary>
        /// <exception cref="InvalidOperationException">The resolver does not support inserting types.</exception>
        protected void EnsureSupportsInsert()
-		{
-			if (SupportsInsert == false)
+        {
+            if (SupportsInsert == false)
                throw new InvalidOperationException("This resolver does not support inserting new types");
-		}
+        }

        /// <summary>
        /// Gets a value indicating whether the resolver supports adding types.
        /// </summary>
 		protected virtual bool SupportsAdd
-		{
-			get { return true; }
-		}
+        {
+            get { return true; }
+        }

        /// <summary>
        /// Gets a value indicating whether the resolver supports inserting types.
        /// </summary>
        protected virtual bool SupportsInsert
-		{
-			get { return true; }
-		}
+        {
+            get { return true; }
+        }

        /// <summary>
        /// Gets a value indicating whether the resolver supports clearing types.
        /// </summary>
        protected virtual bool SupportsClear
-		{
-			get { return true; }
-		}
+        {
+            get { return true; }
+        }

        /// <summary>
        /// Gets a value indicating whether the resolver supports removing types.
        /// </summary>
        protected virtual bool SupportsRemove
-		{
-			get { return true; }
+        {
+            get { return true; }
        }

        #endregion
@@ -1,25 +1,25 @@
-using System;
-
-namespace Umbraco.Core.ObjectResolution
-{
-	/// <summary>
-	/// Indicates the relative weight of a resolved object type.
-	/// </summary>
-	[AttributeUsage(AttributeTargets.Class, AllowMultiple = false)]
-	internal class WeightedPluginAttribute : Attribute
-	{
-		/// <summary>
-		/// Initializes a new instance of the <see cref="WeightedPluginAttribute"/> class with a weight.
-		/// </summary>
-		/// <param name="weight">The object type weight.</param>
-		public WeightedPluginAttribute(int weight)
-		{
-			Weight = weight;
-		}
-
-		/// <summary>
-		/// Gets or sets the weight of the object type.
-		/// </summary>
-		public int Weight { get; private set; }
-	}
+using System;
+
+namespace Umbraco.Core.ObjectResolution
+{
+	/// <summary>
+	/// Indicates the relative weight of a resolved object type.
+	/// </summary>
+	[AttributeUsage(AttributeTargets.Class, AllowMultiple = false, Inherited = false)]
+	public class WeightAttribute : Attribute
+	{
+        /// <summary>
+		/// Initializes a new instance of the <see cref="WeightAttribute"/> class with a weight.
+		/// </summary>
+		/// <param name="weight">The object type weight.</param>
+	    public WeightAttribute(int weight)
+	    {
+	        Weight = weight;
+	    }
+
+		/// <summary>
+		/// Gets or sets the weight of the object type.
+		/// </summary>
+		public int Weight { get; private set; }
+	}
 }
@@ -9,7 +9,7 @@ namespace Umbraco.Core.Persistence.DatabaseModelDefinitions
 {
    internal static class DefinitionFactory
    {
-        public static TableDefinition GetTableDefinition(Type modelType)
+        public static TableDefinition GetTableDefinition(ISqlSyntaxProvider syntaxProvider, Type modelType)
        {
            //Looks for PetaPoco's TableNameAtribute for the name of the table
            //If no attribute is set we use the name of the Type as the default convention
@@ -32,7 +32,7 @@ namespace Umbraco.Core.Persistence.DatabaseModelDefinitions
                //Otherwise use the name of the property itself as the default convention
                var columnAttribute = propertyInfo.FirstAttribute<ColumnAttribute>();
                string columnName = columnAttribute != null ? columnAttribute.Name : propertyInfo.Name;
-                var columnDefinition = GetColumnDefinition(modelType, propertyInfo, columnName, tableName);
+                var columnDefinition = GetColumnDefinition(syntaxProvider, modelType, propertyInfo, columnName, tableName);
                tableDefinition.Columns.Add(columnDefinition);

                //Creates a foreignkey definition and adds it to the collection on the table definition
@@ -58,7 +58,7 @@ namespace Umbraco.Core.Persistence.DatabaseModelDefinitions
            return tableDefinition;
        }

-        public static ColumnDefinition GetColumnDefinition(Type modelType, PropertyInfo propertyInfo, string columnName, string tableName)
+        public static ColumnDefinition GetColumnDefinition(ISqlSyntaxProvider syntaxProvider, Type modelType, PropertyInfo propertyInfo, string columnName, string tableName)
        {
            var definition = new ColumnDefinition{ Name = columnName, TableName = tableName, ModificationType = ModificationType.Create };

@@ -110,7 +110,7 @@ namespace Umbraco.Core.Persistence.DatabaseModelDefinitions
            {
                //Special case for MySQL as it can't have multiple default DateTime values, which 
                //is what the umbracoServer table definition is trying to create
-                if (SqlSyntaxContext.SqlSyntaxProvider is MySqlSyntaxProvider && definition.TableName == "umbracoServer" &&
+                if (syntaxProvider is MySqlSyntaxProvider && definition.TableName == "umbracoServer" &&
                        definition.TableName.ToLowerInvariant() == "lastNotifiedDate".ToLowerInvariant())
                    return definition;

@@ -99,7 +99,7 @@ namespace Umbraco.Core.Persistence

        public void CreateTable(bool overwrite, Type modelType)
        {
-            var tableDefinition = DefinitionFactory.GetTableDefinition(modelType);
+            var tableDefinition = DefinitionFactory.GetTableDefinition(_syntaxProvider, modelType);
            var tableName = tableDefinition.Name;

            string createSql = _syntaxProvider.Format(tableDefinition);
@@ -4,9 +4,11 @@ using System.Configuration;
 using System.Data;
 using System.Data.Common;
 using System.Data.SqlClient;
+using System.Data.SqlServerCe;
 using System.Linq;
 using System.Text;
 using System.Threading.Tasks;
+using MySql.Data.MySqlClient;
 using Umbraco.Core.Logging; 

 namespace Umbraco.Core.Persistence
@@ -69,6 +71,38 @@ namespace Umbraco.Core.Persistence
            }
        }

+        public static string GetConnStringExSecurityInfo(this IDbConnection connection)
+        {
+            try
+            {
+                if (connection is SqlConnection)
+                {
+                    var builder = new SqlConnectionStringBuilder(connection.ConnectionString);
+                    return string.Format("DataSource: {0}, InitialCatalog: {1}", builder.DataSource, builder.InitialCatalog);
+                }
+
+                if (connection is SqlCeConnection)
+                {
+                    var builder = new SqlCeConnectionStringBuilder(connection.ConnectionString);
+                    return string.Format("DataSource: {0}", builder.DataSource);
+                }
+
+                if (connection is MySqlConnection)
+                {
+                    var builder = new MySqlConnectionStringBuilder(connection.ConnectionString);
+                    return string.Format("Server: {0}, Database: {1}", builder.Server, builder.Database);
+                }
+            }
+            catch (Exception ex)
+            {
+                LogHelper.WarnWithException(typeof(DbConnectionExtensions),
+                    "Could not resolve connection string parameters", ex);
+                return "(Could not resolve)";
+            }
+
+            throw new ArgumentException(string.Format("The connection type {0} is not supported", connection.GetType()));
+        }
+
        public static bool IsAvailable(this IDbConnection connection)
        {
            try
@@ -79,7 +113,8 @@ namespace Umbraco.Core.Persistence
            catch (DbException exc)
            {
                // Don't swallow this error, the exception is super handy for knowing "why" its not available
-                LogHelper.WarnWithException<IDbConnection>("Configured database is reporting as not being available!", exc);
+                LogHelper.WarnWithException(typeof(DbConnectionExtensions),
+                    "Configured database is reporting as not being available! {0}", exc, connection.GetConnStringExSecurityInfo);
                return false;
            }

@@ -27,15 +27,28 @@ namespace Umbraco.Core.Persistence.Factories

        #region Implementation of IEntityFactory<IContent,DocumentDto>

-        public IContent BuildEntity(DocumentDto dto)
+        /// <summary>
+        /// Builds a IContent item from the dto(s) and content type
+        /// </summary>
+        /// <param name="dto">
+        /// This DTO can contain all of the information to build an IContent item, however in cases where multiple entities are being built,
+        /// a separate <see cref="DocumentPublishedReadOnlyDto"/> publishedDto entity will be supplied in place of the <see cref="DocumentDto"/>'s own 
+        /// ResultColumn DocumentPublishedReadOnlyDto
+        /// </param>
+        /// <param name="contentType"></param>
+        /// <param name="publishedDto">
+        /// When querying for multiple content items the main DTO will not contain the ResultColumn DocumentPublishedReadOnlyDto and a separate publishedDto instance will be supplied
+        /// </param>
+        /// <returns></returns>
+        public static IContent BuildEntity(DocumentDto dto, IContentType contentType, DocumentPublishedReadOnlyDto publishedDto = null)
        {
-            var content = new Content(dto.Text, dto.ContentVersionDto.ContentDto.NodeDto.ParentId, _contentType);
+            var content = new Content(dto.Text, dto.ContentVersionDto.ContentDto.NodeDto.ParentId, contentType);

            try
            {
                content.DisableChangeTracking();

-                content.Id = _id;
+                content.Id = dto.NodeId;
                content.Key = dto.ContentVersionDto.ContentDto.NodeDto.UniqueId;
                content.Name = dto.Text;
                content.NodeName = dto.ContentVersionDto.ContentDto.NodeDto.Text;
@@ -49,11 +62,19 @@ namespace Umbraco.Core.Persistence.Factories
                content.Published = dto.Published;
                content.CreateDate = dto.ContentVersionDto.ContentDto.NodeDto.CreateDate;
                content.UpdateDate = dto.ContentVersionDto.VersionDate;
-                content.ExpireDate = dto.ExpiresDate.HasValue ? dto.ExpiresDate.Value : (DateTime?) null;
-                content.ReleaseDate = dto.ReleaseDate.HasValue ? dto.ReleaseDate.Value : (DateTime?) null;
+                content.ExpireDate = dto.ExpiresDate.HasValue ? dto.ExpiresDate.Value : (DateTime?)null;
+                content.ReleaseDate = dto.ReleaseDate.HasValue ? dto.ReleaseDate.Value : (DateTime?)null;
                content.Version = dto.ContentVersionDto.VersionId;
+
                content.PublishedState = dto.Published ? PublishedState.Published : PublishedState.Unpublished;
-                content.PublishedVersionGuid = dto.DocumentPublishedReadOnlyDto == null ? default(Guid) : dto.DocumentPublishedReadOnlyDto.VersionId;
+
+                //Check if the publishedDto has been supplied, if not the use the dto's own DocumentPublishedReadOnlyDto value
+                content.PublishedVersionGuid = publishedDto == null
+                    ? (dto.DocumentPublishedReadOnlyDto == null ? default(Guid) : dto.DocumentPublishedReadOnlyDto.VersionId)
+                    : publishedDto.VersionId;
+                content.PublishedDate = publishedDto == null
+                    ? (dto.DocumentPublishedReadOnlyDto == null ? default(DateTime) : dto.DocumentPublishedReadOnlyDto.VersionDate)
+                    : publishedDto.VersionDate;

                //on initial construction we don't want to have dirty properties tracked
                // http://issues.umbraco.org/issue/U4-1946
@@ -64,6 +85,13 @@ namespace Umbraco.Core.Persistence.Factories
            {
                content.EnableChangeTracking();
            }
+
+        }
+
+        [Obsolete("Use the static BuildEntity instead so we don't have to allocate one of these objects everytime we want to map values")]
+        public IContent BuildEntity(DocumentDto dto)
+        {
+            return BuildEntity(dto, _contentType);
        }

        public DocumentDto BuildDto(IContent entity)
@@ -122,7 +122,7 @@ namespace Umbraco.Core.Persistence.Factories
            else if (entity is IMemberType)
                nodeObjectType = Constants.ObjectTypes.MemberTypeGuid;
            else
-                throw new Exception("oops: invalid entity.");
+                throw new Exception("Invalid entity.");

            var contentTypeDto = new ContentTypeDto
            {
@@ -27,15 +27,15 @@ namespace Umbraco.Core.Persistence.Factories

        #region Implementation of IEntityFactory<IMedia,ContentVersionDto>

-        public IMedia BuildEntity(ContentVersionDto dto)
+        public static IMedia BuildEntity(ContentVersionDto dto, IMediaType contentType)
        {
-            var media = new Models.Media(dto.ContentDto.NodeDto.Text, dto.ContentDto.NodeDto.ParentId, _contentType);
+            var media = new Models.Media(dto.ContentDto.NodeDto.Text, dto.ContentDto.NodeDto.ParentId, contentType);

            try
            {
                media.DisableChangeTracking();

-                media.Id = _id;
+                media.Id = dto.NodeId;
                media.Key = dto.ContentDto.NodeDto.UniqueId;
                media.Path = dto.ContentDto.NodeDto.Path;
                media.CreatorId = dto.ContentDto.NodeDto.UserId.Value;
@@ -55,6 +55,13 @@ namespace Umbraco.Core.Persistence.Factories
            {
                media.EnableChangeTracking();
            }
+
+        }
+
+        [Obsolete("Use the static BuildEntity instead so we don't have to allocate one of these objects everytime we want to map values")]
+        public IMedia BuildEntity(ContentVersionDto dto)
+        {
+            return BuildEntity(dto, _contentType);
        }

        public ContentVersionDto BuildDto(IMedia entity)
@@ -28,17 +28,17 @@ namespace Umbraco.Core.Persistence.Factories

        #region Implementation of IEntityFactory<IMedia,ContentVersionDto>

-        public IMember BuildEntity(MemberDto dto)
+        public static IMember BuildEntity(MemberDto dto, IMemberType contentType)
        {
            var member = new Member(
                dto.ContentVersionDto.ContentDto.NodeDto.Text,
-                dto.Email, dto.LoginName, dto.Password, _contentType);
+                dto.Email, dto.LoginName, dto.Password, contentType);

            try
            {
                member.DisableChangeTracking();

-                member.Id = _id;
+                member.Id = dto.NodeId;
                member.Key = dto.ContentVersionDto.ContentDto.NodeDto.UniqueId;
                member.Path = dto.ContentVersionDto.ContentDto.NodeDto.Path;
                member.CreatorId = dto.ContentVersionDto.ContentDto.NodeDto.UserId.Value;
@@ -62,6 +62,12 @@ namespace Umbraco.Core.Persistence.Factories
            }
        }

+        [Obsolete("Use the static BuildEntity instead so we don't have to allocate one of these objects everytime we want to map values")]
+        public IMember BuildEntity(MemberDto dto)
+        {
+            return BuildEntity(dto, _contentType);
+        }
+
        public MemberDto BuildDto(IMember entity)
        {
            var dto = new MemberDto
@@ -79,7 +79,7 @@ namespace Umbraco.Core.Persistence.Factories
                {
                    // note: no idea why Id is nullable here, but better check
                    if (groupDto.Id.HasValue == false)
-                        throw new Exception("oops: groupDto.Id has no value.");
+                        throw new Exception("GroupDto.Id has no value.");
                    group.Id = groupDto.Id.Value;
                }

@@ -30,11 +30,11 @@ namespace Umbraco.Core.Persistence.Factories
            _updateDate = updateDate;
        }

-        public IEnumerable<Property> BuildEntity(PropertyDataDto[] dtos)
+        public static IEnumerable<Property> BuildEntity(IReadOnlyCollection<PropertyDataDto> dtos, PropertyType[] compositionTypeProperties, DateTime createDate, DateTime updateDate)
        {
            var properties = new List<Property>();

-            foreach (var propertyType in _compositionTypeProperties)
+            foreach (var propertyType in compositionTypeProperties)
            {
                var propertyDataDto = dtos.LastOrDefault(x => x.PropertyTypeId == propertyType.Id);
                var property = propertyDataDto == null
@@ -47,8 +47,8 @@ namespace Umbraco.Core.Persistence.Factories
                    //on initial construction we don't want to have dirty properties tracked
                    property.DisableChangeTracking();

-                    property.CreateDate = _createDate;
-                    property.UpdateDate = _updateDate;
+                    property.CreateDate = createDate;
+                    property.UpdateDate = updateDate;
                    // http://issues.umbraco.org/issue/U4-1946
                    property.ResetDirtyProperties(false);
                    properties.Add(property);
@@ -57,12 +57,18 @@ namespace Umbraco.Core.Persistence.Factories
                {
                    property.EnableChangeTracking();
                }
-                
+
            }

            return properties;
        }

+        [Obsolete("Use the static method instead, there's no reason to allocate one of these classes everytime we want to map values")]
+        public IEnumerable<Property> BuildEntity(PropertyDataDto[] dtos)
+        {
+            return BuildEntity(dtos, _compositionTypeProperties, _createDate, _updateDate);
+        }
+
        public IEnumerable<PropertyDataDto> BuildDto(IEnumerable<Property> properties)
        {
            var propertyDataDtos = new List<PropertyDataDto>();
@@ -6,6 +6,7 @@ using System.Reflection;
 using Umbraco.Core.Models;
 using Umbraco.Core.Models.EntityBase;
 using Umbraco.Core.Persistence.Repositories;
+using Umbraco.Core.Strings;

 namespace Umbraco.Core.Persistence.Factories
 {
@@ -17,7 +18,7 @@ namespace Umbraco.Core.Persistence.Factories
            
            //figure out what extra properties we have that are not on the IUmbracoEntity and add them to additional data
            foreach (var k in originalEntityProperties.Keys
-                .Select(x => new { orig = x, title = x.ConvertCase(StringAliasCaseType.PascalCase) })
+                .Select(x => new { orig = x, title = x.ToCleanString(CleanStringType.PascalCase | CleanStringType.Ascii | CleanStringType.ConvertCase) })
                .Where(x => entityProps.InvariantContains(x.title) == false))
            {
                entity.AdditionalData[k.title] = originalEntityProperties[k.orig];
@@ -75,65 +76,6 @@ namespace Umbraco.Core.Persistence.Factories
                entity.EnableChangeTracking();
            }
        }
-
-        public UmbracoEntity BuildEntity(EntityRepository.UmbracoEntityDto dto)
-        {
-            var entity = new UmbracoEntity(dto.Trashed)
-                             {
-                                 CreateDate = dto.CreateDate,
-                                 CreatorId = dto.UserId.Value,
-                                 Id = dto.NodeId,
-                                 Key = dto.UniqueId,
-                                 Level = dto.Level,
-                                 Name = dto.Text,
-                                 NodeObjectTypeId = dto.NodeObjectType.Value,
-                                 ParentId = dto.ParentId,
-                                 Path = dto.Path,
-                                 SortOrder = dto.SortOrder,
-                                 HasChildren = dto.Children > 0,
-                                 ContentTypeAlias = dto.Alias ?? string.Empty,
-                                 ContentTypeIcon = dto.Icon ?? string.Empty,
-                                 ContentTypeThumbnail = dto.Thumbnail ?? string.Empty,
-                             };
-
-            entity.IsPublished = dto.PublishedVersion != default(Guid) || (dto.NewestVersion != default(Guid) && dto.PublishedVersion == dto.NewestVersion);
-            entity.IsDraft = dto.NewestVersion != default(Guid) && (dto.PublishedVersion == default(Guid) || dto.PublishedVersion != dto.NewestVersion);
-            entity.HasPendingChanges = (dto.PublishedVersion != default(Guid) && dto.NewestVersion != default(Guid)) && dto.PublishedVersion != dto.NewestVersion;
-
-            if (dto.UmbracoPropertyDtos != null)
-            {
-                foreach (var propertyDto in dto.UmbracoPropertyDtos)
-                {
-                    entity.AdditionalData[propertyDto.PropertyAlias] = new UmbracoEntity.EntityProperty
-                    {
-                        PropertyEditorAlias = propertyDto.PropertyEditorAlias,
-                        Value = propertyDto.NTextValue.IsNullOrWhiteSpace()
-                            ? propertyDto.NVarcharValue
-                            : propertyDto.NTextValue.ConvertToJsonIfPossible()
-                    };
-                }
-            }
-
-            return entity;
-        }
-
-        public EntityRepository.UmbracoEntityDto BuildDto(UmbracoEntity entity)
-        {
-            var node = new EntityRepository.UmbracoEntityDto
-                           {
-                               CreateDate = entity.CreateDate,
-                               Level = short.Parse(entity.Level.ToString(CultureInfo.InvariantCulture)),
-                               NodeId = entity.Id,
-                               NodeObjectType = entity.NodeObjectTypeId,
-                               ParentId = entity.ParentId,
-                               Path = entity.Path,
-                               SortOrder = entity.SortOrder,
-                               Text = entity.Name,
-                               Trashed = entity.Trashed,
-                               UniqueId = entity.Key,
-                               UserId = entity.CreatorId
-                           };
-            return node;
-        }
+        
    }
 }
@@ -8,7 +8,17 @@ namespace Umbraco.Core.Persistence.Mappers
 {
    public abstract class BaseMapper
    {
-        
+        private readonly ISqlSyntaxProvider _sqlSyntax;
+
+        protected BaseMapper() : this(SqlSyntaxContext.SqlSyntaxProvider)
+        {            
+        }
+
+        protected BaseMapper(ISqlSyntaxProvider sqlSyntax)
+        {
+            _sqlSyntax = sqlSyntax;
+        }
+
        internal abstract ConcurrentDictionary<string, DtoMapModel> PropertyInfoCache { get; }

        internal abstract void BuildMap();
@@ -58,8 +68,8 @@ namespace Umbraco.Core.Persistence.Mappers
            string columnName = columnAttribute.Name;

            string columnMap = string.Format("{0}.{1}",
-                                             SqlSyntaxContext.SqlSyntaxProvider.GetQuotedTableName(tableName),
-                                             SqlSyntaxContext.SqlSyntaxProvider.GetQuotedColumnName(columnName));
+                                             _sqlSyntax.GetQuotedTableName(tableName),
+                                             _sqlSyntax.GetQuotedColumnName(columnName));
            return columnMap;
        }
    }
@@ -3,6 +3,7 @@ using System.Collections.Concurrent;
 using System.Linq.Expressions;
 using Umbraco.Core.Models;
 using Umbraco.Core.Models.Rdbms;
+using Umbraco.Core.Persistence.SqlSyntax;

 namespace Umbraco.Core.Persistence.Mappers
 {
@@ -16,6 +17,11 @@ namespace Umbraco.Core.Persistence.Mappers
    {
        private static readonly ConcurrentDictionary<string, DtoMapModel> PropertyInfoCacheInstance = new ConcurrentDictionary<string, DtoMapModel>();

+        public ContentMapper(ISqlSyntaxProvider sqlSyntax) : base(sqlSyntax)
+        {
+            
+        }
+
        //NOTE: its an internal class but the ctor must be public since we're using Activator.CreateInstance to create it
        // otherwise that would fail because there is no public constructor.
        public ContentMapper()
@@ -31,7 +31,7 @@ namespace Umbraco.Core.Persistence.Mappers
        /// </summary>
        /// <param name="type"></param>
        /// <returns></returns>
-        internal BaseMapper ResolveMapperByType(Type type)
+        public virtual BaseMapper ResolveMapperByType(Type type)
        {
            return _mapperCache.GetOrAdd(type, type1 =>
                {
@@ -43,7 +43,7 @@ namespace Umbraco.Core.Persistence.Mappers
                    {
                        return byAttribute.Result;
                    }
-                    throw new Exception("Invalid Type: A Mapper could not be resolved based on the passed in Type");
+                    throw new Exception("Invalid Type: A Mapper could not be resolved based on the passed in Type " + type);
                });
        }

@@ -67,7 +67,7 @@ namespace Umbraco.Core.Persistence.Mappers
            return Attempt<BaseMapper>.Succeed(mapper);
        }  

-        internal string GetMapping(Type type, string propertyName)
+        public virtual string GetMapping(Type type, string propertyName)
        {
            var mapper = ResolveMapperByType(type);
            var result = mapper.Map(propertyName);
@@ -149,8 +149,8 @@ namespace Umbraco.Core.Persistence.Migrations.Initial
        private void CreateCmsContentTypeData()
        {
            _database.Insert("cmsContentType", "pk", false, new ContentTypeDto { PrimaryKey = 532, NodeId = 1031, Alias = Constants.Conventions.MediaTypes.Folder, Icon = "icon-folder", Thumbnail = "icon-folder", IsContainer = false, AllowAtRoot = true });
-            _database.Insert("cmsContentType", "pk", false, new ContentTypeDto { PrimaryKey = 533, NodeId = 1032, Alias = Constants.Conventions.MediaTypes.Image, Icon = "icon-picture", Thumbnail = "icon-picture" });
-            _database.Insert("cmsContentType", "pk", false, new ContentTypeDto { PrimaryKey = 534, NodeId = 1033, Alias = Constants.Conventions.MediaTypes.File, Icon = "icon-document", Thumbnail = "icon-document" });
+            _database.Insert("cmsContentType", "pk", false, new ContentTypeDto { PrimaryKey = 533, NodeId = 1032, Alias = Constants.Conventions.MediaTypes.Image, Icon = "icon-picture", Thumbnail = "icon-picture", AllowAtRoot = true });
+            _database.Insert("cmsContentType", "pk", false, new ContentTypeDto { PrimaryKey = 534, NodeId = 1033, Alias = Constants.Conventions.MediaTypes.File, Icon = "icon-document", Thumbnail = "icon-document", AllowAtRoot = true });
            _database.Insert("cmsContentType", "pk", false, new ContentTypeDto { PrimaryKey = 531, NodeId = 1044, Alias = Constants.Conventions.MemberTypes.DefaultAlias, Icon = "icon-user", Thumbnail = "icon-user" });
        }

@@ -159,7 +159,7 @@ namespace Umbraco.Core.Persistence.Migrations.Initial

            foreach (var item in OrderedTables.OrderBy(x => x.Key))
            {
-                var tableDefinition = DefinitionFactory.GetTableDefinition(item.Value);
+                var tableDefinition = DefinitionFactory.GetTableDefinition(_sqlSyntaxProvider, item.Value);
                result.TableDefinitions.Add(tableDefinition);
            }

@@ -0,0 +1,25 @@
+using Umbraco.Core.Configuration;
+using Umbraco.Core.Logging;
+using Umbraco.Core.Persistence.SqlSyntax;
+
+namespace Umbraco.Core.Persistence.Migrations.Upgrades.TargetVersionSevenFiveFive
+{
+    /// <summary>
+    /// See: http://issues.umbraco.org/issue/U4-4196
+    /// </summary>
+    [Migration("7.5.5", 1, GlobalSettings.UmbracoMigrationName)]
+    public class UpdateAllowedMediaTypesAtRoot : MigrationBase
+    {
+        public UpdateAllowedMediaTypesAtRoot(ISqlSyntaxProvider sqlSyntax, ILogger logger)
+            : base(sqlSyntax, logger)
+        { }
+
+        public override void Up()
+        {
+            Execute.Sql("UPDATE cmsContentType SET allowAtRoot = 1 WHERE nodeId = 1032 OR nodeId = 1033");
+        }
+
+        public override void Down()
+        { }
+    }
+}
@@ -0,0 +1,38 @@
+using System.Linq;
+using Umbraco.Core.Configuration;
+using Umbraco.Core.Logging;
+using Umbraco.Core.Persistence.SqlSyntax;
+
+namespace Umbraco.Core.Persistence.Migrations.Upgrades.TargetVersionSevenSixZero
+{
+    /// <summary>
+    /// See: http://issues.umbraco.org/issue/U4-9188
+    /// </summary>
+    [Migration("7.6.0", 0, GlobalSettings.UmbracoMigrationName)]
+    public class UpdateUniqueIndexOnCmsPropertyData : MigrationBase
+    {
+        public UpdateUniqueIndexOnCmsPropertyData(ISqlSyntaxProvider sqlSyntax, ILogger logger)
+            : base(sqlSyntax, logger)
+        {
+        }
+
+        public override void Up()
+        {            
+            //tuple = tablename, indexname, columnname, unique
+            var indexes = SqlSyntax.GetDefinedIndexes(Context.Database).ToArray();
+            var found = indexes.FirstOrDefault(
+                x => x.Item1.InvariantEquals("cmsPropertyData")
+                     && x.Item2.InvariantEquals("IX_cmsPropertyData"));
+
+            if (found != null)
+            {
+                //drop the index
+                Delete.Index("IX_cmsPropertyData").OnTable("cmsPropertyData");             
+            }
+        }
+
+        public override void Down()
+        {
+        }
+    }
+}
@@ -739,6 +739,11 @@ namespace Umbraco.Core.Persistence
 	    /// <param name="take"></param>
 	    internal virtual void BuildSqlDbSpecificPagingQuery(DBType databaseType, long skip, long take, string sql, string sqlSelectRemoved, string sqlOrderBy, ref object[] args, out string sqlPage)
 	    {
+            // this is overriden in UmbracoDatabase, and if running SqlServer >=2012, the database type
+            // is switched from SqlServer to SqlServerCE in order to use the better paging syntax that
+            // SqlCE supports, and SqlServer >=2012 too.
+            // so the first case is actually for SqlServer <2012, and second case is CE *and* SqlServer >=2012
+
            if (databaseType == DBType.SqlServer || databaseType == DBType.Oracle)
            {
                sqlSelectRemoved = rxOrderBy.Replace(sqlSelectRemoved, "");
@@ -746,8 +751,16 @@ namespace Umbraco.Core.Persistence
                {
                    sqlSelectRemoved = "peta_inner.* FROM (SELECT " + sqlSelectRemoved + ") peta_inner";
                }
-                sqlPage = string.Format("SELECT * FROM (SELECT ROW_NUMBER() OVER ({0}) peta_rn, {1}) peta_paged WHERE peta_rn>@{2} AND peta_rn<=@{3}",
-                                        sqlOrderBy == null ? "ORDER BY (SELECT NULL)" : sqlOrderBy, sqlSelectRemoved, args.Length, args.Length + 1);
+
+                // split to ensure that peta_rn is the last field to be selected, else Page<int> would fail
+                // the resulting sql is not perfect, NPoco has a much nicer way to do it, but it would require
+                // importing large parts of NPoco
+                var pos = sqlSelectRemoved.IndexOf("FROM");
+                var sqlColumns = sqlSelectRemoved.Substring(0, pos);
+                var sqlFrom = sqlSelectRemoved.Substring(pos);
+
+                sqlPage = string.Format("SELECT * FROM (SELECT {0}, ROW_NUMBER() OVER ({1}) peta_rn {2}) peta_paged WHERE peta_rn>@{3} AND peta_rn<=@{4}",
+                                        sqlColumns, sqlOrderBy ?? "ORDER BY (SELECT NULL)", sqlFrom, args.Length, args.Length + 1);
                args = args.Concat(new object[] { skip, skip + take }).ToArray();
            }
            else if (databaseType == DBType.SqlServerCE)
@@ -774,7 +787,7 @@ namespace Umbraco.Core.Persistence
 				throw new Exception("Unable to parse SQL statement for paged query");
 			if (_dbType == DBType.Oracle && sqlSelectRemoved.StartsWith("*"))
                throw new Exception("Query must alias '*' when performing a paged query.\neg. select t.* from table t order by t.id");
-            
+
 		    BuildSqlDbSpecificPagingQuery(_dbType, skip, take, sql, sqlSelectRemoved, sqlOrderBy, ref args, out sqlPage);
 		}

@@ -3,9 +3,13 @@ using System.Collections.Generic;
 using System.Data;
 using System.Data.Common;
 using System.Data.SqlClient;
+using System.Data.SqlServerCe;
 using System.Linq;
 using System.Text.RegularExpressions;
+using MySql.Data.MySqlClient;
+using StackExchange.Profiling.Data;
 using Umbraco.Core.Logging;
+using Umbraco.Core.Persistence.DatabaseModelDefinitions;
 using Umbraco.Core.Persistence.Querying;
 using Umbraco.Core.Persistence.SqlSyntax;

@@ -56,7 +60,7 @@ namespace Umbraco.Core.Persistence
        /// Safely inserts a record, or updates if it exists, based on a unique constraint.
        /// </summary>
        /// <param name="db"></param>
-        /// <param name="poco"></param>       
+        /// <param name="poco"></param>
        /// <returns>The action that executed, either an insert or an update. If an insert occurred and a PK value got generated, the poco object
        /// passed in will contain the updated value.</returns>
        /// <remarks>
@@ -78,7 +82,7 @@ namespace Umbraco.Core.Persistence
        /// </summary>
        /// <param name="db"></param>
        /// <param name="poco"></param>
-        /// <param name="updateArgs"></param>      
+        /// <param name="updateArgs"></param>
        /// <param name="updateCommand">If the entity has a composite key they you need to specify the update command explicitly</param>
        /// <returns>The action that executed, either an insert or an update. If an insert occurred and a PK value got generated, the poco object
        /// passed in will contain the updated value.</returns>
@@ -130,7 +134,7 @@ namespace Umbraco.Core.Persistence
                    if (rowCount > 0)
                        return RecordPersistenceType.Update;

-                    // failed: does not exist (due to race cond RC2), need to insert 
+                    // failed: does not exist (due to race cond RC2), need to insert
                    // loop
                }
            }
@@ -150,7 +154,7 @@ namespace Umbraco.Core.Persistence
            {
                //this fancy regex will only match a single @ not a double, etc...
                var regex = new Regex("(?<!@)@(?!@)");
-                return regex.Replace(value, "@@");    
+                return regex.Replace(value, "@@");
            }
            return value;

@@ -172,6 +176,13 @@ namespace Umbraco.Core.Persistence
            creator.CreateTable<T>(overwrite);
        }

+        /// <summary>
+        /// Performs the bulk insertion in the context of a current transaction with an optional parameter to complete the transaction
+        /// when finished
+        /// </summary>
+        /// <typeparam name="T"></typeparam>
+        /// <param name="db"></param>
+        /// <param name="collection"></param>
        public static void BulkInsertRecords<T>(this Database db, IEnumerable<T> collection)
        {
            //don't do anything if there are no records.
@@ -180,7 +191,7 @@ namespace Umbraco.Core.Persistence

            using (var tr = db.GetTransaction())
            {
-                db.BulkInsertRecords(collection, tr, true);
+                db.BulkInsertRecords(collection, tr, SqlSyntaxContext.SqlSyntaxProvider, true, true); // use native, commit
            }
        }

@@ -192,55 +203,98 @@ namespace Umbraco.Core.Persistence
        /// <param name="db"></param>
        /// <param name="collection"></param>
        /// <param name="tr"></param>
+        /// <param name="syntaxProvider"></param>
+        /// <param name="useNativeSqlPlatformBulkInsert">
+        /// If this is false this will try to just generate bulk insert statements instead of using the current SQL platform's bulk
+        /// insert logic. For SQLCE, bulk insert statements do not work so if this is false it will insert one at a time.
+        /// </param>
        /// <param name="commitTrans"></param>
-        public static void BulkInsertRecords<T>(this Database db, IEnumerable<T> collection, Transaction tr, bool commitTrans = false)
+        /// <returns>The number of items inserted</returns>
+        public static int BulkInsertRecords<T>(this Database db,
+            IEnumerable<T> collection,
+            Transaction tr,
+            ISqlSyntaxProvider syntaxProvider,
+            bool useNativeSqlPlatformBulkInsert = true,
+            bool commitTrans = false)
        {
+
            //don't do anything if there are no records.
            if (collection.Any() == false)
-                return;
+            {
+                return 0;
+            }
+
+            var pd = Database.PocoData.ForType(typeof(T));
+            if (pd == null) throw new InvalidOperationException("Could not find PocoData for " + typeof(T));

            try
            {
-                //if it is sql ce or it is a sql server version less than 2008, we need to do individual inserts.
-                var sqlServerSyntax = SqlSyntaxContext.SqlSyntaxProvider as SqlServerSyntaxProvider;
+                int processed = 0;

-                if ((sqlServerSyntax != null && (int)sqlServerSyntax.GetVersionName(db) < (int)SqlServerVersionName.V2008) 
-                    || SqlSyntaxContext.SqlSyntaxProvider is SqlCeSyntaxProvider)
-                {
-                    //SqlCe doesn't support bulk insert statements!
+                var usedNativeSqlPlatformInserts = useNativeSqlPlatformBulkInsert
+                    && NativeSqlPlatformBulkInsertRecords(db, syntaxProvider, pd, collection, out processed);

-                    foreach (var poco in collection)
-                    {
-                        db.Insert(poco);
-                    }
-                }
-                else
+                if (usedNativeSqlPlatformInserts == false)
                {
-                    string[] sqlStatements;
-                    var cmds = db.GenerateBulkInsertCommand(collection, db.Connection, out sqlStatements);
-                    for (var i = 0; i < sqlStatements.Length; i++)
+                    //if it is sql ce or it is a sql server version less than 2008, we need to do individual inserts.
+                    var sqlServerSyntax = syntaxProvider as SqlServerSyntaxProvider;
+                    if ((sqlServerSyntax != null && (int) sqlServerSyntax.GetVersionName(db) < (int) SqlServerVersionName.V2008)
+                        || syntaxProvider is SqlCeSyntaxProvider)
                    {
-                        using (var cmd = cmds[i])
+                        //SqlCe doesn't support bulk insert statements!
+                        foreach (var poco in collection)
                        {
-                            cmd.CommandText = sqlStatements[i];
-                            cmd.ExecuteNonQuery();
+                            db.Insert(poco);
+                        }
+                    }
+                    else
+                    {
+                        //we'll need to generate insert statements instead
+
+                        string[] sqlStatements;
+                        var cmds = db.GenerateBulkInsertCommand(pd, collection, out sqlStatements);
+                        for (var i = 0; i < sqlStatements.Length; i++)
+                        {
+                            using (var cmd = cmds[i])
+                            {
+                                cmd.CommandText = sqlStatements[i];
+                                cmd.ExecuteNonQuery();
+                                processed++;
+                            }
                        }
                    }
                }

                if (commitTrans)
                {
-                    tr.Complete();    
+                    tr.Complete();
                }
+                return processed;
            }
            catch
            {
                if (commitTrans)
                {
-                    tr.Dispose();    
+                    tr.Dispose();
                }
                throw;
            }
+
+        }
+
+        /// <summary>
+        /// Performs the bulk insertion in the context of a current transaction with an optional parameter to complete the transaction
+        /// when finished
+        /// </summary>
+        /// <typeparam name="T"></typeparam>
+        /// <param name="db"></param>
+        /// <param name="collection"></param>
+        /// <param name="tr"></param>
+        /// <param name="commitTrans"></param>
+        [Obsolete("Use the method that specifies an SqlSyntaxContext instance instead")]
+        public static void BulkInsertRecords<T>(this Database db, IEnumerable<T> collection, Transaction tr, bool commitTrans = false)
+        {
+            db.BulkInsertRecords<T>(collection, tr, SqlSyntaxContext.SqlSyntaxProvider, commitTrans);
        }

        /// <summary>
@@ -249,45 +303,36 @@ namespace Umbraco.Core.Persistence
        /// <typeparam name="T"></typeparam>
        /// <param name="db"></param>
        /// <param name="collection"></param>
-        /// <param name="connection"></param>        
        /// <param name="sql"></param>
+        /// <param name="pd"></param>
        /// <returns>Sql commands with populated command parameters required to execute the sql statement</returns>
        /// <remarks>
-        /// The limits for number of parameters are 2100 (in sql server, I think there's many more allowed in mysql). So 
-        /// we need to detect that many params and split somehow. 
-        /// For some reason the 2100 limit is not actually allowed even though the exception from sql server mentions 2100 as a max, perhaps it is 2099 
+        /// The limits for number of parameters are 2100 (in sql server, I think there's many more allowed in mysql). So
+        /// we need to detect that many params and split somehow.
+        /// For some reason the 2100 limit is not actually allowed even though the exception from sql server mentions 2100 as a max, perhaps it is 2099
        /// that is max. I've reduced it to 2000 anyways.
        /// </remarks>
        internal static IDbCommand[] GenerateBulkInsertCommand<T>(
-            this Database db, 
-            IEnumerable<T> collection, 
-            IDbConnection connection,             
+            this Database db,
+            Database.PocoData pd,
+            IEnumerable<T> collection,
            out string[] sql)
        {
-            //A filter used below a few times to get all columns except result cols and not the primary key if it is auto-incremental
-            Func<Database.PocoData, KeyValuePair<string, Database.PocoColumn>, bool> includeColumn = (data, column) =>
-                {
-                    if (column.Value.ResultColumn) return false;
-                    if (data.TableInfo.AutoIncrement && column.Key == data.TableInfo.PrimaryKey) return false;
-                    return true;
-                };
-
-            var pd = Database.PocoData.ForType(typeof(T));
            var tableName = db.EscapeTableName(pd.TableInfo.TableName);

            //get all columns to include and format for sql
-            var cols = string.Join(", ", 
+            var cols = string.Join(", ",
                pd.Columns
-                .Where(c => includeColumn(pd, c))
+                .Where(c => IncludeColumn(pd, c))
                .Select(c => tableName + "." + db.EscapeSqlIdentifier(c.Key)).ToArray());

            var itemArray = collection.ToArray();

            //calculate number of parameters per item
-            var paramsPerItem = pd.Columns.Count(i => includeColumn(pd, i));
-            
+            var paramsPerItem = pd.Columns.Count(i => IncludeColumn(pd, i));
+
            //Example calc:
-            // Given: we have 4168 items in the itemArray, each item contains 8 command parameters (values to be inserterted)                
+            // Given: we have 4168 items in the itemArray, each item contains 8 command parameters (values to be inserterted)
            // 2100 / 8 = 262.5
            // Math.Floor(2100 / 8) = 262 items per trans
            // 4168 / 262 = 15.908... = there will be 16 trans in total
@@ -306,14 +351,14 @@ namespace Umbraco.Core.Persistence
                    .Skip(tIndex * (int)itemsPerTrans)
                    .Take((int)itemsPerTrans);

-                var cmd = db.CreateCommand(connection, "");
+                var cmd = db.CreateCommand(db.Connection, string.Empty);
                var pocoValues = new List<string>();
                var index = 0;
                foreach (var poco in itemsForTrans)
                {
                    var values = new List<string>();
                    //get all columns except result cols and not the primary key if it is auto-incremental
-                    foreach (var i in pd.Columns.Where(x => includeColumn(pd, x)))
+                    foreach (var i in pd.Columns.Where(x => IncludeColumn(pd, x)))
                    {
                        db.AddParam(cmd, i.Value.GetValue(poco), "@");
                        values.Add(string.Format("{0}{1}", "@", index++));
@@ -321,14 +366,211 @@ namespace Umbraco.Core.Persistence
                    pocoValues.Add("(" + string.Join(",", values.ToArray()) + ")");
                }

-                var sqlResult = string.Format("INSERT INTO {0} ({1}) VALUES {2}", tableName, cols, string.Join(", ", pocoValues)); 
+                var sqlResult = string.Format("INSERT INTO {0} ({1}) VALUES {2}", tableName, cols, string.Join(", ", pocoValues));
                sqlQueries.Add(sqlResult);
                commands.Add(cmd);
            }

            sql = sqlQueries.ToArray();

-            return commands.ToArray();    
+            return commands.ToArray();
+        }
+
+        /// <summary>
+        /// A filter used below a few times to get all columns except result cols and not the primary key if it is auto-incremental
+        /// </summary>
+        /// <param name="data"></param>
+        /// <param name="column"></param>
+        /// <returns></returns>
+        private static bool IncludeColumn(Database.PocoData data, KeyValuePair<string, Database.PocoColumn> column)
+        {
+            if (column.Value.ResultColumn) return false;
+            if (data.TableInfo.AutoIncrement && column.Key == data.TableInfo.PrimaryKey) return false;
+            return true;
+        }
+
+        /// <summary>
+        /// Bulk insert records with Sql BulkCopy or TableDirect or whatever sql platform specific bulk insert records should be used
+        /// </summary>
+        /// <param name="db"></param>
+        /// <param name="syntaxProvider"></param>
+        /// <param name="pd"></param>
+        /// <param name="collection"></param>
+        /// <param name="processed">The number of records inserted</param>
+        private static bool NativeSqlPlatformBulkInsertRecords<T>(Database db, ISqlSyntaxProvider syntaxProvider, Database.PocoData pd, IEnumerable<T> collection, out int processed)
+        {
+
+            var dbConnection = db.Connection;
+
+            //unwrap the profiled connection if there is one
+            var profiledConnection = dbConnection as ProfiledDbConnection;
+            if (profiledConnection != null)
+            {
+                dbConnection = profiledConnection.InnerConnection;
+            }
+
+            //check if it's SQL or SqlCe
+
+            var sqlConnection = dbConnection as SqlConnection;
+            if (sqlConnection != null)
+            {
+                processed = BulkInsertRecordsSqlServer(db, (SqlServerSyntaxProvider)syntaxProvider, pd, collection);
+                return true;
+            }
+
+            var sqlCeConnection = dbConnection as SqlCeConnection;
+            if (sqlCeConnection != null)
+            {
+                processed = BulkInsertRecordsSqlCe(db, pd, collection);
+                return true;
+            }
+
+            //could not use the SQL server's specific bulk insert operations
+            processed = 0;
+            return false;
+
+        }
+
+        /// <summary>
+        /// Logic used to perform bulk inserts with SqlCe's TableDirect
+        /// </summary>
+        /// <typeparam name="T"></typeparam>
+        /// <param name="db"></param>
+        /// <param name="pd"></param>
+        /// <param name="collection"></param>
+        /// <returns></returns>
+        internal static int BulkInsertRecordsSqlCe<T>(Database db,
+            Database.PocoData pd,
+            IEnumerable<T> collection)
+        {
+            var cols = pd.Columns.ToArray();
+
+            using (var cmd = db.CreateCommand(db.Connection, string.Empty))
+            {
+                cmd.CommandText = pd.TableInfo.TableName;
+                cmd.CommandType = CommandType.TableDirect;
+                //cmd.Transaction = GetTypedTransaction<SqlCeTransaction>(db.Connection.);
+
+                //get the real command
+                using (var sqlCeCommand = GetTypedCommand<SqlCeCommand>(cmd))
+                {
+                    // This seems to cause problems, I think this is primarily used for retrieval, not
+                    // inserting. see: https://msdn.microsoft.com/en-us/library/system.data.sqlserverce.sqlcecommand.indexname%28v=vs.100%29.aspx?f=255&MSPPError=-2147217396
+                    //sqlCeCommand.IndexName = pd.TableInfo.PrimaryKey;
+
+                    var count = 0;
+                    using (var rs = sqlCeCommand.ExecuteResultSet(ResultSetOptions.Updatable))
+                    {
+                        var rec = rs.CreateRecord();
+
+                        foreach (var item in collection)
+                        {
+                            for (var i = 0; i < cols.Length; i++)
+                            {
+                                //skip the index if this shouldn't be included (i.e. PK)
+                                if (IncludeColumn(pd, cols[i]))
+                                {
+                                    var val = cols[i].Value.GetValue(item);
+                                    rec.SetValue(i, val);
+                                }
+                            }
+                            rs.Insert(rec);
+                            count++;
+                        }
+                    }
+                    return count;
+                }
+
+            }
+        }
+
+        /// <summary>
+        /// Logic used to perform bulk inserts with SqlServer's BulkCopy
+        /// </summary>
+        /// <typeparam name="T"></typeparam>
+        /// <param name="db"></param>
+        /// <param name="sqlSyntaxProvider"></param>
+        /// <param name="pd"></param>
+        /// <param name="collection"></param>
+        /// <returns></returns>
+        internal static int BulkInsertRecordsSqlServer<T>(Database db, SqlServerSyntaxProvider sqlSyntaxProvider,
+            Database.PocoData pd, IEnumerable<T> collection)
+        {
+            //NOTE: We need to use the original db.Connection here to create the command, but we need to pass in the typed
+            // connection below to the SqlBulkCopy
+            using (var cmd = db.CreateCommand(db.Connection, string.Empty))
+            {
+                using (var copy = new SqlBulkCopy(
+                    GetTypedConnection<SqlConnection>(db.Connection),
+                    SqlBulkCopyOptions.Default,
+                    GetTypedTransaction<SqlTransaction>(cmd.Transaction))
+                {
+                    BulkCopyTimeout = 10000,
+                    DestinationTableName = pd.TableInfo.TableName
+                })
+                {
+                    //var cols = pd.Columns.Where(x => IncludeColumn(pd, x)).Select(x => x.Value).ToArray();
+
+                    using (var bulkReader = new PocoDataDataReader<T, SqlServerSyntaxProvider>(collection, pd, sqlSyntaxProvider))
+                    {
+                        copy.WriteToServer(bulkReader);
+
+                        return bulkReader.RecordsAffected;
+                    }
+                }
+            }
+        }
+
+
+        /// <summary>
+        /// Returns the underlying connection as a typed connection - this is used to unwrap the profiled mini profiler stuff
+        /// </summary>
+        /// <typeparam name="TConnection"></typeparam>
+        /// <param name="connection"></param>
+        /// <returns></returns>
+        private static TConnection GetTypedConnection<TConnection>(IDbConnection connection)
+            where TConnection : class, IDbConnection
+        {
+            var profiled = connection as ProfiledDbConnection;
+            if (profiled != null)
+            {
+                return profiled.InnerConnection as TConnection;
+            }
+            return connection as TConnection;
+        }
+
+        /// <summary>
+        /// Returns the underlying connection as a typed connection - this is used to unwrap the profiled mini profiler stuff
+        /// </summary>
+        /// <typeparam name="TTransaction"></typeparam>
+        /// <param name="connection"></param>
+        /// <returns></returns>
+        private static TTransaction GetTypedTransaction<TTransaction>(IDbTransaction connection)
+            where TTransaction : class, IDbTransaction
+        {
+            var profiled = connection as ProfiledDbTransaction;
+            if (profiled != null)
+            {
+                return profiled.WrappedTransaction as TTransaction;
+            }
+            return connection as TTransaction;
+        }
+
+        /// <summary>
+        /// Returns the underlying connection as a typed connection - this is used to unwrap the profiled mini profiler stuff
+        /// </summary>
+        /// <typeparam name="TCommand"></typeparam>
+        /// <param name="command"></param>
+        /// <returns></returns>
+        private static TCommand GetTypedCommand<TCommand>(IDbCommand command)
+            where TCommand : class, IDbCommand
+        {
+            var profiled = command as ProfiledDbCommand;
+            if (profiled != null)
+            {
+                return profiled.InternalCommand as TCommand;
+            }
+            return command as TCommand;
        }

        [Obsolete("Use the DatabaseSchemaHelper instead")]
@@ -405,8 +647,8 @@ namespace Umbraco.Core.Persistence
            return ApplicationContext.Current.DatabaseContext.DatabaseProvider;
        }

-        
+
    }

-    
+
 }
@@ -27,18 +27,42 @@ namespace Umbraco.Core.Persistence
            return sql.From(sqlSyntax.GetQuotedTableName(tableName));
        }

+        [Obsolete("Use the overload specifying ISqlSyntaxProvider instead")]
        public static Sql Where<T>(this Sql sql, Expression<Func<T, bool>> predicate)
        {
-            var expresionist = new PocoToSqlExpressionHelper<T>();
+            var expresionist = new PocoToSqlExpressionVisitor<T>();
            var whereExpression = expresionist.Visit(predicate);
            return sql.Where(whereExpression, expresionist.GetSqlParameters());
        }

+        public static Sql Where<T>(this Sql sql, Expression<Func<T, bool>> predicate, ISqlSyntaxProvider sqlSyntax)
+        {
+            var expresionist = new PocoToSqlExpressionVisitor<T>(sqlSyntax);
+            var whereExpression = expresionist.Visit(predicate);
+            return sql.Where(whereExpression, expresionist.GetSqlParameters());
+        }
+
+        private static string GetFieldName<T>(Expression<Func<T, object>> fieldSelector, ISqlSyntaxProvider sqlSyntax)
+        {
+            var field = ExpressionHelper.FindProperty(fieldSelector) as PropertyInfo;
+            var fieldName = field.GetColumnName();
+
+            var type = typeof(T);
+            var tableName = type.GetTableName();
+
+            return sqlSyntax.GetQuotedTableName(tableName) + "." + sqlSyntax.GetQuotedColumnName(fieldName);
+        }
+
+        [Obsolete("Use the overload specifying ISqlSyntaxProvider instead")]
        public static Sql WhereIn<T>(this Sql sql, Expression<Func<T, object>> fieldSelector, IEnumerable values)
        {
-            var expresionist = new PocoToSqlExpressionHelper<T>();
-            var fieldExpression = expresionist.Visit(fieldSelector);
-            return sql.Where(fieldExpression + " IN (@values)", new {@values = values});
+            return sql.WhereIn(fieldSelector, values, SqlSyntaxContext.SqlSyntaxProvider);
+        }
+
+        public static Sql WhereIn<T>(this Sql sql, Expression<Func<T, object>> fieldSelector, IEnumerable values, ISqlSyntaxProvider sqlSyntax)
+        {
+            var fieldName = GetFieldName(fieldSelector, sqlSyntax);
+            return sql.Where(fieldName + " IN (@values)", new { values });
        }

        [Obsolete("Use the overload specifying ISqlSyntaxProvider instead")]
@@ -49,17 +73,7 @@ namespace Umbraco.Core.Persistence

        public static Sql OrderBy<TColumn>(this Sql sql, Expression<Func<TColumn, object>> columnMember, ISqlSyntaxProvider sqlSyntax)
        {
-            var column = ExpressionHelper.FindProperty(columnMember) as PropertyInfo;
-            var columnName = column.GetColumnName();
-
-            var type = typeof(TColumn);
-            var tableName = type.GetTableName();
-
-            //need to ensure the order by is in brackets, see: https://github.com/toptensoftware/PetaPoco/issues/177
-            var syntax = string.Format("({0}.{1})",
-                sqlSyntax.GetQuotedTableName(tableName),
-                sqlSyntax.GetQuotedColumnName(columnName));
-
+            var syntax = "(" + GetFieldName(columnMember, sqlSyntax) + ")";
            return sql.OrderBy(syntax);
        }

@@ -0,0 +1,159 @@
+using System;
+using System.Collections.Generic;
+using System.Data;
+using System.Linq;
+using Umbraco.Core.Persistence.DatabaseAnnotations;
+using Umbraco.Core.Persistence.DatabaseModelDefinitions;
+using Umbraco.Core.Persistence.SqlSyntax;
+
+namespace Umbraco.Core.Persistence
+{
+    /// <summary>
+    /// A data reader used for reading collections of PocoData entity types
+    /// </summary>
+    /// <remarks>
+    /// We are using a custom data reader so that tons of memory is not consumed when rebuilding this table, previously
+    /// we'd generate SQL insert statements, but we'd have to put all of the XML structures into memory first. Alternatively
+    /// we can use .net's DataTable, but this also requires putting everything into memory. By using a DataReader we don't have to 
+    /// store every content item and it's XML structure in memory to get it into the DB, we can stream it into the db with this 
+    /// reader.
+    /// </remarks>
+    internal class PocoDataDataReader<T, TSyntax> : BulkDataReader 
+        where TSyntax : ISqlSyntaxProvider
+    {        
+        private readonly MicrosoftSqlSyntaxProviderBase<TSyntax> _sqlSyntaxProvider;
+        private readonly TableDefinition _tableDefinition;
+        private readonly Database.PocoColumn[] _readerColumns;
+        private readonly IEnumerator<T> _enumerator;
+        private readonly ColumnDefinition[] _columnDefinitions;
+        private int _recordsAffected = -1;
+
+        public PocoDataDataReader(
+            IEnumerable<T> dataSource,             
+            Database.PocoData pd,
+            MicrosoftSqlSyntaxProviderBase<TSyntax> sqlSyntaxProvider)
+        {
+            if (dataSource == null) throw new ArgumentNullException("dataSource");            
+            if (sqlSyntaxProvider == null) throw new ArgumentNullException("sqlSyntaxProvider");
+
+            _tableDefinition = DefinitionFactory.GetTableDefinition(sqlSyntaxProvider, pd.type);
+            if (_tableDefinition == null) throw new InvalidOperationException("No table definition found for type " + pd.type);
+            
+            _readerColumns = pd.Columns.Select(x => x.Value).ToArray();
+            _sqlSyntaxProvider = sqlSyntaxProvider;
+            _enumerator = dataSource.GetEnumerator();
+            _columnDefinitions = _tableDefinition.Columns.ToArray();
+            
+        }
+
+        protected override string SchemaName
+        {
+            get { return _tableDefinition.SchemaName; }
+        }
+
+        protected override string TableName
+        {
+            get { return _tableDefinition.Name; }
+        }
+
+        public override int RecordsAffected
+        {
+            get { return _recordsAffected <= 0 ? -1 : _recordsAffected; }
+        }
+
+        /// <summary>
+        /// This will automatically add the schema rows based on the Poco table definition and the columns passed in
+        /// </summary>
+        protected override void AddSchemaTableRows()
+        {
+            //var colNames = _readerColumns.Select(x => x.ColumnName).ToArray();
+            //foreach (var col in _columnDefinitions.Where(x => colNames.Contains(x.Name, StringComparer.OrdinalIgnoreCase)))
+            foreach (var col in _columnDefinitions)
+            {
+                var sqlDbType = SqlDbType.NVarChar;
+                if (col.HasSpecialDbType)
+                {
+                    //get the SqlDbType from the 'special type'
+                    switch (col.DbType)
+                    {
+                        case SpecialDbTypes.NTEXT:
+                            sqlDbType = SqlDbType.NText;
+                            break;
+                        case SpecialDbTypes.NCHAR:
+                            sqlDbType = SqlDbType.NChar;
+                            break;
+                        default:
+                            throw new ArgumentOutOfRangeException();
+                    }
+                }
+                else if (col.Type.HasValue)
+                {
+                    //get the SqlDbType from the DbType
+                    sqlDbType = _sqlSyntaxProvider.GetSqlDbType(col.Type.Value);
+                }
+                else
+                {
+                    //get the SqlDbType from the clr type
+                    sqlDbType = _sqlSyntaxProvider.GetSqlDbType(col.PropertyType);
+                }
+                
+                AddSchemaTableRow(
+                    col.Name, 
+                    col.Size > 0 ? (int?)col.Size : null,
+                    col.Precision > 0  ? (short?)col.Precision : null, 
+                    null, col.IsUnique, col.IsIdentity, col.IsNullable, sqlDbType, 
+                    null, null, null, null, null);                
+            }
+
+        }
+
+        /// <summary>
+        /// Get the value from the column index for the current object
+        /// </summary>
+        /// <param name="i"></param>
+        /// <returns></returns>
+        public override object GetValue(int i)
+        {
+            if (_enumerator.Current != null)
+            {
+                return _readerColumns[i].GetValue(_enumerator.Current);
+                //return _columnDefinitions[i]. .GetValue(_enumerator.Current);
+            }
+
+            return null;
+            //TODO: Or throw ?
+        }
+
+        /// <summary>
+        /// Advance the cursor
+        /// </summary>
+        /// <returns></returns>
+        public override bool Read()
+        {
+            var result = _enumerator.MoveNext();
+            if (result)
+            {
+                if (_recordsAffected == -1)
+                {
+                    _recordsAffected = 0;
+                }
+                _recordsAffected++;
+            }
+            return result;
+        }
+
+        /// <summary>
+        /// Ensure the enumerator is disposed
+        /// </summary>
+        /// <param name="disposing"></param>
+        protected override void Dispose(bool disposing)
+        {
+            base.Dispose(disposing);
+
+            if (disposing)
+            {
+                _enumerator.Dispose();
+            }
+        }
+    }
+}
@@ -1,31 +1,8 @@
 using System;
-using System.Collections.Generic;
 using System.Linq.Expressions;

 namespace Umbraco.Core.Persistence.Querying
 {
-    /// <summary>
-    /// SD: This is a horrible hack but unless we break compatibility with anyone who's actually implemented IQuery{T} there's not much we can do.
-    /// The IQuery{T} interface is useless without having a GetWhereClauses method and cannot be used for tests.
-    /// We have to wait till v8 to make this change I suppose.
-    /// </summary>
-    internal static class QueryExtensions
-    {
-        /// <summary>
-        /// Returns all translated where clauses and their sql parameters
-        /// </summary>
-        /// <returns></returns>
-        public static IEnumerable<Tuple<string, object[]>> GetWhereClauses<T>(this IQuery<T> query)
-        {
-            var q = query as Query<T>;
-            if (q == null)
-            {
-                throw new NotSupportedException(typeof(IQuery<T>) + " cannot be cast to " + typeof(Query<T>));
-            }
-            return q.GetWhereClauses();
-        }
-    }
-
    /// <summary>
    /// Represents a query for building Linq translatable SQL queries
    /// </summary>
@@ -1,62 +0,0 @@
-using System;
-using System.Collections.Generic;
-using System.Collections.ObjectModel;
-using System.Diagnostics;
-using System.Globalization;
-using System.Linq;
-using System.Linq.Expressions;
-using System.Text;
-using Umbraco.Core.Persistence.Mappers;
-using Umbraco.Core.Persistence.SqlSyntax;
-
-namespace Umbraco.Core.Persistence.Querying
-{
-    internal class ModelToSqlExpressionHelper<T> : BaseExpressionHelper<T>
-    {
-
-        private readonly BaseMapper _mapper;
-
-        public ModelToSqlExpressionHelper()
-        {
-            _mapper = MappingResolver.Current.ResolveMapperByType(typeof(T));
-        }
-        
-        protected override string VisitMemberAccess(MemberExpression m)
-        {
-            if (m.Expression != null && 
-                m.Expression.NodeType == ExpressionType.Parameter 
-                && m.Expression.Type == typeof(T))
-            {
-                var field = _mapper.Map(m.Member.Name, true);
-                if (field.IsNullOrWhiteSpace()) 
-                    throw new InvalidOperationException("The mapper returned an empty field for the member name: " + m.Member.Name);
-                return field;                
-            }
-
-            if (m.Expression != null && m.Expression.NodeType == ExpressionType.Convert)
-            {
-                var field = _mapper.Map(m.Member.Name, true);
-                if (field.IsNullOrWhiteSpace()) 
-                    throw new InvalidOperationException("The mapper returned an empty field for the member name: " + m.Member.Name);
-                return field;
-            }
-            
-            var member = Expression.Convert(m, typeof(object));
-            var lambda = Expression.Lambda<Func<object>>(member);
-            var getter = lambda.Compile();
-            object o = getter();
-
-            SqlParameters.Add(o);
-            return string.Format("@{0}", SqlParameters.Count - 1);
-
-            //return GetQuotedValue(o, o != null ? o.GetType() : null);
-
-        }
-    
-        //protected bool IsFieldName(string quotedExp)
-        //{
-        //    //Not entirely sure this is reliable, but its better then simply returning true
-        //    return quotedExp.LastIndexOf("'", StringComparison.InvariantCultureIgnoreCase) + 1 != quotedExp.Length;
-        //}
-    }
-}
@@ -0,0 +1,126 @@
+using System;
+using System.Linq.Expressions;
+using Umbraco.Core.Models.EntityBase;
+using Umbraco.Core.Persistence.Mappers;
+using Umbraco.Core.Persistence.SqlSyntax;
+
+namespace Umbraco.Core.Persistence.Querying
+{
+    /// <summary>
+    /// An expression tree parser to create SQL statements and SQL parameters based on a strongly typed expression,
+    /// based on Umbraco's business logic models.
+    /// </summary>
+    /// <remarks>This object is stateful and cannot be re-used to parse an expression.</remarks>
+    internal class ModelToSqlExpressionVisitor<T> : ExpressionVisitorBase
+    {
+        private readonly MappingResolver _mappingResolver;
+        private readonly BaseMapper _mapper;
+
+        public ModelToSqlExpressionVisitor(ISqlSyntaxProvider sqlSyntax, MappingResolver mappingResolver)
+            : base(sqlSyntax)
+        {
+            _mapper = mappingResolver.ResolveMapperByType(typeof(T));
+            _mappingResolver = mappingResolver;
+        }
+
+        [Obsolete("Use the overload the specifies a SqlSyntaxProvider")]
+        public ModelToSqlExpressionVisitor()
+            : this(SqlSyntaxContext.SqlSyntaxProvider, MappingResolver.Current)
+        { }
+
+        protected override string VisitMemberAccess(MemberExpression m)
+        {
+            if (m.Expression != null &&
+                m.Expression.NodeType == ExpressionType.Parameter
+                && m.Expression.Type == typeof(T))
+            {
+                //don't execute if compiled
+                if (Visited == false)
+                {
+                    var field = _mapper.Map(m.Member.Name, true);
+                    if (field.IsNullOrWhiteSpace())
+                        throw new InvalidOperationException(string.Format("The mapper returned an empty field for the member name: {0} for type: {1}", m.Member.Name, m.Expression.Type));
+                    return field;
+                }
+                //already compiled, return
+                return string.Empty;
+            }
+
+            if (m.Expression != null && m.Expression.NodeType == ExpressionType.Convert)
+            {
+                //don't execute if compiled
+                if (Visited == false)
+                {
+                    var field = _mapper.Map(m.Member.Name, true);
+                    if (field.IsNullOrWhiteSpace())
+                        throw new InvalidOperationException(string.Format("The mapper returned an empty field for the member name: {0} for type: {1}", m.Member.Name, m.Expression.Type));
+                    return field;
+                }
+                //already compiled, return
+                return string.Empty;
+            }
+
+            if (m.Expression != null 
+                && m.Expression.Type != typeof(T) 
+                && TypeHelper.IsTypeAssignableFrom<IUmbracoEntity>(m.Expression.Type)
+                && EndsWithConstant(m) == false)
+            {
+                //if this is the case, it means we have a sub expression / nested property access, such as: x.ContentType.Alias == "Test";
+                //and since the sub type (x.ContentType) is not the same as x, we need to resolve a mapper for x.ContentType to get it's mapped SQL column
+
+                //don't execute if compiled
+                if (Visited == false)
+                {
+                    var subMapper = _mappingResolver.ResolveMapperByType(m.Expression.Type);
+                    if (subMapper == null)
+                        throw new NullReferenceException("No mapper found for type " + m.Expression.Type);
+                    var field = subMapper.Map(m.Member.Name, true);
+                    if (field.IsNullOrWhiteSpace())
+                        throw new InvalidOperationException(string.Format("The mapper returned an empty field for the member name: {0} for type: {1}", m.Member.Name, m.Expression.Type));
+                    return field;
+                }
+                //already compiled, return
+                return string.Empty;
+            }
+
+            //TODO: When m.Expression.NodeType == ExpressionType.Constant and it's an expression like: content => aliases.Contains(content.ContentType.Alias);
+            // then an SQL parameter will be added for aliases as an array, however in SqlIn on the subclass it will manually add these SqlParameters anyways,
+            // however the query will still execute because the SQL that is written will only contain the correct indexes of SQL parameters, this would be ignored,
+            // I'm just unsure right now due to time constraints how to make it correct. It won't matter right now and has been working already with this bug but I've
+            // only just discovered what it is actually doing.
+
+            var member = Expression.Convert(m, typeof(object));
+            var lambda = Expression.Lambda<Func<object>>(member);
+            var getter = lambda.Compile();
+            var o = getter();
+
+            SqlParameters.Add(o);
+
+            //don't execute if compiled
+            if (Visited == false)
+                return string.Format("@{0}", SqlParameters.Count - 1);
+            //already compiled, return
+            return string.Empty;
+
+        }
+
+        /// <summary>
+        /// Determines if the MemberExpression ends in a Constant value
+        /// </summary>
+        /// <param name="m"></param>
+        /// <returns></returns>
+        private bool EndsWithConstant(MemberExpression m)
+        {
+            Expression expr = m;
+            
+            while (expr is MemberExpression)
+            {
+                var memberExpr = expr as MemberExpression;
+                expr = memberExpr.Expression;
+            }
+            
+            var constExpr = expr as ConstantExpression;
+            return constExpr != null;
+        }
+    }
+}
@@ -1,62 +0,0 @@
-using System;
-using System.Collections;
-using System.Collections.Generic;
-using System.Collections.ObjectModel;
-using System.Linq;
-using System.Linq.Expressions;
-using System.Text;
-using Umbraco.Core.Persistence.SqlSyntax;
-
-namespace Umbraco.Core.Persistence.Querying
-{
-    internal class PocoToSqlExpressionHelper<T> : BaseExpressionHelper<T>
-    {
-        private readonly Database.PocoData _pd;
-
-        public PocoToSqlExpressionHelper()
-        {
-            _pd = new Database.PocoData(typeof(T));
-        }
-        
-        protected override string VisitMemberAccess(MemberExpression m)
-        {
-            if (m.Expression != null &&
-               m.Expression.NodeType == ExpressionType.Parameter
-               && m.Expression.Type == typeof(T))
-            {
-                string field = GetFieldName(_pd, m.Member.Name);
-                return field;
-            }
-
-            if (m.Expression != null && m.Expression.NodeType == ExpressionType.Convert)
-            {
-                string field = GetFieldName(_pd, m.Member.Name);
-                return field;
-            }
-
-            var member = Expression.Convert(m, typeof(object));
-            var lambda = Expression.Lambda<Func<object>>(member);
-            var getter = lambda.Compile();
-            object o = getter();
-
-            SqlParameters.Add(o);
-            return string.Format("@{0}", SqlParameters.Count - 1);
-
-            //return GetQuotedValue(o, o != null ? o.GetType() : null);
-
-        }
-        
-        protected virtual string GetFieldName(Database.PocoData pocoData, string name)
-        {
-            var column = pocoData.Columns.FirstOrDefault(x => x.Value.PropertyInfo.Name == name);
-            return string.Format("{0}.{1}",
-                SqlSyntaxContext.SqlSyntaxProvider.GetQuotedTableName(pocoData.TableInfo.TableName),
-                SqlSyntaxContext.SqlSyntaxProvider.GetQuotedColumnName(column.Value.ColumnName));
-        }
-        
-        //protected bool IsFieldName(string quotedExp)
-        //{
-        //    return true;
-        //}
-    }
-}
@@ -0,0 +1,80 @@
+using System;
+using System.Linq;
+using System.Linq.Expressions;
+using Umbraco.Core.Persistence.SqlSyntax;
+
+namespace Umbraco.Core.Persistence.Querying
+{
+    /// <summary>
+    /// An expression tree parser to create SQL statements and SQL parameters based on a strongly typed expression,
+    /// based on Umbraco's DTOs.
+    /// </summary>
+    /// <remarks>This object is stateful and cannot be re-used to parse an expression.</remarks>
+    internal class PocoToSqlExpressionVisitor<T> : ExpressionVisitorBase
+    {
+        private readonly Database.PocoData _pd;
+
+
+        public PocoToSqlExpressionVisitor(ISqlSyntaxProvider syntaxProvider)
+            : base(syntaxProvider)
+        {
+            _pd = new Database.PocoData(typeof(T));
+        }
+
+        [Obsolete("Use the overload the specifies a SqlSyntaxProvider")]
+        public PocoToSqlExpressionVisitor()
+            : this(SqlSyntaxContext.SqlSyntaxProvider)
+        {            
+        }
+
+        protected override string VisitMemberAccess(MemberExpression m)
+        {
+            if (m.Expression != null &&
+               m.Expression.NodeType == ExpressionType.Parameter
+               && m.Expression.Type == typeof(T))
+            {
+                //don't execute if compiled
+                if (Visited == false)
+                {
+                    string field = GetFieldName(_pd, m.Member.Name);
+                    return field;
+                }
+                //already compiled, return
+                return string.Empty;
+            }
+
+            if (m.Expression != null && m.Expression.NodeType == ExpressionType.Convert)
+            {
+                //don't execute if compiled
+                if (Visited == false)
+                {
+                    string field = GetFieldName(_pd, m.Member.Name);
+                    return field;
+                }
+                //already compiled, return
+                return string.Empty;
+            }
+
+            var member = Expression.Convert(m, typeof(object));
+            var lambda = Expression.Lambda<Func<object>>(member);
+            var getter = lambda.Compile();
+            var o = getter();
+
+            SqlParameters.Add(o);
+
+            //don't execute if compiled
+            if (Visited == false)
+                return string.Format("@{0}", SqlParameters.Count - 1);
+            //already compiled, return
+            return string.Empty;
+        }
+
+        protected virtual string GetFieldName(Database.PocoData pocoData, string name)
+        {
+            var column = pocoData.Columns.FirstOrDefault(x => x.Value.PropertyInfo.Name == name);
+            return string.Format("{0}.{1}",
+                SqlSyntax.GetQuotedTableName(pocoData.TableInfo.TableName),
+                SqlSyntax.GetQuotedColumnName(column.Value.ColumnName));
+        }
+    }
+}
@@ -30,7 +30,8 @@ namespace Umbraco.Core.Persistence.Querying
        {
            if (predicate != null)
            {
-                var expressionHelper = new ModelToSqlExpressionHelper<T>();
+                //TODO: This should have an SqlSyntax object passed in, this ctor is relying on a singleton
+                var expressionHelper = new ModelToSqlExpressionVisitor<T>();
                string whereExpression = expressionHelper.Visit(predicate);

                _wheres.Add(new Tuple<string, object[]>(whereExpression, expressionHelper.GetSqlParameters()));
@@ -0,0 +1,27 @@
+using System;
+using System.Collections.Generic;
+
+namespace Umbraco.Core.Persistence.Querying
+{
+    /// <summary>
+    /// SD: This is a horrible hack but unless we break compatibility with anyone who's actually implemented IQuery{T} there's not much we can do.
+    /// The IQuery{T} interface is useless without having a GetWhereClauses method and cannot be used for tests.
+    /// We have to wait till v8 to make this change I suppose.
+    /// </summary>
+    internal static class QueryExtensions
+    {
+        /// <summary>
+        /// Returns all translated where clauses and their sql parameters
+        /// </summary>
+        /// <returns></returns>
+        public static IEnumerable<Tuple<string, object[]>> GetWhereClauses<T>(this IQuery<T> query)
+        {
+            var q = query as Query<T>;
+            if (q == null)
+            {
+                throw new NotSupportedException(typeof(IQuery<T>) + " cannot be cast to " + typeof(Query<T>));
+            }
+            return q.GetWhereClauses();
+        }
+    }
+}
@@ -1,4 +1,5 @@
-using System;
+using System.Collections.Generic;
+using System.Linq;
 using System.Text.RegularExpressions;

 namespace Umbraco.Core.Persistence.Querying
@@ -6,8 +7,13 @@ namespace Umbraco.Core.Persistence.Querying
    /// <summary>
    /// String extension methods used specifically to translate into SQL
    /// </summary>
-    internal static class SqlStringExtensions
+    internal static class SqlExpressionExtensions
    {
+        public static bool SqlIn<T>(this IEnumerable<T> collection, T item)
+        {
+            return collection.Contains(item);
+        }
+
        public static bool SqlWildcard(this string str, string txt, TextColumnType columnType)
        {
            var wildcardmatch = new Regex("^" + Regex.Escape(txt).
@@ -0,0 +1,31 @@
+namespace Umbraco.Core.Persistence.Repositories
+{
+    internal enum BaseQueryType
+    {
+        /// <summary>
+        /// A query to return all information for a single item
+        /// </summary>
+        /// <remarks>
+        /// In some cases this will be the same as <see cref="FullMultiple"/>
+        /// </remarks>
+        FullSingle,
+
+        /// <summary>
+        /// A query to return all information for multiple items
+        /// </summary>
+        /// <remarks>
+        /// In some cases this will be the same as <see cref="FullSingle"/>
+        /// </remarks>
+        FullMultiple,
+
+        /// <summary>
+        /// A query to return the ids for items
+        /// </summary>
+        Ids,
+
+        /// <summary>
+        /// A query to return the count for items 
+        /// </summary>
+        Count
+    }
+}
@@ -1,22 +1,15 @@
 using System;
 using System.Collections.Generic;
-using System.Data;
+using System.Collections.ObjectModel;
 using System.Globalization;
 using System.Linq;
-using System.Linq.Expressions;
-using System.Net.Http.Headers;
-using System.Text;
 using System.Xml;
 using System.Xml.Linq;
-using Umbraco.Core.Configuration;
-using Umbraco.Core.Dynamics;
-using Umbraco.Core.IO;
 using Umbraco.Core.Logging;
 using Umbraco.Core.Models;
 using Umbraco.Core.Models.EntityBase;
 using Umbraco.Core.Models.Membership;
 using Umbraco.Core.Models.Rdbms;
-
 using Umbraco.Core.Persistence.DatabaseModelDefinitions;
 using Umbraco.Core.Persistence.Factories;
 using Umbraco.Core.Persistence.Querying;
@@ -61,9 +54,9 @@ namespace Umbraco.Core.Persistence.Repositories

        protected override IContent PerformGet(int id)
        {
-            var sql = GetBaseQuery(false)
+            var sql = GetBaseQuery(BaseQueryType.FullSingle)
                .Where(GetBaseWhereClause(), new { Id = id })
-                .Where<DocumentDto>(x => x.Newest)
+                .Where<DocumentDto>(x => x.Newest, SqlSyntax)
                .OrderByDescending<ContentVersionDto>(x => x.VersionDate, SqlSyntax);

            var dto = Database.Fetch<DocumentDto, ContentVersionDto, ContentDto, NodeDto, DocumentPublishedReadOnlyDto>(SqlSyntax.SelectTop(sql, 1)).FirstOrDefault();
@@ -71,70 +64,112 @@ namespace Umbraco.Core.Persistence.Repositories
            if (dto == null)
                return null;

-            var content = CreateContentFromDto(dto, dto.ContentVersionDto.VersionId, sql);
+            var content = CreateContentFromDto(dto, sql);

            return content;
        }

        protected override IEnumerable<IContent> PerformGetAll(params int[] ids)
        {
-            var sql = GetBaseQuery(false);
-            if (ids.Any())
+            Func<Sql, Sql> translate = s =>
            {
-                sql.Where("umbracoNode.id in (@ids)", new { ids = ids });
-            }
+                if (ids.Any())
+                {
+                    s.Where("umbracoNode.id in (@ids)", new { ids });
+                }
+                //we only want the newest ones with this method
+                s.Where<DocumentDto>(x => x.Newest, SqlSyntax);
+                return s;
+            };
+            
+            var sqlBaseFull = GetBaseQuery(BaseQueryType.FullMultiple);
+            var sqlBaseIds = GetBaseQuery(BaseQueryType.Ids);

-            //we only want the newest ones with this method
-            sql.Where<DocumentDto>(x => x.Newest);
-
-            return ProcessQuery(sql);
+            return ProcessQuery(translate(sqlBaseFull), new PagingSqlQuery(translate(sqlBaseIds)));
        }

        protected override IEnumerable<IContent> PerformGetByQuery(IQuery<IContent> query)
        {
-            var sqlClause = GetBaseQuery(false);
-            var translator = new SqlTranslator<IContent>(sqlClause, query);
-            var sql = translator.Translate()
-                                .Where<DocumentDto>(x => x.Newest)
-                                .OrderByDescending<ContentVersionDto>(x => x.VersionDate)
-                                .OrderBy<NodeDto>(x => x.SortOrder);
+            var sqlBaseFull = GetBaseQuery(BaseQueryType.FullMultiple);
+            var sqlBaseIds = GetBaseQuery(BaseQueryType.Ids);

-            return ProcessQuery(sql);
+            Func<SqlTranslator<IContent>, Sql> translate = (translator) =>
+            {
+                return translator.Translate()
+                    .Where<DocumentDto>(x => x.Newest, SqlSyntax)
+                    .OrderByDescending<ContentVersionDto>(x => x.VersionDate, SqlSyntax)
+                    .OrderBy<NodeDto>(x => x.SortOrder, SqlSyntax);
+            };
+
+            var translatorFull = new SqlTranslator<IContent>(sqlBaseFull, query);
+            var translatorIds = new SqlTranslator<IContent>(sqlBaseIds, query);
+
+            return ProcessQuery(translate(translatorFull), new PagingSqlQuery(translate(translatorIds)));
        }

-        #endregion
+        #endregion        

        #region Overrides of PetaPocoRepositoryBase<IContent>

-
-        protected override Sql GetBaseQuery(bool isCount)
+        /// <summary>
+        /// Returns the base query to return Content
+        /// </summary>
+        /// <param name="queryType"></param>
+        /// <returns></returns>
+        /// <remarks>
+        /// Content queries will differ depending on what needs to be returned:
+        /// * FullSingle: When querying for a single document, this will include the Outer join to fetch the content item's published version info
+        /// * FullMultiple: When querying for multiple documents, this will exclude the Outer join to fetch the content item's published version info - this info would need to be fetched separately
+        /// * Ids: This would essentially be the same as FullMultiple however the columns specified will only return the Ids for the documents
+        /// * Count: A query to return the count for documents
+        /// </remarks>
+        protected override Sql GetBaseQuery(BaseQueryType queryType)
        {
-            var sqlx = string.Format("LEFT OUTER JOIN {0} {1} ON ({1}.{2}={0}.{2} AND {1}.{3}=1)",
+            var sql = new Sql();
+            sql.Select(queryType == BaseQueryType.Count ? "COUNT(*)" : (queryType == BaseQueryType.Ids ? "cmsDocument.nodeId" : "*"))
+                .From<DocumentDto>(SqlSyntax)
+                .InnerJoin<ContentVersionDto>(SqlSyntax)
+                .On<DocumentDto, ContentVersionDto>(SqlSyntax, left => left.VersionId, right => right.VersionId)
+                .InnerJoin<ContentDto>(SqlSyntax)
+                .On<ContentVersionDto, ContentDto>(SqlSyntax, left => left.NodeId, right => right.NodeId)
+                .InnerJoin<NodeDto>(SqlSyntax)
+                .On<ContentDto, NodeDto>(SqlSyntax, left => left.NodeId, right => right.NodeId);
+            //TODO: IF we want to enable querying on content type information this will need to be joined
+            //.InnerJoin<ContentTypeDto>(SqlSyntax)
+            //.On<ContentDto, ContentTypeDto>(SqlSyntax, left => left.ContentTypeId, right => right.NodeId, SqlSyntax);
+
+            if (queryType == BaseQueryType.FullSingle)
+            {
+                //The only reason we apply this left outer join is to be able to pull back the DocumentPublishedReadOnlyDto
+                //information with the entire data set, so basically this will get both the latest document and also it's published
+                //version if it has one. When performing a count or when retrieving Ids like in paging, this is unecessary 
+                //and causes huge performance overhead for the SQL server, especially when sorting the result. 
+                //We also don't include this outer join when querying for multiple entities since it is much faster to fetch this information
+                //in a separate query. For a single entity this is ok.
+
+                var sqlx = string.Format("LEFT OUTER JOIN {0} {1} ON ({1}.{2}={0}.{2} AND {1}.{3}=1)",
                SqlSyntax.GetQuotedTableName("cmsDocument"),
                SqlSyntax.GetQuotedTableName("cmsDocument2"),
                SqlSyntax.GetQuotedColumnName("nodeId"),
                SqlSyntax.GetQuotedColumnName("published"));

-            var sql = new Sql();
-            sql.Select(isCount ? "COUNT(*)" : "*")
-                .From<DocumentDto>()
-                .InnerJoin<ContentVersionDto>()
-                .On<DocumentDto, ContentVersionDto>(left => left.VersionId, right => right.VersionId)
-                .InnerJoin<ContentDto>()
-                .On<ContentVersionDto, ContentDto>(left => left.NodeId, right => right.NodeId)
-                .InnerJoin<NodeDto>()
-                .On<ContentDto, NodeDto>(left => left.NodeId, right => right.NodeId)
-
                // cannot do this because PetaPoco does not know how to alias the table
                //.LeftOuterJoin<DocumentPublishedReadOnlyDto>()
                //.On<DocumentDto, DocumentPublishedReadOnlyDto>(left => left.NodeId, right => right.NodeId)
                // so have to rely on writing our own SQL
-                .Append(sqlx/*, new { @published = true }*/)
+                sql.Append(sqlx /*, new { @published = true }*/);
+            }
+
+            sql.Where<NodeDto>(x => x.NodeObjectType == NodeObjectTypeId, SqlSyntax);

-                .Where<NodeDto>(x => x.NodeObjectType == NodeObjectTypeId);
            return sql;
        }

+        protected override Sql GetBaseQuery(bool isCount)
+        {
+            return GetBaseQuery(isCount ? BaseQueryType.Count : BaseQueryType.FullSingle);
+        }
+
        protected override string GetBaseWhereClause()
        {
            return "umbracoNode.id = @Id";
@@ -173,104 +208,123 @@ namespace Umbraco.Core.Persistence.Repositories

        #region Overrides of VersionableRepositoryBase<IContent>

-        public void RebuildXmlStructures(Func<IContent, XElement> serializer, int groupSize = 5000, IEnumerable<int> contentTypeIds = null)
+        public void RebuildXmlStructures(Func<IContent, XElement> serializer, int groupSize = 200, IEnumerable<int> contentTypeIds = null)
        {
+            // the previous way of doing this was to run it all in one big transaction,
+            // and to bulk-insert groups of xml rows - which works, until the transaction
+            // times out - and besides, because v7 transactions are ReadCommited, it does
+            // not bring much safety - so this reverts to updating each record individually,
+            // and it may be slower in the end, but should be more resilient.

-            //Ok, now we need to remove the data and re-insert it, we'll do this all in one transaction too.
-            using (var tr = Database.GetTransaction())
+            var contentTypeIdsA = contentTypeIds == null ? new int[0] : contentTypeIds.ToArray();
+
+            Func<int, Sql, Sql> translate = (bId, sql) =>
            {
-                //Remove all the data first, if anything fails after this it's no problem the transaction will be reverted
-                if (contentTypeIds == null)
+                if (contentTypeIdsA.Length > 0)
                {
-                    var subQuery = new Sql()
-                            .Select("DISTINCT cmsContentXml.nodeId")
-                            .From<ContentXmlDto>()
-                            .InnerJoin<DocumentDto>()
-                            .On<ContentXmlDto, DocumentDto>(left => left.NodeId, right => right.NodeId);
-
-                    var deleteSql = SqlSyntax.GetDeleteSubquery("cmsContentXml", "nodeId", subQuery);
-                    Database.Execute(deleteSql);
+                    sql.WhereIn<ContentDto>(x => x.ContentTypeId, contentTypeIdsA, SqlSyntax);
                }
-                else
-                {
-                    foreach (var id in contentTypeIds)
-                    {
-                        var id1 = id;
-                        var subQuery = new Sql()
-                            .Select("cmsDocument.nodeId")
-                            .From<DocumentDto>()
-                            .InnerJoin<ContentDto>()
-                            .On<DocumentDto, ContentDto>(left => left.NodeId, right => right.NodeId)
-                            .Where<DocumentDto>(dto => dto.Published)
-                            .Where<ContentDto>(dto => dto.ContentTypeId == id1);

-                        var deleteSql = SqlSyntax.GetDeleteSubquery("cmsContentXml", "nodeId", subQuery);
-                        Database.Execute(deleteSql);
+                sql
+                    .Where<NodeDto>(x => x.NodeId > bId && x.Trashed == false, SqlSyntax)
+                    .Where<DocumentDto>(x => x.Published, SqlSyntax)
+                    .OrderBy<NodeDto>(x => x.NodeId, SqlSyntax);
+
+                return sql;
+            };
+
+            var baseId = 0;
+            
+            while (true)
+            {
+                // get the next group of nodes
+                var sqlFull = translate(baseId, GetBaseQuery(BaseQueryType.FullMultiple));
+                var sqlIds = translate(baseId, GetBaseQuery(BaseQueryType.Ids));
+
+                var xmlItems = ProcessQuery(SqlSyntax.SelectTop(sqlFull, groupSize), new PagingSqlQuery(SqlSyntax.SelectTop(sqlIds, groupSize)))
+                    .Select(x => new ContentXmlDto { NodeId = x.Id, Xml = serializer(x).ToString() })
+                    .ToList();
+
+                // no more nodes, break
+                if (xmlItems.Count == 0) break;
+
+                foreach (var xmlItem in xmlItems)
+                {
+                    try
+                    {
+                        // should happen in most cases, then it tries to insert, and it should work
+                        // unless the node has been deleted, and we just report the exception
+                        Database.InsertOrUpdate(xmlItem);
+                    }
+                    catch (Exception e)
+                    {
+                        Logger.Error<MediaRepository>("Could not rebuild XML for nodeId=" + xmlItem.NodeId, e);
                    }
                }
-
-                //now insert the data, again if something fails here, the whole transaction is reversed
-                if (contentTypeIds == null)
-                {
-                    var query = Query<IContent>.Builder.Where(x => x.Published == true);
-                    RebuildXmlStructuresProcessQuery(serializer, query, tr, groupSize);
-                }
-                else
-                {
-                    foreach (var contentTypeId in contentTypeIds)
-                    {
-                        //copy local
-                        var id = contentTypeId;
-                        var query = Query<IContent>.Builder.Where(x => x.Published == true && x.ContentTypeId == id && x.Trashed == false);
-                        RebuildXmlStructuresProcessQuery(serializer, query, tr, groupSize);
-                    }
-                }
-
-                tr.Complete();
+                baseId = xmlItems[xmlItems.Count - 1].NodeId;
            }
+
+            //now delete the items that shouldn't be there
+            var sqlAllIds = translate(0, GetBaseQuery(BaseQueryType.Ids));
+            var allContentIds = Database.Fetch<int>(sqlAllIds);
+            var docObjectType = Guid.Parse(Constants.ObjectTypes.Document);
+            var xmlIdsQuery = new Sql()
+                .Select("DISTINCT cmsContentXml.nodeId")
+                .From<ContentXmlDto>(SqlSyntax)
+                .InnerJoin<NodeDto>(SqlSyntax)
+                .On<ContentXmlDto, NodeDto>(SqlSyntax, left => left.NodeId, right => right.NodeId);
+
+            if (contentTypeIdsA.Length > 0)
+            {
+                xmlIdsQuery.InnerJoin<ContentDto>(SqlSyntax)
+                    .On<ContentDto, NodeDto>(SqlSyntax, left => left.NodeId, right => right.NodeId)
+                    .InnerJoin<ContentTypeDto>(SqlSyntax)
+                    .On<ContentTypeDto, ContentDto>(SqlSyntax, left => left.NodeId, right => right.ContentTypeId)
+                    .WhereIn<ContentDto>(x => x.ContentTypeId, contentTypeIdsA, SqlSyntax);
+            }
+
+            xmlIdsQuery.Where<NodeDto>(dto => dto.NodeObjectType == docObjectType, SqlSyntax);
+            
+            var allXmlIds = Database.Fetch<int>(xmlIdsQuery);
+
+            var toRemove = allXmlIds.Except(allContentIds).ToArray();
+            if (toRemove.Length > 0)
+            {
+                foreach (var idGroup in toRemove.InGroupsOf(2000))
+                {
+                    Database.Execute("DELETE FROM cmsContentXml WHERE nodeId IN (@ids)", new { ids = idGroup });
+                }                
+            }
+                
        }

-        private void RebuildXmlStructuresProcessQuery(Func<IContent, XElement> serializer, IQuery<IContent> query, Transaction tr, int pageSize)
+        public override IEnumerable<IContent> GetAllVersions(int id)
        {
-            var pageIndex = 0;
-            var total = long.MinValue;
-            var processed = 0;
-            do
+            Func<Sql, Sql> translate = s =>
            {
-                //NOTE: This is an important call, we cannot simply make a call to:
-                //  GetPagedResultsByQuery(query, pageIndex, pageSize, out total, "Path", Direction.Ascending);
-                // because that method is used to query 'latest' content items where in this case we don't necessarily
-                // want latest content items because a pulished content item might not actually be the latest.
-                // see: http://issues.umbraco.org/issue/U4-6322 & http://issues.umbraco.org/issue/U4-5982
-                var descendants = GetPagedResultsByQuery<DocumentDto, Content>(query, pageIndex, pageSize, out total,
-                    new Tuple<string, string>("cmsDocument", "nodeId"),
-                    ProcessQuery, "Path", Direction.Ascending, true);
+                return s.Where(GetBaseWhereClause(), new {Id = id})
+                    .OrderByDescending<ContentVersionDto>(x => x.VersionDate, SqlSyntax);
+            };

-                var xmlItems = (from descendant in descendants
-                                let xml = serializer(descendant)
-                                select new ContentXmlDto { NodeId = descendant.Id, Xml = xml.ToDataString() }).ToArray();
-
-                //bulk insert it into the database
-                Database.BulkInsertRecords(xmlItems, tr);
-
-                processed += xmlItems.Length;
-
-                pageIndex++;
-            } while (processed < total);
+            var sqlFull = translate(GetBaseQuery(BaseQueryType.FullMultiple));
+            var sqlIds = translate(GetBaseQuery(BaseQueryType.Ids));
+            
+            return ProcessQuery(sqlFull, new PagingSqlQuery(sqlIds), true,  includeAllVersions:true);
        }

        public override IContent GetByVersion(Guid versionId)
        {
-            var sql = GetBaseQuery(false);
+            var sql = GetBaseQuery(BaseQueryType.FullSingle);
+            //TODO: cmsContentVersion.VersionId has a Unique Index constraint applied, seems silly then to also add OrderByDescending since it would be impossible to return more than one.
            sql.Where("cmsContentVersion.VersionId = @VersionId", new { VersionId = versionId });
-            sql.OrderByDescending<ContentVersionDto>(x => x.VersionDate);
+            sql.OrderByDescending<ContentVersionDto>(x => x.VersionDate, SqlSyntax);

            var dto = Database.Fetch<DocumentDto, ContentVersionDto, ContentDto, NodeDto, DocumentPublishedReadOnlyDto>(sql).FirstOrDefault();

            if (dto == null)
                return null;

-            var content = CreateContentFromDto(dto, versionId, sql);
+            var content = CreateContentFromDto(dto, sql);

            return content;
        }
@@ -279,10 +333,11 @@ namespace Umbraco.Core.Persistence.Repositories
        {
            var sql = new Sql()
                .Select("*")
-                .From<DocumentDto>()
-                .InnerJoin<ContentVersionDto>().On<ContentVersionDto, DocumentDto>(left => left.VersionId, right => right.VersionId)
-                .Where<ContentVersionDto>(x => x.VersionId == versionId)
-                .Where<DocumentDto>(x => x.Newest != true);
+                .From<DocumentDto>(SqlSyntax)
+                .InnerJoin<ContentVersionDto>(SqlSyntax)
+                .On<ContentVersionDto, DocumentDto>(SqlSyntax, left => left.VersionId, right => right.VersionId)
+                .Where<ContentVersionDto>(x => x.VersionId == versionId, SqlSyntax)
+                .Where<DocumentDto>(x => x.Newest != true, SqlSyntax);
            var dto = Database.Fetch<DocumentDto, ContentVersionDto>(sql).FirstOrDefault();

            if (dto == null) return;
@@ -300,7 +355,8 @@ namespace Umbraco.Core.Persistence.Repositories
            var sql = new Sql()
                .Select("*")
                .From<DocumentDto>()
-                .InnerJoin<ContentVersionDto>().On<ContentVersionDto, DocumentDto>(left => left.VersionId, right => right.VersionId)
+                .InnerJoin<ContentVersionDto>()
+                .On<ContentVersionDto, DocumentDto>(left => left.VersionId, right => right.VersionId)
                .Where<ContentVersionDto>(x => x.NodeId == id)
                .Where<ContentVersionDto>(x => x.VersionDate < versionDate)
                .Where<DocumentDto>(x => x.Newest != true);
@@ -383,6 +439,7 @@ namespace Umbraco.Core.Persistence.Repositories

            //Update with new correct path
            nodeDto.Path = string.Concat(parent.Path, ",", nodeDto.NodeId);
+            nodeDto.ValidatePathWithException();
            Database.Update(nodeDto);

            //Update entity with correct values
@@ -456,11 +513,13 @@ namespace Umbraco.Core.Persistence.Repositories
                dto.DocumentPublishedReadOnlyDto = new DocumentPublishedReadOnlyDto
                {
                    VersionId = dto.VersionId,
+                    VersionDate = dto.UpdateDate,
                    Newest = true,
                    NodeId = dto.NodeId,
-                    Published = true
+                    Published = true               
                };
-                ((Content)entity).PublishedVersionGuid = dto.VersionId;
+                ((Content) entity).PublishedVersionGuid = dto.VersionId;
+                ((Content) entity).PublishedDate = dto.UpdateDate;
            }

            entity.ResetDirtyProperties();
@@ -520,6 +579,7 @@ namespace Umbraco.Core.Persistence.Repositories

            //Updates the (base) node data - umbracoNode
            var nodeDto = dto.ContentVersionDto.ContentDto.NodeDto;
+            nodeDto.ValidatePathWithException();
            var o = Database.Update(nodeDto);

            //Only update this DTO if the contentType has actually changed
@@ -538,6 +598,7 @@ namespace Umbraco.Core.Persistence.Repositories
            //if (((ICanBeDirty)entity).IsPropertyDirty("Published") && (entity.Published || publishedState == PublishedState.Unpublished))
            if (entity.ShouldClearPublishedFlagForPreviousVersions(publishedState, shouldCreateNewVersion))
            {
+                //TODO: This perf can be improved, it could easily be UPDATE WHERE.... (one SQL call instead of many)
                var publishedDocs = Database.Fetch<DocumentDto>("WHERE nodeId = @Id AND published = @IsPublished", new { Id = entity.Id, IsPublished = true });
                foreach (var doc in publishedDocs)
                {
@@ -551,6 +612,7 @@ namespace Umbraco.Core.Persistence.Repositories
            }

            //Look up (newest) entries by id in cmsDocument table to set newest = false
+            //TODO: This perf can be improved, it could easily be UPDATE WHERE.... (one SQL call instead of many)
            var documentDtos = Database.Fetch<DocumentDto>("WHERE nodeId = @Id AND newest = @IsNewest", new { Id = entity.Id, IsNewest = true });
            foreach (var documentDto in documentDtos)
            {
@@ -627,22 +689,26 @@ namespace Umbraco.Core.Persistence.Repositories
                dto.DocumentPublishedReadOnlyDto = new DocumentPublishedReadOnlyDto
                {
                    VersionId = dto.VersionId,
+                    VersionDate = dto.UpdateDate,
                    Newest = true,
                    NodeId = dto.NodeId,
                    Published = true
                };
-                ((Content)entity).PublishedVersionGuid = dto.VersionId;
+                ((Content) entity).PublishedVersionGuid = dto.VersionId;
+                ((Content) entity).PublishedDate = dto.UpdateDate;
            }
            else if (publishedStateChanged)
            {
                dto.DocumentPublishedReadOnlyDto = new DocumentPublishedReadOnlyDto
                {
-                    VersionId = default(Guid),
+                    VersionId = default (Guid),
+                    VersionDate = default (DateTime),
                    Newest = false,
                    NodeId = dto.NodeId,
                    Published = false
                };
-                ((Content)entity).PublishedVersionGuid = default(Guid);
+                ((Content) entity).PublishedVersionGuid = default(Guid);
+                ((Content) entity).PublishedDate = default (DateTime);
            }

            entity.ResetDirtyProperties();
@@ -655,41 +721,27 @@ namespace Umbraco.Core.Persistence.Repositories

        public IEnumerable<IContent> GetByPublishedVersion(IQuery<IContent> query)
        {
+            Func<SqlTranslator<IContent>, Sql> translate = t =>
+            {
+                return t.Translate()
+                    .Where<DocumentDto>(x => x.Published, SqlSyntax)
+                    .OrderBy<NodeDto>(x => x.Level, SqlSyntax)
+                    .OrderBy<NodeDto>(x => x.SortOrder, SqlSyntax);
+            };
+
            // we WANT to return contents in top-down order, ie parents should come before children
            // ideal would be pure xml "document order" which can be achieved with:
            // ORDER BY substring(path, 1, len(path) - charindex(',', reverse(path))), sortOrder
            // but that's probably an overkill - sorting by level,sortOrder should be enough

-            var sqlClause = GetBaseQuery(false);
-            var translator = new SqlTranslator<IContent>(sqlClause, query);
-            var sql = translator.Translate()
-                                .Where<DocumentDto>(x => x.Published)
-                                .OrderBy<NodeDto>(x => x.Level, SqlSyntax)
-                                .OrderBy<NodeDto>(x => x.SortOrder, SqlSyntax);
+            var sqlFull = GetBaseQuery(BaseQueryType.FullMultiple);
+            var translatorFull = new SqlTranslator<IContent>(sqlFull, query);
+            var sqlIds = GetBaseQuery(BaseQueryType.Ids);
+            var translatorIds = new SqlTranslator<IContent>(sqlIds, query);

-            //NOTE: This doesn't allow properties to be part of the query
-            var dtos = Database.Fetch<DocumentDto, ContentVersionDto, ContentDto, NodeDto, DocumentPublishedReadOnlyDto>(sql);
-
-            foreach (var dto in dtos)
-            {
-                //Check in the cache first. If it exists there AND it is published
-                // then we can use that entity. Otherwise if it is not published (which can be the case
-                // because we only store the 'latest' entries in the cache which might not be the published
-                // version)
-                var fromCache = RuntimeCache.GetCacheItem<IContent>(GetCacheIdKey<IContent>(dto.NodeId));
-                //var fromCache = TryGetFromCache(dto.NodeId);
-                if (fromCache != null && fromCache.Published)
-                {
-                    yield return fromCache;
-                }
-                else
-                {
-                    yield return CreateContentFromDto(dto, dto.VersionId, sql);
-                }
-            }
+            return ProcessQuery(translate(translatorFull), new PagingSqlQuery(translate(translatorIds)), true);
        }
-
-
+        
        /// <summary>
        /// This builds the Xml document used for the XML cache
        /// </summary>
@@ -721,10 +773,14 @@ namespace Umbraco.Core.Persistence.Repositories
            parent.Attributes.Append(pIdAtt);
            xmlDoc.AppendChild(parent);

-            const string sql = @"select umbracoNode.id, umbracoNode.parentID, umbracoNode.sortOrder, cmsContentXml.xml, umbracoNode.level from umbracoNode
+            //Ensure that only nodes that have published versions are selected
+            var sql = string.Format(@"select umbracoNode.id, umbracoNode.parentID, umbracoNode.sortOrder, cmsContentXml.{0}, umbracoNode.{1} from umbracoNode
 inner join cmsContentXml on cmsContentXml.nodeId = umbracoNode.id and umbracoNode.nodeObjectType = @type
 where umbracoNode.id in (select cmsDocument.nodeId from cmsDocument where cmsDocument.published = 1)
-order by umbracoNode.level, umbracoNode.parentID, umbracoNode.sortOrder";
+order by umbracoNode.{2}, umbracoNode.parentID, umbracoNode.sortOrder",
+                SqlSyntax.GetQuotedColumnName("xml"),
+                SqlSyntax.GetQuotedColumnName("level"),
+                SqlSyntax.GetQuotedColumnName("level"));

            XmlElement last = null;

@@ -775,13 +831,8 @@ order by umbracoNode.level, umbracoNode.parentID, umbracoNode.sortOrder";

        public void ClearPublished(IContent content)
        {
-            // race cond!
-            var documentDtos = Database.Fetch<DocumentDto>("WHERE nodeId=@id AND published=@published", new { id = content.Id, published = true });
-            foreach (var documentDto in documentDtos)
-            {
-                documentDto.Published = false;
-                Database.Update(documentDto);
-            }
+            var sql = "UPDATE cmsDocument SET published=0 WHERE nodeId=@id AND published=1";
+            Database.Execute(sql, new {id = content.Id});
        }

        /// <summary>
@@ -861,9 +912,9 @@ order by umbracoNode.level, umbracoNode.parentID, umbracoNode.sortOrder";
            
            Func<Tuple<string, object[]>> filterCallback = () => new Tuple<string, object[]>(filterSql.SQL, filterSql.Arguments);

-            return GetPagedResultsByQuery<DocumentDto, Content>(query, pageIndex, pageSize, out totalRecords,
+            return GetPagedResultsByQuery<DocumentDto>(query, pageIndex, pageSize, out totalRecords,
                new Tuple<string, string>("cmsDocument", "nodeId"),
-                ProcessQuery, orderBy, orderDirection, orderBySystemField,
+                (sqlFull, pagingSqlQuery) => ProcessQuery(sqlFull, pagingSqlQuery), orderBy, orderDirection, orderBySystemField,
                filterCallback);

        }
@@ -894,99 +945,155 @@ order by umbracoNode.level, umbracoNode.parentID, umbracoNode.sortOrder";
            return base.GetDatabaseFieldNameForOrderBy(orderBy);
        }

-        private IEnumerable<IContent> ProcessQuery(Sql sql)
-        {
-            //NOTE: This doesn't allow properties to be part of the query
-            var dtos = Database.Fetch<DocumentDto, ContentVersionDto, ContentDto, NodeDto, DocumentPublishedReadOnlyDto>(sql);
-
-            //nothing found
-            if (dtos.Any() == false) return Enumerable.Empty<IContent>();
-
-            //content types
-            //NOTE: This should be ok for an SQL 'IN' statement, there shouldn't be an insane amount of content types
-            var contentTypes = _contentTypeRepository.GetAll(dtos.Select(x => x.ContentVersionDto.ContentDto.ContentTypeId).ToArray())
-                .ToArray();
-
-
-            var ids = dtos
-                .Where(dto => dto.TemplateId.HasValue && dto.TemplateId.Value > 0)
-                .Select(x => x.TemplateId.Value).ToArray();
-
-            //NOTE: This should be ok for an SQL 'IN' statement, there shouldn't be an insane amount of content types
-            var templates = ids.Length == 0 ? Enumerable.Empty<ITemplate>() : _templateRepository.GetAll(ids).ToArray();
-
-            var dtosWithContentTypes = dtos
-                //This select into and null check are required because we don't have a foreign damn key on the contentType column
-                // http://issues.umbraco.org/issue/U4-5503
-                .Select(x => new { dto = x, contentType = contentTypes.FirstOrDefault(ct => ct.Id == x.ContentVersionDto.ContentDto.ContentTypeId) })
-                .Where(x => x.contentType != null)
-                .ToArray();
-
-            //Go get the property data for each document
-            var docDefs = dtosWithContentTypes.Select(d => new DocumentDefinition(
-                d.dto.NodeId,
-                d.dto.VersionId,
-                d.dto.ContentVersionDto.VersionDate,
-                d.dto.ContentVersionDto.ContentDto.NodeDto.CreateDate,
-                d.contentType));
-
-            var propertyData = GetPropertyCollection(sql, docDefs);
-
-            return dtosWithContentTypes.Select(d => CreateContentFromDto(
-                d.dto,
-                contentTypes.First(ct => ct.Id == d.dto.ContentVersionDto.ContentDto.ContentTypeId),
-                templates.FirstOrDefault(tem => tem.Id == (d.dto.TemplateId.HasValue ? d.dto.TemplateId.Value : -1)),
-                propertyData[d.dto.NodeId]));
-        }
-
        /// <summary>
-        /// Private method to create a content object from a DocumentDto, which is used by Get and GetByVersion.
+        /// This is the underlying method that processes most queries for this repository
        /// </summary>
-        /// <param name="dto"></param>
-        /// <param name="contentType"></param>
-        /// <param name="template"></param>
-        /// <param name="propCollection"></param>
+        /// <param name="sqlFull">
+        /// The FullMultiple SQL without the outer join to return all data required to create an IContent excluding it's published state data which this will query separately
+        /// </param>
+        /// <param name="pagingSqlQuery">
+        /// The Id SQL without the outer join to just return all document ids - used to process the properties for the content item
+        /// </param>
+        /// <param name="withCache"></param>
+        /// <param name="includeAllVersions">
+        /// Generally when querying for content we only want to return the most recent version of the content item, however in some cases like when 
+        /// we want to return all versions of a content item, we can't simply return the latest
+        /// </param>
        /// <returns></returns>
-        private IContent CreateContentFromDto(DocumentDto dto,
-            IContentType contentType,
-            ITemplate template,
-            Models.PropertyCollection propCollection)
+        private IEnumerable<IContent> ProcessQuery(Sql sqlFull, PagingSqlQuery pagingSqlQuery, bool withCache = false, bool includeAllVersions = false)
        {
-            var factory = new ContentFactory(contentType, NodeObjectTypeId, dto.NodeId);
-            var content = factory.BuildEntity(dto);
+            // fetch returns a list so it's ok to iterate it in this method
+            var dtos = Database.Fetch<DocumentDto, ContentVersionDto, ContentDto, NodeDto>(sqlFull);
+            if (dtos.Count == 0) return Enumerable.Empty<IContent>();
+            
+            //Go and get all of the published version data separately for this data, this is because when we are querying
+            //for multiple content items we don't include the outer join to fetch this data in the same query because
+            //it is insanely slow. Instead we just fetch the published version data separately in one query.

-            //Check if template id is set on DocumentDto, and get ITemplate if it is.
-            if (dto.TemplateId.HasValue && dto.TemplateId.Value > 0)
+            //we need to parse the original SQL statement and reduce the columns to just cmsDocument.nodeId so that we can use
+            // the statement to go get the published data for all of the items by using an inner join
+            var parsedOriginalSql = "SELECT cmsDocument.nodeId " + sqlFull.SQL.Substring(sqlFull.SQL.IndexOf("FROM", StringComparison.Ordinal));
+            //now remove everything from an Orderby clause and beyond
+            if (parsedOriginalSql.InvariantContains("ORDER BY "))
            {
-                content.Template = template ?? _templateRepository.Get(dto.TemplateId.Value);
-            }
-            else
+                parsedOriginalSql = parsedOriginalSql.Substring(0, parsedOriginalSql.LastIndexOf("ORDER BY ", StringComparison.Ordinal));
+            }            
+
+            //order by update date DESC, if there is corrupted published flags we only want the latest!
+            var publishedSql = new Sql(@"SELECT cmsDocument.nodeId, cmsDocument.published, cmsDocument.versionId, cmsDocument.updateDate, cmsDocument.newest
+FROM cmsDocument INNER JOIN cmsContentVersion ON cmsContentVersion.VersionId = cmsDocument.versionId
+WHERE cmsDocument.published = 1 AND cmsDocument.nodeId IN 
+(" + parsedOriginalSql + @")
+ORDER BY cmsContentVersion.id DESC
+", sqlFull.Arguments);
+
+            //go and get the published version data, we do a Query here and not a Fetch so we are
+            //not allocating a whole list to memory just to allocate another list in memory since
+            //we are assigning this data to a keyed collection for fast lookup below
+            var publishedData = Database.Query<DocumentPublishedReadOnlyDto>(publishedSql);
+            var publishedDataCollection = new DocumentPublishedReadOnlyDtoCollection();
+            foreach (var publishedDto in publishedData)
            {
-                //ensure there isn't one set.
-                content.Template = null;
+                //double check that there's no corrupt db data, there should only be a single published item
+                if (publishedDataCollection.Contains(publishedDto.NodeId) == false)
+                    publishedDataCollection.Add(publishedDto);
            }

-            content.Properties = propCollection;
+            //This is a tuple list identifying if the content item came from the cache or not
+            var content = new List<Tuple<IContent, bool>>();
+            var defs = new DocumentDefinitionCollection(includeAllVersions);
+            var templateIds = new List<int>();
+            
+            //track the looked up content types, even though the content types are cached
+            // they still need to be deep cloned out of the cache and we don't want to add
+            // the overhead of deep cloning them on every item in this loop
+            var contentTypes = new Dictionary<int, IContentType>();
+            
+            foreach (var dto in dtos)
+            {
+                DocumentPublishedReadOnlyDto publishedDto;
+                publishedDataCollection.TryGetValue(dto.NodeId, out publishedDto);

-            //on initial construction we don't want to have dirty properties tracked
-            // http://issues.umbraco.org/issue/U4-1946
-            ((Entity)content).ResetDirtyProperties(false);
-            return content;
+                // if the cache contains the published version, use it
+                if (withCache)
+                {
+                    var cached = RuntimeCache.GetCacheItem<IContent>(GetCacheIdKey<IContent>(dto.NodeId));
+                    //only use this cached version if the dto returned is also the publish version, they must match and be teh same version
+                    if (cached != null && cached.Version == dto.VersionId && cached.Published && dto.Published)
+                    {
+                        content.Add(new Tuple<IContent, bool>(cached, true));
+                        continue;
+                    }
+                }
+
+                // else, need to fetch from the database
+                // content type repository is full-cache so OK to get each one independently
+
+                IContentType contentType;
+                if (contentTypes.ContainsKey(dto.ContentVersionDto.ContentDto.ContentTypeId))
+                {
+                    contentType = contentTypes[dto.ContentVersionDto.ContentDto.ContentTypeId];
+                }
+                else
+                {
+                    contentType = _contentTypeRepository.Get(dto.ContentVersionDto.ContentDto.ContentTypeId);
+                    contentTypes[dto.ContentVersionDto.ContentDto.ContentTypeId] = contentType;
+                }
+
+                // track the definition and if it's successfully added or updated then processed
+                if (defs.AddOrUpdate(new DocumentDefinition(dto, contentType)))
+                {
+                    // assign template
+                    if (dto.TemplateId.HasValue && dto.TemplateId.Value > 0)
+                        templateIds.Add(dto.TemplateId.Value);
+
+                    content.Add(new Tuple<IContent, bool>(ContentFactory.BuildEntity(dto, contentType, publishedDto), false));
+                }
+            }
+
+            // load all required templates in 1 query
+            var templates = _templateRepository.GetAll(templateIds.ToArray())
+                .ToDictionary(x => x.Id, x => x);
+
+            // load all properties for all documents from database in 1 query
+            var propertyData = GetPropertyCollection(pagingSqlQuery, defs);
+
+            // assign template and property data
+            foreach (var contentItem in content)
+            {
+                var cc = contentItem.Item1;
+                var fromCache = contentItem.Item2;
+
+                //if this has come from cache, we do not need to build up it's structure
+                if (fromCache) continue;
+
+                var def = defs[includeAllVersions ? (ValueType)cc.Version : cc.Id];
+
+                ITemplate template = null;
+                if (def.DocumentDto.TemplateId.HasValue)
+                    templates.TryGetValue(def.DocumentDto.TemplateId.Value, out template); // else null
+                cc.Template = template;
+                cc.Properties = propertyData[cc.Version];
+
+                //on initial construction we don't want to have dirty properties tracked
+                // http://issues.umbraco.org/issue/U4-1946
+                cc.ResetDirtyProperties(false);
+            }
+
+            return content.Select(x => x.Item1).ToArray();
        }

        /// <summary>
        /// Private method to create a content object from a DocumentDto, which is used by Get and GetByVersion.
        /// </summary>
        /// <param name="dto"></param>
-        /// <param name="versionId"></param>
        /// <param name="docSql"></param>
        /// <returns></returns>
-        private IContent CreateContentFromDto(DocumentDto dto, Guid versionId, Sql docSql)
+        private IContent CreateContentFromDto(DocumentDto dto, Sql docSql)
        {
            var contentType = _contentTypeRepository.Get(dto.ContentVersionDto.ContentDto.ContentTypeId);

-            var factory = new ContentFactory(contentType, NodeObjectTypeId, dto.NodeId);
-            var content = factory.BuildEntity(dto);
+            var content = ContentFactory.BuildEntity(dto, contentType);

            //Check if template id is set on DocumentDto, and get ITemplate if it is.
            if (dto.TemplateId.HasValue && dto.TemplateId.Value > 0)
@@ -994,11 +1101,11 @@ order by umbracoNode.level, umbracoNode.parentID, umbracoNode.sortOrder";
                content.Template = _templateRepository.Get(dto.TemplateId.Value);
            }

-            var docDef = new DocumentDefinition(dto.NodeId, versionId, content.UpdateDate, content.CreateDate, contentType);
+            var docDef = new DocumentDefinition(dto, contentType);

            var properties = GetPropertyCollection(docSql, new[] { docDef });

-            content.Properties = properties[dto.NodeId];
+            content.Properties = properties[dto.VersionId];

            //on initial construction we don't want to have dirty properties tracked
            // http://issues.umbraco.org/issue/U4-1946
@@ -1037,7 +1144,7 @@ order by umbracoNode.level, umbracoNode.parentID, umbracoNode.sortOrder";

            return currentName;
        }
-
+        
        /// <summary>
        /// Dispose disposable properties
        /// </summary>
@@ -1052,5 +1159,26 @@ order by umbracoNode.level, umbracoNode.parentID, umbracoNode.sortOrder";
            _contentPreviewRepository.Dispose();
            _contentXmlRepository.Dispose();
        }
+
+        /// <summary>
+        /// A keyed collection for fast lookup when retrieving a separate list of published version data
+        /// </summary>
+        private class DocumentPublishedReadOnlyDtoCollection : KeyedCollection<int, DocumentPublishedReadOnlyDto>
+        {
+            protected override int GetKeyForItem(DocumentPublishedReadOnlyDto item)
+            {
+                return item.NodeId;
+            }
+
+            public bool TryGetValue(int key, out DocumentPublishedReadOnlyDto val)
+            {
+                if (Dictionary == null)
+                {
+                    val = null;
+                    return false;
+                }
+                return Dictionary.TryGetValue(key, out val);
+            }
+        }
    }
 }
@@ -1250,5 +1250,19 @@ WHERE cmsContentType." + aliasColumn + @" LIKE @pattern",
            while (aliases.Contains(test = alias + i)) i++;
            return test;
        }
+
+        /// <summary>
+        /// Given the path of a content item, this will return true if the content item exists underneath a list view content item
+        /// </summary>
+        /// <param name="contentPath"></param>
+        /// <returns></returns>
+        public bool HasContainerInPath(string contentPath)
+        {
+            var ids = contentPath.Split(',').Select(int.Parse);
+            var sql = new Sql(@"SELECT COUNT(*) FROM cmsContentType
+INNER JOIN cmsContent ON cmsContentType.nodeId=cmsContent.contentType 
+WHERE cmsContent.nodeId IN (@ids) AND cmsContentType.isContainer=@isContainer", new { ids, isContainer = true });
+            return Database.ExecuteScalar<int>(sql) > 0;
+        }
    }
 }
@@ -2,19 +2,13 @@
 using System.Collections.Generic;
 using System.Linq;
 using Umbraco.Core.Cache;
-using Umbraco.Core.Events;
-using Umbraco.Core.Exceptions;
 using Umbraco.Core.Logging;
 using Umbraco.Core.Models;
 using Umbraco.Core.Models.EntityBase;
 using Umbraco.Core.Models.Rdbms;
-
-using Umbraco.Core.Persistence.Factories;
 using Umbraco.Core.Persistence.Querying;
-using Umbraco.Core.Persistence.Relators;
 using Umbraco.Core.Persistence.SqlSyntax;
 using Umbraco.Core.Persistence.UnitOfWork;
-using Umbraco.Core.Services;

 namespace Umbraco.Core.Persistence.Repositories
 {
@@ -118,7 +112,7 @@ namespace Umbraco.Core.Persistence.Repositories

            if (objectTypes.Any())
            {
-                sql = sql.Where("umbracoNode.nodeObjectType IN (@objectTypes)", objectTypes);
+                sql = sql.Where("umbracoNode.nodeObjectType IN (@objectTypes)", new {objectTypes = objectTypes});
            }

            return Database.Fetch<string>(sql);
@@ -34,7 +34,7 @@ namespace Umbraco.Core.Persistence.Repositories
            : base(work, cache, logger, sqlSyntax)
        {
            _contentTypeRepository = contentTypeRepository;
-            _preValRepository = new DataTypePreValueRepository(work, CacheHelper.CreateDisabledCacheHelper(), logger, sqlSyntax);            
+            _preValRepository = new DataTypePreValueRepository(work, CacheHelper.CreateDisabledCacheHelper(), logger, sqlSyntax);
        }

        #region Overrides of RepositoryBase<int,DataTypeDefinition>
@@ -276,47 +276,33 @@ AND umbracoNode.id <> @id",

        public PreValueCollection GetPreValuesCollectionByDataTypeId(int dataTypeId)
        {
-            var cached = RuntimeCache.GetCacheItemsByKeySearch<PreValueCollection>(GetPrefixedCacheKey(dataTypeId));
-            if (cached != null && cached.Any())
-            {
-                //return from the cache, ensure it's a cloned result
-                return (PreValueCollection)cached.First().DeepClone();
-            }
-
-            return GetAndCachePreValueCollection(dataTypeId);
-        }
-
-        internal static string GetCacheKeyRegex(int preValueId)
-        {
-            return CacheKeys.DataTypePreValuesCacheKey + @"[-\d]+-([\d]*,)*" + preValueId + @"(?!\d)[,\d$]*";
+            var collection = GetCachedPreValueCollection(dataTypeId);
+            return collection;
        }

+        /// <summary>
+        /// Gets a specific PreValue by its Id
+        /// </summary>
+        /// <param name="preValueId">Id of the PreValue to retrieve the value from</param>
+        /// <returns>PreValue as a string</returns>
        public string GetPreValueAsString(int preValueId)
        {
-            //We need to see if we can find the cached PreValueCollection based on the cache key above
+            var collections = RuntimeCache.GetCacheItemsByKeySearch<PreValueCollection>(CacheKeys.DataTypePreValuesCacheKey + "_");

-            var cached = RuntimeCache.GetCacheItemsByKeyExpression<PreValueCollection>(GetCacheKeyRegex(preValueId));
-            if (cached != null && cached.Any())
-            {
-                //return from the cache
-                var collection = cached.First();
-				var preVal = collection.FormatAsDictionary().Single(x => x.Value.Id == preValueId);
-				return preVal.Value.Value;
-            }
+            var preValue = collections.SelectMany(x => x.FormatAsDictionary().Values).FirstOrDefault(x => x.Id == preValueId);
+            if (preValue != null)
+                return preValue.Value;

-            //go and find the data type id for the pre val id passed in
-
-            var dto = Database.FirstOrDefault<DataTypePreValueDto>("WHERE id = @preValueId", new { preValueId = preValueId });
+            var dto = Database.FirstOrDefault<DataTypePreValueDto>("WHERE id = @preValueId", new { preValueId });
            if (dto == null)
-            {
                return string.Empty;
-            }
-            // go cache the collection
-            var preVals = GetAndCachePreValueCollection(dto.DataTypeNodeId);

-            //return the single value for this id
-            var pv = preVals.FormatAsDictionary().Single(x => x.Value.Id == preValueId);
-            return pv.Value.Value;
+            var collection = GetCachedPreValueCollection(dto.DataTypeNodeId);
+            if (collection == null)
+                return string.Empty;
+
+            preValue = collection.FormatAsDictionary().Values.FirstOrDefault(x => x.Id == preValueId);
+            return preValue == null ? string.Empty : preValue.Value;
        }

        public void AddOrUpdatePreValues(int dataTypeId, IDictionary<string, PreValue> values)
@@ -441,40 +427,28 @@ AND umbracoNode.id <> @id",

                sortOrder++;
            }
-
        }

-        private string GetPrefixedCacheKey(int dataTypeId)
+        private static string GetPrefixedCacheKey(int dataTypeId)
        {
-            return CacheKeys.DataTypePreValuesCacheKey + dataTypeId + "-";
+            return CacheKeys.DataTypePreValuesCacheKey + "_" + dataTypeId;
        }

-        private PreValueCollection GetAndCachePreValueCollection(int dataTypeId)
+        private PreValueCollection GetCachedPreValueCollection(int datetypeId)
        {
-            //go get the data
-            var dtos = Database.Fetch<DataTypePreValueDto>("WHERE datatypeNodeId = @Id", new { Id = dataTypeId });
-            var list = dtos.Select(x => new Tuple<PreValue, string, int>(new PreValue(x.Id, x.Value, x.SortOrder), x.Alias, x.SortOrder)).ToList();
-            var collection = PreValueConverter.ConvertToPreValuesCollection(list);
-
-            //now create the cache key, this needs to include all pre-value ids so that we can use this cached item in the GetPreValuesAsString method
-            //the key will be: "UmbracoPreValDATATYPEID-CSVOFPREVALIDS
-
-            var key = GetPrefixedCacheKey(dataTypeId)
-                      + string.Join(",", collection.FormatAsDictionary().Select(x => x.Value.Id).ToArray());
-
-            //store into cache
-            RuntimeCache.InsertCacheItem(key, () => collection,
-                //30 mins
-                new TimeSpan(0, 0, 30),
-                //sliding is true
-                true);
-
-            return collection;
+            var key = GetPrefixedCacheKey(datetypeId);
+            return RuntimeCache.GetCacheItem<PreValueCollection>(key, () =>
+            {
+                var dtos = Database.Fetch<DataTypePreValueDto>("WHERE datatypeNodeId = @Id", new { Id = datetypeId });
+                var list = dtos.Select(x => new Tuple<PreValue, string, int>(new PreValue(x.Id, x.Value, x.SortOrder), x.Alias, x.SortOrder)).ToList();
+                var collection = PreValueConverter.ConvertToPreValuesCollection(list);
+                return collection;
+            }, TimeSpan.FromMinutes(20), isSliding: true);
        }

        private string EnsureUniqueNodeName(string nodeName, int id = 0)
        {
-         
+

            var sql = new Sql();
            sql.Select("*")
@@ -576,7 +550,7 @@ AND umbracoNode.id <> @id",
                }

                //NOTE: We used to check that the Alias was unique for the given DataTypeNodeId prevalues list, BUT
-                // in reality there is no need to check the uniqueness of this alias because the only way that this code executes is 
+                // in reality there is no need to check the uniqueness of this alias because the only way that this code executes is
                // based on an IDictionary<string, PreValue> dictionary being passed to this repository and a dictionary
                // must have unique aliases by definition, so there is no need for this additional check

@@ -596,10 +570,10 @@ AND umbracoNode.id <> @id",
                {
                    throw new InvalidOperationException("Cannot update a pre value for a data type that has no identity");
                }
-                
+
                //NOTE: We used to check that the Alias was unique for the given DataTypeNodeId prevalues list, BUT
                // this causes issues when sorting the pre-values (http://issues.umbraco.org/issue/U4-5670) but in reality
-                // there is no need to check the uniqueness of this alias because the only way that this code executes is 
+                // there is no need to check the uniqueness of this alias because the only way that this code executes is
                // based on an IDictionary<string, PreValue> dictionary being passed to this repository and a dictionary
                // must have unique aliases by definition, so there is no need for this additional check

@@ -614,7 +588,7 @@ AND umbracoNode.id <> @id",
                Database.Update(dto);
            }

-            
+
        }

        internal static class PreValueConverter
@@ -62,17 +62,27 @@ namespace Umbraco.Core.Persistence.Repositories

        protected override IEnumerable<EntityContainer> PerformGetAll(params int[] ids)
        {
-            //we need to batch these in groups of 2000 so we don't exceed the max 2100 limit
-            return ids.InGroupsOf(2000).SelectMany(@group =>
+            if (ids.Any())
+            {
+                //we need to batch these in groups of 2000 so we don't exceed the max 2100 limit
+                return ids.InGroupsOf(2000).SelectMany(@group =>
+                {
+                    var sql = GetBaseQuery(false)
+                        .Where("nodeObjectType=@umbracoObjectTypeId", new {umbracoObjectTypeId = NodeObjectTypeId})
+                        .Where(string.Format("{0} IN (@ids)", SqlSyntax.GetQuotedColumnName("id")), new {ids = @group});
+
+                    sql.OrderBy<NodeDto>(x => x.Level, SqlSyntax);
+
+                    return Database.Fetch<NodeDto>(sql).Select(CreateEntity);
+                });
+            }
+            else
            {
                var sql = GetBaseQuery(false)
-                    .Where("nodeObjectType=@umbracoObjectTypeId", new { umbracoObjectTypeId = NodeObjectTypeId })
-                    .Where(string.Format("{0} IN (@ids)", SqlSyntax.GetQuotedColumnName("id")), new { ids = @group });
-
+                    .Where("nodeObjectType=@umbracoObjectTypeId", new {umbracoObjectTypeId = NodeObjectTypeId});
                sql.OrderBy<NodeDto>(x => x.Level, SqlSyntax);
-
                return Database.Fetch<NodeDto>(sql).Select(CreateEntity);
-            });
+            }
        }

        protected override IEnumerable<EntityContainer> PerformGetByQuery(IQuery<EntityContainer> query)
@@ -1,18 +1,13 @@
 using System;
 using System.Collections.Generic;
-using System.Dynamic;
-using System.Globalization;
+using System.Collections.ObjectModel;
 using System.Linq;
-using System.Reflection;
-using System.Text;
 using Umbraco.Core.Models;
-using Umbraco.Core;
 using Umbraco.Core.Models.EntityBase;
 using Umbraco.Core.Models.Rdbms;
 using Umbraco.Core.Persistence.Factories;
 using Umbraco.Core.Persistence.Querying;
 using Umbraco.Core.Persistence.UnitOfWork;
-using Umbraco.Core.Strings;

 namespace Umbraco.Core.Persistence.Repositories
 {
@@ -68,11 +63,12 @@ namespace Umbraco.Core.Persistence.Repositories
            bool isMedia = objectTypeId == new Guid(Constants.ObjectTypes.Media);

            var sql = GetFullSqlForEntityType(key, isContent, isMedia, objectTypeId);
-            
+
+            var factory = new UmbracoEntityFactory();
+
            if (isMedia)
            {
-                //for now treat media differently
-                //TODO: We should really use this methodology for Content/Members too!! since it includes properties and ALL of the dynamic db fields
+                //for now treat media differently and include all property data too  
                var entities = _work.Database.Fetch<dynamic, UmbracoPropertyDto, UmbracoEntity>(
                    new UmbracoEntityRelator().Map, sql);

@@ -80,14 +76,16 @@ namespace Umbraco.Core.Persistence.Repositories
            }
            else
            {
-                var nodeDto = _work.Database.FirstOrDefault<dynamic>(sql);
-                if (nodeDto == null)
-                    return null;

-                var factory = new UmbracoEntityFactory();
-                var entity = factory.BuildEntityFromDynamic(nodeDto);
-
-                return entity;
+                //query = read forward data reader, do not load everything into mem
+                var dtos = _work.Database.Query<dynamic>(sql);
+                var collection = new EntityDefinitionCollection();
+                foreach (var dto in dtos)
+                {
+                    collection.AddOrUpdate(new EntityDefinition(factory, dto, isContent, false));
+                }
+                var found = collection.FirstOrDefault();
+                return found != null ? found.BuildFromDynamic() : null;                
            }
            
            
@@ -112,29 +110,29 @@ namespace Umbraco.Core.Persistence.Repositories
            bool isMedia = objectTypeId == new Guid(Constants.ObjectTypes.Media);

            var sql = GetFullSqlForEntityType(id, isContent, isMedia, objectTypeId);
-            
+
+            var factory = new UmbracoEntityFactory();
+
            if (isMedia)
            {
-                //for now treat media differently
-                //TODO: We should really use this methodology for Content/Members too!! since it includes properties and ALL of the dynamic db fields
+                //for now treat media differently and include all property data too  
                var entities = _work.Database.Fetch<dynamic, UmbracoPropertyDto, UmbracoEntity>(
                    new UmbracoEntityRelator().Map, sql);

                return entities.FirstOrDefault();
            }
            else
-            {
-                var nodeDto = _work.Database.FirstOrDefault<dynamic>(sql);
-                if (nodeDto == null)
-                    return null;
-
-                var factory = new UmbracoEntityFactory();
-                var entity = factory.BuildEntityFromDynamic(nodeDto);
-
-                return entity;
+            {                
+                //query = read forward data reader, do not load everything into mem
+                var dtos = _work.Database.Query<dynamic>(sql);
+                var collection = new EntityDefinitionCollection();
+                foreach (var dto in dtos)
+                {
+                    collection.AddOrUpdate(new EntityDefinition(factory, dto, isContent, false));
+                }
+                var found = collection.FirstOrDefault();
+                return found != null ? found.BuildFromDynamic() : null;
            }
-
-            
        }

        public virtual IEnumerable<IUmbracoEntity> GetAll(Guid objectTypeId, params int[] ids)
@@ -171,22 +169,21 @@ namespace Umbraco.Core.Persistence.Repositories

            if (isMedia)
            {
-                //for now treat media differently
-                //TODO: We should really use this methodology for Content/Members too!! since it includes properties and ALL of the dynamic db fields
+                //for now treat media differently and include all property data too                
                var entities = _work.Database.Fetch<dynamic, UmbracoPropertyDto, UmbracoEntity>(
                    new UmbracoEntityRelator().Map, sql);
-                foreach (var entity in entities)
-                {
-                    yield return entity;
-                }
+                return entities;
            }
            else
            {
-                var dtos = _work.Database.Fetch<dynamic>(sql);
-                foreach (var entity in dtos.Select(dto => factory.BuildEntityFromDynamic(dto)))
+                //query = read forward data reader, do not load everything into mem
+                var dtos = _work.Database.Query<dynamic>(sql);
+                var collection = new EntityDefinitionCollection();
+                foreach (var dto in dtos)
                {
-                    yield return entity;
+                    collection.AddOrUpdate(new EntityDefinition(factory, dto, isContent, false));
                }
+                return collection.Select(x => x.BuildFromDynamic()).ToList();                
            }
        }

@@ -231,8 +228,7 @@ namespace Umbraco.Core.Persistence.Repositories
                    }
                });

-                //treat media differently for now 
-                //TODO: We should really use this methodology for Content/Members too!! since it includes properties and ALL of the dynamic db fields
+                //for now treat media differently and include all property data too    
                var entities = _work.Database.Fetch<dynamic, UmbracoPropertyDto, UmbracoEntity>(
                    new UmbracoEntityRelator().Map, mediaSql);
                return entities;
@@ -241,8 +237,15 @@ namespace Umbraco.Core.Persistence.Repositories
            {
                //use dynamic so that we can get ALL properties from the SQL so we can chuck that data into our AdditionalData
                var finalSql = entitySql.Append(GetGroupBy(isContent, false));
-                var dtos = _work.Database.Fetch<dynamic>(finalSql);
-                return dtos.Select(factory.BuildEntityFromDynamic).Cast<IUmbracoEntity>().ToList();
+
+                //query = read forward data reader, do not load everything into mem
+                var dtos = _work.Database.Query<dynamic>(finalSql);
+                var collection = new EntityDefinitionCollection();
+                foreach (var dto in dtos)
+                {
+                    collection.AddOrUpdate(new EntityDefinition(factory, dto, isContent, false));
+                }
+                return collection.Select(x => x.BuildFromDynamic()).ToList();
            }
        }

@@ -315,25 +318,31 @@ namespace Umbraco.Core.Persistence.Repositories
        protected virtual Sql GetBase(bool isContent, bool isMedia, Action<Sql> customFilter)
        {
            var columns = new List<object>
-                              {
-                                  "umbracoNode.id",
-                                  "umbracoNode.trashed",
-                                  "umbracoNode.parentID",
-                                  "umbracoNode.nodeUser",
-                                  "umbracoNode.level",
-                                  "umbracoNode.path",
-                                  "umbracoNode.sortOrder",
-                                  "umbracoNode.uniqueID",
-                                  "umbracoNode.text",
-                                  "umbracoNode.nodeObjectType",
-                                  "umbracoNode.createDate",
-                                  "COUNT(parent.parentID) as children"
-                              };
+            {
+                "umbracoNode.id",
+                "umbracoNode.trashed",
+                "umbracoNode.parentID",
+                "umbracoNode.nodeUser",
+                "umbracoNode.level",
+                "umbracoNode.path",
+                "umbracoNode.sortOrder",
+                "umbracoNode.uniqueID",
+                "umbracoNode.text",
+                "umbracoNode.nodeObjectType",
+                "umbracoNode.createDate",
+                "COUNT(parent.parentID) as children"
+            };

            if (isContent || isMedia)
            {
-                columns.Add("published.versionId as publishedVersion");
-                columns.Add("latest.versionId as newestVersion");
+                if (isContent)
+                {
+                    //only content has/needs this info
+                    columns.Add("published.versionId as publishedVersion");
+                    columns.Add("document.versionId as newestVersion");
+                    columns.Add("contentversion.id as versionId");
+                }
+                
                columns.Add("contenttype.alias");
                columns.Add("contenttype.icon");
                columns.Add("contenttype.thumbnail");
@@ -345,17 +354,22 @@ namespace Umbraco.Core.Persistence.Repositories
            var entitySql = new Sql()
                .Select(columns.ToArray())
                .From("umbracoNode umbracoNode");
-                
+
            if (isContent || isMedia)
            {
-                entitySql.InnerJoin("cmsContent content").On("content.nodeId = umbracoNode.id")
-                   .LeftJoin("cmsContentType contenttype").On("contenttype.nodeId = content.contentType")
-                   .LeftJoin(
-                       "(SELECT nodeId, versionId FROM cmsDocument WHERE published = 1 GROUP BY nodeId, versionId) as published")
-                   .On("umbracoNode.id = published.nodeId")
-                   .LeftJoin(
-                       "(SELECT nodeId, versionId FROM cmsDocument WHERE newest = 1 GROUP BY nodeId, versionId) as latest")
-                   .On("umbracoNode.id = latest.nodeId");
+                entitySql.InnerJoin("cmsContent content").On("content.nodeId = umbracoNode.id");
+
+                if (isContent)
+                {
+                    //only content has/needs this info
+                    entitySql                        
+                        .InnerJoin("cmsDocument document").On("document.nodeId = umbracoNode.id")
+                        .InnerJoin("cmsContentVersion contentversion").On("contentversion.VersionId = document.versionId")
+                        .LeftJoin("(SELECT nodeId, versionId FROM cmsDocument WHERE published = 1) as published")
+                        .On("umbracoNode.id = published.nodeId");
+                }
+
+                entitySql.LeftJoin("cmsContentType contenttype").On("contenttype.nodeId = content.contentType");
            }

            entitySql.LeftJoin("umbracoNode parent").On("parent.parentID = umbracoNode.id");
@@ -372,22 +386,42 @@ namespace Umbraco.Core.Persistence.Repositories
        {
            var sql = baseQuery(isContent, isMedia, filter)
                .Where("umbracoNode.nodeObjectType = @NodeObjectType", new { NodeObjectType = nodeObjectType });
+
+            if (isContent)
+            {
+                sql.Where("document.newest = 1");
+            }
+
            return sql;
        }

        protected virtual Sql GetBaseWhere(Func<bool, bool, Action<Sql>, Sql> baseQuery, bool isContent, bool isMedia, int id)
        {
            var sql = baseQuery(isContent, isMedia, null)
-                .Where("umbracoNode.id = @Id", new { Id = id })
-                .Append(GetGroupBy(isContent, isMedia));
+                .Where("umbracoNode.id = @Id", new { Id = id });
+
+            if (isContent)
+            {
+                sql.Where("document.newest = 1");
+            }
+
+            sql.Append(GetGroupBy(isContent, isMedia));
+
            return sql;
        }

        protected virtual Sql GetBaseWhere(Func<bool, bool, Action<Sql>, Sql> baseQuery, bool isContent, bool isMedia, Guid key)
        {
            var sql = baseQuery(isContent, isMedia, null)
-                .Where("umbracoNode.uniqueID = @UniqueID", new { UniqueID = key })
-                .Append(GetGroupBy(isContent, isMedia));
+                .Where("umbracoNode.uniqueID = @UniqueID", new {UniqueID = key});
+
+            if (isContent)
+            {
+                sql.Where("document.newest = 1");
+            }
+
+            sql.Append(GetGroupBy(isContent, isMedia));
+
            return sql;
        }

@@ -396,6 +430,12 @@ namespace Umbraco.Core.Persistence.Repositories
            var sql = baseQuery(isContent, isMedia, null)
                .Where("umbracoNode.id = @Id AND umbracoNode.nodeObjectType = @NodeObjectType",
                       new {Id = id, NodeObjectType = nodeObjectType});
+
+            if (isContent)
+            {
+                sql.Where("document.newest = 1");
+            }
+
            return sql;
        }

@@ -404,36 +444,46 @@ namespace Umbraco.Core.Persistence.Repositories
            var sql = baseQuery(isContent, isMedia, null)
                .Where("umbracoNode.uniqueID = @UniqueID AND umbracoNode.nodeObjectType = @NodeObjectType",
                       new { UniqueID = key, NodeObjectType = nodeObjectType });
+
+            if (isContent)
+            {
+                sql.Where("document.newest = 1");
+            }
+
            return sql;
        }

        protected virtual Sql GetGroupBy(bool isContent, bool isMedia, bool includeSort = true)
        {
            var columns = new List<object>
-                              {
-                                  "umbracoNode.id",
-                                  "umbracoNode.trashed",
-                                  "umbracoNode.parentID",
-                                  "umbracoNode.nodeUser",
-                                  "umbracoNode.level",
-                                  "umbracoNode.path",
-                                  "umbracoNode.sortOrder",
-                                  "umbracoNode.uniqueID",
-                                  "umbracoNode.text",
-                                  "umbracoNode.nodeObjectType",
-                                  "umbracoNode.createDate"
-                              };
+            {
+                "umbracoNode.id",
+                "umbracoNode.trashed",
+                "umbracoNode.parentID",
+                "umbracoNode.nodeUser",
+                "umbracoNode.level",
+                "umbracoNode.path",
+                "umbracoNode.sortOrder",
+                "umbracoNode.uniqueID",
+                "umbracoNode.text",
+                "umbracoNode.nodeObjectType",
+                "umbracoNode.createDate"
+            };

            if (isContent || isMedia)
            {
-                columns.Add("published.versionId");
-                columns.Add("latest.versionId");
+                if (isContent)
+                {
+                    columns.Add("published.versionId");
+                    columns.Add("document.versionId");
+                    columns.Add("contentversion.id");
+                }
                columns.Add("contenttype.alias");
                columns.Add("contenttype.icon");
                columns.Add("contenttype.thumbnail");
-                columns.Add("contenttype.isContainer");
+                columns.Add("contenttype.isContainer");                
            }
-            
+
            var sql = new Sql()
                .GroupBy(columns.ToArray());

@@ -444,7 +494,7 @@ namespace Umbraco.Core.Persistence.Repositories

            return sql;
        }
-        
+
        #endregion

        /// <summary>
@@ -458,33 +508,7 @@ namespace Umbraco.Core.Persistence.Repositories
            UnitOfWork.DisposeIfDisposable();
        }

-        #region umbracoNode POCO - Extends NodeDto
-        [TableName("umbracoNode")]
-        [PrimaryKey("id")]
-        [ExplicitColumns]
-        internal class UmbracoEntityDto : NodeDto
-        {
-            [Column("children")]
-            public int Children { get; set; }
-
-            [Column("publishedVersion")]
-            public Guid PublishedVersion { get; set; }
-
-            [Column("newestVersion")]
-            public Guid NewestVersion { get; set; }
-
-            [Column("alias")]
-            public string Alias { get; set; }
-
-            [Column("icon")]
-            public string Icon { get; set; }
-
-            [Column("thumbnail")]
-            public string Thumbnail { get; set; }
-
-            [ResultColumn]
-            public List<UmbracoPropertyDto> UmbracoPropertyDtos { get; set; }
-        }
+        #region private classes
        
        [ExplicitColumns]
        internal class UmbracoPropertyDto
@@ -568,6 +592,99 @@ namespace Umbraco.Core.Persistence.Repositories
                return prev;
            }
        }
+
+        private class EntityDefinitionCollection : KeyedCollection<int, EntityDefinition>
+        {
+            protected override int GetKeyForItem(EntityDefinition item)
+            {
+                return item.Id;
+            }
+
+            /// <summary>
+            /// if this key already exists if it does then we need to check
+            /// if the existing item is 'older' than the new item and if that is the case we'll replace the older one
+            /// </summary>
+            /// <param name="item"></param>
+            /// <returns></returns>
+            public bool AddOrUpdate(EntityDefinition item)
+            {
+                if (Dictionary == null)
+                {
+                    base.Add(item);
+                    return true;
+                }
+
+                var key = GetKeyForItem(item);
+                EntityDefinition found;
+                if (TryGetValue(key, out found))
+                {
+                    //it already exists and it's older so we need to replace it
+                    if (item.VersionId > found.VersionId)
+                    {
+                        var currIndex = Items.IndexOf(found);
+                        if (currIndex == -1)
+                            throw new IndexOutOfRangeException("Could not find the item in the list: " + found.Id);
+
+                        //replace the current one with the newer one
+                        SetItem(currIndex, item);
+                        return true;
+                    }
+                    //could not add or update
+                    return false;
+                }
+
+                base.Add(item);
+                return true;
+            }
+
+            private bool TryGetValue(int key, out EntityDefinition val)
+            {
+                if (Dictionary == null)
+                {
+                    val = null;
+                    return false;
+                }
+                return Dictionary.TryGetValue(key, out val);
+            }
+        }
+
+        private class EntityDefinition
+        {
+            private readonly UmbracoEntityFactory _factory;
+            private readonly dynamic _entity;
+            private readonly bool _isContent;
+            private readonly bool _isMedia;
+
+            public EntityDefinition(UmbracoEntityFactory factory, dynamic entity, bool isContent, bool isMedia)
+            {
+                _factory = factory;
+                _entity = entity;
+                _isContent = isContent;
+                _isMedia = isMedia;
+            }
+
+            public IUmbracoEntity BuildFromDynamic()
+            {
+                return _factory.BuildEntityFromDynamic(_entity);
+            }
+
+            public int Id
+            {
+                get { return _entity.id; }
+            }
+
+            public int VersionId
+            {
+                get
+                {
+                    if (_isContent || _isMedia)
+                    {
+                        return _entity.versionId;
+                    }
+                    return _entity.id;
+                }
+            }
+        }
        #endregion
    }
 }
@@ -96,5 +96,6 @@ namespace Umbraco.Core.Persistence.Repositories
        /// <returns>An Enumerable list of <see cref="IContent"/> objects</returns>
        IEnumerable<IContent> GetPagedResultsByQuery(IQuery<IContent> query, long pageIndex, int pageSize, out long totalRecords,
            string orderBy, Direction orderDirection, bool orderBySystemField, IQuery<IContent> filter = null);
+        
    }
 }
@@ -8,6 +8,13 @@ namespace Umbraco.Core.Persistence.Repositories
 {
    public interface IContentTypeRepository : IContentTypeCompositionRepository<IContentType>
    {
+        /// <summary>
+        /// Given the path of a content item, this will return true if the content item exists underneath a list view content item
+        /// </summary>
+        /// <param name="contentPath"></param>
+        /// <returns></returns>
+        bool HasContainerInPath(string contentPath);
+
        /// <summary>
        /// Gets all entities of the specified <see cref="PropertyType"/> query
        /// </summary>
@@ -38,5 +38,6 @@ namespace Umbraco.Core.Persistence.Repositories
        /// <returns>An Enumerable list of <see cref="IMedia"/> objects</returns>
        IEnumerable<IMedia> GetPagedResultsByQuery(IQuery<IMedia> query, long pageIndex, int pageSize, out long totalRecords,
            string orderBy, Direction orderDirection, bool orderBySystemField, string filter = "");
+        
    }
 }
@@ -73,6 +73,6 @@ namespace Umbraco.Core.Persistence.Repositories
        /// <param name="content"></param>
        /// <param name="xml"></param>
        void AddOrUpdatePreviewXml(IMember content, Func<IMember, XElement> xml);
-
+        
    }
 }
--- a/Show More
+++ b/Show More
				`@@ -0,0 +1 @@`
				`<processing xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform" xdt:Transform="SetAttributes(preserveExifMetaData)" preserveExifMetaData="true" />`