[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
/*
|
|
|
|
|
* Copyright (C) 1998-2005 ReactOS Team (and the authors from the programmers section)
|
|
|
|
|
*
|
|
|
|
|
* This program is free software; you can redistribute it and/or
|
|
|
|
|
* modify it under the terms of the GNU General Public License
|
|
|
|
|
* as published by the Free Software Foundation; either version 2
|
|
|
|
|
* of the License, or (at your option) any later version.
|
|
|
|
|
*
|
|
|
|
|
* This program is distributed in the hope that it will be useful,
|
|
|
|
|
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
|
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
|
|
|
* GNU General Public License for more details.
|
|
|
|
|
*
|
|
|
|
|
* You should have received a copy of the GNU General Public License
|
|
|
|
|
* along with this program; if not, write to the Free Software
|
|
|
|
|
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
|
|
|
|
*
|
|
|
|
|
*
|
|
|
|
|
* PROJECT: ReactOS kernel
|
2015-10-04 11:54:25 +00:00
|
|
|
* FILE: ntoskrnl/cache/section/data.c
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
* PURPOSE: Implements section objects
|
|
|
|
|
*
|
|
|
|
|
* PROGRAMMERS: Rex Jolliff
|
|
|
|
|
* David Welch
|
|
|
|
|
* Eric Kohl
|
|
|
|
|
* Emanuele Aliberti
|
|
|
|
|
* Eugene Ingerman
|
|
|
|
|
* Casper Hornstrup
|
|
|
|
|
* KJK::Hyperion
|
|
|
|
|
* Guido de Jong
|
|
|
|
|
* Ge van Geldorp
|
|
|
|
|
* Royce Mitchell III
|
|
|
|
|
* Filip Navara
|
|
|
|
|
* Aleksey Bragin
|
|
|
|
|
* Jason Filby
|
|
|
|
|
* Thomas Weidenmueller
|
|
|
|
|
* Gunnar Andre' Dalsnes
|
|
|
|
|
* Mike Nordell
|
|
|
|
|
* Alex Ionescu
|
|
|
|
|
* Gregor Anich
|
|
|
|
|
* Steven Edwards
|
|
|
|
|
* Herve Poussineau
|
|
|
|
|
*/
|
|
|
|
|
|
2012-03-29 04:43:44 +00:00
|
|
|
/*
|
|
|
|
|
|
|
|
|
|
A note on this code:
|
|
|
|
|
|
|
|
|
|
Unlike the previous section code, this code does not rely on an active map
|
|
|
|
|
for a page to exist in a data segment. Each mapping contains a large integer
|
|
|
|
|
offset to map at, and the segment always represents the entire section space
|
|
|
|
|
from zero to the maximum long long. This allows us to associate one single
|
|
|
|
|
page map with each file object, and to let each mapping view an offset into
|
|
|
|
|
the overall mapped file. Temporarily unmapping the file has no effect on the
|
|
|
|
|
section membership.
|
|
|
|
|
|
|
|
|
|
This necessitates a change in the section page table implementation, which is
|
|
|
|
|
now an RtlGenericTable. This will be elaborated more in sptab.c. One upshot
|
|
|
|
|
of this change is that a mapping of a small files takes a bit more than 1/4
|
|
|
|
|
of the size in nonpaged kernel space as it did previously.
|
|
|
|
|
|
|
|
|
|
When we need other threads that may be competing for the same page fault to
|
|
|
|
|
wait, we have a mechanism seperate from PageOps for dealing with that, which
|
|
|
|
|
was suggested by Travis Geiselbrecht after a conversation I had with Alex
|
|
|
|
|
Ionescu. That mechanism is the MM_WAIT_ENTRY, which is the all-ones SWAPENTRY.
|
|
|
|
|
|
|
|
|
|
When we wish for other threads to know that we're waiting and will finish
|
|
|
|
|
handling a page fault, we place the swap entry MM_WAIT_ENTRY in the page table
|
|
|
|
|
at the fault address (this works on either the section page table or a process
|
|
|
|
|
address space), perform any blocking operations required, then replace the
|
2012-03-29 06:04:34 +00:00
|
|
|
entry.
|
2012-03-29 04:43:44 +00:00
|
|
|
|
|
|
|
|
*/
|
|
|
|
|
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
/* INCLUDES *****************************************************************/
|
|
|
|
|
|
|
|
|
|
#include <ntoskrnl.h>
|
|
|
|
|
#include "newmm.h"
|
2014-11-10 16:26:55 +00:00
|
|
|
#include <cache/newcc.h>
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
#define NDEBUG
|
|
|
|
|
#include <debug.h>
|
2014-11-10 16:26:55 +00:00
|
|
|
#include <mm/ARM3/miarm.h>
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
|
|
|
|
|
#define DPRINTC DPRINT
|
|
|
|
|
|
[NEWCC]
A reintegration checkpoint for the NewCC branch, brought to you by Team NewCC.
Differences with current ReactOS trunk:
* A new memory area type, MEMORY_AREA_CACHE, is added, which represents a mapped region of a file. In NEWCC mode, user sections are MEMORY_AREA_CACHE type as well, and obey new semantics. In non-NEWCC mode, they aren't used.
* A way of claiming a page entry for a specific thread's work is added. Placing the special SWAPENTRY value MM_WAIT_ENTRY in a page table, or in a section page table should indicate that memory management code is intended to wait for another thread to make some status change before checking the state of the page entry again. In code that uses this convention, a return value of STATUS_SUCCESS + 1 is used to indicate that the caller should use the MiWaitForPageEvent macro to wait until somebody has change the state of a wait entry before checking again. This is a lighter weight mechanism than PAGEOPs.
* A way of asking the caller to perform some blocking operation without locks held is provided. This replaces some spaghettified code in which locks are repeatedly taken and broken by code that performs various blocking operations. Using this mechanism, it is possible to do a small amount of non-blocking work, fill in a request, then return STATUS_MORE_PROCESSING_REQUIRED to request that locks be dropped and the blocking operation be carried out. A MM_REQUIRED_RESOURCES structure is provided to consumers of this contract to use to accumulate state across many blocking operations. Several functions wrapping blocking operations are provided in ntoskrnl/cache/reqtools.c.
* Image section pages are no longer direct mapped. This is done to simplify consolidation of ownership of pages under the data section system. At a later time, it may be possible to make data pages directly available to image sections for the same file. This is likely the only direct performance impact this code makes on non-NEWCC mode.
RMAPs:
* A new type of RMAP entry is introduced, distinguished by RMAP_IS_SEGMENT(Address) of the rmap entry. This kind of entry contains a pointer to a section page table node in the Process pointer, which in turn links back to the MM_SECTION_SEGMENT it belongs to. Therefore, a page belonging only to a segment (that is, a segment page that isn't mapped) can exist and be evicted using the normal page eviction mechanism in balance.c. Each of the rmap function has been modified to deal with segment rmaps.
* The low 8 bits of the Address field in a segment rmap denote the entry number in the generic table node pointed to by Process that points to the page the rmap belongs to. By combining them, you can determine the file offset the page belongs to.
* In NEWCC mode, MmSharePageEntry/UnsharePageEntry are not used, and instead the page reference count is used to keep track of the number of mappings of a page, allowing the last reference expiring to allow the page to be recycled without much intervention. These are still used in non-NEWCC mode. One change has been made, the count fields have been narrowed by 1 bit to make room for a dirty bit in SSE entries, needed when a page is present but unmapped.
Section page tables:
* The section page tables are now implemented using RtlGenericTables. This enables a fairly compact representation of section page tables without having the existence of a section object imply 4k of fake PDEs. In addition, each node in the generic table has a wide file offset that is a multiple of 256 pages, or 1 megabyte total. Besides needing wide file offsets, the only other visible change caused by the switch to generic tables for section page tables is the need to lock the section segment before interacting with the section page table.
Eviction:
* Page eviction in cache sections is accomplished by MmpPageOutPhysicalAddress. In the case of a shared page, it tries to remove all mappings of the indicated page. If this process fails at any point, the page will simply be drawn back into the target address spaces. After succeeding at this, if TRUE has been accumulated into the page's dirty bit in the section page table, it is written back, and then permanently removed.
NewCC mode:
* NEWCC mode is introduced, which rewrites the file cache to a set of cache stripes actively mapped, along with unmapped section data.
* NewCC is more authentic in its interpretation of the external interface to the windows cache than the current cache manager, implementing each of the cache manager functions according to the documented interface with no preconceived ideas about how anything should be implemented internally. Cache stripes are implemented on top of section objects, using the same memory manager paths, and therefore economizing code and complexity. This replaces a rather complicated system in which pages can be owned by the cache manager and the memory manager simultaneously and they must cooperate in a fairly sophisticated way to manage them. Since they're quite interdependent in the current code, modifying either is very difficult. In NEWCC, they have a clear division of labor and thus can be worked on independently.
* Several third party filesystems that use the kernel Cc interface work properly using NEWCC, including matt wu's ext3 driver.
* In contrast with code that tries to make CcInitializeCacheMap and CcUninitializeCacheMap into a pair that supports reference counting, NEWCC lazily initializes the shared and private cache maps as needed and uses the presence of a PrivateCacheMap on at least one file pointing to the SharedCacheMap as an indication that the FILE_OBJECT reference in the SharedCacheMap should still be held. When the last PrivateCacheMap is discarded, that's the appropriate time to tear down caching for a specific file, as the SharedCacheMap data is allowed to be saved and reused. We honor this by making the SharedCacheMap into a depot for keeping track of the PrivateCacheMap objects associated with views of a file.
svn path=/trunk/; revision=55833
2012-02-23 12:03:06 +00:00
|
|
|
LIST_ENTRY MiSegmentList;
|
|
|
|
|
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
extern KEVENT MpwThreadEvent;
|
|
|
|
|
extern KSPIN_LOCK MiSectionPageTableLock;
|
2012-02-24 10:05:06 +00:00
|
|
|
extern PMMWSL MmWorkingSetList;
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
|
|
|
|
|
/* FUNCTIONS *****************************************************************/
|
|
|
|
|
|
|
|
|
|
/* Note: Mmsp prefix denotes "Memory Manager Section Private". */
|
|
|
|
|
|
|
|
|
|
VOID
|
|
|
|
|
NTAPI
|
[NEWCC]
A reintegration checkpoint for the NewCC branch, brought to you by Team NewCC.
Differences with current ReactOS trunk:
* A new memory area type, MEMORY_AREA_CACHE, is added, which represents a mapped region of a file. In NEWCC mode, user sections are MEMORY_AREA_CACHE type as well, and obey new semantics. In non-NEWCC mode, they aren't used.
* A way of claiming a page entry for a specific thread's work is added. Placing the special SWAPENTRY value MM_WAIT_ENTRY in a page table, or in a section page table should indicate that memory management code is intended to wait for another thread to make some status change before checking the state of the page entry again. In code that uses this convention, a return value of STATUS_SUCCESS + 1 is used to indicate that the caller should use the MiWaitForPageEvent macro to wait until somebody has change the state of a wait entry before checking again. This is a lighter weight mechanism than PAGEOPs.
* A way of asking the caller to perform some blocking operation without locks held is provided. This replaces some spaghettified code in which locks are repeatedly taken and broken by code that performs various blocking operations. Using this mechanism, it is possible to do a small amount of non-blocking work, fill in a request, then return STATUS_MORE_PROCESSING_REQUIRED to request that locks be dropped and the blocking operation be carried out. A MM_REQUIRED_RESOURCES structure is provided to consumers of this contract to use to accumulate state across many blocking operations. Several functions wrapping blocking operations are provided in ntoskrnl/cache/reqtools.c.
* Image section pages are no longer direct mapped. This is done to simplify consolidation of ownership of pages under the data section system. At a later time, it may be possible to make data pages directly available to image sections for the same file. This is likely the only direct performance impact this code makes on non-NEWCC mode.
RMAPs:
* A new type of RMAP entry is introduced, distinguished by RMAP_IS_SEGMENT(Address) of the rmap entry. This kind of entry contains a pointer to a section page table node in the Process pointer, which in turn links back to the MM_SECTION_SEGMENT it belongs to. Therefore, a page belonging only to a segment (that is, a segment page that isn't mapped) can exist and be evicted using the normal page eviction mechanism in balance.c. Each of the rmap function has been modified to deal with segment rmaps.
* The low 8 bits of the Address field in a segment rmap denote the entry number in the generic table node pointed to by Process that points to the page the rmap belongs to. By combining them, you can determine the file offset the page belongs to.
* In NEWCC mode, MmSharePageEntry/UnsharePageEntry are not used, and instead the page reference count is used to keep track of the number of mappings of a page, allowing the last reference expiring to allow the page to be recycled without much intervention. These are still used in non-NEWCC mode. One change has been made, the count fields have been narrowed by 1 bit to make room for a dirty bit in SSE entries, needed when a page is present but unmapped.
Section page tables:
* The section page tables are now implemented using RtlGenericTables. This enables a fairly compact representation of section page tables without having the existence of a section object imply 4k of fake PDEs. In addition, each node in the generic table has a wide file offset that is a multiple of 256 pages, or 1 megabyte total. Besides needing wide file offsets, the only other visible change caused by the switch to generic tables for section page tables is the need to lock the section segment before interacting with the section page table.
Eviction:
* Page eviction in cache sections is accomplished by MmpPageOutPhysicalAddress. In the case of a shared page, it tries to remove all mappings of the indicated page. If this process fails at any point, the page will simply be drawn back into the target address spaces. After succeeding at this, if TRUE has been accumulated into the page's dirty bit in the section page table, it is written back, and then permanently removed.
NewCC mode:
* NEWCC mode is introduced, which rewrites the file cache to a set of cache stripes actively mapped, along with unmapped section data.
* NewCC is more authentic in its interpretation of the external interface to the windows cache than the current cache manager, implementing each of the cache manager functions according to the documented interface with no preconceived ideas about how anything should be implemented internally. Cache stripes are implemented on top of section objects, using the same memory manager paths, and therefore economizing code and complexity. This replaces a rather complicated system in which pages can be owned by the cache manager and the memory manager simultaneously and they must cooperate in a fairly sophisticated way to manage them. Since they're quite interdependent in the current code, modifying either is very difficult. In NEWCC, they have a clear division of labor and thus can be worked on independently.
* Several third party filesystems that use the kernel Cc interface work properly using NEWCC, including matt wu's ext3 driver.
* In contrast with code that tries to make CcInitializeCacheMap and CcUninitializeCacheMap into a pair that supports reference counting, NEWCC lazily initializes the shared and private cache maps as needed and uses the presence of a PrivateCacheMap on at least one file pointing to the SharedCacheMap as an indication that the FILE_OBJECT reference in the SharedCacheMap should still be held. When the last PrivateCacheMap is discarded, that's the appropriate time to tear down caching for a specific file, as the SharedCacheMap data is allowed to be saved and reused. We honor this by making the SharedCacheMap into a depot for keeping track of the PrivateCacheMap objects associated with views of a file.
svn path=/trunk/; revision=55833
2012-02-23 12:03:06 +00:00
|
|
|
_MmLockSectionSegment(PMM_SECTION_SEGMENT Segment, const char *file, int line)
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
{
|
2012-03-28 18:39:21 +00:00
|
|
|
//DPRINT("MmLockSectionSegment(%p,%s:%d)\n", Segment, file, line);
|
|
|
|
|
ExAcquireFastMutex(&Segment->Lock);
|
|
|
|
|
Segment->Locked = TRUE;
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
VOID
|
|
|
|
|
NTAPI
|
[NEWCC]
A reintegration checkpoint for the NewCC branch, brought to you by Team NewCC.
Differences with current ReactOS trunk:
* A new memory area type, MEMORY_AREA_CACHE, is added, which represents a mapped region of a file. In NEWCC mode, user sections are MEMORY_AREA_CACHE type as well, and obey new semantics. In non-NEWCC mode, they aren't used.
* A way of claiming a page entry for a specific thread's work is added. Placing the special SWAPENTRY value MM_WAIT_ENTRY in a page table, or in a section page table should indicate that memory management code is intended to wait for another thread to make some status change before checking the state of the page entry again. In code that uses this convention, a return value of STATUS_SUCCESS + 1 is used to indicate that the caller should use the MiWaitForPageEvent macro to wait until somebody has change the state of a wait entry before checking again. This is a lighter weight mechanism than PAGEOPs.
* A way of asking the caller to perform some blocking operation without locks held is provided. This replaces some spaghettified code in which locks are repeatedly taken and broken by code that performs various blocking operations. Using this mechanism, it is possible to do a small amount of non-blocking work, fill in a request, then return STATUS_MORE_PROCESSING_REQUIRED to request that locks be dropped and the blocking operation be carried out. A MM_REQUIRED_RESOURCES structure is provided to consumers of this contract to use to accumulate state across many blocking operations. Several functions wrapping blocking operations are provided in ntoskrnl/cache/reqtools.c.
* Image section pages are no longer direct mapped. This is done to simplify consolidation of ownership of pages under the data section system. At a later time, it may be possible to make data pages directly available to image sections for the same file. This is likely the only direct performance impact this code makes on non-NEWCC mode.
RMAPs:
* A new type of RMAP entry is introduced, distinguished by RMAP_IS_SEGMENT(Address) of the rmap entry. This kind of entry contains a pointer to a section page table node in the Process pointer, which in turn links back to the MM_SECTION_SEGMENT it belongs to. Therefore, a page belonging only to a segment (that is, a segment page that isn't mapped) can exist and be evicted using the normal page eviction mechanism in balance.c. Each of the rmap function has been modified to deal with segment rmaps.
* The low 8 bits of the Address field in a segment rmap denote the entry number in the generic table node pointed to by Process that points to the page the rmap belongs to. By combining them, you can determine the file offset the page belongs to.
* In NEWCC mode, MmSharePageEntry/UnsharePageEntry are not used, and instead the page reference count is used to keep track of the number of mappings of a page, allowing the last reference expiring to allow the page to be recycled without much intervention. These are still used in non-NEWCC mode. One change has been made, the count fields have been narrowed by 1 bit to make room for a dirty bit in SSE entries, needed when a page is present but unmapped.
Section page tables:
* The section page tables are now implemented using RtlGenericTables. This enables a fairly compact representation of section page tables without having the existence of a section object imply 4k of fake PDEs. In addition, each node in the generic table has a wide file offset that is a multiple of 256 pages, or 1 megabyte total. Besides needing wide file offsets, the only other visible change caused by the switch to generic tables for section page tables is the need to lock the section segment before interacting with the section page table.
Eviction:
* Page eviction in cache sections is accomplished by MmpPageOutPhysicalAddress. In the case of a shared page, it tries to remove all mappings of the indicated page. If this process fails at any point, the page will simply be drawn back into the target address spaces. After succeeding at this, if TRUE has been accumulated into the page's dirty bit in the section page table, it is written back, and then permanently removed.
NewCC mode:
* NEWCC mode is introduced, which rewrites the file cache to a set of cache stripes actively mapped, along with unmapped section data.
* NewCC is more authentic in its interpretation of the external interface to the windows cache than the current cache manager, implementing each of the cache manager functions according to the documented interface with no preconceived ideas about how anything should be implemented internally. Cache stripes are implemented on top of section objects, using the same memory manager paths, and therefore economizing code and complexity. This replaces a rather complicated system in which pages can be owned by the cache manager and the memory manager simultaneously and they must cooperate in a fairly sophisticated way to manage them. Since they're quite interdependent in the current code, modifying either is very difficult. In NEWCC, they have a clear division of labor and thus can be worked on independently.
* Several third party filesystems that use the kernel Cc interface work properly using NEWCC, including matt wu's ext3 driver.
* In contrast with code that tries to make CcInitializeCacheMap and CcUninitializeCacheMap into a pair that supports reference counting, NEWCC lazily initializes the shared and private cache maps as needed and uses the presence of a PrivateCacheMap on at least one file pointing to the SharedCacheMap as an indication that the FILE_OBJECT reference in the SharedCacheMap should still be held. When the last PrivateCacheMap is discarded, that's the appropriate time to tear down caching for a specific file, as the SharedCacheMap data is allowed to be saved and reused. We honor this by making the SharedCacheMap into a depot for keeping track of the PrivateCacheMap objects associated with views of a file.
svn path=/trunk/; revision=55833
2012-02-23 12:03:06 +00:00
|
|
|
_MmUnlockSectionSegment(PMM_SECTION_SEGMENT Segment, const char *file, int line)
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
{
|
2012-03-28 18:39:21 +00:00
|
|
|
ASSERT(Segment->Locked);
|
|
|
|
|
Segment->Locked = FALSE;
|
|
|
|
|
ExReleaseFastMutex(&Segment->Lock);
|
|
|
|
|
//DPRINT("MmUnlockSectionSegment(%p,%s:%d)\n", Segment, file, line);
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
}
|
|
|
|
|
|
2020-10-23 14:16:51 +00:00
|
|
|
#ifdef NEWCC
|
2012-03-29 04:43:44 +00:00
|
|
|
/*
|
|
|
|
|
|
|
|
|
|
MiFlushMappedSection
|
|
|
|
|
|
|
|
|
|
Called from cache code to cause dirty pages of a section
|
|
|
|
|
to be written back. This doesn't affect the mapping.
|
|
|
|
|
|
|
|
|
|
BaseOffset is the base at which to start writing in file space.
|
|
|
|
|
FileSize is the length of the file as understood by the cache.
|
|
|
|
|
|
|
|
|
|
*/
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
NTSTATUS
|
|
|
|
|
NTAPI
|
2012-03-28 18:39:21 +00:00
|
|
|
_MiFlushMappedSection(PVOID BaseAddress,
|
|
|
|
|
PLARGE_INTEGER BaseOffset,
|
|
|
|
|
PLARGE_INTEGER FileSize,
|
|
|
|
|
BOOLEAN WriteData,
|
|
|
|
|
const char *File,
|
|
|
|
|
int Line)
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
{
|
2012-03-28 18:39:21 +00:00
|
|
|
NTSTATUS Status = STATUS_SUCCESS;
|
|
|
|
|
ULONG_PTR PageAddress;
|
|
|
|
|
PMMSUPPORT AddressSpace = MmGetKernelAddressSpace();
|
|
|
|
|
PMEMORY_AREA MemoryArea;
|
|
|
|
|
PMM_SECTION_SEGMENT Segment;
|
|
|
|
|
ULONG_PTR BeginningAddress, EndingAddress;
|
|
|
|
|
LARGE_INTEGER ViewOffset;
|
|
|
|
|
LARGE_INTEGER FileOffset;
|
|
|
|
|
PFN_NUMBER Page;
|
|
|
|
|
PPFN_NUMBER Pages;
|
|
|
|
|
KIRQL OldIrql;
|
|
|
|
|
|
2013-02-16 17:37:17 +00:00
|
|
|
DPRINT("MiFlushMappedSection(%p,%I64x,%I64x,%u,%s:%d)\n",
|
2012-03-28 18:39:21 +00:00
|
|
|
BaseAddress,
|
2013-02-16 17:37:17 +00:00
|
|
|
BaseOffset->QuadPart,
|
|
|
|
|
FileSize ? FileSize->QuadPart : 0,
|
2012-03-28 18:39:21 +00:00
|
|
|
WriteData,
|
|
|
|
|
File,
|
|
|
|
|
Line);
|
|
|
|
|
|
|
|
|
|
MmLockAddressSpace(AddressSpace);
|
|
|
|
|
MemoryArea = MmLocateMemoryAreaByAddress(AddressSpace, BaseAddress);
|
|
|
|
|
if (!MemoryArea || MemoryArea->Type != MEMORY_AREA_CACHE || MemoryArea->DeleteInProgress)
|
|
|
|
|
{
|
|
|
|
|
MmUnlockAddressSpace(AddressSpace);
|
|
|
|
|
DPRINT("STATUS_NOT_MAPPED_DATA\n");
|
|
|
|
|
return STATUS_NOT_MAPPED_DATA;
|
|
|
|
|
}
|
2015-05-16 20:10:03 +00:00
|
|
|
BeginningAddress = PAGE_ROUND_DOWN(MA_GetStartingAddress(MemoryArea));
|
2015-05-16 20:10:26 +00:00
|
|
|
EndingAddress = PAGE_ROUND_UP(MA_GetEndingAddress(MemoryArea));
|
2012-03-28 18:39:21 +00:00
|
|
|
Segment = MemoryArea->Data.SectionData.Segment;
|
|
|
|
|
ViewOffset.QuadPart = MemoryArea->Data.SectionData.ViewOffset.QuadPart;
|
|
|
|
|
|
|
|
|
|
ASSERT(ViewOffset.QuadPart == BaseOffset->QuadPart);
|
|
|
|
|
|
|
|
|
|
MmLockSectionSegment(Segment);
|
|
|
|
|
|
|
|
|
|
Pages = ExAllocatePool(NonPagedPool,
|
|
|
|
|
sizeof(PFN_NUMBER) * ((EndingAddress - BeginningAddress) >> PAGE_SHIFT));
|
|
|
|
|
|
|
|
|
|
if (!Pages)
|
|
|
|
|
{
|
|
|
|
|
ASSERT(FALSE);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
//DPRINT("Getting pages in range %08x-%08x\n", BeginningAddress, EndingAddress);
|
|
|
|
|
|
|
|
|
|
for (PageAddress = BeginningAddress;
|
|
|
|
|
PageAddress < EndingAddress;
|
|
|
|
|
PageAddress += PAGE_SIZE)
|
|
|
|
|
{
|
2012-03-28 19:41:40 +00:00
|
|
|
ULONG_PTR Entry;
|
2012-03-28 18:39:21 +00:00
|
|
|
FileOffset.QuadPart = ViewOffset.QuadPart + PageAddress - BeginningAddress;
|
|
|
|
|
Entry = MmGetPageEntrySectionSegment(MemoryArea->Data.SectionData.Segment,
|
|
|
|
|
&FileOffset);
|
|
|
|
|
Page = PFN_FROM_SSE(Entry);
|
|
|
|
|
if (Entry != 0 && !IS_SWAP_FROM_SSE(Entry) &&
|
|
|
|
|
(MmIsDirtyPageRmap(Page) || IS_DIRTY_SSE(Entry)) &&
|
|
|
|
|
FileOffset.QuadPart < FileSize->QuadPart)
|
|
|
|
|
{
|
2017-11-21 22:33:42 +00:00
|
|
|
OldIrql = MiAcquirePfnLock();
|
2012-03-28 18:39:21 +00:00
|
|
|
MmReferencePage(Page);
|
2017-11-21 22:33:42 +00:00
|
|
|
MiReleasePfnLock(OldIrql);
|
2012-03-28 18:39:21 +00:00
|
|
|
Pages[(PageAddress - BeginningAddress) >> PAGE_SHIFT] = Entry;
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
Pages[(PageAddress - BeginningAddress) >> PAGE_SHIFT] = 0;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
MmUnlockSectionSegment(Segment);
|
|
|
|
|
MmUnlockAddressSpace(AddressSpace);
|
|
|
|
|
|
|
|
|
|
for (PageAddress = BeginningAddress;
|
|
|
|
|
PageAddress < EndingAddress;
|
|
|
|
|
PageAddress += PAGE_SIZE)
|
|
|
|
|
{
|
2012-03-28 19:41:40 +00:00
|
|
|
ULONG_PTR Entry;
|
2012-03-28 18:39:21 +00:00
|
|
|
FileOffset.QuadPart = ViewOffset.QuadPart + PageAddress - BeginningAddress;
|
|
|
|
|
Entry = Pages[(PageAddress - BeginningAddress) >> PAGE_SHIFT];
|
|
|
|
|
Page = PFN_FROM_SSE(Entry);
|
|
|
|
|
if (Page)
|
|
|
|
|
{
|
|
|
|
|
if (WriteData) {
|
|
|
|
|
//DPRINT("MiWriteBackPage(%wZ,addr %x,%08x%08x)\n", &Segment->FileObject->FileName, PageAddress, FileOffset.u.HighPart, FileOffset.u.LowPart);
|
|
|
|
|
Status = MiWriteBackPage(Segment->FileObject, &FileOffset, PAGE_SIZE, Page);
|
|
|
|
|
} else
|
|
|
|
|
Status = STATUS_SUCCESS;
|
|
|
|
|
|
|
|
|
|
if (NT_SUCCESS(Status)) {
|
|
|
|
|
MmLockAddressSpace(AddressSpace);
|
|
|
|
|
MmSetCleanAllRmaps(Page);
|
|
|
|
|
|
|
|
|
|
MmSetPageProtect(MmGetAddressSpaceOwner(AddressSpace),
|
|
|
|
|
(PVOID)PageAddress,
|
|
|
|
|
PAGE_READONLY);
|
|
|
|
|
|
|
|
|
|
MmLockSectionSegment(Segment);
|
|
|
|
|
Entry = MmGetPageEntrySectionSegment(Segment, &FileOffset);
|
|
|
|
|
|
|
|
|
|
if (Entry && !IS_SWAP_FROM_SSE(Entry) && PFN_FROM_SSE(Entry) == Page)
|
|
|
|
|
MmSetPageEntrySectionSegment(Segment, &FileOffset, CLEAN_SSE(Entry));
|
|
|
|
|
|
|
|
|
|
MmUnlockSectionSegment(Segment);
|
|
|
|
|
MmUnlockAddressSpace(AddressSpace);
|
|
|
|
|
} else {
|
|
|
|
|
DPRINT("Writeback from section flush %08x%08x (%x) %x@%x (%08x%08x:%wZ) failed %x\n",
|
|
|
|
|
FileOffset.u.HighPart,
|
|
|
|
|
FileOffset.u.LowPart,
|
|
|
|
|
(ULONG)(FileSize->QuadPart - FileOffset.QuadPart),
|
|
|
|
|
PageAddress,
|
|
|
|
|
Page,
|
|
|
|
|
FileSize->u.HighPart,
|
|
|
|
|
FileSize->u.LowPart,
|
|
|
|
|
&Segment->FileObject->FileName,
|
|
|
|
|
Status);
|
|
|
|
|
}
|
|
|
|
|
MmReleasePageMemoryConsumer(MC_CACHE, Page);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
ExFreePool(Pages);
|
|
|
|
|
|
|
|
|
|
return Status;
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
}
|
|
|
|
|
|
2012-03-29 04:43:44 +00:00
|
|
|
/*
|
|
|
|
|
|
|
|
|
|
This deletes a segment entirely including its page map.
|
|
|
|
|
It must have been unmapped in every address space.
|
|
|
|
|
|
|
|
|
|
*/
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
VOID
|
|
|
|
|
NTAPI
|
[NEWCC]
A reintegration checkpoint for the NewCC branch, brought to you by Team NewCC.
Differences with current ReactOS trunk:
* A new memory area type, MEMORY_AREA_CACHE, is added, which represents a mapped region of a file. In NEWCC mode, user sections are MEMORY_AREA_CACHE type as well, and obey new semantics. In non-NEWCC mode, they aren't used.
* A way of claiming a page entry for a specific thread's work is added. Placing the special SWAPENTRY value MM_WAIT_ENTRY in a page table, or in a section page table should indicate that memory management code is intended to wait for another thread to make some status change before checking the state of the page entry again. In code that uses this convention, a return value of STATUS_SUCCESS + 1 is used to indicate that the caller should use the MiWaitForPageEvent macro to wait until somebody has change the state of a wait entry before checking again. This is a lighter weight mechanism than PAGEOPs.
* A way of asking the caller to perform some blocking operation without locks held is provided. This replaces some spaghettified code in which locks are repeatedly taken and broken by code that performs various blocking operations. Using this mechanism, it is possible to do a small amount of non-blocking work, fill in a request, then return STATUS_MORE_PROCESSING_REQUIRED to request that locks be dropped and the blocking operation be carried out. A MM_REQUIRED_RESOURCES structure is provided to consumers of this contract to use to accumulate state across many blocking operations. Several functions wrapping blocking operations are provided in ntoskrnl/cache/reqtools.c.
* Image section pages are no longer direct mapped. This is done to simplify consolidation of ownership of pages under the data section system. At a later time, it may be possible to make data pages directly available to image sections for the same file. This is likely the only direct performance impact this code makes on non-NEWCC mode.
RMAPs:
* A new type of RMAP entry is introduced, distinguished by RMAP_IS_SEGMENT(Address) of the rmap entry. This kind of entry contains a pointer to a section page table node in the Process pointer, which in turn links back to the MM_SECTION_SEGMENT it belongs to. Therefore, a page belonging only to a segment (that is, a segment page that isn't mapped) can exist and be evicted using the normal page eviction mechanism in balance.c. Each of the rmap function has been modified to deal with segment rmaps.
* The low 8 bits of the Address field in a segment rmap denote the entry number in the generic table node pointed to by Process that points to the page the rmap belongs to. By combining them, you can determine the file offset the page belongs to.
* In NEWCC mode, MmSharePageEntry/UnsharePageEntry are not used, and instead the page reference count is used to keep track of the number of mappings of a page, allowing the last reference expiring to allow the page to be recycled without much intervention. These are still used in non-NEWCC mode. One change has been made, the count fields have been narrowed by 1 bit to make room for a dirty bit in SSE entries, needed when a page is present but unmapped.
Section page tables:
* The section page tables are now implemented using RtlGenericTables. This enables a fairly compact representation of section page tables without having the existence of a section object imply 4k of fake PDEs. In addition, each node in the generic table has a wide file offset that is a multiple of 256 pages, or 1 megabyte total. Besides needing wide file offsets, the only other visible change caused by the switch to generic tables for section page tables is the need to lock the section segment before interacting with the section page table.
Eviction:
* Page eviction in cache sections is accomplished by MmpPageOutPhysicalAddress. In the case of a shared page, it tries to remove all mappings of the indicated page. If this process fails at any point, the page will simply be drawn back into the target address spaces. After succeeding at this, if TRUE has been accumulated into the page's dirty bit in the section page table, it is written back, and then permanently removed.
NewCC mode:
* NEWCC mode is introduced, which rewrites the file cache to a set of cache stripes actively mapped, along with unmapped section data.
* NewCC is more authentic in its interpretation of the external interface to the windows cache than the current cache manager, implementing each of the cache manager functions according to the documented interface with no preconceived ideas about how anything should be implemented internally. Cache stripes are implemented on top of section objects, using the same memory manager paths, and therefore economizing code and complexity. This replaces a rather complicated system in which pages can be owned by the cache manager and the memory manager simultaneously and they must cooperate in a fairly sophisticated way to manage them. Since they're quite interdependent in the current code, modifying either is very difficult. In NEWCC, they have a clear division of labor and thus can be worked on independently.
* Several third party filesystems that use the kernel Cc interface work properly using NEWCC, including matt wu's ext3 driver.
* In contrast with code that tries to make CcInitializeCacheMap and CcUninitializeCacheMap into a pair that supports reference counting, NEWCC lazily initializes the shared and private cache maps as needed and uses the presence of a PrivateCacheMap on at least one file pointing to the SharedCacheMap as an indication that the FILE_OBJECT reference in the SharedCacheMap should still be held. When the last PrivateCacheMap is discarded, that's the appropriate time to tear down caching for a specific file, as the SharedCacheMap data is allowed to be saved and reused. We honor this by making the SharedCacheMap into a depot for keeping track of the PrivateCacheMap objects associated with views of a file.
svn path=/trunk/; revision=55833
2012-02-23 12:03:06 +00:00
|
|
|
MmFinalizeSegment(PMM_SECTION_SEGMENT Segment)
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
{
|
2012-03-28 18:39:21 +00:00
|
|
|
KIRQL OldIrql = 0;
|
|
|
|
|
|
|
|
|
|
DPRINT("Finalize segment %p\n", Segment);
|
|
|
|
|
|
|
|
|
|
MmLockSectionSegment(Segment);
|
|
|
|
|
RemoveEntryList(&Segment->ListOfSegments);
|
|
|
|
|
if (Segment->Flags & MM_DATAFILE_SEGMENT) {
|
|
|
|
|
KeAcquireSpinLock(&Segment->FileObject->IrpListLock, &OldIrql);
|
|
|
|
|
if (Segment->Flags & MM_SEGMENT_FINALIZE) {
|
|
|
|
|
KeReleaseSpinLock(&Segment->FileObject->IrpListLock, OldIrql);
|
|
|
|
|
MmUnlockSectionSegment(Segment);
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
Segment->Flags |= MM_SEGMENT_FINALIZE;
|
|
|
|
|
DPRINTC("Finalizing data file segment %p\n", Segment);
|
|
|
|
|
|
|
|
|
|
Segment->FileObject->SectionObjectPointer->DataSectionObject = NULL;
|
|
|
|
|
KeReleaseSpinLock(&Segment->FileObject->IrpListLock, OldIrql);
|
|
|
|
|
MmFreePageTablesSectionSegment(Segment, MiFreeSegmentPage);
|
|
|
|
|
MmUnlockSectionSegment(Segment);
|
|
|
|
|
DPRINT("Dereference file object %wZ\n", &Segment->FileObject->FileName);
|
|
|
|
|
ObDereferenceObject(Segment->FileObject);
|
|
|
|
|
DPRINT("Done with %wZ\n", &Segment->FileObject->FileName);
|
|
|
|
|
Segment->FileObject = NULL;
|
|
|
|
|
} else {
|
|
|
|
|
DPRINTC("Finalizing segment %p\n", Segment);
|
|
|
|
|
MmFreePageTablesSectionSegment(Segment, MiFreeSegmentPage);
|
|
|
|
|
MmUnlockSectionSegment(Segment);
|
|
|
|
|
}
|
|
|
|
|
DPRINTC("Segment %p destroy\n", Segment);
|
|
|
|
|
ExFreePoolWithTag(Segment, TAG_MM_SECTION_SEGMENT);
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
NTSTATUS
|
|
|
|
|
NTAPI
|
[NEWCC]
A reintegration checkpoint for the NewCC branch, brought to you by Team NewCC.
Differences with current ReactOS trunk:
* A new memory area type, MEMORY_AREA_CACHE, is added, which represents a mapped region of a file. In NEWCC mode, user sections are MEMORY_AREA_CACHE type as well, and obey new semantics. In non-NEWCC mode, they aren't used.
* A way of claiming a page entry for a specific thread's work is added. Placing the special SWAPENTRY value MM_WAIT_ENTRY in a page table, or in a section page table should indicate that memory management code is intended to wait for another thread to make some status change before checking the state of the page entry again. In code that uses this convention, a return value of STATUS_SUCCESS + 1 is used to indicate that the caller should use the MiWaitForPageEvent macro to wait until somebody has change the state of a wait entry before checking again. This is a lighter weight mechanism than PAGEOPs.
* A way of asking the caller to perform some blocking operation without locks held is provided. This replaces some spaghettified code in which locks are repeatedly taken and broken by code that performs various blocking operations. Using this mechanism, it is possible to do a small amount of non-blocking work, fill in a request, then return STATUS_MORE_PROCESSING_REQUIRED to request that locks be dropped and the blocking operation be carried out. A MM_REQUIRED_RESOURCES structure is provided to consumers of this contract to use to accumulate state across many blocking operations. Several functions wrapping blocking operations are provided in ntoskrnl/cache/reqtools.c.
* Image section pages are no longer direct mapped. This is done to simplify consolidation of ownership of pages under the data section system. At a later time, it may be possible to make data pages directly available to image sections for the same file. This is likely the only direct performance impact this code makes on non-NEWCC mode.
RMAPs:
* A new type of RMAP entry is introduced, distinguished by RMAP_IS_SEGMENT(Address) of the rmap entry. This kind of entry contains a pointer to a section page table node in the Process pointer, which in turn links back to the MM_SECTION_SEGMENT it belongs to. Therefore, a page belonging only to a segment (that is, a segment page that isn't mapped) can exist and be evicted using the normal page eviction mechanism in balance.c. Each of the rmap function has been modified to deal with segment rmaps.
* The low 8 bits of the Address field in a segment rmap denote the entry number in the generic table node pointed to by Process that points to the page the rmap belongs to. By combining them, you can determine the file offset the page belongs to.
* In NEWCC mode, MmSharePageEntry/UnsharePageEntry are not used, and instead the page reference count is used to keep track of the number of mappings of a page, allowing the last reference expiring to allow the page to be recycled without much intervention. These are still used in non-NEWCC mode. One change has been made, the count fields have been narrowed by 1 bit to make room for a dirty bit in SSE entries, needed when a page is present but unmapped.
Section page tables:
* The section page tables are now implemented using RtlGenericTables. This enables a fairly compact representation of section page tables without having the existence of a section object imply 4k of fake PDEs. In addition, each node in the generic table has a wide file offset that is a multiple of 256 pages, or 1 megabyte total. Besides needing wide file offsets, the only other visible change caused by the switch to generic tables for section page tables is the need to lock the section segment before interacting with the section page table.
Eviction:
* Page eviction in cache sections is accomplished by MmpPageOutPhysicalAddress. In the case of a shared page, it tries to remove all mappings of the indicated page. If this process fails at any point, the page will simply be drawn back into the target address spaces. After succeeding at this, if TRUE has been accumulated into the page's dirty bit in the section page table, it is written back, and then permanently removed.
NewCC mode:
* NEWCC mode is introduced, which rewrites the file cache to a set of cache stripes actively mapped, along with unmapped section data.
* NewCC is more authentic in its interpretation of the external interface to the windows cache than the current cache manager, implementing each of the cache manager functions according to the documented interface with no preconceived ideas about how anything should be implemented internally. Cache stripes are implemented on top of section objects, using the same memory manager paths, and therefore economizing code and complexity. This replaces a rather complicated system in which pages can be owned by the cache manager and the memory manager simultaneously and they must cooperate in a fairly sophisticated way to manage them. Since they're quite interdependent in the current code, modifying either is very difficult. In NEWCC, they have a clear division of labor and thus can be worked on independently.
* Several third party filesystems that use the kernel Cc interface work properly using NEWCC, including matt wu's ext3 driver.
* In contrast with code that tries to make CcInitializeCacheMap and CcUninitializeCacheMap into a pair that supports reference counting, NEWCC lazily initializes the shared and private cache maps as needed and uses the presence of a PrivateCacheMap on at least one file pointing to the SharedCacheMap as an indication that the FILE_OBJECT reference in the SharedCacheMap should still be held. When the last PrivateCacheMap is discarded, that's the appropriate time to tear down caching for a specific file, as the SharedCacheMap data is allowed to be saved and reused. We honor this by making the SharedCacheMap into a depot for keeping track of the PrivateCacheMap objects associated with views of a file.
svn path=/trunk/; revision=55833
2012-02-23 12:03:06 +00:00
|
|
|
MmCreateCacheSection(PROS_SECTION_OBJECT *SectionObject,
|
2012-03-28 18:39:21 +00:00
|
|
|
ACCESS_MASK DesiredAccess,
|
|
|
|
|
POBJECT_ATTRIBUTES ObjectAttributes,
|
|
|
|
|
PLARGE_INTEGER UMaximumSize,
|
|
|
|
|
ULONG SectionPageProtection,
|
|
|
|
|
ULONG AllocationAttributes,
|
|
|
|
|
PFILE_OBJECT FileObject)
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
/*
|
2012-03-29 04:43:44 +00:00
|
|
|
* Create a section backed by a data file.
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
*/
|
|
|
|
|
{
|
2012-03-28 18:39:21 +00:00
|
|
|
PROS_SECTION_OBJECT Section;
|
|
|
|
|
NTSTATUS Status;
|
2012-03-28 19:41:40 +00:00
|
|
|
LARGE_INTEGER MaximumSize;
|
2012-03-28 18:39:21 +00:00
|
|
|
PMM_SECTION_SEGMENT Segment;
|
|
|
|
|
IO_STATUS_BLOCK Iosb;
|
|
|
|
|
CC_FILE_SIZES FileSizes;
|
|
|
|
|
FILE_STANDARD_INFORMATION FileInfo;
|
|
|
|
|
KIRQL OldIrql;
|
|
|
|
|
|
|
|
|
|
DPRINT("MmCreateDataFileSection\n");
|
|
|
|
|
|
|
|
|
|
/* Create the section */
|
|
|
|
|
Status = ObCreateObject(ExGetPreviousMode(),
|
|
|
|
|
MmSectionObjectType,
|
|
|
|
|
ObjectAttributes,
|
|
|
|
|
ExGetPreviousMode(),
|
|
|
|
|
NULL,
|
|
|
|
|
sizeof(ROS_SECTION_OBJECT),
|
|
|
|
|
0,
|
|
|
|
|
0,
|
|
|
|
|
(PVOID*)(PVOID)&Section);
|
|
|
|
|
if (!NT_SUCCESS(Status))
|
|
|
|
|
{
|
|
|
|
|
DPRINT("Failed: %x\n", Status);
|
2017-04-20 08:58:45 +00:00
|
|
|
ObDereferenceObject(FileObject);
|
2012-03-28 18:39:21 +00:00
|
|
|
return Status;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* Initialize it */
|
|
|
|
|
RtlZeroMemory(Section, sizeof(ROS_SECTION_OBJECT));
|
|
|
|
|
Section->Type = 'SC';
|
|
|
|
|
Section->Size = 'TN';
|
|
|
|
|
Section->SectionPageProtection = SectionPageProtection;
|
|
|
|
|
Section->AllocationAttributes = AllocationAttributes;
|
|
|
|
|
Section->Segment = NULL;
|
|
|
|
|
|
|
|
|
|
Section->FileObject = FileObject;
|
|
|
|
|
|
|
|
|
|
DPRINT("Getting original file size\n");
|
|
|
|
|
/* A hack: If we're cached, we can overcome deadlocking with the upper
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
* layer filesystem call by retriving the object sizes from the cache
|
|
|
|
|
* which is made to keep track. If I had to guess, they were figuring
|
|
|
|
|
* out a similar problem.
|
|
|
|
|
*/
|
2012-03-28 18:39:21 +00:00
|
|
|
if (!CcGetFileSizes(FileObject, &FileSizes))
|
|
|
|
|
{
|
2012-03-28 19:41:40 +00:00
|
|
|
ULONG Information;
|
2012-03-28 18:39:21 +00:00
|
|
|
/*
|
|
|
|
|
* FIXME: This is propably not entirely correct. We can't look into
|
|
|
|
|
* the standard FCB header because it might not be initialized yet
|
|
|
|
|
* (as in case of the EXT2FS driver by Manoj Paul Joseph where the
|
|
|
|
|
* standard file information is filled on first request).
|
|
|
|
|
*/
|
|
|
|
|
DPRINT("Querying info\n");
|
|
|
|
|
Status = IoQueryFileInformation(FileObject,
|
|
|
|
|
FileStandardInformation,
|
|
|
|
|
sizeof(FILE_STANDARD_INFORMATION),
|
|
|
|
|
&FileInfo,
|
2012-03-28 19:41:40 +00:00
|
|
|
&Information);
|
|
|
|
|
Iosb.Information = Information;
|
2012-03-28 18:39:21 +00:00
|
|
|
DPRINT("Query => %x\n", Status);
|
2013-06-02 19:04:02 +00:00
|
|
|
DBG_UNREFERENCED_LOCAL_VARIABLE(Iosb);
|
2012-03-28 18:39:21 +00:00
|
|
|
|
|
|
|
|
if (!NT_SUCCESS(Status))
|
|
|
|
|
{
|
|
|
|
|
DPRINT("Status %x\n", Status);
|
|
|
|
|
ObDereferenceObject(Section);
|
2017-04-20 08:58:45 +00:00
|
|
|
ObDereferenceObject(FileObject);
|
2012-03-28 18:39:21 +00:00
|
|
|
return Status;
|
|
|
|
|
}
|
|
|
|
|
ASSERT(Status != STATUS_PENDING);
|
|
|
|
|
|
|
|
|
|
FileSizes.ValidDataLength = FileInfo.EndOfFile;
|
|
|
|
|
FileSizes.FileSize = FileInfo.EndOfFile;
|
|
|
|
|
}
|
2014-12-26 11:38:31 +00:00
|
|
|
DPRINT("Got %I64x\n", FileSizes.ValidDataLength.QuadPart);
|
2012-03-28 18:39:21 +00:00
|
|
|
|
|
|
|
|
/*
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
* FIXME: Revise this once a locking order for file size changes is
|
|
|
|
|
* decided
|
2012-03-28 18:39:21 +00:00
|
|
|
*
|
|
|
|
|
* We're handed down a maximum size in every case. Should we still check at all?
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
*/
|
2012-03-28 18:39:21 +00:00
|
|
|
if (UMaximumSize != NULL && UMaximumSize->QuadPart)
|
|
|
|
|
{
|
2014-12-26 11:38:31 +00:00
|
|
|
DPRINT("Taking maximum %I64x\n", UMaximumSize->QuadPart);
|
2012-03-28 18:39:21 +00:00
|
|
|
MaximumSize.QuadPart = UMaximumSize->QuadPart;
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
2014-12-26 11:38:31 +00:00
|
|
|
DPRINT("Got file size %I64x\n", FileSizes.FileSize.QuadPart);
|
2012-03-28 18:39:21 +00:00
|
|
|
MaximumSize.QuadPart = FileSizes.FileSize.QuadPart;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* Mapping zero-sized files isn't allowed. */
|
|
|
|
|
if (MaximumSize.QuadPart == 0)
|
|
|
|
|
{
|
|
|
|
|
DPRINT("Zero size file\n");
|
|
|
|
|
ObDereferenceObject(Section);
|
2017-04-20 08:58:45 +00:00
|
|
|
ObDereferenceObject(FileObject);
|
2012-03-28 18:39:21 +00:00
|
|
|
return STATUS_FILE_INVALID;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
Segment = ExAllocatePoolWithTag(NonPagedPool,
|
|
|
|
|
sizeof(MM_SECTION_SEGMENT),
|
|
|
|
|
TAG_MM_SECTION_SEGMENT);
|
|
|
|
|
if (Segment == NULL)
|
|
|
|
|
{
|
|
|
|
|
DPRINT("Failed: STATUS_NO_MEMORY\n");
|
|
|
|
|
ObDereferenceObject(Section);
|
2017-04-20 08:58:45 +00:00
|
|
|
ObDereferenceObject(FileObject);
|
2012-03-28 18:39:21 +00:00
|
|
|
return STATUS_NO_MEMORY;
|
|
|
|
|
}
|
|
|
|
|
|
2013-02-16 17:37:17 +00:00
|
|
|
DPRINT("Zeroing %p\n", Segment);
|
2012-03-28 18:39:21 +00:00
|
|
|
RtlZeroMemory(Segment, sizeof(MM_SECTION_SEGMENT));
|
|
|
|
|
ExInitializeFastMutex(&Segment->Lock);
|
|
|
|
|
|
|
|
|
|
Segment->ReferenceCount = 1;
|
|
|
|
|
Segment->Locked = TRUE;
|
|
|
|
|
RtlZeroMemory(&Segment->Image, sizeof(Segment->Image));
|
|
|
|
|
Section->Segment = Segment;
|
|
|
|
|
|
|
|
|
|
KeAcquireSpinLock(&FileObject->IrpListLock, &OldIrql);
|
|
|
|
|
/*
|
[NEWCC]
A reintegration checkpoint for the NewCC branch, brought to you by Team NewCC.
Differences with current ReactOS trunk:
* A new memory area type, MEMORY_AREA_CACHE, is added, which represents a mapped region of a file. In NEWCC mode, user sections are MEMORY_AREA_CACHE type as well, and obey new semantics. In non-NEWCC mode, they aren't used.
* A way of claiming a page entry for a specific thread's work is added. Placing the special SWAPENTRY value MM_WAIT_ENTRY in a page table, or in a section page table should indicate that memory management code is intended to wait for another thread to make some status change before checking the state of the page entry again. In code that uses this convention, a return value of STATUS_SUCCESS + 1 is used to indicate that the caller should use the MiWaitForPageEvent macro to wait until somebody has change the state of a wait entry before checking again. This is a lighter weight mechanism than PAGEOPs.
* A way of asking the caller to perform some blocking operation without locks held is provided. This replaces some spaghettified code in which locks are repeatedly taken and broken by code that performs various blocking operations. Using this mechanism, it is possible to do a small amount of non-blocking work, fill in a request, then return STATUS_MORE_PROCESSING_REQUIRED to request that locks be dropped and the blocking operation be carried out. A MM_REQUIRED_RESOURCES structure is provided to consumers of this contract to use to accumulate state across many blocking operations. Several functions wrapping blocking operations are provided in ntoskrnl/cache/reqtools.c.
* Image section pages are no longer direct mapped. This is done to simplify consolidation of ownership of pages under the data section system. At a later time, it may be possible to make data pages directly available to image sections for the same file. This is likely the only direct performance impact this code makes on non-NEWCC mode.
RMAPs:
* A new type of RMAP entry is introduced, distinguished by RMAP_IS_SEGMENT(Address) of the rmap entry. This kind of entry contains a pointer to a section page table node in the Process pointer, which in turn links back to the MM_SECTION_SEGMENT it belongs to. Therefore, a page belonging only to a segment (that is, a segment page that isn't mapped) can exist and be evicted using the normal page eviction mechanism in balance.c. Each of the rmap function has been modified to deal with segment rmaps.
* The low 8 bits of the Address field in a segment rmap denote the entry number in the generic table node pointed to by Process that points to the page the rmap belongs to. By combining them, you can determine the file offset the page belongs to.
* In NEWCC mode, MmSharePageEntry/UnsharePageEntry are not used, and instead the page reference count is used to keep track of the number of mappings of a page, allowing the last reference expiring to allow the page to be recycled without much intervention. These are still used in non-NEWCC mode. One change has been made, the count fields have been narrowed by 1 bit to make room for a dirty bit in SSE entries, needed when a page is present but unmapped.
Section page tables:
* The section page tables are now implemented using RtlGenericTables. This enables a fairly compact representation of section page tables without having the existence of a section object imply 4k of fake PDEs. In addition, each node in the generic table has a wide file offset that is a multiple of 256 pages, or 1 megabyte total. Besides needing wide file offsets, the only other visible change caused by the switch to generic tables for section page tables is the need to lock the section segment before interacting with the section page table.
Eviction:
* Page eviction in cache sections is accomplished by MmpPageOutPhysicalAddress. In the case of a shared page, it tries to remove all mappings of the indicated page. If this process fails at any point, the page will simply be drawn back into the target address spaces. After succeeding at this, if TRUE has been accumulated into the page's dirty bit in the section page table, it is written back, and then permanently removed.
NewCC mode:
* NEWCC mode is introduced, which rewrites the file cache to a set of cache stripes actively mapped, along with unmapped section data.
* NewCC is more authentic in its interpretation of the external interface to the windows cache than the current cache manager, implementing each of the cache manager functions according to the documented interface with no preconceived ideas about how anything should be implemented internally. Cache stripes are implemented on top of section objects, using the same memory manager paths, and therefore economizing code and complexity. This replaces a rather complicated system in which pages can be owned by the cache manager and the memory manager simultaneously and they must cooperate in a fairly sophisticated way to manage them. Since they're quite interdependent in the current code, modifying either is very difficult. In NEWCC, they have a clear division of labor and thus can be worked on independently.
* Several third party filesystems that use the kernel Cc interface work properly using NEWCC, including matt wu's ext3 driver.
* In contrast with code that tries to make CcInitializeCacheMap and CcUninitializeCacheMap into a pair that supports reference counting, NEWCC lazily initializes the shared and private cache maps as needed and uses the presence of a PrivateCacheMap on at least one file pointing to the SharedCacheMap as an indication that the FILE_OBJECT reference in the SharedCacheMap should still be held. When the last PrivateCacheMap is discarded, that's the appropriate time to tear down caching for a specific file, as the SharedCacheMap data is allowed to be saved and reused. We honor this by making the SharedCacheMap into a depot for keeping track of the PrivateCacheMap objects associated with views of a file.
svn path=/trunk/; revision=55833
2012-02-23 12:03:06 +00:00
|
|
|
* If this file hasn't been mapped as a data file before then allocate a
|
|
|
|
|
* section segment to describe the data file mapping
|
|
|
|
|
*/
|
2012-03-28 18:39:21 +00:00
|
|
|
if (FileObject->SectionObjectPointer->DataSectionObject == NULL)
|
|
|
|
|
{
|
|
|
|
|
FileObject->SectionObjectPointer->DataSectionObject = (PVOID)Segment;
|
|
|
|
|
KeReleaseSpinLock(&FileObject->IrpListLock, OldIrql);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Set the lock before assigning the segment to the file object
|
|
|
|
|
*/
|
|
|
|
|
ExAcquireFastMutex(&Segment->Lock);
|
|
|
|
|
|
|
|
|
|
DPRINT("Filling out Segment info (No previous data section)\n");
|
|
|
|
|
ObReferenceObject(FileObject);
|
|
|
|
|
Segment->FileObject = FileObject;
|
|
|
|
|
Segment->Protection = SectionPageProtection;
|
|
|
|
|
Segment->Flags = MM_DATAFILE_SEGMENT;
|
|
|
|
|
memset(&Segment->Image, 0, sizeof(Segment->Image));
|
|
|
|
|
Segment->WriteCopy = FALSE;
|
|
|
|
|
|
|
|
|
|
if (AllocationAttributes & SEC_RESERVE)
|
|
|
|
|
{
|
|
|
|
|
Segment->Length.QuadPart = Segment->RawLength.QuadPart = 0;
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
Segment->RawLength.QuadPart = MaximumSize.QuadPart;
|
|
|
|
|
Segment->Length.QuadPart = PAGE_ROUND_UP(Segment->RawLength.QuadPart);
|
|
|
|
|
}
|
|
|
|
|
MiInitializeSectionPageTable(Segment);
|
|
|
|
|
InsertHeadList(&MiSegmentList, &Segment->ListOfSegments);
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
KeReleaseSpinLock(&FileObject->IrpListLock, OldIrql);
|
2013-02-16 17:37:17 +00:00
|
|
|
DPRINTC("Free Segment %p\n", Segment);
|
2012-09-28 12:17:23 +00:00
|
|
|
ExFreePoolWithTag(Segment, TAG_MM_SECTION_SEGMENT);
|
2012-03-28 18:39:21 +00:00
|
|
|
|
|
|
|
|
DPRINT("Filling out Segment info (previous data section)\n");
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If the file is already mapped as a data file then we may need
|
|
|
|
|
* to extend it
|
|
|
|
|
*/
|
|
|
|
|
Segment = (PMM_SECTION_SEGMENT)FileObject->SectionObjectPointer->DataSectionObject;
|
|
|
|
|
Section->Segment = Segment;
|
|
|
|
|
(void)InterlockedIncrementUL(&Segment->ReferenceCount);
|
|
|
|
|
|
|
|
|
|
MmLockSectionSegment(Segment);
|
|
|
|
|
|
|
|
|
|
if (MaximumSize.QuadPart > Segment->RawLength.QuadPart &&
|
[NEWCC]
A reintegration checkpoint for the NewCC branch, brought to you by Team NewCC.
Differences with current ReactOS trunk:
* A new memory area type, MEMORY_AREA_CACHE, is added, which represents a mapped region of a file. In NEWCC mode, user sections are MEMORY_AREA_CACHE type as well, and obey new semantics. In non-NEWCC mode, they aren't used.
* A way of claiming a page entry for a specific thread's work is added. Placing the special SWAPENTRY value MM_WAIT_ENTRY in a page table, or in a section page table should indicate that memory management code is intended to wait for another thread to make some status change before checking the state of the page entry again. In code that uses this convention, a return value of STATUS_SUCCESS + 1 is used to indicate that the caller should use the MiWaitForPageEvent macro to wait until somebody has change the state of a wait entry before checking again. This is a lighter weight mechanism than PAGEOPs.
* A way of asking the caller to perform some blocking operation without locks held is provided. This replaces some spaghettified code in which locks are repeatedly taken and broken by code that performs various blocking operations. Using this mechanism, it is possible to do a small amount of non-blocking work, fill in a request, then return STATUS_MORE_PROCESSING_REQUIRED to request that locks be dropped and the blocking operation be carried out. A MM_REQUIRED_RESOURCES structure is provided to consumers of this contract to use to accumulate state across many blocking operations. Several functions wrapping blocking operations are provided in ntoskrnl/cache/reqtools.c.
* Image section pages are no longer direct mapped. This is done to simplify consolidation of ownership of pages under the data section system. At a later time, it may be possible to make data pages directly available to image sections for the same file. This is likely the only direct performance impact this code makes on non-NEWCC mode.
RMAPs:
* A new type of RMAP entry is introduced, distinguished by RMAP_IS_SEGMENT(Address) of the rmap entry. This kind of entry contains a pointer to a section page table node in the Process pointer, which in turn links back to the MM_SECTION_SEGMENT it belongs to. Therefore, a page belonging only to a segment (that is, a segment page that isn't mapped) can exist and be evicted using the normal page eviction mechanism in balance.c. Each of the rmap function has been modified to deal with segment rmaps.
* The low 8 bits of the Address field in a segment rmap denote the entry number in the generic table node pointed to by Process that points to the page the rmap belongs to. By combining them, you can determine the file offset the page belongs to.
* In NEWCC mode, MmSharePageEntry/UnsharePageEntry are not used, and instead the page reference count is used to keep track of the number of mappings of a page, allowing the last reference expiring to allow the page to be recycled without much intervention. These are still used in non-NEWCC mode. One change has been made, the count fields have been narrowed by 1 bit to make room for a dirty bit in SSE entries, needed when a page is present but unmapped.
Section page tables:
* The section page tables are now implemented using RtlGenericTables. This enables a fairly compact representation of section page tables without having the existence of a section object imply 4k of fake PDEs. In addition, each node in the generic table has a wide file offset that is a multiple of 256 pages, or 1 megabyte total. Besides needing wide file offsets, the only other visible change caused by the switch to generic tables for section page tables is the need to lock the section segment before interacting with the section page table.
Eviction:
* Page eviction in cache sections is accomplished by MmpPageOutPhysicalAddress. In the case of a shared page, it tries to remove all mappings of the indicated page. If this process fails at any point, the page will simply be drawn back into the target address spaces. After succeeding at this, if TRUE has been accumulated into the page's dirty bit in the section page table, it is written back, and then permanently removed.
NewCC mode:
* NEWCC mode is introduced, which rewrites the file cache to a set of cache stripes actively mapped, along with unmapped section data.
* NewCC is more authentic in its interpretation of the external interface to the windows cache than the current cache manager, implementing each of the cache manager functions according to the documented interface with no preconceived ideas about how anything should be implemented internally. Cache stripes are implemented on top of section objects, using the same memory manager paths, and therefore economizing code and complexity. This replaces a rather complicated system in which pages can be owned by the cache manager and the memory manager simultaneously and they must cooperate in a fairly sophisticated way to manage them. Since they're quite interdependent in the current code, modifying either is very difficult. In NEWCC, they have a clear division of labor and thus can be worked on independently.
* Several third party filesystems that use the kernel Cc interface work properly using NEWCC, including matt wu's ext3 driver.
* In contrast with code that tries to make CcInitializeCacheMap and CcUninitializeCacheMap into a pair that supports reference counting, NEWCC lazily initializes the shared and private cache maps as needed and uses the presence of a PrivateCacheMap on at least one file pointing to the SharedCacheMap as an indication that the FILE_OBJECT reference in the SharedCacheMap should still be held. When the last PrivateCacheMap is discarded, that's the appropriate time to tear down caching for a specific file, as the SharedCacheMap data is allowed to be saved and reused. We honor this by making the SharedCacheMap into a depot for keeping track of the PrivateCacheMap objects associated with views of a file.
svn path=/trunk/; revision=55833
2012-02-23 12:03:06 +00:00
|
|
|
!(AllocationAttributes & SEC_RESERVE))
|
2012-03-28 18:39:21 +00:00
|
|
|
{
|
|
|
|
|
Segment->RawLength.QuadPart = MaximumSize.QuadPart;
|
|
|
|
|
Segment->Length.QuadPart = PAGE_ROUND_UP(Segment->RawLength.QuadPart);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
MmUnlockSectionSegment(Segment);
|
|
|
|
|
|
|
|
|
|
Section->MaximumSize.QuadPart = MaximumSize.QuadPart;
|
|
|
|
|
|
|
|
|
|
/* Extend file if section is longer */
|
2013-02-16 17:37:17 +00:00
|
|
|
DPRINT("MaximumSize %I64x ValidDataLength %I64x\n",
|
|
|
|
|
MaximumSize.QuadPart,
|
|
|
|
|
FileSizes.ValidDataLength.QuadPart);
|
2012-03-28 18:39:21 +00:00
|
|
|
if (MaximumSize.QuadPart > FileSizes.ValidDataLength.QuadPart)
|
|
|
|
|
{
|
2013-02-16 17:37:17 +00:00
|
|
|
DPRINT("Changing file size to %I64x, segment %p\n",
|
|
|
|
|
MaximumSize.QuadPart,
|
2012-03-28 18:39:21 +00:00
|
|
|
Segment);
|
|
|
|
|
|
|
|
|
|
Status = IoSetInformation(FileObject,
|
|
|
|
|
FileEndOfFileInformation,
|
|
|
|
|
sizeof(LARGE_INTEGER),
|
|
|
|
|
&MaximumSize);
|
|
|
|
|
|
|
|
|
|
DPRINT("Change: Status %x\n", Status);
|
|
|
|
|
if (!NT_SUCCESS(Status))
|
|
|
|
|
{
|
|
|
|
|
DPRINT("Could not expand section\n");
|
|
|
|
|
ObDereferenceObject(Section);
|
|
|
|
|
return Status;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2013-02-16 17:37:17 +00:00
|
|
|
DPRINTC("Segment %p created (%x)\n", Segment, Segment->Flags);
|
2012-03-28 18:39:21 +00:00
|
|
|
|
|
|
|
|
*SectionObject = Section;
|
|
|
|
|
return STATUS_SUCCESS;
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
NTSTATUS
|
|
|
|
|
NTAPI
|
2012-03-28 18:39:21 +00:00
|
|
|
_MiMapViewOfSegment(PMMSUPPORT AddressSpace,
|
|
|
|
|
PMM_SECTION_SEGMENT Segment,
|
|
|
|
|
PVOID* BaseAddress,
|
|
|
|
|
SIZE_T ViewSize,
|
|
|
|
|
ULONG Protect,
|
|
|
|
|
PLARGE_INTEGER ViewOffset,
|
|
|
|
|
ULONG AllocationType,
|
|
|
|
|
const char *file,
|
|
|
|
|
int line)
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
{
|
2012-03-28 18:39:21 +00:00
|
|
|
PMEMORY_AREA MArea;
|
|
|
|
|
NTSTATUS Status;
|
|
|
|
|
|
|
|
|
|
Status = MmCreateMemoryArea(AddressSpace,
|
|
|
|
|
MEMORY_AREA_CACHE,
|
|
|
|
|
BaseAddress,
|
|
|
|
|
ViewSize,
|
|
|
|
|
Protect,
|
|
|
|
|
&MArea,
|
|
|
|
|
AllocationType,
|
2014-06-09 23:35:16 +00:00
|
|
|
*BaseAddress ?
|
|
|
|
|
PAGE_SIZE : MM_ALLOCATION_GRANULARITY);
|
2012-03-28 18:39:21 +00:00
|
|
|
|
|
|
|
|
if (!NT_SUCCESS(Status))
|
|
|
|
|
{
|
2013-02-16 17:37:17 +00:00
|
|
|
DPRINT("Mapping between 0x%p and 0x%p failed (%X).\n",
|
2012-03-28 18:39:21 +00:00
|
|
|
(*BaseAddress),
|
|
|
|
|
(char*)(*BaseAddress) + ViewSize,
|
|
|
|
|
Status);
|
|
|
|
|
|
|
|
|
|
return Status;
|
|
|
|
|
}
|
|
|
|
|
|
2013-02-16 17:37:17 +00:00
|
|
|
DPRINTC("MiMapViewOfSegment %p %p %p %I64x %Ix %wZ %s:%d\n",
|
2012-03-28 18:39:21 +00:00
|
|
|
MmGetAddressSpaceOwner(AddressSpace),
|
|
|
|
|
*BaseAddress,
|
|
|
|
|
Segment,
|
2013-02-16 17:37:17 +00:00
|
|
|
ViewOffset ? ViewOffset->QuadPart : 0,
|
2012-03-28 18:39:21 +00:00
|
|
|
ViewSize,
|
|
|
|
|
Segment->FileObject ? &Segment->FileObject->FileName : NULL,
|
|
|
|
|
file,
|
|
|
|
|
line);
|
|
|
|
|
|
|
|
|
|
MArea->Data.SectionData.Segment = Segment;
|
|
|
|
|
if (ViewOffset)
|
|
|
|
|
MArea->Data.SectionData.ViewOffset = *ViewOffset;
|
|
|
|
|
else
|
|
|
|
|
MArea->Data.SectionData.ViewOffset.QuadPart = 0;
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
|
|
|
|
|
#if 0
|
2012-03-28 18:39:21 +00:00
|
|
|
MArea->NotPresent = MmNotPresentFaultPageFile;
|
|
|
|
|
MArea->AccessFault = MiCowSectionPage;
|
|
|
|
|
MArea->PageOut = MmPageOutPageFileView;
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
#endif
|
|
|
|
|
|
2012-03-28 18:39:21 +00:00
|
|
|
MmInitializeRegion(&MArea->Data.SectionData.RegionListHead,
|
|
|
|
|
ViewSize,
|
|
|
|
|
0,
|
|
|
|
|
Protect);
|
[NEWCC]
A reintegration checkpoint for the NewCC branch, brought to you by Team NewCC.
Differences with current ReactOS trunk:
* A new memory area type, MEMORY_AREA_CACHE, is added, which represents a mapped region of a file. In NEWCC mode, user sections are MEMORY_AREA_CACHE type as well, and obey new semantics. In non-NEWCC mode, they aren't used.
* A way of claiming a page entry for a specific thread's work is added. Placing the special SWAPENTRY value MM_WAIT_ENTRY in a page table, or in a section page table should indicate that memory management code is intended to wait for another thread to make some status change before checking the state of the page entry again. In code that uses this convention, a return value of STATUS_SUCCESS + 1 is used to indicate that the caller should use the MiWaitForPageEvent macro to wait until somebody has change the state of a wait entry before checking again. This is a lighter weight mechanism than PAGEOPs.
* A way of asking the caller to perform some blocking operation without locks held is provided. This replaces some spaghettified code in which locks are repeatedly taken and broken by code that performs various blocking operations. Using this mechanism, it is possible to do a small amount of non-blocking work, fill in a request, then return STATUS_MORE_PROCESSING_REQUIRED to request that locks be dropped and the blocking operation be carried out. A MM_REQUIRED_RESOURCES structure is provided to consumers of this contract to use to accumulate state across many blocking operations. Several functions wrapping blocking operations are provided in ntoskrnl/cache/reqtools.c.
* Image section pages are no longer direct mapped. This is done to simplify consolidation of ownership of pages under the data section system. At a later time, it may be possible to make data pages directly available to image sections for the same file. This is likely the only direct performance impact this code makes on non-NEWCC mode.
RMAPs:
* A new type of RMAP entry is introduced, distinguished by RMAP_IS_SEGMENT(Address) of the rmap entry. This kind of entry contains a pointer to a section page table node in the Process pointer, which in turn links back to the MM_SECTION_SEGMENT it belongs to. Therefore, a page belonging only to a segment (that is, a segment page that isn't mapped) can exist and be evicted using the normal page eviction mechanism in balance.c. Each of the rmap function has been modified to deal with segment rmaps.
* The low 8 bits of the Address field in a segment rmap denote the entry number in the generic table node pointed to by Process that points to the page the rmap belongs to. By combining them, you can determine the file offset the page belongs to.
* In NEWCC mode, MmSharePageEntry/UnsharePageEntry are not used, and instead the page reference count is used to keep track of the number of mappings of a page, allowing the last reference expiring to allow the page to be recycled without much intervention. These are still used in non-NEWCC mode. One change has been made, the count fields have been narrowed by 1 bit to make room for a dirty bit in SSE entries, needed when a page is present but unmapped.
Section page tables:
* The section page tables are now implemented using RtlGenericTables. This enables a fairly compact representation of section page tables without having the existence of a section object imply 4k of fake PDEs. In addition, each node in the generic table has a wide file offset that is a multiple of 256 pages, or 1 megabyte total. Besides needing wide file offsets, the only other visible change caused by the switch to generic tables for section page tables is the need to lock the section segment before interacting with the section page table.
Eviction:
* Page eviction in cache sections is accomplished by MmpPageOutPhysicalAddress. In the case of a shared page, it tries to remove all mappings of the indicated page. If this process fails at any point, the page will simply be drawn back into the target address spaces. After succeeding at this, if TRUE has been accumulated into the page's dirty bit in the section page table, it is written back, and then permanently removed.
NewCC mode:
* NEWCC mode is introduced, which rewrites the file cache to a set of cache stripes actively mapped, along with unmapped section data.
* NewCC is more authentic in its interpretation of the external interface to the windows cache than the current cache manager, implementing each of the cache manager functions according to the documented interface with no preconceived ideas about how anything should be implemented internally. Cache stripes are implemented on top of section objects, using the same memory manager paths, and therefore economizing code and complexity. This replaces a rather complicated system in which pages can be owned by the cache manager and the memory manager simultaneously and they must cooperate in a fairly sophisticated way to manage them. Since they're quite interdependent in the current code, modifying either is very difficult. In NEWCC, they have a clear division of labor and thus can be worked on independently.
* Several third party filesystems that use the kernel Cc interface work properly using NEWCC, including matt wu's ext3 driver.
* In contrast with code that tries to make CcInitializeCacheMap and CcUninitializeCacheMap into a pair that supports reference counting, NEWCC lazily initializes the shared and private cache maps as needed and uses the presence of a PrivateCacheMap on at least one file pointing to the SharedCacheMap as an indication that the FILE_OBJECT reference in the SharedCacheMap should still be held. When the last PrivateCacheMap is discarded, that's the appropriate time to tear down caching for a specific file, as the SharedCacheMap data is allowed to be saved and reused. We honor this by making the SharedCacheMap into a depot for keeping track of the PrivateCacheMap objects associated with views of a file.
svn path=/trunk/; revision=55833
2012-02-23 12:03:06 +00:00
|
|
|
|
2013-02-16 17:37:17 +00:00
|
|
|
DPRINTC("MiMapViewOfSegment(P %p, A %p, T %x)\n",
|
2012-03-28 18:39:21 +00:00
|
|
|
MmGetAddressSpaceOwner(AddressSpace),
|
|
|
|
|
*BaseAddress,
|
|
|
|
|
MArea->Type);
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
|
2012-03-28 18:39:21 +00:00
|
|
|
return STATUS_SUCCESS;
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
}
|
2020-10-23 14:16:51 +00:00
|
|
|
#endif
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
|
2012-03-29 04:43:44 +00:00
|
|
|
/*
|
|
|
|
|
|
|
|
|
|
Completely remove the page at FileOffset in Segment. The page must not
|
|
|
|
|
be mapped.
|
|
|
|
|
|
|
|
|
|
*/
|
|
|
|
|
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
VOID
|
|
|
|
|
NTAPI
|
2012-03-28 18:39:21 +00:00
|
|
|
MiFreeSegmentPage(PMM_SECTION_SEGMENT Segment,
|
|
|
|
|
PLARGE_INTEGER FileOffset)
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
{
|
2012-03-28 19:41:40 +00:00
|
|
|
ULONG_PTR Entry;
|
2012-03-28 18:39:21 +00:00
|
|
|
PFILE_OBJECT FileObject = Segment->FileObject;
|
|
|
|
|
|
|
|
|
|
Entry = MmGetPageEntrySectionSegment(Segment, FileOffset);
|
2013-02-16 17:37:17 +00:00
|
|
|
DPRINTC("MiFreeSegmentPage(%p:%I64x -> Entry %Ix\n",
|
2012-03-28 18:39:21 +00:00
|
|
|
Segment,
|
2013-02-16 17:37:17 +00:00
|
|
|
FileOffset->QuadPart,
|
2012-03-28 18:39:21 +00:00
|
|
|
Entry);
|
|
|
|
|
|
|
|
|
|
if (Entry && !IS_SWAP_FROM_SSE(Entry))
|
|
|
|
|
{
|
|
|
|
|
// The segment is carrying a dirty page.
|
|
|
|
|
PFN_NUMBER OldPage = PFN_FROM_SSE(Entry);
|
|
|
|
|
if (IS_DIRTY_SSE(Entry) && FileObject)
|
|
|
|
|
{
|
2013-02-16 17:37:17 +00:00
|
|
|
DPRINT("MiWriteBackPage(%p,%wZ,%I64x)\n",
|
2012-03-28 18:39:21 +00:00
|
|
|
Segment,
|
|
|
|
|
&FileObject->FileName,
|
2013-02-16 17:37:17 +00:00
|
|
|
FileOffset->QuadPart);
|
2012-03-28 18:39:21 +00:00
|
|
|
|
|
|
|
|
MiWriteBackPage(FileObject, FileOffset, PAGE_SIZE, OldPage);
|
|
|
|
|
}
|
2013-02-16 17:37:17 +00:00
|
|
|
DPRINTC("Free page %Ix (off %I64x from %p) (ref ct %lu, ent %Ix, dirty? %s)\n",
|
2012-03-28 18:39:21 +00:00
|
|
|
OldPage,
|
2013-02-16 17:37:17 +00:00
|
|
|
FileOffset->QuadPart,
|
2012-03-28 18:39:21 +00:00
|
|
|
Segment,
|
2021-05-20 09:48:02 +00:00
|
|
|
MmGetReferenceCountPageWithoutLock(OldPage),
|
2012-03-28 18:39:21 +00:00
|
|
|
Entry,
|
|
|
|
|
IS_DIRTY_SSE(Entry) ? "true" : "false");
|
|
|
|
|
|
|
|
|
|
MmSetPageEntrySectionSegment(Segment, FileOffset, 0);
|
|
|
|
|
MmReleasePageMemoryConsumer(MC_CACHE, OldPage);
|
|
|
|
|
}
|
|
|
|
|
else if (IS_SWAP_FROM_SSE(Entry))
|
|
|
|
|
{
|
|
|
|
|
DPRINT("Free swap\n");
|
|
|
|
|
MmFreeSwapPage(SWAPENTRY_FROM_SSE(Entry));
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
DPRINT("Done\n");
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
VOID
|
2012-03-28 18:39:21 +00:00
|
|
|
MmFreeCacheSectionPage(PVOID Context,
|
|
|
|
|
MEMORY_AREA* MemoryArea,
|
|
|
|
|
PVOID Address,
|
|
|
|
|
PFN_NUMBER Page,
|
|
|
|
|
SWAPENTRY SwapEntry,
|
|
|
|
|
BOOLEAN Dirty)
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
{
|
2012-03-28 19:41:40 +00:00
|
|
|
ULONG_PTR Entry;
|
2012-03-28 18:39:21 +00:00
|
|
|
PVOID *ContextData = Context;
|
|
|
|
|
PMMSUPPORT AddressSpace;
|
|
|
|
|
PEPROCESS Process;
|
|
|
|
|
PMM_SECTION_SEGMENT Segment;
|
|
|
|
|
LARGE_INTEGER Offset;
|
|
|
|
|
|
2013-02-16 17:37:17 +00:00
|
|
|
DPRINT("MmFreeSectionPage(%p,%p,%Ix,%Ix,%u)\n",
|
2012-03-28 18:39:21 +00:00
|
|
|
MmGetAddressSpaceOwner(ContextData[0]),
|
|
|
|
|
Address,
|
|
|
|
|
Page,
|
|
|
|
|
SwapEntry,
|
|
|
|
|
Dirty);
|
|
|
|
|
|
|
|
|
|
AddressSpace = ContextData[0];
|
|
|
|
|
Process = MmGetAddressSpaceOwner(AddressSpace);
|
|
|
|
|
Address = (PVOID)PAGE_ROUND_DOWN(Address);
|
|
|
|
|
Segment = ContextData[1];
|
2015-05-16 20:10:03 +00:00
|
|
|
Offset.QuadPart = (ULONG_PTR)Address - MA_GetStartingAddress(MemoryArea) +
|
2012-03-28 18:39:21 +00:00
|
|
|
MemoryArea->Data.SectionData.ViewOffset.QuadPart;
|
|
|
|
|
|
|
|
|
|
Entry = MmGetPageEntrySectionSegment(Segment, &Offset);
|
|
|
|
|
|
|
|
|
|
if (Page != 0 && PFN_FROM_SSE(Entry) == Page && Dirty)
|
|
|
|
|
{
|
2013-02-16 17:37:17 +00:00
|
|
|
DPRINT("Freeing section page %p:%I64x -> %Ix\n", Segment, Offset.QuadPart, Entry);
|
2012-03-28 18:39:21 +00:00
|
|
|
MmSetPageEntrySectionSegment(Segment, &Offset, DIRTY_SSE(Entry));
|
|
|
|
|
}
|
|
|
|
|
if (Page)
|
|
|
|
|
{
|
2013-02-16 17:37:17 +00:00
|
|
|
DPRINT("Removing page %p:%I64x -> %x\n", Segment, Offset.QuadPart, Entry);
|
2012-03-28 18:39:21 +00:00
|
|
|
MmSetSavedSwapEntryPage(Page, 0);
|
|
|
|
|
MmDeleteRmap(Page, Process, Address);
|
2014-10-05 07:16:01 +00:00
|
|
|
MmDeleteVirtualMapping(Process, Address, NULL, NULL);
|
2012-03-28 18:39:21 +00:00
|
|
|
MmReleasePageMemoryConsumer(MC_CACHE, Page);
|
|
|
|
|
}
|
|
|
|
|
if (SwapEntry != 0)
|
|
|
|
|
{
|
|
|
|
|
MmFreeSwapPage(SwapEntry);
|
|
|
|
|
}
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
}
|
|
|
|
|
|
2020-10-23 14:16:51 +00:00
|
|
|
#ifdef NEWCC
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
NTSTATUS
|
|
|
|
|
NTAPI
|
2012-03-28 18:39:21 +00:00
|
|
|
MmUnmapViewOfCacheSegment(PMMSUPPORT AddressSpace,
|
|
|
|
|
PVOID BaseAddress)
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
{
|
2012-03-28 18:39:21 +00:00
|
|
|
PVOID Context[2];
|
|
|
|
|
PMEMORY_AREA MemoryArea;
|
|
|
|
|
PMM_SECTION_SEGMENT Segment;
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
|
2012-03-28 18:39:21 +00:00
|
|
|
MemoryArea = MmLocateMemoryAreaByAddress(AddressSpace, BaseAddress);
|
|
|
|
|
if (MemoryArea == NULL || MemoryArea->DeleteInProgress)
|
|
|
|
|
{
|
|
|
|
|
ASSERT(MemoryArea);
|
|
|
|
|
return STATUS_UNSUCCESSFUL;
|
|
|
|
|
}
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
|
2012-03-28 18:39:21 +00:00
|
|
|
MemoryArea->DeleteInProgress = TRUE;
|
|
|
|
|
Segment = MemoryArea->Data.SectionData.Segment;
|
|
|
|
|
MemoryArea->Data.SectionData.Segment = NULL;
|
|
|
|
|
|
|
|
|
|
MmLockSectionSegment(Segment);
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
|
2012-03-28 18:39:21 +00:00
|
|
|
Context[0] = AddressSpace;
|
|
|
|
|
Context[1] = Segment;
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
|
2013-02-16 17:37:17 +00:00
|
|
|
DPRINT("MmFreeMemoryArea(%p,%p)\n",
|
2012-03-28 18:39:21 +00:00
|
|
|
MmGetAddressSpaceOwner(AddressSpace),
|
2015-05-16 20:10:03 +00:00
|
|
|
MA_GetStartingAddress(MemoryArea));
|
2012-03-28 18:39:21 +00:00
|
|
|
|
2013-06-02 19:12:22 +00:00
|
|
|
MmLockAddressSpace(AddressSpace);
|
|
|
|
|
|
2012-03-28 18:39:21 +00:00
|
|
|
MmFreeMemoryArea(AddressSpace, MemoryArea, MmFreeCacheSectionPage, Context);
|
|
|
|
|
|
2013-06-02 19:12:22 +00:00
|
|
|
MmUnlockAddressSpace(AddressSpace);
|
|
|
|
|
|
2012-03-28 18:39:21 +00:00
|
|
|
MmUnlockSectionSegment(Segment);
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
|
2013-02-16 17:37:17 +00:00
|
|
|
DPRINTC("MiUnmapViewOfSegment %p %p %p\n",
|
2012-03-28 18:39:21 +00:00
|
|
|
MmGetAddressSpaceOwner(AddressSpace),
|
|
|
|
|
BaseAddress,
|
|
|
|
|
Segment);
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
|
2012-03-28 18:39:21 +00:00
|
|
|
return STATUS_SUCCESS;
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
NTSTATUS
|
|
|
|
|
NTAPI
|
2012-03-28 18:39:21 +00:00
|
|
|
MmExtendCacheSection(PROS_SECTION_OBJECT Section,
|
|
|
|
|
PLARGE_INTEGER NewSize,
|
|
|
|
|
BOOLEAN ExtendFile)
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
{
|
2012-03-28 18:39:21 +00:00
|
|
|
LARGE_INTEGER OldSize;
|
|
|
|
|
PMM_SECTION_SEGMENT Segment = Section->Segment;
|
2013-02-16 17:37:17 +00:00
|
|
|
DPRINT("Extend Segment %p\n", Segment);
|
2012-03-28 18:39:21 +00:00
|
|
|
|
|
|
|
|
MmLockSectionSegment(Segment);
|
|
|
|
|
OldSize.QuadPart = Segment->RawLength.QuadPart;
|
|
|
|
|
MmUnlockSectionSegment(Segment);
|
|
|
|
|
|
2013-02-16 17:37:17 +00:00
|
|
|
DPRINT("OldSize 0x%I64x NewSize 0x%I64x\n",
|
|
|
|
|
OldSize.QuadPart,
|
|
|
|
|
NewSize->QuadPart);
|
2012-03-28 18:39:21 +00:00
|
|
|
|
|
|
|
|
if (ExtendFile && OldSize.QuadPart < NewSize->QuadPart)
|
|
|
|
|
{
|
|
|
|
|
NTSTATUS Status;
|
|
|
|
|
|
|
|
|
|
Status = IoSetInformation(Segment->FileObject,
|
|
|
|
|
FileEndOfFileInformation,
|
|
|
|
|
sizeof(LARGE_INTEGER),
|
|
|
|
|
NewSize);
|
|
|
|
|
|
|
|
|
|
if (!NT_SUCCESS(Status)) return Status;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
MmLockSectionSegment(Segment);
|
|
|
|
|
Segment->RawLength.QuadPart = NewSize->QuadPart;
|
|
|
|
|
Segment->Length.QuadPart = MAX(Segment->Length.QuadPart,
|
2012-05-04 11:32:07 +00:00
|
|
|
(LONG64)PAGE_ROUND_UP(Segment->RawLength.QuadPart));
|
2012-03-28 18:39:21 +00:00
|
|
|
MmUnlockSectionSegment(Segment);
|
|
|
|
|
return STATUS_SUCCESS;
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
}
|
|
|
|
|
|
2012-03-28 18:39:21 +00:00
|
|
|
NTSTATUS
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
NTAPI
|
2012-03-28 18:39:21 +00:00
|
|
|
MmMapCacheViewInSystemSpaceAtOffset(IN PMM_SECTION_SEGMENT Segment,
|
|
|
|
|
OUT PVOID *MappedBase,
|
|
|
|
|
PLARGE_INTEGER FileOffset,
|
|
|
|
|
IN OUT PULONG ViewSize)
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
{
|
|
|
|
|
PMMSUPPORT AddressSpace;
|
|
|
|
|
NTSTATUS Status;
|
2012-03-28 18:39:21 +00:00
|
|
|
|
2013-02-16 17:37:17 +00:00
|
|
|
DPRINT("MmMapViewInSystemSpaceAtOffset() called offset 0x%I64x\n",
|
|
|
|
|
FileOffset->QuadPart);
|
2012-03-28 18:39:21 +00:00
|
|
|
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
AddressSpace = MmGetKernelAddressSpace();
|
2012-03-28 18:39:21 +00:00
|
|
|
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
MmLockAddressSpace(AddressSpace);
|
[NEWCC]
A reintegration checkpoint for the NewCC branch, brought to you by Team NewCC.
Differences with current ReactOS trunk:
* A new memory area type, MEMORY_AREA_CACHE, is added, which represents a mapped region of a file. In NEWCC mode, user sections are MEMORY_AREA_CACHE type as well, and obey new semantics. In non-NEWCC mode, they aren't used.
* A way of claiming a page entry for a specific thread's work is added. Placing the special SWAPENTRY value MM_WAIT_ENTRY in a page table, or in a section page table should indicate that memory management code is intended to wait for another thread to make some status change before checking the state of the page entry again. In code that uses this convention, a return value of STATUS_SUCCESS + 1 is used to indicate that the caller should use the MiWaitForPageEvent macro to wait until somebody has change the state of a wait entry before checking again. This is a lighter weight mechanism than PAGEOPs.
* A way of asking the caller to perform some blocking operation without locks held is provided. This replaces some spaghettified code in which locks are repeatedly taken and broken by code that performs various blocking operations. Using this mechanism, it is possible to do a small amount of non-blocking work, fill in a request, then return STATUS_MORE_PROCESSING_REQUIRED to request that locks be dropped and the blocking operation be carried out. A MM_REQUIRED_RESOURCES structure is provided to consumers of this contract to use to accumulate state across many blocking operations. Several functions wrapping blocking operations are provided in ntoskrnl/cache/reqtools.c.
* Image section pages are no longer direct mapped. This is done to simplify consolidation of ownership of pages under the data section system. At a later time, it may be possible to make data pages directly available to image sections for the same file. This is likely the only direct performance impact this code makes on non-NEWCC mode.
RMAPs:
* A new type of RMAP entry is introduced, distinguished by RMAP_IS_SEGMENT(Address) of the rmap entry. This kind of entry contains a pointer to a section page table node in the Process pointer, which in turn links back to the MM_SECTION_SEGMENT it belongs to. Therefore, a page belonging only to a segment (that is, a segment page that isn't mapped) can exist and be evicted using the normal page eviction mechanism in balance.c. Each of the rmap function has been modified to deal with segment rmaps.
* The low 8 bits of the Address field in a segment rmap denote the entry number in the generic table node pointed to by Process that points to the page the rmap belongs to. By combining them, you can determine the file offset the page belongs to.
* In NEWCC mode, MmSharePageEntry/UnsharePageEntry are not used, and instead the page reference count is used to keep track of the number of mappings of a page, allowing the last reference expiring to allow the page to be recycled without much intervention. These are still used in non-NEWCC mode. One change has been made, the count fields have been narrowed by 1 bit to make room for a dirty bit in SSE entries, needed when a page is present but unmapped.
Section page tables:
* The section page tables are now implemented using RtlGenericTables. This enables a fairly compact representation of section page tables without having the existence of a section object imply 4k of fake PDEs. In addition, each node in the generic table has a wide file offset that is a multiple of 256 pages, or 1 megabyte total. Besides needing wide file offsets, the only other visible change caused by the switch to generic tables for section page tables is the need to lock the section segment before interacting with the section page table.
Eviction:
* Page eviction in cache sections is accomplished by MmpPageOutPhysicalAddress. In the case of a shared page, it tries to remove all mappings of the indicated page. If this process fails at any point, the page will simply be drawn back into the target address spaces. After succeeding at this, if TRUE has been accumulated into the page's dirty bit in the section page table, it is written back, and then permanently removed.
NewCC mode:
* NEWCC mode is introduced, which rewrites the file cache to a set of cache stripes actively mapped, along with unmapped section data.
* NewCC is more authentic in its interpretation of the external interface to the windows cache than the current cache manager, implementing each of the cache manager functions according to the documented interface with no preconceived ideas about how anything should be implemented internally. Cache stripes are implemented on top of section objects, using the same memory manager paths, and therefore economizing code and complexity. This replaces a rather complicated system in which pages can be owned by the cache manager and the memory manager simultaneously and they must cooperate in a fairly sophisticated way to manage them. Since they're quite interdependent in the current code, modifying either is very difficult. In NEWCC, they have a clear division of labor and thus can be worked on independently.
* Several third party filesystems that use the kernel Cc interface work properly using NEWCC, including matt wu's ext3 driver.
* In contrast with code that tries to make CcInitializeCacheMap and CcUninitializeCacheMap into a pair that supports reference counting, NEWCC lazily initializes the shared and private cache maps as needed and uses the presence of a PrivateCacheMap on at least one file pointing to the SharedCacheMap as an indication that the FILE_OBJECT reference in the SharedCacheMap should still be held. When the last PrivateCacheMap is discarded, that's the appropriate time to tear down caching for a specific file, as the SharedCacheMap data is allowed to be saved and reused. We honor this by making the SharedCacheMap into a depot for keeping track of the PrivateCacheMap objects associated with views of a file.
svn path=/trunk/; revision=55833
2012-02-23 12:03:06 +00:00
|
|
|
MmLockSectionSegment(Segment);
|
2012-03-28 18:39:21 +00:00
|
|
|
|
|
|
|
|
Status = MiMapViewOfSegment(AddressSpace,
|
|
|
|
|
Segment,
|
|
|
|
|
MappedBase,
|
|
|
|
|
*ViewSize,
|
|
|
|
|
PAGE_READWRITE,
|
|
|
|
|
FileOffset,
|
|
|
|
|
0);
|
|
|
|
|
|
[NEWCC]
A reintegration checkpoint for the NewCC branch, brought to you by Team NewCC.
Differences with current ReactOS trunk:
* A new memory area type, MEMORY_AREA_CACHE, is added, which represents a mapped region of a file. In NEWCC mode, user sections are MEMORY_AREA_CACHE type as well, and obey new semantics. In non-NEWCC mode, they aren't used.
* A way of claiming a page entry for a specific thread's work is added. Placing the special SWAPENTRY value MM_WAIT_ENTRY in a page table, or in a section page table should indicate that memory management code is intended to wait for another thread to make some status change before checking the state of the page entry again. In code that uses this convention, a return value of STATUS_SUCCESS + 1 is used to indicate that the caller should use the MiWaitForPageEvent macro to wait until somebody has change the state of a wait entry before checking again. This is a lighter weight mechanism than PAGEOPs.
* A way of asking the caller to perform some blocking operation without locks held is provided. This replaces some spaghettified code in which locks are repeatedly taken and broken by code that performs various blocking operations. Using this mechanism, it is possible to do a small amount of non-blocking work, fill in a request, then return STATUS_MORE_PROCESSING_REQUIRED to request that locks be dropped and the blocking operation be carried out. A MM_REQUIRED_RESOURCES structure is provided to consumers of this contract to use to accumulate state across many blocking operations. Several functions wrapping blocking operations are provided in ntoskrnl/cache/reqtools.c.
* Image section pages are no longer direct mapped. This is done to simplify consolidation of ownership of pages under the data section system. At a later time, it may be possible to make data pages directly available to image sections for the same file. This is likely the only direct performance impact this code makes on non-NEWCC mode.
RMAPs:
* A new type of RMAP entry is introduced, distinguished by RMAP_IS_SEGMENT(Address) of the rmap entry. This kind of entry contains a pointer to a section page table node in the Process pointer, which in turn links back to the MM_SECTION_SEGMENT it belongs to. Therefore, a page belonging only to a segment (that is, a segment page that isn't mapped) can exist and be evicted using the normal page eviction mechanism in balance.c. Each of the rmap function has been modified to deal with segment rmaps.
* The low 8 bits of the Address field in a segment rmap denote the entry number in the generic table node pointed to by Process that points to the page the rmap belongs to. By combining them, you can determine the file offset the page belongs to.
* In NEWCC mode, MmSharePageEntry/UnsharePageEntry are not used, and instead the page reference count is used to keep track of the number of mappings of a page, allowing the last reference expiring to allow the page to be recycled without much intervention. These are still used in non-NEWCC mode. One change has been made, the count fields have been narrowed by 1 bit to make room for a dirty bit in SSE entries, needed when a page is present but unmapped.
Section page tables:
* The section page tables are now implemented using RtlGenericTables. This enables a fairly compact representation of section page tables without having the existence of a section object imply 4k of fake PDEs. In addition, each node in the generic table has a wide file offset that is a multiple of 256 pages, or 1 megabyte total. Besides needing wide file offsets, the only other visible change caused by the switch to generic tables for section page tables is the need to lock the section segment before interacting with the section page table.
Eviction:
* Page eviction in cache sections is accomplished by MmpPageOutPhysicalAddress. In the case of a shared page, it tries to remove all mappings of the indicated page. If this process fails at any point, the page will simply be drawn back into the target address spaces. After succeeding at this, if TRUE has been accumulated into the page's dirty bit in the section page table, it is written back, and then permanently removed.
NewCC mode:
* NEWCC mode is introduced, which rewrites the file cache to a set of cache stripes actively mapped, along with unmapped section data.
* NewCC is more authentic in its interpretation of the external interface to the windows cache than the current cache manager, implementing each of the cache manager functions according to the documented interface with no preconceived ideas about how anything should be implemented internally. Cache stripes are implemented on top of section objects, using the same memory manager paths, and therefore economizing code and complexity. This replaces a rather complicated system in which pages can be owned by the cache manager and the memory manager simultaneously and they must cooperate in a fairly sophisticated way to manage them. Since they're quite interdependent in the current code, modifying either is very difficult. In NEWCC, they have a clear division of labor and thus can be worked on independently.
* Several third party filesystems that use the kernel Cc interface work properly using NEWCC, including matt wu's ext3 driver.
* In contrast with code that tries to make CcInitializeCacheMap and CcUninitializeCacheMap into a pair that supports reference counting, NEWCC lazily initializes the shared and private cache maps as needed and uses the presence of a PrivateCacheMap on at least one file pointing to the SharedCacheMap as an indication that the FILE_OBJECT reference in the SharedCacheMap should still be held. When the last PrivateCacheMap is discarded, that's the appropriate time to tear down caching for a specific file, as the SharedCacheMap data is allowed to be saved and reused. We honor this by making the SharedCacheMap into a depot for keeping track of the PrivateCacheMap objects associated with views of a file.
svn path=/trunk/; revision=55833
2012-02-23 12:03:06 +00:00
|
|
|
MmUnlockSectionSegment(Segment);
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
MmUnlockAddressSpace(AddressSpace);
|
2012-03-28 18:39:21 +00:00
|
|
|
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
return Status;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* @implemented
|
|
|
|
|
*/
|
|
|
|
|
NTSTATUS NTAPI
|
|
|
|
|
MmUnmapCacheViewInSystemSpace (IN PVOID MappedBase)
|
|
|
|
|
{
|
2012-03-28 18:39:21 +00:00
|
|
|
PMMSUPPORT AddressSpace;
|
|
|
|
|
NTSTATUS Status;
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
|
2012-03-28 18:39:21 +00:00
|
|
|
DPRINT("MmUnmapViewInSystemSpace() called\n");
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
|
2012-03-28 18:39:21 +00:00
|
|
|
AddressSpace = MmGetKernelAddressSpace();
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
|
2012-03-28 18:39:21 +00:00
|
|
|
Status = MmUnmapViewOfCacheSegment(AddressSpace, MappedBase);
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
|
2012-03-28 18:39:21 +00:00
|
|
|
return Status;
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
}
|
2020-10-23 14:16:51 +00:00
|
|
|
#endif /* NEWCC */
|
[CACHE]
The cache manager rewrite I started years ago has finally appeared in
ReactOS' trunk and although at this point it's not quite perfectly
integrated, it's enough to boot up the bootcd or livecd. To check out
the more mature original, check out arty-newcc-reactos, branch
arty-newcc on bitbucket.org . Amine Khaldi encouraged me quite a bit
to not give up on it, and was able to reach out and be an advocate
when i really wasn't able to. Others agree that the time has come to
begin removing the old cache manager. I expect the remaining problems
in the version going to trunk will be taken care of relatively
quickly.
The motivation for this effort lies in the particularly hairy
relationship between ReactOS' cache manager and data sections. This
code completely removes page sharing between cache manager and section
and reimagines cache manager as being a facility layered on the memory
manager, not really caring about individual pages, but simply managing
data section objects where caching might occur.
It took me about 2 years to do the first pass of this rewrite and most
of this year to fix some lingering issues, properly implement demand
paging in ReactOS (code which didn't come with this patch in a
recognizable form), and finish getting the PrivateCacheMap and
SharedCacheMap relationship correct.
Currently, the new ntoskrnl/cache directory contains an own
implementation of data file sections. After things have settled down,
we can begin to deprecate and remove the parts of ReactOS' section
implementation that depend on a close relationship with cache
manager. Eventually, I think that the extra code added to
ntoskrnl/cache/section will be removed and ReactOS' own sections will
replace the use of the special MM_CACHE_SECTION_SEGMENT in the cache
path.
Note also, that this makes all cache manager (and new section parts)
use wide file offsets. If my section code were to take over other
parts of the ReactOS memory manager, they would also benefit from
these improvements.
I invite anyone who wants to to peek at this code and fix whatever
bugs can be found.
svn path=/trunk/; revision=49423
2010-11-02 02:32:39 +00:00
|
|
|
|
|
|
|
|
/* EOF */
|