Commit graph

109 commits

Author SHA1 Message Date
Timo Kreuzer bf95b7e8e5 [NTOS:CC] Do not access VACB after decrementing it's reference count
It might already be deleted by a different thread.
2023-11-17 20:07:55 +02:00
Doug Lyons 2b14056600
[NTOS:CC][NTOS:MM] Add back CcRosTrimCache and add Delay for MM to work. (#5630)
MM/CC Add back CcRosTrimCache as suggested by Thomas Faber which was removed in 0.4.15-dev-1717-g 	d8cdb89fb0
and call it once in a while also during read-operations.

fixes JIRA issue: CORE-17624 'Cannot copy files > RAMsize anymore using TotalCommander'


1st testbot results on top of 0.4.15-dev-6526-g8d35887
VBox: https://reactos.org/testman/compare.php?ids=89111,89120 (additional random reboot in winhttp:winhttp)
KVM: https://reactos.org/testman/compare.php?ids=89110,89119
We do assume that reboot to be unrelated.

2nd testbot results on top of 0.4.15-dev-6526-g8d35887
VBox: https://reactos.org/testman/compare.php?ids=89111,89232
KVM: https://reactos.org/testman/compare.php?ids=89110,89233
2023-09-06 13:34:25 +02:00
Doug Lyons e6ca3952e0
[NTOS:CC] Fix locking of CcRosVacbIncRefCount in CcRosCreateVacb (#5527)
CORE-17624
2023-08-12 19:10:34 +02:00
Jérôme Gardou 69daff72d2 NTOS:CC Free some VACBs when we're under memory pressure 2023-07-29 14:00:44 +03:00
Hermès Bélusca-Maïto 579eab8a31
[NTOS] Include kdbg/kdb.h only in the files that really need it. 2023-04-11 00:44:10 +02:00
Serge Gautherie c2e454b439 [NTOS:CC] CcRosFlushVacb(): Fix Iosb annotation
Addendum to 2ba1926.
2021-06-22 10:33:45 +02:00
Timo Kreuzer dd08ae2c0f [NTOS:CC] Fix use of unintialized variable (caught by RTC1) 2021-05-24 22:00:11 +02:00
Jérôme Gardou 0187c1e113 [NTOS:MM] Fix PFN tracing 2021-03-30 16:26:43 +02:00
Jérôme Gardou 2a962eaf8c [NTOS:CC] Keep a reference on the shared cache map of the file when we are in lazy write
This should fix "Leaking VACB" debug prints
2021-02-19 15:48:31 +01:00
Jérôme Gardou cc9607e94e [NTOS:CC] Fix use of uninitialized variable 2021-02-03 13:35:17 +01:00
Jérôme Gardou b7eb0fddf3 Address PR review 2021-02-03 09:41:24 +01:00
Jérôme Gardou 2ba1926037 [NTOS:MM][NTOS:CC] Performance improvement again
Read files by 64kb chunks instead of page-sized chunks.
2021-02-03 09:41:23 +01:00
Jérôme Gardou d0bf98663b [NTOS:CC] Be sure to flush the whole file in CcFlushCache 2021-02-03 09:41:23 +01:00
Jérôme Gardou 5949d5095d [NTOS:CC][NTOS:MM] Try respecting ValidDataLength 2021-02-03 09:41:23 +01:00
Jérôme Gardou bdb73edab7 [NTOS:CC] Flush the whole VACB
Let Mm know what it has to do.
2021-02-03 09:41:23 +01:00
Jérôme Gardou 804f5a41ed [NTOS:CC] Improve trace messages 2021-02-03 09:41:23 +01:00
Jérôme Gardou 20fe42c9e9 [NTOS:CC] Simplify CcFlushCache implementation 2021-02-03 09:41:23 +01:00
Jérôme Gardou 9b6240ee03 [NTOS:CC] Get rid of ROS_VACB:Valid 2021-02-03 09:41:23 +01:00
Jérôme Gardou 33cde28312 [NTOS:CC] Simplify CcRosDeleteFileCache 2021-02-03 09:41:23 +01:00
Jérôme Gardou a9193b5cc2 [NTOS:CC] Remove dead code 2021-02-03 09:41:23 +01:00
Jérôme Gardou 57ee31ee33 [NTOS:CC] Perform sanity checks before doing anything else 2021-02-03 09:41:23 +01:00
Jérôme Gardou 8287a098b9 [NTOS:CC] Fix potnetial use-after-free 2021-02-03 09:41:23 +01:00
Jérôme Gardou 1505abbc09 [NTOS:CC] Do not write behind concurrently the same file 2021-02-03 09:41:23 +01:00
Jérôme Gardou 347a4f146b [NTOS] Loop again and again until the whole cache is empty when sutting down 2021-02-03 09:41:23 +01:00
Jérôme Gardou 6d97d8d2e1 [NTOS:CC] Fix some tests, complain where the current implementation won't let us do the right thing 2021-02-03 09:41:22 +01:00
Jérôme Gardou 30f71c7fc0 [NTOS] Zero data unconditionally after segment end, unless section is created with SEC_RESERVE
Use a SEC_RESERVE section in Cc
2021-02-03 09:41:22 +01:00
Jérôme Gardou f8aa14ce4e [NTOS:CC] Acquire file for flush when flushing if necessary 2021-02-03 09:41:22 +01:00
Jérôme Gardou 70c62aa2c9 [NTOS:CC] Fix Vacb size usage & check 2021-02-03 09:41:22 +01:00
Jérôme Gardou c74cbf0c0b [NTOS/CC] Be more precise when notifying Mm about dirty pages 2021-02-03 09:41:22 +01:00
Jérôme Gardou 36e18aab35 [NTOS:CC] Remove unused functions 2021-02-03 09:41:22 +01:00
Jérôme Gardou d8cdb89fb0 [NTOSKRNL] Overhaul Cc and Mm relationship
Previously, when creating a file section, Mm requested Cc to cache the file, then Cc would request pages from Mm, then Mm would request them back to serve its file-mapping role
Now, Mm does it all by itself. If file cahcing is requested by the FS driver, then Cc creates a file mapping and uses that to serve its purpose.

This is a rewrite of Cc
2021-02-03 09:41:22 +01:00
Jérôme Gardou 8631e75837 [NTOS:CC] Acquire the master lock after freeing the VACB in CcRosFlushDirtyPages
Fixes a random ASSERT
2020-12-09 18:06:42 +01:00
Victor Perevertkin 5c7ce4475e
[REACTOS] Cleanup INIT and some PAGE section allocations
- Change INIT_FUNCTION and INIT_SECTION to CODE_SEG("INIT") and DATA_SEG("INIT") respectively
- Remove INIT_FUNCTION from function prototypes
- Remove alloc_text pragma calls as they are not needed anymore
2020-11-02 21:45:31 +03:00
Jérôme Gardou 1c528cbf84 Revert "[NTOS/MM]
- Fix PFNs tracing
     - Add private pages to the process working set"

This reverts commit 4c5351bf55.
Not ready for prime time
2020-10-20 15:56:21 +02:00
Jérôme Gardou 4c5351bf55 [NTOS/MM]
- Fix PFNs tracing
 - Add private pages to the process working set
2020-10-20 15:20:59 +02:00
Serge Gautherie b20f815126
[NTOSKRNL] Place INIT_FUNCTION before the return type (#2823)
(but after 'static' or SAL 2 annotation.)
Follow-up to 71fefa32, which mentions that it's actually required by the compiler in some circumstances.
2020-05-23 15:56:10 +02:00
Pierre Schweitzer 77b6899d89
[NTOSKRNL] Don't set VACB dirty on release if already dirty
CORE-15954
2019-04-20 11:23:35 +02:00
Pierre Schweitzer 1a267045f8
[NTOSKRNL] Honor files that shouldn't be lazy written 2018-12-23 12:10:58 +01:00
Pierre Schweitzer 1435ff95b4
[NTOSKRNL] Don't call AcquireForLazyWrite with the master lock held
This incorrect behavior was leading to a call at too high IRQL for paged code.
This was triggered by MS FastFAT.

ReleaseFromLazyWrite call was already correctly called to that regard.

CORE-11819
2018-12-21 08:46:40 +01:00
Pierre Schweitzer 4f8b041bf0
[NTOSKRNL] Drop the ViewLock mutex in favour of the master spin lock
This will allow Cc calls during DPC, which is required by MS FastFAT

CORE-11819
2018-12-19 22:51:45 +01:00
Pierre Schweitzer 182cc5c5ab
[NTOSKRNL] Don't dereference VACB when allocating its memory area fails
This avoids performing a double-free (even though that's hidden by the
fact we use lookaside allocations for VACB), and it avoids freeing
a memory address at an uninitialized address.
We don't care about references here, the VACB was just allocated, never
linked and we're its only user.

CORE-15413
2018-12-08 19:56:03 +01:00
Pierre Schweitzer f284947622
[NTOSKRNL] Move the PinCount out of the VACB to the BCB
Given current ReactOS implementation, a VACB can be pinned
several times, with different BCB. In next commits, a single
BCB will be able to be pinned several times. That would
lead to severe inconsistencies in counting and thus corruption.
2018-10-05 21:26:16 +02:00
Pierre Schweitzer e17f61138c
[NTOSKRNL] When allocating a new BCB, save it in a list
This list is stored in the shared map. Later, this will allow
reusing BCB when appropriate
2018-09-05 22:06:25 +02:00
Thomas Faber 1d398057a3
[NTOS:CC] Access SectionObjectPointers without lock in CcRosInitializeFileCache. CORE-14691
kmtest:NtCreateSection calls CcInitializeCacheMap with a
NULL value for SectionObjectPointers. This will cause an exception when
trying to access it, which in Windows can be handled gracefully.
However accessing it while holding ViewLock means the lock will not be
released, leading to an APC_INDEX_MISMATCH bugcheck.

This solves the problem by allocating SharedCacheMap outside the lock,
then freeing it again under lock if another thread has updated SharedCacheMap
in the mean time. This is also What Windows Does(TM).
2018-06-05 16:24:13 +02:00
Pierre Schweitzer 2cf9a69bce
[NTOSKRNL] Addendum to 8a8cb4d: don't print uninit pointer. 2018-05-23 08:44:43 +02:00
Pierre Schweitzer 8a8cb4d890
[NTOSKRNL] Only consider SharedCacheMap value once ViewLock is acquired.
This avoids a really nasty race condition in our cache controler where
two concurrents could try to initialize cache on the same file.
This had two nasty effects: first shared map was purely leaked and erased
by the second one. And the private cache map, allocated on the first shared
cache map couldn't be freed and was leading to Mm BSOD (free in a middle of
a block).

This was often triggered while building ReactOS on ReactOS (with multi threads).
With that patch, I cannot crash anylonger while building ReactOS.

CORE-14634
2018-05-23 08:41:46 +02:00
Pierre Schweitzer 54c049bd6e
[NTOKSNRL] Always flush dirty VACB.
Recent changes seem to show that it's not
required to be exclusive on VACB to be able
to flush it.

This commit goes with f2c44aa and fixes the
last issues going with copying huge files.
There are no longer BSODs (be it in Mm or Cc).
I could, with 750MB RAM extract a 2GB file from
a 53MB archive and copy a 2,5GB file from a VBox
share to the disk. Note that writes are often
deferred, so if copy works, it's not that fast for now.

Note that it also brings some beloved behavior from
Windows: copy times are totally unreliable now when
writes are deferred. Little remaining times when
actively copying, high remaining times when deferred
writes in action. And goes between both... Sorry! ;-)

https://xkcd.com/612/

CORE-9696
CORE-11175
2018-04-30 22:24:30 +02:00
Pierre Schweitzer 74c5d8b6bd
[NTOSKRNL] Free unused VACB when required.
Same mechanism exists in Windows (even their Cc
is way different from ours...) where when Cc is
out of memory (in their case, out of VACB), we
will start scavenge old & unused VACB to free
some of the memory.

It's useful in case we're operating we big files
operations, we may run out of memory where to map
VACB for them, so start to scavenge VACB to free
some of that memory.

With this, I am able to install Qt 4.8.6 with 2,5GB of RAM,
scavenging acting when needed!

CORE-12081
CORE-14582
2018-04-30 12:10:24 +02:00
Pierre Schweitzer cc54e51495
[NTOSKRNL] Unmark dirty first, and then write.
This will avoid trying to flush twice a dirty VACB under
high IOs pressure.

CORE-14584
2018-04-30 10:36:19 +02:00
Pierre Schweitzer f2c44aa483
[NTOSKRNL] Fix lazy writer for in-use VACB.
Adjusting refcount and enabling lazy-write for pinned
VACB makes it actually more efficient, often purging
data to disk, reducing memory stress for the system.

This is required for defering writes.

This commit unfortunately (?) reverts a previous revert.

CORE-12081
CORE-14582
CORE-14313
2018-04-29 20:42:53 +02:00