This is a PoC of what it's possible to realize thanks to an
already existing hack in ntoskrnl :-).
With this extension, on the kdb:> prompt, you're able to type
in commands that will be handled by the FastFAT driver and not
by the kernel, allowing internal debug, not possible otherwise.
So far, three commands exist:
- ?fat.vols: lists all the mounted volumes by FastFAT
- ?fat.files: lists all the files on a specific volume (with their attributes)
- ?fat.setdbgfile: allows watching on specifics files lifetime
This is obviously only the begin and could be greatly improved.
For instance, this is what allowed to debug CORE-14557
Once a directory is crossed (opened or a child is opened), associated
FCB structure is created in FastFAT, but also a stream FO for caching.
Up to now, due to an extra reference taken by the stream file object,
even when the directory was no longer used, the directory was kept in
memory: the FCB was never deleted, the file object was never dereferenced,
and the cache never released.
The immediate effect of this bug is that our FAT driver was leaking every
directory that was used affecting the whole OS situation. In case of
directories intensive operation (like extraction the ReactOS source code
in ReactOS ;-)), we were just killin the whole OS RAM without any way to
release it and recover.
The other side effects: IOs were faster as half of the FS was always
permanant in RAM.
This commit fixes the issue by forcing the FSD to release the FO,
and the cache when a directory is no longer used, leading to its
destruction in RAM.
Downside: on IO intensive operation, expect slowdowns, obviously,
there's less caching now. But more efficient!
CORE-14557
This makes the function return the error instead of continuing and
performing use-after-frees operations.
This is likely to be a forgotten return!
CID 1434254, 1434268
is set before locking is actually attempted for Cc worker thread (lazy write/read ahead).
So in case locking fails, the top level IRP is set and never unset, and latter
call will hit the assert where it expects the top level IRP to be NULL.
Should be reported upstream (tm).
CORE-14315
This is assorted with misc fixes to make this code closer to what we currently have in FastFAT.
This also allows fixing a memory leak in case of single entry return.
CORE-13367
leaves volumes in half-(dis)mounted state and thus they get remounted while not
completely dismounted.
This can lead to major breakage and data corruption.
This requires deeper fixes (let's just drop that driver!).
CORE-14124
CORE-14126
CORE-14133
This avoids issues when these partitions are formatted to FAT for setup.
For now, this commit doesn't change anything, but once IopParseDevice hack
gets removed, this will make a difference!
CORE-6305
It fixes assertion failure in vfatDestroyFCB() where we would have release parent before child.
This is still not perfect, but less bug prone...
With this commits (and ENABLE_SWAPOUT defined), ReactOS seems to unmount FAT volumes quite nice! :-)
(Tried with fsutil volume dismount X:)
AddNewMftEntry() - Zero the (larger) buffer for the bitmap prior to reading the bitmap.
BrowseIndexEntries() - Check sub-nodes before checking an index entry. Read and use the index bitmap when checking sub-nodes.
+BrowseSubNodeIndexEntries() - Called for sub-nodes of an index when browsing index entries.
+NtfsDumpData() - Diagnostic function which allows for visualizing a series of bytes.
svn path=/branches/GSoC_2016/NTFS/; revision=75817
+AddBitmap() - adds a $BITMAP attribute to a file record.
+AddIndexAllocation() - adds an $INDEX_ALLOCATION attribute to a file record.
+CountBTreeKeys() - Counts the number of linked B-Tree keys.
CreateIndexBufferFromBTreeNode() - Set INDEX_NODE_LARGE if the node has sub-nodes.
CreateIndexRootFromBTree() - Simplify the usage and math of MaxIndexSize; make it only account for the cumulative size of the index entries.
+DemoteBTreeRoot() - Replaces the contents of an index root with a dummy key, and puts those contents in a new node, which is made a child of the dummy key. This is done when an index root grows too large.
+GetIndexEntryVCN() - Retrieves the VCN from an index entry.
NtfsAddFilenameToDirectory() - Fix math for MaxIndexRootSize.
NtfsInsertKey() - Add support for splitting a B-Tree node. Don't check size of index root (that will be handled later).
+SplitBTreeNode() - Called when a B-Tree node grows too large.
UpdateIndexAllocation() - Create an $I30 index allocation attribute and bitmap attribute if needed.
UpdateIndexNode() - Update children before updating the current node. Store VCN of child nodes in the index entries of their respective keys.
svn path=/branches/GSoC_2016/NTFS/; revision=75707
IncreaseMftSize() - Add some fixes. Write blank records to newly-allocated mft entries, and update $MFTMirr when finished; these changes are needed for chkdsk. Increase size by 64 records instead of 8.
+UpdateMftMirror() - Backs up the first ~4 master file table entries to the $MFTMirr file.
svn path=/branches/GSoC_2016/NTFS/; revision=75694
+AddIndexRoot() - Creates an $INDEX_ROOT attribute and adds it to a file record.
AddNewMftEntry() - Make sure the buffer used by RtlInitializeBitmap() is ULONG-aligned, and a ULONG-multiple in size, per MSDN.
AllocateIndexNode() - Calculate BytesNeeded correctly. Read $BITMAP attribute before increasing its length, in anticipation of a future commit that will check for a free bit before assigning a new index record to the end of the allocation. Use appropriate Set*AttributeDataLength() function, as $BITMAP can be resident or non-resident.
B_TREE_FILENAME_NODE - Give two members more accurate names: change "ExistsOnDisk" member to "HasValidVCN" and rename "NodeNumber" member "VCN."
+CreateEmptyBTree() - Creates a B-Tree to represent an empty directory (for AddIndexRoot).
+NtfsCreateEmptyFileRecord() - Creates an empty file record in memory, with no attributes.
CreateIndexRootFromBTree() - Fix TotalSizeOfEntries calculation.
+NtfsCreateDirectory() - Creates a file record for an empty directory and adds it to the mft.
svn path=/branches/GSoC_2016/NTFS/; revision=75692
-Don't add a preceding backslash when creating files on root.
-Use NTFS_FILE_NAME_POSIX name type if CaseSensitive option is specified.
-Don't try to create a file when a folder is requested (folder creation is still TODO).
svn path=/branches/GSoC_2016/NTFS/; revision=75671
+AllocateIndexNode() - Allocates a new index record in an index allocation.
+CreateDummyKey() - Creates the final B_TREE_KEY for a B_TREE_FILENAME_NODE. Also creates the associated index entry.
GetSizeOfIndexEntries() - Sums the size of each index entry in every key in a B-Tree node.
+SetIndexEntryVCN() - Sets the VCN of a given IndexEntry.
NtfsInsertKey() - Handle instance when the index root grows too large. If it does, add its contents to a new sub-node, and replace contents with a dummy-key whose child is the new node.
UpdateIndexAllocation() - Update index entry if a key has just been assigned a child allocation.
UpdateIndexNode() - Make sure the node exists on disk, and allocate an index record for it if it doesn't.
svn path=/branches/GSoC_2016/NTFS/; revision=75557
AddRun() - Allow for resizing the size of the data runs when the attribute isn't the last in the file record. Fix some comments.
CreateIndexBufferFromBTreeNode(), CreateIndexRootFromBTree - Fix math of IndexSize when checking if the index buffer is too large.
InternalSetResidentAttributeLength() - Allow changing the length of an attribute in the middle of a file record. Adjust the position of every attribute after the one being resized.
+MoveAttributes() - Moves a block of attributes to a new location in the file Record.
PrintAllVCNs() - Add consideration for an index allocation with a size of 0.
WriteAttribute() - Add optional parameter for a pointer to the file record being written to. If passed a file record, WriteAttribute() will skip reading the file record from disk, and will update the file record in memory before returning. This helps callers that use the file record after writing an attribute to stay in-sync with what's on disk.
svn path=/branches/GSoC_2016/NTFS/; revision=75554
-PrepareAttributeContext() - update to allocate memory for pRecord. Don't assume allocations are succeeding.
-ReleaseAttributeContext() - update to free memory for pRecord.
-InternalSetResidentAttributeLength() - Increase size of AttrContext->pRecord as needed. Update to return an NTSTATUS.
-SetResidentAttributeDataLength() - Fix bug that could occur when migrating resident attributes to non-resident if AttrContext->pRecord is too small for the new attribute.
-AddRun() - Fix a bug by reallocating AttrContext->pRecord if the record needs to be enlarged.
svn path=/branches/GSoC_2016/NTFS/; revision=75493
-CreateBTreeFromIndex() - Fix memory allocation; allocate sizeof(B_TREE_KEY) bytes for the key, not the size of the pointer. Add support for child nodes. Update parameter list.
-CreateIndexRootFromBTree() - Update Header->Flags if any of index root's keys have sub-nodes.
+CreateIndexBufferFromBTreeNode() - Converts a B-Tree node to an index buffer to be written to the index allocation.
+UpdateIndexAllocation() - Updates all of the stale nodes in an index allocation based on a PB_TREE.
+UpdateIndexNode() - Writes a B-Tree node into the index allocation.
+CreateBTreeKeyFromFilename() - Creates a PB_TREE_KEY based on a FILENAME_ATTRIBUTE.
-DestroyBTreeKey() - Destroy a child node, if present.
-DumpBTreeKey() - Dump a child node, if present.
-DumpBTreeNode() - Include Node->KeyCount in output.
-NtfsAddFilenameToDirectory() - Call UpdateIndexAllocation() prior to CreateIndexRootFromBTree().
-Update B_TREE_FILENAME_NODE with members to keep track of its existence on disk.
svn path=/branches/GSoC_2016/NTFS/; revision=75428
+PrintAllVCNs() - Diagnostic function which prints VCN of every node in an index allocation.
+GetAllocationOffsetFromVCN() - Calculates location of an index buffer from the node's VCN.
+GetInfoClassName() - Gets a string representation of an info class enumeration, to speed up development of unimplemented classes.
-NtfsSetInformation() & NtfsQueryInformation() - Use GetInfoClassName to report unhandled information classes.
-CompareTreeKeys() - Add a comment and clarify some comments.
-NtfsAddFilenameToDirectory() - Don't try to update the size of Index Root on disk if the attribute length hasn't changed.
svn path=/branches/GSoC_2016/NTFS/; revision=75424
-Add define for indexed flag for resident attributes, RA_INDEXED.
-CreateBTreeFromIndex() - Don't try to read index entries beyond attribute length.
-Move NtfsAddFileNameToDirectory() from dirctl.c to mft.c (no changes to function).
-SetResidentAttributeDataLength() - Don't try to free AttribData if it hasn't been allocated. Cast IndexRecord to PUCHAR for WriteAttribute(), and cast BitmapData to PCHAR for ReadAttribute() (make gcc happy).
-Replace magic number 16 with a define, NTFS_FILE_FIRST_USER_FILE.
svn path=/branches/GSoC_2016/NTFS/; revision=75371
-Add ATTR_RECORD_ALIGNMENT define to replace magic value of 8 when we need to adjust an attribute's beginning or length.
-Don't use int's.
-GetPackedByteCount() - Remove unused "bytes" variable.
svn path=/branches/GSoC_2016/NTFS/; revision=75288
FindAttribute() - Use * sizeof(WCHAR) in place bit-shifting wizardry.
IncreaseMftSize() - Remove two useless casts. Return error code if WriteAttribute() fails.
SetNonResidentAttributeDataLength() - Assert that AttrContext parameter describes a NonResident attribute. When migrating a resident attribute to non-resident, don't leak memory if FsRtlInitializeLargeMcb() throws an exception, and don't mark the attribute as non-resident until after the Mcb is initialized.
InternalSetResidentAttributeLength() and SetResidentAttributeLength() - Assert that AttrContext parameter describes a resident attribute.
AddNewMftEntry() - Simplify math of backing up and restoring reserved bits by changing BitmapData to PUCHAR type. Don't attempt to modify a volume with more than 2^32 clusters.
AddFixupArray() - Use ULONG in place of int.
svn path=/branches/GSoC_2016/NTFS/; revision=75284
NtfsSetEndOfFile() - Make fileNameAttribute and filename variables uppercase. Don't leak FileRecord if we can't truncate the file. Don't leak memory if there's no FileName attribute.
svn path=/branches/GSoC_2016/NTFS/; revision=75283
-NtfsOpenFile() - Replace an ExFreePool() with ExFreePoolWithTag().
-NtfsCreateFile() - Fix broken cast with BooleanFlagOn() macro.
-NtfsAddFilenameToDirectory() - Remove an extra cast. Return an error if we fail to allocate I30IndexRoot.
-NtfsGetNextPathElement(), NtfsWSubString(), NtfsGetFCBForFile() - Use PCWSTR in place of const PWCHAR or PWCHAR where it makes sense.
svn path=/branches/GSoC_2016/NTFS/; revision=75281
-CompareTreeKeys() - Assert that the first key isn't the dummy key.
-CreateIndexRootFromBTree() - Assert that CurrentKey->IndexEntry->Length isn't 0.
-DumpBTreeKey() - Use sizeof(WCHAR) in place of magic 2.
-NtfsInsertKey() - Check for allocation failure of NewKey.
svn path=/branches/GSoC_2016/NTFS/; revision=75280
-AddFileName() - Don't use dedicated buffer for FileNameNoPath, just point into FileObject->FileName's buffer. Don't use RtlCopyUnicodeString(), just update pointers.
-AddRun() - Fix an SEH2_YIELD with missing return statement.
-FreeClusters() - Use ULONG_MAX in place of 0xffffffff.
-NtfsDumpIndexRootAttribute() - Use consistent UpperCase naming convention. Use BooleanFlagOn() macro where appropriate. Replace magic 8 with sizeof(ULONGLONG).
-GetFileNameAttributeLength() - Add documentation.
svn path=/branches/GSoC_2016/NTFS/; revision=75279
NtfsAddFilenameToDirectory() - Add CaseSensitive parameter. Update to use new B-Tree code: First, the index is read and converted to a B-Tree in memory. Next, a key for the new file is inserted into the tree. Finally, the tree is converted back to an index root attribute which is written to disk.
+btree.c - Includes functions related to B-Trees (AKA B*Trees).
ntfs.h - Added several structures for representing B-Trees in memory.
Known limitations: For simplicity, only trees with a depth of one are currently supported (i.e. an ordered list of filenames). Directories that have or will require an index allocation to store all their filenames are still TODO. As a consequence, the user will only be able to create about 6 files in a directory.
svn path=/branches/GSoC_2016/NTFS/; revision=75223
Improve, add, or fix some DPRINTs. In particular, ULONG's should use %lu, not %u. Also, don't be silent about filesystem corruption.
NtfsFindMftRecord() - move CaseSensitive parameter before output parameter in parameter list.
svn path=/branches/GSoC_2016/NTFS/; revision=75199
Whether or not a filename should be interpreted as case-sensitive is dependent on a flag passed to the driver when a file is created (opened); it's separate from the namespace associated with the file being accessed.
svn path=/branches/GSoC_2016/NTFS/; revision=75178
-AddNewMftEntry() - Increase size of MFT as needed. Fix math for bitmap length. Don't assign file records to MFT indices 0x10 - 0x17; In Windows, these records aren't used unless they have to be, even though they are marked as unused in the bitmap.
+IncreaseMftSize() - Adds room for additional file records in the master file table.
-BrowseIndexEntries() - allow for the rare situation when a non-system file has an MFT index of 0x10.
svn path=/branches/GSoC_2016/NTFS/; revision=75056
-SetAttributeDataLength() has been split into two functions, SetNonResidentAttributeDataLength() and SetResidentAttributeDataLength(). This should improve code readibility and allows for resizing an attribute when there's no FileObject associated with it.
-Added "MftDataOffset" member to DEVICE_EXTENSION, which stores the offset of the Mft's $DATA attribute. (I'm starting to think it's better to add a member for offset to NTFS_ATTR_CONTEXT directly, but I'll save that level of restructuring for a future commit.)
svn path=/branches/GSoC_2016/NTFS/; revision=75055
AddFileName() - Add a parameter to receive the Mft index of the parent directory. Fix so the name of the file will be stored in the attribute, not the name of the directory.
NtfsCreateFile() - Open a file that was successfully created, instead of assuming failure.
NtfsCreateFileRecord() - Add the filename attribute of the created file to the parent directory's index.
+NtfsAddFilenameToDirectory() - Adds a $FILE_NAME attribute to a given directory index. Presently, a file can be created in an empty directory only.
AddNewMftEntry() - Add a parameter to receive the mft index where the new entry was stored.
svn path=/branches/GSoC_2016/NTFS/; revision=74970
[BOOTDATA] - Add a commented-out section to hivesys.inf which can add the required key to enable NTFS write support.
svn path=/branches/GSoC_2016/NTFS/; revision=74685
-Handle IRP_MJ_SET_INFORMATION IRP requests.
+NtfsSetEndOfFile() - Sets the end of file (file size) for a given file.
+NtfsSetInformation() - Sets the specified file information. At this point, only FileEndOfFileInformation is fully implemented. FileAllocationInformation is handled the same way and not truly implemented, but this works well enough for SetEndOfFile().
Overwriting a file in NTFS should now work in the majority of use cases.
svn path=/branches/GSoC_2016/NTFS/; revision=74675
-Modify SetAttributeDataLength() to allow a resident attribute to migrate to non-resident if the attribute grows too large to remain resident.
-Fix values returned by NtfsAllocateClusters() in case of error; return error codes, not 0.
svn path=/branches/GSoC_2016/NTFS/; revision=74524
-Add a new member to the NTFS_ATTR_CONTEXT struct, a LARGE_MCB. This allows an attribute context to describe the cluster mapping of a non-resident file while allowing that mapping to change dynamically, without the context itself needing to be resized. This fixes problems which sometimes arose from resizing files.
-Remove hacky code from NtfsWriteFile() for dealing with "stale" contexts. This fixes that issue.
-Update SetDataAttributeLength(), PrepareAttributeContext(), ReleaseAttributeContext(), FreeClusters(), and AddRun() for the new member.
-Update ReadAttribute() and WriteAttribute() to work with the changed structure. A very-soon-to-come commit will overhaul these functions so they'll operate directly on the LARGE_MCB, instead of converting to and from a packed list of data runs. (Sparse files are broken until then.)
-Rename "RunBufferOffset" to "RunBufferSize" in several places where appropriate.
-Fix, improve, and add some comments.
svn path=/branches/GSoC_2016/NTFS/; revision=74523
+AddStandardInformation(), +AddData(), +AddFileName() - Add attributes to a file record
+NtfsCreateFileRecord() - Creates a new file record and saves it to the MFT.
+AddNewMftEntry() - Adds a file record to the MFT.
NtfsCreateFile() - Modified to create a file record on a file-creation request (file creation is still unsupported; the created file needs to be added to the parent's directory index).
+SetFileRecordEnd() - Establishes a new file record size
UpdateFileRecord() - Improved documentation
InternalSetResidentAttributeLength() - Updated to use SetFileRecordEnd().
svn path=/branches/GSoC_2016/NTFS/; revision=74321
Add some fixes to attrib.c, as suggested by Pierre Schweitzer:
*ConvertLargeMCBToDataRuns() - Use MS portable type, ULONG, for variable i.
*FreeClusters(), AddRun() - Check for invalid parameter before allocating memory, and confirm the memory is allocated.
*ConvertDataRunsToLargeMCB(), AddRun() - Avoid code duplication by using ExRaiseStatus() in try block, and return accurate status via _SEH2_GetExceptionCode().
svn path=/branches/GSoC_2016/NTFS/; revision=72422
Implement AddRun(). Add support functions and documentation.
+ConvertDataRunsToLargeMCB()
+ConvertLargeMCBToDataRuns()
*SetAttributeDataLength(), *NtfsWriteFile() - Update for AddRun() implementation. Add hack to SetAttributeDataLength() to allow notepad.exe to save files until freeing clusters is implemented.
svn path=/branches/GSoC_2016/NTFS/; revision=71942
Add support functions, fix warning from previous commit.
+NtfsDumpDataRuns(), +NtfsDumpDataRunData() - Provide diagnostic output.
+GetPackedByteCount() - Used to encode data runs.
svn path=/branches/GSoC_2016/NTFS/; revision=71921
Add ability to write to resident attributes.
SetAttributeDataLength() - Check if the file is memory mapped before truncating
+InternalSetResidentAttributeLength() - Used by SetAttributeDataLength()
svn path=/branches/GSoC_2016/NTFS/; revision=71820
Allow for an existing file to be opened with FILE_OVERWRITE, FILE_OVERWRITE_IF, or FILE_SUPERSEDE dispositions, and truncate that file. This allows for a file to be opened and saved in Notepad.exe [provided that file is non-resident and its allocation size doesn't need to change].
svn path=/branches/GSoC_2016/NTFS/; revision=71680
Update a file's size in the relevant $FILE_NAME attribute of the index entry in the parent directory.
+UpdateFileNameRecord() - Searches a parent directory for the proper index entry, then updates the file sizes in that entry.
+UpdateIndexEntryFileNameSize() - Recursively searches directory index and applies the size update.
svn path=/branches/GSoC_2016/NTFS/; revision=71664
Fix Up AddFixupArray - It needs to accept a PNTFS_RECORD_HEADER for parameter 2, not a PFILE_RECORD_HEADER.
svn path=/branches/GSoC_2016/NTFS/; revision=71662
When writing to a file, increase the file size if trying to write past the end.
*FindAttribute() has been given an optional pointer to a ULONG that will receive the offset of the found attribute from the beginning of the record. This is to allow for found attributes to be written back into their file records.
+SetAttributeDataLength()
+UpdateFileRecord() - Updates a file record in the master file table at a given index.
+AddFixupArray() - Prepares a file record or directory index for writing to the disk.
svn path=/branches/GSoC_2016/NTFS/; revision=71660
Add fixes to WriteAttribute():
-Remove erroneous check for end of run (before writing to the last run returned)
-Properly dereference RealLengthWritten pointer, as reported in CR-90
svn path=/branches/GSoC_2016/NTFS/; revision=71616
This is likely due to a copy paste error where long name was copied twice and short never.
Fun fact: this was not affecting FATX volumes
Fun fact2: this was defeating a buffer overflow check and thus was allowing buffer overflow!
CORE-14088
This commit fixes weird behavior in our FastFAT implementation. It was mixing two classes:
FileNameInformation and FileNamesInformation. It was handling FileNameInformation like
FileNamesInformation and was filling buffer with FILE_NAMES_INFORMATION structure instead
of FILE_NAME_INFORMATION structure (how many things did that break?!).
Also, it wasn't implementing the FileNamesInformation class at all. This is required by
ntdll_winetest:directory which doesn't expect it to fail and thus, attempts to read
never filled in memory.
This commit fixes the winetest crash, and may fix other weird FS behavior.
CORE-13367
To be reverted if it brings in too many regressions, even though I tried to do my best to address them.
Leaving the old driver in place, for now; if no regressions are reported regarding CDFS in 0.4.8, then old driver could be dropped for 0.4.9.
- SIZE_T -> ULONG in KsecGatherEntropyData
- Add missing ZwQueryInformationProcess() prototype for x64 build of btrfs
- Fix ml.exe path for VS 2017 Community Edition
- Add missing Handle32ToHandle to basetsd.h
To check that these changes are correct, checkout in a directory (let's call it "ros_svn") the /trunk/reactos/ of our read-only SVN repo r76032 and in /trunk/reactos/modules/, the rosapps, rostests and wallpapers.
In a second directory (let's call it "ros_git"), clone the corresponding Git-converted ReactOS directory.
Before applying this patch (and the previous one that added back the empty directories), you should see additional files in ros_git that are not in ros_svn, corresponding to these files I'm deleting here (plus some .gitignore files),
and you should also see additional files in ros_svn that do not appear in ros_git: these are the empty directories I've restored in my previous patch.
Now, after the application of both the previous patch that restores the empty directories (and deletes the .gitignore files), and this patch that removes the ghost files, you should only see that the only differences
between ros_git and ros_svn are the extra .keep files in the empty directories, and that's all!
Command-line for the tests:
diff --strip-trailing-cr -r ros_svn ros_git > diff_svn2git.txt
"-r" means recursive, and "--strip-trailing-cr" ignores the CR-LF vs. LF (or CR) EOLs.
(*): by "ghost" old(*) files I understand files that existed previously in the far past, that then were deleted long ago in SVN, and that popped out back during the Git migration.
