Inspired by some changes made in game of trees, I've
implemented a number of speedups in git9.
First, hashing the chunks during deltification with
murmurhash instead of sha1 speeds up the delta search
significantly.
The stretch function was micro-optimized a bit as well,
since that was taking a large portion of the time when
chunking.
Finally, the full path is not stored. We only care about
grouping files with the same name and path. We don't care
about the ordering. Therefore, only the hash of the path
xored with the hash of the diretory is kept, which saves
a bunch of mallocs and string munging.
This reduces the time spent repacking some test repos
significantly.
9front:
% time git/repack
deltifying 97473 objects: 100%
writing 97473 objects: 100%
indexing 97473 objects: 100%
58.85u 1.39s 61.82r git/repack
% time /sys/src/cmd/git/6.repack
deltifying 97473 objects: 100%
writing 97473 objects: 100%
indexing 97473 objects: 100%
43.86u 1.29s 47.51r /sys/src/cmd/git/6.repack
openbsd:
% time git/repack
deltifying 2092325 objects: 100%
writing 2092325 objects: 100%
indexing 2092325 objects: 100%
1589.48u 45.03s 1729.18r git/repack
% time /sys/src/cmd/git/6.repack
deltifying 2092325 objects: 100%
writing 2092325 objects: 100%
indexing 2092325 objects: 100%
1238.68u 41.49s 1373.15r /sys/src/cmd/git/6.repack
go:
% time git/repack
deltifying 529507 objects: 100%
writing 529507 objects: 100%
indexing 529507 objects: 100%
345.32u 7.71s 369.25r git/repack
% time /sys/src/cmd/git/6.repack
deltifying 529507 objects: 100%
writing 529507 objects: 100%
indexing 529507 objects: 100%
248.07u 4.47s 257.59r /sys/src/cmd/git/6.repack
Checking the range of c|0x60 incorrectly classifies many characters
as alphabetic (digits, control characters 0x01-0x20, and punctuation
characters '!'-':'). This prevents tmparse from parsing dates with
a timezone bounded by those characters (for example, "12:11:56 (PDT)").
Instead, just reuse the isalpha macro provided by ctype.h.
When diffing non-regular files, like /dev/null,
pipes, and similar, diff will generate a temp
file to diff against. This is the right thing
to do, but the temp file leaks into the diff.
This patch retains the original file name all
the way through to diff output.
This code is checking the return of devwalk for
a walk resulting in a clone of an open pipe file. However,
devclone ensures that the chan we are cloning is not
currently open.
This is mostly a copy of port/usbxhci.c with PCIWADDR() replaced
by PADDR() and the pci specific code stripped out.
This could be refactored at a later time.
There is a gpio line for the main hub reset that needs to be
asserted and some power management functions that are currently
done by u-boot (using "usb start" command).
We will do these ourselfs once we have the infrastructure for
it in place.
The getserv function has remained mostly unchanged since
1ed. These checks are for handling string local ports
or lack thereof. This was used by datakit at the time,
where 'ports' were service strings. This made the default
datakit service 'login' when a connector did not give
a specific service. The checks for directory traversal also
seemed specifically to guard against a connector from providing
a malicious service string. With datakit gone all current protocols
use numerics as ports so these checks and defaults become uneeded.
1ed reference:
https://github.com/plan9foundation/plan9/tree/1e-1992-09-21
This is a work in progress port to the mntreform2 laptop.
Working so far:
- mmu (same as raspberry pi 3b+)
- arm generic timer
- gicv3
- uart1
- enet
With access to the uart, one can netboot this kernel in u-boot
using the following commands:
> dhcp
> bootm
devproc allows changing the noteid of another process
which opens a race condition in sysrfork(), when deciding
to inherit the noteid of "up" to the child and calling
pidalloc() later to take the reference, the noteid could
have been changed and the childs noteid could have been
freed already in the process.
this bug can only happen when one writes the /proc/n/noteid
file of a another process than your own that is in the
process of forking.
the noteid changing functionality of devproc seems questinable
and seems to be only used by ape's setpgrid() implementation.
Scrollbar was drawn using a wide dark gutter over a white background
whereas other applications (window, sam, ...) use a thinner scrollbar with an
inverse colorscheme.
This makes the scrollbar more consistent with other 9front applications.
When writing to a large variable in /env, we
spam 'Write error', in spite of the env var
working. This is new behavior, as of commit
b90036a062.
It produces a lot of scary, noisy warnings,
which are probably bogus.
When pushing, git/send would sometimes decide we had all the
objects that we'd need to update the remote, and would try
to pack and send the entire history of the repository. This
is because we only set the 'theirs' ref when we had the object.
If we didn't have the object, we would set a zero hash,
then when deciding if we needed to force, we would think
that we were updating a new branch and send everything,
which would fail to update the remote.
A while ago, qwx noticed that we clobbered the exec
bit when merging files. This is not what we want, so
we changed the operator precedence to avoid merging
dirty files implicitly.
But we do want to merge, because it's convenient for
maintaining permissions. So, instead, we should do a
3 way merge of the exec bit.
This patch does that, as well as reverting the rollback
of that change.
While we're here, we adjust the timestamps correctly
in git/branch.
This requires changes to git/fs, because without an open
handler, lib9p allows opening any file with any mode,
which confuses 'test -x'.
there was a diff that went in a while ago to improve
this, but it got backed out because it encounters a
bug in upstream git -- the spec says that a single
ACK should be sent when not using multi-ack modes,
but they send back multiple ones.
This commit brings back the functionality, and works
around the upstream git bug in two different ways.
First, it skips the packets up until it finds the
start of a pack header.
Second, it deduplicates the want messages, which
is what seems to trigger the duplicate ACKs that
cause us trouble.
This patch makes sam's = cmd output what seems to me a more
useful (plumbable) format: /full/path/to/file:addr , where
addr is line(s) under = and rune(s) under =#.
Avoid calling sdgetdev() for every I/O. Instead,
put the SDunit pointer for #S/sdXX/* files in Chan.aux
and keep a reference to SDev between sdopen()/sdclose().
This avoids having to do the sdindex() lookup and
qlock(),incref(),decref() on every read/write
operation. Removal of SDev's is quite rare and only
can happen with pcmcia ide controllers, and i assume
that for that we can assume thet fileservers having
been exited properly and closed their files before
we attempt to remove a device.
The rest is improving waserror() codepaths, making
sure we release the locks for any of the interface
callbacks (verify/online).
Also get rid of tas() and instead only change the
unit's rawopen flag while holding raw qlock.
The change 775a4bea43
"libsec: various changes to tls"
...
4. simply prf code...
... broke the TLS1.0 prf function, missing the fact
that the prf ouput for sha1 and md5 need to be
xored together.
pci uarts are detected late and usually do not contain
the console= parameter logic.
for these, we can just enable them when devuart is reset,
and replay the boot messages once enabled.
this is usefull as it allows us to use these uarts for
kernel debugging in interrupt context.
MBC3 write switches on a>>13, so the RTC register is 5 (0xA000-0xBFFF).
Mask off upper bits of DH register when updating the timer. Only the
lowest bit is part of the day counter.
Use uint for x in timerforward() so that we don't set negative values
for timer registers if it happens to overflow.
Update timer and then latch rather than the other way around.
Otherwise, timer remains static and will overflow after 512 days.
Although git9 always uses the same commit date and author date, other
implementation do make a distinction. Since commit date is more
representative of the commit graph order, use this as a traversal hint
instead of author date.
If the server only supports the dumb protocol, the first 4 bytes of
response will be the initial part of the hash of the first ref.
The http-protocol documentation says that we should fall back to the
dumb protocol when we don't see a content-type of
application/x-$servicename-advertisement. Check this before
attempting to read a smart git packet.
We now keep track of 3 sets during traversal:
- keep: commits we've reached from head commits
- drop: commits we've reached from tail commits
- skip: ancestors of commits in both 'keep' and 'drop'
Commits in 'keep' and/or 'drop' may be added later to the 'skip' set
if we discover later that they are part of a common subgraph of the
head and tail commits.
From these sets we can calculate the commits we are interested in:
lca commits are those in 'keep' and 'drop', but not in 'skip'.
findtwixt commits are those in 'keep', but not in 'drop' or 'skip'.
The "LCA" commit returned is a common ancestor such that there are no
other common ancestors that can reach that commit. Although there can
be multiple commits that meet this criteria, where one is technically
lower on the commit-graph than the other, these cases only happen in
complex merge arrangements and any choice is likely a decent merge
base.
Repainting is now done in paint() directly. When we find a boundary
commit, we switch our paint color to 'skip'. 'skip' painting does
not stop when it hits another color; we continue until we are left
with only 'skip' commits on the queue.
This fixes several mishandled cases in the current algorithm:
1. If we hit the common subgraph from tail commits first (if the tail
commit was newer than the head commit), we ended up traversing the
entire commit graph. This is because we couldn't distinguish
between 'drop' commits that were part of the common subgraph, and
those that were still looking for it.
2. If we traversed through an initial part of the common subgraph from
head commits before reaching it from tail commits, these commits
were returned from findtwixt even though they were also reachable
from tail commits.
3. In the same case as 2, we might end up choosing an incorrect
commit as the LCA, which is an ancestor of the real LCA.
When loading an acme dump file that contains a window with only one
tag line, there are cases where acme hides that window (i.e. not even
its tag is visible).
The following commands reproduce the issue:
% ed <<EOE
1
i
/tmp
/lib/font/bit/pelm/unicode.8.font
/lib/font/bit/pelm/unicode.8.font
0
f 0 5 175 175 1
5 40 175 1 0 /sys/src/cmd/acme/ Del Snarf Get | Look
f 0 4 330 330 3
4 27 330 1 0 /tmp/ Del Snarf Get | Look
.
,w /tmp/test.dump
Q
EOE
% window -dx 900 -dy 600 'acme -l /tmp/test.dump'
This issue was introduced in commit 47b7dc5ccd.
This avoids ipconfig having to explicitely specify the tag
when we want to set route type, as the tag can be provided
implicitely thru the "tag" command.
This adds a new route "t"-flag that enables network address translation,
replacing the source address (and local port) of a forwarded packet to
one of the outgoing interface.
The state for a translation is kept in a new Translation structure,
which contains two Iphash entries, so it can be inserted into the
per protocol 4-tuple hash table, requiering no extra lookups.
Translations have a low overhead (~200 bytes on amd64),
so we can have many of them. They get reused after 5 minutes
of inactivity or when the per protocol limit of 1000 entries
is reached (then the one with longest inactivity is reused).
The protocol needs to export a "forward" function that is responsible
for modifying the forwarded packet, and then handle translations in
its input function for iphash hits with Iphash.trans != 0.
This patch also fixes a few minor things found during development:
- Include the Iphash in the Conv structure, avoiding estra malloc
- Fix ttl exceeded check (ttl < 1 -> ttl <= 1)
- Router should not reply with ttl exceeded for multicast flows
- Extra checks for icmp advice to avoid protocol confusions.
the bug happens when we did the fast exit thru "done" label,
where we would not make sure that theres space in the buffer
for the NUL terminator.
instead, avoid the fast exit and always do the final
adjbuf() that makes sure we have space for the NUL terminator.
remove the pointless pb checks, they'r wrong (should'v
been bp >= buf+bufsz) and adjbuf() already makes sure this
can never happen.
EDID 1.3 section 5 gives a table describing the priority order of
timing information. Use this ordering when constructing the EDID
mode list.
Since aux/vga selects the first mode in the modelist that matches
the given size, it will now select the mode of that size with the
highest preference. Or, if you set vgasize=auto (or some other
string without an 'x'), aux/vga will select the Preferred Detailed
Timing.
This should make it unnecessary to modify vgadb in many cases.
when reverting files, 'cp -x' updates the mtime
to the time the file was committed. this prevents
'mk' from rebuilding the file, leading to stale
builds.
this change touches the file on revert, so that
we rebuild the file.
Use an RWlock so readers can work in parallel in
the common case (no cache updates).
When a reader needs to update the cache to add
a new learned source mac address, it will drop
the rlock and aquire the wlock to do the update.
When we get a read error, we now unbind the
port to avoid further packets being forwarded
to it.
This is usefull for hotplug ethernet devices
like usb ones or tunnels.
Simplify the unbind, getting rid of the refcount,
by having only the reader proc call freeport().
Avoid holding the bridge lock while opening
and closing ethernet/tunnel device files during
bind and unbind.
Dont use smalloc() (especially when holding locks).
Allocate bridges dynamically, so we do not waste
the memory when we do not need them.
Reject non-hostowner from allocating new bridges.
Use consistent naming: port -> port
Use consistent comment style: // -> /* */
The altsetting was handled only for a single endpoint
(per interface number), but has to be handled for each
endpoint (per interface *AND* altsetting number).
A multi function device (like a disk) can have
multiple interfaces, all with the same interface number
but varying altsetting numbers and each of these
interfaces would list distict endpoint configurations.
Multiple interfaces can even share some endpoints (they
use the same endpoint addresses), but
we still have to duplicate them for each
interface+altsetting number (as they'r part of
actually distict interfaces with distict endpoint
configurations).
It is also important to *NOT* make endpoints bi-directional
(dir == Eboth) when only one direction is used in a
interface/altsetting and the other direction in another.
This was the case for nusb/disk with some seagate drive
where endpoints where shared between the UAS and
usb storage class interface (but with distict altsettings).
The duplicate endpoints (as in using the same endpoint address)
are chained together by a next pointer and the head
is stored in Usbdev.ep[addr], where addr is the endpoint
address. These Ep structures will have distinct endpoint
numbers Ep.id (when they have conflicting types), but all
will share the endpoint address (lower 4 bits of the
endpoint number).
The consequence is that all of the endpoints configuration
(attributes, interval) is now stored in the Ep struct and
no more Altc struct is present.
A pointer to the Ep struct has to be passed to openep()
for it to configure the endpoint.
For the Iface struct, we will now create multiple of them:
one for each interface *AND* altsetting nunber,
chained together on a next pointer and the head being
stored in conf->iface[ifaceid].
--
cinap
Wlock()'ing the ifc causes a deadlock with Medium
bind/unbind as the routine can walk /net, while
ndb/dns or ndb/cs are currently blocked enumerating
/net/ipifc/*.
The fix is to have a fake medium, called "unbound",
that is set temporarily during the call of Medium
bind and unbind.
That way, the interface rwlock can be released while
bind/unbind is in progress.
The ipifcunbind() routine will refuse to unbind a
ifc that is currently assigned to the "unbound"
medium, preventing any accidents.
Pattern matching with lists no longer works:
; ls /tmp/*.c
/tmp/npage.c
/tmp/pagedebug.c
/tmp/pageold.c
/tmp/scheduler.c
/tmp/writeimagetest.c
; ls /tmp/^(*.c)
ls: /tmp/*.c: '/tmp/*.c' directory entry not found
; 9fs dump
; bind /n/dump/2021/1002/amd64/bin/rc /bin/rc
; rc
; ls /tmp/^(*.c)
/tmp/npage.c
/tmp/pagedebug.c
/tmp/pageold.c
/tmp/scheduler.c
/tmp/writeimagetest.c
the fix:
we have to propagate the glob attribute thru lists
as well. before it was only handled for single words
and propagated thru concatenations...
the Xglob instruction now works on list, and we
propagate the glob attribute thru PAREN and WORDS
and ARGLIST nodes.
also, avoid using negative numbers for the Tree.glob
field as char might be unsigned on some targets.
2021-08-14 17:50 GMT, kemal <kemalinanc8@gmail.com>:
> 1- as driver reads 8 bytes from nvm instead of 6 so fw doesn't
> spit us an ADVANCED_SYSASSERT, it was reading 2 more
> extra bytes. apparently those 2 extra bytes were put to
> the first 2 bytes of our buffer, so we got to skip that.
some more thoughts on this, i think as 0x15*2 is not multiple
of 8, fw rounds the offset to 0x14*2. i have touched to code
to read data from 0x14*2 then ignore the first 2 bytes, just
so it's not confusing. if this causes mac to be read wrong again,
report.
also, some more changes:
1. set the fwname at iwlpci, just to align the behavior with 8000+.
this is a cosmetic change.
2. i have discovered that on device boot/reset/shutdown functions,
our driver slept way much more than it should. the reason for that is,
driver used the function delay() on places where it needs to use
microdelay() instead. i have modified the code to use microdelay().
wpi likely needs similar changes too. i hope that this does not
break the code.
3. zzz a bit more on tx/rx scheduler shutdowns and niclock.
4. openbsd's iwm and linux apparently does not check if ownership
was obtained anymore in their handover functions. instead they
just loop until the hw is ready. aligned the behavior.
see linux commit: 289e5501c3141191dd830957f1d764d3dc14a54f
5. don't take antenna masks from nvm. it's apparently empty
in some cards from 7k family. we will rely on what the fw file gives
us.
6. when the calibration is completed, wakeup the proc that runs
postboot. otherwise that thing sleeps for like 2 whole seconds
even if calibration completed earlier.
i honestly don't think any of these changes will fix 7260 not
being able to get calibration results, but i don't see anything
wrong at all in postboot7000 at this point. i will just hope
these changes somehow make it get calibration results.
NOTE: latest patch on the 9front ml, posted Mon, 14 Feb 2022 15:26:55 +0300
(non functional as of yet)
Some gba cartridges use flash chips produced by a few different
manufacturers for backup memory. A game rom may then try to read the
device id for its flash chip and break if it's the wrong one. There's
no way for us to know which chip is supposed to be there, save for
looking it up in some database. These chips are either 64k or 128k
big. gba(1) hardcodes the id for the SST 64k chip, but games like the
Pokémon ones use a 128k Macronix chip and expect a different id,
completely breaking them.
This patch sets a different default for 128k flash backups, and gives
the ability to set the exact type when creating the save file if it's
really necessary. This new default fixes the Pokémons. The type is
saved in the save file, but this will *not* break existing ones. It
may be that we don't need to worry about the type and just have an id
that corresponds to the right chip size, which is why these defaults
make sense, but I haven't tested more than a dozen roms, mostly since
I don't really know which games use 128k flash.
Before applying this patch the following will fail to open ed
in the '/tmp/s p a c e' folder:
<snip>
% mkdir '/tmp/s p a c e'
% window -cd '/tmp/s p a c e' ed
!pwd
/tmp/s p a c e
!
q
<snap>
After applying the patch the above sequence works as expected,
opening ed in the '/tmp/s p a c e' folder, printing the present
working directory, and quitting ed.
The root cause was a faulty computation of the pointer `s`,
being off by one, leading to any arguments after the
directory path to be skipped.
This regression was introduced in revision:
• 614f1d6268
Thanks umbraticus for finding and reporting the issue.
The device address is highly variable and depends on
all prior enumerated devices.
This can happen with some devices that do not have
a serial number and all devices of the same type
having the same hname.
Using a counter of collisions per hname makes more sense
and is more stable (given that the order devices are
enumerated is deterministic).
This allows mapping incoming filenames to a different name
using regular expressions, followed by subtitutions
of the %[ICE] format strings.
I needed this to have individual cmdline.txt files for
netbooted raspberry pi's. In this example, i map cmdline.txt
to %C, which gets substituted for /cfg/pxe/$ether of the client.
The transitional PCI device ID for block devices is 0x1001, and the
virtio spec says that devices must have the transitional device ID or
0x1040 + the virtio device ID (2).
Mothra does not currently render text inside <samp> tags inline
similar to <code>, but instead treats them like <pre> which is actually
incorrect behavior. The following small patch should fixes issue.
Sometimes, there is the one-off occation when one needs to
pass a huge list in rc...
This change makes devenv track total memory consumption
of environment groups allowing them to grow up to 1MB in
size (including overhead).
(Before, only the variable size was restricted, but
not the amount of files being created).
The maximum value size of a single environment variable
is set to half of the total size, which allows the
occational large value. (But not many of them).
Because we track all memory consuption, it is also
now possible to create around 10k small environment
variales.
A hashtable is added for name lookups and the qid.path
was changed to allow direct indexing into the entry
array without needing a scan lookup.
All smalloc() calls have been removed, exhaustion is
handled with error(Enomem) avoiding deadlock
in case we run out of kernel memory.
This does not have any adverse effect, since yylex never calls mpatov
with a string with leading 0 (and not 0x) that contains non-octal
digits, but the condition was wrong regardless.
Strings for existing codes in the most used server (OpenSSH) just
repeat the error code name. OTOH we like to have wording of the
strings under our control as much as possible, so we can easier find
and process them. Error strings are still usefull as fallback for
compatibility with future versions of the server.
- fix showpage1 only decrementing proc counter once limit is reached;
this blocked having more than one loadpages process after NPROC calls,
since the next one has to wait until the last has exited
- allow procs to skip pages currently being loaded by others; this
forced processes to wait for each other at the same page
- bump NPROC from 4 to 8
- (hack) immediately fork a few times after adding all pages at
startup to force loading a batch of pages in parallel
To avoid a MAXMACH limit of 32 and make
txtflush into an array for the bitmap.
Provide portable macros for testing and clearing
the bits: needtxtflush(), donetxtflush().
On pc/pc64, define inittxtflush()/settxtflush()
as no-op macros, avoiding the storage overhead of
the txtflush array alltogether.
The new rc's exit status will be '' for a successfull
pipeline execution instead of '|'.
This is a bit too tightly coupled, so just use if()
statement instead, handling this in a portable way.
programs that try to use /srv would choke when running
under iostats, because we intercepted operations on the
special, magic fd passing; we should instead give them
access to the real /srv.
when running outside of a repository, we would try to
remove '$msgfile.tmp', but we had never actually set
'$msgfile'.
the error is harmless, but annoying.
The following patch fixes acme display glitches at the bottom fringe
of columns when adding/moving/resizing windows.
Here an example of an easy to reproduce case:
• https://invidio.xamh.de/watch?v=iLekQrxycaM
…opening acme and resizing a column to the right is all that is needed.
The functions winresize(…) and textresize(…) are extended with an
additional parameter `fillfringe` to indicate if a window/tag shall
fill a potential fringe area that would otherwise remain white.
The changes have been inspired by the approach taken in plan9port
acme.
Eckard's test case:
cat <<! | cat
asdf
!
The issue is that we have to continue parsing until we see
the '\n' before consuming the here document.
So we revert to the old approach of having two functions:
heredoc() which remembers if we'v seen a heredoc redirection
and a second readhere() function that reads the doc from
the lexers input and sets Tree.str on thee REDIR node.
due to the way the size of buf was calculated, the parent
file had one trailing null byte for each parent. also, while
we're here, replace the sprint with seprint, and compute the
size from how much we printed in.
