The patch does the following:
1. Adds recognition of executable script (shebang) files.
2. Returns correct MIME type for mbox files (RFC 4155).
3. Returns XML instead of HTML type in some cases.
changeset: 8411:19f6a88ea241
branch: mbp-2011
user: Romano <unobe@cpan.org>
date: Sat Apr 17 14:35:21 2021 -0700
files: sys/src/cmd/upas/fs/imap.c
description:
When an imap fetch fails, it's helpful at times to know the underlying
cause. This provides more details by providing the underlying error
message.
unlike other tools like iconv(1), a crop(1) without arguments or with
ones resulting in a no-op, like `-t 0 0', errors out. other options
like `-i 0' do not error. this breaks assumptions and results in
tedious intermediary steps or hacks like:
foo | {crop -t $1 $2 >[2]/null || cat} > baz.bit
instead, just ignore the check. subsequent code doesn't make
assumptions on that.
This patch adds dirmodefmt from fcall.h to pretty-print file
permissions, similarly to ls -l. I didn't notice any performance
degradation.
I hope no-one relied on the old behaviour.
To reproduce the suicide try running the following in acme:
• 'Edit B <ls lib'
by select and middle clicking in a window that is in your $home.
There is a very high chance acme will commit suicide like this:
<snip>
cpu% broke
echo kill>/proc/333310/ctl # acme
cpu% acid 333310
/proc/333310/text:amd64 plan 9 executable
/sys/lib/acid/port
/sys/lib/acid/amd64
acid: lstk()
edittext(nr=0x31,q=0x0,r=0x45aa10)+0x8 /sys/src/cmd/acme/ecmd.c:135
xfidwrite(x=0x461230)+0x28a /sys/src/cmd/acme/xfid.c:479
w=0x0
qid=0x5
fc=0x461390
t=0x1
nr=0x100000031
r=0x45aa10
eval=0x3100000000
a=0x405621
nb=0x500000001
err=0x419310
q0=0x100000000
tq0=0x80
tq1=0x8000000000
buf=0x41e8d800000000
xfidctl(arg=0x461230)+0x35 /sys/src/cmd/acme/xfid.c:52
x=0x461230
launcheramd64(arg=0x461230,f=0x22357e)+0x10 /sys/src/libthread/amd64.c:11
0xfefefefefefefefe ?file?:0
</snap>
The suicide issue is caused by the following chain of events:
• /sys/src/cmd/acme/ecmd.c:/^edittext is called at
/sys/src/cmd/acme/xfid.c:479 passing nil as its first parameter:
<snip>
...
case QWeditout:
r = fullrunewrite(x, &nr);
if(w)
err = edittext(w, w->wrselrange.q1, r, nr);
else
err = edittext(nil, 0, r, nr);
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
</snap>
...and /sys/src/cmd/acme/ecmd.c:/^edittext dereferences the
first parameter that is *nil* at the first statement:
<snip>
char*
edittext(Window *w, int q, Rune *r, int nr)
{
File *f;
f = w->body.file;
^^^^^^^^^^^^^^^^^^^^^
This will crash if 'w' is *nil*
switch(editing){
...
</snap>
Moving the the derefernce of 'w' into the case where it is
needed (see above patch) fixes the suicude.
The memory leak is fixed in /sys/src/cmd/acme/ecmd.c:/^filelist. The
current implementation of filelist(...) breaks its contract with its
caller, thereby leading to a memory leak in /sys/src/cmd/acme/ecmd.c:/^B_cmd
and /sys/src/cmd/acme/ecmd.c:/^D_cmd.
The contract /sys/src/cmd/acme/ecmd.c:/^filelist seems to have with
its callers is that in case of success it fills up a 'collection' that
callers can then clear with a call to clearcollection(...).
The fix above honours this contract and thereby removes the leak.
After you apply the patch the following two tests should succeed:
• Execute by select and middle click in a Tag:
'Edit B lib/profile'
• Execute by select and middle click in a Tag:
'Edit B <ls lib'
The former lead to a resource leak that is now fixed.
The latter lead to a suicide that is now fixed by moving the statement
that dereferences the parameter to the location where it is needed,
which is not the path used in the case of 'Edit B <ls'.
Cheers,
Igor
The expression value of the assignment operation was
returned implicitely by relying on regalloc() on the
right hand side "nod" borrowing the register from nn.
But this only works if nn is a register.
In case of 6c, it can also be a ONAME from a .safe
rathole returned by regsalloc().
This change adds explicit gmove() calls to assign the
expression value. Note that gmove() checks if source
and destination are the same register so it wont emit
redundant move operations in the common case.
The same is applied also to OPREINC and OPOSTINC operations.
Mutating lists that are being iterated is needlessly error
prone, and we were removing the wrong message in some cases
if it the dummy got inserted in the right place.
Separating deletion into a redraw/relink and zap phase
simplifies the problem.
Switching window focus used to be non deterministic
as the current window in focus (Window *input) was set
concurrently while processing window messages such as
Resized and Topped.
This implements a new approach where wcurrent() and
wuncurrent() are responsible for the synchronization
and switch of the input.
It is implemented by sending a Repaint message to the
old input window first, neccesarily waiting until that
window releases the focus and then input is updated
and then a Topped or Reshaped message is send to the
new input window.
Note, that when the whole screen is resized that no
input changes need to happening anymore.
When deleting messages that came in just
the right order, we would end up stuck in
a loop deleting and reinserting a dummy
parent, rather than the messages we wanted
to remove.
Sacrifice some of the sub-millisecond timer precision in favor of less
cpu load when the timer is about to be kicked a bit early. Result is
visible *especially* when the guest idling.
Timer proc *still* has to send to the channel (in order to kick PIT
and RTC logic), which takes time, and compensates a bit for possibly
early runs.
tftpd currently unconditionally sets its namespace via /lib/namespace
(newns("none", nil)), which stymied my attempts to pxe boot the
openbsd installer without creating a real /etc dir on 9front, which
would've been gross.
I tried working around this with -h (and -r for good measure), but
again hit issues because the namespace is rebuilt from scratch -- any
binds of /386, /amd64, /cfg/pxe, etc. into the tftp-specific directory
disappeared from tftpd's namespace and rendered my *9front* boxes
unable to boot. I could maintain copies of the needed files in the
tftp-specific directory, but that'd be kind of a drag.
The following patch adds a -n flag to allow the specification of a
namespace file in place of /lib/namespace; similar to ip/ftpd.
I thought about setting up a /lib/namespace.tftp to act as a default
rather than continuing to use /lib/namespace by default (which
security-wise is about the same as allowing 9p mounts by user none,
which I also have disabled), but I had trouble coming up with a sane
default. Maybe someone more experienced would like to try that out.
- sam-d
When the save folder did not exist, and we could not create
it, we would handle up to one Biobuf worth of message, and
then fail, due to a failed tee. The sequence of events leading
up to this was:
openfolder() -> error
tee(0, fd, -1) -> wait for read
write(0, data) ->
write(fd, data) -> ok
write(-1, data) -> error, tee terminates
write(0, attachment) -> error
This change prevents us from writing to a closed fd, and
therefore from erroring out when sending.
We also warn the user.
---
To: 9front@9front.org
Date: Sun, 07 Feb 2021 14:56:39 +0100
From: kvik@a-b.xyz
Subject: Re: [9front] transient dns errors cause smtp failure
Reply-To: 9front@9front.org
I think I found a reason for DNS failing on known good domains.
/sys/src/cmd/ndb/dns.h:156,157
/* tune; was 60*1000; keep it short */
Maxreqtm= 8*1000, /* max. ms to process a request */
So, 8 seconds is how much the resolver will bother with a request it
has been handed, before dropping it on the floor with little
explanation.
It seems quite possible that this is too short a timeout on a machine
during a spam queue run, which predictably stresses the compute and
network resources.
In turn, negative response caching might explain why a particular
unlucky domain would basically stop receiveing any mail for a while.
I'm dying to know if bumping this limit would clear up the queue of
such DNS errors.
---
[narrator: it did.]
fseeko returns 0 on success, not the new stream position.
This allows flacenc to update the streaminfo block when it is finished
(for example to set the number of samples and checksum).
There may be two iso endpoints with the same ID if it is asynchronous
or adaptive (one for data, one for feedback), and rw iso endpoints are
unusable (error out with "iso i/o is half-duplex").
There may be two iso endpoints with the same ID if it is asynchronous
or adaptive (one for data, one for feedback), and rw iso endpoints are
unusable (error out with "iso i/o is half-duplex").
validateattachment has no business with the mime boundary; it is not
part of the attachment itself.
Also, it causes the boundary to be dropped in the message output from
upas/vf, effectively dropping the following attachment (though the
content is still present after the last boundary of the wrapped first
attachment part).
Consider the following sequence of events:
1. upas/vf is run on a message containing two attachments.
2. The first attachment does not have a known extension, so is saved
to a temporary file *including* the following mime boundary.
3. This file is opened as p->tmpbuf, which is used for subsequent
reads until switching back to stdin.
4. The attachment fails validateattachment, so upas/vf wraps it in a
multipart with a warning message.
5. problemchild() calls passbody(p, 0), which copies from p->tmpbuf
until it hits the outer boundary line, which it excludes, seeks
back one line, then returns the outer multipart.
6. problemchild() then writes its own boundary, and then copies one
line from *stdin* to stdout, expecting the outer boundary.
However, this boundary was already read from stdin in 2, so it ends
up reading the first line of the subsequent part instead.
To fix this, pass 0 to passbody() in save() to exclude it from the
attachment file and make it available in stdin when expected.
Reading nested subparts of messages into the root
message array allows deeply nested multipart trees
of messages to show correctly in the message view.
upas/vf was converted to use tmdate, but the formatter was never
installed. This caused it to send attachments to validateattachment
with header `From virusfilter %τ%`, which always failed since upas/fs
would just skip over the message.
The old parser code was rubbish and only worked for trivial
expressions. The new code properly handles complex expressions,
including short circuit evaluation.
As such, the BUGS section has been removed from the test(1) man page.
The description of an unimplemented feature has also been removed.
Despite pervious efforts, mk clean still doesn't remove libcommon.a*
files from cmd/upas/common/. To fix this, let's tell cmd/mklib to do
the job instead.
Runq spawns a number of processes, and wait()s for them
in 2 different places. Because of the way that the exit
handling is done, the wait can get the wrong message.
It turns out that only one place in the code needs to
wait for the child, and in all other cases, it's just
muddling the problem.
This change adds the RFNOWAIT call to all the processes
we don't need to wait for, so that the places that do
need wait will always get the correct child.
Screenlock should use libdraw(2) to init the display
and create the window, instead of looking at the screen
file directly. Also, to prevent new windows from popping
up over screenlock, bring it to the top periodically.
on arm32, we can do one of 4 shifts
by a constant:
reg<<(0..31)
reg>>(1..32)
((u32int)reg)>>(1..32)
reg ROT (0..31)
There's no way to encode a 0 bit right
shift, so when encoding reg>>0, flip
it to the equivalent nop reg<<0, which
can be encoded.
This prevents an incorrect warning for a comparison such as `0 < x`,
where x is an unsigned type. Previously, this would get normalized as
`x >= 0` rather than `x > 0` when checking the comparison.
With ntlm auth, we were trying to set 0 bytes of
the auth struct to its size. The args were clearly
swapped. Fix it.
While we're here, remove some dead code.
When running a mail queue, it's useful to run it with limited
parallelism. This helps mailing lists process messages in a
reasonable time.
At the same time, we can remove the load balancing from runq,
since the kinds of systems that this matters on no longer
exist, and running multiple queues at once can be better
done through xargs.
Querying battery (or temperature) using ACPI takes quite some
resources, which makes the battery discharge faster. It doesn't make
much sense to have it queried as often either. So, when using ACPI:
1) set battery query period to 10s minimum
2) set temperature query period to 5s minimum
When invoking with dd with an invalid size suffix, we
silently accept the suffix. This can lead to confusion,
because lines like:
dd -bs 1K
dd -bs 1m
will silently copy in 1-byte increments. This has caught
people by surprise. While we're at it, megabytes are
convenient, so let's have them too.
Passwd used to produce a very confusing error
about DES not being enabled whenever the password
was mistyped. This happened because we attempted
to guess what authentication method to use, and
preseneted the error from the wrong one on failure.
This puts the legacy mode behind a flag, so that
we don't even try the old method unless it's
explicitly requested.
when loading large binaries such as netsurf, with many
symbols, our hash table fills up with collisions and
loading the symbol table gets very slow. Bumping it up
drops the time to lstk() in acid on netsurf from 4 minutes
to 8 seconds.
Call exits(0) instead of returning from main. Also call sysfatal if
writing of image data fails. Previously, qr(1) would exit with
default non-nil status "main" unconditionally as a result of returning
from main.
On 12/18/20, Jacob Moody wrote:
> Hello,
>
> I recently ran in to some issues with pointing an unbound server towards a
> 9front dns server as its upstream.
> The parsing seemed to fail when ndb/dns received a DNSKEY RR from it's own
> upstream source on behalf of unbound.
> This patch catches and stores the DNSKEY from the upstream server to prevent
> this.
vt sets several environment variables ($TERM, $COLS, $LINES)
after exiting. This change rforks the environment so that this
detritus doesn't get left behind.
The initial protocol handling in exportfs for
cpu and import services is a huge mess.
Saparate the code out into its own program with
its own oexportfs(4) manpage.
The -v flag now does not create a new rio window,
while -w flag does (restores the old behaviour).
This allows vmx to run under vncs and is in general
mode aligned to other emulators and programs.
The previous resize optimization now means that the wfill()
is skipped on resize for libdraw programs.
So do it once /dev/mouse is closed and the window processes
the Refresh message.
"" looks for patterns in the form 'prompt;' or 'prompt%',
and gets confused when proof emits 'illegal;'. This change
replaces the ';' with a ':', which both matches other
conventional error outputs and prevents "" from getting
confused.
As long as the client as the mouse file open
and maintains reading the winname file of the window
after a resize we will avoid drawing the text frame
on a resize as it will be overdrawn by the client.
This reduces flicker on resize somewhat for slow systems.
Fixes 3 issues in our upas mkfiles:
- mk/mkfile and send/mkfile were rebuilding
only the rfc822.tab.$O, even though the
header also needed to be rebuilt.
- CLEANFILES had a pattern that would not
get expanded.
- Third, ../upas/mkfile was being included
in the wrong place and making the wrong
rule default.
dc crashes because a Blk* sometimes ends getting double freed.
To make it crash, any of these lines will do:
(each line is a separate input to dc):
1 sa 2 :a le d sa v :a
1 sa 2 :a le d sa :a
1 sa 2 :a le d sa c
Fix by assigning p to sptr->val before EMTPY causes a jump.
Additionally, dcgetwd() can return 0. all other uses check for
0 ptr; Also fix a buffer overflow.
the hid 1.11 specification says that for hid devices which arent in
the boot subclass (subclass 1), it is only optional to support the set
protocol command. for my devices, trying to set protocol results in a
stall error and unusable devices.
fixes my Tex Shinobi keyboard and Playstation 4 controller.
Before the "native" awk work, a call to the fflush function resulted
in one or more calls to the APE fflush(2).
Calling fflush on a stream open for reading has different behavior
based on the environment: within APE, it's a no-op¹; on OpenBSD, it's
an error²; in musl, it depends on whether or not the underlying file
descriptor is seekable³; etc. I'm sure glibc is subtly different.
Now that awk uses libbio, things are different: calling Bflush(2) on a
file open for reading simply discards any data in the buffer. This
explains why we're seeing truncated input. When awk attempts to read
in the next record, there's nothing in the buffer and no more data to
read so it gets EOF and exits normally. Note that this behavior is not
documented in bio(2). It was added in the second edition but I haven't
figured out why or what depends on it.
The simple fix is to have awk only call Bflush on files that were
opened for writing. You could argue that this is the only correct
behavior according to the awk(1) manual and it is, in fact, how GNU
awk behaves⁴.
1. /sys/src/ape/lib/ap/stdio/fflush.c
2. https://cvsweb.openbsd.org/src/lib/libc/stdio/fflush.c?rev=1.9
3. https://git.musl-libc.org/cgit/musl/tree/src/stdio/fflush.c
4. https://git.savannah.gnu.org/cgit/gawk.git/tree/io.c#n1492
Changeset 50ad211fb12f broke the libcommon rule in
mkupas. Deleting the 'mk clean' in the recipe fixes
this.
Cleanup includes deleting UPDATE vars from all mkfiles,
reorganization of vars in TARG,LIB,OFILE,HFILE order,
and deletion of extra vars used for UPDATE.
The fsdestroyfid() is called regardless if the open succeeded
or failed. This causes erroneous videoclose() when opening
the frame or video file while the camera is active.
Upas/marshal -F was broken with the '-8' command, and silly
without it: It used aliases passed on the command line, so
the destination address was ignored with -8 was passed.
In addition, it would create a new mailbox for any aliases
being sent to, instead of putting them all in one location.
The new -S option is similar to -F, but specifies where the
message should go.
libcommon.a$O doesn't end with a .a, so mk
doesn't know how to look inside it in order
to check if the files are up to date.
This means that when 'mk clean' is run,
libcommon.a$O looks up to date:
% mk clean
...
% mk
mk: 'default' is up to date
Deleting the library works around this problem.
This prevents VESA bios from accessing the pci
CONFIG_ADDRESS/CONFIG_DATA registers (0xCF8/0xCFC)
directly to access pci config space.
This makes sure the access to pci config space is
properly serialized by the kernel.
vt chording behaves slightly differently from other
applications: a chord must be fully released before
the next chord can be applied. This makes any change
in chord apply the action.
supplying a non-ip address in ADD_EDGE crashes the unix tincd.
the reason was that we where misreporting ADD_EDGE messages;
ignoring the information from our peers; and always supplying
the Address string from our configuration instead of the
connections ip address.
now we just report the edge information as is.
When moving messages between folders, mbappend,
deliver, and nedmail were trying to parse the
timestamp ouut of the message. They were doing
it incorrectly, trying to include the user name
as part of the date format.
Change to pass just the date to the date parser.
I have the problem that i need to delegate a subdomain
to another name server that is confused about its own zone
(and its own name) returning unusable ns records.
With this, one can make up a nameserver entry in ndb that
is authoritative and owned by us for that nameserver,
and then put it in the soa=delegated ns entry.
This promotes the ns record in the soa=delegated to
Authoritative, which avoids overriding the ns rr's from
the confused server for the delegated zone.
we used to only allocate ports from 6881 to 6890,
which limits the maximum of parallel torrents to 9.
this change make it go up to 9000, which gives us
at best 2120 ports, which is overkill but ports might
be randomly occupied by other connections.
the new date format introduced by the previous commit;
using numeric timezone offsets; needs one character more,
so increase the date format buffer to 31 characters.
Complete the conversion of upas to remove ctime,
use the new date library, and print time zones
in +hhmm format, instead of NNN format.
This may affect code that expects specific names
for timezones. Fix that code.
Recently the script which generates tab.h and the code including it got
incompatibly changed. People reported problems involving syntax errors
when trying to rebuild the system following a sysupdate.
The problem was with the script being embedded within a mkfile rule,
meaning that mk didn't notice it changing and therefore didn't rebuild
the target file. For people who were rebuilding the system this meant
that the old tab.h got included, causing syntax errors.
This patch moves the codegen script into a file and tells mk about this
new dependency, so that tab.h will get rebuilt for everyone. I also
took an opportunity to rewrite the script, hopefuly making it easier to
follow.
RFC959 says:
"The ALLO command should be treated as a NOOP (no
operation) by those servers which do not require
that the maximum size of the file be declared
beforehand..."
Import the following improvements and bugfixes from plan9port:
4650064a acme: scale window bodies on resize, not including tag space
d28913a9 acme: save/restore multiline tags in Dump/Load
d2df5d6c acme: fix crash in X |cat with multiple windows
3d6e5cb5 acme: preserve window position and selection during Get
The format produced by `diff -u` is inferior to that
produced by `diff -c`, but it's what ape/patch and
unix patch expect, so it's useful to generate it.
This patch adds `diff -u`.
Following is a list of functional changes:
* The -o flag outputs the entire buffer to the length returned
by the syscall, or, in case of fd2path(2) and errstr(2), to '\0'.
* The -x flag is removed; the above makes it possible to pipe
into xd(1) to get the same result.
* The -s flag uses dirfmt(2) to format the stat message, instead
of trying to imitate ls(1).
* Stderr reports are normalized and made easier to parse.
The code also suffered a number of stylistic changes.
Right now, upasfs exposes header lines as is, without stripping
out new lines. It also documents that it provides one header per
line in the info file.
As a result, when we get a mail with headers that span lines,
our tools get confused.
These split lines are not semantically meaningful. From RFC5322:
2.2.3. Long Header Fields
Each header field is logically a single line of characters comprising
the field name, the colon, and the field body. For convenience
however, and to deal with the 998/78 character limitations per line,
the field body portion of a header field can be split into a
multiple-line representation; this is called "folding". The general
rule is that wherever this specification allows for folding white
space (not simply WSP characters), a CRLF may be inserted before any
WSP.
As a result, to simplify processing, we should just strip out the
line separators when exposing the headers from upasfs.
do not try to parse the m->unixfrom field, it only contains
the unix mail address.
instead, have parseunix() save a pointer into the unixheader
after the unix mail address for the unixdate, and later use
it to derive the mails timestamp.
In addition to being more code, this date parser would
treat local times as local, but anything that wasn't a
local time would get parsed as gmt, due to a quirk of
how tm2sec used to work.
This moves the code to tmparse, and fixes timezone parsing
at the same time.
aux/na was comparing the return of putc with <0, when it should
have been comparing against EOF, which is not specified as -ve.
aux/ms2 was zero-extending the mask for the address when it
should have been sign extended.
C99 integer constants with no type suffix promote differently
depending on the way that they're written: hex and oct consts
promote as int => uint => long => ulong => vlong => uvlong.
Decimal constants are always signed.
We used to promote all values to uint on overflow, and never
went wider. This change fixes that, and adds a warning when
a decimal constant that would have been promoted to uint in
the past gets promoted to int.
The de-duplication of txt, nullrr, cert, key and sig records
reduced all records to a single one.
Also, dblookup1() missed the txt record case and did not return
a unique list of rr's.
Now we consider these records unique if their value is different.
The new txtequiv() function does that for TXT records, which is
a bit tricky as it needs to take different segmentation into account.
the previous patch broke 64-bit ops as the type for the
operation is determined from the first argument to
gopcode() (nod1.type), not the type the result (nod.type).
so we need to include the conversion of nod1 type to
the type of nod.
the bug can be reproduced with the following test case:
#include <u.h>
#include <libc.h>
void
main()
{
int size = 1;
size*=1.5;
exits(0);
}
this produces the following assembly:
TEXT main+0(SB),0,$16
MOVW $1,R1
FCVTZSDW $1.50000000000000000e+00,R2 <- tries to convert rhs to int??
MULW R2,R1,R2 <- multiplication done in int? bug!
MOV $0,R0
BL ,exits+0(SB)
RETURN ,
END ,
the confusion comes from the *= operation using the wrong type
for the multiplication. in this case we should use the float
type of the rhs, do the operation, and then convert the result
back to int type of the lhs.
this change ports the same logic from 5c's getasop().
Characters greater than 0X80 will cause a read beyond the bounds of the
array chars[]. For particular unicode characters this can cause deroff
to segfault.
A minimal example:
$ deroff
.EQ
u∈
Segmentation fault
Throughout deroff, charclass() is used instead of directly indexing
chars[] so I presume this was just missed.
version(5) says:
If the server does not understand the client's version
string, it should respond with an Rversion message (not
Rerror) with the version string the 7 characters
``unknown''.
Pre-lib9p file servers -- all except cwfs(4) -- do return Rerror.
lib9p(2) follows the above spec, although ignoring the next part
concerning comparison after period-stripping. It assumes an
Fcall.version starting with "9P" is correctly formed and returns
the only supported version of the protocol, which seems alright.
This patch brings pre-lib9p servers in accordance with the spec.
The current logging prints a debug line for every
message in an inbox, which is unusably verbose.
This removes the prints for unchanged messages,
and adds a print for flag changes.
The $SCRIPTS were added to $TARG, which complicates the all rule, as
each script's object file must be suppressed.
Fix by removing $SCRIPTS from $TARG, removing the script object file
suppression rule, and overriding the install rule.
The script bin install rule assumes that only one script install is
called at a time. Valid calls like 'mk -a /$objtype/replica/changes
/$objtype/replica/pull' will fail.
Fix by adding a for loop.
Remove the unused $UPDATE variable.
Handle cases where parameterless macros expand to each other:
#define FOO BAR
#define BAR FOO
FOO
There were cases where the macros didn't make it into the hidesets,
and we would recurse infinitely. This fixes that.
