USENET-Format Poll

OpenPGP is really an encoding for signatures and not much else. It doesn't say how to define USENET signatures or all the many attributes a USENET certificate needs. How do you say "This keyholder is allowed to newgroup in comp.*" or "This keyholder is the moderator of rec.humor.funny" in OpenPGP?

You don't, except by extending it. What we would get from OpenPGP before extending it is almost nothing -- a packed binary syntax for a few basic parameters. USENET tends not to use packed binary syntaxes, preferring readable, editable forms even though they are larger. -BradT

Anglocentric as it is, it's simply more efficient, with the vast majority of groups being in English, to have that as a default for this header. We want to reduce the mandatory headers.

Alternately, we could specify that "newsgroups" have a default and say only that postings not in the default language of the newsgroup must contain such a header. -BradT

No spec should have two equivalent means of doing something, except as a migration from one to another. MIME is clearly better than uuencode, and now as most readers have the ability to use MIME, it's time to make a decision to do one and not both. -BradT

It must be realized that the primary use of cancel on the net today is not to cancel one's articles, but to cancel net abuse, namely spam. Cancel lock, implemented without a corresponding 3rd party cancel system, creates uncancellable spam, at least within the spec.

In the rush to stop abusive forged cancels, we should not install a one half of a facility, bringing on an even worse problem.

Yes, there are some ad-hoc 3rd party cancel systems out there. But it is not right for a spec to knowingly create a problem and just hope that ad-hoc systems will fix it. If we know we need a 3rd party signed cancel (or rather 3rd party signed message) system, we should have it.

Cancel locks are easier to implement. Drafted without a spec for general signed messages, implementers will implement only them.

You'll be able to easily spot spam after that -- spam will be the first stuff to have cancel locks.

Cancel locks are useful, but they are not useful enough for this group to endorse only half (or rather even less than half) of an answer to the security question. It's all or nothing. -BradT

There are two ways to secure cancel. The cancel-lock/unlock pair, with one string on the original message and another string that hashes to the first one on the cancel, and public key digital signature of the cancel, with a certificate indicating the keyholder has authority to cancel the original message.

The cancel lock is easier to code, and requires no public key generation and no certificates. Thus it may make sense for simpler news systems to implement it for users.

But it has its costs. It requires that the lock string be placed on all messages when they are posted. As such it is a lot bulkier. In addition, one has to do this in advance. It is not possible to post an article with old software, realize later one wishes to cancel it, and download software to cancel it. This can, however be done with PK signed cancels.

Cancel locks only allow the originator to cancel a message, if they planned in advance. PK signed cancels can allow any party you want to authorize, and in particular allow the cancellation of forgeries by the victim. Cancel locks only allow cancellation of forgeries by the perpetrator. (This is both a feature and a bug)

However, when it comes to sites cancelling articles injected at the site, the PK signed cancel is a big win. For sites to add an extra 35 bytes to every article is not trivial in size. For moderators and their sites to add it is even larger.

Sites rarely have to cancel, and they rarely know in advance they will need to cancel. The most common causes for a site wanting to cancel are spam, and a forged message appearing to come from their site which is not authorized to (often also spam, but not necessarily.)

With cancel locks, the locks must have been added (perhaps on top of a user's lock) when the message was posted. One can't be running an older, or simpler injector when the spam is posted and then fix the problem later. With PK signed cancels, one can do that. One downloads a spam canceller, gets a site certificate and issues the site cancel, after the fact.

With PK signed cancels, one can cancel forgeries as well.

PK signed cancels also offer a major efficiency if the posted messages have a message-id using the site's domain name. One can verify such a cancel without having to have first received the message, or having to fetch it from disk. One simply receives a cancel for a message-id, which of course contains a domain, and checks if the canceller is authorized to cancel message-ids in that domain. This won't always work, but in well over 90% of cases the message-id will be from the domain and things will be a lot more efficient.

A cancel lock system requires that in the common case that the message has not yet arrived (very common with a cancelled message), the cancel with unlock key be stored in a pending queue, then verified when the actual message arrives to see if it is authorized.

There is only one downside to using PK cancels for site or moderator cancels. The site admin or moderator must, before issuing the cancel, get a certificate saying they are the master of their domain. A moderator can be presumed to already have one, simply to moderate the group, since we want moderated groups to be secured.

The site should get one in advance, but if they don't, there are ways to build tools to allow the quick fetching of such a certificate for short term use. (Somewhat complex before secured DNS but still doable.) After secured DNS becomes common -- and it is going to -- all sites will have a site certificate already. -BradT

Look how many travails we had this time because we couldn't do a subtle change to the semantics of a header. All formats should be extensible as much as possible. And tagged parameters are much better than positional. I hope it's a no brainer to define new headers as extensible, but I also say let the new parsers also parse (and ignore) any extensions on old headers where this doesn't break something.

Some day, far from now, the old software will be gone and people will be able to extend those headers, not define a redundant one as we are doing for supersedes. They will thank us then. If we don't do it now, then those headers will never be extensible. -BradT

I had originally proposed the prefixes. Now I think the more generalizable extension makes the most sense. These are all new headers, no problem having extensible options on them. This also means that if we want, the attribute can be selectable (ie. header is sometimes persistent sometimes not, or state can be enforced.) We can also remove the attribute later if all software is deemed to know about it. The double-extended headers like X-P-V-Hello just seem too nuts to me. -BradT

I'm saying our software should handle binary. I know we can't use it yet. Just every bone of my software designer being says that bits is bits, and since it's really not that hard to design software that can handle arbitrary bits, we owe it to the future to require it of software now. The bulk of the volume of USENET is now jpegs and other raw binary. We would save a ton of overhead to eventually have groups where raw binaries can be present.

We can't have it now, but we'll never have it if we don't plan it now. -BradT

I believe that body structure should be formal. Software should never have to guess at body structure, and it should be very hard to have users accidentally generate body structure elements. That's the main problem with "-- ". It can appear by accident. It can appear by including text of another article without indent.

Good formal specs are formal specs. So any of the formal methods, be it a MIME tag, a Sig-Len header or an out of band tagging method are preferred by me. -BradT

This is a close call, but for USENET, the header makes more sense. Using multipart/signed is ugly even to users of MIME aware software, because the headers that are signed must be repeated in the body, and the transports must now parse the body to get those headers. To make things look nice, the headers need to be at the end of the article, which is a further burden on the transport.

With mult/signed it is complex as to how to do multiple signature. Especially if the 2nd signature adds some new data it wasnts to sign. The only current way to do that is to have multipart signed within multipart signed, perhaps with intervening multipart related. Very cumbersome and ugly for the user of software not fully aware of how to deal with it. It's also bulky.

Doing it in the header has a couple of disadvantages. It's easier to get the software wrong, because one must precisely distinguish between headers included in the sig and those not, possibly for multiple different sigs that sign different headers! And one must handle header re-ordering and always get the headers in the right order. (Though a sort is not that hard to do.)

And it doesn't fit existing IETF standards. However, I think USENET's signing needs are differnet enough that this is something we have to accept. USENET articles are distributed hundreds of thousands of times. We want to make things compact and efficient.

-BradT

Message/partial is there to allow a message to get into a system that for technical reasons can't handle messages over a certain size.

If we require that all net software be able to handle a certain size, like 1MB or 5MB, then there is no technical reason to use message/partial on articles below that size. And lots of reasons not to -- it's complex, hard to code, and runs the risk of missing a part.

So the only reason to use message/partial on these smaller articles is to get into sites that reject large articles for policy reasons. To trick them, in effect, into taking your large article when they have said they don't want it.

I can see no value in that.

So this means message/partial should only be permitted on articles above a certain size which we have declared all software can already handle. Tools that want to be able to handle over that size should implement message/partial. -BradT

To help implement size based policy filtering, it is necessary that one know the size of the final article before accepting or feeding any given part.

As such, articles split into chunks should contain within them the final size. I propose we add a parameter to the Content-type line of the form final-size=kilobytes.

We should send this off to the MIME group. If they accept it, great. However, even if they don't accept it, we can still insist on it, rejecting any article that doesn't have it. USENET message/partial groups will not be generated by E-mail software but only by news-aware software, and we are allowed to require extra parameters. -BradT

While it's clear that multipart/alternative is inefficient and undesired in some groups both for that and because older newsreaders can't handle it, all this was known to the MIME group when they designed it.

They figured having a way to migrate from one form to another was worth the cost, or at least can be worth it. To deprecate this useful feature in a long term spec has no technical basis. It is a policy decision, and should be made within subsets of the net, not globally for all USENET-compatible software. -BradT

I am not sure why need the functionality of the new syntax, and it will break some old software. -BradT

Named articles should not be confused with things like FAQs. While one can implement FAQs as named articles, and those may wish all their updates to be visible to the readers of a group, many named articles such as policy files, topic lists, help screens, menus, group descriptions and so on should not be visible to readers ordinarily.

It should be up to the poster whether article updates are by default seen by the users. Those users who want to see such articles can do so by tracking the control messages in the group, but frankly it's hard to credit that more than a tiny minority will want to track these updates. What the majority want should be the default and easy thing.

If instead the articles are visible but contain a header with the request to not be shown, only users of newer newsreaders will get to ignore them. The vast majority of users for some time will see them -- which defeats the purpose of posting them as database updates.

USENET has a mechanism for invisible messages that affect databases rather than present information to readers. It is control messages, and that is what should be used.

For FAQs and articles which are expected to be seen, those should be posted as visible articles. There can either be a header to give them a name, or they can be posted in both visible form and invisible form, the former for readers to see and the latter for storage in the database for fetch on demand.

This actually makes more sense even though it's less efficient. The invisible forms can be, for example, multipart/alternative articles with HTML while the visible ones would be plain text for some time. Since the invisible would only be seen by new newsreaders, they can take advantage of things like HTML. And many FAQs today are maintained in HTML for good reason, the links are quite useful.

-BradT

The overview system is cumbersome and gains little. To force all named article fetches to require an overview fetch, scan and article fetch is slow, complex and inefficient, when in fact quite commonly the newsreader or posting program will know the article it wants and just want to fetch it.

Requiring an overview scan defeats several of the key purposes of named articles, including fast customization. I envision reading programs doing things like fetching a 3 line description of each group when a user enters the group to put in a window at the top. You don't want to do an overview fetch to do this, you just want to say, "get me the description."

Using free-form names, which can contain dots for delimiters to turn them into a tree, allows the overview method, it just doesn't forbid simple fetch by name. And it's a lot simpler. -BradT

In-place replacement will work for both new newsreaders and old. The only thing it makes harder is any effort to try to track down old versions. Frankly, I find it hard to accept arguments that more than a tiny few people will want to routinely do this.

If we do in-place replacement, the code is done once, in the relative small number of servers. There are only a few server systems, and they are maintained by system admins, not ordinary users. And there are orders of magnitude more clients, maintained by ordinary users.

Replacement in-place gives the benefits of replacement to users immediately, as soon as their server upgrades. Otherwise it will be many years before one can reliably do a replaces, and that will cause a chicken and egg problem about getting the feature used.

-BradT

The proposed author-id stored an injector-id (already in the new path line) and some site-specific encoding of the user. The tail of the path is otherwise unusued. Who needs a new bulky header copied a million times to provide info that can only be decoded locally anyway? -BradT

I think a bulk cancel solves all the problems of multi-cancel for chains of articles and spam cancel. It should allow an arbitrary number, not just what you can get on the control line. Put it in the body, I don't care a lot whether it's a new control message or a variant of the old. -BradT

In a way, USENET in its RFC822 mail header format, has always followed the philosophy that data should be tagged and fre-form rather than positionally formatted. The old A news used positional headers. That's hard to read, and hard to extend. And easier to get wrong.

Tagging is the way to go. Parameters for headers should move to a tagged format instead of a positional format for their parameters. We can't change the old messages which had positional parameters like From of newgroup, but we can fix this error for the new ones. -BradT

I see no reason to not leave them in, once they require authentication. With the potential for abuse gone, they can be used for mapping again. They are not hard to implement, and mapping is valuable. -BradT

I haven't heard any reports of people still using ihave/sendme in this fashion. This belongs in the transfer protocol not the news format.

However, a globalized system, where central sites broadcast "complete" lists of message-ids, along with their newsgroups/distribution and hash, has a lot of value, as it will let sites detect when they miss an article (or their feeds miss an article) and then fetch it by some means. -BradT

We get so many newsgroup wars because USENET has nothing in between the newsgroup and the thread. Most other systems do have such tools, and USENET needs one.

This would allow easy creation of subtobics to follow closely or to avoid. Their permanence would make it easier for users to pick them and tools to work with them. Today if there is a major subtopic within a group there is no way to reliably track it or avoid it. -BradT

Distributing complex software is close to impossible. You will never be able to write policy checking software to run at every client or even every server that is consistent.

Even if you could, it's harder, since you need to put the code in scores of clients and keep it up to date.

No, it's a lot easier, and far more reliable to have policy checking done by one program, written once and maintained in one place (or a small number of places.) That program can reject or cancelbot things that don't obey policy.

The remaining hard part is to get this message back to the poster. One simple way is E-mail, though it is not immediate enough. So if necessary one can try to find a way to distribute policy information in some language to posting programs, but frankly this seems a daunting task. -BradT

The specification should say nothing about policy issues. The spec says how to do it, not what to do. It is up to other bodies to decide even the default states of what is forbidden and permitted. -BradT