Networking design of a decentralized Internet

 

Networking design of a decentralized Internet

 

 

Hey everyone,

I've been thinking through mesh net concepts and how a new, more decentralized, the internet could be built. Or merged in, rather. This will probably consist of a few decent thoughts mixed in with some unbacked rants and statements, so please bear with me.

I have little concept of wireless networking and effective mesh networking, so not all of my ideas may come together. Ideally, we'd have a mix of both. Run some CAT-5 to your neighbor, bury some fiber to your friend a km away, and connect to a mesh network. Wireless is extremely complex but lets us work without wires where impractical, too illegal, and where we're not ready for them. We should find a way to mix and blend our layer 1 friends without making the others suffer.

Layer 3 thoughts:

Scrap the notion of IPv4 altogether. We can do fancy things with IPv6 and run into less chance of collision if we start adding random addresses, not knowing of who else has them. No point building the internet on something that's already running out of addresses as it is. That being said, it may not too hard to dual-stack or agnostic-stack (layer 2) much of it.

There are a few important things we need to work out:

IP addresses and "authority". Route propagation/distribution. Route optimization/pathfinding. Routing tables. Multicast. DNS, in a decentralized world. Physical interconnects.

Currently, we have IANA, ARIN, and DNS registry all very centralized. You buy your IP blocks from ARIN if you're in the US (or can from a private party, not really sure how that process works), and an ASN. Probably a pretty expensive investment, and very controlled.

Route propagation is usually done through BGP, which at least has a nice, spider-web feel to it. DNS is very hierarchal, which makes it a bit trickier in my opinion to decentralize.

Many will disagree, but I personally feel that encryption is not something to put into the design. It should be an option at Layer 3, 4, or inside layer 5. Not to say that I'm against cryptography/signatures with route distribution and IP selection/authority, but we should not make it more ridiculously complex than it already is. Besides, encryption on Linux ISOs and public Reddit pages is usually just needless overhead.

I'm going to assume we have multiple physical networks interconnects in some router devices, connecting to other routers' devices. I'm also going to forget about BGP or an alternative right now. BATMAN does look pretty cool, though.

As far as IP address distribution goes, I think we should pick an IPv6 prefix. Maybe 1337::/16, 2364::/16, or a /32 if you're feeling stingy. Something not in use, something unlikely to be used (so maybe 1337::/16 is a bad idea). We should keep our feet out of existing IP space so we can get the best of both worlds. With enough usage, no one will want to step into our realm, even if unregistered and we haven't paid anyone a dime.

So who gets what address blocks? I'd say they should be allocated in /48s and /64s, mostly /64s. Or maybe just /64s, I'm not sure. We can have a "central authority" of Redditors who gives them out or some sort of open registry. We could also expect people to be harmonious and have no true central authority. People just start using "random enough" prefixes and hope no one else is doing the same... probably not given the number of bits. Participation in registries is more optional but encouraged.

The last option I've thought of is cryptographically-backed blocks. Like Namecoins, but bootstrapping poses a huge problem and I think chain length might make things tricky in the long run. If we had some sort of bitcoin-y resource with which people registered blocks (or were given arbitrarily with the ipcoin), you could have a private key on the router which advertises its routes and is validated against the public. This could be mixed in later with unencrypted routes to bootstrap with, or be required from the start.

I'm leaning towards anarchistic allocation in 2364::/16 or something like namecoins. Or maybe a mix, with weights for validated routes. People could use any number of open registries for claiming their prefixes. Of course, none would be all-encompassing, but maybe some would rise to be de facto. At least, we could get up and running, only to quibble about such things later. /64 conflicts from across the world could get interesting, though. Namecoin-style would help authenticate and keep people from claiming prefixes that aren't theirs, but we don't have too much of that problem as it is with very vulnerable BGP. Government sabotage may be a potential issue to consider, but could likely be fought by savvy network administrators as it comes up.

Route propagation could be done through some layer 2 broadcast system, relying on recursively distributed routes. Bootstrapping to more advanced functionality over layer 5 traversing layer 3 might be possible, too.

As far as DNS goes, here are some simpler options over my others: Keep the same system and hope for the best. Ignore root servers and distribute lists of TLD servers. So decentralize root, centralize TLD name servers. I don't mind this, even if it's imperfect. Combo of traditional DNS and nameserver resolution through Namecoin.

As far as Namecoins go, I'm very concerned about its scalability. I don't know much about the architecture, but it seems like you need a full chain with all NS records per TLD (just .bit for the moment, I think). This is impractical on little routers, even a little awkward on bigger ones. Maybe we could have ffxe::dns-style multicast listeners for Namecoin-aware recursive resolution? This gets a bit more into my other idea. Could also just point the routers at big DNS servers which you can hopefully trust, and do this work for you (which you'd be doing with multicast, but would be less manual). Very concerned about multi-GB blockchains in the future. I prefer recursive resolution when possible, rather than having every detail about anything since it's so much smaller.

So, here's where gets crazy. If we can find a way to mix a reliable multicast setup into our new interwebs, what if we resolved domains with multicast almost entirely?

ffxe:dnsprefix::/32 or less would be the overhead. Then we get 96 bits of pure DNS goodness. Or.. we could hope for the best and do ffxe::/16 and try not to conflict. Anyways, there are a few things we could do from here.

ffxe:DNS(I know it's not hex and invalid)::(TLD in hex, not sure what base xx format to use)

Could this work at all? Hmm. Seems like the DNS spec is case-sensitive, but everything acts as case-insensitive. Let's say it's insensitive. We have English letters a-z, hyphens, and numbers. That should be 37 characters if my math is right. We'd have other overhead, but per character, we should be able to fit it into 6 bits since 2^6 = 64. Half that at 5 bits is 32, which is just too little. 6 bits is pretty wasteful but may give us some more flexibility. We might need a ".", anyways.

96 available bits divided by 6 bits is 16 characters! Not bad at all, and there might even be room for optimization.

So how can we use this?

Bob coerces the software on his laptop to attempt to try pulling down a website. Let's say, reddit.com.

Super awesome DNS resolver library on the laptop could do this a few ways: Check /etc/resolv.conf and query the stated nameserver. Resolve to the local namecoin database. Multicast-resolve to the TLD's nameservers, so .com. Ignore hierarchy, and say that all domains are more or less their own entity if they "end" with a certain TLD. So it has logic to multicast-resolve for domains ending in .pzq/.com/.bit/whatever, going straight for the second level.

The last two are more exciting. Let's start with resolving the TLD. UDP packet destined for port 53 goes out to ffxe:dns::com. ffxe:dns::com writes back, directing you to ns1.reddit.com, with an IP of 2364::1337. You then query 2364::1337 and ask for your domain.

What if we go straight for the goods, and want reddit.com? UDP packet destined for port 53 with the usual DNS-protocol spheel, queries ffxe:dns::reddit.com for a resource record (actually not sure exactly what it's called off the top of my head). Reddit's nameservers respond, claiming to be authoritative, giving back an A (or AAAA if we made such a query), and stating the nameserver is itself at 2364::1337. If we wanted www.reddit.com, the resolver would then cache that record, then query 2364::1337 for www.reddit.com and probably give you something very similar.

We have many potential problems with a decentralized internet. What if connectivity is broken for a large region, and you can't reach everything needing to resolve the domain that points to your neighbor? We should consider the design of a fully connected, disconnected, and partially connected internet. TLD multicast (or maybe I'm thinking of anycast, having multiple TLD nameservers respond back is very bad) is nice, but what if not all TLDs get on board? It's also much more centralized, possibly more than we like.

If we go for second level-direct multicast resolution, the biggest limitation I can think of is 16 character domains. We could restrict second-level registration to only be for a certain TLD, so we go for fixed:dns::reddit, which saves us some letters. I'd lean towards even shorter domains over TLD-restricted, but alternatively, we could have different DNS prefixes to use other TLDs. Could also have ffxe:(16 bits identifier)(16 bit TLD lookup table):: domain.

So what about security in this sense? Anyone who pops up a multi-cast listener for the domain can easily respond, maybe before the other true server. For this, we could maybe implement Namecoin-style distributed keys for domains, as well as locally set keys for domains you trust and already have keys for. As far as specifics go, I've run DNSSEC and hate it. I used it years ago and found it to be far too bloated, complicated, and needless. I think DNSCurve is a much better option and makes a lot more sense. It's also much lighter and has tiny keys to distribute, versus the monsters of DNSSEC.

So pretty much, I haven't proposed any true model, but I've given a lot of specifics and what I lean towards. Here's an example world of how this might all play out:

SOPA passes, domains are blocked from most ISPs through packet inspection of DNS packets and doesn't forward queries, only responds with its own records pointing to A records with HTTP servers that serve "You've been blocked!" pages. Setting to other nameservers obviously won't help. Pushing DNS over an SSH tunnel with a VPS someplace that isn't SOPA-enforced might work.

Anyways, Alice is sick of this and wants to connect up with her cool Reddit friends. Has some sort of Linux 2U router which she connects to Jane's wireless AP, Bob's ethernet, and a PPP over serial to Jason. Jason has direct ethernet to Fred, and Fred is connected with Jane over fiber. Jane has a default route-worthy "internet connection", unlike the rest of them.

An unlikely scenario, but bear with me. Alice pops up her router and advertises herself as 2364::a1c1:3::/64. Super magical routing daemons over layer 2 convince her router to add routes for 2364::jane/64 (getting lazy, I know it's invalid) over wireless, 2364::b0b/64 over ethernet, and 2364::j4s0:n::/64 over the slow serial connection. Also has an alternative-priority route for Jason which jumps through Jane's wireless, to Fred's fiber, and then Jason over ethernet. Depending on the algorithms in place, it might pick that route over serial, which it probably should. Alice will also have a route to Fred through Jane or Jason and is not directly connected. Her default route is to Jane, for all other traffic.

Alice is a kind internet user and wishes to "register" her prefix with the Reddit Registry. The Reddit Registry hasn't been blocked by any means and is available at redditregistry.com. Typical routing and DNS resolution connect her with the registry to lay claim to 2364::a1c1:e::/64, hoping no other Alices take it.

Then, Alice wants to get to Bob's public FTP server with music on it. She knows it's accessible via music.bob.pzq, so runs lftp ftp://music.bob.pzq. /etc/resolv.conf is set to resolve to localhost, which is a modified dnrb that works with multicast-typical TLDs. Her libc of choice sends a query to dnrb. dnrb sends the query over ffxe::b0b.pzq. The router has some sort of multicast routing daemon that already worked out listeners from the other networks (or, it queries all locally connected or just the default for where to send it). The router forwards traffic to Bob over ethernet. Bob's router broadcasts the request to Bob's layer 2 switching network. Bob's music server responds to the request saying the nameserver is ns.b0b.pzq at 2364::b0b:50 and sends the query back to Bob's router, back over ethernet, then to Alice's router. Alice's device then queries 2364::b0b:50 for music.bob.pzq, and it gives a AAAA record for 2364::b0b:50. Alice SYNs to 2364::b0b:50, and it goes from there.

So I didn't really discuss security. We could have had DNSCurve on Bob's domain server whose key was published over namecoin, and possibly the nameserver/IP info as well. The DNS query would then be sent as DNSCurve-encrypted, only to be opened and responded to by a legitimate private key on Bob's FTP server. For ensuring the routes are legitimate, something like namecoin could also be used after an initial bootstrap, to prove Bob's router is "authoritative" for 2364:b0b::/64. These authoritative packets might be able to be passed over multiple hops, too. Routes advertised without signatures that have a Namecoin entry would be ignored.

For the FTP connection, IPSEC using opportunistic encryption with a certain key record in DNS could have been used. Alternatively, SFTP could be used safely.

I also don't quite understand how multicast would fit in. I've just played with it a little bit on link-local, never beyond. I think all that I've stated is doable.

In summary, I'd like a solution that is: As simple as can be, only complexifying as needed. Bootstrappable and does not create a chicken-and-egg problem. Efficient, and as viable for point to point as for mesh (if possible, might not be). Encryption is on top, not on the bottom. Compatible with the current internet.

I realize I've probably misused many technical terms and concepts, but I think most of the ideas hold somewhat true.

What do you guys think? Comment on Reddit, or come talk to me as sega01 on Freenode.

Editor's note: I'm not as often on Freenode. Email is best. This could use some updates that I probably won't do unless I'm bugged about it.