The details of networking -- like anything really -- can be quite complicated. But the basic ideas of how it all works are surprisingly simple, and that's what we're going to study.
The Internet is like a global phone system for computers: a computer can "call" another computer on the internet to get or send a little information. Suppose your laptop is connected to the internet, and you type "http://www.nytimes.com" into your browser -- what happens? Your computer contacts the computer "www.nytimes.com" -- placing a "call" in effect -- and sends a request for the main web page. The request is small, about 1KB (1 kilobyte). The www.nytimes.com machine sends back a large response which is the web page -- maybe 200KB -- and ends the call. Your browser gets back all this data and formats it for your screen so you can read the text, click links etc. We'll look at this fetch-web-page example a few different ways to see how the internet works.
Ethernet is an extremely common and influential wired networking standard, so we'll start there. Ethernet is a LAN (Local Area Network) networking technology, used to connect small groups of, say, 20 computers. Ethernet cable lengths are typically limited to 100 meters, in keeping with its "local" orientation. A typical LAN application is networking the computers in one room or in one floor of a building. The most common form of ethernet wiring is 100base-T which uses wires with a "RJ-45" connector the size of your pinkie finger, like a larger phone wire plug.
We'll start with the simplest case of two computer connected with an ethernet cable, and we want to send a 50KB jpeg image file from one computer to the other. This is the "one-hop" case .. networking between two computers separated only by an ethernet cable. Below we'll scale this up to the full Internet case of two computers on separate sides of the world. The first question is: how are the bytes of the image file on one computer sent to the other computer over the rather skinny ethernet cable?
For transmission, the 50KB of the image is divided into packets. The packet is the natural unit of transmission within networking. In this case, say each packet is about 1500 bytes (a typical ethernet size), then the bytes of the 50KB image could be divided into about 32 packets. It is not required that all the packets be the same size, just that every byte of the image is sent in one packet or another.
Ethernet provides a basic facility to transmit a packet between two computers connected by the ethernet cable (I'm simplifying a a bit here). Here's one way it could work. Say we have a packet of 1500 bytes of information we want to send. Each byte is 8 bits, so that's 12000 bits to send, where each bit is a 0 or 1. Here's an oversimplification that captures how it works: the ethernet cable contains two wires connecting the computers. The sending computer could read through the 12000 bits in order, and for each 1 bit, put 3 volts between the wires, and for each 0 bit, put 0 volts between the wires. The receiving computer can follow along, noting the 3v/0v pattern on the wires over time and so receive the 12000 bits. In reality the ethernet contains 4 pairs of wires and supports sending information in both directions. However, this pattern of going through the bits and varying the voltage to "send" each bit is basically how it works.
Math aside: currently the most common speed of ethernet is "100 megabit" ethernet, which signals at a rate of 100 million bits per second (i.e. this is that rate that 0 vs. 1 voltages are put on the wires). At that rate our 12000 bit packet will take 12000 / 100 million = .00012 seconds, or 120 microseconds. Multiplying by the 32 packets, that's about .0038 seconds for all the packets. In reality, there is significant overhead for each packet, so we'll double this and figure sending the whole 50KB image takes about .008 seconds, or 8 milliseconds. The faster standard "gigabit ethernet" (also known as "gigE") runs about 10 times faster. Note that wireless connections tend to be slower than wired connections.
We divide larger data into into packets for networking for 2 main reasons:
Suppose your computer is currently sending an email, and sending a web page request, and uploading a file, and this is all going over your one ethernet cable. How can all these simultaneous activities share the one ethernet connection? Because each of these "send" operations is divided into packets, the ethernet can operate like a big input hopper .. all the packets to send are dumped in there, and the ethernet can work through them, sending one packet at a time. So it might send a packet of your file upload, then two packets of your email, then another packet of the file upload, and so on. Because the packets are so small, each taking very little time, it all appears to be happening simultaneously.
Each packet includes extra checksum bytes, so the receiver of the packet can detect if some of the bits in the packet got corrupted in transmission. A simple example checksum scheme would be: go through all the bytes, and add them all up. The checksum is the last 3 digits of the sum of all the bytes; send that checksum along with the rest of the packet data. The receiver can do the same computation -- adding up all the bytes -- to check that they get the same checksum. The actual checksum algorithm is more complex than just adding up the bytes, and is more capable of detecting errors. The checksum is probabilistic, not detecting 100% of errors; there is a minute chance that an error occurs but the checksum does not catch it.
The checksum allows the the receiver to notice which packets did not come through right, and get the sender to re-send just those packets. Most packets get through fine, but re-sending few happens all the time.
Ethernet, even in its original simple form, is an interesting and elegant design for a LAN network, connecting a bunch of local computers together with one shared wire. It demonstrates the paradigm of getting many actors to cooperate without any central control. Ethernet was created by Bob Metcalfe at Xerox PARC in 1975. Here is how the original Ethernet worked, although newer versions are more complex and have higher performance.
Ethernet is a nice example of getting multiple distributed actors to cooperate without relying on a central authority.
The LAN examples above solve the one-hop problem -- getting a packet from one computer to a nearby computer with, say, an ethernet connection between the two. Now we will scale the problem up to send packets between any two computers on earth.
The worldwide Internet is built on the TCP/IP family of standards (Transmission Control Protocol / Internet Protocol) which solves the larger problem of sending packets between computers across the whole internet. These are free and open, vendor-neutral standards which is one reason why they have been so incredibly successful.
Every computer on the internet has an "IP address" that identifies it (like a phone number). The IP address is 4 bytes, written between dots, like "171.64.2.3". The left part of the address encodes in part where that IP address is in the whole internet -- for example any 171.64.(anything) is part of Stanford (like the area code of a phone number). More specifically, in my part of the Gates building, all the IP addresses begin 171.64.64.XX varying only in that last byte.
Sandra Bullock: there's a blooper in the movie The Net where the IP address "23.75.345.200" is shown. This is not a good address, since 345 is larger than the largest possible byte value which is 255.
The most common way for a computer to be "on the internet" is to establish a connection with a "router" which is already on the internet. The computer establishes a connection via, say, ethernet to communicate packets with the router. The router is "upstream" of the computer, connecting the computer to the whole internet. For example, the computer in my Stanford office has IP address 171.64.64.166, and it has a one-hop ethernet connection to its router upstream at 171.64.64.1, and this router handles packets for my computer. As a convention, the router will often have the ".1" address, ending in x.x.x.1, and the computers it supports will use the larger numbers like x.x.x.166.
Suppose my computer at 171.64.64.166 wants to send a packet to a computer at 173.255.219.70 somewhere out on the internet (actually that's the codingbat.com server I administer). The Internet is essentially made of a big web of routers talking to each other.
1. My computer prepares an IP packet which includes in particular From:/To: information as IP addresses, like this: (IP Packet From:171.64.64.166 To:173.255.219.70 data data data data).
2. My computer sends that IP packet to my upstream router, one hop, over ethernet. This is the "first hop" of the packet on its journey.
3. The router looks at the To:/From: of the packet and forward it to the next router, one hop closer to its ultimate destination. Essentially, the router has its own upstream router which is bigger and knows more about the layout of the internet. The packet is forwarded, one hop at a time, until it reaches its ultimate destination. Each router does not need to know the whole route to the destination; each router just needs to know which way to send the packet to get it one-hop closer to its destination. The routers look at the left part of the IP address to get the packet to the right neighborhood -- 171.64 -- with the right part of the address -- .64.166 -- coming into play only when the packet is near its ultimate destination.
The routing of a packet from your computer is like a capillary/artery system .. your computer is down at the capillary level, your packet gets forwarded up to larger and larger arteries, makes its way over to the right area, and then down to smaller and smaller capillaries again, finally arriving at its destination. The ultimate destination puts all the packets back together in the right order to recover the original image file or whatever. The routers at the ends have a trivial upstream/downstream configuration, so the next hop for a packet is pretty simple. More central routers tend to have several possible outgoing connections, so they have a more complicated choice about which link to use for the next hop.
The routers, collectively, measure what networks are reachable over what links, and dynamically adjust what links to use for each packet. One simple metric would be to route packets the way that takes the fewest number of hops. In reality, the metrics used are more complex than this. The routing system resilient to router hardware failures, overloading of certain links due to normal traffic, and links going down. The path taken by an IP packet can change from minute to minute. The routers are another example of a distributed, cooperative system. The old joke is that the backhoe is the IP packet's natural predator in the wild. I supposed authoritarian/dictatorship type governments that wish to control what their populations can see and do would be the other natural enemy of the IP packet.
Note that my computer does not need to know the layout of the internet. My computer just needs to have a connection to its upstream router, and the router, and its upstream router etc., will handle the routing from there.
Very broadly speaking, most data you get or send on the Internet goes in packets which take more than 10 but less than 30 hops from origin to destination.
Part of the TCP/IP standard provides end-to-end detection and correction for the packets. Each packet has checksum information, so the recipient can detect if the packet is good or corrupted. As part of TCP/IP routing, the ultimate recipient sends back to the sender ACK (acknowledgement) information, letting the sender know which packets have been successfully received (the packets have serial numbers 0, 1, 2 ..). In this way, the sender knows which packets to re-send. This is why networking when you are just within range of a wireless access point can be so slow. Because the radio signals are at their range limit, the packets are more likely to get corrupted, require many re-sends. You see slowness, not actually corrupt/incorrect data. The ACK traffic also provides "flow control" -- slowing the sender down if it is sending packets faster than the destination can process them.
There are actually multiple layers of checksums protecting the data. Ethernet has a checksum scheme that operates just one hop at a time. TCP/IP, working at the level of the entire route, has an independent checksum scheme. These error detection and correction schemes are complementary, making unfixed errors unlikely, although no impossible.
So what does it mean for a computer to be on the internet? Typically it means the computer has established a connection with a router. The commonly used DHCP standard (Dynamic Host Configuration Protocol), facilitates connecting to a router; establishing a temporary connection, and the router gives your computer an IP address to use temporarily. Typically DHCP is used when you connect to a Wi-Fi access point.
Exercise: bring up the networking control panel of your computer. It should show what IP address you are currently using and the IP address of your router. You will probably see some text mentioning that DHCP is being used.
"Ping" is a old and very simple internet utility. Your computer sends a "ping" packet to any computer on the internet, and the computer responds with a "ping" reply (not all computers respond to ping). In this way, you can check if the other computer is functioning and if the network path between you and it works. As a verb, "ping" has now entered regular English usage, meaning a quick check-in with someone.
Exercise: Try pinging www.google.com or 171.64.64.166 (nick's desk computer). On the Mac, try Applications - Utilities - Network Utility. Or on the command line of your computer, there is probably a "ping" program.
Traceroute is a program that will attempt to identify all the routers in between you and some other computer out on the internet. Some routers are visible to traceroute and some not, so it does not provide completely reliable output. However, it is a neat reflection of how IP routing works. Here's an example traceroute from my office to codingbat.com:
$ traceroute codingbat.com # typing this at the command line traceroute to codingbat.com (173.255.219.70), 30 hops max, 40 byte packets 1 yoza-vlan64.stanford.edu (171.64.64.2) 1.547 ms 0.622 ms 0.441 ms 2 bbra-rtr-a.stanford.edu (171.64.255.129) 0.462 ms 0.598 ms 0.454 ms 3 boundarya-rtr.stanford.edu (172.20.4.2) 0.518 ms 0.59 ms 0.485 ms 4 dca-rtr.stanford.edu (68.65.168.51) 0.471 ms 0.482 ms 0.494 ms 5 dc-svl-agg1--stanford-10ge.cenic.net (137.164.50.157) 0.848 ms 0.841 ms 0.852 ms 6 dc-svl-core1--svl-agg1-10ge.cenic.net (137.164.47.121) 1.38 ms 1.174 ms 1.114 ms 7 dc-svl-px1--svl-core1-10ge-1.cenic.net (137.164.46.205) 1.065 ms 1.024 ms 0.907 ms 8 te4-4-482-tr01-plalca01.transitrail.net (137.164.131.61) 1.133 ms 1.162 ms 1.143 ms 9 he-peer.plalca01.transitrail.net (137.164.130.58) 3.133 ms 2.61 ms 1.434 ms 10 10gigabitethernet1-2.core1.fmt1.he.net (66.160.158.241) 2.36 ms 9.455 ms 2.415 ms 11 linode-llc.10gigabitethernet2-3.core1.fmt1.he.net (64.62.250.6) 2.097 ms 2.103 ms 2.166 ms 12 li229-70.members.linode.com (173.255.219.70) 3.944 ms 19.323 ms 3.786 ms
The numbers down the left side are the number of "hops" to that machine. The "ms" figures are the number of milliseconds (1 thousandth of a second) it took for the send/reply. Notice that as the hops get further away, it does roughly take more milliseconds. The first few hops are Stanford addresses, then the route goes over some provider, until it arrives at Linode, which is the company that provides the hardware where codingbat.com currently lives. Small mystery: it seems like the first hop should be 171.64.64.1 which is the first router from my office; apparently that router is invisible to traceroute.
Wi-Fi is an enormously successful LAN standard, operating like an ethernet connection using radio waves. Formally, Wi-Fi refers to the 802.11 set of standards. Like ethernet, Wi-Fi is a LAN (local) networking technology, most commonly just connecting computers within about 100 feet. The common pattern is that a Wi-Fi access point combines radios with a router. Your computer connects with the Wi-Fi access point, registering its presence, getting information about what radio frequencies to use etc., and using DHCP to get an IP address for itself and the router IP address. Sending packets works similarly to ethernet, but now the packets are transmitted over the radio. In particular, the notion of a shared medium with everyone listening, and a "wait for silence before transmitting" carries over naturally from ethernet to radio. Fun fact: historically, ethernet itself was influenced by ideas from an earlier radio-based network, so it's just coming full circle.
Wi-Fi uses the "unlicensed" 2.4Ghz and 5Ghz radio bands, which anyone in the US is free to use so long as their signals are not very strong and they follow certain other rules. The weak signals are why Wi-Fi range is only about 100ft, and why Wi-Fi networks in adjacent houses mostly don't interfere with each other. Fun fact: 2.4Ghz radio noise can be emitted by microwave ovens, so you may notice that your networking doesn't work so well when you are running the microwave.
In contrast, the cell phone companies own their radio bands, and only they may use them. Using the free radio bands for short-range computer networking has worked out incredibly well. (Aside: there's an interesting economics research idea here .. usually pricing something at $0 doesn't work well, but in this case it works brilliantly. I suspect it works well since it is mostly limited to one house at a time. If there were a city-wide free radio band, it probably would be crowded with some spam-type traffic to the point of uselessness.)
The 802.11g standard does signaling at 54mbps (millions of bits per second), and the newer 802.11n standards can operate at speeds of 150Mbps and 300Mbps. Note that overhead for addressing, error detection etc. in these standards is high, so the usable bandwidth is roughly around half of the quoted numbers.
Here is a famous type of Wi-Fi router, the Linksys WRT54g. It combines a Wi-Fi access point with a router. It also has 4 RJ-45 jacks on the back, allowing computers to connect to it by ethernet as an alternative to Wi-Fi. This type of router is a bit famous, since the software that runs on it was available as open source. Hobbyists started with that open source code and added features and fixed bugs on it, ultimately resulting in a free DD-WRT software that was significantly better than the software available from the professional programmers working for Linksys itself -- a signal example that there are cases where the free community is better suited for a problem than the traditional non-free route. The Linksys corporation (now owned by Cisco) seems to have mixed feelings about this; I think they are making an error, failing to take advantage of the natural energy of the open source community for certain problems. We'll talk about this more when we talk about open source software. Note on the back of the router how there are 4 RJ-45 jacks for computers to connect to the router, and the 1 special jack (yellow wire shown) is for the upstream connection, for the router to connect to its router.
The simplest model of TCP/IP is that every computer on the internet has its own, unique IP address. However, there are a couple special forms of IP address that is widely used.
The IP address 127.0.0.1 is special, referring back to the computer itself. So if a piece of software wanted to send a packet to another piece of software on the same computer, it could address the packet to the IP address 127.0.0.1.
There are special reserved "non-routable" IP address forms -- the IP addresses 192.168.x.x and 10.x.x.x are designated as special and are reserved. These addresses are not used as the IP address of any computer on the public internet. Therefore, they can be used within an organization as a sort of private, internal-only IP address. In particular, addresses like 192.168.x.x will not be routed out on the open internet. They can only be used in a special way within an organization, as outlined below.
Normally the non-routable addresses would be an obscure detail, but they are used very often by Wi-Fi routers to share an internet connection (the Stanford campus wide Wi-Fi does it, for example). In particular, an Wi-Fi access point can have one IP address, e.g. 171.64.64.1, and yet allow all the "clients" connected to the Wi-Fi access point to share that one IP address. This is called NAT (Network Address Translation), and here's how it works:
Thus far IP networking has been defined in terms of IP addresses, like 171.64.64.166, and indeed at the low level that's how IP networking works. However, it does not make sense for humans to have to remember IP addresses. The DNS, Domain Name System, is an internet-wide system which maps human names like "codingbat.com" or "www.google.com" to IP addresses like 173.255.219.70 or 74.125.224.83. The human readable name "codingbat.com" is the "domain name" for that IP address.
When you type "http://codingbat.com" into your browser, the "codingbat.com" part is a domain name. The browser first does a quick DNS lookup, to map the domain name to an IP address. Then, with the IP address in hand, the browser can start sending packets to that site for you. TCP/IP networking uses IP addresses for everything, so your computer must look up the IP address of the other computer before it can initiate anything.
When you connect your computer to a router, the router also typically tells your computer what DNS servers to use for domain lookups (this is generally done through DHCP as part of connecting with the router). You really need both a router and a DNS server to be meaningfully on the internet. You may see a message "looking up codingbat.com..." at the bottom of your browser window briefly .. this corresponds to the DNS lookup. If the message just stays there .. your DNS setup may not be working, so the browser is just stuck at that step. Aside: Google provides free, highly reliable DNS servers at 8.8.8.8 and 8.8.4.4 -- you could use them if your DNS setup is having problem or for testing or whatever.
The DNS system is a distributed system; it does not depend on a single computer which knows all the domain names. Instead, there is a hierarchy of computers, each storing some of the domain names. The DNS servers cooperate, passing requests to each other, to collectively be able to look up any domain name.
Currently, maintaining a domain name in the system costs about $15/year. It's a first-come, first-serve system. However, if a company has a trademark, such as "McDonalds", then they may have priority for the mcdonalds.com domain. Many domain names are bought up by "domain squatters" who figure the domain would be valuable to someone, but they just sit on it uselessly in the hopes of selling it to someone someday. In my opinion, the domain squatters are leeches, not creating any value. However, I cannot think of a better system .. it's not like you could have some committee that decides who is worthy of having which domain name.
Networking equipment is usually rated by its bit "bandwidth" .. how fast can it pass bits when running flat out.
There is significant overhead in packet networking -- some bits are used up for addressing, checksums etc., and starting and ending each packet takes some time. Therefore, if you have a 1 Mbps connection, that does not mean that 1 million bits per second of your data will make it through .. there is 10-30% overhead. As a quick rule of thumb, if you figure each byte takes 10 bits (rather than the true number, 8), this allows for some overhead and makes the math easy. By that measure, a 1 Mbps connection can send about 100,000 bytes per second, i.e. about 100 KB per second.
A typical 2011 home ADSL (Asymmetric Digital Subscriber Line) connection in the US provides 1.5 Mbps connection downstream (receiving into the home), and 384 Kbps upstream (sending from the home) over the existing copper phone lines. The service is implemented by a "DSL modem" which connects to the phone line on one side, and provides an ethernet connection on the other. Note the asymmetry in speeds -- this is why downloading images is much faster than, say, uploading them when you send an email with an image in it. The current maximum ADSL speed is about 6Mbps, but this only works for houses that are near to the central phone-company location.
There is a business/competition problem here .. the local phone company has the wires to the home already, so its very hard to get competition (fortunately, cable-TV does provide one competitor in many areas). This is the so-called "last mile" problem. Countries with better policies have much faster and cheaper internet access to the home, such as 20 Mbps, with South Korea being the famous example. Yet another reason to dislike monopolies, which can be relied on to provide mediocre service at a high price.
The original ethernet design functioned with just a dumb wire connecting all the computers. However, the most common modern design is a hub-and-spoke configuration, where there is a cheap central hub/router, and each computer has its own ethernet cable leading to the hub. The hub increases the efficiency of the system by keeping the wires isolated, so that each computer can transmit at the same time without interference (contrast this to the original shared-wire ethernet). A typical arrangement in a building is that each office has an ethernet connection, and all those wires converge on a small "wiring closet", with all the wires connecting into a hub installed there.
