An HTTP expert breaks down the most important architectural aspects of the web and HTTP, such as caching and servers, and explains how it all fits together.
In the first article of the series, we talked about the basic concepts of HTTP. Now that we have a foundation to build upon, we can talk about some of the architectural aspects of HTTP. There is more to HTTP than just sending and receiving data.
HTTP cannot function by itself as an application protocol. It needs infrastructure in the form of hardware and software solutions that provide different services and make communication over the World Wide Web possible and efficient.
Let’s start.
As the first article explained, the primary function of a web server is to store resources and to serve them upon receiving requests. You access the web server using a web client (aka web browser) and in return get the requested resource or change the state of existing ones. Web servers can be accessed automatically too, using web crawlers, that we will talk about later in the article.
Some of the most popular web servers out there, and probably the ones you have heard of, are Apache HTTP Server, NGINX, IIS, Glassfish, etc.
Web servers can vary from very simple and easy to use, to sophisticated and complicated pieces of software. Modern web servers are capable of performing a lot of different tasks. Basic tasks that a web server should be able to perform, include:
I will break up the basic flow of the web server in a few different phases. These phases represent a very simplified version of the web server flow.
When a web client wants to access the web server, it must try to open a new TCP connection. On the other side, the server tries to extract the IP address of the client. After that, it is up to the server to decide to open or close the TCP connection to that client.
If the server accepts the connection, it adds it to the list of existing connections and watches the data on that connection.
It can also close the connection if the client is not authorized or blacklisted (malicious) .
The server can also try to identify the hostname of the client by using the “reverse DNS.” This information can help when logging the messages, but hostname lookups can take a while, slowing the transactions.
When parsing the incoming requests, web servers parse the information from the message request line, headers, and body (if provided) . One thing to note is that the connection can pause at any time, and in that case, the server must store the information temporarily until it receives the rest of the data.
High-end web servers should be able to open many simultaneous connections. This includes multiple simultaneous connections from the same client. A typical web page can request many different resources from the server.
Since web servers are primarily the resource providers, they have multiple ways to map and access the resources.
The simplest way is to map the resource is to use the request URI to find the file in the web server’s filesystem. Typically, the resources are contained in a special folder on the server, called docroot. For example, docroot on a Windows server can be located on F: \WebResources\. If a GET request wants to access the file on the /images/codemazeblog.txt, the server translates this to F: \WebResources\images\codemazeblog.txt and returns that file in the response message. When more than one website is hosted on a web server, each one can have its separate docroot.
If a web server receives a request for a directory instead of a file, it can resolve it in a few ways. It can return an error message, return a default index file instead of the directory, or scan the directory and return the HTML file with content.
The server may also map the request URI to the dynamic resource – a software application that generates some result. There is a whole class of servers called application servers that are used to connect web servers to complicated software solutions and serve dynamic content.
Once the server has identified the resource it needs to serve, it forms the response message. The response message contains the status code, response headers, and response body if one was needed.
If the body is present in the response, the message usually contains the Content-Lengthheader describing the size of the body and the Content-Type header describing the MIME type of the returned resource.
After generating the response, the server chooses the client to which it needs to send the response. For the nonpersistent connections, the server needs to close the connection when the entire response message is sent.
When the transaction is complete, the server logs all the transaction information in the file. Many servers provide logging customizations.
Proxy servers (proxies) are the intermediary servers. They are often found between the web server and web client. Due to their nature, proxy servers need to behave both like a web client and a web server.
But why do we need Proxy servers? Why don’ t we just communicate directly between web clients and web servers? Isn’ t that much simpler and faster?
Well, simple it may be, but faster, not really. But we will come to that.
The forward proxy acts as a proxy for the client requesting a resource from a web server. It protects the client by filtering requests through the firewall or hiding the information about the client. The reverse proxy, on the other hand, works exactly the opposite way. It is usually placed behind the firewall and protects the web servers. For all the clients know, they talk to the real web server and remain unaware of the network behind the reverse proxy.
Proxies are very useful and their application is pretty wide. Let’s go through some of the ways the proxy servers are used.
As you can see, proxies can be very versatile and flexible.
Web caches are devices that automatically make copies of the requested data and save them in local storage.
By doing this, they can:
So you can clearly say that web caches improve both user experience and web server performance. And of course, potentially save a lot of money.
Here is what the basic web cache workflow looks like:
In time, as the HTTP matured, people found many different ways to use it. HTTP became useful as a framework to connect different applications and protocols.
Let’s see how.
Gateways refer to pieces of hardware that can enable HTTP to communicate with different protocols and applications by abstracting a way to get a resource. They are also called the protocol converters and are far more complex than routers or switches due to the usage of multiple protocols.
You can, for example, use a gateway to get the file over FTP by sending an HTTP request. Or you can receive an encrypted message over SSL and convert it to HTTP (Client-Side Security Accelerator Gateways) or convert HTTP to more secure HTTPS messages (Server-Side Security Gateways) .
Tunnels make use of the CONNECT request method. Tunnels enable sending non-HTTP data over HTTP. The CONNECT method asks the tunnel to open a connection to the destination server and to relay the data between client and server.
