HttpClient - HttpClient preference architecture and configuration guide
Author: Oleg Kalnichevski

##### Overview

- Features
- News
- Status
- Download
- Wiki
- User Guide
  - Authentication Guide
  - Character Encodings
  - Cookies
  - Exception Handling
  - Logging Guide
  - Methods
  - Optimization Guide
  - Preference Architecture
  - Redirects Handling
  - Sample Code
  - SSL Guide
  - Threading
  - Trouble Shooting
  - Tutorial
- Developer Guide

##### Project Documentation

- About
- Project Info
- Project Reports
- Development Process

##### Legend

- External Link
- Opens in a new window

## Table of contents

- HttpClient preference architecture
  - HTTP parameters
  - HTTP parameter hierarchy
- Supported parameters
  - HTTP method parameters
  - HTTP connection parameters
  - HTTP connection manager parameters
  - HTTP client parameters

## HttpClient preference architecture

Quality and extent of the `HTTP/1.0` and `HTTP/1.1` spec compliance vary significantly among commonly used HTTP agents and HTTP servers. That requires of HttpClient to be able to

- mimic (mis-)behavior of widely used web browsers;
- support flexible and configurable level of leniency toward non-critical protocol violations especially in those gray areas of the specification subject to different, at times conflicting, interpretations;
- apply a different set of parameters to individual HTTP methods, hosts, or client instances using common interface;

### HTTP parameters

As of version 3 HttpClient sports a new preference API based on HttpParams interface. All major components of the HttpClient toolkit (agents, host configurations, methods, connections, connection managers) contain a collection of HTTP parameters, which determine the runtime behavior of those components.

    HttpClient httpclient = new HttpClient();
    HttpVersion ver = (HttpVersion)httpclient.getParams().getParameter("http.protocol.version");

In a nutshell HTTP parameters is a collection of name/object pairs that can be linked with other collections to form a hierarchy. If a particular parameter value has not been explicitly defined in the collection itself, its value will be drawn from the upper level collection of parameters.

    HttpClient httpclient = new HttpClient();
    httpclient.getParams().setParameter("http.protocol.version", HttpVersion.HTTP_1_1);
    httpclient.getParams().setParameter("http.socket.timeout", new Integer(1000));
    httpclient.getParams().setParameter("http.protocol.content-charset", "UTF-8");

    HostConfiguration hostconfig = new HostConfiguration();
    hostconfig.setHost("www.yahoo.com");
    hostconfig.getParams().setParameter("http.protocol.version", HttpVersion.HTTP_1_0);

    GetMethod httpget = new GetMethod("/");
    httpget.getParams().setParameter("http.socket.timeout", new Integer(5000));

    try {
      // Internally the parameter collections will be linked together
      // by performing the following operations:
      // hostconfig.getParams().setDefaults(httpclient.getParams());
      // httpget.getParams().setDefaults(hostconfig.getParams());
      httpclient.executeMethod(hostconfig, httpget);
      System.out.println(httpget.getParams().getParameter("http.protocol.version"));
      System.out.println(httpget.getParams().getParameter("http.socket.timeout"));
      System.out.println(httpget.getParams().getParameter("http.protocol.content-charset"));
    } finally {
      httpget.releaseConnection();
    }

The code above will produce the following output:

    HTTP/1.0
    5000
    UTF-8

When resolving a parameter HttpClient uses the following algorithm:

- start parameter lookup from the lowest level at which this parameter applies
- if the parameter is undefined at the current level, defer its resolution to the next level up in the hierarchy
- return parameter value from the lowest level in the hierarchy the parameter defined at
- return null if the parameter is undefined

This architecture enables the users to define generic parameters at a higher level (for instance, at the agent level or host level) and selectively override specific parameters at a lower level (for instance, at the method level). Whenever a parameter is not explicitly defined at a given level, the defaults of the upper levels will apply.

### HTTP parameter hierarchy

Presently HttpClient provides the following parameter hierarchy:

    global--+                            | DefaultHttpParams
            |                            |
          client                         | HttpClient
            |                            |
            +-- connection manager       | HttpConnectionManager
            |     |                      |
            |     +-- connection         | HttpConnection
            |                            |
            +-- host                     | HostConfiguration
                  |                      |
                  +-- method             | HttpMethod

## Supported parameters

### HTTP method parameters

Applicable at the following levels: global -> client -> host -> method

Name | Type | Description | Default
--- | --- | --- | ---
http.useragent | String | The content of the `User-Agent` header used by the HTTP methods. | official release name, e.g. "Jakarta Commons-HttpClient/3.0"
http.protocol.version | HttpVersion | The HTTP protocol version used per default by the HTTP methods. | `HttpVersion.HTTP_1_1`
http.protocol.unambiguous-statusline | Boolean | Defines whether HTTP methods should reject ambiguous HTTP status line. | `<undefined>`
http.protocol.single-cookie-header | Boolean | Defines whether cookies should be put on a single response header. | `<undefined>`
http.protocol.strict-transfer-encoding | Boolean | Defines whether responses with an invalid `Transfer-Encoding` header should be rejected. | `<undefined>`
http.protocol.reject-head-body | Boolean | Defines whether the content body sent in response to `HEAD` request should be rejected. | `<undefined>`
http.protocol.head-body-timeout | Integer | Sets period of time in milliseconds to wait for a content body sent in response to `HEAD` response from a non-compliant server. If the parameter is not set or set to `-1` non-compliant response body check is disabled. | `<undefined>`
http.protocol.expect-continue | Boolean | Activates 'Expect: 100-Continue' handshake for the entity enclosing methods. The 'Expect: 100-Continue' handshake allows a client that is sending a request message with a request body to determine if the origin server is willing to accept the request (based on the request headers) before the client sends the request body. The use of the 'Expect: 100-continue' handshake can result in noticeable performance improvement for entity enclosing requests (such as `POST` and `PUT`) that require the target server's authentication. 'Expect: 100-continue' handshake should be used with caution, as it may cause problems with HTTP servers and proxies that do not support `HTTP/1.1` protocol. | `<undefined>`
http.protocol.credential-charset | String | The charset to be used when encoding credentials. If not defined then the value of the 'http.protocol.element-charset' should be used. | `<undefined>`
http.protocol.element-charset | String | The charset to be used for encoding/decoding HTTP protocol elements (status line and headers). | 'US-ASCII'
http.protocol.content-charset | String | The charset to be used for encoding content body. | 'ISO-8859-1'
http.protocol.cookie-policy | String | The cookie policy to be used for cookie management. | `CookiePolicy.RFC_2109`
http.protocol.warn-extra-input | Boolean | Defines HttpClient's behavior when a response provides more bytes than expected (specified with `Content-Length` header, for example). Such surplus data makes the HTTP connection unreliable for keep-alive requests, as malicious response data (faked headers etc.) can lead to undesired results on the next request using that connection. If this parameter is set to `true`, any detection of extra input data will generate a warning in the log. | `<undefined>`
http.protocol.status-line-garbage-limit | Integer | Defines the maximum number of ignorable lines before we expect a HTTP response's status code. With HTTP/1.1 persistent connections, the problem arises that broken scripts could return a wrong `Content-Length` (there are more bytes sent than specified). Unfortunately, in some cases, this is not possible after the bad response, but only before the next one. So, HttpClient must be able to skip those surplus lines this way. Set this to `0` to disallow any garbage/empty lines before the status line. To specify no limit, use `Integer#MAX_VALUE`. | `<undefined>`
http.socket.timeout | Integer | Sets the socket timeout (`SO_TIMEOUT`) in milliseconds to be used when executing the method. A timeout value of zero is interpreted as an infinite timeout. | `<undefined>`
http.method.retry-handler | HttpMethodRetryHandler | The method retry handler used for retrying failed methods. For details see the Exception handling guide. | default implementation
http.dateparser.patterns | Collection | Date patterns used for parsing. The patterns are stored in a Collection and must be compatible with SimpleDateFormat. | 'EEE, dd MMM yyyy HH:mm:ss zzz', 'EEEE, dd-MMM-yy HH:mm:ss zzz', 'EEE MMM d HH:mm:ss yyyy', 'EEE, dd-MMM-yyyy HH:mm:ss z', 'EEE, dd-MMM-yyyy HH-mm-ss z', 'EEE, dd MMM yy HH:mm:ss z', 'EEE dd-MMM-yyyy HH:mm:ss z', 'EEE dd MMM yyyy HH:mm:ss z', 'EEE dd-MMM-yyyy HH-mm-ss z', 'EEE dd-MMM-yy HH:mm:ss z', 'EEE dd MMM yy HH:mm:ss z', 'EEE,dd-MMM-yy HH:mm:ss z', 'EEE,dd-MMM-yyyy HH:mm:ss z', 'EEE, dd-MM-yyyy HH:mm:ss z'
http.method.response.buffer.warnlimit | Integer | The maximum buffered response size (in bytes) that triggers no warning. Buffered responses exceeding this size will trigger a warning in the log. If not set, the limit is 1 MB. | `<undefined>`
http.method.multipart.boundary | String | The multipart boundary string to use in conjunction with the MultipartRequestEntity. When not set a random value will be generated for each request. | `<undefined>`

Whenever a parameter is left undefined (no value is explicitly set anywhere in the parameter hierarchy) HttpClient will use its best judgment to pick up a value. This default behavior is likely to provide the best compatibility with widely used HTTP servers.

### HTTP connection parameters

Applicable at the following levels: global -> client -> connection manager -> connection

Name | Type | Description | Default
--- | --- | --- | ---
http.socket.timeout | Integer | The default socket timeout (`SO_TIMEOUT`) in milliseconds which is the timeout for waiting for data. A timeout value of zero is interpreted as an infinite timeout. This value is used when no socket timeout is set in the HTTP method parameters. | `<undefined>`
http.tcp.nodelay | Boolean | Determines whether Nagle's algorithm is to be used. The Nagle's algorithm tries to conserve bandwidth by minimizing the number of segments that are sent. When applications wish to decrease network latency and increase performance, they can disable Nagle's algorithm (by enabling `TCP_NODELAY`). Data will be sent earlier, at the cost of an increase in bandwidth consumption and number of packets. | `<undefined>`
http.socket.sendbuffer | Integer | The value to set on Socket.setSendBufferSize(int). This value is a suggestion to the kernel from the application about the size of buffers to use for the data to be sent over the socket. | `<undefined>`
http.socket.receivebuffer | Integer | The value to set on Socket.setReceiveBufferSize(int). This value is a suggestion to the kernel from the application about the size of buffers to use for the data to be received over the socket. | `<undefined>`
http.socket.linger | Integer | The linger time (`SO_LINGER`) in seconds. This option disables/enables immediate return from a close() of a TCP Socket. Enabling this option with a non-zero Integer timeout means that a close() will block pending the transmission and acknowledgement of all data written to the peer, at which point the socket is closed gracefully. Value `0` implies that the option is disabled. Value `-1` implies that the JRE default is used. | `<undefined>`
http.connection.timeout | Integer | The timeout until a connection is established. A value of zero means the timeout is not used. | `<undefined>`
http.connection.stalecheck | Boolean | Determines whether stale connection check is to be used. Disabling stale connection check may result in slight performance improvement at the risk of getting an I/O error when executing a request over a connection that has been closed at the server side. | `<undefined>`

Whenever a parameter is left undefined (no value is explicitly set anywhere in the parameter hierarchy) HttpClient will use its best judgment to pick up a value. This default behavior is likely to provide the best compatibility with widely used HTTP servers.

### HTTP connection manager parameters

Applicable at the following levels: global -> client -> connection manager

Name | Type | Description | Default
--- | --- | --- | ---
http.connection-manager.max-per-host | Map | Defines the maximum number of connections allowed per host configuration. These values only apply to the number of connections from a particular instance of HttpConnectionManager. This parameter expects a value of type `Map`. The value should map instances of `HostConfiguration` to `Integer`s. The default value can be specified using `ANY_HOST_CONFIGURATION`. | `<undefined>`
http.connection-manager.max-total | Integer | Defines the maximum number of connections allowed overall. This value only applies to the number of connections from a particular instance of HttpConnectionManager. | `<undefined>`

Whenever a parameter is left undefined (no value is explicitly set anywhere in the parameter hierarchy) HttpClient will use its best judgment to pick up a value. This default behavior is likely to provide the best compatibility with widely used HTTP servers.

### Host configuration parameters

Applicable at the following levels: global -> client -> host

Name | Type | Description | Default
--- | --- | --- | ---
http.default-headers | Collection | The request headers to be sent per default with each request. This parameter expects a value of type Collection. The collection is expected to contain HTTP headers | `<undefined>`

Whenever a parameter is left undefined (no value is explicitly set anywhere in the parameter hierarchy) HttpClient will use its best judgment to pick up a value. This default behavior is likely to provide the best compatibility with widely used HTTP servers.

### HTTP client parameters

Applicable at the following levels: global -> client

Name | Type | Description | Default
--- | --- | --- | ---
http.connection-manager.timeout | Long | The timeout in milliseconds used when retrieving an HTTP connection from the HTTP connection manager. 0 means to wait indefinitely. | `<undefined>`
http.connection-manager.class | Class | The default HTTP connection manager class. | `SimpleHttpConnectionManager` class
http.authentication.preemptive | Boolean | Defines whether authentication should be attempted preemptively. See authentication guide. | `<undefined>`
http.protocol.reject-relative-redirect | Boolean | Defines whether relative redirects should be rejected. Although redirects are supposed to be absolute it is common internet practice to use relative URLs. | `<undefined>`
http.protocol.max-redirects | Integer | Defines the maximum number of redirects to be followed. The limit on number of redirects is intended to prevent infinite loops. | `<undefined>`
http.protocol.allow-circular-redirects | Boolean | Defines whether circular redirects (redirects to the same location) should be allowed. The HTTP spec is not sufficiently clear whether circular redirects are permitted, therefore optionally they can be enabled. | `<undefined>`

Whenever a parameter is left undefined (no value is explicitly set anywhere in the parameter hierarchy) HttpClient will use its best judgment to pick up a value. This default behavior is likely to provide the best compatibility with widely used HTTP servers.