HttpClient - Character Encodings
Author: Adrian Sutton

##### Overview

- Features
- News
- Status
- Download
- Wiki
- User Guide
  - Authentication Guide
  - Character Encodings
  - Cookies
  - Exception Handling
  - Logging Guide
  - Methods
  - Optimization Guide
  - Preference Architecture
  - Redirects Handling
  - Sample Code
  - SSL Guide
  - Threading
  - Trouble Shooting
  - Tutorial
- Developer Guide

##### Project Documentation

- About
- Project Info
- Project Reports
- Development Process

##### Legend

- External Link
- Opens in a new window

## Introduction

This document provides an overview of how HttpClient handles character encodings and how to use HttpClient in an encoding safe way. There are three main sections: HTTP Headers, Request/Response Body and URLs.

## HTTP Headers

The headers of a HTTP request or response must be in `US-ASCII` format. It is not possible to use non `US-ASCII` characters in the header of a request or response. Generally this is not an issue however, because the HTTP headers are designed to facilite the transfer of data rather than to actually transfer the data itself.

One exception however are cookies. Since cookies are transfered as HTTP Headers they are confined to the `US-ASCII` character set. See the Cookie Guide for more information.

## Request/Response Body

The request or response body can be any encoding, but by default is `ISO-8859-1`. The encoding may be specified in the `Content-Type` header, for example:

> Content-Type: text/html; charset=UTF-8

In this case the application should be careful to use UTF-8 encoding when converting the body to a String or some characters may be corrupt. You can set the content type header for a request with the `addRequestHeader` method in each method and retrieve the encoding for the response body with the `getResponseCharSet` method.

If the response is known to be a String, you can use the `getResponseBodyAsString` method which will automatically use the encoding specified in the `Content-Type` header or `ISO-8859-1` if no charset is specified.

Note that some document types, such as HTML and XML allow the author to specify the content type of the file. In this case, you should consult the appropriate standards regarding how to resovle any conflicts in the reported charsets.

## URLs

The standard for URLs (RFC1738) explictly states that URLs may only contain graphic printable characters of the `US-ASCII` coded character set and is defined in terms of octets. The octets `80-FF` hexadecimal are not used in `US-ASCII` and the octets `OO-1F` hexadecimal represent control characters; characters in these ranges must be encoded.

Characters which cannot be represented by an 8-bit ASCII code, can not be used in an URL as there is no way to reliably encode them (the encoding scheme for URLs is based off of octets). Despite this, some servers do support varying means of encoding double byte characters in URLs, the most common technique seems to be to use UTF-8 encoding and encode each octet separately even if a pair of octets represents one character. This however, is not specified by the standard and is highly prone to error, so it is recommended that URLs be restricted to the 8-bit ASCII range.