http-client

An HTTP/HTTPS client for Carp with streaming support, chunked transfer-encoding, and SSE-friendly response reading.

Built on socket, http, and tls.

Installation

(load "git@github.com:carpentry-org/http-client@0.5.4")

Requires OpenSSL for HTTPS support (via the tls library). Plain HTTP works without OpenSSL.

Usage

GET

(match (Client.get "https://example.com/")
  (Result.Success r) (println* (Response.body &r))
  (Result.Error e) (IO.errorln &e))

POST with headers

(match (Client.post "https://api.example.com/data"
         {@"Content-Type" [@"application/json"]}
         "{\"key\": 1}")
  (Result.Success r) (println* (Response.body &r))
  (Result.Error e) (IO.errorln &e))

Custom verb

(Client.request "PATCH" "https://api.example.com/x"
                {@"Authorization" [@"Bearer ..."]}
                "{}")

Timeouts and redirect limits

Create a RequestConfig to set connect/read timeouts (in seconds) and a redirect limit:

(let [cfg (RequestConfig.init 5 10 10)]  ; 5s connect, 10s read, 10 redirects
  (match (Client.get-with-config "https://example.com/" &cfg)
    (Result.Success r) (println* (Response.body &r))
    (Result.Error e) (IO.errorln &e)))

All -with-config variants accept a &RequestConfig as the last argument. A timeout of 0 or negative (the default) means no timeout. Connect-timeout applies only to plain HTTP; HTTPS connections go through TlsStream.connect, which does not support a timeout parameter.

Streaming

For chunked or long-running responses, use Client.request-stream to get a ResponseStream you can poll for chunks as they arrive:

(match (Client.request-stream "POST" url headers body)
  (Result.Success stream)
    (do
      (while-do true
        (match (ResponseStream.poll &stream)
          (Maybe.Nothing) (break)
          (Maybe.Just chunk) (IO.print &chunk)))
      (ResponseStream.close stream))
  (Result.Error e) (IO.errorln &e))

ResponseStream handles Transfer-Encoding: chunked automatically and implements the poll interface from the streams library.

Cookie jar

Use a CookieJar to store cookies from responses and replay them on subsequent requests automatically:

(let-do [jar (CookieJar.create)]
  (match (Client.get-with-jar "https://example.com/login" &jar)
    (Result.Success r) (println* (Response.body &r))
    (Result.Error e) (IO.errorln &e))
  ; jar now has cookies from the login response
  (match (Client.get-with-jar "https://example.com/dashboard" &jar)
    (Result.Success r) (println* (Response.body &r))
    (Result.Error e) (IO.errorln &e)))

The jar follows RFC 6265 §5.3 and §5.4. A cookie that arrives with no Domain attribute is host-only: it goes back to the host that set it and to no subdomain. A Domain attribute the responding host does not domain-match is rejected outright, and so is a single-label one such as Domain=com. On top of that the jar enforces path matching, the Secure flag, and expiry; cookies are deduplicated by name+domain+path and serialized longest path first. During redirects, cookies from every hop are stored and re-applied for each new URL.

Multipart uploads

(match (Client.post-multipart "https://example.com/upload"
                              (the (Map String (Array String)) {})
                              &[(Multipart.text-part "field" "value")
                                (Multipart.file-part "upload" "test.txt"
                                                     "text/plain"
                                                     "file contents")])
  (Result.Success r) (println* (Response.code &r))
  (Result.Error e) (IO.errorln &e))

post-multipart picks the boundary with Multipart.boundary-for, which checks it against the parts and extends it until it occurs in none of them. RFC 2046 §5.1.1 requires that, and without it any upload whose contents happen to contain the delimiter is split in the wrong places by the receiver.

To build the body yourself, pick the boundary the same way:

(let [parts [(Multipart.text-part "name" "Carp")]
      boundary (Multipart.boundary-for &parts)]
  (Client.post url
    {@"Content-Type" [(Multipart.content-type-header &boundary)]}
    &(Multipart.encode &parts &boundary)))

A CR or LF in a part name, filename or content type is percent-encoded as %0D and %0A, so an untrusted field name cannot inject header lines or a further part into the body. Quotes are backslash-escaped. Values without those characters are emitted unchanged.

API

Client

Function	Purpose
Client.get url	HTTP GET
Client.post url headers body	HTTP POST (auto-sets Content-Length)
Client.put url headers body	HTTP PUT
Client.del url	HTTP DELETE
Client.head url	HTTP HEAD
Client.patch url headers body	HTTP PATCH
Client.request verb url headers body	Generic request
Client.request-with-max-redirects verb url headers body n	Generic request with custom redirect limit
Client.request-stream verb url headers body	Returns a ResponseStream
Client.request-stream-with-max-redirects verb url headers body n	Streaming with custom redirect limit
Client.get-with-config url config	GET with request config
Client.post-with-config url headers body config	POST with request config
Client.put-with-config url headers body config	PUT with request config
Client.del-with-config url config	DELETE with request config
Client.head-with-config url config	HEAD with request config
Client.patch-with-config url headers body config	PATCH with request config
Client.post-multipart url headers parts	POST a multipart/form-data body
Client.post-multipart-with-config url headers parts config	Multipart POST with request config
Client.request-with-config verb url headers body config	Generic request with request config
Client.request-stream-with-config verb url headers body config	Streaming with request config
Client.get-with-jar url jar	GET with cookie jar
Client.post-with-jar url headers body jar	POST with cookie jar
Client.put-with-jar url headers body jar	PUT with cookie jar
Client.del-with-jar url jar	DELETE with cookie jar
Client.head-with-jar url jar	HEAD with cookie jar
Client.patch-with-jar url headers body jar	PATCH with cookie jar
Client.request-with-jar verb url headers body jar	Generic request with cookie jar
Client.request-stream-with-jar verb url headers body jar	Streaming with cookie jar
Client.request-with-jar-and-config verb url headers body jar config	Generic request with jar and config
Client.request-stream-with-jar-and-config verb url headers body jar config	Streaming with jar and config

All return (Result Response String) (or (Result ResponseStream String) for the streaming variants).

All methods follow HTTP redirects automatically (up to Client.default-max-redirects, which is 10). For 301/302/303 responses the method is changed to GET and the body is dropped. For 307/308 responses the original method and body are preserved. Use the -with-max-redirects variants to control the limit, or pass 0 to disable.

A relative Location is resolved against the URL of the hop that produced it, following RFC 3986 §5. A Location that carries its own scheme is followed as given.

Multipart

Function	Purpose
Multipart.text-part name value	A text form field
Multipart.file-part name filename content-type data	A file upload part
Multipart.boundary-for parts	A boundary that occurs in no part
Multipart.generate-boundary	A boundary from the clock, unchecked against any payload
Multipart.content-type-header boundary	The Content-Type value for a boundary
Multipart.encode parts boundary	The encoded body

RequestConfig

Function	Purpose
RequestConfig.init connect-timeout read-timeout max-redirects	Create a config (timeouts in seconds, 0 = none)
RequestConfig.default	Config with no timeouts and 10 max redirects

CookieJar

Function	Purpose
CookieJar.create	Create an empty jar
CookieJar.store! jar cookie	Store a cookie as a domain cookie, replacing duplicates by name+domain+path
CookieJar.store-response! jar response url	Store a response's cookies, applying RFC 6265 §5.3's origin checks
CookieJar.matching jar url	Return cookies matching the URL by domain, path, security, and expiry, longest path first
CookieJar.cookie-header jar url	Build a Cookie header value, or Nothing if no cookies match
CookieJar.apply-to-headers jar url headers	Add a Cookie header to the headers map
CookieJar.size jar	Number of stored cookies
CookieJar.clear! jar	Remove all cookies

Connection

A union of (Plain TcpStream) and (Secure TlsStream). Used internally for transport dispatch, but exposed in case you want lower-level control.

ResponseStream

Function	Purpose
ResponseStream.poll stream	Returns (Maybe String) — next decoded chunk, or Nothing when done
ResponseStream.close stream	Close the underlying connection
ResponseStream.status-code stream	The HTTP status code from the response headers

Notes

• All requests send Connection: close for predictable HTTP/1.1 behavior. No keep-alive support yet.
• HTTP/2 is not supported.

Testing

carp -x test/http-client.carp

Tests hit example.com and httpbin.org.

---

Have fun!