Imported Upstream version 0.5.0

parents
test/data
test/scratch
gitlab-workhorse
before_script:
- rm -rf /usr/local/go
- apt-get update -qq
- apt-get install -y curl unzip bzip2
- curl -O https://storage.googleapis.com/golang/go1.5.2.linux-amd64.tar.gz
- echo 'cae87ed095e8d94a81871281d35da7829bd1234e go1.5.2.linux-amd64.tar.gz' | shasum -c -
- tar -C /usr/local -xzf go1.5.2.linux-amd64.tar.gz
- export PATH=/usr/local/go/bin:$PATH
test:
script: make clean test
# Changelog for gitlab-workhorse
Formerly known as 'gitlab-git-http-server'.
0.5.0
Send ALL GitLab requests through gitlab-workhorse.
0.4.2
Return response to client when uploading Git LFS object.
0.4.1
Add support for Build Artifacts and Git LFS. The GitLab-Workhorse
offloads file uploading and downloading by providing support for
rewriting multipart form data and X-Sendfile.
Other changes:
- add header Gitlab-Workhorse to all requests to indicate from where
they originated
0.4.0
Rename the project to gitlab-workhorse. The old name had become too
specific.
Other changes:
- pass LD_LIBRARY_PATH to Git commands
- accomodate broken HTTP clients by spelling 'Www-Authenticate' as
'WWW-Authenticate'
0.3.1
Add support for Unix domain socket connections to the authBackend.
0.3.0
In 0.3.0 we also handle 'git archive' downloads for GitLab 8.1+.
This has lead to some breaking API changes, making 0.3.0 incompatible
with GitLab 8.0. We now expect the 'auth backend' (GitLab) to
provide us with much more information about each request, such as
the path on disk to the Git repository the client is requesting.
This makes the REPO_ROOT command line argument obsolete.
0.2.14
This is the last version that works with GitLab 8.0.
\ No newline at end of file
The MIT License (MIT)
Copyright (c) 2015 GitLab B.V.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
PREFIX=/usr/local
VERSION=$(shell git describe)-$(shell date -u +%Y%m%d.%H%M%S)
gitlab-workhorse: $(wildcard *.go)
go build -ldflags "-X main.Version ${VERSION}" -o gitlab-workhorse
install: gitlab-workhorse
install gitlab-workhorse ${PREFIX}/bin/
.PHONY: test
test: test/data/group/test.git clean-workhorse gitlab-workhorse
go fmt | awk '{ print "Please run go fmt"; exit 1 }'
go test
coverage: test/data/group/test.git
go test -cover -coverprofile=test.coverage
go tool cover -html=test.coverage -o coverage.html
rm -f test.coverage
test/data/group/test.git: test/data
git clone --bare https://gitlab.com/gitlab-org/gitlab-test.git test/data/group/test.git
test/data:
mkdir -p test/data
.PHONY: clean
clean: clean-workhorse
rm -rf test/data test/scratch
.PHONY: clean-workhorse
clean-workhorse:
rm -f gitlab-workhorse
# gitlab-workhorse
gitlab-workhorse was designed to unload Git HTTP traffic from
the GitLab Rails app (Unicorn) to a separate daemon. It also serves
'git archive' downloads for GitLab. All authentication and
authorization logic is still handled by the GitLab Rails app.
Architecture: Git client -> NGINX -> gitlab-workhorse (makes
auth request to GitLab Rails app) -> git-upload-pack
## Usage
```
gitlab-workhorse [OPTIONS]
Options:
-authBackend string
Authentication/authorization backend (default "http://localhost:8080")
-authSocket string
Optional: Unix domain socket to dial authBackend at
-listenAddr string
Listen address for HTTP server (default "localhost:8181")
-listenNetwork string
Listen 'network' (tcp, tcp4, tcp6, unix) (default "tcp")
-listenUmask int
Umask for Unix socket, default: 022 (default 18)
-pprofListenAddr string
pprof listening address, e.g. 'localhost:6060'
-version
Print version and exit
```
gitlab-workhorse allows Git HTTP clients to push and pull to
and from Git repositories. Each incoming request is first replayed
(with an empty request body) to an external authentication/authorization
HTTP server: the 'auth backend'. The auth backend is expected to
be a GitLab Unicorn process. The 'auth response' is a JSON message
which tells gitlab-workhorse the path of the Git repository
to read from/write to.
gitlab-workhorse can listen on either a TCP or a Unix domain socket. It
can also open a second listening TCP listening socket with the Go
[net/http/pprof profiler server](http://golang.org/pkg/net/http/pprof/).
## Installation
To install into `/usr/local/bin` run `make install`.
```
make install
```
To install into `/foo/bin` set the PREFIX variable.
```
make install PREFIX=/foo
```
## Tests
```
make clean test
```
## Try it out
You can try out the Git server without authentication as follows:
```
# Start a fake auth backend that allows everything/everybody
make test/data/test.git
go run support/fake-auth-backend.go ~+/test/data/test.git &
# Start gitlab-workhorse
make
./gitlab-workhorse
```
Now you can try things like:
```
git clone http://localhost:8181/test.git
curl -JO http://localhost:8181/test/repository/archive.zip
```
## Example request flow
- start POST repo.git/git-receive-pack to NGINX
- ..start POST repo.git/git-receive-pack to gitlab-workhorse
- ....start POST repo.git/git-receive-pack to Unicorn for auth
- ....end POST to Unicorn for auth
- ....start git-receive-pack process from gitlab-workhorse
- ......start POST /api/v3/internal/allowed to Unicorn from Git hook (check protected branches)
- ......end POST to Unicorn from Git hook
- ....end git-receive-pack process
- ..end POST to gitlab-workhorse
- end POST to NGINX
## License
This code is distributed under the MIT license, see the LICENSE file.
/*
In this file we handle 'git archive' downloads
*/
package main
import (
"fmt"
"io"
"io/ioutil"
"log"
"net/http"
"os"
"os/exec"
"path"
"path/filepath"
"time"
)
func handleGetArchive(w http.ResponseWriter, r *gitRequest) {
var format string
urlPath := r.URL.Path
switch filepath.Base(urlPath) {
case "archive.zip":
format = "zip"
case "archive.tar":
format = "tar"
case "archive", "archive.tar.gz":
format = "tar.gz"
case "archive.tar.bz2":
format = "tar.bz2"
default:
fail500(w, fmt.Errorf("handleGetArchive: invalid format: %s", urlPath))
return
}
archiveFilename := path.Base(r.ArchivePath)
if cachedArchive, err := os.Open(r.ArchivePath); err == nil {
defer cachedArchive.Close()
log.Printf("Serving cached file %q", r.ArchivePath)
setArchiveHeaders(w, format, archiveFilename)
// Even if somebody deleted the cachedArchive from disk since we opened
// the file, Unix file semantics guarantee we can still read from the
// open file in this process.
http.ServeContent(w, r.Request, "", time.Unix(0, 0), cachedArchive)
return
}
// We assume the tempFile has a unique name so that concurrent requests are
// safe. We create the tempfile in the same directory as the final cached
// archive we want to create so that we can use an atomic link(2) operation
// to finalize the cached archive.
tempFile, err := prepareArchiveTempfile(path.Dir(r.ArchivePath), archiveFilename)
if err != nil {
fail500(w, fmt.Errorf("handleGetArchive: create tempfile: %v", err))
return
}
defer tempFile.Close()
defer os.Remove(tempFile.Name())
compressCmd, archiveFormat := parseArchiveFormat(format)
archiveCmd := gitCommand("", "git", "--git-dir="+r.RepoPath, "archive", "--format="+archiveFormat, "--prefix="+r.ArchivePrefix+"/", r.CommitId)
archiveStdout, err := archiveCmd.StdoutPipe()
if err != nil {
fail500(w, fmt.Errorf("handleGetArchive: archive stdout: %v", err))
return
}
defer archiveStdout.Close()
if err := archiveCmd.Start(); err != nil {
fail500(w, fmt.Errorf("handleGetArchive: start %v: %v", archiveCmd.Args, err))
return
}
defer cleanUpProcessGroup(archiveCmd) // Ensure brute force subprocess clean-up
var stdout io.ReadCloser
if compressCmd == nil {
stdout = archiveStdout
} else {
compressCmd.Stdin = archiveStdout
stdout, err = compressCmd.StdoutPipe()
if err != nil {
fail500(w, fmt.Errorf("handleGetArchive: compress stdout: %v", err))
return
}
defer stdout.Close()
if err := compressCmd.Start(); err != nil {
fail500(w, fmt.Errorf("handleGetArchive: start %v: %v", compressCmd.Args, err))
return
}
defer compressCmd.Wait()
archiveStdout.Close()
}
// Every Read() from stdout will be synchronously written to tempFile
// before it comes out the TeeReader.
archiveReader := io.TeeReader(stdout, tempFile)
// Start writing the response
setArchiveHeaders(w, format, archiveFilename)
w.WriteHeader(200) // Don't bother with HTTP 500 from this point on, just return
if _, err := io.Copy(w, archiveReader); err != nil {
logError(fmt.Errorf("handleGetArchive: read: %v", err))
return
}
if err := archiveCmd.Wait(); err != nil {
logError(fmt.Errorf("handleGetArchive: archiveCmd: %v", err))
return
}
if compressCmd != nil {
if err := compressCmd.Wait(); err != nil {
logError(fmt.Errorf("handleGetArchive: compressCmd: %v", err))
return
}
}
if err := finalizeCachedArchive(tempFile, r.ArchivePath); err != nil {
logError(fmt.Errorf("handleGetArchive: finalize cached archive: %v", err))
return
}
}
func setArchiveHeaders(w http.ResponseWriter, format string, archiveFilename string) {
w.Header().Add("Content-Disposition", fmt.Sprintf(`attachment; filename="%s"`, archiveFilename))
if format == "zip" {
w.Header().Add("Content-Type", "application/zip")
} else {
w.Header().Add("Content-Type", "application/octet-stream")
}
w.Header().Add("Content-Transfer-Encoding", "binary")
w.Header().Add("Cache-Control", "private")
}
func parseArchiveFormat(format string) (*exec.Cmd, string) {
switch format {
case "tar":
return nil, "tar"
case "tar.gz":
return exec.Command("gzip", "-c", "-n"), "tar"
case "tar.bz2":
return exec.Command("bzip2", "-c"), "tar"
case "zip":
return nil, "zip"
}
return nil, "unknown"
}
func prepareArchiveTempfile(dir string, prefix string) (*os.File, error) {
if err := os.MkdirAll(dir, 0700); err != nil {
return nil, err
}
return ioutil.TempFile(dir, prefix)
}
func finalizeCachedArchive(tempFile *os.File, archivePath string) error {
if err := tempFile.Close(); err != nil {
return err
}
return os.Link(tempFile.Name(), archivePath)
}
package main
func artifactsAuthorizeHandler(handleFunc serviceHandleFunc) serviceHandleFunc {
return preAuthorizeHandler(handleFunc, "/authorize")
}
package main
import (
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
)
func (u *upstream) newUpstreamRequest(r *http.Request, body io.Reader, suffix string) (*http.Request, error) {
url := u.authBackend + r.URL.RequestURI() + suffix
authReq, err := http.NewRequest(r.Method, url, body)
if err != nil {
return nil, err
}
// Forward all headers from our client to the auth backend. This includes
// HTTP Basic authentication credentials (the 'Authorization' header).
for k, v := range r.Header {
authReq.Header[k] = v
}
// Clean some headers when issuing a new request without body
if body == nil {
authReq.Header.Del("Content-Type")
authReq.Header.Del("Content-Encoding")
authReq.Header.Del("Content-Length")
authReq.Header.Del("Content-Disposition")
authReq.Header.Del("Accept-Encoding")
// Hop-by-hop headers. These are removed when sent to the backend.
// http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html
authReq.Header.Del("Transfer-Encoding")
authReq.Header.Del("Connection")
authReq.Header.Del("Keep-Alive")
authReq.Header.Del("Proxy-Authenticate")
authReq.Header.Del("Proxy-Authorization")
authReq.Header.Del("Te")
authReq.Header.Del("Trailers")
authReq.Header.Del("Upgrade")
}
// Also forward the Host header, which is excluded from the Header map by the http libary.
// This allows the Host header received by the backend to be consistent with other
// requests not going through gitlab-workhorse.
authReq.Host = r.Host
// Set a custom header for the request. This can be used in some
// configurations (Passenger) to solve auth request routing problems.
authReq.Header.Set("Gitlab-Workhorse", Version)
return authReq, nil
}
func preAuthorizeHandler(handleFunc serviceHandleFunc, suffix string) serviceHandleFunc {
return func(w http.ResponseWriter, r *gitRequest) {
authReq, err := r.u.newUpstreamRequest(r.Request, nil, suffix)
if err != nil {
fail500(w, fmt.Errorf("preAuthorizeHandler: newUpstreamRequest: %v", err))
return
}
authResponse, err := r.u.httpClient.Do(authReq)
if err != nil {
fail500(w, fmt.Errorf("preAuthorizeHandler: do %v: %v", authReq.URL.Path, err))
return
}
defer authResponse.Body.Close()
if authResponse.StatusCode != 200 {
// The Git request is not allowed by the backend. Maybe the
// client needs to send HTTP Basic credentials. Forward the
// response from the auth backend to our client. This includes
// the 'WWW-Authenticate' header that acts as a hint that
// Basic auth credentials are needed.
for k, v := range authResponse.Header {
// Accomodate broken clients that do case-sensitive header lookup
if k == "Www-Authenticate" {
w.Header()["WWW-Authenticate"] = v
} else {
w.Header()[k] = v
}
}
w.WriteHeader(authResponse.StatusCode)
io.Copy(w, authResponse.Body)
return
}
// The auth backend validated the client request and told us additional
// request metadata. We must extract this information from the auth
// response body.
if err := json.NewDecoder(authResponse.Body).Decode(&r.authorizationResponse); err != nil {
fail500(w, fmt.Errorf("preAuthorizeHandler: decode authorization response: %v", err))
return
}
// Don't hog a TCP connection in CLOSE_WAIT, we can already close it now
authResponse.Body.Close()
// Negotiate authentication (Kerberos) may need to return a WWW-Authenticate
// header to the client even in case of success as per RFC4559.
for k, v := range authResponse.Header {
// Case-insensitive comparison as per RFC7230
if strings.EqualFold(k, "WWW-Authenticate") {
w.Header()[k] = v
}
}
handleFunc(w, r)
}
}
package main
import (
"fmt"
"net/http"
"net/http/httptest"
"regexp"
"testing"
)
func okHandler(w http.ResponseWriter, r *gitRequest) {
w.WriteHeader(201)
fmt.Fprint(w, "{\"status\":\"ok\"}")
}
func runPreAuthorizeHandler(t *testing.T, suffix string, url *regexp.Regexp, authorizationResponse interface{}, returnCode, expectedCode int) *httptest.ResponseRecorder {
// Prepare test server and backend
ts := testAuthServer(url, returnCode, authorizationResponse)
defer ts.Close()
// Create http request
httpRequest, err := http.NewRequest("GET", "/address", nil)
if err != nil {
t.Fatal(err)
}
request := gitRequest{
Request: httpRequest,
u: newUpstream(ts.URL, nil),
}
response := httptest.NewRecorder()
preAuthorizeHandler(okHandler, suffix)(response, &request)
assertResponseCode(t, response, expectedCode)
return response
}
func TestPreAuthorizeHappyPath(t *testing.T) {
runPreAuthorizeHandler(
t, "/authorize",
regexp.MustCompile(`/authorize\z`),
&authorizationResponse{},
200, 201)
}
func TestPreAuthorizeSuffix(t *testing.T) {
runPreAuthorizeHandler(
t, "/different-authorize",
regexp.MustCompile(`/authorize\z`),
&authorizationResponse{},
200, 404)
}
func TestPreAuthorizeJsonFailure(t *testing.T) {
runPreAuthorizeHandler(
t, "/authorize",
regexp.MustCompile(`/authorize\z`),
"not-json",
200, 500)
}
package main
import (
"io/ioutil"
"net/http"
"path/filepath"
)
func handleDeployPage(documentRoot *string, handler serviceHandleFunc) serviceHandleFunc {
return func(w http.ResponseWriter, r *gitRequest) {
deployPage := filepath.Join(*documentRoot, "index.html")
data, err := ioutil.ReadFile(deployPage)
if err != nil {
handler(w, r)
return
}
setNoCacheHeaders(w.Header())
w.Header().Set("Content-Type", "text/html; charset=utf-8")
w.WriteHeader(http.StatusOK)
w.Write(data)
}
}
package main
import (
"io/ioutil"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
)
func TestIfNoDeployPageExist(t *testing.T) {
dir, err := ioutil.TempDir("", "deploy")
if err != nil {
t.Fatal(err)
}
defer os.RemoveAll(dir)
w := httptest.NewRecorder()
executed := false
handleDeployPage(&dir, func(w http.ResponseWriter, r *gitRequest) {
executed = true
})(w, nil)
if !executed {
t.Error("The handler should get executed")
}
}
func TestIfDeployPageExist(t *testing.T) {
dir, err := ioutil.TempDir("", "deploy")
if err != nil {
t.Fatal(err)
}
defer os.RemoveAll(dir)
deployPage := "DEPLOY"
ioutil.WriteFile(filepath.Join(dir, "index.html"), []byte(deployPage), 0600)
w := httptest.NewRecorder()
executed := false
handleDeployPage(&dir, func(w http.ResponseWriter, r *gitRequest) {
executed = true
})(w, nil)
if executed {
t.Error("The handler should not get executed")
}
w.Flush()
assertResponseCode(t, w, 200)
assertResponseBody(t, w, deployPage)
}
package main
import "net/http"
func handleDevelopmentMode(developmentMode *bool, handler serviceHandleFunc) serviceHandleFunc {
return func(w http.ResponseWriter, r *gitRequest) {
if !*developmentMode {
http.NotFound(w, r.Request)
return
}
handler(w, r)
}
}
package main
import (
"net/http"
"net/http/httptest"
"testing"
)
func TestDevelopmentModeEnabled(t *testing.T) {
developmentMode := true
r, _ := http.NewRequest("GET", "/something", nil)
w := httptest.NewRecorder()
executed := false
handleDevelopmentMode(&developmentMode, func(w http.ResponseWriter, r *gitRequest) {
executed = true
})(w, &gitRequest{Request: r})
if !executed {
t.Error("The handler should get executed")
}
}
func TestDevelopmentModeDisabled(t *testing.T) {
developmentMode := false
r, _ := http.NewRequest("GET", "/something", nil)
w := httptest.NewRecorder()
executed := false
handleDevelopmentMode(&developmentMode, func(w http.ResponseWriter, r *gitRequest) {
executed = true
})(w, &gitRequest{Request: r})
if executed {
t.Error("The handler should not get executed")
}
assertResponseCode(t, w, 404)
}
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"path/filepath"
)
type errorPageResponseWriter struct {
rw http.ResponseWriter
status int
hijacked bool
path *string
}
func (s *errorPageResponseWriter) Header() http.Header {
return s.rw.Header()
}
func (s *errorPageResponseWriter) Write(data []byte) (n int, err error) {
if s.status == 0 {
s.WriteHeader(http.StatusOK)
}
if s.hijacked {
return 0, nil
}
return s.rw.Write(data)
}
func (s *errorPageResponseWriter) WriteHeader(status int) {
if s.status != 0 {
return
}
s.status = status
if 400 <= s.status && s.status <= 599 {
errorPageFile := filepath.Join(*s.path, fmt.Sprintf("%d.html", s.status))
// check if custom error page exists, serve this page instead
if data, err := ioutil.ReadFile(errorPageFile); err == nil {
s.hijacked = true
log.Printf("ErrorPage: serving predefined error page: %d", s.status)