Commit aff2bb58 authored by Dmitry Smirnov's avatar Dmitry Smirnov

initial commit

parent d5943f37
# This file lists authors for copyright purposes. This file is distinct from
# the CONTRIBUTORS files. See the latter for an explanation.
#
# Names should be added to this file as:
# Name or Organization <email address>
#
# The email address is not required for organizations.
#
# Please keep the list sorted.
CZ.NIC z.s.p.o. <kontakt@nic.cz>
Jan Mercl <0xjnml@gmail.com>
Aaron Bieber <deftly@gmail.com>
# This file lists people who contributed code to this repository. The AUTHORS
# file lists the copyright holders; this file lists people.
#
# Names should be added to this file like so:
# Name <email address>
#
# Please keep the list sorted.
Bill Thiede <xinu.tv>
Gary Burd <gary@beagledreams.com>
Jan Mercl <0xjnml@gmail.com>
Nick Owens <mischief@offblast.org>
Tamás Gulácsi <gt-dev@gthomas.eu>
Aaron Bieber <deftly@gmail.com>
Copyright (c) 2014 The fileutil Authors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the names of the authors nor the names of the
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
# Copyright (c) 2014 The fileutil authors. All rights reserved.
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file.
.PHONY: all clean editor todo
all: editor
go vet
golint .
go install
make todo
editor:
go fmt
go test -i
go test
go build
todo:
@grep -n ^[[:space:]]*_[[:space:]]*=[[:space:]][[:alpha:]][[:alnum:]]* *.go || true
@grep -n TODO *.go || true
@grep -n BUG *.go || true
@grep -n println *.go || true
clean:
@go clean
rm -f y.output
This is a goinstall-able mirror of modified code already published at:
http://git.nic.cz/redmine/projects/gofileutil/repository
Packages in this repository:
Install: $go get github.com/cznic/fileutil
Godocs: http://godoc.org/github.com/cznic/fileutil
Install: $go get github.com/cznic/fileutil/storage
Godocs: http://godoc.org/github.com/cznic/fileutil/storage
Install: $go get github.com/cznic/fileutil/falloc
Godocs: http://godoc.org/github.com/cznic/fileutil/falloc
Install: $go get github.com/cznic/fileutil/hdb
Godocs: http://godoc.org/github.com/cznic/fileutil/hdb
// Copyright (c) 2014 The fileutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package fileutil
import (
"os"
"path/filepath"
"strings"
"testing"
)
func TestTempFile(t *testing.T) {
f, err := TempFile("", "abc", "mno.xyz")
if err != nil {
t.Fatal(err)
}
n := f.Name()
t.Log(n)
defer func() {
f.Close()
os.Remove(n)
}()
base := filepath.Base(n)
if base == "abcmno.xyz" {
t.Fatal(base)
}
if !strings.HasPrefix(base, "abc") {
t.Fatal(base)
}
if !strings.HasSuffix(base, "mno.xyz") {
t.Fatal(base)
}
}
golang-github-cznic-fileutil (0.0~git20150708.0.1c9c88f-1) UNRELEASED; urgency=medium
* Initial release (Closes: TODO)
-- Dmitry Smirnov <onlyjob@debian.org> Thu, 05 Nov 2015 00:47:03 +1100
Source: golang-github-cznic-fileutil
Section: devel
Priority: extra
Maintainer: Debian Go Packaging Team <pkg-go-maintainers@lists.alioth.debian.org>
Uploaders: Dmitry Smirnov <onlyjob@debian.org>
Build-Depends: debhelper (>= 9),
dh-golang,
golang-go,
golang-github-cznic-mathutil-dev
Standards-Version: 3.9.6
Homepage: https://github.com/cznic/fileutil
Vcs-Browser: https://anonscm.debian.org/cgit/pkg-go/packages/golang-github-cznic-fileutil.git
Vcs-Git: https://anonscm.debian.org/pkg-go/packages/golang-github-cznic-fileutil.git
XS-Go-Import-Path: github.com/cznic/fileutil
Package: golang-github-cznic-fileutil-dev
Architecture: all
Depends: ${misc:Depends},
golang-go,
golang-github-cznic-mathutil-dev
Description: Golang file utility functions
"Fileutil collects some file utility functions".
Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: fileutil
Source: https://github.com/cznic/fileutil
Files: *
Copyright:
2014 The fileutil Authors
2011 CZ.NIC z.s.p.o. <kontakt@nic.cz>
2013 jnml
License: BSD-3-Clause
Files: debian/*
Copyright: 2015-2016 Dmitry Smirnov <onlyjob@debian.org>
License: GPL-3+
Files: debian/patches/*
Copyright: 2016 Dmitry Smirnov <onlyjob@debian.org>
License: GPL-3+ or BSD-3-clause
Comment: patches can be licensed under the same terms as upstream.
License: BSD-3-Clause
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
.
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the names of the authors nor the names of the
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
License: GPL-3+
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
The complete text of the GNU General Public License version 3
can be found in "/usr/share/common-licenses/GPL-3".
[buildpackage]
overlay = True
export-dir = ../build-area/
tarball-dir = ../
[dch]
id-length = 0
[import-orig]
pristine-tar = True
merge = False
#!/usr/bin/make -f
%:
dh $@ --buildsystem=golang --with=golang
Copyright (c) 2011 CZ.NIC z.s.p.o. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of CZ.NIC nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
This is a goinstall-able mirror of modified code already published at:
https://git.nic.cz/redmine/projects/gofileutil/repository/show/falloc
Install: $go get github.com/cznic/fileutil/falloc
Godocs: http://gopkgdoc.appspot.com/pkg/github.com/cznic/fileutil/falloc
This diff is collapsed.
// Copyright (c) 2011 CZ.NIC z.s.p.o. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// blame: jnml, labs.nic.cz
/*
WIP: Package falloc provides allocation/deallocation of space within a
file/store (WIP, unstable API).
Overall structure:
File == n blocks.
Block == n atoms.
Atom == 16 bytes.
x6..x0 == least significant 7 bytes of a 64 bit integer, highest (7th) byte is
0 and is not stored in the file.
Block first byte
Aka block type tag.
------------------------------------------------------------------------------
0xFF: Free atom (free block of size 1).
+------++---------++---------++------+
| 0 || 1...7 || 8...14 || 15 |
+------++---------++---------++------+
| 0xFF || p6...p0 || n6...n0 || 0xFF |
+------++---------++---------++------+
Link to the previous free block (atom addressed) is p6...p0, next dtto in
n6...n0. Doubly linked lists of "compatible" free blocks allows for free space
reclaiming and merging. "Compatible" == of size at least some K. Heads of all
such lists are organized per K or intervals of Ks elsewhere.
------------------------------------------------------------------------------
0xFE: Free block, size == s6...s0 atoms.
+------++---------++---------++---------++--
| +0 || 1...7 || 8...14 || 15...21 || 22...16*size-1
+------++---------++---------++---------++--
| 0xFE || p6...p0 || n6...n0 || s6...s0 || ...
+------++---------++---------++---------++--
Prev and next links as in the 0xFF first byte case. End of this block - see
"Block last byte": 0xFE bellow. Data between == undefined.
------------------------------------------------------------------------------
0xFD: Relocated block.
+------++---------++-----------++------+
| 0 || 1...7 || 8...14 || 15 |
+------++---------++-----------++------+
| 0xFD || r6...r0 || undefined || 0x00 | // == used block
+------++---------++-----------++------+
Relocation link is r6..r0 == atom address. Relocations MUST NOT chain and MUST
point to a "content" block, i.e. one with the first byte in 0x00...0xFC.
Relocated block allows to permanently assign a handle/file pointer ("atom"
address) to some content and resize the content anytime afterwards w/o having
to update all the possible existing references to the original handle.
------------------------------------------------------------------------------
0xFC: Used long block.
+------++---------++--------------------++---------+---+
| 0 || 1...2 || 3...N+2 || | |
+------++---------++--------------------++---------+---+
| 0xFC || n1...n0 || N bytes of content || padding | Z |
+------++---------++--------------------++---------+---+
This block type is used for content of length in N == 238...61680 bytes. N is
encoded as a 2 byte unsigned integer n1..n0 in network byte order. Values
bellow 238 are reserved, those content lengths are to be carried by the
0x00..0xFB block types.
1. n in 0x00EE...0xF0F0 is used for content under the same rules
as in the 0x01..0xED type.
2. If the last byte of the content is not the last byte of an atom then
the last byte of the block is 0x00.
3. If the last byte of the content IS the last byte of an atom:
3.1 If the last byte of content is in 0x00..0xFD then everything is OK.
3.2 If the last byte of content is 0xFE or 0xFF then the escape
via n > 0xF0F0 MUST be used AND the block's last byte is 0x00 or 0x01,
meaning value 0xFE and 0xFF respectively.
4. n in 0xF0F1...0xFFFF is like the escaped 0xEE..0xFB block.
N == 13 + 16(n - 0xF0F1).
Discussion of the padding and Z fields - see the 0x01..0xED block type.
------------------------------------------------------------------------------
0xEE...0xFB: Used escaped short block.
+---++----------------------++---+
| 0 || 1...N-1 || |
+---++----------------------++---+
| X || N-1 bytes of content || Z |
+---++----------------------++---+
N == 15 + 16(X - 0xEE). Z is the content last byte encoded as follows.
case Z == 0x00: The last byte of content is 0xFE
case Z == 0x01: The last byte of content is 0xFF
------------------------------------------------------------------------------
0x01...0xED: Used short block.
+---++--------------------++---------+---+
| 0 || 1...N || | |
+---++--------------------++---------+---+
| N || N bytes of content || padding | Z |
+---++--------------------++---------+---+
This block type is used for content of length in 1...237 bytes. The value of
the "padding" field, if of non zero length, is undefined.
If the last byte of content is the last byte of an atom (== its file byte
offset & 0xF == 0xF) then such last byte MUST be in 0x00...0xFD.
If the last byte of content is the last byte of an atom AND the last byte of
content is 0xFE or 0xFF then the short escape block type (0xEE...0xFB) MUST be
used.
If the last byte of content is not the last byte of an atom, then the last byte
of such block, i.e. the Z field, which is also a last byte of some atom, MUST
be 0x00 (i.e. the used block marker). Other "tail" values are reserved.
------------------------------------------------------------------------------
0x00: Used empty block.
+------++-----------++------+
| 0 || 1...14 || 15 |
+------++-----------++------+
| 0x00 || undefined || 0x00 | // == used block, other "tail" values reserved.
+------++-----------++------+
All of the rules for 0x01..0xED applies. Depicted only for its different
semantics (e.g. an allocated [existing] string but with length of zero).
==============================================================================
Block last byte
------------------------------------------------------------------------------
0xFF: Free atom. Layout - see "Block first byte": FF.
------------------------------------------------------------------------------
0xFE: Free block, size n atoms. Preceding 7 bytes == size (s6...s0) of the free
block in atoms, network byte order
--++---------++------+
|| -8...-2 || -1 |
--++---------++------+
... || s6...s0 || 0xFE | <- block's last byte
--++---------++------+
Layout at start of this block - see "Block first byte": FE.
------------------------------------------------------------------------------
0x00...0xFD: Used (non free) block.
==============================================================================
Free lists table
The free lists table content is stored in the standard layout of a used block.
A table item is a 7 byte size field followed by a 7 byte atom address field
(both in network byte order), thus every item is 14 contiguous bytes. The
item's address field is pointing to a free block. The size field determines
the minimal size (in atoms) of free blocks on that list.
The free list table is n above items, thus the content has 14n bytes. Note that
the largest block content is 61680 bytes and as there are 14 bytes per table
item, so the table is limited to at most 4405 entries.
Items in the table do not have to be sorted according to their size field values.
No two items can have the same value of the size field.
When freeing blocks, the block MUST be linked into an item list with the
highest possible size field, which is less or equal to the number of atoms in
the new free block.
When freeing a block, the block MUST be first merged with any adjacent free
blocks (thus possibly creating a bigger free block) using information derived
from the adjacent blocks first and last bytes. Such merged free blocks MUST be
removed from their original doubly linked lists. Afterwards the new bigger free
block is put to the free list table in the appropriate item.
Items with address field == 0 are legal. Such item is a placeholder for a empty
list of free blocks of the item's size.
Items with size field == 0 are legal. Such item is a placeholder, used e.g. to
avoid further reallocations/redirecting of the free lists table.
The largest possible allocation request (for content length 61680 bytes) is
0xF10 (3856) atoms. All free blocks of this or bigger size are presumably put
into a single table item with the size 3856. It may be useful to additionally
have a free lists table item which links free blocks of some bigger size (say
1M+) and then use the OS sparse file support (if present) to save the physical
space used by such free blocks.
Smaller (<3856 atoms) free blocks can be organized exactly (every distinct size
has its table item) or the sizes can run using other schema like e.g. "1, 2,
4, 8, ..." (powers of 2) or "1, 2, 3, 5, 8, 13, ..." (the Fibonacci sequence)
or they may be fine tuned to a specific usage pattern.
==============================================================================
Header
The first block of a file (atom address == file offset == 0) is the file header.
The header block has the standard layout of a used short non escaped block.
Special conditions apply: The header block and its content MUST be like this:
+------+---------+---------+------+
| 0 | 1...7 | 8...14 | 15 |
+------+---------+---------+------+
| 0x0F | m6...m0 | f6...f0 | FLTT |
+------+---------+---------+------+
m6..m0 is a "magic" value 0xF1C1A1FE51B1E.
f6...f0 is the atom address of the free lists table (discussed elsewhere).
If f6...f0 == 0x00 the there is no free lists table (yet).
FLTT describes the type of the Free List Table. Currently defined values:
------------------------------------------------------------------------------
FLTT == 0: Free List Table is fixed at atom address 2. It has a fixed size for 3856 entries
for free list of size 1..3855 atoms and the last is for the list of free block >= 3856 atoms.
*/
package falloc
const (
INVALID_HANDLE = Handle(-1)
)
// Copyright (c) 2011 CZ.NIC z.s.p.o. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// blame: jnml, labs.nic.cz
package falloc
import "fmt"
// EBadRequest is an error produced for invalid operation, e.g. for data of more than maximum allowed.
type EBadRequest struct {
Name string
Size int
}
func (e *EBadRequest) Error() string {
return fmt.Sprintf("%s: size %d", e.Name, e.Size)
}
// EClose is a file/store close error.
type EClose struct {
Name string
Err error
}
func (e *EClose) Error() string {
return fmt.Sprintf("%sx: %s", e.Name, e.Err)
}
// ECorrupted is a file/store format error.
type ECorrupted struct {
Name string
Ofs int64
}
func (e *ECorrupted) Error() string {
return fmt.Sprintf("%s: corrupted data @%#x", e.Name, e.Ofs)
}
// ECreate is a file/store create error.
type ECreate struct {
Name string
Err error
}
func (e *ECreate) Error() string {
return fmt.Sprintf("%s: %s", e.Name, e.Err)
}
// EFreeList is a file/store format error.
type EFreeList struct {
Name string
Size int64
Block int64
}
func (e *EFreeList) Error() string {
return fmt.Sprintf("%s: invalid free list item, size %#x, block %#x", e.Name, e.Size, e.Block)
}
// EHandle is an error type reported for invalid Handles.
type EHandle struct {
Name string
Handle Handle
}
func (e EHandle) Error() string {
return fmt.Sprintf("%s: invalid handle %#x", e.Name, e.Handle)
}
// EHeader is a file/store format error.
type EHeader struct {
Name string
Header []byte
Expected []byte
}
func (e *EHeader) Error() string {
return fmt.Sprintf("%s: invalid header, got [% x], expected [% x]", e.Name, e.Header, e.Expected)
}
// ENullHandle is a file/store access error via a null handle.
type ENullHandle string
func (e ENullHandle) Error() string {
return fmt.Sprintf("%s: access via null handle", e)
}
// EOpen is a file/store open error.
type EOpen struct {
Name string
Err error
}
func (e *EOpen) Error() string {
return fmt.Sprintf("%s: %s", e.Name, e.Err)
}
// ERead is a file/store read error.
type ERead struct {
Name string
Ofs int64
Err error
}
func (e *ERead) Error() string {
return fmt.Sprintf("%s, %#x: %s", e.Name, e.Ofs, e.Err)
}
// ESize is a file/store size error.
type ESize struct {
Name string
Size int64
}
func (e *ESize) Error() string {
return fmt.Sprintf("%s: invalid size %#x(%d), size %%16 != 0", e.Name, e.Size, e.Size)
}
// EWrite is a file/store write error.
type EWrite struct {
Name string
Ofs int64
Err error
}
func (e *EWrite) Error() string {
return fmt.Sprintf("%s, %#x: %s", e.Name, e.Ofs, e.Err)
}
This diff is collapsed.
// Copyright (c) 2011 CZ.NIC z.s.p.o. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// blame: jnml, labs.nic.cz
package falloc
// Pull test dependencies too.
// Enables easy 'go test X' after 'go get X'
import (
_ "github.com/cznic/fileutil"
_ "github.com/cznic/fileutil/storage"
_ "github.com/cznic/mathutil"
)
// Copyright (c) 2014 The fileutil Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package fileutil collects some file utility functions.
package fileutil
import (
"fmt"
"io"
"os"
"path/filepath"
"runtime"
"strconv"
"sync"
"time"
)
// GoMFile is a concurrent access safe version of MFile.
type GoMFile struct {
mfile *MFile
mutex sync.Mutex
}
// NewGoMFile return a newly created GoMFile.
func NewGoMFile(fname string, flag int, perm os.FileMode, delta_ns int64) (m *GoMFile, err error) {
m = &GoMFile{}
if m.mfile, err = NewMFile(fname, flag, perm, delta_ns); err != nil {
m = nil
}
return
}
func (m *GoMFile) File() (file *os.File, err error) {
m.mutex.Lock()
defer m.mutex.Unlock()
return m.mfile.File()
}
func (m *GoMFile) SetChanged() {
m.mutex.Lock()
defer m.mutex.Unlock()
m.mfile.SetChanged()
}
func (m *GoMFile) SetHandler(h MFileHandler) {
m.mutex.Lock()
defer m.mutex.Unlock()
m.mfile.SetHandler(h)
}
// MFileHandler resolves modifications of File.
// Possible File context is expected to be a part of the handler's closure.
type MFileHandler func(*os.File) error
// MFile represents an os.File with a guard/handler on change/modification.
// Example use case is an app with a configuration file which can be modified at any time
// and have to be reloaded in such event prior to performing something configurable by that
// file. The checks are made only on access to the MFile file by
// File() and a time threshold/hysteresis value can be chosen on creating a new MFile.
type MFile struct {
file *os.File