swtor/ssn-installer
0
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

🏗 Add xdelta3 dependency

Browse source
This commit is contained in:
C-3PO 2018-09-14 03:16:31 +02:00
parent 80f2439239
commit ae59ce2256
Signed by: c3po
GPG key ID: 62993C4BB4D86F24
97 changed files with 45332 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

14
README.md
View file

@ -49,4 +49,16 @@ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE. THE SOFTWARE.
## Xdelta3 ## Xdelta3 <[https://github.com/jmacd/xdelta](https://github.com/jmacd/xdelta)>
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

176
lib/xdelta3/LICENSE Normal file
View file

@ -0,0 +1,176 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS

192
lib/xdelta3/Makefile.am Normal file
View file

@ -0,0 +1,192 @@
ACLOCAL_AMFLAGS = -I m4
AUTOMAKE_OPTIONS = subdir-objects
bin_PROGRAMS = xdelta3
noinst_PROGRAMS = xdelta3regtest xdelta3decode xdelta3checksum
export AFL_HARDEN
common_SOURCES = \
xdelta3-blkcache.h \
xdelta3-decode.h \
xdelta3-djw.h \
xdelta3-fgk.h \
xdelta3-hash.h \
xdelta3-internal.h \
xdelta3-list.h \
xdelta3-lzma.h \
xdelta3-main.h \
xdelta3-merge.h \
xdelta3-second.h \
xdelta3-test.h \
xdelta3-cfgs.h \
xdelta3.h
xdelta3_SOURCES = $(common_SOURCES) xdelta3.c
xdelta3decode_SOURCES = $(common_SOURCES) xdelta3.c
xdelta3regtest_SOURCES = $(common_SOURCES) \
testing/cmp.h \
testing/delta.h \
testing/file.h \
testing/modify.h \
testing/random.h \
testing/regtest.cc \
testing/regtest_c.c \
testing/segment.h \
testing/sizes.h \
testing/test.h
xdelta3checksum_SOURCES = $(common_SOURCES) \
testing/checksum_test.cc \
testing/checksum_test_c.c
# These sources constitute a regression test written in Go, that is
# not automatically built or run. Install Go-1.5.x or later, add
# `pwd`/go in $GOPATH, and (cd go/src && go run regtest.go).
# TODO(jmacd): replace hard-coded path names in regtest.go w/ flags.
GOLANG_SRCS = \
go/src/xdelta/test.go \
go/src/xdelta/rstream.go \
go/src/xdelta/tgroup.go \
go/src/xdelta/run.go \
go/src/regtest.go
# Note: for extra sanity checks, enable -Wconversion. Note there
# are a lot of false positives.
WFLAGS = -Wall -Wshadow -fno-builtin -Wextra -Wsign-compare \
-Wformat=2 -Wno-format-nonliteral \
-Wno-unused-parameter -Wno-unused-function
# -Weverything \
# -Wc++11-compat-reserved-user-defined-literal \
# -Wno-padded \
# -Wno-format-nonliteral \
# -Wno-cast-align \
# -Wno-unused-parameter \
# -Wno-sign-conversion \
# -Wno-conversion \
# -Wno-switch-enum \
# -Wno-covered-switch-default \
# -Wno-disabled-macro-expansion \
# -Wno-variadic-macros \
# -Wno-c++98-compat-pedantic
C_WFLAGS = $(WFLAGS) -std=c99
CXX_WFLAGS = $(WFLAGS) -std=c++11
common_CFLAGS = \
-DREGRESSION_TEST=1 \
-DSECONDARY_DJW=1 \
-DSECONDARY_FGK=1 \
-DXD3_MAIN=1
if DEBUG_SYMBOLS
common_CFLAGS += -g
endif
#common_CFLAGS += -fsanitize=address -fno-omit-frame-pointer
#common_CFLAGS += -O2
# For additional debugging, add -DXD3_DEBUG=1, 2, 3, ...
xdelta3_CFLAGS = $(C_WFLAGS) $(common_CFLAGS) -DXD3_DEBUG=0
xdelta3_LDADD = -lm
xdelta3decode_CFLAGS = \
$(C_WFLAGS) \
-DREGRESSION_TEST=0 \
-DSECONDARY_DJW=0 \
-DSECONDARY_FGK=0 \
-DSECONDARY_LZMA=0 \
-DXD3_MAIN=1 \
-DXD3_ENCODER=0 \
-DXD3_STDIO=1 \
-DEXTERNAL_COMPRESSION=0 \
-DVCDIFF_TOOLS=0
xdelta3regtest_CXXFLAGS = \
$(CXX_WFLAGS) $(common_CFLAGS) -DNOT_MAIN=1 -DXD3_DEBUG=1
xdelta3regtest_CFLAGS = \
$(C_WFLAGS) $(common_CFLAGS) -DNOT_MAIN=1 -DXD3_DEBUG=1
xdelta3regtest_LDADD = -lm
xdelta3checksum_CXXFLAGS = \
$(CXX_WFLAGS) $(common_CFLAGS) -DNOT_MAIN=1 -DXD3_MAIN=1 -std=c++11
xdelta3checksum_CFLAGS = \
$(C_WFLAGS) $(common_CFLAGS) -DNOT_MAIN=1 -DXD3_MAIN=1
man1_MANS = xdelta3.1
EXTRA_DIST = \
README.md \
run_release.sh \
draft-korn-vcdiff.txt \
$(GOLANG_SRCS) \
examples/Makefile \
examples/README.md \
examples/compare_test.c \
examples/encode_decode_test.c \
examples/small_page_test.c \
examples/speed_test.c \
examples/test.h \
examples/iOS/xdelta3-ios-test/xdelta3-ios-test.xcodeproj/project.pbxproj \
examples/iOS/xdelta3-ios-test/xdelta3-ios-test/Xd3iOSAppDelegate.h \
examples/iOS/xdelta3-ios-test/xdelta3-ios-test/Xd3iOSAppDelegate.m \
examples/iOS/xdelta3-ios-test/xdelta3-ios-test/Xd3iOSViewController.h \
examples/iOS/xdelta3-ios-test/xdelta3-ios-test/Xd3iOSViewController.m \
examples/iOS/xdelta3-ios-test/xdelta3-ios-test/en.lproj/InfoPlist.strings \
examples/iOS/xdelta3-ios-test/xdelta3-ios-test/en.lproj/MainStoryboard_iPad.storyboard \
examples/iOS/xdelta3-ios-test/xdelta3-ios-test/en.lproj/MainStoryboard_iPhone.storyboard \
examples/iOS/xdelta3-ios-test/xdelta3-ios-test/file_v1.bin \
examples/iOS/xdelta3-ios-test/xdelta3-ios-test/file_v1_to_v2.bin \
examples/iOS/xdelta3-ios-test/xdelta3-ios-test/file_v2.bin \
examples/iOS/xdelta3-ios-test/xdelta3-ios-test/main.m \
examples/iOS/xdelta3-ios-test/xdelta3-ios-test/xdelta3-ios-test-Info.plist \
examples/iOS/xdelta3-ios-test/xdelta3-ios-test/xdelta3-ios-test-Prefix.pch \
cpp-btree/CMakeLists.txt \
cpp-btree/COPYING \
cpp-btree/README \
cpp-btree/btree.h \
cpp-btree/btree_bench.cc \
cpp-btree/btree_container.h \
cpp-btree/btree_map.h \
cpp-btree/btree_set.h \
cpp-btree/btree_test.cc \
cpp-btree/btree_test.h \
cpp-btree/btree_test_flags.cc \
cpp-btree/safe_btree.h \
cpp-btree/safe_btree_map.h \
cpp-btree/safe_btree_set.h \
cpp-btree/safe_btree_test.cc \
testing/xdelta3-regtest.py \
testing/xdelta3-test.py \
xdelta3.1 \
xdelta3.i \
xdelta3.vcxproj \
xdelta3.wxi \
xdelta3.wxs
# Broken, removed from distribution:
# xdelta3_pywrap.c
# xdelta3.py
#PYFILES = xdelta3_pywrap.c xdelta3.py
#XDELTA3PY = xdelta3.py
#XDELTA3PYLIB = xdelta3.la
#BUILT_SOURCES = $(PYFILES)
#xdelta3_pywrap.c xdelta3.py : xdelta3.i
# $(SWIG) -python -o xdelta3_pywrap.c xdelta3.i
# OS X for some reason requires:
# pythondir = $(PYTHON_SITE_PKG)
# pyexecdir = $(PYTHON_SITE_PKG)
#python_PYTHON = $(XDELTA3PY)
#pyexec_LTLIBRARIES = $(XDELTA3PYLIB)
#_xdelta3_la_SOURCES = $(srcdir)/xdelta3_pywrap.c $(xdelta3_SOURCES)
#_xdelta3_la_CFLAGS = $(common_CFLAGS) -DNOT_MAIN=1 $(PYTHON_CPPFLAGS)
#_xdelta3_la_LDFLAGS = -module

37
lib/xdelta3/README.md Normal file
View file

@ -0,0 +1,37 @@
Xdelta 3.x readme.txt
Copyright (C) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008,
2009, 2010, 2011, 2012, 2013, 2014, 2015
<josh.macdonald@gmail.com>
Thanks for downloading Xdelta!
This directory contains the Xdelta3 command-line interface (CLI) and source
distribution for VCDIFF differential compression, a.k.a. delta
compression. The latest information and downloads are available here:
http://xdelta.org/
http://github.com/jmacd/xdelta/
Xdelta can be configured to use XZ Utils for secondary compression:
http://tukaani.org/xz/
The command-line syntax is detailed here:
https://github.com/jmacd/xdelta/blob/wiki/CommandLineSyntax.md
Run 'xdelta3 -h' for brief help. Run 'xdelta3 test' for built-in tests.
Sample commands (like gzip, -e means encode, -d means decode)
xdelta3 -9 -S lzma -e -f -s OLD_FILE NEW_FILE DELTA_FILE
xdelta3 -d -s OLD_FILE DELTA_FILE DECODED_FILE
File bug reports and browse open support issues here:
https://github.com/jmacd/xdelta/issues
The source distribution contains the C/C++/Python APIs, Unix, Microsoft VC++
and Cygwin builds. Xdelta3 is covered under the terms of the APL, see
LICENSE.

158
lib/xdelta3/badcopy.c Normal file
View file

@ -0,0 +1,158 @@
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#define BUFSZ (1 << 22)
#ifdef WIN32
// whatever
static
double drand48() {
double r = rand() / (double)RAND_MAX;
return r;
}
long lrand48() {
long l = 0;
int i;
for (i = 0; i < 32; i++) {
l = l ^ (l << 2) ^ (l << 1) ^ rand();
}
return l;
}
#endif
#ifdef _WIN32
#define XD3_WIN32 1
#else
#define XD3_POSIX 1
#endif
#define XD3_MAIN 1
#define main notmain
#define EXTERNAL_COMPRESSION 0
#define XD3_USE_LARGEFILE64 1
#include "xdelta3.c"
#undef main
double error_prob = 0.0001;
usize_t mean_change = 100;
xoff_t total_change = 0;
xoff_t total_size = 0;
usize_t max_change = 0;
usize_t num_change = 0;
static usize_t
edist (usize_t mean, usize_t max)
{
double mean_d = mean;
double erand = log (1.0 / drand48 ());
usize_t x = (usize_t) (mean_d * erand + 0.5);
return (x < max) ? (x > 0 ? x : 1) : max;
}
void modify (char *buf, usize_t size)
{
usize_t bufpos = 0, j;
usize_t last_end = 0;
for (;; /* bufpos and j are incremented in the inner loop */)
{
/* The size of the next modification. */
usize_t next_size = edist (mean_change, 1 << 31);
/* The expected interval of such a change. */
double expect_interval = ((double) next_size * (1.0 - error_prob)) / error_prob;
/* The number of bytes until the next modification. */
usize_t next_mod = edist ((usize_t)expect_interval, 1 << 31);
if (next_size + next_mod + bufpos > size) { break; }
if (max_change < next_size) { max_change = next_size; }
bufpos += next_mod;
fprintf (stderr, "COPY: %I64u-%I64u (%u)\n",
total_size + (xoff_t)last_end,
total_size + (xoff_t)bufpos,
bufpos - last_end);
fprintf (stderr, "ADD: %I64u-%I64u (%u) is change %u\n",
total_size + (xoff_t)bufpos,
total_size + (xoff_t)(bufpos + next_size),
next_size, num_change);
total_change += next_size;
num_change += 1;
for (j = 0; j < next_size; j += 1, bufpos += 1)
{
buf[bufpos] = (char)(lrand48 () >> 3);
}
last_end = bufpos;
}
fprintf (stderr, "COPY: %I64u-%I64u (%u)\n",
total_size + last_end,
total_size + size, size - last_end);
total_size += size;
}
int main(int argc, char **argv)
{
main_file inp, out;
char *buf = malloc(BUFSZ);
int c, ret;
main_file_init(&inp);
main_file_init(&out);
option_force = 1;
if (argc > 5)
{
fprintf (stderr, "usage: badcopy [byte_error_prob [mean_error_size]]\n");
return 1;
}
if (argc > 4) { mean_change = atoi (argv[4]); }
if (argc > 3) { error_prob = atof (argv[3]); }
fprintf (stderr, "mean change = %u; error_prob = %0.10f\n", mean_change, error_prob);
if ((ret = main_file_open (&inp, argv[1], XO_READ)) != 0) {
return 1;
}
if ((ret = main_file_open (&out, argv[2], XO_WRITE)) != 0) {
return 1;
}
if (error_prob < 0.0 || error_prob > 1.0)
{
fprintf (stderr, "warning: error probability out of range\n");
return 1;
}
do
{
if ((ret = main_file_read (&inp, buf, BUFSZ, &c, "read failed")) != 0) {
return 1;
}
if (c == 0) { break; }
modify (buf, c);
if ((ret = main_file_write (&out, buf, c, "write failed")) != 0) {
return 1;
}
}
while (c == BUFSZ);
if ((ret = main_file_close (&out)))
{
return 1;
}
fprintf (stderr, "add_prob %f; %u adds; total_change %u of %u bytes; add percentage %f; max add size %u\n",
error_prob, num_change, total_change, total_size, (double) total_change / (double) total_size, max_change);
return 0;
}

51
lib/xdelta3/configure.ac Normal file
View file

@ -0,0 +1,51 @@
AC_INIT([Xdelta3], [3.1.1], [josh.macdonald@gmail.com],
[xdelta3], [http://xdelta.org/])
AC_PREREQ([2.68])
AC_CONFIG_MACRO_DIR([m4])
LT_INIT
AM_INIT_AUTOMAKE([1.15 no-define foreign tar-ustar subdir-objects])
AC_CONFIG_MACRO_DIRS([m4])
AX_CHECK_ALIGNED_ACCESS_REQUIRED
AC_PROG_CC
AC_PROG_CXX
AC_CHECK_SIZEOF(size_t)
AC_CHECK_SIZEOF(unsigned int)
AC_CHECK_SIZEOF(unsigned long)
AC_CHECK_SIZEOF(unsigned long long)
AC_ARG_WITH(
[liblzma],
[AC_HELP_STRING(
[--with-liblzma],
[build with liblzma support @<:@default=autodetect@:>@])],
[USE_LIBLZMA=$withval],
[USE_LIBLZMA=auto])
if test "x$USE_LIBLZMA" != xno ; then
AC_CHECK_HEADERS([lzma.h],,[
if test "x$with_liblzma" = xyes ; then
AC_MSG_FAILURE([liblzma includes were not found])
fi])
AC_CHECK_LIB([lzma], [lzma_easy_buffer_encode],,[
if test "x$with_liblzma" = xyes ; then
AC_MSG_FAILURE([liblzma library were not found])
fi])
fi
#AM_PATH_PYTHON(,, [:])
#AM_CONDITIONAL([HAVE_PYTHON], [test "$PYTHON" != :])
#AX_PYTHON_DEVEL()
#AX_PKG_SWIG(2.0.0,,)
#AX_SWIG_PYTHON
dnl --enable-debug-symbols : build with debug symbols?
AC_ARG_ENABLE(debug-symbols,
AS_HELP_STRING(--enable-debug-symbols,[Build with debug symbols (default is NO)]),,enableval=no)
AM_CONDITIONAL([DEBUG_SYMBOLS], [test ${enableval} = "yes"])
AC_CONFIG_HEADERS([config.h])
AC_CONFIG_FILES([Makefile])
AC_OUTPUT

40
lib/xdelta3/cpp-btree/CMakeLists.txt Normal file
View file

@ -0,0 +1,40 @@
# Copyright 2013 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
cmake_minimum_required(VERSION 2.6)
project(cppbtree CXX)
option(build_tests "Build B-tree tests" OFF)
add_definitions(-std=c++11)
set(CMAKE_CXX_FLAGS "-g -O2")
# CMake doesn't have a way to pure template library,
# add_library(cppbtree btree.h btree_map.h btree_set.h
# safe_btree.h safe_btree_map.h safe_btree_set.h)
# set_target_properties(cppbtree PROPERTIES LINKER_LANGUAGE CXX)
if(build_tests)
enable_testing()
include_directories($ENV{GTEST_ROOT}/include)
link_directories($ENV{GTEST_ROOT})
include_directories($ENV{GFLAGS_ROOT}/include)
link_directories($ENV{GFLAGS_ROOT}/lib)
add_executable(btree_test btree_test.cc btree_test_flags.cc)
add_executable(safe_btree_test safe_btree_test.cc btree_test_flags.cc)
add_executable(btree_bench btree_bench.cc btree_test_flags.cc)
target_link_libraries(btree_test gtest_main gtest gflags)
target_link_libraries(safe_btree_test gtest_main gtest gflags)
target_link_libraries(btree_bench gflags gtest)
endif()

202
lib/xdelta3/cpp-btree/COPYING Normal file
View file

@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

31
lib/xdelta3/cpp-btree/README Normal file
View file

@ -0,0 +1,31 @@
This library is a C++ template library and, as such, there is no
library to build and install. Copy the .h files and use them!
See http://code.google.com/p/cpp-btree/wiki/UsageInstructions for
details.
----
To build and run the provided tests, however, you will need to install
CMake, the Google C++ Test framework, and the Google flags package.
Download and install CMake from http://www.cmake.org
Download and build the GoogleTest framework from
http://code.google.com/p/googletest
Download and install gflags from https://code.google.com/p/gflags
Set GTEST_ROOT to the directory where GTEST was built.
Set GFLAGS_ROOT to the directory prefix where GFLAGS is installed.
export GTEST_ROOT=/path/for/gtest-x.y
export GFLAGS_ROOT=/opt
cmake . -Dbuild_tests=ON
For example, to build on a Unix system with the clang++ compiler,
export GTEST_ROOT=$(HOME)/src/googletest
export GFLAGS_ROOT=/opt
cmake . -G "Unix Makefiles" -Dbuild_tests=ON -DCMAKE_CXX_COMPILER=clang++

2394
lib/xdelta3/cpp-btree/btree.h Normal file
View file

File diff suppressed because it is too large Load diff

593
lib/xdelta3/cpp-btree/btree_bench.cc Normal file
View file

@ -0,0 +1,593 @@
// Copyright 2013 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include <stdint.h>
#include <stdlib.h>
#include <algorithm>
#include <functional>
#include <map>
#include <set>
#include <string>
#include <sys/time.h>
#include <type_traits>
#include <vector>
#include "gflags/gflags.h"
#include "btree_map.h"
#include "btree_set.h"
#include "btree_test.h"
DEFINE_int32(test_random_seed, 123456789, "Seed for srand()");
DEFINE_int32(benchmark_max_iters, 10000000, "Maximum test iterations");
DEFINE_int32(benchmark_min_iters, 100, "Minimum test iterations");
DEFINE_int32(benchmark_target_seconds, 1,
"Attempt to benchmark for this many seconds");
using std::allocator;
using std::less;
using std::map;
using std::max;
using std::min;
using std::multimap;
using std::multiset;
using std::set;
using std::string;
using std::vector;
namespace btree {
namespace {
struct RandGen {
typedef ptrdiff_t result_type;
RandGen(result_type seed) {
srand(seed);
}
result_type operator()(result_type l) {
return rand() % l;
}
};
struct BenchmarkRun {
BenchmarkRun(const char *name, void (*func)(int));
void Run();
void Stop();
void Start();
void Reset();
BenchmarkRun *next_benchmark;
const char *benchmark_name;
void (*benchmark_func)(int);
int64_t accum_micros;
int64_t last_started;
};
BenchmarkRun *first_benchmark;
BenchmarkRun *current_benchmark;
int64_t get_micros () {
timeval tv;
gettimeofday(&tv, NULL);
return tv.tv_sec * 1000000 + tv.tv_usec;
}
BenchmarkRun::BenchmarkRun(const char *name, void (*func)(int))
: next_benchmark(first_benchmark),
benchmark_name(name),
benchmark_func(func),
accum_micros(0),
last_started(0) {
first_benchmark = this;
}
#define BTREE_BENCHMARK(name) \
BTREE_BENCHMARK2(#name, name, __COUNTER__)
#define BTREE_BENCHMARK2(name, func, counter) \
BTREE_BENCHMARK3(name, func, counter)
#define BTREE_BENCHMARK3(name, func, counter) \
BenchmarkRun bench ## counter (name, func)
void StopBenchmarkTiming() {
current_benchmark->Stop();
}
void StartBenchmarkTiming() {
current_benchmark->Start();
}
void RunBenchmarks() {
for (BenchmarkRun *bench = first_benchmark; bench;
bench = bench->next_benchmark) {
bench->Run();
}
}
void BenchmarkRun::Start() {
assert(!last_started);
last_started = get_micros();
}
void BenchmarkRun::Stop() {
if (last_started == 0) {
return;
}
accum_micros += get_micros() - last_started;
last_started = 0;
}
void BenchmarkRun::Reset() {
last_started = 0;
accum_micros = 0;
}
void BenchmarkRun::Run() {
assert(current_benchmark == NULL);
current_benchmark = this;
int iters = FLAGS_benchmark_min_iters;
for (;;) {
Reset();
Start();
benchmark_func(iters);
Stop();
if (accum_micros > FLAGS_benchmark_target_seconds * 1000000 ||
iters >= FLAGS_benchmark_max_iters) {
break;
} else if (accum_micros == 0) {
iters *= 100;
} else {
int64_t target_micros = FLAGS_benchmark_target_seconds * 1000000;
iters = target_micros * iters / accum_micros;
}
iters = min(iters, FLAGS_benchmark_max_iters);
}
std::cout << benchmark_name << "\t"
<< accum_micros * 1000 / iters << "\t"
<< iters;
current_benchmark = NULL;
}
// Used to avoid compiler optimizations for these benchmarks.
template <typename T>
void sink(const T& t0) {
volatile T t = t0;
}
// Benchmark insertion of values into a container.
template <typename T>
void BM_Insert(int n) {
typedef typename std::remove_const<typename T::value_type>::type V;
typename KeyOfValue<typename T::key_type, V>::type key_of_value;
// Disable timing while we perform some initialization.
StopBenchmarkTiming();
T container;
vector<V> values = GenerateValues<V>(FLAGS_benchmark_values);
for (int i = 0; i < values.size(); i++) {
container.insert(values[i]);
}
for (int i = 0; i < n; ) {
// Remove and re-insert 10% of the keys
int m = min(n - i, FLAGS_benchmark_values / 10);
for (int j = i; j < i + m; j++) {
int x = j % FLAGS_benchmark_values;
container.erase(key_of_value(values[x]));
}
StartBenchmarkTiming();
for (int j = i; j < i + m; j++) {
int x = j % FLAGS_benchmark_values;
container.insert(values[x]);
}
StopBenchmarkTiming();
i += m;
}
}
// Benchmark lookup of values in a container.
template <typename T>
void BM_Lookup(int n) {
typedef typename std::remove_const<typename T::value_type>::type V;
typename KeyOfValue<typename T::key_type, V>::type key_of_value;
// Disable timing while we perform some initialization.
StopBenchmarkTiming();
T container;
vector<V> values = GenerateValues<V>(FLAGS_benchmark_values);
for (int i = 0; i < values.size(); i++) {
container.insert(values[i]);
}
V r = V();
StartBenchmarkTiming();
for (int i = 0; i < n; i++) {
int m = i % values.size();
r = *container.find(key_of_value(values[m]));
}
StopBenchmarkTiming();
sink(r); // Keep compiler from optimizing away r.
}
// Benchmark lookup of values in a full container, meaning that values
// are inserted in-order to take advantage of biased insertion, which
// yields a full tree.
template <typename T>
void BM_FullLookup(int n) {
typedef typename std::remove_const<typename T::value_type>::type V;
typename KeyOfValue<typename T::key_type, V>::type key_of_value;
// Disable timing while we perform some initialization.
StopBenchmarkTiming();
T container;
vector<V> values = GenerateValues<V>(FLAGS_benchmark_values);
vector<V> sorted(values);
sort(sorted.begin(), sorted.end());
for (int i = 0; i < sorted.size(); i++) {
container.insert(sorted[i]);
}
V r = V();
StartBenchmarkTiming();
for (int i = 0; i < n; i++) {
int m = i % values.size();
r = *container.find(key_of_value(values[m]));
}
StopBenchmarkTiming();
sink(r); // Keep compiler from optimizing away r.
}
// Benchmark deletion of values from a container.
template <typename T>
void BM_Delete(int n) {
typedef typename std::remove_const<typename T::value_type>::type V;
typename KeyOfValue<typename T::key_type, V>::type key_of_value;
// Disable timing while we perform some initialization.
StopBenchmarkTiming();
T container;
vector<V> values = GenerateValues<V>(FLAGS_benchmark_values);
for (int i = 0; i < values.size(); i++) {
container.insert(values[i]);
}
for (int i = 0; i < n; ) {
// Remove and re-insert 10% of the keys
int m = min(n - i, FLAGS_benchmark_values / 10);
StartBenchmarkTiming();
for (int j = i; j < i + m; j++) {
int x = j % FLAGS_benchmark_values;
container.erase(key_of_value(values[x]));
}
StopBenchmarkTiming();
for (int j = i; j < i + m; j++) {
int x = j % FLAGS_benchmark_values;
container.insert(values[x]);
}
i += m;
}
}
// Benchmark steady-state insert (into first half of range) and remove
// (from second second half of range), treating the container
// approximately like a queue with log-time access for all elements.
// This benchmark does not test the case where insertion and removal
// happen in the same region of the tree. This benchmark counts two
// value constructors.
template <typename T>
void BM_QueueAddRem(int n) {
typedef typename std::remove_const<typename T::value_type>::type V;
typename KeyOfValue<typename T::key_type, V>::type key_of_value;
// Disable timing while we perform some initialization.
StopBenchmarkTiming();
assert(FLAGS_benchmark_values % 2 == 0);
T container;
const int half = FLAGS_benchmark_values / 2;
vector<int> remove_keys(half);
vector<int> add_keys(half);
for (int i = 0; i < half; i++) {
remove_keys[i] = i;
add_keys[i] = i;
}
RandGen rand(FLAGS_test_random_seed);
random_shuffle(remove_keys.begin(), remove_keys.end(), rand);
random_shuffle(add_keys.begin(), add_keys.end(), rand);
Generator<V> g(FLAGS_benchmark_values + FLAGS_benchmark_max_iters);
for (int i = 0; i < half; i++) {
container.insert(g(add_keys[i]));
container.insert(g(half + remove_keys[i]));
}
// There are three parts each of size "half":
// 1 is being deleted from [offset - half, offset)
// 2 is standing [offset, offset + half)
// 3 is being inserted into [offset + half, offset + 2 * half)
int offset = 0;
StartBenchmarkTiming();
for (int i = 0; i < n; i++) {
int idx = i % half;
if (idx == 0) {
StopBenchmarkTiming();
random_shuffle(remove_keys.begin(), remove_keys.end(), rand);
random_shuffle(add_keys.begin(), add_keys.end(), rand);
offset += half;
StartBenchmarkTiming();
}
int e = container.erase(key_of_value(g(offset - half + remove_keys[idx])));
assert(e == 1);
container.insert(g(offset + half + add_keys[idx]));
}
StopBenchmarkTiming();
}
// Mixed insertion and deletion in the same range using pre-constructed values.
template <typename T>
void BM_MixedAddRem(int n) {
typedef typename std::remove_const<typename T::value_type>::type V;
typename KeyOfValue<typename T::key_type, V>::type key_of_value;
// Disable timing while we perform some initialization.
StopBenchmarkTiming();
assert(FLAGS_benchmark_values % 2 == 0);
T container;
RandGen rand(FLAGS_test_random_seed);
vector<V> values = GenerateValues<V>(FLAGS_benchmark_values * 2);
// Create two random shuffles
vector<int> remove_keys(FLAGS_benchmark_values);
vector<int> add_keys(FLAGS_benchmark_values);
// Insert the first half of the values (already in random order)
for (int i = 0; i < FLAGS_benchmark_values; i++) {
container.insert(values[i]);
// remove_keys and add_keys will be swapped before each round,
// therefore fill add_keys here w/ the keys being inserted, so
// they'll be the first to be removed.
remove_keys[i] = i + FLAGS_benchmark_values;
add_keys[i] = i;
}
StartBenchmarkTiming();
for (int i = 0; i < n; i++) {
int idx = i % FLAGS_benchmark_values;
if (idx == 0) {
StopBenchmarkTiming();
remove_keys.swap(add_keys);
random_shuffle(remove_keys.begin(), remove_keys.end(), rand);
random_shuffle(add_keys.begin(), add_keys.end(), rand);
StartBenchmarkTiming();
}
int e = container.erase(key_of_value(values[remove_keys[idx]]));
assert(e == 1);
container.insert(values[add_keys[idx]]);
}
StopBenchmarkTiming();
}
// Insertion at end, removal from the beginning. This benchmark
// counts two value constructors.
template <typename T>
void BM_Fifo(int n) {
typedef typename std::remove_const<typename T::value_type>::type V;
// Disable timing while we perform some initialization.
StopBenchmarkTiming();
T container;
Generator<V> g(FLAGS_benchmark_values + FLAGS_benchmark_max_iters);
for (int i = 0; i < FLAGS_benchmark_values; i++) {
container.insert(g(i));
}
StartBenchmarkTiming();
for (int i = 0; i < n; i++) {
container.erase(container.begin());
container.insert(container.end(), g(i + FLAGS_benchmark_values));
}
StopBenchmarkTiming();
}
// Iteration (forward) through the tree
template <typename T>
void BM_FwdIter(int n) {
typedef typename std::remove_const<typename T::value_type>::type V;
// Disable timing while we perform some initialization.
StopBenchmarkTiming();
T container;
vector<V> values = GenerateValues<V>(FLAGS_benchmark_values);
for (int i = 0; i < FLAGS_benchmark_values; i++) {
container.insert(values[i]);
}
typename T::iterator iter;
V r = V();
StartBenchmarkTiming();
for (int i = 0; i < n; i++) {
int idx = i % FLAGS_benchmark_values;
if (idx == 0) {
iter = container.begin();
}
r = *iter;
++iter;
}
StopBenchmarkTiming();
sink(r); // Keep compiler from optimizing away r.
}
typedef set<int32_t> stl_set_int32;
typedef set<int64_t> stl_set_int64;
typedef set<string> stl_set_string;
typedef map<int32_t, intptr_t> stl_map_int32;
typedef map<int64_t, intptr_t> stl_map_int64;
typedef map<string, intptr_t> stl_map_string;
typedef multiset<int32_t> stl_multiset_int32;
typedef multiset<int64_t> stl_multiset_int64;
typedef multiset<string> stl_multiset_string;
typedef multimap<int32_t, intptr_t> stl_multimap_int32;
typedef multimap<int64_t, intptr_t> stl_multimap_int64;
typedef multimap<string, intptr_t> stl_multimap_string;
#define MY_BENCHMARK_TYPES2(value, name, size) \
typedef btree ## _set<value, less<value>, allocator<value>, size> \
btree ## _ ## size ## _set_ ## name; \
typedef btree ## _map<value, int, less<value>, allocator<value>, size> \
btree ## _ ## size ## _map_ ## name; \
typedef btree ## _multiset<value, less<value>, allocator<value>, size> \
btree ## _ ## size ## _multiset_ ## name; \
typedef btree ## _multimap<value, int, less<value>, allocator<value>, size> \
btree ## _ ## size ## _multimap_ ## name
#define MY_BENCHMARK_TYPES(value, name) \
MY_BENCHMARK_TYPES2(value, name, 128); \
MY_BENCHMARK_TYPES2(value, name, 160); \
MY_BENCHMARK_TYPES2(value, name, 192); \
MY_BENCHMARK_TYPES2(value, name, 224); \
MY_BENCHMARK_TYPES2(value, name, 256); \
MY_BENCHMARK_TYPES2(value, name, 288); \
MY_BENCHMARK_TYPES2(value, name, 320); \
MY_BENCHMARK_TYPES2(value, name, 352); \
MY_BENCHMARK_TYPES2(value, name, 384); \
MY_BENCHMARK_TYPES2(value, name, 416); \
MY_BENCHMARK_TYPES2(value, name, 448); \
MY_BENCHMARK_TYPES2(value, name, 480); \
MY_BENCHMARK_TYPES2(value, name, 512); \
MY_BENCHMARK_TYPES2(value, name, 1024); \
MY_BENCHMARK_TYPES2(value, name, 1536); \
MY_BENCHMARK_TYPES2(value, name, 2048)
MY_BENCHMARK_TYPES(int32_t, int32);
MY_BENCHMARK_TYPES(int64_t, int64);
MY_BENCHMARK_TYPES(string, string);
#define MY_BENCHMARK4(type, name, func) \
void BM_ ## type ## _ ## name(int n) { BM_ ## func <type>(n); } \
BTREE_BENCHMARK(BM_ ## type ## _ ## name)
// Define NODESIZE_TESTING when running btree_perf.py.
#ifdef NODESIZE_TESTING
#define MY_BENCHMARK3(tree, type, name, func) \
MY_BENCHMARK4(tree ## _128_ ## type, name, func); \
MY_BENCHMARK4(tree ## _160_ ## type, name, func); \
MY_BENCHMARK4(tree ## _192_ ## type, name, func); \
MY_BENCHMARK4(tree ## _224_ ## type, name, func); \
MY_BENCHMARK4(tree ## _256_ ## type, name, func); \
MY_BENCHMARK4(tree ## _288_ ## type, name, func); \
MY_BENCHMARK4(tree ## _320_ ## type, name, func); \
MY_BENCHMARK4(tree ## _352_ ## type, name, func); \
MY_BENCHMARK4(tree ## _384_ ## type, name, func); \
MY_BENCHMARK4(tree ## _416_ ## type, name, func); \
MY_BENCHMARK4(tree ## _448_ ## type, name, func); \
MY_BENCHMARK4(tree ## _480_ ## type, name, func); \
MY_BENCHMARK4(tree ## _512_ ## type, name, func); \
MY_BENCHMARK4(tree ## _1024_ ## type, name, func); \
MY_BENCHMARK4(tree ## _1536_ ## type, name, func); \
MY_BENCHMARK4(tree ## _2048_ ## type, name, func)
#else
#define MY_BENCHMARK3(tree, type, name, func) \
MY_BENCHMARK4(tree ## _256_ ## type, name, func); \
MY_BENCHMARK4(tree ## _2048_ ## type, name, func)
#endif
#define MY_BENCHMARK2(type, name, func) \
MY_BENCHMARK4(stl_ ## type, name, func); \
MY_BENCHMARK3(btree, type, name, func)
#define MY_BENCHMARK(type) \
MY_BENCHMARK2(type, insert, Insert); \
MY_BENCHMARK2(type, lookup, Lookup); \
MY_BENCHMARK2(type, fulllookup, FullLookup); \
MY_BENCHMARK2(type, delete, Delete); \
MY_BENCHMARK2(type, queueaddrem, QueueAddRem); \
MY_BENCHMARK2(type, mixedaddrem, MixedAddRem); \
MY_BENCHMARK2(type, fifo, Fifo); \
MY_BENCHMARK2(type, fwditer, FwdIter)
MY_BENCHMARK(set_int32);
MY_BENCHMARK(map_int32);
MY_BENCHMARK(set_int64);
MY_BENCHMARK(map_int64);
MY_BENCHMARK(set_string);
MY_BENCHMARK(map_string);
MY_BENCHMARK(multiset_int32);
MY_BENCHMARK(multimap_int32);
MY_BENCHMARK(multiset_int64);
MY_BENCHMARK(multimap_int64);
MY_BENCHMARK(multiset_string);
MY_BENCHMARK(multimap_string);
} // namespace
} // namespace btree
int main(int argc, char **argv) {
btree::RunBenchmarks();
return 0;
}

349
lib/xdelta3/cpp-btree/btree_container.h Normal file
View file

@ -0,0 +1,349 @@
// Copyright 2013 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#ifndef UTIL_BTREE_BTREE_CONTAINER_H__
#define UTIL_BTREE_BTREE_CONTAINER_H__
#include <iosfwd>
#include <utility>
#include "btree.h"
namespace btree {
// A common base class for btree_set, btree_map, btree_multiset and
// btree_multimap.
template <typename Tree>
class btree_container {
typedef btree_container<Tree> self_type;
public:
typedef typename Tree::params_type params_type;
typedef typename Tree::key_type key_type;
typedef typename Tree::value_type value_type;
typedef typename Tree::key_compare key_compare;
typedef typename Tree::allocator_type allocator_type;
typedef typename Tree::pointer pointer;
typedef typename Tree::const_pointer const_pointer;
typedef typename Tree::reference reference;
typedef typename Tree::const_reference const_reference;
typedef typename Tree::size_type size_type;
typedef typename Tree::difference_type difference_type;
typedef typename Tree::iterator iterator;
typedef typename Tree::const_iterator const_iterator;
typedef typename Tree::reverse_iterator reverse_iterator;
typedef typename Tree::const_reverse_iterator const_reverse_iterator;
public:
// Default constructor.
btree_container(const key_compare &comp, const allocator_type &alloc)
: tree_(comp, alloc) {
}
// Copy constructor.
btree_container(const self_type &x)
: tree_(x.tree_) {
}
// Iterator routines.
iterator begin() { return tree_.begin(); }
const_iterator begin() const { return tree_.begin(); }
iterator end() { return tree_.end(); }
const_iterator end() const { return tree_.end(); }
reverse_iterator rbegin() { return tree_.rbegin(); }
const_reverse_iterator rbegin() const { return tree_.rbegin(); }
reverse_iterator rend() { return tree_.rend(); }
const_reverse_iterator rend() const { return tree_.rend(); }
// Lookup routines.
iterator lower_bound(const key_type &key) {
return tree_.lower_bound(key);
}
const_iterator lower_bound(const key_type &key) const {
return tree_.lower_bound(key);
}
iterator upper_bound(const key_type &key) {
return tree_.upper_bound(key);
}
const_iterator upper_bound(const key_type &key) const {
return tree_.upper_bound(key);
}
std::pair<iterator,iterator> equal_range(const key_type &key) {
return tree_.equal_range(key);
}
std::pair<const_iterator,const_iterator> equal_range(const key_type &key) const {
return tree_.equal_range(key);
}
// Utility routines.
void clear() {
tree_.clear();
}
void swap(self_type &x) {
tree_.swap(x.tree_);
}
void dump(std::ostream &os) const {
tree_.dump(os);
}
void verify() const {
tree_.verify();
}
// Size routines.
size_type size() const { return tree_.size(); }
size_type max_size() const { return tree_.max_size(); }
bool empty() const { return tree_.empty(); }
size_type height() const { return tree_.height(); }
size_type internal_nodes() const { return tree_.internal_nodes(); }
size_type leaf_nodes() const { return tree_.leaf_nodes(); }
size_type nodes() const { return tree_.nodes(); }
size_type bytes_used() const { return tree_.bytes_used(); }
static double average_bytes_per_value() {
return Tree::average_bytes_per_value();
}
double fullness() const { return tree_.fullness(); }
double overhead() const { return tree_.overhead(); }
bool operator==(const self_type& x) const {
if (size() != x.size()) {
return false;
}
for (const_iterator i = begin(), xi = x.begin(); i != end(); ++i, ++xi) {
if (*i != *xi) {
return false;
}
}
return true;
}
bool operator!=(const self_type& other) const {
return !operator==(other);
}
protected:
Tree tree_;
};
template <typename T>
inline std::ostream& operator<<(std::ostream &os, const btree_container<T> &b) {
b.dump(os);
return os;
}
// A common base class for btree_set and safe_btree_set.
template <typename Tree>
class btree_unique_container : public btree_container<Tree> {
typedef btree_unique_container<Tree> self_type;
typedef btree_container<Tree> super_type;
public:
typedef typename Tree::key_type key_type;
typedef typename Tree::value_type value_type;
typedef typename Tree::size_type size_type;
typedef typename Tree::key_compare key_compare;
typedef typename Tree::allocator_type allocator_type;
typedef typename Tree::iterator iterator;
typedef typename Tree::const_iterator const_iterator;
public:
// Default constructor.
btree_unique_container(const key_compare &comp = key_compare(),
const allocator_type &alloc = allocator_type())
: super_type(comp, alloc) {
}
// Copy constructor.
btree_unique_container(const self_type &x)
: super_type(x) {
}
// Range constructor.
template <class InputIterator>
btree_unique_container(InputIterator b, InputIterator e,
const key_compare &comp = key_compare(),
const allocator_type &alloc = allocator_type())
: super_type(comp, alloc) {
insert(b, e);
}
// Lookup routines.
iterator find(const key_type &key) {
return this->tree_.find_unique(key);
}
const_iterator find(const key_type &key) const {
return this->tree_.find_unique(key);
}
size_type count(const key_type &key) const {
return this->tree_.count_unique(key);
}
// Insertion routines.
std::pair<iterator,bool> insert(const value_type &x) {
return this->tree_.insert_unique(x);
}
iterator insert(iterator position, const value_type &x) {
return this->tree_.insert_unique(position, x);
}
template <typename InputIterator>
void insert(InputIterator b, InputIterator e) {
this->tree_.insert_unique(b, e);
}
// Deletion routines.
int erase(const key_type &key) {
return this->tree_.erase_unique(key);
}
// Erase the specified iterator from the btree. The iterator must be valid
// (i.e. not equal to end()). Return an iterator pointing to the node after
// the one that was erased (or end() if none exists).
iterator erase(const iterator &iter) {
return this->tree_.erase(iter);
}
void erase(const iterator &first, const iterator &last) {
this->tree_.erase(first, last);
}
};
// A common base class for btree_map and safe_btree_map.
template <typename Tree>
class btree_map_container : public btree_unique_container<Tree> {
typedef btree_map_container<Tree> self_type;
typedef btree_unique_container<Tree> super_type;
public:
typedef typename Tree::key_type key_type;
typedef typename Tree::data_type data_type;
typedef typename Tree::value_type value_type;
typedef typename Tree::mapped_type mapped_type;
typedef typename Tree::key_compare key_compare;
typedef typename Tree::allocator_type allocator_type;
private:
// A pointer-like object which only generates its value when
// dereferenced. Used by operator[] to avoid constructing an empty data_type
// if the key already exists in the map.
struct generate_value {
generate_value(const key_type &k)
: key(k) {
}
value_type operator*() const {
return std::make_pair(key, data_type());
}
const key_type &key;
};
public:
// Default constructor.
btree_map_container(const key_compare &comp = key_compare(),
const allocator_type &alloc = allocator_type())
: super_type(comp, alloc) {
}
// Copy constructor.
btree_map_container(const self_type &x)
: super_type(x) {
}
// Range constructor.
template <class InputIterator>
btree_map_container(InputIterator b, InputIterator e,
const key_compare &comp = key_compare(),
const allocator_type &alloc = allocator_type())
: super_type(b, e, comp, alloc) {
}
// Insertion routines.
data_type& operator[](const key_type &key) {
return this->tree_.insert_unique(key, generate_value(key)).first->second;
}
};
// A common base class for btree_multiset and btree_multimap.
template <typename Tree>
class btree_multi_container : public btree_container<Tree> {
typedef btree_multi_container<Tree> self_type;
typedef btree_container<Tree> super_type;
public:
typedef typename Tree::key_type key_type;
typedef typename Tree::value_type value_type;
typedef typename Tree::size_type size_type;
typedef typename Tree::key_compare key_compare;
typedef typename Tree::allocator_type allocator_type;
typedef typename Tree::iterator iterator;
typedef typename Tree::const_iterator const_iterator;
public:
// Default constructor.
btree_multi_container(const key_compare &comp = key_compare(),
const allocator_type &alloc = allocator_type())
: super_type(comp, alloc) {
}
// Copy constructor.
btree_multi_container(const self_type &x)
: super_type(x) {
}
// Range constructor.
template <class InputIterator>
btree_multi_container(InputIterator b, InputIterator e,
const key_compare &comp = key_compare(),
const allocator_type &alloc = allocator_type())
: super_type(comp, alloc) {
insert(b, e);
}
// Lookup routines.
iterator find(const key_type &key) {
return this->tree_.find_multi(key);
}
const_iterator find(const key_type &key) const {
return this->tree_.find_multi(key);
}
size_type count(const key_type &key) const {
return this->tree_.count_multi(key);
}
// Insertion routines.
iterator insert(const value_type &x) {
return this->tree_.insert_multi(x);
}
iterator insert(iterator position, const value_type &x) {
return this->tree_.insert_multi(position, x);
}
template <typename InputIterator>
void insert(InputIterator b, InputIterator e) {
this->tree_.insert_multi(b, e);
}
// Deletion routines.
int erase(const key_type &key) {
return this->tree_.erase_multi(key);
}
// Erase the specified iterator from the btree. The iterator must be valid
// (i.e. not equal to end()). Return an iterator pointing to the node after
// the one that was erased (or end() if none exists).
iterator erase(const iterator &iter) {
return this->tree_.erase(iter);
}
void erase(const iterator &first, const iterator &last) {
this->tree_.erase(first, last);
}
};
} // namespace btree
#endif // UTIL_BTREE_BTREE_CONTAINER_H__

130
lib/xdelta3/cpp-btree/btree_map.h Normal file
View file

@ -0,0 +1,130 @@
// Copyright 2013 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
// A btree_map<> implements the STL unique sorted associative container
// interface and the pair associative container interface (a.k.a map<>) using a
// btree. A btree_multimap<> implements the STL multiple sorted associative
// container interface and the pair associtive container interface (a.k.a
// multimap<>) using a btree. See btree.h for details of the btree
// implementation and caveats.
#ifndef UTIL_BTREE_BTREE_MAP_H__
#define UTIL_BTREE_BTREE_MAP_H__
#include <algorithm>
#include <functional>
#include <memory>
#include <string>
#include <utility>
#include "btree.h"
#include "btree_container.h"
namespace btree {
// The btree_map class is needed mainly for its constructors.
template <typename Key, typename Value,
typename Compare = std::less<Key>,
typename Alloc = std::allocator<std::pair<const Key, Value> >,
int TargetNodeSize = 256>
class btree_map : public btree_map_container<
btree<btree_map_params<Key, Value, Compare, Alloc, TargetNodeSize> > > {
typedef btree_map<Key, Value, Compare, Alloc, TargetNodeSize> self_type;
typedef btree_map_params<
Key, Value, Compare, Alloc, TargetNodeSize> params_type;
typedef btree<params_type> btree_type;
typedef btree_map_container<btree_type> super_type;
public:
typedef typename btree_type::key_compare key_compare;
typedef typename btree_type::allocator_type allocator_type;
public:
// Default constructor.
btree_map(const key_compare &comp = key_compare(),
const allocator_type &alloc = allocator_type())
: super_type(comp, alloc) {
}
// Copy constructor.
btree_map(const self_type &x)
: super_type(x) {
}
// Range constructor.
template <class InputIterator>
btree_map(InputIterator b, InputIterator e,
const key_compare &comp = key_compare(),
const allocator_type &alloc = allocator_type())
: super_type(b, e, comp, alloc) {
}
};
template <typename K, typename V, typename C, typename A, int N>
inline void swap(btree_map<K, V, C, A, N> &x,
btree_map<K, V, C, A, N> &y) {
x.swap(y);
}
// The btree_multimap class is needed mainly for its constructors.
template <typename Key, typename Value,
typename Compare = std::less<Key>,
typename Alloc = std::allocator<std::pair<const Key, Value> >,
int TargetNodeSize = 256>
class btree_multimap : public btree_multi_container<
btree<btree_map_params<Key, Value, Compare, Alloc, TargetNodeSize> > > {
typedef btree_multimap<Key, Value, Compare, Alloc, TargetNodeSize> self_type;
typedef btree_map_params<
Key, Value, Compare, Alloc, TargetNodeSize> params_type;
typedef btree<params_type> btree_type;
typedef btree_multi_container<btree_type> super_type;
public:
typedef typename btree_type::key_compare key_compare;
typedef typename btree_type::allocator_type allocator_type;
typedef typename btree_type::data_type data_type;
typedef typename btree_type::mapped_type mapped_type;
public:
// Default constructor.
btree_multimap(const key_compare &comp = key_compare(),
const allocator_type &alloc = allocator_type())
: super_type(comp, alloc) {
}
// Copy constructor.
btree_multimap(const self_type &x)
: super_type(x) {
}
// Range constructor.
template <class InputIterator>
btree_multimap(InputIterator b, InputIterator e,
const key_compare &comp = key_compare(),
const allocator_type &alloc = allocator_type())
: super_type(b, e, comp, alloc) {
}
};
template <typename K, typename V, typename C, typename A, int N>
inline void swap(btree_multimap<K, V, C, A, N> &x,
btree_multimap<K, V, C, A, N> &y) {
x.swap(y);
}
} // namespace btree
#endif // UTIL_BTREE_BTREE_MAP_H__

121
lib/xdelta3/cpp-btree/btree_set.h Normal file
View file

@ -0,0 +1,121 @@
// Copyright 2013 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
// A btree_set<> implements the STL unique sorted associative container
// interface (a.k.a set<>) using a btree. A btree_multiset<> implements the STL
// multiple sorted associative container interface (a.k.a multiset<>) using a
// btree. See btree.h for details of the btree implementation and caveats.
#ifndef UTIL_BTREE_BTREE_SET_H__
#define UTIL_BTREE_BTREE_SET_H__
#include <functional>
#include <memory>
#include <string>
#include "btree.h"
#include "btree_container.h"
namespace btree {
// The btree_set class is needed mainly for its constructors.
template <typename Key,
typename Compare = std::less<Key>,
typename Alloc = std::allocator<Key>,
int TargetNodeSize = 256>
class btree_set : public btree_unique_container<
btree<btree_set_params<Key, Compare, Alloc, TargetNodeSize> > > {
typedef btree_set<Key, Compare, Alloc, TargetNodeSize> self_type;
typedef btree_set_params<Key, Compare, Alloc, TargetNodeSize> params_type;
typedef btree<params_type> btree_type;
typedef btree_unique_container<btree_type> super_type;
public:
typedef typename btree_type::key_compare key_compare;
typedef typename btree_type::allocator_type allocator_type;
public:
// Default constructor.
btree_set(const key_compare &comp = key_compare(),
const allocator_type &alloc = allocator_type())
: super_type(comp, alloc) {
}
// Copy constructor.
btree_set(const self_type &x)
: super_type(x) {
}
// Range constructor.
template <class InputIterator>
btree_set(InputIterator b, InputIterator e,
const key_compare &comp = key_compare(),
const allocator_type &alloc = allocator_type())
: super_type(b, e, comp, alloc) {
}
};
template <typename K, typename C, typename A, int N>
inline void swap(btree_set<K, C, A, N> &x, btree_set<K, C, A, N> &y) {
x.swap(y);
}
// The btree_multiset class is needed mainly for its constructors.
template <typename Key,
typename Compare = std::less<Key>,
typename Alloc = std::allocator<Key>,
int TargetNodeSize = 256>
class btree_multiset : public btree_multi_container<
btree<btree_set_params<Key, Compare, Alloc, TargetNodeSize> > > {
typedef btree_multiset<Key, Compare, Alloc, TargetNodeSize> self_type;
typedef btree_set_params<Key, Compare, Alloc, TargetNodeSize> params_type;
typedef btree<params_type> btree_type;
typedef btree_multi_container<btree_type> super_type;
public:
typedef typename btree_type::key_compare key_compare;
typedef typename btree_type::allocator_type allocator_type;
public:
// Default constructor.
btree_multiset(const key_compare &comp = key_compare(),
const allocator_type &alloc = allocator_type())
: super_type(comp, alloc) {
}
// Copy constructor.
btree_multiset(const self_type &x)
: super_type(x) {
}
// Range constructor.
template <class InputIterator>
btree_multiset(InputIterator b, InputIterator e,
const key_compare &comp = key_compare(),
const allocator_type &alloc = allocator_type())
: super_type(b, e, comp, alloc) {
}
};
template <typename K, typename C, typename A, int N>
inline void swap(btree_multiset<K, C, A, N> &x,
btree_multiset<K, C, A, N> &y) {
x.swap(y);
}
} // namespace btree
#endif // UTIL_BTREE_BTREE_SET_H__

270
lib/xdelta3/cpp-btree/btree_test.cc Normal file
View file

@ -0,0 +1,270 @@
// Copyright 2013 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include "gtest/gtest.h"
#include "btree_map.h"
#include "btree_set.h"
#include "btree_test.h"
namespace btree {
namespace {
template <typename K, int N>
void SetTest() {
typedef TestAllocator<K> TestAlloc;
ASSERT_EQ(sizeof(btree_set<K>), sizeof(void*));
BtreeTest<btree_set<K, std::less<K>, std::allocator<K>, N>, std::set<K> >();
BtreeAllocatorTest<btree_set<K, std::less<K>, TestAlloc, N> >();
}
template <typename K, int N>
void MapTest() {
typedef TestAllocator<K> TestAlloc;
ASSERT_EQ(sizeof(btree_map<K, K>), sizeof(void*));
BtreeTest<btree_map<K, K, std::less<K>, std::allocator<K>, N>, std::map<K, K> >();
BtreeAllocatorTest<btree_map<K, K, std::less<K>, TestAlloc, N> >();
BtreeMapTest<btree_map<K, K, std::less<K>, std::allocator<K>, N> >();
}
TEST(Btree, set_int32_32) { SetTest<int32_t, 32>(); }
TEST(Btree, set_int32_64) { SetTest<int32_t, 64>(); }
TEST(Btree, set_int32_128) { SetTest<int32_t, 128>(); }
TEST(Btree, set_int32_256) { SetTest<int32_t, 256>(); }
TEST(Btree, set_int64_256) { SetTest<int64_t, 256>(); }
TEST(Btree, set_string_256) { SetTest<std::string, 256>(); }
TEST(Btree, set_pair_256) { SetTest<std::pair<int, int>, 256>(); }
TEST(Btree, map_int32_256) { MapTest<int32_t, 256>(); }
TEST(Btree, map_int64_256) { MapTest<int64_t, 256>(); }
TEST(Btree, map_string_256) { MapTest<std::string, 256>(); }
TEST(Btree, map_pair_256) { MapTest<std::pair<int, int>, 256>(); }
// Large-node tests
TEST(Btree, map_int32_1024) { MapTest<int32_t, 1024>(); }
TEST(Btree, map_int32_1032) { MapTest<int32_t, 1032>(); }
TEST(Btree, map_int32_1040) { MapTest<int32_t, 1040>(); }
TEST(Btree, map_int32_1048) { MapTest<int32_t, 1048>(); }
TEST(Btree, map_int32_1056) { MapTest<int32_t, 1056>(); }
TEST(Btree, map_int32_2048) { MapTest<int32_t, 2048>(); }
TEST(Btree, map_int32_4096) { MapTest<int32_t, 4096>(); }
TEST(Btree, set_int32_1024) { SetTest<int32_t, 1024>(); }
TEST(Btree, set_int32_2048) { SetTest<int32_t, 2048>(); }
TEST(Btree, set_int32_4096) { SetTest<int32_t, 4096>(); }
TEST(Btree, map_string_1024) { MapTest<std::string, 1024>(); }
TEST(Btree, map_string_2048) { MapTest<std::string, 2048>(); }
TEST(Btree, map_string_4096) { MapTest<std::string, 4096>(); }
TEST(Btree, set_string_1024) { SetTest<std::string, 1024>(); }
TEST(Btree, set_string_2048) { SetTest<std::string, 2048>(); }
TEST(Btree, set_string_4096) { SetTest<std::string, 4096>(); }
template <typename K, int N>
void MultiSetTest() {
typedef TestAllocator<K> TestAlloc;
ASSERT_EQ(sizeof(btree_multiset<K>), sizeof(void*));
BtreeMultiTest<btree_multiset<K, std::less<K>, std::allocator<K>, N>,
std::multiset<K> >();
BtreeAllocatorTest<btree_multiset<K, std::less<K>, TestAlloc, N> >();
}
template <typename K, int N>
void MultiMapTest() {
typedef TestAllocator<K> TestAlloc;
ASSERT_EQ(sizeof(btree_multimap<K, K>), sizeof(void*));
BtreeMultiTest<btree_multimap<K, K, std::less<K>, std::allocator<K>, N>,
std::multimap<K, K> >();
BtreeMultiMapTest<btree_multimap<K, K, std::less<K>, std::allocator<K>, N> >();
BtreeAllocatorTest<btree_multimap<K, K, std::less<K>, TestAlloc, N> >();
}
TEST(Btree, multiset_int32_256) { MultiSetTest<int32_t, 256>(); }
TEST(Btree, multiset_int64_256) { MultiSetTest<int64_t, 256>(); }
TEST(Btree, multiset_string_256) { MultiSetTest<std::string, 256>(); }
TEST(Btree, multiset_pair_256) { MultiSetTest<std::pair<int, int>, 256>(); }
TEST(Btree, multimap_int32_256) { MultiMapTest<int32_t, 256>(); }
TEST(Btree, multimap_int64_256) { MultiMapTest<int64_t, 256>(); }
TEST(Btree, multimap_string_256) { MultiMapTest<std::string, 256>(); }
TEST(Btree, multimap_pair_256) { MultiMapTest<std::pair<int, int>, 256>(); }
// Large-node tests
TEST(Btree, multimap_int32_1024) { MultiMapTest<int32_t, 1024>(); }
TEST(Btree, multimap_int32_2048) { MultiMapTest<int32_t, 2048>(); }
TEST(Btree, multimap_int32_4096) { MultiMapTest<int32_t, 4096>(); }
TEST(Btree, multiset_int32_1024) { MultiSetTest<int32_t, 1024>(); }
TEST(Btree, multiset_int32_2048) { MultiSetTest<int32_t, 2048>(); }
TEST(Btree, multiset_int32_4096) { MultiSetTest<int32_t, 4096>(); }
TEST(Btree, multimap_string_1024) { MultiMapTest<std::string, 1024>(); }
TEST(Btree, multimap_string_2048) { MultiMapTest<std::string, 2048>(); }
TEST(Btree, multimap_string_4096) { MultiMapTest<std::string, 4096>(); }
TEST(Btree, multiset_string_1024) { MultiSetTest<std::string, 1024>(); }
TEST(Btree, multiset_string_2048) { MultiSetTest<std::string, 2048>(); }
TEST(Btree, multiset_string_4096) { MultiSetTest<std::string, 4096>(); }
// Verify that swapping btrees swaps the key comparision functors.
struct SubstringLess {
SubstringLess() : n(2) {}
SubstringLess(size_t length)
: n(length) {
}
bool operator()(const std::string &a, const std::string &b) const {
std::string as(a.data(), std::min(n, a.size()));
std::string bs(b.data(), std::min(n, b.size()));
return as < bs;
}
size_t n;
};
TEST(Btree, SwapKeyCompare) {
typedef btree_set<std::string, SubstringLess> SubstringSet;
SubstringSet s1(SubstringLess(1), SubstringSet::allocator_type());
SubstringSet s2(SubstringLess(2), SubstringSet::allocator_type());
ASSERT_TRUE(s1.insert("a").second);
ASSERT_FALSE(s1.insert("aa").second);
ASSERT_TRUE(s2.insert("a").second);
ASSERT_TRUE(s2.insert("aa").second);
ASSERT_FALSE(s2.insert("aaa").second);
swap(s1, s2);
ASSERT_TRUE(s1.insert("b").second);
ASSERT_TRUE(s1.insert("bb").second);
ASSERT_FALSE(s1.insert("bbb").second);
ASSERT_TRUE(s2.insert("b").second);
ASSERT_FALSE(s2.insert("bb").second);
}
TEST(Btree, UpperBoundRegression) {
// Regress a bug where upper_bound would default-construct a new key_compare
// instead of copying the existing one.
typedef btree_set<std::string, SubstringLess> SubstringSet;
SubstringSet my_set(SubstringLess(3));
my_set.insert("aab");
my_set.insert("abb");
// We call upper_bound("aaa"). If this correctly uses the length 3
// comparator, aaa < aab < abb, so we should get aab as the result.
// If it instead uses the default-constructed length 2 comparator,
// aa == aa < ab, so we'll get abb as our result.
SubstringSet::iterator it = my_set.upper_bound("aaa");
ASSERT_TRUE(it != my_set.end());
EXPECT_EQ("aab", *it);
}
TEST(Btree, IteratorIncrementBy) {
// Test that increment_by returns the same position as increment.
const int kSetSize = 2341;
btree_set<int32_t> my_set;
for (int i = 0; i < kSetSize; ++i) {
my_set.insert(i);
}
{
// Simple increment vs. increment by.
btree_set<int32_t>::iterator a = my_set.begin();
btree_set<int32_t>::iterator b = my_set.begin();
a.increment();
b.increment_by(1);
EXPECT_EQ(*a, *b);
}
btree_set<int32_t>::iterator a = my_set.begin();
for (int i = 1; i < kSetSize; ++i) {
++a;
// increment_by
btree_set<int32_t>::iterator b = my_set.begin();
b.increment_by(i);
EXPECT_EQ(*a, *b) << ": i=" << i;
}
}
TEST(Btree, Comparison) {
const int kSetSize = 1201;
btree_set<int64_t> my_set;
for (int i = 0; i < kSetSize; ++i) {
my_set.insert(i);
}
btree_set<int64_t> my_set_copy(my_set);
EXPECT_TRUE(my_set_copy == my_set);
EXPECT_TRUE(my_set == my_set_copy);
EXPECT_FALSE(my_set_copy != my_set);
EXPECT_FALSE(my_set != my_set_copy);
my_set.insert(kSetSize);
EXPECT_FALSE(my_set_copy == my_set);
EXPECT_FALSE(my_set == my_set_copy);
EXPECT_TRUE(my_set_copy != my_set);
EXPECT_TRUE(my_set != my_set_copy);
my_set.erase(kSetSize - 1);
EXPECT_FALSE(my_set_copy == my_set);
EXPECT_FALSE(my_set == my_set_copy);
EXPECT_TRUE(my_set_copy != my_set);
EXPECT_TRUE(my_set != my_set_copy);
btree_map<std::string, int64_t> my_map;
for (int i = 0; i < kSetSize; ++i) {
my_map[std::string(i, 'a')] = i;
}
btree_map<std::string, int64_t> my_map_copy(my_map);
EXPECT_TRUE(my_map_copy == my_map);
EXPECT_TRUE(my_map == my_map_copy);
EXPECT_FALSE(my_map_copy != my_map);
EXPECT_FALSE(my_map != my_map_copy);
++my_map_copy[std::string(7, 'a')];
EXPECT_FALSE(my_map_copy == my_map);
EXPECT_FALSE(my_map == my_map_copy);
EXPECT_TRUE(my_map_copy != my_map);
EXPECT_TRUE(my_map != my_map_copy);
my_map_copy = my_map;
my_map["hello"] = kSetSize;
EXPECT_FALSE(my_map_copy == my_map);
EXPECT_FALSE(my_map == my_map_copy);
EXPECT_TRUE(my_map_copy != my_map);
EXPECT_TRUE(my_map != my_map_copy);
my_map.erase(std::string(kSetSize - 1, 'a'));
EXPECT_FALSE(my_map_copy == my_map);
EXPECT_FALSE(my_map == my_map_copy);
EXPECT_TRUE(my_map_copy != my_map);
EXPECT_TRUE(my_map != my_map_copy);
}
TEST(Btree, RangeCtorSanity) {
typedef btree_set<int, std::less<int>, std::allocator<int>, 256> test_set;
typedef btree_map<int, int, std::less<int>, std::allocator<int>, 256>
test_map;
typedef btree_multiset<int, std::less<int>, std::allocator<int>, 256>
test_mset;
typedef btree_multimap<int, int, std::less<int>, std::allocator<int>, 256>
test_mmap;
std::vector<int> ivec;
ivec.push_back(1);
std::map<int, int> imap;
imap.insert(std::make_pair(1, 2));
test_mset tmset(ivec.begin(), ivec.end());
test_mmap tmmap(imap.begin(), imap.end());
test_set tset(ivec.begin(), ivec.end());
test_map tmap(imap.begin(), imap.end());
EXPECT_EQ(1, tmset.size());
EXPECT_EQ(1, tmmap.size());
EXPECT_EQ(1, tset.size());
EXPECT_EQ(1, tmap.size());
}
} // namespace
} // namespace btree

940
lib/xdelta3/cpp-btree/btree_test.h Normal file
View file

@ -0,0 +1,940 @@
// Copyright 2013 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#ifndef UTIL_BTREE_BTREE_TEST_H__
#define UTIL_BTREE_BTREE_TEST_H__
#include <stdio.h>
#include <algorithm>
#include <functional>
#include <type_traits>
#include <iosfwd>
#include <map>
#include <set>
#include <sstream>
#include <string>
#include <utility>
#include <vector>
#include "gtest/gtest.h"
#include "gflags/gflags.h"
#include "btree_container.h"
DECLARE_int32(test_values);
DECLARE_int32(benchmark_values);
namespace std {
// Provide operator<< support for std::pair<T, U>.
template <typename T, typename U>
ostream& operator<<(ostream &os, const std::pair<T, U> &p) {
os << "(" << p.first << "," << p.second << ")";
return os;
}
// Provide pair equality testing that works as long as x.first is comparable to
// y.first and x.second is comparable to y.second. Needed in the test for
// comparing std::pair<T, U> to std::pair<const T, U>.
template <typename T, typename U, typename V, typename W>
bool operator==(const std::pair<T, U> &x, const std::pair<V, W> &y) {
return x.first == y.first && x.second == y.second;
}
// Partial specialization of remove_const that propagates the removal through
// std::pair.
template <typename T, typename U>
struct remove_const<pair<T, U> > {
typedef pair<typename remove_const<T>::type,
typename remove_const<U>::type> type;
};
} // namespace std
namespace btree {
// Select the first member of a pair.
template <class _Pair>
struct select1st : public std::unary_function<_Pair, typename _Pair::first_type> {
const typename _Pair::first_type& operator()(const _Pair& __x) const {
return __x.first;
}
};
// Utility class to provide an accessor for a key given a value. The default
// behavior is to treat the value as a pair and return the first element.
template <typename K, typename V>
struct KeyOfValue {
typedef select1st<V> type;
};
template <typename T>
struct identity {
inline const T& operator()(const T& t) const { return t; }
};
// Partial specialization of KeyOfValue class for when the key and value are
// the same type such as in set<> and btree_set<>.
template <typename K>
struct KeyOfValue<K, K> {
typedef identity<K> type;
};
// Counts the number of occurances of "c" in a buffer.
inline ptrdiff_t strcount(const char* buf_begin, const char* buf_end, char c) {
if (buf_begin == NULL)
return 0;
if (buf_end <= buf_begin)
return 0;
ptrdiff_t num = 0;
for (const char* bp = buf_begin; bp != buf_end; bp++) {
if (*bp == c)
num++;
}
return num;
}
// for when the string is not null-terminated.
inline ptrdiff_t strcount(const char* buf, size_t len, char c) {
return strcount(buf, buf + len, c);
}
inline ptrdiff_t strcount(const std::string& buf, char c) {
return strcount(buf.c_str(), buf.size(), c);
}
// The base class for a sorted associative container checker. TreeType is the
// container type to check and CheckerType is the container type to check
// against. TreeType is expected to be btree_{set,map,multiset,multimap} and
// CheckerType is expected to be {set,map,multiset,multimap}.
template <typename TreeType, typename CheckerType>
class base_checker {
typedef base_checker<TreeType, CheckerType> self_type;
public:
typedef typename TreeType::key_type key_type;
typedef typename TreeType::value_type value_type;
typedef typename TreeType::key_compare key_compare;
typedef typename TreeType::pointer pointer;
typedef typename TreeType::const_pointer const_pointer;
typedef typename TreeType::reference reference;
typedef typename TreeType::const_reference const_reference;
typedef typename TreeType::size_type size_type;
typedef typename TreeType::difference_type difference_type;
typedef typename TreeType::iterator iterator;
typedef typename TreeType::const_iterator const_iterator;
typedef typename TreeType::reverse_iterator reverse_iterator;
typedef typename TreeType::const_reverse_iterator const_reverse_iterator;
public:
// Default constructor.
base_checker()
: const_tree_(tree_) {
}
// Copy constructor.
base_checker(const self_type &x)
: tree_(x.tree_),
const_tree_(tree_),
checker_(x.checker_) {
}
// Range constructor.
template <typename InputIterator>
base_checker(InputIterator b, InputIterator e)
: tree_(b, e),
const_tree_(tree_),
checker_(b, e) {
}
// Iterator routines.
iterator begin() { return tree_.begin(); }
const_iterator begin() const { return tree_.begin(); }
iterator end() { return tree_.end(); }
const_iterator end() const { return tree_.end(); }
reverse_iterator rbegin() { return tree_.rbegin(); }
const_reverse_iterator rbegin() const { return tree_.rbegin(); }
reverse_iterator rend() { return tree_.rend(); }
const_reverse_iterator rend() const { return tree_.rend(); }
// Helper routines.
template <typename IterType, typename CheckerIterType>
IterType iter_check(
IterType tree_iter, CheckerIterType checker_iter) const {
if (tree_iter == tree_.end()) {
EXPECT_EQ(checker_iter, checker_.end());
} else {
EXPECT_EQ(*tree_iter, *checker_iter);
}
return tree_iter;
}
template <typename IterType, typename CheckerIterType>
IterType riter_check(
IterType tree_iter, CheckerIterType checker_iter) const {
if (tree_iter == tree_.rend()) {
EXPECT_EQ(checker_iter, checker_.rend());
} else {
EXPECT_EQ(*tree_iter, *checker_iter);
}
return tree_iter;
}
void value_check(const value_type &x) {
typename KeyOfValue<typename TreeType::key_type,
typename TreeType::value_type>::type key_of_value;
const key_type &key = key_of_value(x);
EXPECT_EQ(*find(key), x);
lower_bound(key);
upper_bound(key);
equal_range(key);
count(key);
}
void erase_check(const key_type &key) {
EXPECT_TRUE(tree_.find(key) == const_tree_.end());
EXPECT_TRUE(const_tree_.find(key) == tree_.end());
EXPECT_TRUE(tree_.equal_range(key).first ==
const_tree_.equal_range(key).second);
}
// Lookup routines.
iterator lower_bound(const key_type &key) {
return iter_check(tree_.lower_bound(key), checker_.lower_bound(key));
}
const_iterator lower_bound(const key_type &key) const {
return iter_check(tree_.lower_bound(key), checker_.lower_bound(key));
}
iterator upper_bound(const key_type &key) {
return iter_check(tree_.upper_bound(key), checker_.upper_bound(key));
}
const_iterator upper_bound(const key_type &key) const {
return iter_check(tree_.upper_bound(key), checker_.upper_bound(key));
}
std::pair<iterator,iterator> equal_range(const key_type &key) {
std::pair<typename CheckerType::iterator,
typename CheckerType::iterator> checker_res =
checker_.equal_range(key);
std::pair<iterator, iterator> tree_res = tree_.equal_range(key);
iter_check(tree_res.first, checker_res.first);
iter_check(tree_res.second, checker_res.second);
return tree_res;
}
std::pair<const_iterator,const_iterator> equal_range(const key_type &key) const {
std::pair<typename CheckerType::const_iterator,
typename CheckerType::const_iterator> checker_res =
checker_.equal_range(key);
std::pair<const_iterator, const_iterator> tree_res = tree_.equal_range(key);
iter_check(tree_res.first, checker_res.first);
iter_check(tree_res.second, checker_res.second);
return tree_res;
}
iterator find(const key_type &key) {
return iter_check(tree_.find(key), checker_.find(key));
}
const_iterator find(const key_type &key) const {
return iter_check(tree_.find(key), checker_.find(key));
}
size_type count(const key_type &key) const {
size_type res = checker_.count(key);
EXPECT_EQ(res, tree_.count(key));
return res;
}
// Assignment operator.
self_type& operator=(const self_type &x) {
tree_ = x.tree_;
checker_ = x.checker_;
return *this;
}
// Deletion routines.
int erase(const key_type &key) {
int size = tree_.size();
int res = checker_.erase(key);
EXPECT_EQ(res, tree_.count(key));
EXPECT_EQ(res, tree_.erase(key));
EXPECT_EQ(tree_.count(key), 0);
EXPECT_EQ(tree_.size(), size - res);
erase_check(key);
return res;
}
iterator erase(iterator iter) {
key_type key = iter.key();
int size = tree_.size();
int count = tree_.count(key);
typename CheckerType::iterator checker_iter = checker_.find(key);
for (iterator tmp(tree_.find(key)); tmp != iter; ++tmp) {
++checker_iter;
}
typename CheckerType::iterator checker_next = checker_iter;
++checker_next;
checker_.erase(checker_iter);
iter = tree_.erase(iter);
EXPECT_EQ(tree_.size(), checker_.size());
EXPECT_EQ(tree_.size(), size - 1);
EXPECT_EQ(tree_.count(key), count - 1);
if (count == 1) {
erase_check(key);
}
return iter_check(iter, checker_next);
}
void erase(iterator begin, iterator end) {
int size = tree_.size();
int count = distance(begin, end);
typename CheckerType::iterator checker_begin = checker_.find(begin.key());
for (iterator tmp(tree_.find(begin.key())); tmp != begin; ++tmp) {
++checker_begin;
}
typename CheckerType::iterator checker_end =
end == tree_.end() ? checker_.end() : checker_.find(end.key());
if (end != tree_.end()) {
for (iterator tmp(tree_.find(end.key())); tmp != end; ++tmp) {
++checker_end;
}
}
checker_.erase(checker_begin, checker_end);
tree_.erase(begin, end);
EXPECT_EQ(tree_.size(), checker_.size());
EXPECT_EQ(tree_.size(), size - count);
}
// Utility routines.
void clear() {
tree_.clear();
checker_.clear();
}
void swap(self_type &x) {
tree_.swap(x.tree_);
checker_.swap(x.checker_);
}
void verify() const {
tree_.verify();
EXPECT_EQ(tree_.size(), checker_.size());
// Move through the forward iterators using increment.
typename CheckerType::const_iterator
checker_iter(checker_.begin());
const_iterator tree_iter(tree_.begin());
for (; tree_iter != tree_.end();
++tree_iter, ++checker_iter) {
EXPECT_EQ(*tree_iter, *checker_iter);
}
// Move through the forward iterators using decrement.
for (int n = tree_.size() - 1; n >= 0; --n) {
iter_check(tree_iter, checker_iter);
--tree_iter;
--checker_iter;
}
EXPECT_TRUE(tree_iter == tree_.begin());
EXPECT_TRUE(checker_iter == checker_.begin());
// Move through the reverse iterators using increment.
typename CheckerType::const_reverse_iterator
checker_riter(checker_.rbegin());
const_reverse_iterator tree_riter(tree_.rbegin());
for (; tree_riter != tree_.rend();
++tree_riter, ++checker_riter) {
EXPECT_EQ(*tree_riter, *checker_riter);
}
// Move through the reverse iterators using decrement.
for (int n = tree_.size() - 1; n >= 0; --n) {
riter_check(tree_riter, checker_riter);
--tree_riter;
--checker_riter;
}
EXPECT_EQ(tree_riter, tree_.rbegin());
EXPECT_EQ(checker_riter, checker_.rbegin());
}
// Access to the underlying btree.
const TreeType& tree() const { return tree_; }
// Size routines.
size_type size() const {
EXPECT_EQ(tree_.size(), checker_.size());
return tree_.size();
}
size_type max_size() const { return tree_.max_size(); }
bool empty() const {
EXPECT_EQ(tree_.empty(), checker_.empty());
return tree_.empty();
}
size_type height() const { return tree_.height(); }
size_type internal_nodes() const { return tree_.internal_nodes(); }
size_type leaf_nodes() const { return tree_.leaf_nodes(); }
size_type nodes() const { return tree_.nodes(); }
size_type bytes_used() const { return tree_.bytes_used(); }
double fullness() const { return tree_.fullness(); }
double overhead() const { return tree_.overhead(); }
protected:
TreeType tree_;
const TreeType &const_tree_;
CheckerType checker_;
};
// A checker for unique sorted associative containers. TreeType is expected to
// be btree_{set,map} and CheckerType is expected to be {set,map}.
template <typename TreeType, typename CheckerType>
class unique_checker : public base_checker<TreeType, CheckerType> {
typedef base_checker<TreeType, CheckerType> super_type;
typedef unique_checker<TreeType, CheckerType> self_type;
public:
typedef typename super_type::iterator iterator;
typedef typename super_type::value_type value_type;
public:
// Default constructor.
unique_checker()
: super_type() {
}
// Copy constructor.
unique_checker(const self_type &x)
: super_type(x) {
}
// Range constructor.
template <class InputIterator>
unique_checker(InputIterator b, InputIterator e)
: super_type(b, e) {
}
// Insertion routines.
std::pair<iterator,bool> insert(const value_type &x) {
int size = this->tree_.size();
std::pair<typename CheckerType::iterator,bool> checker_res =
this->checker_.insert(x);
std::pair<iterator,bool> tree_res = this->tree_.insert(x);
EXPECT_EQ(*tree_res.first, *checker_res.first);
EXPECT_EQ(tree_res.second, checker_res.second);
EXPECT_EQ(this->tree_.size(), this->checker_.size());
EXPECT_EQ(this->tree_.size(), size + tree_res.second);
return tree_res;
}
iterator insert(iterator position, const value_type &x) {
int size = this->tree_.size();
std::pair<typename CheckerType::iterator,bool> checker_res =
this->checker_.insert(x);
iterator tree_res = this->tree_.insert(position, x);
EXPECT_EQ(*tree_res, *checker_res.first);
EXPECT_EQ(this->tree_.size(), this->checker_.size());
EXPECT_EQ(this->tree_.size(), size + checker_res.second);
return tree_res;
}
template <typename InputIterator>
void insert(InputIterator b, InputIterator e) {
for (; b != e; ++b) {
insert(*b);
}
}
};
// A checker for multiple sorted associative containers. TreeType is expected
// to be btree_{multiset,multimap} and CheckerType is expected to be
// {multiset,multimap}.
template <typename TreeType, typename CheckerType>
class multi_checker : public base_checker<TreeType, CheckerType> {
typedef base_checker<TreeType, CheckerType> super_type;
typedef multi_checker<TreeType, CheckerType> self_type;
public:
typedef typename super_type::iterator iterator;
typedef typename super_type::value_type value_type;
public:
// Default constructor.
multi_checker()
: super_type() {
}
// Copy constructor.
multi_checker(const self_type &x)
: super_type(x) {
}
// Range constructor.
template <class InputIterator>
multi_checker(InputIterator b, InputIterator e)
: super_type(b, e) {
}
// Insertion routines.
iterator insert(const value_type &x) {
int size = this->tree_.size();
typename CheckerType::iterator checker_res = this->checker_.insert(x);
iterator tree_res = this->tree_.insert(x);
EXPECT_EQ(*tree_res, *checker_res);
EXPECT_EQ(this->tree_.size(), this->checker_.size());
EXPECT_EQ(this->tree_.size(), size + 1);
return tree_res;
}
iterator insert(iterator position, const value_type &x) {
int size = this->tree_.size();
typename CheckerType::iterator checker_res = this->checker_.insert(x);
iterator tree_res = this->tree_.insert(position, x);
EXPECT_EQ(*tree_res, *checker_res);
EXPECT_EQ(this->tree_.size(), this->checker_.size());
EXPECT_EQ(this->tree_.size(), size + 1);
return tree_res;
}
template <typename InputIterator>
void insert(InputIterator b, InputIterator e) {
for (; b != e; ++b) {
insert(*b);
}
}
};
char* GenerateDigits(char buf[16], int val, int maxval) {
EXPECT_LE(val, maxval);
int p = 15;
buf[p--] = 0;
while (maxval > 0) {
buf[p--] = '0' + (val % 10);
val /= 10;
maxval /= 10;
}
return buf + p + 1;
}
template <typename K>
struct Generator {
int maxval;
Generator(int m)
: maxval(m) {
}
K operator()(int i) const {
EXPECT_LE(i, maxval);
return i;
}
};
template <>
struct Generator<std::string> {
int maxval;
Generator(int m)
: maxval(m) {
}
std::string operator()(int i) const {
char buf[16];
return GenerateDigits(buf, i, maxval);
}
};
template <typename T, typename U>
struct Generator<std::pair<T, U> > {
Generator<typename std::remove_const<T>::type> tgen;
Generator<typename std::remove_const<U>::type> ugen;
Generator(int m)
: tgen(m),
ugen(m) {
}
std::pair<T, U> operator()(int i) const {
return std::make_pair(tgen(i), ugen(i));
}
};
// Generate values for our tests and benchmarks. Value range is [0, maxval].
const std::vector<int>& GenerateNumbers(int n, int maxval) {
static std::vector<int> values;
static std::set<int> unique_values;
if (values.size() < n) {
for (int i = values.size(); i < n; i++) {
int value;
do {
value = rand() % (maxval + 1);
} while (unique_values.find(value) != unique_values.end());
values.push_back(value);
unique_values.insert(value);
}
}
return values;
}
// Generates values in the range
// [0, 4 * min(FLAGS_benchmark_values, FLAGS_test_values)]
template <typename V>
std::vector<V> GenerateValues(int n) {
int two_times_max = 2 * std::max(FLAGS_benchmark_values, FLAGS_test_values);
int four_times_max = 2 * two_times_max;
EXPECT_LE(n, two_times_max);
const std::vector<int> &nums = GenerateNumbers(n, four_times_max);
Generator<V> gen(four_times_max);
std::vector<V> vec;
for (int i = 0; i < n; i++) {
vec.push_back(gen(nums[i]));
}
return vec;
}
template <typename T, typename V>
void DoTest(const char *name, T *b, const std::vector<V> &values) {
typename KeyOfValue<typename T::key_type, V>::type key_of_value;
T &mutable_b = *b;
const T &const_b = *b;
// Test insert.
for (int i = 0; i < values.size(); ++i) {
mutable_b.insert(values[i]);
mutable_b.value_check(values[i]);
}
assert(mutable_b.size() == values.size());
const_b.verify();
printf(" %s fullness=%0.2f overhead=%0.2f bytes-per-value=%0.2f\n",
name, const_b.fullness(), const_b.overhead(),
double(const_b.bytes_used()) / const_b.size());
// Test copy constructor.
T b_copy(const_b);
EXPECT_EQ(b_copy.size(), const_b.size());
EXPECT_LE(b_copy.height(), const_b.height());
EXPECT_LE(b_copy.internal_nodes(), const_b.internal_nodes());
EXPECT_LE(b_copy.leaf_nodes(), const_b.leaf_nodes());
for (int i = 0; i < values.size(); ++i) {
EXPECT_EQ(*b_copy.find(key_of_value(values[i])), values[i]);
}
// Test range constructor.
T b_range(const_b.begin(), const_b.end());
EXPECT_EQ(b_range.size(), const_b.size());
EXPECT_LE(b_range.height(), const_b.height());
EXPECT_LE(b_range.internal_nodes(), const_b.internal_nodes());
EXPECT_LE(b_range.leaf_nodes(), const_b.leaf_nodes());
for (int i = 0; i < values.size(); ++i) {
EXPECT_EQ(*b_range.find(key_of_value(values[i])), values[i]);
}
// Test range insertion for values that already exist.
b_range.insert(b_copy.begin(), b_copy.end());
b_range.verify();
// Test range insertion for new values.
b_range.clear();
b_range.insert(b_copy.begin(), b_copy.end());
EXPECT_EQ(b_range.size(), b_copy.size());
EXPECT_EQ(b_range.height(), b_copy.height());
EXPECT_EQ(b_range.internal_nodes(), b_copy.internal_nodes());
EXPECT_EQ(b_range.leaf_nodes(), b_copy.leaf_nodes());
for (int i = 0; i < values.size(); ++i) {
EXPECT_EQ(*b_range.find(key_of_value(values[i])), values[i]);
}
// Test assignment to self. Nothing should change.
b_range.operator=(b_range);
EXPECT_EQ(b_range.size(), b_copy.size());
EXPECT_EQ(b_range.height(), b_copy.height());
EXPECT_EQ(b_range.internal_nodes(), b_copy.internal_nodes());
EXPECT_EQ(b_range.leaf_nodes(), b_copy.leaf_nodes());
// Test assignment of new values.
b_range.clear();
b_range = b_copy;
EXPECT_EQ(b_range.size(), b_copy.size());
EXPECT_EQ(b_range.height(), b_copy.height());
EXPECT_EQ(b_range.internal_nodes(), b_copy.internal_nodes());
EXPECT_EQ(b_range.leaf_nodes(), b_copy.leaf_nodes());
// Test swap.
b_range.clear();
b_range.swap(b_copy);
EXPECT_EQ(b_copy.size(), 0);
EXPECT_EQ(b_range.size(), const_b.size());
for (int i = 0; i < values.size(); ++i) {
EXPECT_EQ(*b_range.find(key_of_value(values[i])), values[i]);
}
b_range.swap(b_copy);
// Test erase via values.
for (int i = 0; i < values.size(); ++i) {
mutable_b.erase(key_of_value(values[i]));
// Erasing a non-existent key should have no effect.
EXPECT_EQ(mutable_b.erase(key_of_value(values[i])), 0);
}
const_b.verify();
EXPECT_EQ(const_b.internal_nodes(), 0);
EXPECT_EQ(const_b.leaf_nodes(), 0);
EXPECT_EQ(const_b.size(), 0);
// Test erase via iterators.
mutable_b = b_copy;
for (int i = 0; i < values.size(); ++i) {
mutable_b.erase(mutable_b.find(key_of_value(values[i])));
}
const_b.verify();
EXPECT_EQ(const_b.internal_nodes(), 0);
EXPECT_EQ(const_b.leaf_nodes(), 0);
EXPECT_EQ(const_b.size(), 0);
// Test insert with hint.
for (int i = 0; i < values.size(); i++) {
mutable_b.insert(mutable_b.upper_bound(key_of_value(values[i])), values[i]);
}
const_b.verify();
// Test dumping of the btree to an ostream. There should be 1 line for each
// value.
std::stringstream strm;
strm << mutable_b.tree();
EXPECT_EQ(mutable_b.size(), strcount(strm.str(), '\n'));
// Test range erase.
mutable_b.erase(mutable_b.begin(), mutable_b.end());
EXPECT_EQ(mutable_b.size(), 0);
const_b.verify();
// First half.
mutable_b = b_copy;
typename T::iterator mutable_iter_end = mutable_b.begin();
for (int i = 0; i < values.size() / 2; ++i) ++mutable_iter_end;
mutable_b.erase(mutable_b.begin(), mutable_iter_end);
EXPECT_EQ(mutable_b.size(), values.size() - values.size() / 2);
const_b.verify();
// Second half.
mutable_b = b_copy;
typename T::iterator mutable_iter_begin = mutable_b.begin();
for (int i = 0; i < values.size() / 2; ++i) ++mutable_iter_begin;
mutable_b.erase(mutable_iter_begin, mutable_b.end());
EXPECT_EQ(mutable_b.size(), values.size() / 2);
const_b.verify();
// Second quarter.
mutable_b = b_copy;
mutable_iter_begin = mutable_b.begin();
for (int i = 0; i < values.size() / 4; ++i) ++mutable_iter_begin;
mutable_iter_end = mutable_iter_begin;
for (int i = 0; i < values.size() / 4; ++i) ++mutable_iter_end;
mutable_b.erase(mutable_iter_begin, mutable_iter_end);
EXPECT_EQ(mutable_b.size(), values.size() - values.size() / 4);
const_b.verify();
mutable_b.clear();
}
template <typename T>
void ConstTest() {
typedef typename T::value_type value_type;
typename KeyOfValue<typename T::key_type, value_type>::type key_of_value;
T mutable_b;
const T &const_b = mutable_b;
// Insert a single value into the container and test looking it up.
value_type value = Generator<value_type>(2)(2);
mutable_b.insert(value);
EXPECT_TRUE(mutable_b.find(key_of_value(value)) != const_b.end());
EXPECT_TRUE(const_b.find(key_of_value(value)) != mutable_b.end());
EXPECT_EQ(*const_b.lower_bound(key_of_value(value)), value);
EXPECT_TRUE(const_b.upper_bound(key_of_value(value)) == const_b.end());
EXPECT_EQ(*const_b.equal_range(key_of_value(value)).first, value);
// We can only create a non-const iterator from a non-const container.
typename T::iterator mutable_iter(mutable_b.begin());
EXPECT_TRUE(mutable_iter == const_b.begin());
EXPECT_TRUE(mutable_iter != const_b.end());
EXPECT_TRUE(const_b.begin() == mutable_iter);
EXPECT_TRUE(const_b.end() != mutable_iter);
typename T::reverse_iterator mutable_riter(mutable_b.rbegin());
EXPECT_TRUE(mutable_riter == const_b.rbegin());
EXPECT_TRUE(mutable_riter != const_b.rend());
EXPECT_TRUE(const_b.rbegin() == mutable_riter);
EXPECT_TRUE(const_b.rend() != mutable_riter);
// We can create a const iterator from a non-const iterator.
typename T::const_iterator const_iter(mutable_iter);
EXPECT_TRUE(const_iter == mutable_b.begin());
EXPECT_TRUE(const_iter != mutable_b.end());
EXPECT_TRUE(mutable_b.begin() == const_iter);
EXPECT_TRUE(mutable_b.end() != const_iter);
typename T::const_reverse_iterator const_riter(mutable_riter);
EXPECT_EQ(const_riter, mutable_b.rbegin());
EXPECT_TRUE(const_riter != mutable_b.rend());
EXPECT_EQ(mutable_b.rbegin(), const_riter);
EXPECT_TRUE(mutable_b.rend() != const_riter);
// Make sure various methods can be invoked on a const container.
const_b.verify();
EXPECT_FALSE(const_b.empty());
EXPECT_EQ(const_b.size(), 1);
EXPECT_GT(const_b.max_size(), 0);
EXPECT_EQ(const_b.height(), 1);
EXPECT_EQ(const_b.count(key_of_value(value)), 1);
EXPECT_EQ(const_b.internal_nodes(), 0);
EXPECT_EQ(const_b.leaf_nodes(), 1);
EXPECT_EQ(const_b.nodes(), 1);
EXPECT_GT(const_b.bytes_used(), 0);
EXPECT_GT(const_b.fullness(), 0);
EXPECT_GT(const_b.overhead(), 0);
}
template <typename T, typename C>
void BtreeTest() {
ConstTest<T>();
typedef typename std::remove_const<typename T::value_type>::type V;
std::vector<V> random_values = GenerateValues<V>(FLAGS_test_values);
unique_checker<T, C> container;
// Test key insertion/deletion in sorted order.
std::vector<V> sorted_values(random_values);
sort(sorted_values.begin(), sorted_values.end());
DoTest("sorted: ", &container, sorted_values);
// Test key insertion/deletion in reverse sorted order.
reverse(sorted_values.begin(), sorted_values.end());
DoTest("rsorted: ", &container, sorted_values);
// Test key insertion/deletion in random order.
DoTest("random: ", &container, random_values);
}
template <typename T, typename C>
void BtreeMultiTest() {
ConstTest<T>();
typedef typename std::remove_const<typename T::value_type>::type V;
const std::vector<V>& random_values = GenerateValues<V>(FLAGS_test_values);
multi_checker<T, C> container;
// Test keys in sorted order.
std::vector<V> sorted_values(random_values);
sort(sorted_values.begin(), sorted_values.end());
DoTest("sorted: ", &container, sorted_values);
// Test keys in reverse sorted order.
reverse(sorted_values.begin(), sorted_values.end());
DoTest("rsorted: ", &container, sorted_values);
// Test keys in random order.
DoTest("random: ", &container, random_values);
// Test keys in random order w/ duplicates.
std::vector<V> duplicate_values(random_values);
duplicate_values.insert(
duplicate_values.end(), random_values.begin(), random_values.end());
DoTest("duplicates:", &container, duplicate_values);
// Test all identical keys.
std::vector<V> identical_values(100);
fill(identical_values.begin(), identical_values.end(), Generator<V>(2)(2));
DoTest("identical: ", &container, identical_values);
}
template <typename T, typename Alloc = std::allocator<T> >
class TestAllocator : public Alloc {
public:
typedef typename Alloc::pointer pointer;
typedef typename Alloc::size_type size_type;
TestAllocator() : bytes_used_(NULL) { }
TestAllocator(int64_t *bytes_used) : bytes_used_(bytes_used) { }
// Constructor used for rebinding
template <class U>
TestAllocator(const TestAllocator<U>& x)
: Alloc(x),
bytes_used_(x.bytes_used()) {
}
pointer allocate(size_type n, std::allocator<void>::const_pointer hint = 0) {
EXPECT_TRUE(bytes_used_ != NULL);
*bytes_used_ += n * sizeof(T);
return Alloc::allocate(n, hint);
}
void deallocate(pointer p, size_type n) {
Alloc::deallocate(p, n);
EXPECT_TRUE(bytes_used_ != NULL);
*bytes_used_ -= n * sizeof(T);
}
// Rebind allows an allocator<T> to be used for a different type
template <class U> struct rebind {
typedef TestAllocator<U, typename Alloc::template rebind<U>::other> other;
};
int64_t* bytes_used() const { return bytes_used_; }
private:
int64_t *bytes_used_;
};
template <typename T>
void BtreeAllocatorTest() {
typedef typename T::value_type value_type;
int64_t alloc1 = 0;
int64_t alloc2 = 0;
T b1(typename T::key_compare(), &alloc1);
T b2(typename T::key_compare(), &alloc2);
// This should swap the allocators!
swap(b1, b2);
for (int i = 0; i < 1000; i++) {
b1.insert(Generator<value_type>(1000)(i));
}
// We should have allocated out of alloc2!
EXPECT_LE(b1.bytes_used(), alloc2 + sizeof(b1));
EXPECT_GT(alloc2, alloc1);
}
template <typename T>
void BtreeMapTest() {
typedef typename T::value_type value_type;
typedef typename T::mapped_type mapped_type;
mapped_type m = Generator<mapped_type>(0)(0);
(void) m;
T b;
// Verify we can insert using operator[].
for (int i = 0; i < 1000; i++) {
value_type v = Generator<value_type>(1000)(i);
b[v.first] = v.second;
}
EXPECT_EQ(b.size(), 1000);
// Test whether we can use the "->" operator on iterators and
// reverse_iterators. This stresses the btree_map_params::pair_pointer
// mechanism.
EXPECT_EQ(b.begin()->first, Generator<value_type>(1000)(0).first);
EXPECT_EQ(b.begin()->second, Generator<value_type>(1000)(0).second);
EXPECT_EQ(b.rbegin()->first, Generator<value_type>(1000)(999).first);
EXPECT_EQ(b.rbegin()->second, Generator<value_type>(1000)(999).second);
}
template <typename T>
void BtreeMultiMapTest() {
typedef typename T::mapped_type mapped_type;
mapped_type m = Generator<mapped_type>(0)(0);
(void) m;
}
} // namespace btree
#endif // UTIL_BTREE_BTREE_TEST_H__

20
lib/xdelta3/cpp-btree/btree_test_flags.cc Normal file
View file

@ -0,0 +1,20 @@
// Copyright 2013 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include "gflags/gflags.h"
DEFINE_int32(test_values, 10000,
"The number of values to use for tests.");
DEFINE_int32(benchmark_values, 1000000,
"The number of values to use for benchmarks.");

395
lib/xdelta3/cpp-btree/safe_btree.h Normal file
View file

@ -0,0 +1,395 @@
// Copyright 2013 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
// A safe_btree<> wraps around a btree<> and removes the caveat that insertion
// and deletion invalidate iterators. A safe_btree<> maintains a generation
// number that is incremented on every mutation. A safe_btree<>::iterator keeps
// a pointer to the safe_btree<> it came from, the generation of the tree when
// it was last validated and the key the underlying btree<>::iterator points
// to. If an iterator is accessed and its generation differs from the tree
// generation it is revalidated.
//
// References and pointers returned by safe_btree iterators are not safe.
//
// See the incorrect usage examples mentioned in safe_btree_set.h and
// safe_btree_map.h.
#ifndef UTIL_BTREE_SAFE_BTREE_H__
#define UTIL_BTREE_SAFE_BTREE_H__
#include <stddef.h>
#include <iosfwd>
#include <utility>
#include "btree.h"
namespace btree {
template <typename Tree, typename Iterator>
class safe_btree_iterator {
public:
typedef typename Iterator::key_type key_type;
typedef typename Iterator::value_type value_type;
typedef typename Iterator::size_type size_type;
typedef typename Iterator::difference_type difference_type;
typedef typename Iterator::pointer pointer;
typedef typename Iterator::reference reference;
typedef typename Iterator::const_pointer const_pointer;
typedef typename Iterator::const_reference const_reference;
typedef typename Iterator::iterator_category iterator_category;
typedef typename Tree::iterator iterator;
typedef typename Tree::const_iterator const_iterator;
typedef safe_btree_iterator<Tree, Iterator> self_type;
void update() const {
if (iter_ != tree_->internal_btree()->end()) {
// A positive generation indicates a valid key.
generation_ = tree_->generation();
key_ = iter_.key();
} else {
// Use a negative generation to indicate iter_ points to end().
generation_ = -tree_->generation();
}
}
public:
safe_btree_iterator()
: generation_(0),
key_(),
iter_(),
tree_(NULL) {
}
safe_btree_iterator(const iterator &x)
: generation_(x.generation()),
key_(x.key()),
iter_(x.iter()),
tree_(x.tree()) {
}
safe_btree_iterator(Tree *tree, const Iterator &iter)
: generation_(),
key_(),
iter_(iter),
tree_(tree) {
update();
}
Tree* tree() const { return tree_; }
int64_t generation() const { return generation_; }
Iterator* mutable_iter() const {
if (generation_ != tree_->generation()) {
if (generation_ > 0) {
// This does the wrong thing for a multi{set,map}. If my iter was
// pointing to the 2nd of 2 values with the same key, then this will
// reset it to point to the first. This is why we don't provide a
// safe_btree_multi{set,map}.
iter_ = tree_->internal_btree()->lower_bound(key_);
update();
} else if (-generation_ != tree_->generation()) {
iter_ = tree_->internal_btree()->end();
generation_ = -tree_->generation();
}
}
return &iter_;
}
const Iterator& iter() const {
return *mutable_iter();
}
// Equality/inequality operators.
bool operator==(const const_iterator &x) const {
return iter() == x.iter();
}
bool operator!=(const const_iterator &x) const {
return iter() != x.iter();
}
// Accessors for the key/value the iterator is pointing at.
const key_type& key() const {
return key_;
}
// This reference value is potentially invalidated by any non-const
// method on the tree; it is NOT safe.
reference operator*() const {
assert(generation_ > 0);
return iter().operator*();
}
// This pointer value is potentially invalidated by any non-const
// method on the tree; it is NOT safe.
pointer operator->() const {
assert(generation_ > 0);
return iter().operator->();
}
// Increment/decrement operators.
self_type& operator++() {
++(*mutable_iter());
update();
return *this;
}
self_type& operator--() {
--(*mutable_iter());
update();
return *this;
}
self_type operator++(int) {
self_type tmp = *this;
++*this;
return tmp;
}
self_type operator--(int) {
self_type tmp = *this;
--*this;
return tmp;
}
private:
// The generation of the tree when "iter" was updated.
mutable int64_t generation_;
// The key the iterator points to.
mutable key_type key_;
// The underlying iterator.
mutable Iterator iter_;
// The tree the iterator is associated with.
Tree *tree_;
};
template <typename Params>
class safe_btree {
typedef safe_btree<Params> self_type;
typedef btree<Params> btree_type;
typedef typename btree_type::iterator tree_iterator;
typedef typename btree_type::const_iterator tree_const_iterator;
public:
typedef typename btree_type::params_type params_type;
typedef typename btree_type::key_type key_type;
typedef typename btree_type::data_type data_type;
typedef typename btree_type::mapped_type mapped_type;
typedef typename btree_type::value_type value_type;
typedef typename btree_type::key_compare key_compare;
typedef typename btree_type::allocator_type allocator_type;
typedef typename btree_type::pointer pointer;
typedef typename btree_type::const_pointer const_pointer;
typedef typename btree_type::reference reference;
typedef typename btree_type::const_reference const_reference;
typedef typename btree_type::size_type size_type;
typedef typename btree_type::difference_type difference_type;
typedef safe_btree_iterator<self_type, tree_iterator> iterator;
typedef safe_btree_iterator<
const self_type, tree_const_iterator> const_iterator;
typedef std::reverse_iterator<const_iterator> const_reverse_iterator;
typedef std::reverse_iterator<iterator> reverse_iterator;
public:
// Default constructor.
safe_btree(const key_compare &comp, const allocator_type &alloc)
: tree_(comp, alloc),
generation_(1) {
}
// Copy constructor.
safe_btree(const self_type &x)
: tree_(x.tree_),
generation_(1) {
}
iterator begin() {
return iterator(this, tree_.begin());
}
const_iterator begin() const {
return const_iterator(this, tree_.begin());
}
iterator end() {
return iterator(this, tree_.end());
}
const_iterator end() const {
return const_iterator(this, tree_.end());
}
reverse_iterator rbegin() {
return reverse_iterator(end());
}
const_reverse_iterator rbegin() const {
return const_reverse_iterator(end());
}
reverse_iterator rend() {
return reverse_iterator(begin());
}
const_reverse_iterator rend() const {
return const_reverse_iterator(begin());
}
// Lookup routines.
iterator lower_bound(const key_type &key) {
return iterator(this, tree_.lower_bound(key));
}
const_iterator lower_bound(const key_type &key) const {
return const_iterator(this, tree_.lower_bound(key));
}
iterator upper_bound(const key_type &key) {
return iterator(this, tree_.upper_bound(key));
}
const_iterator upper_bound(const key_type &key) const {
return const_iterator(this, tree_.upper_bound(key));
}
std::pair<iterator, iterator> equal_range(const key_type &key) {
std::pair<tree_iterator, tree_iterator> p = tree_.equal_range(key);
return std::make_pair(iterator(this, p.first),
iterator(this, p.second));
}
std::pair<const_iterator, const_iterator> equal_range(const key_type &key) const {
std::pair<tree_const_iterator, tree_const_iterator> p = tree_.equal_range(key);
return std::make_pair(const_iterator(this, p.first),
const_iterator(this, p.second));
}
iterator find_unique(const key_type &key) {
return iterator(this, tree_.find_unique(key));
}
const_iterator find_unique(const key_type &key) const {
return const_iterator(this, tree_.find_unique(key));
}
iterator find_multi(const key_type &key) {
return iterator(this, tree_.find_multi(key));
}
const_iterator find_multi(const key_type &key) const {
return const_iterator(this, tree_.find_multi(key));
}
size_type count_unique(const key_type &key) const {
return tree_.count_unique(key);
}
size_type count_multi(const key_type &key) const {
return tree_.count_multi(key);
}
// Insertion routines.
template <typename ValuePointer>
std::pair<iterator, bool> insert_unique(const key_type &key, ValuePointer value) {
std::pair<tree_iterator, bool> p = tree_.insert_unique(key, value);
generation_ += p.second;
return std::make_pair(iterator(this, p.first), p.second);
}
std::pair<iterator, bool> insert_unique(const value_type &v) {
std::pair<tree_iterator, bool> p = tree_.insert_unique(v);
generation_ += p.second;
return std::make_pair(iterator(this, p.first), p.second);
}
iterator insert_unique(iterator position, const value_type &v) {
tree_iterator tree_pos = position.iter();
++generation_;
return iterator(this, tree_.insert_unique(tree_pos, v));
}
template <typename InputIterator>
void insert_unique(InputIterator b, InputIterator e) {
for (; b != e; ++b) {
insert_unique(*b);
}
}
iterator insert_multi(const value_type &v) {
++generation_;
return iterator(this, tree_.insert_multi(v));
}
iterator insert_multi(iterator position, const value_type &v) {
tree_iterator tree_pos = position.iter();
++generation_;
return iterator(this, tree_.insert_multi(tree_pos, v));
}
template <typename InputIterator>
void insert_multi(InputIterator b, InputIterator e) {
for (; b != e; ++b) {
insert_multi(*b);
}
}
self_type& operator=(const self_type &x) {
if (&x == this) {
// Don't copy onto ourselves.
return *this;
}
++generation_;
tree_ = x.tree_;
return *this;
}
// Deletion routines.
void erase(const iterator &begin, const iterator &end) {
tree_.erase(begin.iter(), end.iter());
++generation_;
}
// Erase the specified iterator from the btree. The iterator must be valid
// (i.e. not equal to end()). Return an iterator pointing to the node after
// the one that was erased (or end() if none exists).
iterator erase(iterator iter) {
tree_iterator res = tree_.erase(iter.iter());
++generation_;
return iterator(this, res);
}
int erase_unique(const key_type &key) {
int res = tree_.erase_unique(key);
generation_ += res;
return res;
}
int erase_multi(const key_type &key) {
int res = tree_.erase_multi(key);
generation_ += res;
return res;
}
// Access to the underlying btree.
btree_type* internal_btree() { return &tree_; }
const btree_type* internal_btree() const { return &tree_; }
// Utility routines.
void clear() {
++generation_;
tree_.clear();
}
void swap(self_type &x) {
++generation_;
++x.generation_;
tree_.swap(x.tree_);
}
void dump(std::ostream &os) const {
tree_.dump(os);
}
void verify() const {
tree_.verify();
}
int64_t generation() const {
return generation_;
}
key_compare key_comp() const { return tree_.key_comp(); }
// Size routines.
size_type size() const { return tree_.size(); }
size_type max_size() const { return tree_.max_size(); }
bool empty() const { return tree_.empty(); }
size_type height() const { return tree_.height(); }
size_type internal_nodes() const { return tree_.internal_nodes(); }
size_type leaf_nodes() const { return tree_.leaf_nodes(); }
size_type nodes() const { return tree_.nodes(); }
size_type bytes_used() const { return tree_.bytes_used(); }
static double average_bytes_per_value() {
return btree_type::average_bytes_per_value();
}
double fullness() const { return tree_.fullness(); }
double overhead() const { return tree_.overhead(); }
private:
btree_type tree_;
int64_t generation_;
};
} // namespace btree
#endif // UTIL_BTREE_SAFE_BTREE_H__

89
lib/xdelta3/cpp-btree/safe_btree_map.h Normal file
View file

@ -0,0 +1,89 @@
// Copyright 2013 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
// The safe_btree_map<> is like btree_map<> except that it removes the caveat
// about insertion and deletion invalidating existing iterators at a small cost
// in making iterators larger and slower.
//
// Revalidation occurs whenever an iterator is accessed. References
// and pointers returned by safe_btree_map<> iterators are not stable,
// they are potentially invalidated by any non-const method on the map.
//
// BEGIN INCORRECT EXAMPLE
// for (auto i = safe_map->begin(); i != safe_map->end(); ++i) {
// const T *value = &i->second; // DO NOT DO THIS
// [code that modifies safe_map and uses value];
// }
// END INCORRECT EXAMPLE
#ifndef UTIL_BTREE_SAFE_BTREE_MAP_H__
#define UTIL_BTREE_SAFE_BTREE_MAP_H__
#include <functional>
#include <memory>
#include <utility>
#include "btree_container.h"
#include "btree_map.h"
#include "safe_btree.h"
namespace btree {
// The safe_btree_map class is needed mainly for its constructors.
template <typename Key, typename Value,
typename Compare = std::less<Key>,
typename Alloc = std::allocator<std::pair<const Key, Value> >,
int TargetNodeSize = 256>
class safe_btree_map : public btree_map_container<
safe_btree<btree_map_params<Key, Value, Compare, Alloc, TargetNodeSize> > > {
typedef safe_btree_map<Key, Value, Compare, Alloc, TargetNodeSize> self_type;
typedef btree_map_params<
Key, Value, Compare, Alloc, TargetNodeSize> params_type;
typedef safe_btree<params_type> btree_type;
typedef btree_map_container<btree_type> super_type;
public:
typedef typename btree_type::key_compare key_compare;
typedef typename btree_type::allocator_type allocator_type;
public:
// Default constructor.
safe_btree_map(const key_compare &comp = key_compare(),
const allocator_type &alloc = allocator_type())
: super_type(comp, alloc) {
}
// Copy constructor.
safe_btree_map(const self_type &x)
: super_type(x) {
}
// Range constructor.
template <class InputIterator>
safe_btree_map(InputIterator b, InputIterator e,
const key_compare &comp = key_compare(),
const allocator_type &alloc = allocator_type())
: super_type(b, e, comp, alloc) {
}
};
template <typename K, typename V, typename C, typename A, int N>
inline void swap(safe_btree_map<K, V, C, A, N> &x,
safe_btree_map<K, V, C, A, N> &y) {
x.swap(y);
}
} // namespace btree
#endif // UTIL_BTREE_SAFE_BTREE_MAP_H__

88
lib/xdelta3/cpp-btree/safe_btree_set.h Normal file
View file

@ -0,0 +1,88 @@
// Copyright 2013 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
// The safe_btree_set<> is like btree_set<> except that it removes the caveat
// about insertion and deletion invalidating existing iterators at a small cost
// in making iterators larger and slower.
//
// Revalidation occurs whenever an iterator is accessed. References
// and pointers returned by safe_btree_map<> iterators are not stable,
// they are potentially invalidated by any non-const method on the set.
//
// BEGIN INCORRECT EXAMPLE
// for (auto i = safe_set->begin(); i != safe_set->end(); ++i) {
// const T &value = *i; // DO NOT DO THIS
// [code that modifies safe_set and uses value];
// }
// END INCORRECT EXAMPLE
#ifndef UTIL_BTREE_SAFE_BTREE_SET_H__
#define UTIL_BTREE_SAFE_BTREE_SET_H__
#include <functional>
#include <memory>
#include "btree_container.h"
#include "btree_set.h"
#include "safe_btree.h"
namespace btree {
// The safe_btree_set class is needed mainly for its constructors.
template <typename Key,
typename Compare = std::less<Key>,
typename Alloc = std::allocator<Key>,
int TargetNodeSize = 256>
class safe_btree_set : public btree_unique_container<
safe_btree<btree_set_params<Key, Compare, Alloc, TargetNodeSize> > > {
typedef safe_btree_set<Key, Compare, Alloc, TargetNodeSize> self_type;
typedef btree_set_params<Key, Compare, Alloc, TargetNodeSize> params_type;
typedef safe_btree<params_type> btree_type;
typedef btree_unique_container<btree_type> super_type;
public:
typedef typename btree_type::key_compare key_compare;
typedef typename btree_type::allocator_type allocator_type;
public:
// Default constructor.
safe_btree_set(const key_compare &comp = key_compare(),
const allocator_type &alloc = allocator_type())
: super_type(comp, alloc) {
}
// Copy constructor.
safe_btree_set(const self_type &x)
: super_type(x) {
}
// Range constructor.
template <class InputIterator>
safe_btree_set(InputIterator b, InputIterator e,
const key_compare &comp = key_compare(),
const allocator_type &alloc = allocator_type())
: super_type(b, e, comp, alloc) {
}
};
template <typename K, typename C, typename A, int N>
inline void swap(safe_btree_set<K, C, A, N> &x,
safe_btree_set<K, C, A, N> &y) {
x.swap(y);
}
} // namespace btree
#endif // UTIL_BTREE_SAFE_BTREE_SET_H__

116
lib/xdelta3/cpp-btree/safe_btree_test.cc Normal file
View file

@ -0,0 +1,116 @@
// Copyright 2013 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// TODO(pmattis): Add some tests that iterators are not invalidated by
// insertion and deletion.
#include <functional>
#include <map>
#include <set>
#include <string>
#include <utility>
#include "gtest/gtest.h"
#include "btree_test.h"
#include "safe_btree_map.h"
#include "safe_btree_set.h"
class UnsafeArena;
namespace btree {
namespace {
template <typename K, int N>
void SetTest() {
typedef TestAllocator<K> TestAlloc;
BtreeTest<safe_btree_set<K, std::less<K>, std::allocator<K>, N>, std::set<K> >();
BtreeAllocatorTest<safe_btree_set<K, std::less<K>, TestAlloc, N> >();
}
template <typename K, int N>
void MapTest() {
typedef TestAllocator<K> TestAlloc;
BtreeTest<safe_btree_map<K, K, std::less<K>, std::allocator<K>, N>, std::map<K, K> >();
BtreeAllocatorTest<safe_btree_map<K, K, std::less<K>, TestAlloc, N> >();
BtreeMapTest<safe_btree_map<K, K, std::less<K>, std::allocator<K>, N> >();
}
TEST(SafeBtree, set_int32_32) { SetTest<int32_t, 32>(); }
TEST(SafeBtree, set_int32_64) { SetTest<int32_t, 64>(); }
TEST(SafeBtree, set_int32_128) { SetTest<int32_t, 128>(); }
TEST(SafeBtree, set_int32_256) { SetTest<int32_t, 256>(); }
TEST(SafeBtree, set_int64_256) { SetTest<int64_t, 256>(); }
TEST(SafeBtree, set_string_256) { SetTest<std::string, 256>(); }
TEST(SafeBtree, set_pair_256) { SetTest<std::pair<int, int>, 256>(); }
TEST(SafeBtree, map_int32_256) { MapTest<int32_t, 256>(); }
TEST(SafeBtree, map_int64_256) { MapTest<int64_t, 256>(); }
TEST(SafeBtree, map_string_256) { MapTest<std::string, 256>(); }
TEST(SafeBtree, map_pair_256) { MapTest<std::pair<int, int>, 256>(); }
TEST(SafeBtree, Comparison) {
const int kSetSize = 1201;
safe_btree_set<int64_t> my_set;
for (int i = 0; i < kSetSize; ++i) {
my_set.insert(i);
}
safe_btree_set<int64_t> my_set_copy(my_set);
EXPECT_TRUE(my_set_copy == my_set);
EXPECT_TRUE(my_set == my_set_copy);
EXPECT_FALSE(my_set_copy != my_set);
EXPECT_FALSE(my_set != my_set_copy);
my_set.insert(kSetSize);
EXPECT_FALSE(my_set_copy == my_set);
EXPECT_FALSE(my_set == my_set_copy);
EXPECT_TRUE(my_set_copy != my_set);
EXPECT_TRUE(my_set != my_set_copy);
my_set.erase(kSetSize - 1);
EXPECT_FALSE(my_set_copy == my_set);
EXPECT_FALSE(my_set == my_set_copy);
EXPECT_TRUE(my_set_copy != my_set);
EXPECT_TRUE(my_set != my_set_copy);
safe_btree_map<std::string, int64_t> my_map;
for (int i = 0; i < kSetSize; ++i) {
my_map[std::string(i, 'a')] = i;
}
safe_btree_map<std::string, int64_t> my_map_copy(my_map);
EXPECT_TRUE(my_map_copy == my_map);
EXPECT_TRUE(my_map == my_map_copy);
EXPECT_FALSE(my_map_copy != my_map);
EXPECT_FALSE(my_map != my_map_copy);
++my_map_copy[std::string(7, 'a')];
EXPECT_FALSE(my_map_copy == my_map);
EXPECT_FALSE(my_map == my_map_copy);
EXPECT_TRUE(my_map_copy != my_map);
EXPECT_TRUE(my_map != my_map_copy);
my_map_copy = my_map;
my_map["hello"] = kSetSize;
EXPECT_FALSE(my_map_copy == my_map);
EXPECT_FALSE(my_map == my_map_copy);
EXPECT_TRUE(my_map_copy != my_map);
EXPECT_TRUE(my_map != my_map_copy);
my_map.erase(std::string(kSetSize - 1, 'a'));
EXPECT_FALSE(my_map_copy == my_map);
EXPECT_FALSE(my_map == my_map_copy);
EXPECT_TRUE(my_map_copy != my_map);
EXPECT_TRUE(my_map != my_map_copy);
}
} // namespace
} // namespace btree

1322
lib/xdelta3/draft-korn-vcdiff.txt Normal file
View file

File diff suppressed because it is too large Load diff

32
lib/xdelta3/examples/Makefile Normal file
View file

@ -0,0 +1,32 @@
CFLAGS = -g -Wall -I.. -DXD3_DEBUG=1 -DNDEBUG=0 -DSIZEOF_SIZE_T=8 -DSIZEOF_UNSIGNED_LONG_LONG=8
#CFLAGS = -O3 -Wall -I.. -DXD3_DEBUG=0 -fno-builtin -DNDEBUG=1
# -pg
SOURCES = small_page_test.c encode_decode_test.c speed_test.c
DEPS = ../*.h ../*.c *.h
TARGETS = small_page_test encode_decode_test speed_test32 speed_test64 compare_test checksum_test
all: $(TARGETS)
small_page_test: small_page_test.c $(DEPS)
$(CC) $(CFLAGS) small_page_test.c -o small_page_test -DXD3_USE_LARGEFILE64=0 -DSECONDARY_DJW=1
encode_decode_test: encode_decode_test.c $(DEPS)
$(CC) $(CFLAGS) encode_decode_test.c -o encode_decode_test
speed_test32: speed_test.c $(DEPS)
$(CC) $(CFLAGS) -DXD3_USE_LARGEFILE64=0 speed_test.c -o speed_test32
speed_test64: speed_test.c $(DEPS)
$(CC) $(CFLAGS) -DXD3_USE_LARGEFILE64=1 speed_test.c -o speed_test64
compare_test: compare_test.c
$(CC) $(CFLAGS) compare_test.c -o compare_test
checksum_test: checksum_test.cc
$(CXX) $(CFLAGS) checksum_test.cc -o checksum_test
clean:
rm -r -f *.exe *.stackdump $(TARGETS) *.dSYM *~

8
lib/xdelta3/examples/README.md Normal file
View file

@ -0,0 +1,8 @@
Files in this directory demonstrate how to use the Xdelta3 API. Copyrights
are held by the respective authors.
small_page_test.c -- how to use xdelta3 in an environment such as the kernel
for small pages with little memory
encode_decode_test.c -- how to use xdelta3 to process (encode/decode) data in
multiple windows with the non-blocking API

138
lib/xdelta3/examples/compare_test.c Normal file
View file

@ -0,0 +1,138 @@
/* xdelta3 - delta compression tools and library
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <string.h>
#include <assert.h>
#include "xdelta3.h"
#define NUM (1<<20)
#define ITERS 100
/* From wikipedia on RDTSC */
inline uint64_t rdtsc() {
uint32_t lo, hi;
asm volatile ("rdtsc" : "=a" (lo), "=d" (hi));
return (uint64_t)hi << 32 | lo;
}
typedef int (*test_func)(const char *s1, const char *s2, int n);
void run_test(const char *buf1, const char *buf2,
const char *name, test_func func) {
uint64_t start, end;
uint64_t accum = 0;
int i, x;
for (i = 0; i < ITERS; i++) {
start = rdtsc();
x = func(buf1, buf2, NUM);
end = rdtsc();
accum += end - start;
assert(x == NUM - 1);
}
accum /= ITERS;
printf("%s : %qu cycles\n", name, accum);
}
/* Build w/ -fno-builtin for this to be fast, this assumes that there
* is a difference at s1[n-1] */
int memcmp_fake(const char *s1, const char *s2, int n) {
int x = memcmp(s1, s2, n);
return x < 0 ? n - 1 : n + 1;
}
#define UNALIGNED_OK 1
static inline int
test2(const char *s1c, const char *s2c, int n)
{
int i = 0;
#if UNALIGNED_OK
int nint = n / sizeof(int);
if (nint >> 3)
{
int j = 0;
const int *s1 = (const int*)s1c;
const int *s2 = (const int*)s2c;
int nint_8 = nint - 8;
while (i <= nint_8 &&
s1[i++] == s2[j++] &&
s1[i++] == s2[j++] &&
s1[i++] == s2[j++] &&
s1[i++] == s2[j++] &&
s1[i++] == s2[j++] &&
s1[i++] == s2[j++] &&
s1[i++] == s2[j++] &&
s1[i++] == s2[j++]) { }
i = (i - 1) * sizeof(int);
}
#endif
while (i < n && s1c[i] == s2c[i])
{
i++;
}
return i;
}
static inline int
test1(const char *s1c, const char *s2c, int n) {
int i = 0;
while (i < n && s1c[i] == s2c[i])
{
i++;
}
return i;
}
int main(/*int argc, char **argv*/) {
char *buf1 = malloc(NUM+1);
char *buf2 = malloc(NUM+1);
int i;
for (i = 0; i < NUM; i++) {
buf1[i] = buf2[i] = rand();
}
buf2[NUM-1]++;
printf ("ALIGNED\n");
run_test(buf1, buf2, "memcmp", &memcmp_fake);
run_test(buf1, buf2, "test1", &test1);
run_test(buf1, buf2, "test2", &test2);
for (i = 0; i < NUM; i++) {
buf1[i] = buf2[i+1] = rand();
}
buf2[NUM]++;
printf ("UNALIGNED\n");
run_test(buf1, buf2+1, "memcmp", &memcmp_fake);
run_test(buf1, buf2+1, "test1", &test1);
run_test(buf1, buf2+1, "test2", &test2);
return 0;
}

203
lib/xdelta3/examples/encode_decode_test.c Normal file
View file

@ -0,0 +1,203 @@
// Permission to distribute this example by
// Copyright (C) 2007 Ralf Junker
// Ralf Junker <delphi@yunqa.de>
// http://www.yunqa.de/delphi/
//---------------------------------------------------------------------------
#include <stdio.h>
#include <sys/stat.h>
#include "xdelta3.h"
#include "xdelta3.c"
//---------------------------------------------------------------------------
int code (
int encode,
FILE* InFile,
FILE* SrcFile ,
FILE* OutFile,
int BufSize )
{
int r, ret;
struct stat statbuf;
xd3_stream stream;
xd3_config config;
xd3_source source;
void* Input_Buf;
int Input_Buf_Read;
if (BufSize < XD3_ALLOCSIZE)
BufSize = XD3_ALLOCSIZE;
memset (&stream, 0, sizeof (stream));
memset (&source, 0, sizeof (source));
xd3_init_config(&config, XD3_ADLER32);
config.winsize = BufSize;
xd3_config_stream(&stream, &config);
if (SrcFile)
{
r = fstat(fileno(SrcFile), &statbuf);
if (r)
return r;
source.blksize = BufSize;
source.curblk = malloc(source.blksize);
/* Load 1st block of stream. */
r = fseek(SrcFile, 0, SEEK_SET);
if (r)
return r;
source.onblk = fread((void*)source.curblk, 1, source.blksize, SrcFile);
source.curblkno = 0;
/* Set the stream. */
xd3_set_source(&stream, &source);
}
Input_Buf = malloc(BufSize);
fseek(InFile, 0, SEEK_SET);
do
{
Input_Buf_Read = fread(Input_Buf, 1, BufSize, InFile);
if (Input_Buf_Read < BufSize)
{
xd3_set_flags(&stream, XD3_FLUSH | stream.flags);
}
xd3_avail_input(&stream, Input_Buf, Input_Buf_Read);
process:
if (encode)
ret = xd3_encode_input(&stream);
else
ret = xd3_decode_input(&stream);
switch (ret)
{
case XD3_INPUT:
{
fprintf (stderr,"XD3_INPUT\n");
continue;
}
case XD3_OUTPUT:
{
fprintf (stderr,"XD3_OUTPUT\n");
r = fwrite(stream.next_out, 1, stream.avail_out, OutFile);
if (r != (int)stream.avail_out)
return r;
xd3_consume_output(&stream);
goto process;
}
case XD3_GETSRCBLK:
{
fprintf (stderr,"XD3_GETSRCBLK %qd\n", source.getblkno);
if (SrcFile)
{
r = fseek(SrcFile, source.blksize * source.getblkno, SEEK_SET);
if (r)
return r;
source.onblk = fread((void*)source.curblk, 1,
source.blksize, SrcFile);
source.curblkno = source.getblkno;
}
goto process;
}
case XD3_GOTHEADER:
{
fprintf (stderr,"XD3_GOTHEADER\n");
goto process;
}
case XD3_WINSTART:
{
fprintf (stderr,"XD3_WINSTART\n");
goto process;
}
case XD3_WINFINISH:
{
fprintf (stderr,"XD3_WINFINISH\n");
goto process;
}
default:
{
fprintf (stderr,"!!! INVALID %s %d !!!\n",
stream.msg, ret);
return ret;
}
}
}
while (Input_Buf_Read == BufSize);
free(Input_Buf);
free((void*)source.curblk);
xd3_close_stream(&stream);
xd3_free_stream(&stream);
return 0;
};
int main(int argc, char* argv[])
{
FILE* InFile;
FILE* SrcFile;
FILE* OutFile;
int r;
if (argc != 3) {
fprintf (stderr, "usage: %s source input\n", argv[0]);
return 1;
}
char *input = argv[2];
char *source = argv[1];
const char *output = "encoded.testdata";
const char *decoded = "decoded.testdata";
/* Encode */
InFile = fopen(input, "rb");
SrcFile = fopen(source, "rb");
OutFile = fopen(output, "wb");
r = code (1, InFile, SrcFile, OutFile, 0x1000);
fclose(OutFile);
fclose(SrcFile);
fclose(InFile);
if (r) {
fprintf (stderr, "Encode error: %d\n", r);
return r;
}
/* Decode */
InFile = fopen(output, "rb");
SrcFile = fopen(source, "rb");
OutFile = fopen(decoded, "wb");
r = code (0, InFile, SrcFile, OutFile, 0x1000);
fclose(OutFile);
fclose(SrcFile);
fclose(InFile);
if (r) {
fprintf (stderr, "Decode error: %d\n", r);
return r;
}
return 0;
}

389
lib/xdelta3/examples/iOS/xdelta3-ios-test/xdelta3-ios-test.xcodeproj/project.pbxproj Normal file
View file

@ -0,0 +1,389 @@
// !$*UTF8*$!
{
archiveVersion = 1;
classes = {
};
objectVersion = 46;
objects = {
/* Begin PBXBuildFile section */
B9001B65158D008900B9E855 /* xdelta3.c in Sources */ = {isa = PBXBuildFile; fileRef = B9001B63158D008900B9E855 /* xdelta3.c */; };
B9313C3C158D11BA001C1F28 /* file_v1_to_v2.bin in Resources */ = {isa = PBXBuildFile; fileRef = B9313C39158D11BA001C1F28 /* file_v1_to_v2.bin */; };
B9313C3D158D11BA001C1F28 /* file_v1.bin in Resources */ = {isa = PBXBuildFile; fileRef = B9313C3A158D11BA001C1F28 /* file_v1.bin */; };
B9313C3E158D11BA001C1F28 /* file_v2.bin in Resources */ = {isa = PBXBuildFile; fileRef = B9313C3B158D11BA001C1F28 /* file_v2.bin */; };
B9ADC6BF158CFD36007EF999 /* UIKit.framework in Frameworks */ = {isa = PBXBuildFile; fileRef = B9ADC6BE158CFD36007EF999 /* UIKit.framework */; };
B9ADC6C1158CFD36007EF999 /* Foundation.framework in Frameworks */ = {isa = PBXBuildFile; fileRef = B9ADC6C0158CFD36007EF999 /* Foundation.framework */; };
B9ADC6C3158CFD36007EF999 /* CoreGraphics.framework in Frameworks */ = {isa = PBXBuildFile; fileRef = B9ADC6C2158CFD36007EF999 /* CoreGraphics.framework */; };
B9ADC6C9158CFD36007EF999 /* InfoPlist.strings in Resources */ = {isa = PBXBuildFile; fileRef = B9ADC6C7158CFD36007EF999 /* InfoPlist.strings */; };
B9ADC6CB158CFD36007EF999 /* main.m in Sources */ = {isa = PBXBuildFile; fileRef = B9ADC6CA158CFD36007EF999 /* main.m */; };
B9ADC6CF158CFD36007EF999 /* Xd3iOSAppDelegate.m in Sources */ = {isa = PBXBuildFile; fileRef = B9ADC6CE158CFD36007EF999 /* Xd3iOSAppDelegate.m */; };
B9ADC6D2158CFD36007EF999 /* MainStoryboard_iPhone.storyboard in Resources */ = {isa = PBXBuildFile; fileRef = B9ADC6D0158CFD36007EF999 /* MainStoryboard_iPhone.storyboard */; };
B9ADC6D5158CFD36007EF999 /* MainStoryboard_iPad.storyboard in Resources */ = {isa = PBXBuildFile; fileRef = B9ADC6D3158CFD36007EF999 /* MainStoryboard_iPad.storyboard */; };
B9ADC6D8158CFD36007EF999 /* Xd3iOSViewController.m in Sources */ = {isa = PBXBuildFile; fileRef = B9ADC6D7158CFD36007EF999 /* Xd3iOSViewController.m */; };
/* End PBXBuildFile section */
/* Begin PBXFileReference section */
B9001B56158D008900B9E855 /* xdelta3-blkcache.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = "xdelta3-blkcache.h"; path = "../../../../xdelta3-blkcache.h"; sourceTree = "<group>"; };
B9001B57158D008900B9E855 /* xdelta3-cfgs.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = "xdelta3-cfgs.h"; path = "../../../../xdelta3-cfgs.h"; sourceTree = "<group>"; };
B9001B58158D008900B9E855 /* xdelta3-decode.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = "xdelta3-decode.h"; path = "../../../../xdelta3-decode.h"; sourceTree = "<group>"; };
B9001B59158D008900B9E855 /* xdelta3-djw.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = "xdelta3-djw.h"; path = "../../../../xdelta3-djw.h"; sourceTree = "<group>"; };
B9001B5A158D008900B9E855 /* xdelta3-fgk.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = "xdelta3-fgk.h"; path = "../../../../xdelta3-fgk.h"; sourceTree = "<group>"; };
B9001B5B158D008900B9E855 /* xdelta3-hash.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = "xdelta3-hash.h"; path = "../../../../xdelta3-hash.h"; sourceTree = "<group>"; };
B9001B5C158D008900B9E855 /* xdelta3-internal.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = "xdelta3-internal.h"; path = "../../../../xdelta3-internal.h"; sourceTree = "<group>"; };
B9001B5D158D008900B9E855 /* xdelta3-list.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = "xdelta3-list.h"; path = "../../../../xdelta3-list.h"; sourceTree = "<group>"; };
B9001B5E158D008900B9E855 /* xdelta3-main.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = "xdelta3-main.h"; path = "../../../../xdelta3-main.h"; sourceTree = "<group>"; };
B9001B5F158D008900B9E855 /* xdelta3-merge.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = "xdelta3-merge.h"; path = "../../../../xdelta3-merge.h"; sourceTree = "<group>"; };
B9001B60158D008900B9E855 /* xdelta3-python.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = "xdelta3-python.h"; path = "../../../../xdelta3-python.h"; sourceTree = "<group>"; };
B9001B61158D008900B9E855 /* xdelta3-second.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = "xdelta3-second.h"; path = "../../../../xdelta3-second.h"; sourceTree = "<group>"; };
B9001B62158D008900B9E855 /* xdelta3-test.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = "xdelta3-test.h"; path = "../../../../xdelta3-test.h"; sourceTree = "<group>"; };
B9001B63158D008900B9E855 /* xdelta3.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; name = xdelta3.c; path = ../../../../xdelta3.c; sourceTree = "<group>"; };
B9001B64158D008900B9E855 /* xdelta3.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = xdelta3.h; path = ../../../../xdelta3.h; sourceTree = "<group>"; };
B9313C39158D11BA001C1F28 /* file_v1_to_v2.bin */ = {isa = PBXFileReference; lastKnownFileType = archive.macbinary; path = file_v1_to_v2.bin; sourceTree = "<group>"; };
B9313C3A158D11BA001C1F28 /* file_v1.bin */ = {isa = PBXFileReference; lastKnownFileType = archive.macbinary; path = file_v1.bin; sourceTree = "<group>"; };
B9313C3B158D11BA001C1F28 /* file_v2.bin */ = {isa = PBXFileReference; lastKnownFileType = archive.macbinary; path = file_v2.bin; sourceTree = "<group>"; };
B9ADC6BA158CFD36007EF999 /* xdelta3-ios-test.app */ = {isa = PBXFileReference; explicitFileType = wrapper.application; includeInIndex = 0; path = "xdelta3-ios-test.app"; sourceTree = BUILT_PRODUCTS_DIR; };
B9ADC6BE158CFD36007EF999 /* UIKit.framework */ = {isa = PBXFileReference; lastKnownFileType = wrapper.framework; name = UIKit.framework; path = System/Library/Frameworks/UIKit.framework; sourceTree = SDKROOT; };
B9ADC6C0158CFD36007EF999 /* Foundation.framework */ = {isa = PBXFileReference; lastKnownFileType = wrapper.framework; name = Foundation.framework; path = System/Library/Frameworks/Foundation.framework; sourceTree = SDKROOT; };
B9ADC6C2158CFD36007EF999 /* CoreGraphics.framework */ = {isa = PBXFileReference; lastKnownFileType = wrapper.framework; name = CoreGraphics.framework; path = System/Library/Frameworks/CoreGraphics.framework; sourceTree = SDKROOT; };
B9ADC6C6158CFD36007EF999 /* xdelta3-ios-test-Info.plist */ = {isa = PBXFileReference; lastKnownFileType = text.plist.xml; path = "xdelta3-ios-test-Info.plist"; sourceTree = "<group>"; };
B9ADC6C8158CFD36007EF999 /* en */ = {isa = PBXFileReference; lastKnownFileType = text.plist.strings; name = en; path = en.lproj/InfoPlist.strings; sourceTree = "<group>"; };
B9ADC6CA158CFD36007EF999 /* main.m */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.objc; path = main.m; sourceTree = "<group>"; };
B9ADC6CC158CFD36007EF999 /* xdelta3-ios-test-Prefix.pch */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = "xdelta3-ios-test-Prefix.pch"; sourceTree = "<group>"; };
B9ADC6CD158CFD36007EF999 /* Xd3iOSAppDelegate.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = Xd3iOSAppDelegate.h; sourceTree = "<group>"; };
B9ADC6CE158CFD36007EF999 /* Xd3iOSAppDelegate.m */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.objc; path = Xd3iOSAppDelegate.m; sourceTree = "<group>"; };
B9ADC6D1158CFD36007EF999 /* en */ = {isa = PBXFileReference; lastKnownFileType = file.storyboard; name = en; path = en.lproj/MainStoryboard_iPhone.storyboard; sourceTree = "<group>"; };
B9ADC6D4158CFD36007EF999 /* en */ = {isa = PBXFileReference; lastKnownFileType = file.storyboard; name = en; path = en.lproj/MainStoryboard_iPad.storyboard; sourceTree = "<group>"; };
B9ADC6D6158CFD36007EF999 /* Xd3iOSViewController.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = Xd3iOSViewController.h; sourceTree = "<group>"; };
B9ADC6D7158CFD36007EF999 /* Xd3iOSViewController.m */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.objc; path = Xd3iOSViewController.m; sourceTree = "<group>"; };
/* End PBXFileReference section */
/* Begin PBXFrameworksBuildPhase section */
B9ADC6B7158CFD36007EF999 /* Frameworks */ = {
isa = PBXFrameworksBuildPhase;
buildActionMask = 2147483647;
files = (
B9ADC6BF158CFD36007EF999 /* UIKit.framework in Frameworks */,
B9ADC6C1158CFD36007EF999 /* Foundation.framework in Frameworks */,
B9ADC6C3158CFD36007EF999 /* CoreGraphics.framework in Frameworks */,
);
runOnlyForDeploymentPostprocessing = 0;
};
/* End PBXFrameworksBuildPhase section */
/* Begin PBXGroup section */
B9ADC6AF158CFD36007EF999 = {
isa = PBXGroup;
children = (
B9ADC6C4158CFD36007EF999 /* xdelta3-ios-test */,
B9ADC6BD158CFD36007EF999 /* Frameworks */,
B9ADC6BB158CFD36007EF999 /* Products */,
);
sourceTree = "<group>";
};
B9ADC6BB158CFD36007EF999 /* Products */ = {
isa = PBXGroup;
children = (
B9ADC6BA158CFD36007EF999 /* xdelta3-ios-test.app */,
);
name = Products;
sourceTree = "<group>";
};
B9ADC6BD158CFD36007EF999 /* Frameworks */ = {
isa = PBXGroup;
children = (
B9ADC6BE158CFD36007EF999 /* UIKit.framework */,
B9ADC6C0158CFD36007EF999 /* Foundation.framework */,
B9ADC6C2158CFD36007EF999 /* CoreGraphics.framework */,
);
name = Frameworks;
sourceTree = "<group>";
};
B9ADC6C4158CFD36007EF999 /* xdelta3-ios-test */ = {
isa = PBXGroup;
children = (
B9001B56158D008900B9E855 /* xdelta3-blkcache.h */,
B9001B57158D008900B9E855 /* xdelta3-cfgs.h */,
B9001B58158D008900B9E855 /* xdelta3-decode.h */,
B9001B59158D008900B9E855 /* xdelta3-djw.h */,
B9001B5A158D008900B9E855 /* xdelta3-fgk.h */,
B9001B5B158D008900B9E855 /* xdelta3-hash.h */,
B9001B5C158D008900B9E855 /* xdelta3-internal.h */,
B9001B5D158D008900B9E855 /* xdelta3-list.h */,
B9001B5E158D008900B9E855 /* xdelta3-main.h */,
B9001B5F158D008900B9E855 /* xdelta3-merge.h */,
B9001B60158D008900B9E855 /* xdelta3-python.h */,
B9001B61158D008900B9E855 /* xdelta3-second.h */,
B9001B62158D008900B9E855 /* xdelta3-test.h */,
B9001B63158D008900B9E855 /* xdelta3.c */,
B9001B64158D008900B9E855 /* xdelta3.h */,
B9ADC6CD158CFD36007EF999 /* Xd3iOSAppDelegate.h */,
B9ADC6CE158CFD36007EF999 /* Xd3iOSAppDelegate.m */,
B9ADC6D0158CFD36007EF999 /* MainStoryboard_iPhone.storyboard */,
B9ADC6D3158CFD36007EF999 /* MainStoryboard_iPad.storyboard */,
B9ADC6D6158CFD36007EF999 /* Xd3iOSViewController.h */,
B9ADC6D7158CFD36007EF999 /* Xd3iOSViewController.m */,
B9ADC6C5158CFD36007EF999 /* Supporting Files */,
);
path = "xdelta3-ios-test";
sourceTree = "<group>";
};
B9ADC6C5158CFD36007EF999 /* Supporting Files */ = {
isa = PBXGroup;
children = (
B9313C39158D11BA001C1F28 /* file_v1_to_v2.bin */,
B9313C3A158D11BA001C1F28 /* file_v1.bin */,
B9313C3B158D11BA001C1F28 /* file_v2.bin */,
B9ADC6C6158CFD36007EF999 /* xdelta3-ios-test-Info.plist */,
B9ADC6C7158CFD36007EF999 /* InfoPlist.strings */,
B9ADC6CA158CFD36007EF999 /* main.m */,
B9ADC6CC158CFD36007EF999 /* xdelta3-ios-test-Prefix.pch */,
);
name = "Supporting Files";
sourceTree = "<group>";
};
/* End PBXGroup section */
/* Begin PBXNativeTarget section */
B9ADC6B9158CFD36007EF999 /* xdelta3-ios-test */ = {
isa = PBXNativeTarget;
buildConfigurationList = B9ADC6DB158CFD36007EF999 /* Build configuration list for PBXNativeTarget "xdelta3-ios-test" */;
buildPhases = (
B9ADC6B6158CFD36007EF999 /* Sources */,
B9ADC6B7158CFD36007EF999 /* Frameworks */,
B9ADC6B8158CFD36007EF999 /* Resources */,
);
buildRules = (
);
dependencies = (
);
name = "xdelta3-ios-test";
productName = "xdelta3-ios-test";
productReference = B9ADC6BA158CFD36007EF999 /* xdelta3-ios-test.app */;
productType = "com.apple.product-type.application";
};
/* End PBXNativeTarget section */
/* Begin PBXProject section */
B9ADC6B1158CFD36007EF999 /* Project object */ = {
isa = PBXProject;
attributes = {
LastUpgradeCheck = 0420;
};
buildConfigurationList = B9ADC6B4158CFD36007EF999 /* Build configuration list for PBXProject "xdelta3-ios-test" */;
compatibilityVersion = "Xcode 3.2";
developmentRegion = English;
hasScannedForEncodings = 0;
knownRegions = (
en,
);
mainGroup = B9ADC6AF158CFD36007EF999;
productRefGroup = B9ADC6BB158CFD36007EF999 /* Products */;
projectDirPath = "";
projectRoot = "";
targets = (
B9ADC6B9158CFD36007EF999 /* xdelta3-ios-test */,
);
};
/* End PBXProject section */
/* Begin PBXResourcesBuildPhase section */
B9ADC6B8158CFD36007EF999 /* Resources */ = {
isa = PBXResourcesBuildPhase;
buildActionMask = 2147483647;
files = (
B9ADC6C9158CFD36007EF999 /* InfoPlist.strings in Resources */,
B9ADC6D2158CFD36007EF999 /* MainStoryboard_iPhone.storyboard in Resources */,
B9ADC6D5158CFD36007EF999 /* MainStoryboard_iPad.storyboard in Resources */,
B9313C3C158D11BA001C1F28 /* file_v1_to_v2.bin in Resources */,
B9313C3D158D11BA001C1F28 /* file_v1.bin in Resources */,
B9313C3E158D11BA001C1F28 /* file_v2.bin in Resources */,
);
runOnlyForDeploymentPostprocessing = 0;
};
/* End PBXResourcesBuildPhase section */
/* Begin PBXSourcesBuildPhase section */
B9ADC6B6158CFD36007EF999 /* Sources */ = {
isa = PBXSourcesBuildPhase;
buildActionMask = 2147483647;
files = (
B9ADC6CB158CFD36007EF999 /* main.m in Sources */,
B9ADC6CF158CFD36007EF999 /* Xd3iOSAppDelegate.m in Sources */,
B9ADC6D8158CFD36007EF999 /* Xd3iOSViewController.m in Sources */,
B9001B65158D008900B9E855 /* xdelta3.c in Sources */,
);
runOnlyForDeploymentPostprocessing = 0;
};
/* End PBXSourcesBuildPhase section */
/* Begin PBXVariantGroup section */
B9ADC6C7158CFD36007EF999 /* InfoPlist.strings */ = {
isa = PBXVariantGroup;
children = (
B9ADC6C8158CFD36007EF999 /* en */,
);
name = InfoPlist.strings;
sourceTree = "<group>";
};
B9ADC6D0158CFD36007EF999 /* MainStoryboard_iPhone.storyboard */ = {
isa = PBXVariantGroup;
children = (
B9ADC6D1158CFD36007EF999 /* en */,
);
name = MainStoryboard_iPhone.storyboard;
sourceTree = "<group>";
};
B9ADC6D3158CFD36007EF999 /* MainStoryboard_iPad.storyboard */ = {
isa = PBXVariantGroup;
children = (
B9ADC6D4158CFD36007EF999 /* en */,
);
name = MainStoryboard_iPad.storyboard;
sourceTree = "<group>";
};
/* End PBXVariantGroup section */
/* Begin XCBuildConfiguration section */
B9ADC6D9158CFD36007EF999 /* Debug */ = {
isa = XCBuildConfiguration;
buildSettings = {
ALWAYS_SEARCH_USER_PATHS = NO;
ARCHS = "$(ARCHS_STANDARD_32_BIT)";
CLANG_ENABLE_OBJC_ARC = YES;
"CODE_SIGN_IDENTITY[sdk=iphoneos*]" = "iPhone Developer";
COPY_PHASE_STRIP = NO;
GCC_C_LANGUAGE_STANDARD = gnu99;
GCC_DYNAMIC_NO_PIC = NO;
GCC_INPUT_FILETYPE = sourcecode.c.objc;
GCC_OPTIMIZATION_LEVEL = 0;
GCC_PREPROCESSOR_DEFINITIONS = (
"XD3_USE_LARGEFILE64=0",
"XD3_POSIX=1",
"EXTERNAL_COMPRESSION=0",
"NOT_MAIN=1",
"XD3_MAIN=1",
"SECONDARY_DJW=1",
"XD3_DEBUG=1",
"REGRESSION_TEST=1",
"SHELL_TESTS=0",
"SECONDARY_FGK=1",
"DEBUG=1",
"$(inherited)",
);
GCC_SYMBOLS_PRIVATE_EXTERN = NO;
GCC_VERSION = com.apple.compilers.llvm.clang.1_0;
GCC_WARN_ABOUT_MISSING_PROTOTYPES = YES;
GCC_WARN_ABOUT_RETURN_TYPE = YES;
GCC_WARN_UNUSED_VARIABLE = YES;
IPHONEOS_DEPLOYMENT_TARGET = 5.0;
OTHER_CFLAGS = (
"-DXD3_USE_LARGEFILE64=0",
"-DXD3_POSIX=1",
"-DEXTERNAL_COMPRESSION=0",
"-DNOT_MAIN=1",
"-DXD3_MAIN=1",
"-DSECONDARY_DJW=1",
"-DXD3_DEBUG=1",
"-DREGRESSION_TEST=1",
"-DSHELL_TESTS=0",
"-DSECONDARY_FGK=1",
);
SDKROOT = iphoneos;
TARGETED_DEVICE_FAMILY = "1,2";
};
name = Debug;
};
B9ADC6DA158CFD36007EF999 /* Release */ = {
isa = XCBuildConfiguration;
buildSettings = {
ALWAYS_SEARCH_USER_PATHS = NO;
ARCHS = "$(ARCHS_STANDARD_32_BIT)";
CLANG_ENABLE_OBJC_ARC = YES;
"CODE_SIGN_IDENTITY[sdk=iphoneos*]" = "iPhone Developer";
COPY_PHASE_STRIP = YES;
GCC_C_LANGUAGE_STANDARD = gnu99;
GCC_INPUT_FILETYPE = sourcecode.c.objc;
GCC_PREPROCESSOR_DEFINITIONS = (
"XD3_USE_LARGEFILE64=0",
"XD3_POSIX=1",
"EXTERNAL_COMPRESSION=0",
"NOT_MAIN=1",
"XD3_MAIN=1",
"SECONDARY_DJW=1",
"XD3_DEBUG=1",
"REGRESSION_TEST=1",
"SHELL_TESTS=0",
"SECONDARY_FGK=1",
);
GCC_VERSION = com.apple.compilers.llvm.clang.1_0;
GCC_WARN_ABOUT_MISSING_PROTOTYPES = YES;
GCC_WARN_ABOUT_RETURN_TYPE = YES;
GCC_WARN_UNUSED_VARIABLE = YES;
IPHONEOS_DEPLOYMENT_TARGET = 5.0;
OTHER_CFLAGS = (
"-DXD3_USE_LARGEFILE64=0",
"-DXD3_POSIX=1",
"-DEXTERNAL_COMPRESSION=0",
"-DNOT_MAIN=1",
"-DXD3_MAIN=1",
"-DSECONDARY_DJW=1",
"-DXD3_DEBUG=1",
"-DREGRESSION_TEST=1",
"-DSHELL_TESTS=0",
"-DSECONDARY_FGK=1",
);
SDKROOT = iphoneos;
TARGETED_DEVICE_FAMILY = "1,2";
VALIDATE_PRODUCT = YES;
};
name = Release;
};
B9ADC6DC158CFD36007EF999 /* Debug */ = {
isa = XCBuildConfiguration;
buildSettings = {
GCC_PRECOMPILE_PREFIX_HEADER = YES;
GCC_PREFIX_HEADER = "xdelta3-ios-test/xdelta3-ios-test-Prefix.pch";
INFOPLIST_FILE = "xdelta3-ios-test/xdelta3-ios-test-Info.plist";
OTHER_CFLAGS = "";
PRODUCT_NAME = "$(TARGET_NAME)";
WRAPPER_EXTENSION = app;
};
name = Debug;
};
B9ADC6DD158CFD36007EF999 /* Release */ = {
isa = XCBuildConfiguration;
buildSettings = {
GCC_PRECOMPILE_PREFIX_HEADER = YES;
GCC_PREFIX_HEADER = "xdelta3-ios-test/xdelta3-ios-test-Prefix.pch";
INFOPLIST_FILE = "xdelta3-ios-test/xdelta3-ios-test-Info.plist";
OTHER_CFLAGS = "";
PRODUCT_NAME = "$(TARGET_NAME)";
WRAPPER_EXTENSION = app;
};
name = Release;
};
/* End XCBuildConfiguration section */
/* Begin XCConfigurationList section */
B9ADC6B4158CFD36007EF999 /* Build configuration list for PBXProject "xdelta3-ios-test" */ = {
isa = XCConfigurationList;
buildConfigurations = (
B9ADC6D9158CFD36007EF999 /* Debug */,
B9ADC6DA158CFD36007EF999 /* Release */,
);
defaultConfigurationIsVisible = 0;
defaultConfigurationName = Release;
};
B9ADC6DB158CFD36007EF999 /* Build configuration list for PBXNativeTarget "xdelta3-ios-test" */ = {
isa = XCConfigurationList;
buildConfigurations = (
B9ADC6DC158CFD36007EF999 /* Debug */,
B9ADC6DD158CFD36007EF999 /* Release */,
);
defaultConfigurationIsVisible = 0;
defaultConfigurationName = Release;
};
/* End XCConfigurationList section */
};
rootObject = B9ADC6B1158CFD36007EF999 /* Project object */;
}

23
lib/xdelta3/examples/iOS/xdelta3-ios-test/xdelta3-ios-test/Xd3iOSAppDelegate.h Normal file
View file

@ -0,0 +1,23 @@
/* xdelta3 - delta compression tools and library -*- Mode: objc *-*
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#import <UIKit/UIKit.h>
@interface Xd3iOSAppDelegate : UIResponder <UIApplicationDelegate>
@property (strong, nonatomic) UIWindow *window;
@end

68
lib/xdelta3/examples/iOS/xdelta3-ios-test/xdelta3-ios-test/Xd3iOSAppDelegate.m Normal file
View file

@ -0,0 +1,68 @@
/* xdelta3 - delta compression tools and library -*- Mode: objc *-*
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#import "Xd3iOSAppDelegate.h"
@implementation Xd3iOSAppDelegate
@synthesize window = _window;
- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions
{
// Override point for customization after application launch.
return YES;
}
- (void)applicationWillResignActive:(UIApplication *)application
{
/*
Sent when the application is about to move from active to inactive state. This can occur for certain types of temporary interruptions (such as an incoming phone call or SMS message) or when the user quits the application and it begins the transition to the background state.
Use this method to pause ongoing tasks, disable timers, and throttle down OpenGL ES frame rates. Games should use this method to pause the game.
*/
}
- (void)applicationDidEnterBackground:(UIApplication *)application
{
/*
Use this method to release shared resources, save user data, invalidate timers, and store enough application state information to restore your application to its current state in case it is terminated later.
If your application supports background execution, this method is called instead of applicationWillTerminate: when the user quits.
*/
}
- (void)applicationWillEnterForeground:(UIApplication *)application
{
/*
Called as part of the transition from the background to the inactive state; here you can undo many of the changes made on entering the background.
*/
}
- (void)applicationDidBecomeActive:(UIApplication *)application
{
/*
Restart any tasks that were paused (or not yet started) while the application was inactive. If the application was previously in the background, optionally refresh the user interface.
*/
}
- (void)applicationWillTerminate:(UIApplication *)application
{
/*
Called when the application is about to terminate.
Save data if appropriate.
See also applicationDidEnterBackground:.
*/
}
@end

28
lib/xdelta3/examples/iOS/xdelta3-ios-test/xdelta3-ios-test/Xd3iOSViewController.h Normal file
View file

@ -0,0 +1,28 @@
/* xdelta3 - delta compression tools and library -*- Mode: objc *-*
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#import <UIKit/UIKit.h>
@interface Xd3iOSViewController : UIViewController <UITextViewDelegate> {
NSString *inputSeed;
}
- (IBAction)startTest:(id)sender;
@property (weak, nonatomic) IBOutlet UITextField *theSeed;
@property (weak, nonatomic) IBOutlet UITextView *theView;
@property (atomic, retain) NSMutableString *theOutput;
@property (nonatomic) BOOL inTest;
@end

177
lib/xdelta3/examples/iOS/xdelta3-ios-test/xdelta3-ios-test/Xd3iOSViewController.m Normal file
View file

@ -0,0 +1,177 @@
/* xdelta3 - delta compression tools and library -*- Mode: objc *-*
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#import "Xd3iOSViewController.h"
#include "xdelta3.h"
#include "dispatch/queue.h"
#include "Foundation/NSBundle.h"
extern void (*xprintf_message_func)(const char* msg);
void print_to_view(const char* buf);
int xd3_main_cmdline(int argc, char **argv);
void do_localfile_test(void);
int compare_files(const char* file1, const char* file2);
Xd3iOSViewController *static_ptr;
@implementation Xd3iOSViewController
@synthesize theSeed = _theSeed;
@synthesize theView = _theView;
@synthesize theOutput = _theOutput;
@synthesize inTest = _inTest;
- (void)didReceiveMemoryWarning
{
[super didReceiveMemoryWarning];
}
#pragma mark - View lifecycle
- (void)viewDidLoad
{
[super viewDidLoad];
}
- (void)viewDidUnload
{
[self setTheSeed:nil];
[self setTheView:nil];
[self setTheView:nil];
[super viewDidUnload];
}
- (void)viewWillAppear:(BOOL)animated
{
[super viewWillAppear:animated];
}
- (void)viewDidAppear:(BOOL)animated
{
[super viewDidAppear:animated];
}
- (void)viewWillDisappear:(BOOL)animated
{
[super viewWillDisappear:animated];
}
- (void)viewDidDisappear:(BOOL)animated
{
[super viewDidDisappear:animated];
}
- (BOOL)shouldAutorotateToInterfaceOrientation:(UIInterfaceOrientation)interfaceOrientation
{
switch (interfaceOrientation) {
case UIInterfaceOrientationPortrait:
case UIInterfaceOrientationPortraitUpsideDown:
return YES;
default:
break;
}
return NO;
}
- (BOOL)textFieldShouldReturn:(UITextField*)theTextField {
if (theTextField == self.theSeed) {
[theTextField resignFirstResponder];
}
return YES;
}
- (IBAction)startTest:(id)sender {
if (self.inTest) {
return;
}
self.inTest = YES;
NSString *seedString = self.theSeed.text;
if ([seedString length] == 0) {
seedString = @"RFC3284";
}
static_ptr = self;
xprintf_message_func = &print_to_view;
self.theOutput = [[NSMutableString alloc] initWithFormat:@"Starting test (seed=%@)\n", seedString];
self.theView.text = self.theOutput;
dispatch_queue_t mq = dispatch_get_main_queue();
dispatch_queue_t dq = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
dispatch_async(dq, ^{
do_localfile_test();
char *argv[] = { "xdelta3", "test", NULL };
xd3_main_cmdline(2, argv);
print_to_view("Finished unittest: success");
dispatch_async(mq, ^{
self.inTest = NO;
});
});
}
void printns_to_view(NSString* ns);
void printns_to_view(NSString* ns) {
dispatch_queue_t mq = dispatch_get_main_queue();
dispatch_async(mq, ^{
if ([static_ptr.theOutput length] < 25000) {
[static_ptr.theOutput appendString:ns];
} else {
static_ptr.theOutput = [[NSMutableString alloc] initWithString:ns];
}
static_ptr.theView.text = static_ptr.theOutput;
CGSize size = static_ptr.theView.contentSize;
[static_ptr.theView scrollRectToVisible:CGRectMake(0, size.height - 1, 1, 1) animated:NO];
});
}
void print_to_view(const char* buf) {
NSString *ns = [NSString stringWithCString:buf encoding:NSASCIIStringEncoding];
printns_to_view(ns);
}
void do_localfile_test(void) {
NSBundle *bundle;
bundle = [NSBundle mainBundle];
NSString *localfile1 = [bundle pathForResource:@"file_v1" ofType:@"bin"];
NSString *localfile2 = [bundle pathForResource:@"file_v2" ofType:@"bin"];
NSString *localfiled = [bundle pathForResource:@"file_v1_to_v2" ofType:@"bin"];
printns_to_view([localfile1 stringByAppendingString:@"\n"]);
printns_to_view([localfile2 stringByAppendingString:@"\n"]);
printns_to_view([localfiled stringByAppendingString:@"\n"]);
NSString *tmpdir = NSTemporaryDirectory();
NSString *tmpfile = [tmpdir stringByAppendingPathComponent:@"delta.tmp"];
printns_to_view([tmpfile stringByAppendingString:@"\n"]);
char *argv[] = {
"xdelta3", "-dfvv", "-s",
(char*)[localfile1 UTF8String],
(char*)[localfiled UTF8String],
(char*)[tmpfile UTF8String] };
xd3_main_cmdline(6, argv);
NSFileManager *filemgr;
filemgr = [NSFileManager defaultManager];
if ([filemgr contentsEqualAtPath: localfile2 andPath: tmpfile] == YES) {
printns_to_view(@"File contents match\n");
} else {
NSError *err1 = NULL;
NSDictionary *d1 = [filemgr attributesOfItemAtPath: tmpfile error: &err1];
if (err1 != NULL) {
printns_to_view([@"File localfile2 could not stat %s\n" stringByAppendingString: tmpfile]);
} else {
printns_to_view([@"File contents do not match!!!! tmpfile size=" stringByAppendingString:
[[NSMutableString alloc] initWithFormat:@"%llu\n", [d1 fileSize]]]);
}
compare_files([localfile2 UTF8String], [tmpfile UTF8String]);
}
print_to_view("Finished localfile test.\n");
}
@end

2
lib/xdelta3/examples/iOS/xdelta3-ios-test/xdelta3-ios-test/en.lproj/InfoPlist.strings Normal file
View file

@ -0,0 +1,2 @@
/* Localized versions of Info.plist keys */

77
lib/xdelta3/examples/iOS/xdelta3-ios-test/xdelta3-ios-test/en.lproj/MainStoryboard_iPad.storyboard Normal file
View file

@ -0,0 +1,77 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="1.0" toolsVersion="1938" systemVersion="11C74" targetRuntime="iOS.CocoaTouch.iPad" propertyAccessControl="none" initialViewController="2">
<dependencies>
<development defaultVersion="4200" identifier="xcode"/>
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="933"/>
</dependencies>
<scenes>
<scene sceneID="4">
<objects>
<placeholder placeholderIdentifier="IBFirstResponder" id="3" sceneMemberID="firstResponder"/>
<viewController id="2" customClass="Xd3iOSViewController" sceneMemberID="viewController">
<view key="view" contentMode="scaleToFill" id="5">
<rect key="frame" x="0.0" y="20" width="768" height="1004"/>
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
<subviews>
<button opaque="NO" contentMode="scaleToFill" contentHorizontalAlignment="center" contentVerticalAlignment="center" buttonType="roundedRect" lineBreakMode="middleTruncation" id="d7Y-KS-zOa">
<rect key="frame" x="258" y="28" width="197" height="37"/>
<autoresizingMask key="autoresizingMask" flexibleMaxX="YES" flexibleMaxY="YES"/>
<fontDescription key="fontDescription" type="boldSystem" pointSize="15"/>
<state key="normal" title="Start test">
<color key="titleColor" red="0.19607843459999999" green="0.30980393290000002" blue="0.52156865600000002" alpha="1" colorSpace="calibratedRGB"/>
<color key="titleShadowColor" white="0.5" alpha="1" colorSpace="calibratedWhite"/>
</state>
<state key="highlighted">
<color key="titleColor" white="1" alpha="1" colorSpace="calibratedWhite"/>
</state>
<connections>
<action selector="startTest:" destination="2" eventType="touchUpInside" id="f4X-jg-ZsU"/>
</connections>
</button>
<textField opaque="NO" clipsSubviews="YES" contentMode="scaleToFill" contentHorizontalAlignment="left" contentVerticalAlignment="center" borderStyle="roundedRect" placeholder="Random seed" minimumFontSize="17" id="TZ8-OW-wjf">
<rect key="frame" x="27" y="28" width="197" height="31"/>
<autoresizingMask key="autoresizingMask" flexibleMaxX="YES" flexibleMaxY="YES"/>
<fontDescription key="fontDescription" type="system" pointSize="14"/>
<textInputTraits key="textInputTraits" autocorrectionType="no"/>
<connections>
<outlet property="delegate" destination="2" id="hjY-Ym-Fcw"/>
</connections>
</textField>
<textView clipsSubviews="YES" multipleTouchEnabled="YES" contentMode="scaleToFill" showsHorizontalScrollIndicator="NO" editable="NO" id="LHz-h6-ZBC">
<rect key="frame" x="27" y="88" width="721" height="887"/>
<autoresizingMask key="autoresizingMask" flexibleMaxX="YES" flexibleMaxY="YES"/>
<color key="backgroundColor" red="1" green="1" blue="1" alpha="1" colorSpace="calibratedRGB"/>
<fontDescription key="fontDescription" type="system" pointSize="14"/>
<textInputTraits key="textInputTraits" autocapitalizationType="sentences"/>
<connections>
<outlet property="delegate" destination="2" id="fwY-fT-bCV"/>
</connections>
</textView>
</subviews>
<color key="backgroundColor" red="0.13337372065218178" green="0.1801924475036723" blue="0.21739130434782605" alpha="1" colorSpace="calibratedRGB"/>
</view>
<connections>
<outlet property="theSeed" destination="TZ8-OW-wjf" id="QuA-uT-5IR"/>
<outlet property="theView" destination="LHz-h6-ZBC" id="s64-32-fBA"/>
</connections>
</viewController>
</objects>
<point key="canvasLocation" x="-601" y="-1021"/>
</scene>
</scenes>
<classes>
<class className="Xd3iOSViewController" superclassName="UIViewController">
<source key="sourceIdentifier" type="project" relativePath="./Classes/Xd3iOSViewController.h"/>
<relationships>
<relationship kind="action" name="startTest:"/>
<relationship kind="outlet" name="theSeed" candidateClass="UITextField"/>
<relationship kind="outlet" name="theView" candidateClass="UITextView"/>
</relationships>
</class>
</classes>
<simulatedMetricsContainer key="defaultSimulatedMetrics">
<simulatedStatusBarMetrics key="statusBar" statusBarStyle="blackTranslucent"/>
<simulatedOrientationMetrics key="orientation"/>
<simulatedScreenMetrics key="destination"/>
</simulatedMetricsContainer>
</document>

27
lib/xdelta3/examples/iOS/xdelta3-ios-test/xdelta3-ios-test/en.lproj/MainStoryboard_iPhone.storyboard Normal file
View file

@ -0,0 +1,27 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="1.0" toolsVersion="1906" systemVersion="11A511" targetRuntime="iOS.CocoaTouch" nextObjectID="6" propertyAccessControl="none" initialViewController="2">
<dependencies>
<development defaultVersion="4200" identifier="xcode"/>
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="902"/>
</dependencies>
<scenes>
<scene sceneID="5">
<objects>
<placeholder placeholderIdentifier="IBFirstResponder" id="4" sceneMemberID="firstResponder"/>
<viewController id="2" customClass="Xd3iOSViewController" sceneMemberID="viewController">
<view key="view" contentMode="scaleToFill" id="3">
<rect key="frame" x="0.0" y="20" width="320" height="460"/>
<autoresizingMask key="autoresizingMask" flexibleMaxX="YES" flexibleMaxY="YES"/>
<subviews/>
<color key="backgroundColor" white="1" alpha="1" colorSpace="custom" customColorSpace="calibratedWhite"/>
</view>
</viewController>
</objects>
</scene>
</scenes>
<simulatedMetricsContainer key="defaultSimulatedMetrics">
<simulatedStatusBarMetrics key="statusBar"/>
<simulatedOrientationMetrics key="orientation"/>
<simulatedScreenMetrics key="destination"/>
</simulatedMetricsContainer>
</document>

1378
lib/xdelta3/examples/iOS/xdelta3-ios-test/xdelta3-ios-test/file_v1.bin Normal file
View file

File diff suppressed because it is too large Load diff

BIN
lib/xdelta3/examples/iOS/xdelta3-ios-test/xdelta3-ios-test/file_v1_to_v2.bin Normal file
View file

Binary file not shown.

5419
lib/xdelta3/examples/iOS/xdelta3-ios-test/xdelta3-ios-test/file_v2.bin Normal file
View file

File diff suppressed because it is too large Load diff

25
lib/xdelta3/examples/iOS/xdelta3-ios-test/xdelta3-ios-test/main.m Normal file
View file

@ -0,0 +1,25 @@
/* xdelta3 - delta compression tools and library -*- Mode: objc *-*
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#import <UIKit/UIKit.h>
#import "Xd3iOSAppDelegate.h"
int main(int argc, char *argv[])
{
@autoreleasepool {
return UIApplicationMain(argc, argv, nil, NSStringFromClass([Xd3iOSAppDelegate class]));
}
}

52
lib/xdelta3/examples/iOS/xdelta3-ios-test/xdelta3-ios-test/xdelta3-ios-test-Info.plist Normal file
View file

@ -0,0 +1,52 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>CFBundleDevelopmentRegion</key>
<string>en</string>
<key>CFBundleDisplayName</key>
<string>${PRODUCT_NAME}</string>
<key>CFBundleExecutable</key>
<string>${EXECUTABLE_NAME}</string>
<key>CFBundleIconFiles</key>
<array/>
<key>CFBundleIdentifier</key>
<string>Joshua-MacDonald.${PRODUCT_NAME:rfc1034identifier}</string>
<key>CFBundleInfoDictionaryVersion</key>
<string>6.0</string>
<key>CFBundleName</key>
<string>${PRODUCT_NAME}</string>
<key>CFBundlePackageType</key>
<string>APPL</string>
<key>CFBundleShortVersionString</key>
<string>1.0</string>
<key>CFBundleSignature</key>
<string>????</string>
<key>CFBundleVersion</key>
<string>1.0</string>
<key>LSRequiresIPhoneOS</key>
<true/>
<key>UIMainStoryboardFile</key>
<string>MainStoryboard_iPhone</string>
<key>UIMainStoryboardFile~ipad</key>
<string>MainStoryboard_iPad</string>
<key>UIRequiredDeviceCapabilities</key>
<array>
<string>armv7</string>
</array>
<key>UISupportedInterfaceOrientations</key>
<array>
<string>UIInterfaceOrientationPortrait</string>
<string>UIInterfaceOrientationLandscapeLeft</string>
<string>UIInterfaceOrientationLandscapeRight</string>
<string>UIInterfaceOrientationPortraitUpsideDown</string>
</array>
<key>UISupportedInterfaceOrientations~ipad</key>
<array>
<string>UIInterfaceOrientationPortrait</string>
<string>UIInterfaceOrientationPortraitUpsideDown</string>
<string>UIInterfaceOrientationLandscapeLeft</string>
<string>UIInterfaceOrientationLandscapeRight</string>
</array>
</dict>
</plist>

14
lib/xdelta3/examples/iOS/xdelta3-ios-test/xdelta3-ios-test/xdelta3-ios-test-Prefix.pch Normal file
View file

@ -0,0 +1,14 @@
//
// Prefix header for all source files of the 'xdelta3-ios-test' target in the 'xdelta3-ios-test' project
//
#import <Availability.h>
#ifndef __IPHONE_5_0
#warning "This project uses features only available in iOS SDK 5.0 and later."
#endif
#ifdef __OBJC__
#import <UIKit/UIKit.h>
#import <Foundation/Foundation.h>
#endif

215
lib/xdelta3/examples/small_page_test.c Normal file
View file

@ -0,0 +1,215 @@
/* xdelta3 - delta compression tools and library
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#include <stdio.h>
#define PAGE_SIZE 4096
#define SPACE_MAX 131072 // how much memory per process
#define OUTPUT_MAX 1024 // max size for output
#define XD3_ALLOCSIZE 256 // internal size for various buffers
#define IOPT_SIZE 128 // instruction buffer
// SPACE_MAX of 32K is sufficient for most inputs with XD3_COMPLEVEL_1
// XD3_COMPLEVEL_9 requires about 4x more space than XD3_COMPLEVEL_1
#include "xdelta3.h"
#include "xdelta3.c"
typedef struct _context {
uint8_t *buffer;
int allocated;
} context_t;
static int max_allocated = 0;
void*
process_alloc (void* opaque, usize_t items, usize_t size)
{
context_t *ctx = (context_t*) opaque;
usize_t t = items * size;
void *ret;
if (ctx->allocated + t > SPACE_MAX)
{
return NULL;
}
ret = ctx->buffer + ctx->allocated;
ctx->allocated += t;
return ret;
}
void
process_free (void* opaque, void *ptr)
{
}
int
process_page (int is_encode,
int (*func) (xd3_stream *),
const uint8_t *input,
usize_t input_size,
const uint8_t *source,
uint8_t *output,
usize_t *output_size,
usize_t output_size_max,
int flags) {
/* On my x86 this is 1072 of objects on the stack */
xd3_stream stream;
xd3_config config;
xd3_source src;
context_t *ctx = calloc(SPACE_MAX, 1);
int ret;
memset (&config, 0, sizeof(config));
if (ctx == NULL)
{
printf("calloc failed\n");
return -1;
}
ctx->buffer = (uint8_t*)ctx;
ctx->allocated = sizeof(*ctx);
config.flags = flags;
config.winsize = PAGE_SIZE;
config.sprevsz = PAGE_SIZE;
config.srcwin_maxsz = PAGE_SIZE;
config.iopt_size = IOPT_SIZE;
config.alloc = &process_alloc;
config.freef = &process_free;
config.opaque = (void*) ctx;
src.blksize = PAGE_SIZE;
src.onblk = PAGE_SIZE;
src.curblk = source;
src.curblkno = 0;
if ((ret = xd3_config_stream (&stream, &config)) != 0 ||
(ret = xd3_set_source_and_size (&stream, &src, PAGE_SIZE)) != 0 ||
(ret = xd3_process_stream (is_encode,
&stream,
func, 1,
input, input_size,
output, output_size,
output_size_max)) != 0)
{
if (stream.msg != NULL)
{
fprintf(stderr, "stream message: %s\n", stream.msg);
}
}
xd3_free_stream (&stream);
if (max_allocated < ctx->allocated)
{
max_allocated = ctx->allocated;
fprintf(stderr, "max allocated %d\n", max_allocated);
}
free(ctx);
return ret;
}
int test(int stride, int encode_flags)
{
uint8_t frompg[PAGE_SIZE];
uint8_t topg[PAGE_SIZE];
uint8_t output[OUTPUT_MAX];
uint8_t reout[PAGE_SIZE];
usize_t output_size;
usize_t re_size;
int i, j, ret;
for (i = 0; i < PAGE_SIZE; i++)
{
topg[i] = frompg[i] = (rand() >> 3 ^ rand() >> 6 ^ rand() >> 9);
}
// change 1 byte every stride
if (stride > 0)
{
for (j = stride; j <= PAGE_SIZE; j += stride)
{
topg[j - 1] ^= 0xff;
}
}
if ((ret = process_page (1, xd3_encode_input,
topg, PAGE_SIZE,
frompg, output,
&output_size, OUTPUT_MAX,
encode_flags)) != 0)
{
fprintf (stderr, "encode failed: stride %u flags 0x%x\n",
stride, encode_flags);
return ret;
}
if ((ret = process_page (0, xd3_decode_input,
output, output_size,
frompg, reout,
&re_size, PAGE_SIZE,
0)) != 0)
{
fprintf (stderr, "decode failed: stride %u output_size %u flags 0x%x\n",
stride, output_size, encode_flags);
return ret;
}
if (output_size > OUTPUT_MAX || re_size != PAGE_SIZE)
{
fprintf (stderr, "internal error: %u != %u\n", output_size, re_size);
return -1;
}
for (i = 0; i < PAGE_SIZE; i++)
{
if (reout[i] != topg[i])
{
fprintf (stderr, "encode-decode error: position %d\n", i);
return -1;
}
}
fprintf(stderr, "stride %d flags 0x%x size %u ",
stride, encode_flags, output_size);
fprintf(stderr, "%s\n", (ret == 0) ? "OK" : "FAIL");
return 0;
}
int main()
{
int stride;
int level;
for (level = 1; level < 10; level = (level == 1 ? 3 : level + 3))
{
int lflag = level << XD3_COMPLEVEL_SHIFT;
for (stride = 2; stride <= PAGE_SIZE; stride += 2)
{
test(stride, lflag);
test(stride, lflag | XD3_SEC_DJW);
}
}
return 0;
}

87
lib/xdelta3/examples/speed_test.c Normal file
View file

@ -0,0 +1,87 @@
/* xdelta3 - delta compression tools and library
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#include "test.h"
usize_t bench_speed(const uint8_t *from_buf, const size_t from_len,
const uint8_t *to_buf, const size_t to_len,
uint8_t *delta_buf, const size_t delta_alloc,
int flags) {
usize_t delta_size;
int ret = xd3_encode_memory(to_buf, to_len, from_buf, from_len,
delta_buf, &delta_size, delta_alloc, flags);
if (ret != 0) {
fprintf(stderr, "encode failure: %d: %s\n", ret, xd3_strerror(ret));
abort();
}
return delta_size;
}
int main(int argc, char **argv) {
int repeat, level;
char *from, *to;
uint8_t *from_buf = NULL, *to_buf = NULL, *delta_buf = NULL;
size_t from_len = 0, to_len, delta_alloc, delta_size = 0;
long start, finish;
int i, ret;
int flags;
if (argc != 5) {
fprintf(stderr, "usage: speed_test LEVEL COUNT FROM TO\n");
return 1;
}
level = atoi(argv[1]);
repeat = atoi(argv[2]);
from = argv[3];
to = argv[4];
flags = (level << XD3_COMPLEVEL_SHIFT) & XD3_COMPLEVEL_MASK;
if ((strcmp(from, "null") != 0 &&
(ret = read_whole_file(from, &from_buf, &from_len))) ||
(ret = read_whole_file(to, &to_buf, &to_len))) {
fprintf(stderr, "read_whole_file error\n");
goto exit;
}
delta_alloc = to_len * 11 / 10;
delta_buf = main_malloc(delta_alloc);
start = get_millisecs_now();
for (i = 0; i < repeat; ++i) {
delta_size = bench_speed(from_buf, from_len,
to_buf, to_len, delta_buf, delta_alloc, flags);
}
finish = get_millisecs_now();
fprintf(stderr,
"STAT: encode %3.2f ms from %s to %s repeat %d %zdbit delta %zd\n",
(double)(finish - start) / repeat, from, to, repeat, sizeof (xoff_t) * 8, delta_size);
ret = 0;
if (0) {
exit:
ret = 1;
}
main_free(to_buf);
main_free(from_buf);
main_free(delta_buf);
return ret;
}

56
lib/xdelta3/examples/test.h Normal file
View file

@ -0,0 +1,56 @@
/* xdelta3 - delta compression tools and library
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#define NOT_MAIN 1
#include "xdelta3.h"
#include "xdelta3.c"
static int read_whole_file(const char *name,
uint8_t **buf_ptr,
size_t *buf_len) {
main_file file;
int ret;
xoff_t len;
usize_t nread;
main_file_init(&file);
file.filename = name;
ret = main_file_open(&file, name, XO_READ);
if (ret != 0) {
fprintf(stderr, "open failed\n");
goto exit;
}
ret = main_file_stat(&file, &len);
if (ret != 0) {
fprintf(stderr, "stat failed\n");
goto exit;
}
(*buf_len) = (size_t)len;
(*buf_ptr) = (uint8_t*) main_malloc(*buf_len);
ret = main_file_read(&file, *buf_ptr, *buf_len, &nread,
"read failed");
if (ret == 0 && *buf_len == nread) {
ret = 0;
} else {
fprintf(stderr, "invalid read\n");
ret = XD3_INTERNAL;
}
exit:
main_file_cleanup(&file);
return ret;
}

8
lib/xdelta3/generate_build_files.sh Normal file
View file

@ -0,0 +1,8 @@
#!/bin/sh
aclocal &&
autoreconf --install &&
libtoolize &&
autoconf &&
automake --add-missing &&
automake

274
lib/xdelta3/go/src/regtest.go Normal file
View file

@ -0,0 +1,274 @@
package main
import (
"fmt"
"io"
"path"
"os"
"sort"
"time"
"xdelta"
)
const (
xdataset = "/volume/home/jmacd/src/testdata"
xcompare = "/volume/home/jmacd/src/xdelta-devel/xdelta3/build/x86_64-pc-linux-gnu-m64/xoff64/xdelta3"
xdelta3 = "/volume/home/jmacd/src/xdelta-64bithash/xdelta3/build/x86_64-pc-linux-gnu-m64/usize64/xoff64/xdelta3"
seed = 1422253499919909358
)
type Config struct {
srcbuf_size int64
window_size int64
blocksize int
}
func NewC() Config {
// TODO make these (and above) flags
return Config{1<<26, 1<<22, 1<<16}
}
func (c Config) smokeTest(t *xdelta.TestGroup, p xdelta.Program) {
target := "Hello world!"
source := "Hello world, nice to meet you!"
enc, err := t.Exec("encode", p, true, []string{"-e"})
if err != nil {
t.Panic(err)
}
dec, err := t.Exec("decode", p, true, []string{"-d"})
if err != nil {
t.Panic(err)
}
encodeout := t.Drain(enc.Stdout, "encode.stdout")
decodeout := t.Drain(dec.Stdout, "decode.stdout")
t.Empty(enc.Stderr, "encode")
t.Empty(dec.Stderr, "decode")
t.TestWrite("encode.stdin", enc.Stdin, []byte(target))
t.TestWrite("encode.srcin", enc.Srcin, []byte(source))
t.TestWrite("decode.stdin", dec.Stdin, <-encodeout)
t.TestWrite("decode.srcin", dec.Srcin, []byte(source))
if do := string(<-decodeout); do != target {
t.Panic(fmt.Errorf("It's not working! %s\n!=\n%s\n", do, target))
}
t.Wait(enc, dec)
}
type PairTest struct {
// Input
Config
program xdelta.Program
source, target string
// Output
TestOutput
}
type TestOutput struct {
encoded int64
encDuration time.Duration
decDuration time.Duration
encSysDuration time.Duration
decSysDuration time.Duration
}
func (to *TestOutput) Add(a TestOutput) {
to.encoded += a.encoded
to.encDuration += a.encDuration
to.decDuration += a.decDuration
to.encSysDuration += a.encSysDuration
to.decSysDuration += a.decSysDuration
}
func (to *TestOutput) String() string {
return fmt.Sprintf("SIZE: %v\tT: %v\tTSYS: %v\tDT: %v\tDTSYS: %v",
to.encoded, to.encDuration, to.encSysDuration, to.decDuration, to.encSysDuration)
}
// P is the test program, Q is the reference version.
func (cfg Config) datasetTest(t *xdelta.TestGroup, p, q xdelta.Program) {
dir, err := os.Open(xdataset)
if err != nil {
t.Panic(err)
}
dents, err := dir.Readdir(-1)
if err != nil {
t.Panic(err)
}
paths := make([]string, len(dents))
var total int64
for i, d := range dents {
if !d.Mode().IsRegular() {
continue
}
paths[i] = fmt.Sprint(xdataset, "/", d.Name())
total += d.Size()
}
meansize := total / int64(len(dents))
largest := uint(20)
for ; largest <= 31 && 1<<largest < meansize; largest++ {}
sort.Strings(paths)
testSum := map[uint]*TestOutput{}
compSum := map[uint]*TestOutput{}
for _, in1 := range paths {
for _, in2 := range paths {
if in1 == in2 { continue }
// 1/4, 1/2, and 1 of the power-of-2 rounded-up mean size
for b := largest - 2; b <= largest; b++ {
if _, has := testSum[b]; !has {
testSum[b] = &TestOutput{}
compSum[b] = &TestOutput{}
}
c1 := cfg
c1.srcbuf_size = 1<<b
ptest := &PairTest{c1, p, in1, in2, TestOutput{-1, 0, 0, 0, 0}}
ptest.datasetPairTest(t, 1<<b);
qtest := &PairTest{c1, q, in1, in2, TestOutput{-1, 0, 0, 0, 0}}
qtest.datasetPairTest(t, 1<<b)
testSum[b].Add(ptest.TestOutput)
compSum[b].Add(qtest.TestOutput)
fmt.Printf("%s, %s: %.2f%% %+d/%d\n\tE:%.2f%%/%s(%.2f%%/%s) D:%.2f%%/%s(%.2f%%/%s) [B=%d]\n",
path.Base(in1), path.Base(in2),
float64(ptest.encoded - qtest.encoded) * 100.0 / float64(qtest.encoded),
ptest.encoded - qtest.encoded,
qtest.encoded,
(ptest.encDuration - qtest.encDuration).Seconds() * 100.0 / qtest.encDuration.Seconds(),
qtest.encDuration,
(ptest.decDuration - qtest.decDuration).Seconds() * 100.0 / qtest.decDuration.Seconds(),
qtest.encDuration,
(ptest.encSysDuration - qtest.encSysDuration).Seconds() * 100.0 / qtest.encSysDuration.Seconds(),
qtest.encSysDuration,
(ptest.decSysDuration - qtest.decSysDuration).Seconds() * 100.0 / qtest.decSysDuration.Seconds(),
qtest.decSysDuration,
1<<b)
}
}
}
var keys []uint
for k, _ := range testSum {
keys = append(keys, k)
}
for _, k := range keys {
fmt.Printf("B=%v\nTEST: %v\nCOMP: %v\n", 1<<k, testSum[k], compSum[k])
}
}
func (pt *PairTest) datasetPairTest(t *xdelta.TestGroup, meanSize int64) {
cfg := pt.Config
eargs := []string{"-e", fmt.Sprint("-B", cfg.srcbuf_size), // "-q",
fmt.Sprint("-W", cfg.window_size), "-s", pt.source,
"-I0", "-S", "none", pt.target}
enc, err := t.Exec("encode", pt.program, false, eargs)
if err != nil {
t.Panic(err)
}
dargs := []string{"-dc", fmt.Sprint("-B", cfg.srcbuf_size), //"-q",
fmt.Sprint("-W", cfg.window_size), "-s", pt.source,
"-S", "none"}
dec, err := t.Exec("decode", pt.program, false, dargs)
if err != nil {
t.Panic(err)
}
tgt_check, err := os.Open(pt.target)
if err != nil {
t.Panic(err)
}
tgt_info, err := tgt_check.Stat()
if err != nil {
t.Panic(err)
}
t.Empty(enc.Stderr, "encode")
t.Empty(dec.Stderr, "decode")
t.CopyStreams(enc.Stdout, dec.Stdin, &pt.encoded)
t.CompareStreams(dec.Stdout, tgt_check, tgt_info.Size())
t.Wait(enc, dec)
pt.decDuration = dec.Cmd.ProcessState.UserTime()
pt.encDuration = enc.Cmd.ProcessState.UserTime()
pt.decSysDuration = dec.Cmd.ProcessState.SystemTime()
pt.encSysDuration = enc.Cmd.ProcessState.SystemTime()
}
func (cfg Config) offsetTest(t *xdelta.TestGroup, p xdelta.Program, offset, length int64) {
eargs := []string{"-e", "-0", fmt.Sprint("-B", cfg.srcbuf_size), "-q",
fmt.Sprint("-W", cfg.window_size)}
enc, err := t.Exec("encode", p, true, eargs)
if err != nil {
t.Panic(err)
}
dargs := []string{"-d", fmt.Sprint("-B", cfg.srcbuf_size), "-q",
fmt.Sprint("-W", cfg.window_size)}
dec, err := t.Exec("decode", p, true, dargs)
if err != nil {
t.Panic(err)
}
// The pipe used to read the decoder output and compare
// against the target.
read, write := io.Pipe()
t.Empty(enc.Stderr, "encode")
t.Empty(dec.Stderr, "decode")
var encoded_size int64
t.CopyStreams(enc.Stdout, dec.Stdin, &encoded_size)
t.CompareStreams(dec.Stdout, read, length)
// The decoder output ("read", above) is compared with the
// test-provided output ("write", below). The following
// generates two identical inputs.
t.WriteRstreams("encode", seed, offset, length, enc.Srcin, enc.Stdin)
t.WriteRstreams("decode", seed, offset, length, dec.Srcin, write)
t.Wait(enc, dec)
expect := cfg.srcbuf_size - offset
if float64(encoded_size) < (0.95 * float64(expect)) ||
float64(encoded_size) > (1.05 * float64(expect)) {
t.Fail("encoded size should be ~=", expect, ", actual ", encoded_size)
}
}
func main() {
r, err := xdelta.NewRunner()
if err != nil {
panic(err)
}
defer r.Cleanup()
cfg := NewC()
prog := xdelta.Program{xdelta3}
r.RunTest("smoketest", func(t *xdelta.TestGroup) { cfg.smokeTest(t, prog) })
for i := uint(29); i <= 33; i += 1 {
// The arguments to offsetTest are offset, source
// window size, and file size. The source window size
// is (2 << i) and (in the 3.0x release branch) is
// limited to 2^31, so the the greatest value of i is
// 30.
cfg.srcbuf_size = 2 << i
r.RunTest(fmt.Sprint("offset", i), func(t *xdelta.TestGroup) {
cfg.offsetTest(t, prog, 1 << i, 3 << i) })
}
comp := xdelta.Program{xcompare}
r.RunTest("dataset", func(t *xdelta.TestGroup) { cfg.datasetTest(t, prog, comp) })
}

71
lib/xdelta3/go/src/xdelta/rstream.go Normal file
View file

@ -0,0 +1,71 @@
package xdelta
import (
"io"
"math/rand"
)
const (
blocksize = 1<<17
)
func (t *TestGroup) WriteRstreams(desc string, seed, offset, len int64,
src, tgt io.WriteCloser) {
t.Go("src-write:"+desc, func (g *Goroutine) {
writeOne(g, seed, 0, len, tgt, false)
})
t.Go("tgt-write:"+desc, func (g *Goroutine) {
writeOne(g, seed, offset, len, src, true)
})
}
func writeOne(g *Goroutine, seed, offset, len int64, stream io.WriteCloser, readall bool) {
if !readall {
// Allow the source-read to fail or block until the process terminates.
// This behavior is reserved for the decoder, which is not required to
// read the entire source.
g.OK()
}
if offset != 0 {
// Fill with other random data until the offset
if err := writeRand(g, rand.New(rand.NewSource(^seed)), offset, stream); err != nil {
g.Panic(err)
}
}
if err := writeRand(g, rand.New(rand.NewSource(seed)),
len - offset, stream); err != nil {
g.Panic(err)
}
if err := stream.Close(); err != nil {
g.Panic(err)
}
g.OK()
}
func writeRand(g *Goroutine, r *rand.Rand, len int64, s io.Writer) error {
blk := make([]byte, blocksize)
for len > 0 {
fillRand(r, blk)
c := blocksize
if len < blocksize {
c = int(len)
}
if _, err := s.Write(blk[0:c]); err != nil {
return err
}
len -= int64(c)
}
return nil
}
func fillRand(r *rand.Rand, blk []byte) {
for p := 0; p < len(blk); {
v := r.Int63()
for i := 7; i != 0 && p < len(blk); i-- {
blk[p] = byte(v)
p++
v >>= 8
}
}
}

71
lib/xdelta3/go/src/xdelta/run.go Normal file
View file

@ -0,0 +1,71 @@
package xdelta
import (
"fmt"
"io"
"io/ioutil"
"os"
"os/exec"
)
type Program struct {
Path string
}
type Run struct {
Cmd exec.Cmd
Srcfile string
Stdin io.WriteCloser
Srcin io.WriteCloser
Stdout io.ReadCloser
Stderr io.ReadCloser
}
type Runner struct {
Testdir string
}
func (r *Run) Wait() error {
return r.Cmd.Wait()
}
func NewRunner() (*Runner, error) {
if dir, err := ioutil.TempDir(tmpDir, "xrt"); err != nil {
return nil, err
} else {
return &Runner{dir}, nil
}
}
func (r *Runner) newTestGroup(name string) (*TestGroup) {
tg := &TestGroup{Runner: r}
tg.WaitGroup.Add(1)
g0 := &Goroutine{tg, name, false}
tg.running = append(tg.running, g0)
tg.main = g0
return tg
}
func (r *Runner) Cleanup() {
os.RemoveAll(r.Testdir)
}
func (r *Runner) RunTest(name string, f func (t *TestGroup)) {
t := r.newTestGroup(name)
c := make(chan interface{})
go func() {
defer func() {
rec := recover()
c <- rec
}()
fmt.Println("Testing", name, "...")
f(t)
c <- nil
}()
rec := <- c
if t.errors == nil && rec == nil {
fmt.Println("Success:", name)
} else {
fmt.Println("FAILED:", name, t.errors, rec)
}
}

164
lib/xdelta3/go/src/xdelta/test.go Normal file
View file

@ -0,0 +1,164 @@
package xdelta
import (
"bufio"
"bytes"
"errors"
"fmt"
"io"
"io/ioutil"
"os"
"path"
"sync/atomic"
"golang.org/x/sys/unix"
)
var (
tmpDir = "/tmp"
srcSeq int64
)
func (t *TestGroup) Drain(f io.ReadCloser, desc string) <-chan []byte {
c := make(chan []byte)
t.Go(desc, func(g *Goroutine) {
if b, err := ioutil.ReadAll(f); err != nil {
g.Panic(err)
} else {
c <- b
}
g.OK()
})
return c
}
func (t *TestGroup) Empty(f io.ReadCloser, desc string) *Goroutine {
return t.Go("empty:"+desc, func (g *Goroutine) {
s := bufio.NewScanner(f)
for s.Scan() {
os.Stderr.Write([]byte(fmt.Sprint(desc, ": ", s.Text(), "\n")))
}
err := s.Err()
f.Close()
if err != nil {
g.Panic(err)
}
g.OK()
})
}
func (t *TestGroup) TestWrite(what string, f io.WriteCloser, b []byte) *Goroutine {
return t.Go("write", func(g *Goroutine) {
if _, err := f.Write(b); err != nil {
g.Panic(err)
}
if err := f.Close(); err != nil {
g.Panic(err)
}
g.OK()
})
}
func (t *TestGroup) CopyStreams(r io.ReadCloser, w io.WriteCloser, written *int64) *Goroutine {
return t.Go("copy", func(g *Goroutine) {
nwrite, err := io.Copy(w, r)
if err != nil {
g.Panic(err)
}
err = r.Close()
if err != nil {
g.Panic(err)
}
err = w.Close()
if err != nil {
g.Panic(err)
}
g.OK()
*written = nwrite
})
}
func (t *TestGroup) CompareStreams(r1 io.ReadCloser, r2 io.ReadCloser, length int64) *Goroutine {
return t.Go("compare", func(g *Goroutine) {
b1 := make([]byte, blocksize)
b2 := make([]byte, blocksize)
var idx int64
for length > 0 {
c := blocksize
if length < blocksize {
c = int(length)
}
if _, err := io.ReadFull(r1, b1[0:c]); err != nil {
g.Panic(err)
}
if _, err := io.ReadFull(r2, b2[0:c]); err != nil {
g.Panic(err)
}
if bytes.Compare(b1[0:c], b2[0:c]) != 0 {
fmt.Println("B1 is", string(b1[0:c]))
fmt.Println("B2 is", string(b2[0:c]))
g.Panic(errors.New(fmt.Sprint("Bytes do not compare at ", idx)))
}
length -= int64(c)
idx += int64(c)
}
g.OK()
})
}
func (t *TestGroup) Exec(desc string, p Program, srcfifo bool, flags []string) (*Run, error) {
var err error
run := &Run{}
args := []string{p.Path}
if srcfifo {
num := atomic.AddInt64(&srcSeq, 1)
run.Srcfile = path.Join(t.Runner.Testdir, fmt.Sprint("source", num))
if err = unix.Mkfifo(run.Srcfile, 0600); err != nil {
return nil, err
}
read, write := io.Pipe()
t.writeFifo(run.Srcfile, read)
run.Srcin = write
args = append(args, "-s")
args = append(args, run.Srcfile)
}
if run.Stdin, err = run.Cmd.StdinPipe(); err != nil {
return nil, err
}
if run.Stdout, err = run.Cmd.StdoutPipe(); err != nil {
return nil, err
}
if run.Stderr, err = run.Cmd.StderrPipe(); err != nil {
return nil, err
}
run.Cmd.Path = p.Path
run.Cmd.Args = append(args, flags...)
run.Cmd.Dir = t.Runner.Testdir
if serr := run.Cmd.Start(); serr != nil {
return nil, serr
}
return run, nil
}
func (t *TestGroup) Fail(v ...interface{}) {
panic(fmt.Sprintln(v...))
}
func (t *TestGroup) writeFifo(srcfile string, read io.Reader) *Goroutine {
return t.Go("compare", func(g *Goroutine) {
fifo, err := os.OpenFile(srcfile, os.O_WRONLY, 0600)
if err != nil {
fifo.Close()
g.Panic(err)
}
if _, err := io.Copy(fifo, read); err != nil {
fifo.Close()
g.Panic(err)
}
if err := fifo.Close(); err != nil {
g.Panic(err)
}
g.OK()
})
}

97
lib/xdelta3/go/src/xdelta/tgroup.go Normal file
View file

@ -0,0 +1,97 @@
package xdelta
import (
"fmt"
"runtime"
"sync"
)
type TestGroup struct {
*Runner
main *Goroutine
sync.Mutex
sync.WaitGroup
running []*Goroutine
errors []error
nonerrors []error // For tolerated / expected conditions
}
type Goroutine struct {
*TestGroup
name string
done bool
}
func (g *Goroutine) String() string {
return fmt.Sprint("[", g.name, "]")
}
func (g *Goroutine) finish(err error) {
wait := false
tg := g.TestGroup
sbuf := make([]byte, 4096)
sbuf = sbuf[0:runtime.Stack(sbuf, false)]
if err != nil {
err = fmt.Errorf("%v:%v:%v", g.name, err, string(sbuf))
}
tg.Lock()
if g.done {
if err != nil {
tg.nonerrors = append(tg.nonerrors, err)
}
} else {
wait = true
g.done = true
if err != nil {
tg.errors = append(tg.errors, err)
}
}
tg.Unlock()
if wait {
tg.WaitGroup.Done()
}
}
func (g *Goroutine) OK() {
g.finish(nil)
}
func (g *Goroutine) Panic(err error) {
g.finish(err)
if g != g.TestGroup.main {
runtime.Goexit()
}
}
func (t *TestGroup) Main() *Goroutine { return t.main }
func (t *TestGroup) Panic(err error) { t.Main().Panic(err) }
func (t *TestGroup) Go(name string, f func(*Goroutine)) *Goroutine {
g := &Goroutine{t, name, false}
t.Lock()
t.WaitGroup.Add(1)
t.running = append(t.running, g)
t.Unlock()
go f(g)
return g
}
func (t *TestGroup) Wait(procs... *Run) {
t.Main().OK()
t.WaitGroup.Wait()
for _, p := range procs {
if err := p.Wait(); err != nil {
t.errors = append(t.errors, err)
}
}
for _, err := range t.errors {
fmt.Println(":ERROR:", err)
}
for _, err := range t.nonerrors {
fmt.Println("(ERROR)", err)
}
if len(t.errors) != 0 {
t.Fail("Test failed with", len(t.errors), "errors")
}
}

42
lib/xdelta3/linkxd3lib.c Normal file
View file

@ -0,0 +1,42 @@
#include "xdelta3.h"
extern int VVV;
int VVV;
void use(int r)
{
VVV = r;
}
int main() {
xd3_config config;
xd3_stream stream;
xd3_source source;
xd3_init_config (& config, 0);
use (xd3_config_stream (&stream, &config));
use (xd3_close_stream (&stream));
xd3_abort_stream (&stream);
xd3_free_stream (&stream);
xd3_avail_input (& stream, NULL, 0);
xd3_consume_output (& stream);
use (xd3_set_source (& stream, & source));
xd3_set_flags (& stream, 0);
use (xd3_decode_stream (& stream, NULL, 0, NULL, NULL, 0));
use (xd3_decode_input (&stream));
use (xd3_get_appheader (& stream, NULL, NULL));
#if XD3_ENCODER
use (xd3_encode_input (&stream));
use (xd3_encode_stream (& stream, NULL, 0, NULL, NULL, 0));
use (xd3_set_appheader (& stream));
use (xd3_encoder_used_source (& stream));
use (xd3_encoder_srcbase (& stream));
use (xd3_encoder_srclen (& stream));
#endif
return 0;
}

84
lib/xdelta3/m4/ax_check_aligned_access_required.m4 Normal file
View file

@ -0,0 +1,84 @@
# ====================================================================================
# http://www.gnu.org/software/autoconf-archive/ax_check_aligned_access_required.html
# ====================================================================================
#
# SYNOPSIS
#
# AC_CHECK_ALIGNED_ACCESS_REQUIRED
#
# DESCRIPTION
#
# While the x86 CPUs allow access to memory objects to be unaligned it
# happens that most of the modern designs require objects to be aligned -
# or they will fail with a buserror. That mode is quite known by
# big-endian machines (sparc, etc) however the alpha cpu is little-
# endian.
#
# The following function will test for aligned access to be required and
# set a config.h define HAVE_ALIGNED_ACCESS_REQUIRED (name derived by
# standard usage). Structures loaded from a file (or mmapped to memory)
# should be accessed per-byte in that case to avoid segfault type errors.
#
# LICENSE
#
# Copyright (c) 2008 Guido U. Draheim <guidod@gmx.de>
#
# This program is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the
# Free Software Foundation; either version 3 of the License, or (at your
# option) any later version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
# Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
#
# As a special exception, the respective Autoconf Macro's copyright owner
# gives unlimited permission to copy, distribute and modify the configure
# scripts that are the output of Autoconf when processing the Macro. You
# need not follow the terms of the GNU General Public License when using
# or distributing such scripts, even though portions of the text of the
# Macro appear in them. The GNU General Public License (GPL) does govern
# all other use of the material that constitutes the Autoconf Macro.
#
# This special exception to the GPL applies to versions of the Autoconf
# Macro released by the Autoconf Archive. When you make and distribute a
# modified version of the Autoconf Macro, you may extend this special
# exception to the GPL to apply to your modified version as well.
#serial 7
AC_DEFUN([AX_CHECK_ALIGNED_ACCESS_REQUIRED],
[AC_CACHE_CHECK([if pointers to integers require aligned access],
[ax_cv_have_aligned_access_required],
[AC_TRY_RUN([
#include <stdio.h>
#include <stdlib.h>
int main()
{
char* string = malloc(40);
int i;
for (i=0; i < 40; i++) string[[i]] = i;
{
void* s = string;
int* p = s+1;
int* q = s+2;
if (*p == *q) { return 1; }
}
return 0;
}
],
[ax_cv_have_aligned_access_required=yes],
[ax_cv_have_aligned_access_required=no],
[ax_cv_have_aligned_access_required=no])
])
if test "$ax_cv_have_aligned_access_required" = yes ; then
AC_DEFINE([HAVE_ALIGNED_ACCESS_REQUIRED], [1],
[Define if pointers to integers require aligned access])
fi
])

135
lib/xdelta3/m4/ax_pkg_swig.m4 Normal file
View file

@ -0,0 +1,135 @@
# ===========================================================================
# http://www.gnu.org/software/autoconf-archive/ax_pkg_swig.html
# ===========================================================================
#
# SYNOPSIS
#
# AX_PKG_SWIG([major.minor.micro], [action-if-found], [action-if-not-found])
#
# DESCRIPTION
#
# This macro searches for a SWIG installation on your system. If found,
# then SWIG is AC_SUBST'd; if not found, then $SWIG is empty. If SWIG is
# found, then SWIG_LIB is set to the SWIG library path, and AC_SUBST'd.
#
# You can use the optional first argument to check if the version of the
# available SWIG is greater than or equal to the value of the argument. It
# should have the format: N[.N[.N]] (N is a number between 0 and 999. Only
# the first N is mandatory.) If the version argument is given (e.g.
# 1.3.17), AX_PKG_SWIG checks that the swig package is this version number
# or higher.
#
# As usual, action-if-found is executed if SWIG is found, otherwise
# action-if-not-found is executed.
#
# In configure.in, use as:
#
# AX_PKG_SWIG(1.3.17, [], [ AC_MSG_ERROR([SWIG is required to build..]) ])
# AX_SWIG_ENABLE_CXX
# AX_SWIG_MULTI_MODULE_SUPPORT
# AX_SWIG_PYTHON
#
# LICENSE
#
# Copyright (c) 2008 Sebastian Huber <sebastian-huber@web.de>
# Copyright (c) 2008 Alan W. Irwin <irwin@beluga.phys.uvic.ca>
# Copyright (c) 2008 Rafael Laboissiere <rafael@laboissiere.net>
# Copyright (c) 2008 Andrew Collier <colliera@ukzn.ac.za>
# Copyright (c) 2011 Murray Cumming <murrayc@openismus.com>
#
# This program is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the
# Free Software Foundation; either version 2 of the License, or (at your
# option) any later version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
# Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
#
# As a special exception, the respective Autoconf Macro's copyright owner
# gives unlimited permission to copy, distribute and modify the configure
# scripts that are the output of Autoconf when processing the Macro. You
# need not follow the terms of the GNU General Public License when using
# or distributing such scripts, even though portions of the text of the
# Macro appear in them. The GNU General Public License (GPL) does govern
# all other use of the material that constitutes the Autoconf Macro.
#
# This special exception to the GPL applies to versions of the Autoconf
# Macro released by the Autoconf Archive. When you make and distribute a
# modified version of the Autoconf Macro, you may extend this special
# exception to the GPL to apply to your modified version as well.
#serial 8
AC_DEFUN([AX_PKG_SWIG],[
# Ubuntu has swig 2.0 as /usr/bin/swig2.0
AC_PATH_PROGS([SWIG],[swig swig2.0])
if test -z "$SWIG" ; then
m4_ifval([$3],[$3],[:])
elif test -n "$1" ; then
AC_MSG_CHECKING([SWIG version])
[swig_version=`$SWIG -version 2>&1 | grep 'SWIG Version' | sed 's/.*\([0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\).*/\1/g'`]
AC_MSG_RESULT([$swig_version])
if test -n "$swig_version" ; then
# Calculate the required version number components
[required=$1]
[required_major=`echo $required | sed 's/[^0-9].*//'`]
if test -z "$required_major" ; then
[required_major=0]
fi
[required=`echo $required | sed 's/[0-9]*[^0-9]//'`]
[required_minor=`echo $required | sed 's/[^0-9].*//'`]
if test -z "$required_minor" ; then
[required_minor=0]
fi
[required=`echo $required | sed 's/[0-9]*[^0-9]//'`]
[required_patch=`echo $required | sed 's/[^0-9].*//'`]
if test -z "$required_patch" ; then
[required_patch=0]
fi
# Calculate the available version number components
[available=$swig_version]
[available_major=`echo $available | sed 's/[^0-9].*//'`]
if test -z "$available_major" ; then
[available_major=0]
fi
[available=`echo $available | sed 's/[0-9]*[^0-9]//'`]
[available_minor=`echo $available | sed 's/[^0-9].*//'`]
if test -z "$available_minor" ; then
[available_minor=0]
fi
[available=`echo $available | sed 's/[0-9]*[^0-9]//'`]
[available_patch=`echo $available | sed 's/[^0-9].*//'`]
if test -z "$available_patch" ; then
[available_patch=0]
fi
# Convert the version tuple into a single number for easier comparison.
# Using base 100 should be safe since SWIG internally uses BCD values
# to encode its version number.
required_swig_vernum=`expr $required_major \* 10000 \
\+ $required_minor \* 100 \+ $required_patch`
available_swig_vernum=`expr $available_major \* 10000 \
\+ $available_minor \* 100 \+ $available_patch`
if test $available_swig_vernum -lt $required_swig_vernum; then
AC_MSG_WARN([SWIG version >= $1 is required. You have $swig_version.])
SWIG=''
m4_ifval([$3],[$3],[])
else
AC_MSG_CHECKING([for SWIG library])
SWIG_LIB=`$SWIG -swiglib`
AC_MSG_RESULT([$SWIG_LIB])
m4_ifval([$2],[$2],[])
fi
else
AC_MSG_WARN([cannot determine SWIG version])
SWIG=''
m4_ifval([$3],[$3],[])
fi
fi
AC_SUBST([SWIG_LIB])
])

325
lib/xdelta3/m4/ax_python_devel.m4 Normal file
View file

@ -0,0 +1,325 @@
# ===========================================================================
# http://www.gnu.org/software/autoconf-archive/ax_python_devel.html
# ===========================================================================
#
# SYNOPSIS
#
# AX_PYTHON_DEVEL([version])
#
# DESCRIPTION
#
# Note: Defines as a precious variable "PYTHON_VERSION". Don't override it
# in your configure.ac.
#
# This macro checks for Python and tries to get the include path to
# 'Python.h'. It provides the $(PYTHON_CPPFLAGS) and $(PYTHON_LDFLAGS)
# output variables. It also exports $(PYTHON_EXTRA_LIBS) and
# $(PYTHON_EXTRA_LDFLAGS) for embedding Python in your code.
#
# You can search for some particular version of Python by passing a
# parameter to this macro, for example ">= '2.3.1'", or "== '2.4'". Please
# note that you *have* to pass also an operator along with the version to
# match, and pay special attention to the single quotes surrounding the
# version number. Don't use "PYTHON_VERSION" for this: that environment
# variable is declared as precious and thus reserved for the end-user.
#
# This macro should work for all versions of Python >= 2.1.0. As an end
# user, you can disable the check for the python version by setting the
# PYTHON_NOVERSIONCHECK environment variable to something else than the
# empty string.
#
# If you need to use this macro for an older Python version, please
# contact the authors. We're always open for feedback.
#
# LICENSE
#
# Copyright (c) 2009 Sebastian Huber <sebastian-huber@web.de>
# Copyright (c) 2009 Alan W. Irwin <irwin@beluga.phys.uvic.ca>
# Copyright (c) 2009 Rafael Laboissiere <rafael@laboissiere.net>
# Copyright (c) 2009 Andrew Collier <colliera@ukzn.ac.za>
# Copyright (c) 2009 Matteo Settenvini <matteo@member.fsf.org>
# Copyright (c) 2009 Horst Knorr <hk_classes@knoda.org>
#
# This program is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the
# Free Software Foundation, either version 3 of the License, or (at your
# option) any later version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
# Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
#
# As a special exception, the respective Autoconf Macro's copyright owner
# gives unlimited permission to copy, distribute and modify the configure
# scripts that are the output of Autoconf when processing the Macro. You
# need not follow the terms of the GNU General Public License when using
# or distributing such scripts, even though portions of the text of the
# Macro appear in them. The GNU General Public License (GPL) does govern
# all other use of the material that constitutes the Autoconf Macro.
#
# This special exception to the GPL applies to versions of the Autoconf
# Macro released by the Autoconf Archive. When you make and distribute a
# modified version of the Autoconf Macro, you may extend this special
# exception to the GPL to apply to your modified version as well.
#serial 8
AU_ALIAS([AC_PYTHON_DEVEL], [AX_PYTHON_DEVEL])
AC_DEFUN([AX_PYTHON_DEVEL],[
#
# Allow the use of a (user set) custom python version
#
AC_ARG_VAR([PYTHON_VERSION],[The installed Python
version to use, for example '2.3'. This string
will be appended to the Python interpreter
canonical name.])
AC_PATH_PROG([PYTHON],[python[$PYTHON_VERSION]])
if test -z "$PYTHON"; then
AC_MSG_ERROR([Cannot find python$PYTHON_VERSION in your system path])
PYTHON_VERSION=""
fi
#
# Check for a version of Python >= 2.1.0
#
AC_MSG_CHECKING([for a version of Python >= '2.1.0'])
ac_supports_python_ver=`$PYTHON -c "import sys; \
ver = sys.version.split ()[[0]]; \
print (ver >= '2.1.0')"`
if test "$ac_supports_python_ver" != "True"; then
if test -z "$PYTHON_NOVERSIONCHECK"; then
AC_MSG_RESULT([no])
AC_MSG_FAILURE([
This version of the AC@&t@_PYTHON_DEVEL macro
doesn't work properly with versions of Python before
2.1.0. You may need to re-run configure, setting the
variables PYTHON_CPPFLAGS, PYTHON_LDFLAGS, PYTHON_SITE_PKG,
PYTHON_EXTRA_LIBS and PYTHON_EXTRA_LDFLAGS by hand.
Moreover, to disable this check, set PYTHON_NOVERSIONCHECK
to something else than an empty string.
])
else
AC_MSG_RESULT([skip at user request])
fi
else
AC_MSG_RESULT([yes])
fi
#
# if the macro parameter ``version'' is set, honour it
#
if test -n "$1"; then
AC_MSG_CHECKING([for a version of Python $1])
ac_supports_python_ver=`$PYTHON -c "import sys; \
ver = sys.version.split ()[[0]]; \
print (ver $1)"`
if test "$ac_supports_python_ver" = "True"; then
AC_MSG_RESULT([yes])
else
AC_MSG_RESULT([no])
AC_MSG_ERROR([this package requires Python $1.
If you have it installed, but it isn't the default Python
interpreter in your system path, please pass the PYTHON_VERSION
variable to configure. See ``configure --help'' for reference.
])
PYTHON_VERSION=""
fi
fi
#
# Check if you have distutils, else fail
#
AC_MSG_CHECKING([for the distutils Python package])
ac_distutils_result=`$PYTHON -c "import distutils" 2>&1`
if test -z "$ac_distutils_result"; then
AC_MSG_RESULT([yes])
else
AC_MSG_RESULT([no])
AC_MSG_ERROR([cannot import Python module "distutils".
Please check your Python installation. The error was:
$ac_distutils_result])
PYTHON_VERSION=""
fi
#
# Check for Python include path
#
AC_MSG_CHECKING([for Python include path])
if test -z "$PYTHON_CPPFLAGS"; then
python_path=`$PYTHON -c "import distutils.sysconfig; \
print (distutils.sysconfig.get_python_inc ());"`
if test -n "${python_path}"; then
python_path="-I$python_path"
fi
PYTHON_CPPFLAGS=$python_path
fi
AC_MSG_RESULT([$PYTHON_CPPFLAGS])
AC_SUBST([PYTHON_CPPFLAGS])
#
# Check for Python library path
#
AC_MSG_CHECKING([for Python library path])
if test -z "$PYTHON_LDFLAGS"; then
# (makes two attempts to ensure we've got a version number
# from the interpreter)
ac_python_version=`cat<<EOD | $PYTHON -
# join all versioning strings, on some systems
# major/minor numbers could be in different list elements
from distutils.sysconfig import *
ret = ''
for e in get_config_vars ('VERSION'):
if (e != None):
ret += e
print (ret)
EOD`
if test -z "$ac_python_version"; then
if test -n "$PYTHON_VERSION"; then
ac_python_version=$PYTHON_VERSION
else
ac_python_version=`$PYTHON -c "import sys; \
print (sys.version[[:3]])"`
fi
fi
# Make the versioning information available to the compiler
AC_DEFINE_UNQUOTED([HAVE_PYTHON], ["$ac_python_version"],
[If available, contains the Python version number currently in use.])
# First, the library directory:
ac_python_libdir=`cat<<EOD | $PYTHON -
# There should be only one
import distutils.sysconfig
for e in distutils.sysconfig.get_config_vars ('LIBDIR'):
if e != None:
print (e)
break
EOD`
# Before checking for libpythonX.Y, we need to know
# the extension the OS we're on uses for libraries
# (we take the first one, if there's more than one fix me!):
ac_python_soext=`$PYTHON -c \
"import distutils.sysconfig; \
print (distutils.sysconfig.get_config_vars('SO')[[0]])"`
# Now, for the library:
ac_python_soname=`$PYTHON -c \
"import distutils.sysconfig; \
print (distutils.sysconfig.get_config_vars('LDLIBRARY')[[0]])"`
# Strip away extension from the end to canonicalize its name:
ac_python_library=`echo "$ac_python_soname" | sed "s/${ac_python_soext}$//"`
# This small piece shamelessly adapted from PostgreSQL python macro;
# credits goes to momjian, I think. I'd like to put the right name
# in the credits, if someone can point me in the right direction... ?
#
if test -n "$ac_python_libdir" -a -n "$ac_python_library" \
-a x"$ac_python_library" != x"$ac_python_soname"
then
# use the official shared library
ac_python_library=`echo "$ac_python_library" | sed "s/^lib//"`
PYTHON_LDFLAGS="-L$ac_python_libdir -l$ac_python_library"
else
# old way: use libpython from python_configdir
ac_python_libdir=`$PYTHON -c \
"from distutils.sysconfig import get_python_lib as f; \
import os; \
print (os.path.join(f(plat_specific=1, standard_lib=1), 'config'));"`
PYTHON_LDFLAGS="-L$ac_python_libdir -lpython$ac_python_version"
fi
if test -z "PYTHON_LDFLAGS"; then
AC_MSG_ERROR([
Cannot determine location of your Python DSO. Please check it was installed with
dynamic libraries enabled, or try setting PYTHON_LDFLAGS by hand.
])
fi
fi
AC_MSG_RESULT([$PYTHON_LDFLAGS])
AC_SUBST([PYTHON_LDFLAGS])
#
# Check for site packages
#
AC_MSG_CHECKING([for Python site-packages path])
if test -z "$PYTHON_SITE_PKG"; then
PYTHON_SITE_PKG=`$PYTHON -c "import distutils.sysconfig; \
print (distutils.sysconfig.get_python_lib(0,0));"`
fi
AC_MSG_RESULT([$PYTHON_SITE_PKG])
AC_SUBST([PYTHON_SITE_PKG])
#
# libraries which must be linked in when embedding
#
AC_MSG_CHECKING(python extra libraries)
if test -z "$PYTHON_EXTRA_LIBS"; then
PYTHON_EXTRA_LIBS=`$PYTHON -c "import distutils.sysconfig; \
conf = distutils.sysconfig.get_config_var; \
print (conf('LOCALMODLIBS') + ' ' + conf('LIBS'))"`
fi
AC_MSG_RESULT([$PYTHON_EXTRA_LIBS])
AC_SUBST(PYTHON_EXTRA_LIBS)
#
# linking flags needed when embedding
#
AC_MSG_CHECKING(python extra linking flags)
if test -z "$PYTHON_EXTRA_LDFLAGS"; then
PYTHON_EXTRA_LDFLAGS=`$PYTHON -c "import distutils.sysconfig; \
conf = distutils.sysconfig.get_config_var; \
print (conf('LINKFORSHARED'))"`
fi
AC_MSG_RESULT([$PYTHON_EXTRA_LDFLAGS])
AC_SUBST(PYTHON_EXTRA_LDFLAGS)
#
# final check to see if everything compiles alright
#
AC_MSG_CHECKING([consistency of all components of python development environment])
# save current global flags
ac_save_LIBS="$LIBS"
ac_save_CPPFLAGS="$CPPFLAGS"
LIBS="$ac_save_LIBS $PYTHON_LDFLAGS $PYTHON_EXTRA_LDFLAGS $PYTHON_EXTRA_LIBS"
CPPFLAGS="$ac_save_CPPFLAGS $PYTHON_CPPFLAGS"
AC_LANG_PUSH([C])
AC_LINK_IFELSE([
AC_LANG_PROGRAM([[#include <Python.h>]],
[[Py_Initialize();]])
],[pythonexists=yes],[pythonexists=no])
AC_LANG_POP([C])
# turn back to default flags
CPPFLAGS="$ac_save_CPPFLAGS"
LIBS="$ac_save_LIBS"
AC_MSG_RESULT([$pythonexists])
if test ! "x$pythonexists" = "xyes"; then
AC_MSG_FAILURE([
Could not link test program to Python. Maybe the main Python library has been
installed in some non-standard library path. If so, pass it to configure,
via the LDFLAGS environment variable.
Example: ./configure LDFLAGS="-L/usr/non-standard-path/python/lib"
============================================================================
ERROR!
You probably have to install the development version of the Python package
for your distribution. The exact name of this package varies among them.
============================================================================
])
PYTHON_VERSION=""
fi
#
# all done!
#
])

64
lib/xdelta3/m4/ax_swig_python.m4 Normal file
View file

@ -0,0 +1,64 @@
# ===========================================================================
# http://www.gnu.org/software/autoconf-archive/ax_swig_python.html
# ===========================================================================
#
# SYNOPSIS
#
# AX_SWIG_PYTHON([use-shadow-classes = {no, yes}])
#
# DESCRIPTION
#
# Checks for Python and provides the $(AX_SWIG_PYTHON_CPPFLAGS), and
# $(AX_SWIG_PYTHON_OPT) output variables.
#
# $(AX_SWIG_PYTHON_OPT) contains all necessary SWIG options to generate
# code for Python. Shadow classes are enabled unless the value of the
# optional first argument is exactly 'no'. If you need multi module
# support (provided by the AX_SWIG_MULTI_MODULE_SUPPORT macro) use
# $(AX_SWIG_PYTHON_LIBS) to link against the appropriate library. It
# contains the SWIG Python runtime library that is needed by the type
# check system for example.
#
# LICENSE
#
# Copyright (c) 2008 Sebastian Huber <sebastian-huber@web.de>
# Copyright (c) 2008 Alan W. Irwin <irwin@beluga.phys.uvic.ca>
# Copyright (c) 2008 Rafael Laboissiere <rafael@laboissiere.net>
# Copyright (c) 2008 Andrew Collier <colliera@ukzn.ac.za>
#
# This program is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the
# Free Software Foundation; either version 2 of the License, or (at your
# option) any later version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
# Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
#
# As a special exception, the respective Autoconf Macro's copyright owner
# gives unlimited permission to copy, distribute and modify the configure
# scripts that are the output of Autoconf when processing the Macro. You
# need not follow the terms of the GNU General Public License when using
# or distributing such scripts, even though portions of the text of the
# Macro appear in them. The GNU General Public License (GPL) does govern
# all other use of the material that constitutes the Autoconf Macro.
#
# This special exception to the GPL applies to versions of the Autoconf
# Macro released by the Autoconf Archive. When you make and distribute a
# modified version of the Autoconf Macro, you may extend this special
# exception to the GPL to apply to your modified version as well.
#serial 7
AU_ALIAS([SWIG_PYTHON], [AX_SWIG_PYTHON])
AC_DEFUN([AX_SWIG_PYTHON],[
AC_REQUIRE([AX_PKG_SWIG])
AC_REQUIRE([AX_PYTHON_DEVEL])
test "x$1" != "xno" || swig_shadow=" -noproxy"
AC_SUBST([AX_SWIG_PYTHON_OPT],[-python$swig_shadow])
AC_SUBST([AX_SWIG_PYTHON_CPPFLAGS],[$PYTHON_CPPFLAGS])
])

25
lib/xdelta3/plot.sh Normal file
View file

@ -0,0 +1,25 @@
#!/bin/sh
G=/usr/bin/gnuplot
D=./output_dir
I=$1
O=$D/$2
$G > $O <<EOF
#set terminal jpeg
set terminal png
f(x) = 1331000 + 30000 * (1 / (x - 2.45))
# plot [x=1:10] [1:10] f(x)
# plot sin(x), cos(x)
# , f(x)
plot "$I" using 1:2
EOF
mv "$I" "$D"

1861
lib/xdelta3/rcs_junk.cc Normal file
View file

File diff suppressed because it is too large Load diff

288
lib/xdelta3/run_release.sh Normal file
View file

@ -0,0 +1,288 @@
#!/bin/bash
# Run from the source dir.
SRCDIR=${PWD}
# TODO replace w/ wget
LZMA="xz-5.2.1"
LZMA_FILE="${SRCDIR}/../${LZMA}.tar.gz"
MAKEFLAGS="-j 10"
BUILDDIR=${SRCDIR}/build
LZMASRC=${BUILDDIR}/${LZMA}
NONWIN_CFLAGS=""
MINGW_CFLAGS="-DEXTERNAL_COMPRESSION=0 -DXD3_WIN32=1 -DSHELL_TESTS=0"
MYOS=`uname`
DATE=`date`
CLEAN=""
LINUXTGTS=""
LINUXTEST1=""
LINUXTEST2=""
WINTGTS=""
WINTEST1=""
WINTEST2=""
OSXTGTS=""
OSXTEST1=""
OSXTEST2=""
XTMP="/tmp"
if [ "${TMP}" != "" ]; then
XTMP="${TMP}"
fi
if [ "${TMPDIR}" != "" ]; then
XTMP="${TMPDIR}"
fi
BUILDFILES=`ls -A ${BUILDDIR} 2> /dev/null`
if [ -d "${BUILDDIR}" ]; then
if [ -n "${BUILDFILES}" ]; then
echo "Directory ${BUILDDIR} should be empty"
exit 1
fi
else
mkdir "${BUILDDIR}"
fi
function setup {
libtoolize || glibtoolize
automake --add-missing
aclocal -I m4
autoheader
automake
autoconf
}
function try {
local w=$1
shift
local dir=$1
shift
echo -n " ${w} ... "
(cd "${dir}" && "$@" >${w}.stdout 2>${w}.stderr)
local s=$?
if [ ${s} -eq 0 ]; then
echo " success"
else
echo " failed!"
echo "Error $1 in ${dir}" >&2
fi
return ${s}
}
function buildlzma {
host=$1
march=$2
local target="${BUILDDIR}/lib-${host}${march}"
echo " ... liblzma"
mkdir -p ${target}
try configure-lzma ${target} ${LZMASRC}/configure \
--host=${host} \
--prefix=${target} \
--disable-shared \
"CC=${CC}" \
"CXX=${CXX}" \
"CFLAGS=${march}" \
"CXXFLAGS=${march}" \
"LDFLAGS=${march}"
if [ $? -ne 0 ]; then
return
fi
try build-lzma ${target} make ${MAKEFLAGS}
if [ $? -ne 0 ]; then
return
fi
try install-lzma ${target} make install
if [ $? -ne 0 ]; then
return
fi
}
function buildit {
local host=$1
local march=$2
local usizebits=$3
local offsetbits=$4
local cargs=$5
local afl=$6
local BM="${host}${march}"
local USECC="${CC}"
local USECXX="${CXX}"
local LIBBM="${BM}"
if [ "${afl}" = "1" ]; then
USECC="afl-gcc"
USECXX="afl-g++"
BM="${BM}-afl"
fi
local D="build/${BM}/usize${usizebits}/xoff${offsetbits}"
local BMD="${BM}-${usizebits}-${offsetbits}"
local FULLD="${SRCDIR}/${D}"
local CFLAGS="${march} ${cargs} -I${SRCDIR}/build/lib-${LIBBM}/include"
local CXXFLAGS="${march} ${cargs} -I${SRCDIR}/build/lib-${LIBBM}/include"
local CPPFLAGS="-I${SRCDIR}/build/lib-${LIBBM}/include"
local LDFLAGS="${march} -L${SRCDIR}/build/lib-${LIBBM}/lib"
local EXEC_PREAMBLE=""
local EXEC_SUFFIX=""
case ${host} in
*mingw*)
EXEC_PREAMBLE="wine"
EXEC_SUFFIX=".exe"
;;
esac
mkdir -p ${D}
echo " ... ${BMD}"
cat >> Makefile.test <<EOF
# ${BMD}
# ${CFLAGS}
.PHONY: build-${BMD}
build-${BMD}:
(cd ${D} && make all && make install)
.PHONY: clean-${BMD}
clean-${BMD}:
(cd ${D} && make clean)
.PHONY: regtest-${BMD}
regtest-${BMD}:
(cd ${D} && ${EXEC_PREAMBLE} ./bin/xdelta3regtest${EXEC_SUFFIX} 1> \${TMP}/regtest.${BMD}.stdout 2> \${TMP}/regtest.${BMD}.stderr)
.PHONY: selftest-${BMD}
selftest-${BMD}:
(cd ${D} && ${EXEC_PREAMBLE} ./bin/xdelta3${EXEC_SUFFIX} test 1> \${TMP}/selftest.${BMD}.stdout 2> \${TMP}/selftest.${BMD}.stderr)
EOF
case ${host} in
*linux*)
LINUXTGTS="${LINUXTGTS} build-${BMD}"
LINUXTEST1="${LINUXTEST1} selftest-${BMD}"
LINUXTEST2="${LINUXTEST2} regtest-${BMD}"
;;
*mingw*)
WINTGTS="${WINTGTS} build-${BMD}"
WINTEST1="${WINTEST1} selftest-${BMD}"
WINTEST2="${WINTEST2} regtest-${BMD}"
;;
*apple*)
OSXTGTS="${OSXTGTS} build-${BMD}"
OSXTEST1="${OSXTEST1} selftest-${BMD}"
OSXTEST2="${OSXTEST2} regtest-${BMD}"
;;
esac
CLEAN="${CLEAN} clean-${BMD}"
try configure-xdelta ${FULLD} ${SRCDIR}/configure \
--host=${host} \
--prefix=${FULLD} \
--enable-static \
--disable-shared \
--enable-debug-symbols \
"CFLAGS=${CFLAGS}" \
"CXXFLAGS=${CXXFLAGS}" \
"CPPFLAGS=${CPPFLAGS}" \
"LDFLAGS=${LDFLAGS}" \
"CC=${USECC}" \
"CXX=${USECXX}"
if [ $? -ne 0 ]; then
return
fi
# try build-xdelta ${FULLD} make ${MAKEFLAGS} all
# if [ $? -ne 0 ]; then
# return
# fi
# try install-xdelta ${FULLD} make install
}
function buildall {
echo ""
echo "Host $1$2 afl=$4"
echo ""
buildlzma "$1" "$2"
buildit "$1" "$2" 32 32 "-DXD3_USE_LARGESIZET=0 -DXD3_USE_LARGEFILE64=0 $3" "$4"
buildit "$1" "$2" 32 64 "-DXD3_USE_LARGESIZET=0 -DXD3_USE_LARGEFILE64=1 $3" "$4"
buildit "$1" "$2" 64 64 "-DXD3_USE_LARGESIZET=1 -DXD3_USE_LARGEFILE64=1 $3" "$4"
}
setup
try untar-lzma ${BUILDDIR} tar -xvf "${LZMA_FILE}"
if [ $? -ne 0 ]; then
exit $?
fi
cat > Makefile.test <<EOF
# Auto-generated ${DATE} -*- Mode: Makefile -*-
TMP = ${XTMP}
all: linux windows apple
EOF
# Native compiles
if [ "${MYOS}" == "Linux" ]; then
# Linux
buildall x86_64-pc-linux-gnu -m32 "${NONWIN_CFLAGS}" "0"
buildall x86_64-pc-linux-gnu -m32 "${NONWIN_CFLAGS}" "1"
buildall x86_64-pc-linux-gnu -m64 "${NONWIN_CFLAGS}" "0"
buildall x86_64-pc-linux-gnu -m64 "${NONWIN_CFLAGS}" "1"
fi
if [ "${MYOS}" == "Darwin" ]; then
# OS X
buildall x86_64-apple-darwin -m32 "${NONWIN_CFLAGS}" "0"
buildall x86_64-apple-darwin -m64 "${NONWIN_CFLAGS}" "0"
fi
# Cross compile
buildall i686-w64-mingw32 -mconsole "${MINGW_CFLAGS}" "0"
buildall x86_64-w64-mingw32 -mconsole "${MINGW_CFLAGS}" "0"
cat >> Makefile.test <<EOF
clean: ${CLEAN}
.PHONY: linux windows apple
.PHONY: linux-build windows-build apple-build
.PHONY: linux-selftest windows-selftest apple-selftest
.PHONY: linux-regtest windows-regtest apple-regtest
linux: linux-build linux-selftest linux-regtest
windows: windows-build windows-selftest windows-regtest
apple: apple-build apple-selftest apple-regtest
linux-build: ${LINUXTGTS}
linux-selftest: ${LINUXTEST1}
linux-regtest: ${LINUXTEST2}
windows-build: ${WINTGTS}
windows-selftest: ${WINTEST1}
windows-regtest: ${WINTEST2}
apple-build: ${OSXTGTS}
apple-selftest: ${OSXTEST1}
apple-regtest: ${OSXTEST2}
EOF

8
lib/xdelta3/testing/Makefile Normal file
View file

@ -0,0 +1,8 @@
all:
(cd .. && make all)
xdelta3regtest:
(cd .. && make xdelta3regtest)
xdelta3checksum:
(cd .. && make xdelta3checksum)

770
lib/xdelta3/testing/checksum_test.cc Normal file
View file

@ -0,0 +1,770 @@
/* xdelta3 - delta compression tools and library -*- Mode: C++ -*-
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#include "test.h"
#include <assert.h>
#include <list>
#include <vector>
#include <algorithm>
#include "../cpp-btree/btree_map.h"
extern "C" {
uint32_t xd3_large32_cksum_old (xd3_hash_cfg *cfg, const uint8_t *base, const usize_t look);
uint32_t xd3_large32_cksum_update_old (xd3_hash_cfg *cfg, uint32_t cksum,
const uint8_t *base, const usize_t look);
uint64_t xd3_large64_cksum_old (xd3_hash_cfg *cfg, const uint8_t *base, const usize_t look);
uint64_t xd3_large64_cksum_update_old (xd3_hash_cfg *cfg, uint64_t cksum,
const uint8_t *base, const usize_t look);
}
using btree::btree_map;
using std::list;
using std::vector;
// MLCG parameters
// a, a*
uint32_t good_32bit_values[] = {
1597334677U, // ...
741103597U, 887987685U,
};
// a, a*
uint64_t good_64bit_values[] = {
1181783497276652981ULL, 4292484099903637661ULL,
7664345821815920749ULL, // ...
};
void print_header() {
static int hdr_cnt = 0;
if (hdr_cnt++ % 20 == 0) {
printf("%-32sConf\t\tCount\tUniq\tFull\tCover\tColls"
"\tMB/s\tIters\t#Colls\n", "Name");
}
}
struct true_type { };
struct false_type { };
template <typename Word>
usize_t bitsof();
template<>
usize_t bitsof<unsigned int>() {
return sizeof(unsigned int) * 8;
}
template<>
usize_t bitsof<unsigned long>() {
return sizeof(unsigned long) * 8;
}
template<>
usize_t bitsof<unsigned long long>() {
return sizeof(unsigned long long) * 8;
}
template <typename Word>
struct hhash { // shift "s" bits leaving the high bits as a hash value for
// this checksum, which are the most "distant" in terms of the
// spectral test for the rabin_karp MLCG. For short windows,
// the high bits aren't enough, XOR "mask" worth of these in.
Word operator()(const Word t, const Word s, const Word mask) {
return (t >> s) ^ (t & mask);
}
};
template <typename Word>
Word good_word();
template<>
uint32_t good_word<uint32_t>() {
return good_32bit_values[0];
}
template<>
uint64_t good_word<uint64_t>() {
return good_64bit_values[0];
}
// CLASSES
#define SELF Word, CksumSize, CksumSkip, Hash, Compaction
#define MEMBER template <typename Word, \
int CksumSize, \
int CksumSkip, \
typename Hash, \
int Compaction>
MEMBER
struct cksum_params {
typedef Word word_type;
typedef Hash hash_type;
static const int cksum_size = CksumSize;
static const int cksum_skip = CksumSkip;
static const int compaction = Compaction;
};
MEMBER
struct rabin_karp : public cksum_params<SELF> {
// (a^cksum_size-1 c_0) + (a^cksum_size-2 c_1) ...
rabin_karp()
: powers(make_powers()),
product(powers[0] * good_word<Word>()),
incr_state(0) { }
static Word* make_powers() {
Word *p = new Word[CksumSize];
p[CksumSize - 1] = 1;
for (int i = CksumSize - 2; i >= 0; i--) {
p[i] = p[i + 1] * good_word<Word>();
}
return p;
}
~rabin_karp() {
delete [] powers;
}
Word step(const uint8_t *ptr) {
Word h = 0;
for (int i = 0; i < CksumSize; i++) {
h += (ptr[i]) * powers[i];
}
return h;
}
Word state0(const uint8_t *ptr) {
incr_state = step(ptr);
return incr_state;
}
Word incr(const uint8_t *ptr) {
incr_state = good_word<Word>() * incr_state -
product * (ptr[-1]) + (ptr[CksumSize - 1]);
return incr_state;
}
const Word *const powers;
const Word product;
Word incr_state;
};
MEMBER
struct with_stream : public cksum_params<SELF> {
xd3_stream stream;
with_stream()
{
xd3_config cfg;
memset (&stream, 0, sizeof (stream));
xd3_init_config (&cfg, 0);
cfg.smatch_cfg = XD3_SMATCH_SOFT;
cfg.smatcher_soft.large_look = CksumSize;
cfg.smatcher_soft.large_step = CksumSkip;
cfg.smatcher_soft.small_look = 4;
cfg.smatcher_soft.small_chain = 4;
cfg.smatcher_soft.small_lchain = 4;
cfg.smatcher_soft.max_lazy = 4;
cfg.smatcher_soft.long_enough = 4;
CHECK_EQ(0, xd3_config_stream (&stream, &cfg));
CHECK_EQ(0, xd3_size_hashtable (&stream,
1<<10 /* ignored */,
stream.smatcher.large_look,
& stream.large_hash));
}
~with_stream()
{
xd3_free_stream (&stream);
}
};
MEMBER
struct large_cksum : public with_stream<SELF> {
Word step(const uint8_t *ptr) {
return xd3_large_cksum (&this->stream.large_hash, ptr, CksumSize);
}
Word state0(const uint8_t *ptr) {
incr_state = step(ptr);
return incr_state;
}
Word incr(const uint8_t *ptr) {
incr_state = xd3_large_cksum_update (&this->stream.large_hash,
incr_state, ptr - 1, CksumSize);
return incr_state;
}
Word incr_state;
};
#if SIZEOF_USIZE_T == 4
#define xd3_large_cksum_old xd3_large32_cksum_old
#define xd3_large_cksum_update_old xd3_large32_cksum_update_old
#elif SIZEOF_USIZE_T == 8
#define xd3_large_cksum_old xd3_large64_cksum_old
#define xd3_large_cksum_update_old xd3_large64_cksum_update_old
#endif
MEMBER
struct large_cksum_old : public with_stream<SELF> {
Word step(const uint8_t *ptr) {
return xd3_large_cksum_old (&this->stream.large_hash, ptr, CksumSize);
}
Word state0(const uint8_t *ptr) {
incr_state = step(ptr);
return incr_state;
}
Word incr(const uint8_t *ptr) {
incr_state = xd3_large_cksum_update_old (&this->stream.large_hash,
incr_state, ptr - 1, CksumSize);
return incr_state;
}
Word incr_state;
};
// TESTS
template <typename Word>
struct file_stats {
typedef const uint8_t* ptr_type;
typedef Word word_type;
typedef btree::btree_multimap<word_type, ptr_type> table_type;
typedef typename table_type::iterator table_iterator;
usize_t cksum_size;
usize_t cksum_skip;
usize_t unique;
usize_t unique_values;
usize_t count;
table_type table;
file_stats(usize_t size, usize_t skip)
: cksum_size(size),
cksum_skip(skip),
unique(0),
unique_values(0),
count(0) {
}
void reset() {
unique = 0;
unique_values = 0;
count = 0;
table.clear();
}
void update(word_type word, ptr_type ptr) {
table_iterator t_i = table.find(word);
count++;
if (t_i != table.end()) {
int collisions = 0;
for (table_iterator p_i = t_i;
p_i != table.end() && p_i->first == word;
++p_i) {
if (memcmp(p_i->second, ptr, cksum_size) == 0) {
return;
}
collisions++;
}
if (collisions >= 1000) {
fprintf(stderr, "Something is not right, lots of collisions=%d\n",
collisions);
abort();
}
} else {
unique_values++;
}
unique++;
table.insert(std::make_pair(word, ptr));
return;
}
void freeze() {
table.clear();
}
};
struct test_result_base;
static vector<test_result_base*> all_tests;
struct test_result_base {
virtual ~test_result_base() {
}
virtual void reset() = 0;
virtual void print() = 0;
virtual void get(const uint8_t* buf, const size_t buf_size,
usize_t iters) = 0;
virtual void stat() = 0;
virtual usize_t count() = 0;
virtual usize_t dups() = 0;
virtual double uniqueness() = 0;
virtual double fullness() = 0;
virtual double collisions() = 0;
virtual double coverage() = 0;
virtual double compression() = 0;
virtual double time() = 0;
virtual double total_time() = 0;
virtual usize_t total_count() = 0;
virtual usize_t total_dups() = 0;
};
template <typename Checksum>
struct test_result : public test_result_base {
Checksum cksum;
const char *test_name;
file_stats<typename Checksum::word_type> fstats;
usize_t test_size;
usize_t n_steps;
usize_t n_incrs;
typename Checksum::word_type s_bits;
typename Checksum::word_type s_mask;
usize_t t_entries;
usize_t h_bits;
usize_t h_buckets_full;
char *hash_table;
long accum_millis;
usize_t accum_iters;
// These are not reset
double accum_time;
usize_t accum_count;
usize_t accum_dups;
usize_t accum_colls;
size_t accum_size;
test_result(const char *name)
: test_name(name),
fstats(Checksum::cksum_size, Checksum::cksum_skip),
hash_table(NULL),
accum_millis(0),
accum_iters(0),
accum_time(0.0),
accum_count(0),
accum_dups(0),
accum_colls(0),
accum_size(0) {
all_tests.push_back(this);
}
~test_result() {
reset();
}
void reset() {
// size of file
test_size = 0;
// count
n_steps = 0;
n_incrs = 0;
// four values used by new_table()/summarize_table()
s_bits = 0;
s_mask = 0;
t_entries = 0;
h_bits = 0;
h_buckets_full = 0;
accum_millis = 0;
accum_iters = 0;
fstats.reset();
// temporary
if (hash_table) {
delete(hash_table);
hash_table = NULL;
}
}
usize_t count() {
if (Checksum::cksum_skip == 1) {
return n_incrs;
} else {
return n_steps;
}
}
usize_t dups() {
return fstats.count - fstats.unique;
}
/* Fraction of distinct strings of length cksum_size which are not
* represented in the hash table. */
double collisions() {
return (fstats.unique - fstats.unique_values) / (double) fstats.unique;
}
usize_t colls() {
return (fstats.unique - fstats.unique_values);
}
double uniqueness() {
return 1.0 - (double) dups() / count();
}
double fullness() {
return (double) h_buckets_full / (1 << h_bits);
}
double coverage() {
return (double) h_buckets_full / uniqueness() / count();
}
double compression() {
return 1.0 - coverage();
}
double time() {
return (double) accum_millis / accum_iters;
}
double total_time() {
return accum_time;
}
usize_t total_count() {
return accum_count;
}
usize_t total_dups() {
return accum_dups;
}
usize_t total_colls() {
return accum_dups;
}
void stat() {
accum_time += time();
accum_count += count();
accum_dups += dups();
accum_colls += colls();
accum_size += test_size;
}
void print() {
if (fstats.count != count()) {
fprintf(stderr, "internal error: %" W "d != %" W "d\n", fstats.count, count());
abort();
}
print_header();
printf("%-32s%d/%d 2^%" W "u\t%" W "u\t%0.4f\t%.4f\t%.4f\t%.1e\t%.2f\t"
"%" W "u\t%" W "u\n",
test_name,
Checksum::cksum_size,
Checksum::cksum_skip,
h_bits,
count(),
uniqueness(),
fullness(),
coverage(),
collisions(),
0.001 * accum_iters * test_size / accum_millis,
accum_iters,
colls());
}
usize_t size_log2 (usize_t slots) {
usize_t bits = bitsof<typename Checksum::word_type>() - 1;
usize_t i;
for (i = 3; i <= bits; i += 1) {
if (slots <= (1U << i)) {
return i - Checksum::compaction;
}
}
return bits;
}
void new_table(usize_t entries) {
t_entries = entries;
h_bits = size_log2(entries);
usize_t n = 1 << h_bits;
s_bits = bitsof<typename Checksum::word_type>() - h_bits;
s_mask = n - 1U;
hash_table = new char[n / 8];
memset(hash_table, 0, n / 8);
}
int get_table_bit(usize_t i) {
return hash_table[i/8] & (1 << i%8);
}
int set_table_bit(usize_t i) {
return hash_table[i/8] |= (1 << i%8);
}
void summarize_table() {
usize_t n = 1 << h_bits;
usize_t f = 0;
for (usize_t i = 0; i < n; i++) {
if (get_table_bit(i)) {
f++;
}
}
h_buckets_full = f;
}
void get(const uint8_t* buf, const size_t buf_size, usize_t test_iters) {
typename Checksum::hash_type hash;
const uint8_t *ptr;
const uint8_t *end;
usize_t periods;
int64_t last_offset;
int64_t stop;
test_size = buf_size;
last_offset = buf_size - Checksum::cksum_size;
if (last_offset < 0) {
periods = 0;
n_steps = 0;
n_incrs = 0;
stop = -Checksum::cksum_size;
} else {
periods = last_offset / Checksum::cksum_skip;
n_steps = periods + 1;
n_incrs = last_offset + 1;
stop = last_offset - (periods + 1) * Checksum::cksum_skip;
}
// Compute file stats once.
if (fstats.unique_values == 0) {
if (Checksum::cksum_skip == 1) {
for (size_t i = 0; i <= buf_size - Checksum::cksum_size; i++) {
fstats.update(hash(cksum.step(buf + i), s_bits, s_mask), buf + i);
}
} else {
ptr = buf + last_offset;
end = buf + stop;
for (; ptr != end; ptr -= Checksum::cksum_skip) {
fstats.update(hash(cksum.step(ptr), s_bits, s_mask), ptr);
}
}
fstats.freeze();
}
long start_test = get_millisecs_now();
if (Checksum::cksum_skip != 1) {
new_table(n_steps);
for (usize_t i = 0; i < test_iters; i++) {
ptr = buf + last_offset;
end = buf + stop;
for (; ptr != end; ptr -= Checksum::cksum_skip) {
set_table_bit(hash(cksum.step(ptr), s_bits, s_mask));
}
}
summarize_table();
}
stop = buf_size - Checksum::cksum_size + 1;
if (stop < 0) {
stop = 0;
}
if (Checksum::cksum_skip == 1) {
new_table(n_incrs);
for (usize_t i = 0; i < test_iters; i++) {
ptr = buf;
end = buf + stop;
if (ptr != end) {
set_table_bit(hash(cksum.state0(ptr++), s_bits, s_mask));
}
for (; ptr != end; ptr++) {
typename Checksum::word_type w = cksum.incr(ptr);
CHECK_EQ(w, cksum.step(ptr));
set_table_bit(hash(w, s_bits, s_mask));
}
}
summarize_table();
}
accum_iters += test_iters;
accum_millis += get_millisecs_now() - start_test;
}
};
static int read_whole_file(const char *name,
uint8_t **buf_ptr,
size_t *buf_len) {
main_file file;
int ret;
xoff_t len;
size_t nread;
main_file_init(&file);
file.filename = name;
ret = main_file_open(&file, name, XO_READ);
if (ret != 0) {
fprintf(stderr, "open failed\n");
goto exit;
}
ret = main_file_stat(&file, &len);
if (ret != 0) {
fprintf(stderr, "stat failed\n");
goto exit;
}
(*buf_len) = (size_t)len;
(*buf_ptr) = (uint8_t*) main_malloc(*buf_len);
ret = main_file_read(&file, *buf_ptr, *buf_len, &nread,
"read failed");
if (ret == 0 && *buf_len == nread) {
ret = 0;
} else {
fprintf(stderr, "invalid read\n");
ret = XD3_INTERNAL;
}
exit:
main_file_cleanup(&file);
return ret;
}
int main(int argc, char** argv) {
int i;
uint8_t *buf = NULL;
size_t buf_len = 0;
int ret;
if (argc <= 1) {
fprintf(stderr, "usage: %s file ...\n", argv[0]);
return 1;
}
// TODO: The xdelta3-hash.h code is identical now; add sameness test.
// using rabin_karp<> template.
#define TEST(T,Z,S,C) \
test_result<large_cksum<T,Z,S,hhash<T>,C>> \
_xck_ ## T ## _ ## Z ## _ ## S ## _ ## C \
("xck_" #T "_" #Z "_" #S "_" #C); \
test_result<large_cksum_old<T,Z,S,hhash<T>,C>> \
_old_ ## T ## _ ## Z ## _ ## S ## _ ## C \
("old_" #T "_" #Z "_" #S "_" #C)
#define TESTS(SIZE, SKIP) \
TEST(usize_t, SIZE, SKIP, 1); \
TEST(usize_t, SIZE, SKIP, 2)
TESTS(5, 1);
TESTS(6, 1);
TESTS(7, 1);
TESTS(8, 1);
TESTS(9, 1);
TESTS(10, 1);
TESTS(11, 1);
TESTS(12, 1);
TESTS(13, 1);
TESTS(14, 1);
TESTS(15, 1);
TESTS(16, 1);
TESTS(17, 1);
TESTS(18, 1);
TESTS(19, 1);
TESTS(20, 1);
TESTS(21, 1);
TESTS(22, 1);
TESTS(23, 1);
TESTS(24, 1);
TESTS(25, 1);
TESTS(26, 1);
TESTS(27, 1);
TESTS(28, 1);
TESTS(29, 1);
TESTS(30, 1);
TESTS(31, 1);
TESTS(32, 1);
TESTS(33, 1);
TESTS(34, 1);
TESTS(35, 1);
TESTS(36, 1);
TESTS(37, 1);
TESTS(38, 1);
TESTS(39, 1);
for (i = 1; i < argc; i++) {
if ((ret = read_whole_file(argv[i],
& buf,
& buf_len))) {
return 1;
}
fprintf(stderr, "file %s is %zu bytes\n",
argv[i], buf_len);
double min_time = -1.0;
double min_compression = 0.0;
for (vector<test_result_base*>::iterator iter = all_tests.begin();
iter != all_tests.end(); ++iter) {
test_result_base *test = *iter;
test->reset();
usize_t iters = 1;
long start_test = get_millisecs_now();
do {
test->get(buf, buf_len, iters);
iters *= 3;
iters /= 2;
} while (get_millisecs_now() - start_test < 2000);
test->stat();
if (min_time < 0.0) {
min_compression = test->compression();
min_time = test->time();
}
if (min_time > test->time()) {
min_time = test->time();
}
if (min_compression > test->compression()) {
min_compression = test->compression();
}
test->print();
}
main_free(buf);
buf = NULL;
}
return 0;
}

189
lib/xdelta3/testing/checksum_test_c.c Normal file
View file

@ -0,0 +1,189 @@
/* xdelta3 - delta compression tools and library -*- Mode: C++ -*-
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#include "../xdelta3.c"
// OLD CHECKSUM CODE
#define PERMUTE32(x) (__single_hash32[x])
#define PERMUTE64(x) (__single_hash64[x])
const uint16_t __single_hash32[256] =
{
/* This hashes the input alphabet (Scheme SLIB pseudo-random). */
0xbcd1, 0xbb65, 0x42c2, 0xdffe, 0x9666, 0x431b, 0x8504, 0xeb46,
0x6379, 0xd460, 0xcf14, 0x53cf, 0xdb51, 0xdb08, 0x12c8, 0xf602,
0xe766, 0x2394, 0x250d, 0xdcbb, 0xa678, 0x02af, 0xa5c6, 0x7ea6,
0xb645, 0xcb4d, 0xc44b, 0xe5dc, 0x9fe6, 0x5b5c, 0x35f5, 0x701a,
0x220f, 0x6c38, 0x1a56, 0x4ca3, 0xffc6, 0xb152, 0x8d61, 0x7a58,
0x9025, 0x8b3d, 0xbf0f, 0x95a3, 0xe5f4, 0xc127, 0x3bed, 0x320b,
0xb7f3, 0x6054, 0x333c, 0xd383, 0x8154, 0x5242, 0x4e0d, 0x0a94,
0x7028, 0x8689, 0x3a22, 0x0980, 0x1847, 0xb0f1, 0x9b5c, 0x4176,
0xb858, 0xd542, 0x1f6c, 0x2497, 0x6a5a, 0x9fa9, 0x8c5a, 0x7743,
0xa8a9, 0x9a02, 0x4918, 0x438c, 0xc388, 0x9e2b, 0x4cad, 0x01b6,
0xab19, 0xf777, 0x365f, 0x1eb2, 0x091e, 0x7bf8, 0x7a8e, 0x5227,
0xeab1, 0x2074, 0x4523, 0xe781, 0x01a3, 0x163d, 0x3b2e, 0x287d,
0x5e7f, 0xa063, 0xb134, 0x8fae, 0x5e8e, 0xb7b7, 0x4548, 0x1f5a,
0xfa56, 0x7a24, 0x900f, 0x42dc, 0xcc69, 0x02a0, 0x0b22, 0xdb31,
0x71fe, 0x0c7d, 0x1732, 0x1159, 0xcb09, 0xe1d2, 0x1351, 0x52e9,
0xf536, 0x5a4f, 0xc316, 0x6bf9, 0x8994, 0xb774, 0x5f3e, 0xf6d6,
0x3a61, 0xf82c, 0xcc22, 0x9d06, 0x299c, 0x09e5, 0x1eec, 0x514f,
0x8d53, 0xa650, 0x5c6e, 0xc577, 0x7958, 0x71ac, 0x8916, 0x9b4f,
0x2c09, 0x5211, 0xf6d8, 0xcaaa, 0xf7ef, 0x287f, 0x7a94, 0xab49,
0xfa2c, 0x7222, 0xe457, 0xd71a, 0x00c3, 0x1a76, 0xe98c, 0xc037,
0x8208, 0x5c2d, 0xdfda, 0xe5f5, 0x0b45, 0x15ce, 0x8a7e, 0xfcad,
0xaa2d, 0x4b5c, 0xd42e, 0xb251, 0x907e, 0x9a47, 0xc9a6, 0xd93f,
0x085e, 0x35ce, 0xa153, 0x7e7b, 0x9f0b, 0x25aa, 0x5d9f, 0xc04d,
0x8a0e, 0x2875, 0x4a1c, 0x295f, 0x1393, 0xf760, 0x9178, 0x0f5b,
0xfa7d, 0x83b4, 0x2082, 0x721d, 0x6462, 0x0368, 0x67e2, 0x8624,
0x194d, 0x22f6, 0x78fb, 0x6791, 0xb238, 0xb332, 0x7276, 0xf272,
0x47ec, 0x4504, 0xa961, 0x9fc8, 0x3fdc, 0xb413, 0x007a, 0x0806,
0x7458, 0x95c6, 0xccaa, 0x18d6, 0xe2ae, 0x1b06, 0xf3f6, 0x5050,
0xc8e8, 0xf4ac, 0xc04c, 0xf41c, 0x992f, 0xae44, 0x5f1b, 0x1113,
0x1738, 0xd9a8, 0x19ea, 0x2d33, 0x9698, 0x2fe9, 0x323f, 0xcde2,
0x6d71, 0xe37d, 0xb697, 0x2c4f, 0x4373, 0x9102, 0x075d, 0x8e25,
0x1672, 0xec28, 0x6acb, 0x86cc, 0x186e, 0x9414, 0xd674, 0xd1a5
};
const uint32_t __single_hash64[256] =
{
/* http://random.org 2014.10.24 */
0xd25e9f0a, 0xb1af9d5e, 0xb753dfa2, 0x157050f7, /* 0 */
0xc84b072c, 0xdd14fe7c, 0xf92208c3, 0xdf08a0c0,
0x63a5c118, 0x76f5d90f, 0xa2f8b93e, 0xb6c12d22,
0xaf074957, 0x966fb7d9, 0x62f7b785, 0xb40e8a09,
0x0a811d5d, 0x323a6daa, 0xb62f7c5b, 0xfdcb9a53,
0xf25a9067, 0x4506bc7a, 0xff58a74b, 0x5ae62817,
0x74097675, 0x722c0fd9, 0x116a2a66, 0x65f76728,
0x72c79651, 0xe043cf9d, 0x64b867c7, 0x6604834f,
0xcdca58a6, 0x0f164e2d, 0x24515f05, 0x632cdbf8,
0x18091d4a, 0x3eff4128, 0x673d1c33, 0xd8e10c71,
0x1a3edf11, 0xba52892f, 0xa56949e0, 0xf3e1dd77, /* 10 */
0x86fcbe3e, 0x138d66d0, 0x4fc98359, 0xc22e5dd6,
0xc59f2267, 0x6c6dd739, 0xe03da190, 0x07e8469c,
0xadcfb02c, 0x00d3b0d9, 0xa1f44918, 0x8bd84d87,
0x08ec9ec1, 0xbbcd156f, 0xb57718e3, 0x3177e752,
0xf52a4d70, 0xde7aaad9, 0x075f1da0, 0x21ba00c6,
0xb9469a5c, 0xcf08d5ba, 0x91ac9edc, 0xc6167b63,
0xc1974919, 0xc8c8d195, 0x4b1996dd, 0xeff8991c,
0xf7f66c6b, 0x25b012e2, 0x59d12a98, 0xea40d3cc,
0x41f9970b, 0xec48101a, 0xa3bdcf90, 0x99f16905,
0x27af6c97, 0xc849af37, 0x49cad89b, 0xf48c2278, /* 20 */
0x5529c3d8, 0x9e7d6dce, 0x16feb52d, 0xf1b0aca1,
0xaf28fccb, 0x48e4ce3c, 0xc4436617, 0x64524e3e,
0x61806681, 0x6384f2d7, 0x1172880f, 0x34a5ef5f,
0xcc8cc0a8, 0x66e8f100, 0x2866085f, 0xba9b1b2d,
0x51285949, 0x2be4b574, 0x889b1ef5, 0x3dbe920d,
0x9277a62f, 0x0584a9f6, 0x085d8fc4, 0x4b5d403d,
0x4e46ca78, 0x3294c2f9, 0x29313e70, 0xe4f09b24,
0xe73b331c, 0x072f5552, 0x2e390b78, 0xea0021ca,
0xd8f40320, 0xed0e16fd, 0x7de9cf7a, 0xf17e3d6c,
0x8df1bd85, 0x052cae67, 0x3486e512, 0x3a1c09b8, /* 30 */
0x6c2a7b4e, 0x83455753, 0xbc0353ac, 0x0ffe20b6,
0x5fdcef85, 0x010f506c, 0x595ce972, 0xe28680d0,
0xa7e216b2, 0xa392ee0f, 0x25b73faa, 0x2b1f4983,
0xeeaefe98, 0x1d3d9cbc, 0x6aebe97b, 0x8b7b3584,
0x9e6a9a07, 0xd37f1e99, 0x4ac2a441, 0x8ae9a213,
0x7d0e27d7, 0x5de54b9a, 0x8621de1f, 0xf0f2f866,
0xcb08d275, 0x49c3f87e, 0xd5ee68c1, 0x9802fc77,
0x68be6c5e, 0x65aa8c27, 0xf423d5f7, 0x10ec5502,
0x9909bce1, 0x509cdf1b, 0x338fea72, 0x2733e9bf,
0xf92f4fd7, 0x87738ea2, 0x931a8bbc, 0x0a5c9155, /* 40 */
0xbe5edd9b, 0xadbf5838, 0x0338f8d2, 0x290da210,
0x390c37d8, 0xe7cffae8, 0x20617ebe, 0x464322dd,
0x7b3c4e78, 0xac142dcb, 0x2d5cef76, 0xd8fe49fc,
0x60f4e9a9, 0x7473816f, 0x0dc35f39, 0x5eed80c1,
0x0cb55ab6, 0x1d3ac541, 0x13c7f529, 0x7bffdf4a,
0xe334785b, 0x85263ec1, 0xd132ae56, 0x7c868b9e,
0x47f60638, 0x1012b979, 0x81c31dd3, 0x1af868c8,
0x0c5d0742, 0xd1b3e1a2, 0x5873200a, 0xf848465c,
0x0fc4d596, 0x609c18af, 0xc9f5a480, 0xd1a94a84,
0xa1431a3f, 0x7de8bb1a, 0x25f1256b, 0x1dcc732c, /* 50 */
0x6aa1549a, 0xa2367281, 0x32f2a77e, 0x82e62a0f,
0x045cbb56, 0x74b2027c, 0xd71a32d9, 0x022e7cb5,
0xe99be177, 0x60222fdf, 0xd69681ca, 0x9008ee2c,
0x32923db4, 0xcf82bf97, 0x38960a5b, 0xb3503d5b,
0x9bd4c7f2, 0x33c029c8, 0x1ef504a3, 0xdb249d3b,
0x91e89676, 0x4ca43b36, 0x9191433c, 0x465d5dc4,
0xf4dcb118, 0x9d11dd00, 0xb592f058, 0xdbe5ce30,
0x74790d92, 0x779850a8, 0x7180d25b, 0xfa951d99,
0x5990935a, 0x921cb022, 0x3b7c39bc, 0x6a38a7c7,
0xdc22703b, 0x142bab3b, 0x4e3d9479, 0x44bb8482, /* 60 */
0x8043abce, 0xfebe832a, 0x8e6a2f98, 0x4d43c4fe,
0xd192a70a, 0x802f3c3a, 0x5d11bbab, 0x2665d241,
0xb3f3a680, 0x3a8d223f, 0xcf82cdb4, 0x4ed28743,
};
uint64_t
xd3_large64_cksum_old (xd3_hash_cfg *ignore, const uint8_t *base, const usize_t look)
{
static const uint64_t kBits = 32;
static const uint64_t kMask = 0xffffffff;
usize_t i = 0;
uint64_t low = 0;
uint64_t high = 0;
for (; i < look; i += 1)
{
low += PERMUTE64(*base++);
high += low;
}
return ((high & kMask) << kBits) | (low & kMask);
}
uint64_t
xd3_large64_cksum_update_old (xd3_hash_cfg *ignore, const uint64_t cksum,
const uint8_t *base, const usize_t look)
{
static const uint64_t kBits = 32;
static const uint64_t kMask = 0xffffffff;
uint64_t old_c = PERMUTE64(base[0]);
uint64_t new_c = PERMUTE64(base[look]);
uint64_t low = ((cksum & kMask) - old_c + new_c) & kMask;
uint64_t high = ((cksum >> kBits) - (old_c * look) + low) & kMask;
return (high << kBits) | low;
}
uint32_t
xd3_large32_cksum_old (xd3_hash_cfg *ignore, const uint8_t *base, const usize_t look)
{
static const uint32_t kBits = 16;
static const uint32_t kMask = 0xffff;
usize_t i = 0;
uint32_t low = 0;
uint32_t high = 0;
for (; i < look; i += 1)
{
low += PERMUTE32(*base++);
high += low;
}
return ((high & kMask) << kBits) | (low & kMask);
}
uint32_t
xd3_large32_cksum_update_old (xd3_hash_cfg *ignore, const uint32_t cksum,
const uint8_t *base, const usize_t look)
{
static const uint32_t kBits = 16;
static const uint32_t kMask = 0xffff;
uint32_t old_c = PERMUTE32(base[0]);
uint32_t new_c = PERMUTE32(base[look]);
uint32_t low = ((cksum & kMask) - old_c + new_c) & kMask;
uint32_t high = ((cksum >> kBits) - (old_c * look) + low) & kMask;
return (high << kBits) | low;
}

67
lib/xdelta3/testing/cmp.h Normal file
View file

@ -0,0 +1,67 @@
/* xdelta3 - delta compression tools and library -*- Mode: C++ -*-
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
static size_t CmpDifferentBlockBytes(const Block &a, const Block &b) {
size_t total = 0;
size_t i = 0;
size_t m = min(a.Size(), b.Size());
for (; i < m; i++) {
if (a[i] != b[i]) {
total++;
}
}
total += a.Size() - i;
total += b.Size() - i;
return total;
}
static xoff_t CmpDifferentBytes(const FileSpec &a, const FileSpec &b) {
Block block_a, block_b;
xoff_t total = 0;
typename FileSpec::iterator a_i(a), b_i(b);
for (; !a_i.Done() && !b_i.Done(); a_i.Next(), b_i.Next()) {
a_i.Get(&block_a);
b_i.Get(&block_b);
total += CmpDifferentBlockBytes(block_a, block_b);
}
for (; !a_i.Done(); a_i.Next()) {
total += a_i.BytesOnBlock();
}
for (; !b_i.Done(); b_i.Next()) {
total += b_i.BytesOnBlock();
}
return total;
}
static size_t CmpDifferentBlockBytesAtOffset(const Block &a,
const FileSpec &b_spec,
xoff_t offset) {
Block b;
size_t size = a.Size();
CHECK_LE(offset, b_spec.Size());
if (b_spec.Size() < offset + size) {
size = b_spec.Size() - offset;
}
b_spec.Get(&b, offset, size);
return CmpDifferentBlockBytes(a, b);
}

87
lib/xdelta3/testing/delta.h Normal file
View file

@ -0,0 +1,87 @@
/* xdelta3 - delta compression tools and library -*- Mode: C++ -*-
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
class Delta {
public:
Delta(const Block &block) {
int ret;
xd3_config config;
memset(&stream_, 0, sizeof (stream_));
memset(&config, 0, sizeof (config));
xd3_init_config(&config, XD3_SKIP_EMIT | XD3_ADLER32_NOVER);
CHECK_EQ(0, xd3_config_stream (&stream_, &config));
xd3_avail_input (&stream_, block.Data(), block.Size());
bool done = false;
while (!done) {
ret = xd3_decode_input(&stream_);
switch (ret) {
case XD3_INPUT:
done = true;
break;
case XD3_OUTPUT:
CHECK_EQ(0, xd3_whole_append_window (&stream_));
break;
case XD3_GOTHEADER:
case XD3_WINSTART:
case XD3_WINFINISH:
break;
default:
cerr << "decode: " << done;
abort();
}
}
}
~Delta() {
xd3_free_stream(&stream_);
}
xoff_t AddedBytes() const {
return stream_.whole_target.addslen;
}
xoff_t Windows() const {
return stream_.whole_target.wininfolen;
}
// Note: This does not benefit from -Wformat= checking, due to the
// enclosing template. Further, it was not used.
// void Print() const {
// for (size_t i = 0; i < stream_.whole_target.instlen; i++) {
// xd3_winst &winst = stream_.whole_target.inst[i];
// switch (winst.type) {
// case XD3_RUN:
// DP(RINT, "%" Q "u run %" W "u\n", winst.position, winst.size);
// break;
// case XD3_ADD:
// DP(RINT "%" Q "u add %" W "u\n", winst.position, winst.size);
// break;
// default:
// DP(RINT "%" Q "u copy %" W "u @ %" Q "u (mode %u)\n",
// winst.position, winst.size, winst.addr, winst.mode);
// break;
// }
// }
// }
private:
xd3_stream stream_;
};

399
lib/xdelta3/testing/file.h Normal file
View file

@ -0,0 +1,399 @@
/* xdelta3 - delta compression tools and library -*- Mode: C++ -*-
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
class Block;
class BlockIterator;
class TmpFile;
class Block {
public:
Block()
: data_(NULL),
data_size_(0),
size_(0) { }
~Block() {
if (data_) {
delete [] data_;
}
}
size_t Size() const {
return size_;
}
uint8_t operator[](size_t i) const {
CHECK_LT(i, size_);
return data_[i];
}
uint8_t* Data() const {
if (data_ == NULL) {
CHECK_EQ(0, size_);
data_size_ = 1;
data_ = new uint8_t[1];
}
return data_;
}
// For writing to blocks
void Append(const uint8_t *data, size_t size) {
if (data_ == NULL) {
CHECK_EQ(0, size_);
CHECK_EQ(0, data_size_);
data_ = new uint8_t[Constants::BLOCK_SIZE];
data_size_ = Constants::BLOCK_SIZE;
}
if (size_ + size > data_size_) {
uint8_t *tmp = data_;
while (size_ + size > data_size_) {
data_size_ *= 2;
}
data_ = new uint8_t[data_size_];
memcpy(data_, tmp, size_);
delete [] tmp;
}
memcpy(data_ + size_, data, size);
size_ += size;
}
// For cleaing a block
void Reset() {
size_ = 0;
}
// Note: This does not benefit from -Wformat= checking, due to the
// enclosing template. Further, it was not used.
// void Print() const {
// xoff_t pos = 0;
// for (size_t i = 0; i < Size(); i++) {
// if (pos % 16 == 0) {
// DP(RINT "%5" Q "x: ", pos);
// }
// DP(RINT "%02x ", (*this)[i]);
// if (pos % 16 == 15) {
// DP(RINT "\n");
// }
// pos++;
// }
// DP(RINT "\n");
// }
void WriteTmpFile(TmpFile *f) const {
f->Append(this);
}
void SetSize(size_t size) {
uint8_t *t = NULL;
if (data_size_ < size) {
if (data_) {
t = data_;
}
data_ = new uint8_t[size];
data_size_ = size;
}
if (t && size < size_) {
memcpy(data_, t, size);
}
delete [] t;
size_ = size;
}
private:
friend class BlockIterator;
mutable uint8_t *data_;
mutable size_t data_size_;
size_t size_;
};
class FileSpec {
public:
FileSpec(MTRandom *rand)
: rand_(rand) {
}
// Generates a file with a known size
void GenerateFixedSize(xoff_t size) {
Reset();
for (xoff_t p = 0; p < size; ) {
xoff_t t = min(Constants::BLOCK_SIZE, size - p);
table_.insert(make_pair(p, Segment(t, rand_)));
p += t;
}
}
// Generates a file with exponential-random distributed size
void GenerateRandomSize(xoff_t mean) {
GenerateFixedSize(rand_->ExpRand(mean));
}
// Returns the size of the file
xoff_t Size() const {
if (table_.empty()) {
return 0;
}
ConstSegmentMapIterator i = --table_.end();
return i->first + i->second.Size();
}
// Returns the number of blocks
xoff_t Blocks(size_t blksize = Constants::BLOCK_SIZE) const {
if (table_.empty()) {
return 0;
}
return ((Size() - 1) / blksize) + 1;
}
// Returns the number of segments
xoff_t Segments() const {
return table_.size();
}
// Create a mutation according to "what".
void ModifyTo(const Mutator &mutator,
FileSpec *modify) const {
modify->Reset();
mutator.Mutate(&modify->table_, &table_, rand_);
modify->CheckSegments();
}
void CheckSegments() const {
for (ConstSegmentMapIterator iter(table_.begin());
iter != table_.end(); ) {
ConstSegmentMapIterator iter0(iter++);
if (iter == table_.end()) {
break;
}
CHECK_EQ(iter0->first + iter0->second.Size(), iter->first);
}
}
void Reset() {
table_.clear();
}
void Print() const {
for (ConstSegmentMapIterator iter(table_.begin());
iter != table_.end();
++iter) {
const Segment &seg = iter->second;
cerr << "Segment at " << iter->first
<< " (" << seg.ToString() << ")" << endl;
}
}
void PrintData() const {
Block block;
for (BlockIterator iter(*this); !iter.Done(); iter.Next()) {
iter.Get(&block);
block.Print();
}
}
void WriteTmpFile(TmpFile *f) const {
Block block;
for (BlockIterator iter(*this); !iter.Done(); iter.Next()) {
iter.Get(&block);
f->Append(&block);
}
}
void Get(Block *block, xoff_t offset, size_t size) const {
size_t got = 0;
block->SetSize(size);
ConstSegmentMapIterator pos = table_.upper_bound(offset);
if (pos == table_.begin()) {
CHECK_EQ(0, Size());
return;
}
--pos;
while (got < size) {
CHECK(pos != table_.end());
CHECK_GE(offset, pos->first);
const Segment &seg = pos->second;
// The position of this segment may start before this block starts,
// and then the position of the data may be offset from the seeding
// position.
size_t seg_offset = offset - pos->first;
size_t advance = min(seg.Size() - seg_offset,
size - got);
seg.Fill(seg_offset, advance, block->Data() + got);
got += advance;
offset += advance;
++pos;
}
}
typedef BlockIterator iterator;
private:
friend class BlockIterator;
MTRandom *rand_;
SegmentMap table_;
};
class BlockIterator {
public:
explicit BlockIterator(const FileSpec& spec)
: spec_(spec),
blkno_(0),
blksize_(Constants::BLOCK_SIZE) { }
BlockIterator(const FileSpec& spec,
size_t blksize)
: spec_(spec),
blkno_(0),
blksize_(blksize) { }
bool Done() const {
return blkno_ >= spec_.Blocks(blksize_);
}
void Next() {
blkno_++;
}
xoff_t Blkno() const {
return blkno_;
}
xoff_t Blocks() const {
return spec_.Blocks(blksize_);
}
xoff_t Offset() const {
return blkno_ * blksize_;
}
void SetBlock(xoff_t blkno) {
CHECK_LE(blkno, Blocks());
blkno_ = blkno;
}
void Get(Block *block) const {
spec_.Get(block, blkno_ * blksize_, BytesOnBlock());
}
size_t BytesOnBlock() const {
xoff_t blocks = spec_.Blocks(blksize_);
xoff_t size = spec_.Size();
DCHECK((blkno_ < blocks) ||
(blkno_ == blocks && size % blksize_ == 0));
if (blkno_ == blocks) {
return 0;
}
if (blkno_ + 1 == blocks) {
return ((size - 1) % blksize_) + 1;
}
return blksize_;
}
size_t BlockSize() const {
return blksize_;
}
private:
const FileSpec& spec_;
xoff_t blkno_;
size_t blksize_;
};
class ExtFile {
public:
ExtFile() {
static int static_counter = 0;
pid_t pid = getpid();
char buf[64];
xoff_t xpid = pid;
snprintf(buf, 64, "/tmp/regtest.%" Q "u.%d", xpid, static_counter++);
filename_.append(buf);
unlink(filename_.c_str());
}
~ExtFile() {
unlink(filename_.c_str());
}
const char* Name() const {
return filename_.c_str();
}
// Check whether a real file matches a file spec.
bool EqualsSpec(const FileSpec &spec) const {
main_file t;
main_file_init(&t);
CHECK_EQ(0, main_file_open(&t, Name(), XO_READ));
Block tblock;
Block sblock;
for (BlockIterator iter(spec); !iter.Done(); iter.Next()) {
iter.Get(&sblock);
tblock.SetSize(sblock.Size());
size_t tread;
CHECK_EQ(0, main_file_read(&t,
tblock.Data(),
tblock.Size(), &tread, "read failed"));
CHECK_EQ(0, CmpDifferentBlockBytes(tblock, sblock));
}
CHECK_EQ(0, main_file_close(&t));
main_file_cleanup(&t);
return true;
}
protected:
string filename_;
};
class TmpFile : public ExtFile {
public:
TmpFile() {
main_file_init(&file_);
CHECK_EQ(0, main_file_open(&file_, Name(), XO_WRITE));
}
~TmpFile() {
main_file_cleanup(&file_);
}
void Append(const Block *block) {
CHECK_EQ(0, main_file_write(&file_,
block->Data(), block->Size(),
"tmpfile write failed"));
}
const char* Name() const {
if (main_file_isopen(&file_)) {
CHECK_EQ(0, main_file_close(&file_));
}
return ExtFile::Name();
}
private:
mutable main_file file_;
};

400
lib/xdelta3/testing/modify.h Normal file
View file

@ -0,0 +1,400 @@
/* xdelta3 - delta compression tools and library -*- Mode: C++ -*-
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
class Mutator {
public:
virtual ~Mutator() { }
virtual void Mutate(SegmentMap *table,
const SegmentMap *source_table,
MTRandom *rand) const = 0;
};
class Change {
public:
enum Kind {
MODIFY = 1, // Mutate a certain range w/ random or supplied data
ADD = 2, // Insert random or supplied data
DELRANGE = 3, // Delete a specified range of data
COPY = 4, // Copy from one region, inserting elsewhere
MOVE = 5, // Copy then delete copied-from range
COPYOVER = 6 // Copy then delete copied-to range
// ADD, DELRANGE, and COPY change the file size
// MODIFY, MOVE, COPYOVER preserve the file size
};
// Constructor for modify, add, delete.
Change(Kind kind0, xoff_t size0, xoff_t addr1_0)
: kind(kind0),
size(size0),
addr1(addr1_0),
addr2(0),
insert(NULL) {
CHECK(kind != MOVE && kind != COPY && kind != COPYOVER);
}
// Constructor for modify, add w/ provided data.
Change(Kind kind0, xoff_t size0, xoff_t addr1_0, Segment *insert0)
: kind(kind0),
size(size0),
addr1(addr1_0),
addr2(0),
insert(insert0) {
CHECK(kind != MOVE && kind != COPY && kind != COPYOVER);
}
// Constructor for move, copy, overwrite
Change(Kind kind0, xoff_t size0, xoff_t addr1_0, xoff_t addr2_0)
: kind(kind0),
size(size0),
addr1(addr1_0),
addr2(addr2_0),
insert(NULL) {
CHECK(kind == MOVE || kind == COPY || kind == COPYOVER);
}
Kind kind;
xoff_t size;
xoff_t addr1;
xoff_t addr2;
Segment *insert; // For modify and/or add
};
typedef list<Change> ChangeList;
typedef typename ChangeList::const_iterator ConstChangeListIterator;
typedef typename ChangeList::iterator ChangeListIterator;
class ChangeListMutator : public Mutator {
public:
ChangeListMutator(const ChangeList &cl)
: cl_(cl) { }
ChangeListMutator() { }
void Mutate(SegmentMap *table,
const SegmentMap *source_table,
MTRandom *rand) const {
// The speed of processing gigabytes of data is so slow compared with
// these table-copy operations, no attempt to make this fast.
SegmentMap tmp;
for (ConstChangeListIterator iter(cl_.begin());
iter != cl_.end(); ++iter) {
const Change &ch = *iter;
tmp.clear();
Mutate(ch, &tmp, source_table, rand);
tmp.swap(*table);
source_table = table;
}
}
static void Mutate(const Change &ch,
SegmentMap *table,
const SegmentMap *source_table,
MTRandom *rand) {
switch (ch.kind) {
case Change::ADD:
AddChange(ch, table, source_table, rand);
break;
case Change::MODIFY:
ModifyChange(ch, table, source_table, rand);
break;
case Change::DELRANGE:
DeleteChange(ch, table, source_table, rand);
break;
case Change::COPY:
CopyChange(ch, table, source_table, rand);
break;
case Change::MOVE:
MoveChange(ch, table, source_table, rand);
break;
case Change::COPYOVER:
OverwriteChange(ch, table, source_table, rand);
break;
}
}
static void ModifyChange(const Change &ch,
SegmentMap *table,
const SegmentMap *source_table,
MTRandom *rand) {
xoff_t m_start = ch.addr1;
xoff_t m_end = m_start + ch.size;
xoff_t i_start = 0;
xoff_t i_end = 0;
for (ConstSegmentMapIterator iter(source_table->begin());
iter != source_table->end();
++iter) {
const Segment &seg = iter->second;
i_start = iter->first;
i_end = i_start + seg.Size();
if (i_end <= m_start || i_start >= m_end) {
table->insert(table->end(), make_pair(i_start, seg));
continue;
}
if (i_start < m_start) {
table->insert(table->end(),
make_pair(i_start,
seg.Subseg(0, m_start - i_start)));
}
// Insert the entire segment, even though it may extend into later
// segments. This condition avoids inserting it during later
// segments.
if (m_start >= i_start) {
if (ch.insert != NULL) {
table->insert(table->end(), make_pair(m_start, *ch.insert));
} else {
Segment part(m_end - m_start, rand);
table->insert(table->end(), make_pair(m_start, part));
}
}
if (i_end > m_end) {
table->insert(table->end(),
make_pair(m_end,
seg.Subseg(m_end - i_start, i_end - m_end)));
}
}
// This check verifies that the modify does not extend past the
// source_table EOF.
CHECK_LE(m_end, i_end);
}
static void AddChange(const Change &ch,
SegmentMap *table,
const SegmentMap *source_table,
MTRandom *rand) {
xoff_t m_start = ch.addr1;
xoff_t i_start = 0;
xoff_t i_end = 0;
for (ConstSegmentMapIterator iter(source_table->begin());
iter != source_table->end();
++iter) {
const Segment &seg = iter->second;
i_start = iter->first;
i_end = i_start + seg.Size();
if (i_end <= m_start) {
table->insert(table->end(), make_pair(i_start, seg));
continue;
}
if (i_start > m_start) {
table->insert(table->end(), make_pair(i_start + ch.size, seg));
continue;
}
if (i_start < m_start) {
table->insert(table->end(),
make_pair(i_start,
seg.Subseg(0, m_start - i_start)));
}
if (ch.insert != NULL) {
table->insert(table->end(), make_pair(m_start, *ch.insert));
} else {
Segment addseg(ch.size, rand);
table->insert(table->end(), make_pair(m_start, addseg));
}
if (m_start < i_end) {
table->insert(table->end(),
make_pair(m_start + ch.size,
seg.Subseg(m_start - i_start,
i_end - m_start)));
}
}
CHECK_LE(m_start, i_end);
// Special case for add at end-of-input.
if (m_start == i_end) {
Segment addseg(ch.size, rand);
table->insert(table->end(), make_pair(m_start, addseg));
}
}
static void DeleteChange(const Change &ch,
SegmentMap *table,
const SegmentMap *source_table,
MTRandom *rand) {
xoff_t m_start = ch.addr1;
xoff_t m_end = m_start + ch.size;
xoff_t i_start = 0;
xoff_t i_end = 0;
for (ConstSegmentMapIterator iter(source_table->begin());
iter != source_table->end();
++iter) {
const Segment &seg = iter->second;
i_start = iter->first;
i_end = i_start + seg.Size();
if (i_end <= m_start) {
table->insert(table->end(), make_pair(i_start, seg));
continue;
}
if (i_start >= m_end) {
table->insert(table->end(), make_pair(i_start - ch.size, seg));
continue;
}
if (i_start < m_start) {
table->insert(table->end(),
make_pair(i_start,
seg.Subseg(0, m_start - i_start)));
}
if (i_end > m_end) {
table->insert(table->end(),
make_pair(m_end - ch.size,
seg.Subseg(m_end - i_start, i_end - m_end)));
}
}
CHECK_LT(m_start, i_end);
CHECK_LE(m_end, i_end);
}
// A move is a copy followed by delete of the copied-from range.
static void MoveChange(const Change &ch,
SegmentMap *table,
const SegmentMap *source_table,
MTRandom *rand) {
SegmentMap tmp;
CHECK_NE(ch.addr1, ch.addr2);
CopyChange(ch, &tmp, source_table, rand);
Change d(Change::DELRANGE, ch.size,
ch.addr1 < ch.addr2 ? ch.addr1 : ch.addr1 + ch.size);
DeleteChange(d, table, &tmp, rand);
}
// An overwrite is a copy followed by a delete of the copied-to range.
static void OverwriteChange(const Change &ch,
SegmentMap *table,
const SegmentMap *source_table,
MTRandom *rand) {
SegmentMap tmp;
CHECK_NE(ch.addr1, ch.addr2);
CopyChange(ch, &tmp, source_table, rand);
Change d(Change::DELRANGE, ch.size, ch.addr2 + ch.size);
DeleteChange(d, table, &tmp, rand);
}
static void CopyChange(const Change &ch,
SegmentMap *table,
const SegmentMap *source_table,
MTRandom *ignore) {
xoff_t m_start = ch.addr2;
xoff_t c_start = ch.addr1;
xoff_t i_start = 0;
xoff_t i_end = 0;
// Like AddChange() with AppendCopy instead of a random segment.
for (ConstSegmentMapIterator iter(source_table->begin());
iter != source_table->end();
++iter) {
const Segment &seg = iter->second;
i_start = iter->first;
i_end = i_start + seg.Size();
if (i_end <= m_start) {
table->insert(table->end(), make_pair(i_start, seg));
continue;
}
if (i_start > m_start) {
table->insert(table->end(), make_pair(i_start + ch.size, seg));
continue;
}
if (i_start < m_start) {
table->insert(table->end(),
make_pair(i_start,
seg.Subseg(0, m_start - i_start)));
}
AppendCopy(table, source_table, c_start, m_start, ch.size);
if (m_start < i_end) {
table->insert(table->end(),
make_pair(m_start + ch.size,
seg.Subseg(m_start - i_start, i_end - m_start)));
}
}
CHECK_LE(m_start, i_end);
// Special case for copy to end-of-input.
if (m_start == i_end) {
AppendCopy(table, source_table, c_start, m_start, ch.size);
}
}
static void AppendCopy(SegmentMap *table,
const SegmentMap *source_table,
xoff_t copy_offset,
xoff_t append_offset,
xoff_t length) {
ConstSegmentMapIterator pos(source_table->upper_bound(copy_offset));
--pos;
xoff_t got = 0;
while (got < length) {
size_t seg_offset = copy_offset - pos->first;
size_t advance = min(pos->second.Size() - seg_offset,
(size_t)(length - got));
table->insert(table->end(),
make_pair(append_offset,
pos->second.Subseg(seg_offset,
advance)));
got += advance;
copy_offset += advance;
append_offset += advance;
++pos;
}
}
ChangeList* Changes() {
return &cl_;
}
const ChangeList* Changes() const {
return &cl_;
}
private:
ChangeList cl_;
};
class Modify1stByte : public Mutator {
public:
void Mutate(SegmentMap *table,
const SegmentMap *source_table,
MTRandom *rand) const {
ChangeListMutator::Mutate(Change(Change::MODIFY, 1, 0),
table, source_table, rand);
}
};

157
lib/xdelta3/testing/random.h Normal file
View file

@ -0,0 +1,157 @@
/* xdelta3 - delta compression tools and library -*- Mode: C++ -*-
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#undef MT_LEN
#undef MT_IA
class MTRandom {
public:
enum Constants {
MT_LEN = 624,
MT_IA = 397
};
static const uint32_t TEST_SEED1;
static const uint32_t UPPER_MASK;
static const uint32_t LOWER_MASK;
static const uint32_t MATRIX_A;
MTRandom() {
Init(TEST_SEED1);
}
explicit MTRandom(uint32_t seed) {
Init(seed);
}
/* This Mersenne Twister code is attributed to Michael Brundage. Thanks!
* http://www.qbrundage.com/michaelb/pubs/essays/random_number_generation.html
*/
uint32_t Rand32 () {
uint32_t y;
static unsigned long mag01[2] = {
0 , MATRIX_A
};
if (mt_index_ >= MT_LEN) {
int kk;
for (kk = 0; kk < MT_LEN - MT_IA; kk++) {
y = (mt_buffer_[kk] & UPPER_MASK) | (mt_buffer_[kk + 1] & LOWER_MASK);
mt_buffer_[kk] = mt_buffer_[kk + MT_IA] ^ (y >> 1) ^ mag01[y & 0x1UL];
}
for (;kk < MT_LEN - 1; kk++) {
y = (mt_buffer_[kk] & UPPER_MASK) | (mt_buffer_[kk + 1] & LOWER_MASK);
mt_buffer_[kk] = mt_buffer_[kk + (MT_IA - MT_LEN)] ^ (y >> 1) ^ mag01[y & 0x1UL];
}
y = (mt_buffer_[MT_LEN - 1] & UPPER_MASK) | (mt_buffer_[0] & LOWER_MASK);
mt_buffer_[MT_LEN - 1] = mt_buffer_[MT_IA - 1] ^ (y >> 1) ^ mag01[y & 0x1UL];
mt_index_ = 0;
}
y = mt_buffer_[mt_index_++];
y ^= (y >> 11);
y ^= (y << 7) & 0x9d2c5680UL;
y ^= (y << 15) & 0xefc60000UL;
y ^= (y >> 18);
return y;
}
uint32_t ExpRand32(uint32_t mean) {
double mean_d = mean;
double erand = log (1.0 / (Rand32() / (double)UINT32_MAX));
uint32_t x = (uint32_t) (mean_d * erand + 0.5);
return x;
}
uint64_t Rand64() {
return ((uint64_t)Rand32() << 32) | Rand32();
}
uint64_t ExpRand64(uint64_t mean) {
double mean_d = mean;
double erand = log (1.0 / (Rand64() / (double)UINT32_MAX));
uint64_t x = (uint64_t) (mean_d * erand + 0.5);
return x;
}
template <typename T>
T Rand() {
switch (sizeof(T)) {
case sizeof(uint32_t):
return Rand32();
case sizeof(uint64_t):
return Rand64();
default:
cerr << "Invalid sizeof T" << endl;
abort();
}
}
template <typename T>
T ExpRand(T mean) {
switch (sizeof(T)) {
case sizeof(uint32_t):
return ExpRand32(mean);
case sizeof(uint64_t):
return ExpRand64(mean);
default:
cerr << "Invalid sizeof T" << endl;
abort();
}
}
private:
void Init(uint32_t seed) {
mt_buffer_[0] = seed;
mt_index_ = MT_LEN;
for (int i = 1; i < MT_LEN; i++) {
/* See Knuth TAOCP Vol2. 3rd Ed. P.106 for multiplier. */
/* In the previous versions, MSBs of the seed affect */
/* only MSBs of the array mt[]. */
/* 2002/01/09 modified by Makoto Matsumoto */
mt_buffer_[i] =
(1812433253UL * (mt_buffer_[i-1] ^ (mt_buffer_[i-1] >> 30)) + i);
}
}
int mt_index_;
uint32_t mt_buffer_[MT_LEN];
};
const uint32_t MTRandom::TEST_SEED1 = 5489UL;
const uint32_t MTRandom::UPPER_MASK = 0x80000000;
const uint32_t MTRandom::LOWER_MASK = 0x7FFFFFFF;
const uint32_t MTRandom::MATRIX_A = 0x9908B0DF;
class MTRandom8 {
public:
MTRandom8(MTRandom *rand)
: rand_(rand) {
}
uint8_t Rand8() {
uint32_t r = rand_->Rand32();
// TODO: make this use a single byte at a time?
return (r & 0xff) ^ (r >> 7) ^ (r >> 15) ^ (r >> 21);
}
private:
MTRandom *rand_;
};

1321
lib/xdelta3/testing/regtest.cc Normal file
View file

File diff suppressed because it is too large Load diff

17
lib/xdelta3/testing/regtest_c.c Normal file
View file

@ -0,0 +1,17 @@
/* xdelta3 - delta compression tools and library
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#include "../xdelta3.c"

2
lib/xdelta3/testing/run_release.sh Normal file
View file

@ -0,0 +1,2 @@
#!/bin/sh
(cd .. && ./run_release.sh)

112
lib/xdelta3/testing/segment.h Normal file
View file

@ -0,0 +1,112 @@
/* xdelta3 - delta compression tools and library -*- Mode: C++ -*-
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
class Segment {
public:
Segment(size_t size, MTRandom *rand)
: size_(size),
seed_(rand->Rand32()),
seed_offset_(0),
data_(NULL) {
CHECK_GT(size_, 0);
}
Segment(size_t size, uint32_t seed)
: size_(size),
seed_(seed),
seed_offset_(0),
data_(NULL) {
CHECK_GT(size_, 0);
}
Segment(size_t size, uint8_t *data)
: size_(size),
seed_(0),
seed_offset_(0),
data_(data) {
CHECK_GT(size_, 0);
}
size_t Size() const {
return size_;
}
Segment Subseg(size_t start, size_t size) const {
CHECK_LE(start + size, size_);
if (data_) {
return Segment(size, data_ + start);
} else {
return Segment(size, seed_, seed_offset_ + start);
}
}
void Fill(size_t seg_offset, size_t size, uint8_t *data) const {
CHECK_LE(seg_offset + size, size_);
if (data_) {
memcpy(data, data_ + seg_offset, size);
} else {
size_t skip = seg_offset + seed_offset_;
MTRandom gen(seed_);
MTRandom8 gen8(&gen);
while (skip--) {
gen8.Rand8();
}
for (size_t i = 0; i < size; i++) {
data[i] = gen8.Rand8();
}
}
}
string ToString() const {
string r;
if (data_) {
for (size_t i = 0; i < size_; i++) {
char buf[10];
sprintf(buf, "%02x ", data_[i]);
r.append(buf);
}
} else {
char buf[256];
sprintf(buf, "size=%ld,seed=%ud,skip=%ld", size_, seed_, seed_offset_);
r.append(buf);
}
return r;
}
private:
// Used by Subseg()
Segment(size_t size, uint32_t seed, size_t seed_offset)
: size_(size),
seed_(seed),
seed_offset_(seed_offset),
data_(NULL) {
CHECK_GT(size_, 0);
}
size_t size_; // Size of this segment
// For random segments
uint32_t seed_; // Seed used for generating byte sequence
size_t seed_offset_; // Seed positions the sequence this many bytes
// before its beginning.
// For literal segments (data is not owned)
uint8_t *data_;
};
typedef map<xoff_t, Segment> SegmentMap;
typedef typename SegmentMap::const_iterator ConstSegmentMapIterator;
typedef typename SegmentMap::iterator SegmentMapIterator;

126
lib/xdelta3/testing/sizes.h Normal file
View file

@ -0,0 +1,126 @@
/* xdelta3 - delta compression tools and library -*- Mode: C++ -*-
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
template <typename T, typename U>
class SizeIterator {
public:
SizeIterator(MTRandom *rand, size_t howmany)
: rand_(rand),
count_(0),
fixed_(U::sizes),
fixed_size_(SIZEOF_ARRAY(U::sizes)),
howmany_(howmany) { }
T Get() {
if (count_ < fixed_size_) {
return fixed_[count_];
}
return rand_->Rand<T>() % U::max_value;
}
bool Done() {
return count_ >= fixed_size_ && count_ >= howmany_;
}
void Next() {
count_++;
}
private:
MTRandom *rand_;
size_t count_;
T* fixed_;
size_t fixed_size_;
size_t howmany_;
};
// Small sizes
class SmallSizes {
public:
static size_t sizes[];
static size_t max_value;
};
size_t SmallSizes::sizes[] = {
0, 1, 128 / 4, 3333,
128 - (128 / 3),
128,
128 + (128 / 3),
2 * 128 - (128 / 3),
2 * 128,
2 * 128 + (128 / 3),
};
size_t SmallSizes::max_value = 128 * 3;
// Large sizes
class LargeSizes {
public:
static size_t sizes[];
static size_t max_value;
};
size_t LargeSizes::sizes[] = {
1 << 20,
1 << 18,
1 << 16,
};
size_t LargeSizes::max_value = 1<<20;
// Base constants
struct BaseConstants {
static const size_t TEST_ROUNDS;
};
const size_t BaseConstants::TEST_ROUNDS = 10;
// Regtest<> arguments
struct SmallBlock : public BaseConstants {
static const xoff_t BLOCK_SIZE;
static const size_t WINDOW_SIZE;
typedef SmallSizes Sizes;
};
const xoff_t SmallBlock::BLOCK_SIZE = 1<<7;
const size_t SmallBlock::WINDOW_SIZE = 1<<7;
struct LargeBlock : public BaseConstants {
static const xoff_t BLOCK_SIZE;
static const size_t WINDOW_SIZE;
typedef LargeSizes Sizes;
};
const xoff_t LargeBlock::BLOCK_SIZE = (1 << 13);
const size_t LargeBlock::WINDOW_SIZE = (1 << 13);
struct MixedBlock : public BaseConstants {
static const xoff_t BLOCK_SIZE;
static const size_t WINDOW_SIZE;
typedef SmallSizes Sizes;
};
const xoff_t MixedBlock::BLOCK_SIZE = 1<<7;
const size_t MixedBlock::WINDOW_SIZE = 1<<8;
struct OversizeBlock : public BaseConstants {
static const xoff_t BLOCK_SIZE;
static const size_t WINDOW_SIZE;
typedef SmallSizes Sizes;
};
const xoff_t OversizeBlock::BLOCK_SIZE = 1<<8;
const size_t OversizeBlock::WINDOW_SIZE = 1<<7;

84
lib/xdelta3/testing/test.h Normal file
View file

@ -0,0 +1,84 @@
/* xdelta3 - delta compression tools and library -*- Mode: C++ -*-
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
extern "C" {
#include "../xdelta3.h"
#include "../xdelta3-internal.h"
}
#include <unistd.h>
#include <math.h>
#include <string>
#define CHECK_EQ(x,y) CHECK_OP(x,y,==)
#define CHECK_NE(x,y) CHECK_OP(x,y,!=)
#define CHECK_LT(x,y) CHECK_OP(x,y,<)
#define CHECK_GT(x,y) CHECK_OP(x,y,>)
#define CHECK_LE(x,y) CHECK_OP(x,y,<=)
#define CHECK_GE(x,y) CHECK_OP(x,y,>=)
#define CHECK_OP(x,y,OP) \
do { \
__typeof__(x) _x(x); \
__typeof__(x) _y(y); \
if (!(_x OP _y)) { \
cerr << __FILE__ << ":" << __LINE__ << " Check failed: " << #x " " #OP " " #y << endl; \
cerr << __FILE__ << ":" << __LINE__ << " {0} " << _x << endl; \
cerr << __FILE__ << ":" << __LINE__ << " {1} " << _y << endl; \
abort(); \
} } while (false)
#undef CHECK
#define CHECK(x) \
do {if (!(x)) { \
cerr << __FILE__ << ":" << __LINE__ << " Check failed: " << #x << endl; \
abort(); \
} } while (false)
#define DCHECK(x)
using std::string;
#include <vector>
using std::vector;
inline string CommandToString(const vector<const char*> &v) {
string s(v[0]);
for (size_t i = 1; i < v.size() && v[i] != NULL; i++) {
s.append(" ");
s.append(v[i]);
}
return s;
}
#include <iostream>
using std::cerr;
using std::endl;
using std::ostream;
#include <map>
using std::map;
using std::pair;
#include <list>
using std::list;
template <typename T, typename U>
pair<T, U> make_pair(const T& t, const U& u) {
return pair<T, U>(t, u);
}
using std::min;
using std::max;

1264
lib/xdelta3/testing/xdelta3-regtest.py Normal file
View file

File diff suppressed because it is too large Load diff

153
lib/xdelta3/testing/xdelta3-test.py Normal file
View file

@ -0,0 +1,153 @@
#!/usr/bin/python2.7
# xdelta3 - delta compression tools and library -*- Mode: C++ -*-
# Copyright 2016 Joshua MacDonald
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import xdelta3
# the test data section is expected to be len('target')
source = 'source source input0 source source'
target = 'source source target source source'
#
#
print 'encode: basic ...'
result, patch = xdelta3.xd3_encode_memory(target, source, 50)
assert result == 0
assert len(patch) < len(source)
print 'encode: adler32 ...'
result, patch_adler32 = xdelta3.xd3_encode_memory(target, source, 50,
xdelta3.XD3_ADLER32)
assert result == 0
assert len(patch_adler32) < len(source)
assert len(patch_adler32) > len(patch)
print 'encode: secondary ...'
result, patch_djw = xdelta3.xd3_encode_memory(target, source, 50,
xdelta3.XD3_SEC_DJW)
assert result == 0
# secondary compression doesn't help
assert len(patch_djw) > len(patch)
print 'encode: exact ...'
result, ignore = xdelta3.xd3_encode_memory(target, source, len(patch))
assert result == 0
assert len(ignore) < len(source)
print 'encode: out of space ...'
result, ignore = xdelta3.xd3_encode_memory(target, source, len(patch) - 1)
assert result == 28
assert ignore == None
print 'encode: zero space ...'
result, ignore = xdelta3.xd3_encode_memory(target, source, 0)
assert result == 28
assert ignore == None
print 'encode: no source ...'
result, zdata = xdelta3.xd3_encode_memory(target, None, 50)
assert result == 0
assert len(zdata) > len(patch)
print 'encode: no input ...'
result, ignore = xdelta3.xd3_encode_memory(None, None, 50)
assert result != 0
print 'decode: basic ...'
result, target1 = xdelta3.xd3_decode_memory(patch, source, len(target))
assert result == 0
assert len(target1) == len(target)
assert target1 == target
print 'decode: out of space ...'
result, ignore = xdelta3.xd3_decode_memory(patch, source, len(target) - 1)
assert result == 28
assert ignore == None
print 'decode: zero space ...'
result, ignore = xdelta3.xd3_decode_memory(patch, source, 0)
assert result == 28
assert ignore == None
print 'decode: single byte error ...'
# a few expected single-byte errors, e.g., unused address cache bits, see
# xdelta3-test.h's single-bit error tests
extra_count = 4
noverify_count = 0
for corrupt_pos in range(len(patch_adler32)):
input = ''.join([j == corrupt_pos and '\xff' or patch_adler32[j]
for j in range(len(patch_adler32))])
result, ignore = xdelta3.xd3_decode_memory(input, source, len(target), 0)
assert result == -17712
assert ignore == None
# without adler32 verification, the error may be in the data section which
# in this case is 6 bytes 'target'
result, corrupt = xdelta3.xd3_decode_memory(input, source, len(target),
xdelta3.XD3_ADLER32_NOVER)
if result == 0:
noverify_count = noverify_count + 1
#print "got %s" % corrupt
#end
#end
assert noverify_count == len('target') + extra_count
print 'decode: no source ...'
result, target2 = xdelta3.xd3_decode_memory(zdata, None, len(target))
assert result == 0
assert target == target2
# Test compression level setting via flags. assumes a 9 byte checksum
# and that level 9 steps 2, level 1 steps 15:
# 01234567890123456789012345678901
# level 1 only indexes 2 checksums "abcdefghi" and "ABCDEFGHI"
# outputs 43 vs. 23 bytes
print 'encode: compression level ...'
source = '_la_la_abcdefghi_la_la_ABCDEFGHI'
target = 'la_la_ABCDEFGH__la_la_abcdefgh__'
result1, level1 = xdelta3.xd3_encode_memory(target, source, 50, xdelta3.XD3_COMPLEVEL_1)
result9, level9 = xdelta3.xd3_encode_memory(target, source, 50, xdelta3.XD3_COMPLEVEL_9)
assert result1 == 0 and result9 == 0
assert len(level1) > len(level9)
#
# Issue 65
print 'encode: 65 ...'
source = 'Hello World'
target = 'Hello everyone'
result, patch = xdelta3.xd3_encode_memory(target, source, len(target))
assert result != 0
result, patch = xdelta3.xd3_encode_memory(target, source, 2 * len(target))
assert result == 0
print 'PASS'

557
lib/xdelta3/xdelta3-blkcache.h Normal file
View file

@ -0,0 +1,557 @@
/* xdelta3 - delta compression tools and library
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#include "xdelta3-internal.h"
typedef struct _main_blklru main_blklru;
typedef struct _main_blklru_list main_blklru_list;
#define XD3_INVALID_OFFSET XOFF_T_MAX
struct _main_blklru_list
{
main_blklru_list *next;
main_blklru_list *prev;
};
struct _main_blklru
{
uint8_t *blk;
xoff_t blkno;
usize_t size;
main_blklru_list link;
};
XD3_MAKELIST(main_blklru_list,main_blklru,link);
static usize_t lru_size = 0;
static main_blklru *lru = NULL; /* array of lru_size elts */
static main_blklru_list lru_list;
static int do_src_fifo = 0; /* set to avoid lru */
static int lru_hits = 0;
static int lru_misses = 0;
static int lru_filled = 0;
static void main_lru_reset (void)
{
lru_size = 0;
lru = NULL;
do_src_fifo = 0;
lru_hits = 0;
lru_misses = 0;
lru_filled = 0;
}
static void main_lru_cleanup (void)
{
if (lru != NULL)
{
main_buffree (lru[0].blk);
}
main_free (lru);
lru = NULL;
lru_hits = 0;
lru_misses = 0;
lru_filled = 0;
}
/* This is called at different times for encoding and decoding. The
* encoder calls it immediately, the decoder delays until the
* application header is received. */
static int
main_set_source (xd3_stream *stream, xd3_cmd cmd,
main_file *sfile, xd3_source *source)
{
int ret = 0;
usize_t i;
xoff_t source_size = 0;
usize_t blksize;
XD3_ASSERT (lru == NULL);
XD3_ASSERT (stream->src == NULL);
XD3_ASSERT (option_srcwinsz >= XD3_MINSRCWINSZ);
/* TODO: this code needs refactoring into FIFO, LRU, FAKE. Yuck!
* This is simplified from 3.0z which had issues with sizing the
* source buffer memory allocation and the source blocksize. */
/* LRU-specific */
main_blklru_list_init (& lru_list);
if (allow_fake_source)
{
/* TODO: refactor
* TOOLS/recode-specific: Check "allow_fake_source" mode looks
* broken now. */
sfile->mode = XO_READ;
sfile->realname = sfile->filename;
sfile->nread = 0;
}
else
{
/* Either a regular file (possibly compressed) or a FIFO
* (possibly compressed). */
if ((ret = main_file_open (sfile, sfile->filename, XO_READ)))
{
return ret;
}
/* If the file is regular we know it's size. If the file turns
* out to be externally compressed, size_known may change. */
sfile->size_known = (main_file_stat (sfile, &source_size) == 0);
}
/* Note: The API requires a power-of-two blocksize and srcwinsz
* (-B). The logic here will use a single block if the entire file
* is known to fit into srcwinsz. */
option_srcwinsz = xd3_xoff_roundup (option_srcwinsz);
/* Though called "lru", it is not LRU-specific. We always allocate
* a maximum number of source block buffers. If the entire file
* fits into srcwinsz, this buffer will stay as the only
* (lru_size==1) source block. Otherwise, we know that at least
* option_srcwinsz bytes are available. Split the source window
* into buffers. */
if ((lru = (main_blklru*) main_malloc (MAX_LRU_SIZE *
sizeof (main_blklru))) == NULL)
{
ret = ENOMEM;
return ret;
}
memset (lru, 0, sizeof(lru[0]) * MAX_LRU_SIZE);
/* Allocate the entire buffer. */
if ((lru[0].blk = (uint8_t*) main_bufalloc (option_srcwinsz)) == NULL)
{
ret = ENOMEM;
return ret;
}
/* Main calls main_getblk_func() once before xd3_set_source(). This
* is the point at which external decompression may begin. Set the
* system for a single block. */
lru_size = 1;
lru[0].blkno = XD3_INVALID_OFFSET;
blksize = option_srcwinsz;
main_blklru_list_push_back (& lru_list, & lru[0]);
XD3_ASSERT (blksize != 0);
/* Initialize xd3_source. */
source->blksize = blksize;
source->name = sfile->filename;
source->ioh = sfile;
source->curblkno = XD3_INVALID_OFFSET;
source->curblk = NULL;
source->max_winsize = option_srcwinsz;
if ((ret = main_getblk_func (stream, source, 0)) != 0)
{
XPR(NT "error reading source: %s: %s\n",
sfile->filename,
xd3_mainerror (ret));
return ret;
}
source->onblk = lru[0].size; /* xd3 sets onblk */
/* If the file is smaller than a block, size is known. */
if (!sfile->size_known && source->onblk < blksize)
{
source_size = source->onblk;
source->onlastblk = source_size;
sfile->size_known = 1;
}
/* If the size is not known or is greater than the buffer size, we
* split the buffer across MAX_LRU_SIZE blocks (already allocated in
* "lru"). */
if (!sfile->size_known || source_size > option_srcwinsz)
{
/* Modify block 0, change blocksize. */
blksize = option_srcwinsz / MAX_LRU_SIZE;
source->blksize = blksize;
source->onblk = blksize;
source->onlastblk = blksize;
source->max_blkno = MAX_LRU_SIZE - 1;
lru[0].size = blksize;
lru_size = MAX_LRU_SIZE;
/* Setup rest of blocks. */
for (i = 1; i < lru_size; i += 1)
{
lru[i].blk = lru[0].blk + (blksize * i);
lru[i].blkno = i;
lru[i].size = blksize;
main_blklru_list_push_back (& lru_list, & lru[i]);
}
}
if (! sfile->size_known)
{
/* If the size is not know, we must use FIFO discipline. */
do_src_fifo = 1;
}
/* Call the appropriate set_source method, handle errors, print
* verbose message, etc. */
if (sfile->size_known)
{
ret = xd3_set_source_and_size (stream, source, source_size);
}
else
{
ret = xd3_set_source (stream, source);
}
if (ret)
{
XPR(NT XD3_LIB_ERRMSG (stream, ret));
return ret;
}
XD3_ASSERT (stream->src == source);
XD3_ASSERT (source->blksize == blksize);
if (option_verbose)
{
static shortbuf srcszbuf;
static shortbuf srccntbuf;
static shortbuf winszbuf;
static shortbuf blkszbuf;
static shortbuf nbufs;
if (sfile->size_known)
{
short_sprintf (srcszbuf, "source size %s [%"Q"u]",
main_format_bcnt (source_size, &srccntbuf),
source_size);
}
else
{
short_sprintf (srcszbuf, "%s", "source size unknown");
}
nbufs.buf[0] = 0;
if (option_verbose > 1)
{
short_sprintf (nbufs, " #bufs %"W"u", lru_size);
}
XPR(NT "source %s %s blksize %s window %s%s%s\n",
sfile->filename,
srcszbuf.buf,
main_format_bcnt (blksize, &blkszbuf),
main_format_bcnt (option_srcwinsz, &winszbuf),
nbufs.buf,
do_src_fifo ? " (FIFO)" : "");
}
return 0;
}
static int
main_getblk_lru (xd3_source *source, xoff_t blkno,
main_blklru** blrup, int *is_new)
{
main_blklru *blru = NULL;
usize_t i;
(*is_new) = 0;
if (do_src_fifo)
{
/* Direct lookup assumes sequential scan w/o skipping blocks. */
int idx = blkno % lru_size;
blru = & lru[idx];
if (blru->blkno == blkno)
{
(*blrup) = blru;
return 0;
}
/* No going backwards in a sequential scan. */
if (blru->blkno != XD3_INVALID_OFFSET && blru->blkno > blkno)
{
return XD3_TOOFARBACK;
}
}
else
{
/* Sequential search through LRU. */
for (i = 0; i < lru_size; i += 1)
{
blru = & lru[i];
if (blru->blkno == blkno)
{
main_blklru_list_remove (blru);
main_blklru_list_push_back (& lru_list, blru);
(*blrup) = blru;
IF_DEBUG1 (DP(RINT "[getblk_lru] HIT blkno = %"Q"u lru_size=%"W"u\n",
blkno, lru_size));
return 0;
}
}
IF_DEBUG1 (DP(RINT "[getblk_lru] MISS blkno = %"Q"u lru_size=%"W"u\n",
blkno, lru_size));
}
if (do_src_fifo)
{
int idx = blkno % lru_size;
blru = & lru[idx];
}
else
{
XD3_ASSERT (! main_blklru_list_empty (& lru_list));
blru = main_blklru_list_pop_front (& lru_list);
main_blklru_list_push_back (& lru_list, blru);
}
lru_filled += 1;
(*is_new) = 1;
(*blrup) = blru;
blru->blkno = XD3_INVALID_OFFSET;
return 0;
}
static int
main_read_seek_source (xd3_stream *stream,
xd3_source *source,
xoff_t blkno) {
xoff_t pos = blkno * source->blksize;
main_file *sfile = (main_file*) source->ioh;
main_blklru *blru;
int is_new;
size_t nread = 0;
int ret = 0;
if (!sfile->seek_failed)
{
ret = main_file_seek (sfile, pos);
if (ret == 0)
{
sfile->source_position = pos;
}
}
if (sfile->seek_failed || ret != 0)
{
/* For an unseekable file (or other seek error, does it
* matter?) */
if (sfile->source_position > pos)
{
/* Could assert !IS_ENCODE(), this shouldn't happen
* because of do_src_fifo during encode. */
if (!option_quiet)
{
XPR(NT "source can't seek backwards; requested block offset "
"%"Q"u source position is %"Q"u\n",
pos, sfile->source_position);
}
sfile->seek_failed = 1;
stream->msg = "non-seekable source: "
"copy is too far back (try raising -B)";
return XD3_TOOFARBACK;
}
/* There's a chance here, that an genuine lseek error will cause
* xdelta3 to shift into non-seekable mode, entering a degraded
* condition. */
if (!sfile->seek_failed && option_verbose)
{
XPR(NT "source can't seek, will use FIFO for %s\n",
sfile->filename);
if (option_verbose > 1)
{
XPR(NT "seek error at offset %"Q"u: %s\n",
pos, xd3_mainerror (ret));
}
}
sfile->seek_failed = 1;
if (option_verbose > 1 && pos != sfile->source_position)
{
XPR(NT "non-seekable source skipping %"Q"u bytes @ %"Q"u\n",
pos - sfile->source_position,
sfile->source_position);
}
while (sfile->source_position < pos)
{
xoff_t skip_blkno;
usize_t skip_offset;
xd3_blksize_div (sfile->source_position, source,
&skip_blkno, &skip_offset);
/* Read past unused data */
XD3_ASSERT (pos - sfile->source_position >= source->blksize);
XD3_ASSERT (skip_offset == 0);
if ((ret = main_getblk_lru (source, skip_blkno,
& blru, & is_new)))
{
return ret;
}
XD3_ASSERT (is_new);
blru->blkno = skip_blkno;
if ((ret = main_read_primary_input (sfile,
(uint8_t*) blru->blk,
source->blksize,
& nread)))
{
return ret;
}
if (nread != source->blksize)
{
IF_DEBUG1 (DP(RINT "[getblk] short skip block nread = %"Z"u\n",
nread));
stream->msg = "non-seekable input is short";
return XD3_INVALID_INPUT;
}
sfile->source_position += nread;
blru->size = nread;
IF_DEBUG1 (DP(RINT "[getblk] skip blkno %"Q"u size %"W"u\n",
skip_blkno, blru->size));
XD3_ASSERT (sfile->source_position <= pos);
}
}
return 0;
}
/* This is the callback for reading a block of source. This function
* is blocking and it implements a small LRU.
*
* Note that it is possible for main_input() to handle getblk requests
* in a non-blocking manner. If the callback is NULL then the caller
* of xd3_*_input() must handle the XD3_GETSRCBLK return value and
* fill the source in the same way. See xd3_getblk for details. To
* see an example of non-blocking getblk, see xdelta-test.h. */
static int
main_getblk_func (xd3_stream *stream,
xd3_source *source,
xoff_t blkno)
{
int ret = 0;
xoff_t pos = blkno * source->blksize;
main_file *sfile = (main_file*) source->ioh;
main_blklru *blru;
int is_new;
size_t nread = 0;
if (allow_fake_source)
{
source->curblkno = blkno;
source->onblk = 0;
source->curblk = lru[0].blk;
lru[0].size = 0;
return 0;
}
if ((ret = main_getblk_lru (source, blkno, & blru, & is_new)))
{
return ret;
}
if (!is_new)
{
source->curblkno = blkno;
source->onblk = blru->size;
source->curblk = blru->blk;
lru_hits++;
return 0;
}
lru_misses += 1;
if (pos != sfile->source_position)
{
/* Only try to seek when the position is wrong. This means the
* decoder will fail when the source buffer is too small, but
* only when the input is non-seekable. */
if ((ret = main_read_seek_source (stream, source, blkno)))
{
return ret;
}
}
XD3_ASSERT (sfile->source_position == pos);
if ((ret = main_read_primary_input (sfile,
(uint8_t*) blru->blk,
source->blksize,
& nread)))
{
return ret;
}
/* Save the last block read, used to handle non-seekable files. */
sfile->source_position = pos + nread;
if (option_verbose > 3)
{
if (blru->blkno != XD3_INVALID_OFFSET)
{
if (blru->blkno != blkno)
{
XPR(NT "source block %"Q"u read %"Z"u ejects %"Q"u (lru_hits=%u, "
"lru_misses=%u, lru_filled=%u)\n",
blkno, nread, blru->blkno, lru_hits, lru_misses, lru_filled);
}
else
{
XPR(NT "source block %"Q"u read %"Z"u (lru_hits=%u, "
"lru_misses=%u, lru_filled=%u)\n",
blkno, nread, lru_hits, lru_misses, lru_filled);
}
}
else
{
XPR(NT "source block %"Q"u read %"Z"u (lru_hits=%u, lru_misses=%u, "
"lru_filled=%u)\n", blkno, nread,
lru_hits, lru_misses, lru_filled);
}
}
source->curblk = blru->blk;
source->curblkno = blkno;
source->onblk = nread;
blru->size = nread;
blru->blkno = blkno;
IF_DEBUG1 (DP(RINT "[main_getblk] blkno %"Q"u onblk %"Z"u pos %"Q"u "
"srcpos %"Q"u\n",
blkno, nread, pos, sfile->source_position));
return 0;
}

171
lib/xdelta3/xdelta3-cfgs.h Normal file
View file

@ -0,0 +1,171 @@
/* xdelta3 - delta compression tools and library
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
/******************************************************************
SOFT string matcher
******************************************************************/
#if XD3_BUILD_SOFT
#define TEMPLATE soft
#define LLOOK stream->smatcher.large_look
#define LSTEP stream->smatcher.large_step
#define SLOOK stream->smatcher.small_look
#define SCHAIN stream->smatcher.small_chain
#define SLCHAIN stream->smatcher.small_lchain
#define MAXLAZY stream->smatcher.max_lazy
#define LONGENOUGH stream->smatcher.long_enough
#define SOFTCFG 1
#include "xdelta3.c"
#undef SOFTCFG
#undef TEMPLATE
#undef LLOOK
#undef SLOOK
#undef LSTEP
#undef SCHAIN
#undef SLCHAIN
#undef MAXLAZY
#undef LONGENOUGH
#endif
#define SOFTCFG 0
/************************************************************
FASTEST string matcher
**********************************************************/
#if XD3_BUILD_FASTEST
#define TEMPLATE fastest
#define LLOOK 9
#define LSTEP 26
#define SLOOK 4U
#define SCHAIN 1
#define SLCHAIN 1
#define MAXLAZY 6
#define LONGENOUGH 6
#include "xdelta3.c"
#undef TEMPLATE
#undef LLOOK
#undef SLOOK
#undef LSTEP
#undef SCHAIN
#undef SLCHAIN
#undef MAXLAZY
#undef LONGENOUGH
#endif
/************************************************************
FASTER string matcher
**********************************************************/
#if XD3_BUILD_FASTER
#define TEMPLATE faster
#define LLOOK 9
#define LSTEP 15
#define SLOOK 4U
#define SCHAIN 1
#define SLCHAIN 1
#define MAXLAZY 18
#define LONGENOUGH 18
#include "xdelta3.c"
#undef TEMPLATE
#undef LLOOK
#undef SLOOK
#undef LSTEP
#undef SCHAIN
#undef SLCHAIN
#undef MAXLAZY
#undef LONGENOUGH
#endif
/******************************************************
FAST string matcher
********************************************************/
#if XD3_BUILD_FAST
#define TEMPLATE fast
#define LLOOK 9
#define LSTEP 8
#define SLOOK 4U
#define SCHAIN 4
#define SLCHAIN 1
#define MAXLAZY 18
#define LONGENOUGH 35
#include "xdelta3.c"
#undef TEMPLATE
#undef LLOOK
#undef SLOOK
#undef LSTEP
#undef SCHAIN
#undef SLCHAIN
#undef MAXLAZY
#undef LONGENOUGH
#endif
/**************************************************
SLOW string matcher
**************************************************************/
#if XD3_BUILD_SLOW
#define TEMPLATE slow
#define LLOOK 9
#define LSTEP 2
#define SLOOK 4U
#define SCHAIN 44
#define SLCHAIN 13
#define MAXLAZY 90
#define LONGENOUGH 70
#include "xdelta3.c"
#undef TEMPLATE
#undef LLOOK
#undef SLOOK
#undef LSTEP
#undef SCHAIN
#undef SLCHAIN
#undef MAXLAZY
#undef LONGENOUGH
#endif
/********************************************************
DEFAULT string matcher
************************************************************/
#if XD3_BUILD_DEFAULT
#define TEMPLATE default
#define LLOOK 9
#define LSTEP 3
#define SLOOK 4U
#define SCHAIN 8
#define SLCHAIN 2
#define MAXLAZY 36
#define LONGENOUGH 70
#include "xdelta3.c"
#undef TEMPLATE
#undef LLOOK
#undef SLOOK
#undef LSTEP
#undef SCHAIN
#undef SLCHAIN
#undef MAXLAZY
#undef LONGENOUGH
#endif

1219
lib/xdelta3/xdelta3-decode.h Normal file
View file

File diff suppressed because it is too large Load diff

1835
lib/xdelta3/xdelta3-djw.h Normal file
View file

File diff suppressed because it is too large Load diff

857
lib/xdelta3/xdelta3-fgk.h Normal file
View file

@ -0,0 +1,857 @@
/* xdelta3 - delta compression tools and library
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
For demonstration purposes only.
*/
#ifndef _XDELTA3_FGK_h_
#define _XDELTA3_FGK_h_
/* An implementation of the FGK algorithm described by D.E. Knuth in
* "Dynamic Huffman Coding" in Journal of Algorithms 6. */
/* A 32bit counter (fgk_weight) is used as the frequency counter for
* nodes in the huffman tree. TODO: Need oto test for overflow and/or
* reset stats. */
typedef struct _fgk_stream fgk_stream;
typedef struct _fgk_node fgk_node;
typedef struct _fgk_block fgk_block;
typedef unsigned int fgk_bit;
typedef uint32_t fgk_weight;
struct _fgk_block {
union {
fgk_node *un_leader;
fgk_block *un_freeptr;
} un;
};
#define block_leader un.un_leader
#define block_freeptr un.un_freeptr
/* The code can also support fixed huffman encoding/decoding. */
#define IS_ADAPTIVE 1
/* weight is a count of the number of times this element has been seen
* in the current encoding/decoding. parent, right_child, and
* left_child are pointers defining the tree structure. right and
* left point to neighbors in an ordered sequence of weights. The
* left child of a node is always guaranteed to have weight not
* greater than its sibling. fgk_blockLeader points to the element
* with the same weight as itself which is closest to the next
* increasing weight block. */
struct _fgk_node
{
fgk_weight weight;
fgk_node *parent;
fgk_node *left_child;
fgk_node *right_child;
fgk_node *left;
fgk_node *right;
fgk_block *my_block;
};
/* alphabet_size is the a count of the number of possible leaves in
* the huffman tree. The number of total nodes counting internal
* nodes is ((2 * alphabet_size) - 1). zero_freq_count is the number
* of elements remaining which have zero frequency. zero_freq_exp and
* zero_freq_rem satisfy the equation zero_freq_count =
* 2^zero_freq_exp + zero_freq_rem. root_node is the root of the
* tree, which is initialized to a node with zero frequency and
* contains the 0th such element. free_node contains a pointer to the
* next available fgk_node space. alphabet contains all the elements
* and is indexed by N. remaining_zeros points to the head of the
* list of zeros. */
struct _fgk_stream
{
usize_t alphabet_size;
usize_t zero_freq_count;
usize_t zero_freq_exp;
usize_t zero_freq_rem;
usize_t coded_depth;
usize_t total_nodes;
usize_t total_blocks;
fgk_bit *coded_bits;
fgk_block *block_array;
fgk_block *free_block;
fgk_node *decode_ptr;
fgk_node *remaining_zeros;
fgk_node *alphabet;
fgk_node *root_node;
fgk_node *free_node;
};
/*********************************************************************/
/* Encoder */
/*********************************************************************/
static fgk_stream* fgk_alloc (xd3_stream *stream /*, usize_t alphabet_size */);
static int fgk_init (xd3_stream *stream,
fgk_stream *h,
int is_encode);
static usize_t fgk_encode_data (fgk_stream *h,
usize_t n);
static inline fgk_bit fgk_get_encoded_bit (fgk_stream *h);
static int xd3_encode_fgk (xd3_stream *stream,
fgk_stream *sec_stream,
xd3_output *input,
xd3_output *output,
xd3_sec_cfg *cfg);
/*********************************************************************/
/* Decoder */
/*********************************************************************/
static inline int fgk_decode_bit (fgk_stream *h,
fgk_bit b);
static usize_t fgk_decode_data (fgk_stream *h);
static void fgk_destroy (xd3_stream *stream,
fgk_stream *h);
static int xd3_decode_fgk (xd3_stream *stream,
fgk_stream *sec_stream,
const uint8_t **input,
const uint8_t *const input_end,
uint8_t **output,
const uint8_t *const output_end);
/*********************************************************************/
/* Private */
/*********************************************************************/
static unsigned int fgk_find_nth_zero (fgk_stream *h, usize_t n);
static usize_t fgk_nth_zero (fgk_stream *h, usize_t n);
static void fgk_update_tree (fgk_stream *h, usize_t n);
static fgk_node* fgk_increase_zero_weight (fgk_stream *h, usize_t n);
static void fgk_eliminate_zero (fgk_stream* h, fgk_node *node);
static void fgk_move_right (fgk_stream *h, fgk_node *node);
static void fgk_promote (fgk_stream *h, fgk_node *node);
static void fgk_init_node (fgk_node *node, usize_t i, usize_t size);
static fgk_block* fgk_make_block (fgk_stream *h, fgk_node *l);
static void fgk_free_block (fgk_stream *h, fgk_block *b);
static void fgk_factor_remaining (fgk_stream *h);
static inline void fgk_swap_ptrs (fgk_node **one, fgk_node **two);
/*********************************************************************/
/* Basic Routines */
/*********************************************************************/
/* returns an initialized huffman encoder for an alphabet with the
* given size. returns NULL if enough memory cannot be allocated */
static fgk_stream* fgk_alloc (xd3_stream *stream /*, int alphabet_size0 */)
{
usize_t alphabet_size0 = ALPHABET_SIZE;
fgk_stream *h;
if ((h = (fgk_stream*) xd3_alloc (stream, 1, sizeof (fgk_stream))) == NULL)
{
return NULL;
}
h->total_nodes = (2 * alphabet_size0) - 1;
h->total_blocks = (2 * h->total_nodes);
h->alphabet = (fgk_node*) xd3_alloc (stream, h->total_nodes, sizeof (fgk_node));
h->block_array = (fgk_block*) xd3_alloc (stream, h->total_blocks, sizeof (fgk_block));
h->coded_bits = (fgk_bit*) xd3_alloc (stream, alphabet_size0, sizeof (fgk_bit));
if (h->coded_bits == NULL ||
h->alphabet == NULL ||
h->block_array == NULL)
{
fgk_destroy (stream, h);
return NULL;
}
h->alphabet_size = alphabet_size0;
return h;
}
static int fgk_init (xd3_stream *stream, fgk_stream *h, int is_encode)
{
usize_t ui;
ssize_t si;
h->root_node = h->alphabet;
h->decode_ptr = h->root_node;
h->free_node = h->alphabet + h->alphabet_size;
h->remaining_zeros = h->alphabet;
h->coded_depth = 0;
h->zero_freq_count = h->alphabet_size + 2;
/* after two calls to factor_remaining, zero_freq_count == alphabet_size */
fgk_factor_remaining(h); /* set ZFE and ZFR */
fgk_factor_remaining(h); /* set ZFDB according to prev state */
IF_DEBUG (memset (h->alphabet, 0, sizeof (h->alphabet[0]) * h->total_nodes));
for (ui = 0; ui < h->total_blocks-1; ui += 1)
{
h->block_array[ui].block_freeptr = &h->block_array[ui + 1];
}
h->block_array[h->total_blocks - 1].block_freeptr = NULL;
h->free_block = h->block_array;
/* Zero frequency nodes are inserted in the first alphabet_size
* positions, with Value, weight, and a pointer to the next zero
* frequency node. */
for (si = h->alphabet_size - 1; si >= 0; si -= 1)
{
fgk_init_node (h->alphabet + si, (usize_t) si, h->alphabet_size);
}
return 0;
}
static void fgk_swap_ptrs(fgk_node **one, fgk_node **two)
{
fgk_node *tmp = *one;
*one = *two;
*two = tmp;
}
/* Takes huffman transmitter h and n, the nth elt in the alphabet, and
* returns the number of required to encode n. */
static usize_t fgk_encode_data (fgk_stream* h, usize_t n)
{
fgk_node *target_ptr = h->alphabet + n;
XD3_ASSERT (n < h->alphabet_size);
h->coded_depth = 0;
/* First encode the binary representation of the nth remaining
* zero frequency element in reverse such that bit, which will be
* encoded from h->coded_depth down to 0 will arrive in increasing
* order following the tree path. If there is only one left, it
* is not neccesary to encode these bits. */
if (IS_ADAPTIVE && target_ptr->weight == 0)
{
usize_t where, shift;
usize_t bits;
where = fgk_find_nth_zero(h, n);
shift = 1;
if (h->zero_freq_rem == 0)
{
bits = h->zero_freq_exp;
}
else
{
bits = h->zero_freq_exp + 1;
}
while (bits > 0)
{
h->coded_bits[h->coded_depth++] = (shift & where) && 1;
bits -= 1;
shift <<= 1;
};
target_ptr = h->remaining_zeros;
}
/* The path from root to node is filled into coded_bits in reverse so
* that it is encoded in the right order */
while (target_ptr != h->root_node)
{
h->coded_bits[h->coded_depth++] = (target_ptr->parent->right_child == target_ptr);
target_ptr = target_ptr->parent;
}
if (IS_ADAPTIVE)
{
fgk_update_tree(h, n);
}
return h->coded_depth;
}
/* Should be called as many times as fgk_encode_data returns.
*/
static inline fgk_bit fgk_get_encoded_bit (fgk_stream *h)
{
XD3_ASSERT (h->coded_depth > 0);
return h->coded_bits[--h->coded_depth];
}
/* This procedure updates the tree after alphabet[n] has been encoded
* or decoded.
*/
static void fgk_update_tree (fgk_stream *h, usize_t n)
{
fgk_node *incr_node;
if (h->alphabet[n].weight == 0)
{
incr_node = fgk_increase_zero_weight (h, n);
}
else
{
incr_node = h->alphabet + n;
}
while (incr_node != h->root_node)
{
fgk_move_right (h, incr_node);
fgk_promote (h, incr_node);
incr_node->weight += 1; /* incr the parent */
incr_node = incr_node->parent; /* repeat */
}
h->root_node->weight += 1;
}
static void fgk_move_right (fgk_stream *h, fgk_node *move_fwd)
{
fgk_node **fwd_par_ptr, **back_par_ptr;
fgk_node *move_back, *tmp;
move_back = move_fwd->my_block->block_leader;
if (move_fwd == move_back ||
move_fwd->parent == move_back ||
move_fwd->weight == 0)
{
return;
}
move_back->right->left = move_fwd;
if (move_fwd->left)
{
move_fwd->left->right = move_back;
}
tmp = move_fwd->right;
move_fwd->right = move_back->right;
if (tmp == move_back)
{
move_back->right = move_fwd;
}
else
{
tmp->left = move_back;
move_back->right = tmp;
}
tmp = move_back->left;
move_back->left = move_fwd->left;
if (tmp == move_fwd)
{
move_fwd->left = move_back;
}
else
{
tmp->right = move_fwd;
move_fwd->left = tmp;
}
if (move_fwd->parent->right_child == move_fwd)
{
fwd_par_ptr = &move_fwd->parent->right_child;
}
else
{
fwd_par_ptr = &move_fwd->parent->left_child;
}
if (move_back->parent->right_child == move_back)
{
back_par_ptr = &move_back->parent->right_child;
}
else
{
back_par_ptr = &move_back->parent->left_child;
}
fgk_swap_ptrs (&move_fwd->parent, &move_back->parent);
fgk_swap_ptrs (fwd_par_ptr, back_par_ptr);
move_fwd->my_block->block_leader = move_fwd;
}
/* Shifts node, the leader of its block, into the next block. */
static void fgk_promote (fgk_stream *h, fgk_node *node)
{
fgk_node *my_left, *my_right;
fgk_block *cur_block;
my_right = node->right;
my_left = node->left;
cur_block = node->my_block;
if (node->weight == 0)
{
return;
}
/* if left is right child, parent of remaining zeros case (?), means parent
* has same weight as right child. */
if (my_left == node->right_child &&
node->left_child &&
node->left_child->weight == 0)
{
XD3_ASSERT (node->left_child == h->remaining_zeros);
XD3_ASSERT (node->right_child->weight == (node->weight+1)); /* child weight was already incremented */
if (node->weight == (my_right->weight - 1) && my_right != h->root_node)
{
fgk_free_block (h, cur_block);
node->my_block = my_right->my_block;
my_left->my_block = my_right->my_block;
}
return;
}
if (my_left == h->remaining_zeros)
{
return;
}
/* true if not the leftmost node */
if (my_left->my_block == cur_block)
{
my_left->my_block->block_leader = my_left;
}
else
{
fgk_free_block (h, cur_block);
}
/* node->parent != my_right */
if ((node->weight == (my_right->weight - 1)) && (my_right != h->root_node))
{
node->my_block = my_right->my_block;
}
else
{
node->my_block = fgk_make_block (h, node);
}
}
/* When an element is seen the first time this is called to remove it from the list of
* zero weight elements and introduce a new internal node to the tree. */
static fgk_node* fgk_increase_zero_weight (fgk_stream *h, usize_t n)
{
fgk_node *this_zero, *new_internal, *zero_ptr;
this_zero = h->alphabet + n;
if (h->zero_freq_count == 1)
{
/* this is the last one */
this_zero->right_child = NULL;
if (this_zero->right->weight == 1)
{
this_zero->my_block = this_zero->right->my_block;
}
else
{
this_zero->my_block = fgk_make_block (h, this_zero);
}
h->remaining_zeros = NULL;
return this_zero;
}
zero_ptr = h->remaining_zeros;
new_internal = h->free_node++;
new_internal->parent = zero_ptr->parent;
new_internal->right = zero_ptr->right;
new_internal->weight = 0;
new_internal->right_child = this_zero;
new_internal->left = this_zero;
if (h->remaining_zeros == h->root_node)
{
/* This is the first element to be coded */
h->root_node = new_internal;
this_zero->my_block = fgk_make_block (h, this_zero);
new_internal->my_block = fgk_make_block (h, new_internal);
}
else
{
new_internal->right->left = new_internal;
if (zero_ptr->parent->right_child == zero_ptr)
{
zero_ptr->parent->right_child = new_internal;
}
else
{
zero_ptr->parent->left_child = new_internal;
}
if (new_internal->right->weight == 1)
{
new_internal->my_block = new_internal->right->my_block;
}
else
{
new_internal->my_block = fgk_make_block (h, new_internal);
}
this_zero->my_block = new_internal->my_block;
}
fgk_eliminate_zero (h, this_zero);
new_internal->left_child = h->remaining_zeros;
this_zero->right = new_internal;
this_zero->left = h->remaining_zeros;
this_zero->parent = new_internal;
this_zero->left_child = NULL;
this_zero->right_child = NULL;
h->remaining_zeros->parent = new_internal;
h->remaining_zeros->right = this_zero;
return this_zero;
}
/* When a zero frequency element is encoded, it is followed by the
* binary representation of the index into the remaining elements.
* Sets a cache to the element before it so that it can be removed
* without calling this procedure again. */
static unsigned int fgk_find_nth_zero (fgk_stream* h, usize_t n)
{
fgk_node *target_ptr = h->alphabet + n;
fgk_node *head_ptr = h->remaining_zeros;
unsigned int idx = 0;
while (target_ptr != head_ptr)
{
head_ptr = head_ptr->right_child;
idx += 1;
}
return idx;
}
/* Splices node out of the list of zeros. */
static void fgk_eliminate_zero (fgk_stream* h, fgk_node *node)
{
if (h->zero_freq_count == 1)
{
return;
}
fgk_factor_remaining(h);
if (node->left_child == NULL)
{
h->remaining_zeros = h->remaining_zeros->right_child;
h->remaining_zeros->left_child = NULL;
}
else if (node->right_child == NULL)
{
node->left_child->right_child = NULL;
}
else
{
node->right_child->left_child = node->left_child;
node->left_child->right_child = node->right_child;
}
}
static void fgk_init_node (fgk_node *node, usize_t i, usize_t size)
{
if (i < size - 1)
{
node->right_child = node + 1;
}
else
{
node->right_child = NULL;
}
if (i >= 1)
{
node->left_child = node - 1;
}
else
{
node->left_child = NULL;
}
node->weight = 0;
node->parent = NULL;
node->right = NULL;
node->left = NULL;
node->my_block = NULL;
}
/* The data structure used is an array of blocks, which are unions of
* free pointers and huffnode pointers. free blocks are a linked list
* of free blocks, the front of which is h->free_block. The used
* blocks are pointers to the head of each block. */
static fgk_block* fgk_make_block (fgk_stream *h, fgk_node* lead)
{
fgk_block *ret = h->free_block;
XD3_ASSERT (h->free_block != NULL);
h->free_block = h->free_block->block_freeptr;
ret->block_leader = lead;
return ret;
}
/* Restores the block to the front of the free list. */
static void fgk_free_block (fgk_stream *h, fgk_block *b)
{
b->block_freeptr = h->free_block;
h->free_block = b;
}
/* sets zero_freq_count, zero_freq_rem, and zero_freq_exp to satsity
* the equation given above. */
static void fgk_factor_remaining (fgk_stream *h)
{
unsigned int i;
i = (--h->zero_freq_count);
h->zero_freq_exp = 0;
while (i > 1)
{
h->zero_freq_exp += 1;
i >>= 1;
}
i = 1 << h->zero_freq_exp;
h->zero_freq_rem = h->zero_freq_count - i;
}
/* receives a bit at a time and returns true when a complete code has
* been received.
*/
static inline int fgk_decode_bit (fgk_stream* h, fgk_bit b)
{
XD3_ASSERT (b == 1 || b == 0);
if (IS_ADAPTIVE && h->decode_ptr->weight == 0)
{
usize_t bitsreq;
if (h->zero_freq_rem == 0)
{
bitsreq = h->zero_freq_exp;
}
else
{
bitsreq = h->zero_freq_exp + 1;
}
h->coded_bits[h->coded_depth] = b;
h->coded_depth += 1;
return h->coded_depth >= bitsreq;
}
else
{
if (b)
{
h->decode_ptr = h->decode_ptr->right_child;
}
else
{
h->decode_ptr = h->decode_ptr->left_child;
}
if (h->decode_ptr->left_child == NULL)
{
/* If the weight is non-zero, finished. */
if (h->decode_ptr->weight != 0)
{
return 1;
}
/* zero_freq_count is dropping to 0, finished. */
return h->zero_freq_count == 1;
}
else
{
return 0;
}
}
}
static usize_t fgk_nth_zero (fgk_stream* h, usize_t n)
{
fgk_node *ret = h->remaining_zeros;
/* ERROR: if during this loop (ret->right_child == NULL) then the
* encoder's zero count is too high. Could return an error code
* now, but is probably unnecessary overhead, since the caller
* should check integrity anyway. */
for (; n != 0 && ret->right_child != NULL; n -= 1)
{
ret = ret->right_child;
}
return (usize_t)(ret - h->alphabet);
}
/* once fgk_decode_bit returns 1, this retrieves an index into the
* alphabet otherwise this returns 0, indicating more bits are
* required.
*/
static usize_t fgk_decode_data (fgk_stream* h)
{
usize_t elt = (usize_t)(h->decode_ptr - h->alphabet);
if (IS_ADAPTIVE && h->decode_ptr->weight == 0) {
usize_t i = 0;
usize_t n = 0;
if (h->coded_depth > 0)
{
for (; i < h->coded_depth - 1; i += 1)
{
n |= h->coded_bits[i];
n <<= 1;
}
}
n |= h->coded_bits[i];
elt = fgk_nth_zero(h, n);
}
h->coded_depth = 0;
if (IS_ADAPTIVE)
{
fgk_update_tree(h, elt);
}
h->decode_ptr = h->root_node;
return elt;
}
static void fgk_destroy (xd3_stream *stream,
fgk_stream *h)
{
if (h != NULL)
{
xd3_free (stream, h->alphabet);
xd3_free (stream, h->coded_bits);
xd3_free (stream, h->block_array);
xd3_free (stream, h);
}
}
/*********************************************************************/
/* Xdelta */
/*********************************************************************/
static int
xd3_encode_fgk (xd3_stream *stream, fgk_stream *sec_stream, xd3_output *input, xd3_output *output, xd3_sec_cfg *cfg)
{
bit_state bstate = BIT_STATE_ENCODE_INIT;
xd3_output *cur_page;
int ret;
/* OPT: quit compression early if it looks bad */
for (cur_page = input; cur_page; cur_page = cur_page->next_page)
{
const uint8_t *inp = cur_page->base;
const uint8_t *inp_max = inp + cur_page->next;
while (inp < inp_max)
{
usize_t bits = fgk_encode_data (sec_stream, *inp++);
while (bits--)
{
if ((ret = xd3_encode_bit (stream, & output, & bstate, fgk_get_encoded_bit (sec_stream)))) { return ret; }
}
}
}
return xd3_flush_bits (stream, & output, & bstate);
}
static int
xd3_decode_fgk (xd3_stream *stream,
fgk_stream *sec_stream,
const uint8_t **input_pos,
const uint8_t *const input_max,
uint8_t **output_pos,
const uint8_t *const output_max)
{
bit_state bstate;
uint8_t *output = *output_pos;
const uint8_t *input = *input_pos;
for (;;)
{
if (input == input_max)
{
stream->msg = "secondary decoder end of input";
return XD3_INTERNAL;
}
bstate.cur_byte = *input++;
for (bstate.cur_mask = 1; bstate.cur_mask != 0x100; bstate.cur_mask <<= 1)
{
int done = fgk_decode_bit (sec_stream, (bstate.cur_byte & bstate.cur_mask) ? 1U : 0U);
if (! done) { continue; }
*output++ = fgk_decode_data (sec_stream);
if (output == output_max)
{
/* During regression testing: */
IF_REGRESSION ({
int ret;
bstate.cur_mask <<= 1;
if ((ret = xd3_test_clean_bits (stream, & bstate))) { return ret; }
});
(*output_pos) = output;
(*input_pos) = input;
return 0;
}
}
}
}
#endif /* _XDELTA3_FGK_ */

159
lib/xdelta3/xdelta3-hash.h Normal file
View file

@ -0,0 +1,159 @@
/* xdelta3 - delta compression tools and library
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#ifndef _XDELTA3_HASH_H_
#define _XDELTA3_HASH_H_
#include "xdelta3-internal.h"
#if XD3_DEBUG
#define SMALL_HASH_DEBUG1(s,inp) \
uint32_t debug_state; \
uint32_t debug_hval = xd3_checksum_hash (& (s)->small_hash, \
xd3_scksum (&debug_state, (inp), (s)->smatcher.small_look))
#define SMALL_HASH_DEBUG2(s,inp) \
XD3_ASSERT (debug_hval == xd3_checksum_hash (& (s)->small_hash, \
xd3_scksum (&debug_state, (inp), (s)->smatcher.small_look)))
#else
#define SMALL_HASH_DEBUG1(s,inp)
#define SMALL_HASH_DEBUG2(s,inp)
#endif /* XD3_DEBUG */
#if UNALIGNED_OK
#define UNALIGNED_READ32(dest,src) (*(dest)) = (*(uint32_t*)(src))
#else
#define UNALIGNED_READ32(dest,src) memcpy((dest), (src), 4);
#endif
/* These are good hash multipliers for 32-bit and 64-bit LCGs: see
* "linear congruential generators of different sizes and good lattice
* structure" */
#define xd3_hash_multiplier32 1597334677U
#define xd3_hash_multiplier64 1181783497276652981ULL
/* TODO: small cksum is hard-coded for 4 bytes (i.e., "look" is unused) */
static inline uint32_t
xd3_scksum (uint32_t *state,
const uint8_t *base,
const usize_t look)
{
UNALIGNED_READ32(state, base);
return (*state) * xd3_hash_multiplier32;
}
static inline uint32_t
xd3_small_cksum_update (uint32_t *state,
const uint8_t *base,
usize_t look)
{
UNALIGNED_READ32(state, base+1);
return (*state) * xd3_hash_multiplier32;
}
#if XD3_ENCODER
inline usize_t
xd3_checksum_hash (const xd3_hash_cfg *cfg, const usize_t cksum)
{
return (cksum >> cfg->shift) ^ (cksum & cfg->mask);
}
#if SIZEOF_USIZE_T == 4
inline uint32_t
xd3_large32_cksum (xd3_hash_cfg *cfg, const uint8_t *base, const usize_t look)
{
uint32_t h = 0;
for (usize_t i = 0; i < look; i++) {
h += base[i] * cfg->powers[i];
}
return h;
}
inline uint32_t
xd3_large32_cksum_update (xd3_hash_cfg *cfg, const uint32_t cksum,
const uint8_t *base, const usize_t look)
{
return xd3_hash_multiplier32 * cksum - cfg->multiplier * base[0] + base[look];
}
#endif
#if SIZEOF_USIZE_T == 8
inline uint64_t
xd3_large64_cksum (xd3_hash_cfg *cfg, const uint8_t *base, const usize_t look)
{
uint64_t h = 0;
for (usize_t i = 0; i < look; i++) {
h += base[i] * cfg->powers[i];
}
return h;
}
inline uint64_t
xd3_large64_cksum_update (xd3_hash_cfg *cfg, const uint64_t cksum,
const uint8_t *base, const usize_t look)
{
return xd3_hash_multiplier64 * cksum - cfg->multiplier * base[0] + base[look];
}
#endif
static usize_t
xd3_size_hashtable_bits (usize_t slots)
{
usize_t bits = (SIZEOF_USIZE_T * 8) - 1;
usize_t i;
for (i = 3; i <= bits; i += 1)
{
if (slots < (1U << i))
{
/* Note: this is the compaction=1 setting measured in
* checksum_test */
bits = i - 1;
break;
}
}
return bits;
}
int
xd3_size_hashtable (xd3_stream *stream,
usize_t slots,
usize_t look,
xd3_hash_cfg *cfg)
{
usize_t bits = xd3_size_hashtable_bits (slots);
cfg->size = (1U << bits);
cfg->mask = (cfg->size - 1);
cfg->shift = (SIZEOF_USIZE_T * 8) - bits;
cfg->look = look;
if ((cfg->powers =
(usize_t*) xd3_alloc0 (stream, look, sizeof (usize_t))) == NULL)
{
return ENOMEM;
}
cfg->powers[look-1] = 1;
for (int i = look-2; i >= 0; i--)
{
cfg->powers[i] = cfg->powers[i+1] * xd3_hash_multiplier;
}
cfg->multiplier = cfg->powers[0] * xd3_hash_multiplier;
return 0;
}
#endif /* XD3_ENCODER */
#endif /* _XDELTA3_HASH_H_ */

385
lib/xdelta3/xdelta3-internal.h Normal file
View file

@ -0,0 +1,385 @@
/* xdelta3 - delta compression tools and library
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#ifndef XDELTA3_INTERNAL_H__
#define XDELTA3_INTERNAL_H__
#include "xdelta3.h"
typedef struct _main_file main_file;
typedef struct _main_extcomp main_extcomp;
void main_buffree (void *ptr);
void* main_bufalloc (size_t size);
void main_file_init (main_file *xfile);
int main_file_close (main_file *xfile);
void main_file_cleanup (main_file *xfile);
int main_file_isopen (main_file *xfile);
int main_file_open (main_file *xfile, const char* name, int mode);
int main_file_exists (main_file *xfile);
int main_file_stat (main_file *xfile, xoff_t *size);
int xd3_whole_append_window (xd3_stream *stream);
int xd3_main_cmdline (int argc, char **argv);
int main_file_read (main_file *ifile,
uint8_t *buf,
size_t size,
size_t *nread,
const char *msg);
int main_file_write (main_file *ofile, uint8_t *buf,
usize_t size, const char *msg);
void* main_malloc (size_t size);
void main_free (void *ptr);
int test_compare_files (const char* f0, const char* f1);
usize_t xd3_bytes_on_srcblk (xd3_source *src, xoff_t blkno);
xoff_t xd3_source_eof(const xd3_source *src);
uint32_t xd3_large_cksum_update (uint32_t cksum,
const uint8_t *base,
usize_t look);
int xd3_emit_byte (xd3_stream *stream,
xd3_output **outputp,
uint8_t code);
int xd3_emit_bytes (xd3_stream *stream,
xd3_output **outputp,
const uint8_t *base,
usize_t size);
xd3_output* xd3_alloc_output (xd3_stream *stream,
xd3_output *old_output);
int xd3_encode_init_full (xd3_stream *stream);
usize_t xd3_pow2_roundup (usize_t x);
long get_millisecs_now (void);
int xd3_process_stream (int is_encode,
xd3_stream *stream,
int (*func) (xd3_stream *),
int close_stream,
const uint8_t *input,
usize_t input_size,
uint8_t *output,
usize_t *output_size,
usize_t output_size_max);
#if PYTHON_MODULE || SWIG_MODULE || NOT_MAIN
int xd3_main_cmdline (int argc, char **argv);
#endif
#if REGRESSION_TEST
int xd3_selftest (void);
#endif
/* main_file->mode values */
typedef enum
{
XO_READ = 0,
XO_WRITE = 1
} main_file_modes;
#ifndef XD3_POSIX
#define XD3_POSIX 0
#endif
#ifndef XD3_STDIO
#define XD3_STDIO 0
#endif
#ifndef XD3_WIN32
#define XD3_WIN32 0
#endif
#ifndef NOT_MAIN
#define NOT_MAIN 0
#endif
/* If none are set, default to posix. */
#if (XD3_POSIX + XD3_STDIO + XD3_WIN32) == 0
#undef XD3_POSIX
#define XD3_POSIX 1
#endif
struct _main_file
{
#if XD3_WIN32
HANDLE file;
#elif XD3_STDIO
FILE *file;
#elif XD3_POSIX
int file;
#endif
int mode; /* XO_READ and XO_WRITE */
const char *filename; /* File name or /dev/stdin,
* /dev/stdout, /dev/stderr. */
char *filename_copy; /* File name or /dev/stdin,
* /dev/stdout, /dev/stderr. */
const char *realname; /* File name or /dev/stdin,
* /dev/stdout, /dev/stderr. */
const main_extcomp *compressor; /* External compression struct. */
int flags; /* RD_FIRST, RD_NONEXTERNAL, ... */
xoff_t nread; /* for input position */
xoff_t nwrite; /* for output position */
uint8_t *snprintf_buf; /* internal snprintf() use */
int size_known; /* Set by main_set_souze */
xoff_t source_position; /* for avoiding seek in getblk_func */
int seek_failed; /* after seek fails once, try FIFO */
};
#ifndef UINT32_MAX
#define UINT32_MAX 4294967295U
#endif
#ifndef UINT64_MAX
#define UINT64_MAX 18446744073709551615ULL
#endif
#define UINT32_OFLOW_MASK 0xfe000000U
#define UINT64_OFLOW_MASK 0xfe00000000000000ULL
/*********************************************************************
Integer encoder/decoder functions
**********************************************************************/
/* Consume N bytes of input, only used by the decoder. */
#define DECODE_INPUT(n) \
do { \
stream->total_in += (xoff_t) (n); \
stream->avail_in -= (n); \
stream->next_in += (n); \
} while (0)
#define DECODE_INTEGER_TYPE(PART,OFLOW) \
while (stream->avail_in != 0) \
{ \
usize_t next = stream->next_in[0]; \
\
DECODE_INPUT(1); \
\
if (PART & OFLOW) \
{ \
stream->msg = "overflow in decode_integer"; \
return XD3_INVALID_INPUT; \
} \
\
PART = (PART << 7) | (next & 127); \
\
if ((next & 128) == 0) \
{ \
(*val) = PART; \
PART = 0; \
return 0; \
} \
} \
\
stream->msg = "further input required"; \
return XD3_INPUT
#define READ_INTEGER_TYPE(TYPE, OFLOW) \
TYPE val = 0; \
const uint8_t *inp = (*inpp); \
usize_t next; \
\
do \
{ \
if (inp == maxp) \
{ \
stream->msg = "end-of-input in read_integer"; \
return XD3_INVALID_INPUT; \
} \
\
if (val & OFLOW) \
{ \
stream->msg = "overflow in read_intger"; \
return XD3_INVALID_INPUT; \
} \
\
next = (*inp++); \
val = (val << 7) | (next & 127); \
} \
while (next & 128); \
\
(*valp) = val; \
(*inpp) = inp; \
\
return 0
#define EMIT_INTEGER_TYPE() \
/* max 64-bit value in base-7 encoding is 9.1 bytes */ \
uint8_t buf[10]; \
usize_t bufi = 10; \
\
/* This loop performs division and turns on all MSBs. */ \
do \
{ \
buf[--bufi] = (num & 127) | 128; \
num >>= 7U; \
} \
while (num != 0); \
\
/* Turn off MSB of the last byte. */ \
buf[9] &= 127; \
\
return xd3_emit_bytes (stream, output, buf + bufi, 10 - bufi)
#define IF_SIZEOF32(x) if (num < (1U << (7 * (x)))) return (x);
#define IF_SIZEOF64(x) if (num < (1ULL << (7 * (x)))) return (x);
#if USE_UINT32
static inline uint32_t
xd3_sizeof_uint32_t (uint32_t num)
{
IF_SIZEOF32(1);
IF_SIZEOF32(2);
IF_SIZEOF32(3);
IF_SIZEOF32(4);
return 5;
}
static inline int
xd3_decode_uint32_t (xd3_stream *stream, uint32_t *val)
{ DECODE_INTEGER_TYPE (stream->dec_32part, UINT32_OFLOW_MASK); }
static inline int
xd3_read_uint32_t (xd3_stream *stream, const uint8_t **inpp,
const uint8_t *maxp, uint32_t *valp)
{ READ_INTEGER_TYPE (uint32_t, UINT32_OFLOW_MASK); }
#if XD3_ENCODER
static inline int
xd3_emit_uint32_t (xd3_stream *stream, xd3_output **output, uint32_t num)
{ EMIT_INTEGER_TYPE (); }
#endif /* XD3_ENCODER */
#endif /* USE_UINT32 */
#if USE_UINT64
static inline uint32_t
xd3_sizeof_uint64_t (uint64_t num)
{
IF_SIZEOF64(1);
IF_SIZEOF64(2);
IF_SIZEOF64(3);
IF_SIZEOF64(4);
IF_SIZEOF64(5);
IF_SIZEOF64(6);
IF_SIZEOF64(7);
IF_SIZEOF64(8);
IF_SIZEOF64(9);
return 10;
}
static inline int
xd3_decode_uint64_t (xd3_stream *stream, uint64_t *val)
{ DECODE_INTEGER_TYPE (stream->dec_64part, UINT64_OFLOW_MASK); }
static inline int
xd3_read_uint64_t (xd3_stream *stream, const uint8_t **inpp,
const uint8_t *maxp, uint64_t *valp)
{ READ_INTEGER_TYPE (uint64_t, UINT64_OFLOW_MASK); }
#if XD3_ENCODER
static inline int
xd3_emit_uint64_t (xd3_stream *stream, xd3_output **output, uint64_t num)
{ EMIT_INTEGER_TYPE (); }
#endif /* XD3_ENCODER */
#endif /* USE_UINT64 */
#if SIZEOF_USIZE_T == 4
#define USIZE_T_MAX UINT32_MAX
#define USIZE_T_MAXBLKSZ 0x80000000U
#define XD3_MAXSRCWINSZ (1ULL << 31)
#define xd3_large_cksum xd3_large32_cksum
#define xd3_large_cksum_update xd3_large32_cksum_update
#define xd3_hash_multiplier xd3_hash_multiplier32
static inline uint32_t xd3_sizeof_size (usize_t num)
{ return xd3_sizeof_uint32_t (num); }
static inline int xd3_decode_size (xd3_stream *stream, usize_t *valp)
{ return xd3_decode_uint32_t (stream, (uint32_t*) valp); }
static inline int xd3_read_size (xd3_stream *stream, const uint8_t **inpp,
const uint8_t *maxp, usize_t *valp)
{ return xd3_read_uint32_t (stream, inpp, maxp, (uint32_t*) valp); }
#if XD3_ENCODER
static inline int xd3_emit_size (xd3_stream *stream, xd3_output **output, usize_t num)
{ return xd3_emit_uint32_t (stream, output, num); }
#endif
#elif SIZEOF_USIZE_T == 8
#define USIZE_T_MAX UINT64_MAX
#define USIZE_T_MAXBLKSZ 0x8000000000000000ULL
#define XD3_MAXSRCWINSZ (1ULL << 61)
#define xd3_large_cksum xd3_large64_cksum
#define xd3_large_cksum_update xd3_large64_cksum_update
#define xd3_hash_multiplier xd3_hash_multiplier64
static inline uint32_t xd3_sizeof_size (usize_t num)
{ return xd3_sizeof_uint64_t (num); }
static inline int xd3_decode_size (xd3_stream *stream, usize_t *valp)
{ return xd3_decode_uint64_t (stream, (uint64_t*) valp); }
static inline int xd3_read_size (xd3_stream *stream, const uint8_t **inpp,
const uint8_t *maxp, usize_t *valp)
{ return xd3_read_uint64_t (stream, inpp, maxp, (uint64_t*) valp); }
#if XD3_ENCODER
static inline int xd3_emit_size (xd3_stream *stream, xd3_output **output, usize_t num)
{ return xd3_emit_uint64_t (stream, output, num); }
#endif
#endif /* SIZEOF_USIZE_T */
#if SIZEOF_XOFF_T == 4
#define XOFF_T_MAX UINT32_MAX
static inline int xd3_decode_offset (xd3_stream *stream, xoff_t *valp)
{ return xd3_decode_uint32_t (stream, (uint32_t*) valp); }
#if XD3_ENCODER
static inline int xd3_emit_offset (xd3_stream *stream, xd3_output **output, xoff_t num)
{ return xd3_emit_uint32_t (stream, output, num); }
#endif
#elif SIZEOF_XOFF_T == 8
#define XOFF_T_MAX UINT64_MAX
static inline int xd3_decode_offset (xd3_stream *stream, xoff_t *valp)
{ return xd3_decode_uint64_t (stream, (uint64_t*) valp); }
#if XD3_ENCODER
static inline int xd3_emit_offset (xd3_stream *stream, xd3_output **output, xoff_t num)
{ return xd3_emit_uint64_t (stream, output, num); }
#endif
#endif
#define USIZE_T_OVERFLOW(a,b) ((USIZE_T_MAX - (usize_t) (a)) < (usize_t) (b))
#define XOFF_T_OVERFLOW(a,b) ((XOFF_T_MAX - (xoff_t) (a)) < (xoff_t) (b))
int xd3_size_hashtable (xd3_stream *stream,
usize_t slots,
usize_t look,
xd3_hash_cfg *cfg);
usize_t xd3_checksum_hash (const xd3_hash_cfg *cfg, const usize_t cksum);
#if USE_UINT32
uint32_t xd3_large32_cksum (xd3_hash_cfg *cfg, const uint8_t *base, const usize_t look);
uint32_t xd3_large32_cksum_update (xd3_hash_cfg *cfg, const uint32_t cksum,
const uint8_t *base, const usize_t look);
#endif /* USE_UINT32 */
#if USE_UINT64
uint64_t xd3_large64_cksum (xd3_hash_cfg *cfg, const uint8_t *base, const usize_t look);
uint64_t xd3_large64_cksum_update (xd3_hash_cfg *cfg, const uint64_t cksum,
const uint8_t *base, const usize_t look);
#endif /* USE_UINT64 */
#define MAX_LRU_SIZE 32U
#define XD3_MINSRCWINSZ (XD3_ALLOCSIZE * MAX_LRU_SIZE)
#endif // XDELTA3_INTERNAL_H__

127
lib/xdelta3/xdelta3-list.h Normal file
View file

@ -0,0 +1,127 @@
/* xdelta3 - delta compression tools and library
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#ifndef __XDELTA3_LIST__
#define __XDELTA3_LIST__
#define XD3_MAKELIST(LTYPE,ETYPE,LNAME) \
\
static inline ETYPE* \
LTYPE ## _entry (LTYPE* l) \
{ \
return (ETYPE*) ((char*) l - (ptrdiff_t) &((ETYPE*) 0)->LNAME); \
} \
\
static inline void \
LTYPE ## _init (LTYPE *l) \
{ \
l->next = l; \
l->prev = l; \
} \
\
static inline void \
LTYPE ## _add (LTYPE *prev, LTYPE *next, LTYPE *ins) \
{ \
next->prev = ins; \
prev->next = ins; \
ins->next = next; \
ins->prev = prev; \
} \
\
static inline void \
LTYPE ## _push_back (LTYPE *l, ETYPE *i) \
{ \
LTYPE ## _add (l->prev, l, & i->LNAME); \
} \
\
static inline void \
LTYPE ## _del (LTYPE *next, \
LTYPE *prev) \
{ \
next->prev = prev; \
prev->next = next; \
} \
\
static inline ETYPE* \
LTYPE ## _remove (ETYPE *f) \
{ \
LTYPE *i = f->LNAME.next; \
LTYPE ## _del (f->LNAME.next, f->LNAME.prev); \
return LTYPE ## _entry (i); \
} \
\
static inline ETYPE* \
LTYPE ## _pop_back (LTYPE *l) \
{ \
LTYPE *i = l->prev; \
LTYPE ## _del (i->next, i->prev); \
return LTYPE ## _entry (i); \
} \
\
static inline ETYPE* \
LTYPE ## _pop_front (LTYPE *l) \
{ \
LTYPE *i = l->next; \
LTYPE ## _del (i->next, i->prev); \
return LTYPE ## _entry (i); \
} \
\
static inline int \
LTYPE ## _empty (LTYPE *l) \
{ \
return l == l->next; \
} \
\
static inline ETYPE* \
LTYPE ## _front (LTYPE *f) \
{ \
return LTYPE ## _entry (f->next); \
} \
\
static inline ETYPE* \
LTYPE ## _back (LTYPE *f) \
{ \
return LTYPE ## _entry (f->prev); \
} \
\
static inline int \
LTYPE ## _end (LTYPE *f, ETYPE *i) \
{ \
return f == & i->LNAME; \
} \
\
static inline ETYPE* \
LTYPE ## _next (ETYPE *f) \
{ \
return LTYPE ## _entry (f->LNAME.next); \
} \
\
static inline usize_t \
LTYPE ## _length (LTYPE *l) \
{ \
LTYPE *p; \
usize_t c = 0; \
\
for (p = l->next; p != l; p = p->next) \
{ \
c += 1; \
} \
\
return c; \
} \
\
typedef int unused_ ## LTYPE
#endif

195
lib/xdelta3/xdelta3-lzma.h Normal file
View file

@ -0,0 +1,195 @@
/* xdelta3 - delta compression tools and library
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
/* Note: The use of the _easy_ decoder means we're not calling the
* xd3_stream malloc hooks. TODO(jmacd) Fix if anyone cares. */
#ifndef _XDELTA3_LZMA_H_
#define _XDELTA3_LZMA_H_
#include <lzma.h>
typedef struct _xd3_lzma_stream xd3_lzma_stream;
struct _xd3_lzma_stream {
lzma_stream lzma;
lzma_options_lzma options;
lzma_filter filters[2];
};
static xd3_sec_stream*
xd3_lzma_alloc (xd3_stream *stream)
{
return (xd3_sec_stream*) xd3_alloc (stream, sizeof (xd3_lzma_stream), 1);
}
static void
xd3_lzma_destroy (xd3_stream *stream, xd3_sec_stream *sec_stream)
{
xd3_lzma_stream *ls = (xd3_lzma_stream*) sec_stream;
lzma_end (&ls->lzma);
xd3_free (stream, ls);
}
static int
xd3_lzma_init (xd3_stream *stream, xd3_lzma_stream *sec, int is_encode)
{
int ret;
memset (&sec->lzma, 0, sizeof(sec->lzma));
if (is_encode)
{
uint32_t preset =
(stream->flags & XD3_COMPLEVEL_MASK) >> XD3_COMPLEVEL_SHIFT;
if (lzma_lzma_preset(&sec->options, preset))
{
stream->msg = "invalid lzma preset";
return XD3_INVALID;
}
sec->filters[0].id = LZMA_FILTER_LZMA2;
sec->filters[0].options = &sec->options;
sec->filters[1].id = LZMA_VLI_UNKNOWN;
ret = lzma_stream_encoder (&sec->lzma, &sec->filters[0], LZMA_CHECK_NONE);
}
else
{
ret = lzma_stream_decoder (&sec->lzma, UINT64_MAX, LZMA_TELL_NO_CHECK);
}
if (ret != LZMA_OK)
{
stream->msg = "lzma stream init failed";
return XD3_INTERNAL;
}
return 0;
}
static int xd3_decode_lzma (xd3_stream *stream, xd3_lzma_stream *sec,
const uint8_t **input_pos,
const uint8_t *const input_end,
uint8_t **output_pos,
const uint8_t *const output_end)
{
uint8_t *output = *output_pos;
const uint8_t *input = *input_pos;
size_t avail_in = input_end - input;
size_t avail_out = output_end - output;
sec->lzma.avail_in = avail_in;
sec->lzma.next_in = input;
sec->lzma.avail_out = avail_out;
sec->lzma.next_out = output;
while (1)
{
int lret = lzma_code (&sec->lzma, LZMA_RUN);
switch (lret)
{
case LZMA_NO_CHECK:
case LZMA_OK:
if (sec->lzma.avail_out == 0)
{
(*output_pos) = sec->lzma.next_out;
(*input_pos) = sec->lzma.next_in;
return 0;
}
break;
default:
stream->msg = "lzma decoding error";
return XD3_INTERNAL;
}
}
}
#if XD3_ENCODER
static int xd3_encode_lzma (xd3_stream *stream,
xd3_lzma_stream *sec,
xd3_output *input,
xd3_output *output,
xd3_sec_cfg *cfg)
{
lzma_action action = LZMA_RUN;
cfg->inefficient = 1; /* Can't skip windows */
sec->lzma.next_in = NULL;
sec->lzma.avail_in = 0;
sec->lzma.next_out = (output->base + output->next);
sec->lzma.avail_out = (output->avail - output->next);
while (1)
{
int lret;
size_t nwrite;
if (sec->lzma.avail_in == 0 && input != NULL)
{
sec->lzma.avail_in = input->next;
sec->lzma.next_in = input->base;
if ((input = input->next_page) == NULL)
{
action = LZMA_SYNC_FLUSH;
}
}
lret = lzma_code (&sec->lzma, action);
nwrite = (output->avail - output->next) - sec->lzma.avail_out;
if (nwrite != 0)
{
output->next += nwrite;
if (output->next == output->avail)
{
if ((output = xd3_alloc_output (stream, output)) == NULL)
{
return ENOMEM;
}
sec->lzma.next_out = output->base;
sec->lzma.avail_out = output->avail;
}
}
switch (lret)
{
case LZMA_OK:
break;
case LZMA_STREAM_END:
return 0;
default:
stream->msg = "lzma encoding error";
return XD3_INTERNAL;
}
}
return 0;
}
#endif /* XD3_ENCODER */
#endif /* _XDELTA3_LZMA_H_ */

4062
lib/xdelta3/xdelta3-main.h Normal file
View file

File diff suppressed because it is too large Load diff

583
lib/xdelta3/xdelta3-merge.h Normal file
View file

@ -0,0 +1,583 @@
/* xdelta3 - delta compression tools and library
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#ifndef _XDELTA3_MERGE_H_
#define _XDELTA3_MERGE_H_
int xd3_merge_inputs (xd3_stream *stream,
xd3_whole_state *source,
xd3_whole_state *input);
static int
xd3_whole_state_init (xd3_stream *stream)
{
XD3_ASSERT (stream->whole_target.adds == NULL);
XD3_ASSERT (stream->whole_target.inst == NULL);
XD3_ASSERT (stream->whole_target.wininfo == NULL);
XD3_ASSERT (stream->whole_target.length == 0);
stream->whole_target.adds_alloc = XD3_ALLOCSIZE;
stream->whole_target.inst_alloc = XD3_ALLOCSIZE;
stream->whole_target.wininfo_alloc = XD3_ALLOCSIZE;
if ((stream->whole_target.adds = (uint8_t*)
xd3_alloc (stream, stream->whole_target.adds_alloc, 1)) == NULL ||
(stream->whole_target.inst = (xd3_winst*)
xd3_alloc (stream, stream->whole_target.inst_alloc, 1)) == NULL ||
(stream->whole_target.wininfo = (xd3_wininfo*)
xd3_alloc (stream, stream->whole_target.wininfo_alloc, 1)) == NULL)
{
return ENOMEM;
}
return 0;
}
static void
xd3_swap_whole_state (xd3_whole_state *a,
xd3_whole_state *b)
{
xd3_whole_state tmp;
XD3_ASSERT (a->inst != NULL && a->adds != NULL);
XD3_ASSERT (b->inst != NULL && b->adds != NULL);
XD3_ASSERT (b->wininfo != NULL && b->wininfo != NULL);
memcpy (&tmp, a, sizeof (xd3_whole_state));
memcpy (a, b, sizeof (xd3_whole_state));
memcpy (b, &tmp, sizeof (xd3_whole_state));
}
static int
xd3_realloc_buffer (xd3_stream *stream,
usize_t current_units,
usize_t unit_size,
usize_t new_units,
usize_t *alloc_size,
void **alloc_ptr)
{
usize_t needed;
usize_t new_alloc;
usize_t cur_size;
uint8_t *new_buf;
needed = (current_units + new_units) * unit_size;
if (needed <= *alloc_size)
{
return 0;
}
cur_size = current_units * unit_size;
new_alloc = xd3_round_blksize (needed * 2, XD3_ALLOCSIZE);
if ((new_buf = (uint8_t*) xd3_alloc (stream, new_alloc, 1)) == NULL)
{
return ENOMEM;
}
if (cur_size != 0)
{
memcpy (new_buf, *alloc_ptr, cur_size);
}
if (*alloc_ptr != NULL)
{
xd3_free (stream, *alloc_ptr);
}
*alloc_size = new_alloc;
*alloc_ptr = new_buf;
return 0;
}
/* allocate one new output instruction */
static int
xd3_whole_alloc_winst (xd3_stream *stream,
xd3_winst **winstp)
{
int ret;
if ((ret = xd3_realloc_buffer (stream,
stream->whole_target.instlen,
sizeof (xd3_winst),
1,
& stream->whole_target.inst_alloc,
(void**) & stream->whole_target.inst)))
{
return ret;
}
*winstp = &stream->whole_target.inst[stream->whole_target.instlen++];
return 0;
}
static int
xd3_whole_alloc_adds (xd3_stream *stream,
usize_t count)
{
return xd3_realloc_buffer (stream,
stream->whole_target.addslen,
1,
count,
& stream->whole_target.adds_alloc,
(void**) & stream->whole_target.adds);
}
static int
xd3_whole_alloc_wininfo (xd3_stream *stream,
xd3_wininfo **wininfop)
{
int ret;
if ((ret = xd3_realloc_buffer (stream,
stream->whole_target.wininfolen,
sizeof (xd3_wininfo),
1,
& stream->whole_target.wininfo_alloc,
(void**) & stream->whole_target.wininfo)))
{
return ret;
}
*wininfop = &stream->whole_target.wininfo[stream->whole_target.wininfolen++];
return 0;
}
static int
xd3_whole_append_inst (xd3_stream *stream,
xd3_hinst *inst)
{
int ret;
xd3_winst *winst;
if ((ret = xd3_whole_alloc_winst (stream, &winst)))
{
return ret;
}
winst->type = inst->type;
winst->mode = 0;
winst->size = inst->size;
winst->position = stream->whole_target.length;
stream->whole_target.length += inst->size;
if (((inst->type == XD3_ADD) || (inst->type == XD3_RUN)) &&
(ret = xd3_whole_alloc_adds (stream,
(inst->type == XD3_RUN ? 1 : inst->size))))
{
return ret;
}
switch (inst->type)
{
case XD3_RUN:
winst->addr = stream->whole_target.addslen;
stream->whole_target.adds[stream->whole_target.addslen++] =
*stream->data_sect.buf++;
break;
case XD3_ADD:
winst->addr = stream->whole_target.addslen;
memcpy (stream->whole_target.adds + stream->whole_target.addslen,
stream->data_sect.buf,
inst->size);
stream->data_sect.buf += inst->size;
stream->whole_target.addslen += inst->size;
break;
default:
if (inst->addr < stream->dec_cpylen)
{
winst->mode = SRCORTGT (stream->dec_win_ind);
winst->addr = stream->dec_cpyoff + inst->addr;
}
else
{
winst->addr = (stream->dec_winstart +
inst->addr -
stream->dec_cpylen);
}
break;
}
return 0;
}
int
xd3_whole_append_window (xd3_stream *stream)
{
int ret;
xd3_wininfo *wininfo;
if ((ret = xd3_whole_alloc_wininfo (stream, &wininfo))) { return ret; }
wininfo->length = stream->dec_tgtlen;
wininfo->offset = stream->dec_winstart;
wininfo->adler32 = stream->dec_adler32;
while (stream->inst_sect.buf < stream->inst_sect.buf_max)
{
if ((ret = xd3_decode_instruction (stream)))
{
return ret;
}
if ((stream->dec_current1.type != XD3_NOOP) &&
(ret = xd3_whole_append_inst (stream,
& stream->dec_current1)))
{
return ret;
}
if ((stream->dec_current2.type != XD3_NOOP) &&
(ret = xd3_whole_append_inst (stream,
& stream->dec_current2)))
{
return ret;
}
}
return 0;
}
/* xd3_merge_input_output applies *source to *stream, returns the
* result in stream. */
static int xd3_merge_input_output (xd3_stream *stream,
xd3_whole_state *source)
{
int ret;
xd3_stream tmp_stream;
memset (& tmp_stream, 0, sizeof (tmp_stream));
if ((ret = xd3_config_stream (& tmp_stream, NULL)) ||
(ret = xd3_whole_state_init (& tmp_stream)) ||
(ret = xd3_merge_inputs (& tmp_stream,
source,
& stream->whole_target)))
{
XPR(NT XD3_LIB_ERRMSG (&tmp_stream, ret));
return ret;
}
/* the output is in tmp_stream.whole_state, swap into input */
xd3_swap_whole_state (& stream->whole_target,
& tmp_stream.whole_target);
/* total allocation counts are preserved */
xd3_free_stream (& tmp_stream);
return 0;
}
static int
xd3_merge_run (xd3_stream *stream,
xd3_whole_state *target,
xd3_winst *iinst)
{
int ret;
xd3_winst *oinst;
if ((ret = xd3_whole_alloc_winst (stream, &oinst)) ||
(ret = xd3_whole_alloc_adds (stream, 1)))
{
return ret;
}
oinst->type = iinst->type;
oinst->mode = iinst->mode;
oinst->size = iinst->size;
oinst->addr = stream->whole_target.addslen;
XD3_ASSERT (stream->whole_target.length == iinst->position);
oinst->position = stream->whole_target.length;
stream->whole_target.length += iinst->size;
stream->whole_target.adds[stream->whole_target.addslen++] =
target->adds[iinst->addr];
return 0;
}
static int
xd3_merge_add (xd3_stream *stream,
xd3_whole_state *target,
xd3_winst *iinst)
{
int ret;
xd3_winst *oinst;
if ((ret = xd3_whole_alloc_winst (stream, &oinst)) ||
(ret = xd3_whole_alloc_adds (stream, iinst->size)))
{
return ret;
}
oinst->type = iinst->type;
oinst->mode = iinst->mode;
oinst->size = iinst->size;
oinst->addr = stream->whole_target.addslen;
XD3_ASSERT (stream->whole_target.length == iinst->position);
oinst->position = stream->whole_target.length;
stream->whole_target.length += iinst->size;
memcpy(stream->whole_target.adds + stream->whole_target.addslen,
target->adds + iinst->addr,
iinst->size);
stream->whole_target.addslen += iinst->size;
return 0;
}
static int
xd3_merge_target_copy (xd3_stream *stream,
xd3_winst *iinst)
{
int ret;
xd3_winst *oinst;
if ((ret = xd3_whole_alloc_winst (stream, &oinst)))
{
return ret;
}
XD3_ASSERT (stream->whole_target.length == iinst->position);
memcpy (oinst, iinst, sizeof (*oinst));
return 0;
}
static int
xd3_merge_find_position (xd3_stream *stream,
xd3_whole_state *source,
xoff_t address,
usize_t *inst_num)
{
usize_t low;
usize_t high;
if (address >= source->length)
{
stream->msg = "Invalid copy offset in merge";
return XD3_INVALID_INPUT;
}
low = 0;
high = source->instlen;
while (low != high)
{
xoff_t mid_lpos;
xoff_t mid_hpos;
usize_t mid = low + (high - low) / 2;
mid_lpos = source->inst[mid].position;
if (address < mid_lpos)
{
high = mid;
continue;
}
mid_hpos = mid_lpos + source->inst[mid].size;
if (address >= mid_hpos)
{
low = mid + 1;
continue;
}
*inst_num = mid;
return 0;
}
stream->msg = "Internal error in merge";
return XD3_INTERNAL;
}
static int
xd3_merge_source_copy (xd3_stream *stream,
xd3_whole_state *source,
const xd3_winst *iinst_orig)
{
int ret;
xd3_winst iinst;
usize_t sinst_num;
memcpy (& iinst, iinst_orig, sizeof (iinst));
XD3_ASSERT (iinst.mode == VCD_SOURCE);
if ((ret = xd3_merge_find_position (stream, source,
iinst.addr, &sinst_num)))
{
return ret;
}
while (iinst.size > 0)
{
xd3_winst *sinst;
xd3_winst *minst;
usize_t sinst_offset;
usize_t sinst_left;
usize_t this_take;
XD3_ASSERT (sinst_num < source->instlen);
sinst = &source->inst[sinst_num];
XD3_ASSERT (iinst.addr >= sinst->position);
sinst_offset = (usize_t)(iinst.addr - sinst->position);
XD3_ASSERT (sinst->size > sinst_offset);
sinst_left = sinst->size - sinst_offset;
this_take = xd3_min (iinst.size, sinst_left);
XD3_ASSERT (this_take > 0);
if ((ret = xd3_whole_alloc_winst (stream, &minst)))
{
return ret;
}
minst->size = this_take;
minst->type = sinst->type;
minst->position = iinst.position;
minst->mode = 0;
switch (sinst->type)
{
case XD3_RUN:
if ((ret = xd3_whole_alloc_adds (stream, 1)))
{
return ret;
}
minst->addr = stream->whole_target.addslen;
stream->whole_target.adds[stream->whole_target.addslen++] =
source->adds[sinst->addr];
break;
case XD3_ADD:
if ((ret = xd3_whole_alloc_adds (stream, this_take)))
{
return ret;
}
minst->addr = stream->whole_target.addslen;
memcpy(stream->whole_target.adds + stream->whole_target.addslen,
source->adds + sinst->addr + sinst_offset,
this_take);
stream->whole_target.addslen += this_take;
break;
default:
if (sinst->mode != 0)
{
minst->mode = sinst->mode;
minst->addr = sinst->addr + sinst_offset;
}
else
{
// Note: A better implementation will construct the
// mapping of output ranges, starting from the input
// range, applying deltas in forward order, using an
// interval tree. This code uses recursion to construct
// each copied range, recursively (using binary search
// in xd3_merge_find_position).
//
// TODO: This code can cause stack overflow. Fix as
// described above.
xd3_winst tinst;
tinst.type = XD3_CPY;
tinst.mode = iinst.mode;
tinst.addr = sinst->addr + sinst_offset;
tinst.size = this_take;
tinst.position = iinst.position;
// The instruction allocated in this frame will not be used.
stream->whole_target.instlen -= 1;
if ((ret = xd3_merge_source_copy (stream, source, &tinst)))
{
return ret;
}
}
break;
}
iinst.position += this_take;
iinst.addr += this_take;
iinst.size -= this_take;
sinst_num += 1;
}
return 0;
}
/* xd3_merge_inputs() applies *input to *source, returns its result in
* stream. */
int xd3_merge_inputs (xd3_stream *stream,
xd3_whole_state *source,
xd3_whole_state *input)
{
int ret = 0;
usize_t i;
size_t input_i;
for (i = 0; i < input->wininfolen; ++i) {
xd3_wininfo *copyinfo;
if ((ret = xd3_whole_alloc_wininfo (stream, &copyinfo))) { return ret; }
*copyinfo = input->wininfo[i];
}
/* iterate over each instruction. */
for (input_i = 0; ret == 0 && input_i < input->instlen; ++input_i)
{
xd3_winst *iinst = &input->inst[input_i];
switch (iinst->type)
{
case XD3_RUN:
ret = xd3_merge_run (stream, input, iinst);
break;
case XD3_ADD:
ret = xd3_merge_add (stream, input, iinst);
break;
default:
if (iinst->mode == 0)
{
ret = xd3_merge_target_copy (stream, iinst);
}
else if (iinst->mode == VCD_TARGET)
{
ret = XD3_INVALID_INPUT;
}
else
{
ret = xd3_merge_source_copy (stream, source, iinst);
}
/* The whole_target.length is not updated in the xd3_merge*copy
* routine because of recursion in xd3_merge_source_copy. */
stream->whole_target.length += iinst->size;
break;
}
}
return ret;
}
#endif

321
lib/xdelta3/xdelta3-second.h Normal file
View file

@ -0,0 +1,321 @@
/* xdelta3 - delta compression tools and library
Copyright 2016 Joshua MacDonald
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#ifndef _XDELTA3_SECOND_H_
#define _XDELTA3_SECOND_H_
static inline void xd3_bit_state_encode_init (bit_state *bits)
{
bits->cur_byte = 0;
bits->cur_mask = 1;
}
static inline int xd3_decode_bits (xd3_stream *stream,
bit_state *bits,
const uint8_t **input,
const uint8_t *input_max,
usize_t nbits,
usize_t *valuep)
{
usize_t value = 0;
usize_t vmask = 1 << nbits;
if (bits->cur_mask == 0x100) { goto next_byte; }
for (;;)
{
do
{
vmask >>= 1;
if (bits->cur_byte & bits->cur_mask)
{
value |= vmask;
}
bits->cur_mask <<= 1;
if (vmask == 1) { goto done; }
}
while (bits->cur_mask != 0x100);
next_byte:
if (*input == input_max)
{
stream->msg = "secondary decoder end of input";
return XD3_INTERNAL;
}
bits->cur_byte = *(*input)++;
bits->cur_mask = 1;
}
done:
IF_DEBUG2 (DP(RINT "(d) %"W"u ", value));
(*valuep) = value;
return 0;
}
#if REGRESSION_TEST
/* There may be extra bits at the end of secondary decompression, this macro
* checks for non-zero bits. This is overly strict, but helps pass the
* single-bit-error regression test. */
static int
xd3_test_clean_bits (xd3_stream *stream, bit_state *bits)
{
for (; bits->cur_mask != 0x100; bits->cur_mask <<= 1)
{
if (bits->cur_byte & bits->cur_mask)
{
stream->msg = "secondary decoder garbage";
return XD3_INTERNAL;
}
}
return 0;
}
#endif
static int
xd3_get_secondary (xd3_stream *stream, xd3_sec_stream **sec_streamp,
int is_encode)
{
if (*sec_streamp == NULL)
{
int ret;
if ((*sec_streamp = stream->sec_type->alloc (stream)) == NULL)
{
stream->msg = "error initializing secondary stream";
return XD3_INVALID;
}
if ((ret = stream->sec_type->init (stream, *sec_streamp, is_encode)) != 0)
{
return ret;
}
}
return 0;
}
static int
xd3_decode_secondary (xd3_stream *stream,
xd3_desect *sect,
xd3_sec_stream **sec_streamp)
{
usize_t dec_size;
uint8_t *out_used;
int ret;
if ((ret = xd3_get_secondary (stream, sec_streamp, 0)) != 0)
{
return ret;
}
/* Decode the size, allocate the buffer. */
if ((ret = xd3_read_size (stream, & sect->buf,
sect->buf_max, & dec_size)) ||
(ret = xd3_decode_allocate (stream, dec_size,
& sect->copied2, & sect->alloc2)))
{
return ret;
}
if (dec_size == 0)
{
stream->msg = "secondary decoder invalid output size";
return XD3_INVALID_INPUT;
}
out_used = sect->copied2;
if ((ret = stream->sec_type->decode (stream, *sec_streamp,
& sect->buf, sect->buf_max,
& out_used, out_used + dec_size)))
{
return ret;
}
if (sect->buf != sect->buf_max)
{
stream->msg = "secondary decoder finished with unused input";
return XD3_INTERNAL;
}
if (out_used != sect->copied2 + dec_size)
{
stream->msg = "secondary decoder short output";
return XD3_INTERNAL;
}
sect->buf = sect->copied2;
sect->buf_max = sect->copied2 + dec_size;
sect->size = dec_size;
return 0;
}
#if XD3_ENCODER
static inline int xd3_encode_bit (xd3_stream *stream,
xd3_output **output,
bit_state *bits,
usize_t bit)
{
int ret;
if (bit)
{
bits->cur_byte |= bits->cur_mask;
}
/* OPT: Might help to buffer more than 8 bits at once. */
if (bits->cur_mask == 0x80)
{
if ((ret = xd3_emit_byte (stream, output, bits->cur_byte)) != 0)
{
return ret;
}
bits->cur_mask = 1;
bits->cur_byte = 0;
}
else
{
bits->cur_mask <<= 1;
}
return 0;
}
static inline int xd3_flush_bits (xd3_stream *stream,
xd3_output **output,
bit_state *bits)
{
return (bits->cur_mask == 1) ? 0 :
xd3_emit_byte (stream, output, bits->cur_byte);
}
static inline int xd3_encode_bits (xd3_stream *stream,
xd3_output **output,
bit_state *bits,
usize_t nbits,
usize_t value)
{
int ret;
usize_t mask = 1 << nbits;
XD3_ASSERT (nbits > 0);
XD3_ASSERT (nbits < sizeof (usize_t) * 8);
XD3_ASSERT (value < mask);
do
{
mask >>= 1;
if ((ret = xd3_encode_bit (stream, output, bits, value & mask)))
{
return ret;
}
}
while (mask != 1);
IF_DEBUG2 (DP(RINT "(e) %"W"u ", value));
return 0;
}
static int
xd3_encode_secondary (xd3_stream *stream,
xd3_output **head,
xd3_output **tail,
xd3_sec_stream **sec_streamp,
xd3_sec_cfg *cfg,
int *did_it)
{
xd3_output *tmp_head;
xd3_output *tmp_tail;
usize_t comp_size;
usize_t orig_size;
int ret;
orig_size = xd3_sizeof_output (*head);
if (orig_size < SECONDARY_MIN_INPUT) { return 0; }
if ((ret = xd3_get_secondary (stream, sec_streamp, 1)) != 0)
{
return ret;
}
tmp_head = xd3_alloc_output (stream, NULL);
/* Encode the size, encode the data. Encoding the size makes it
* simpler, but is a little gross. Should not need the entire
* section in contiguous memory, but it is much easier this way. */
if ((ret = xd3_emit_size (stream, & tmp_head, orig_size)) ||
(ret = stream->sec_type->encode (stream, *sec_streamp, *head,
tmp_head, cfg)))
{
goto getout;
}
/* If the secondary compressor determines it's no good, it returns
* XD3_NOSECOND. */
/* Setup tmp_tail, comp_size */
tmp_tail = tmp_head;
comp_size = tmp_head->next;
while (tmp_tail->next_page != NULL)
{
tmp_tail = tmp_tail->next_page;
comp_size += tmp_tail->next;
}
XD3_ASSERT (comp_size == xd3_sizeof_output (tmp_head));
XD3_ASSERT (tmp_tail != NULL);
if (comp_size < (orig_size - SECONDARY_MIN_SAVINGS) || cfg->inefficient)
{
if (comp_size < orig_size)
{
IF_DEBUG1(DP(RINT "[encode_secondary] saved %"W"u bytes: %"W"u -> %"W"u (%0.2f%%)\n",
orig_size - comp_size, orig_size, comp_size,
100.0 * (double) comp_size / (double) orig_size));
}
xd3_free_output (stream, *head);
*head = tmp_head;
*tail = tmp_tail;
*did_it = 1;
}
else
{
getout:
if (ret == XD3_NOSECOND) { ret = 0; }
xd3_free_output (stream, tmp_head);
}
return ret;
}
#endif /* XD3_ENCODER */
#endif /* _XDELTA3_SECOND_H_ */

3022
lib/xdelta3/xdelta3-test.h Normal file
View file

File diff suppressed because it is too large Load diff

153
lib/xdelta3/xdelta3.1 Normal file
View file

@ -0,0 +1,153 @@
.TH XDELTA3 "1" "August 2009" "Xdelta3"
.SH NAME
xdelta3 \- VCDIFF (RFC 3284) binary diff tool
.SH SYNOPSIS
.B xdelta3
.RI [ command ]
.RI [ options ]
.RI [ input
.RI [ output ]]
.SH DESCRIPTION
.B xdelta3
is a binary diff tool that uses the VCDIFF (RFC 3284) format and compression.
.SH COMMANDS
.TP
.BI config
prints xdelta3 configuration
.TP
.BI decode
decompress the input, also set by -d
.TP
.BI encode
compress the input, also set by -e (default)
.TP
.BI test
run the builtin tests
.TP
.BI printdelta
print information about the entire delta
.TP
.BI printhdr
print information about the first window
.TP
.BI printhdrs
print information about all windows
.TP
.BI recode
encode with new application/secondary settings
.SH OPTIONS
standard options:
.TP
.BI "\-0 .. \-9"
compression level
.TP
.BI "\-c"
use stdout
.TP
.BI "\-d"
decompress
.TP
.BI \-e
compress
.TP
.BI \-f
force overwrite
.TP
.BI \-h
show help
.TP
.BI \-q
be quiet
.TP
.BI \-v
be verbose (max 2)
.TP
.BI \-V
show version
.TP
memory options:
.TP
.BI \-B
.RI bytes
source window size
.TP
.BI \-W
.RI bytes
input window size
.TP
.BI \-P
.RI size
compression duplicates window
.TP
.BI \-I
.RI size
instruction buffer size (0 = unlimited)
.TP
compression options:
.TP
.BI \-s
.RI source
source file to copy from (if any)
.TP
.BI "\-S " [djw|fgk]
enable/disable secondary compression
.TP
.BI \-N
disable small string-matching compression
.TP
.BI \-D
disable external decompression (encode/decode)
.TP
.BI \-R
disable external recompression (decode)
.TP
.BI \-n
disable checksum (encode/decode)
.TP
.BI \-C
soft config (encode, undocumented)
.TP
.BI "\-A " [apphead]
disable/provide application header (encode)
.TP
.BI \-J
disable output (check/compute only)
.TP
.BI \-T
use alternate code table (test)
.SH NOTES
The
.B XDELTA
environment variable may contain extra args:
.RS
XDELTA="-s source-x.y.tar.gz" \\
.br
tar --use-compress-program=xdelta3 -cf \\
.br
target-x.z.tar.gz.vcdiff target-x.y/
.SH EXAMPLES
Compress the differences between SOURCE and TARGET, yielding OUT,
using "djw" secondary compression:
xdelta3 -S djw -s SOURCE TARGET OUT
Do the same, using standard input and output:
xdelta3 -S djw -s SOURCE < TARGET > OUT
To decompress OUT, using SOURCE, yielding TARGET:
xdelta3 -d -s SOURCE OUT TARGET
.SH AUTHOR
xdelta3 was written by Josh MacDonald <josh.macdonald@gmail.com>.
.PP
This manual page was written by Leo 'costela' Antunes <costela@debian.org>
for the Debian project (but may be used by others).

4819
lib/xdelta3/xdelta3.c Normal file
View file

File diff suppressed because it is too large Load diff

1476
lib/xdelta3/xdelta3.h Normal file
View file

File diff suppressed because it is too large Load diff

85
lib/xdelta3/xdelta3.i Normal file
View file

@ -0,0 +1,85 @@
%module xdelta3
%import cstring.i
%import argcargv.i
%{
#include "xdelta3.h"
int xd3_main_cmdline (int ARGC, char **ARGV);
%}
%cstring_input_binary(const char *input, unsigned int input_size);
%cstring_input_binary(const char *source, unsigned int source_size);
%define %max_output_withsize(TYPEMAP, SIZE, MAXSIZE)
%typemap(in) MAXSIZE (unsigned int alloc_size) {
$1 = alloc_size = PyInt_AsLong(obj2);
}
%typemap(in,numinputs=0) (TYPEMAP, SIZE) {
}
%typemap(check) (TYPEMAP, SIZE) {
// alloc_size input is #7th position in xd3_xxcode_memory()
$1 = malloc(alloc_size7);
$2 = &alloc_size7;
}
%typemap(argout,fragment="t_output_helper") (TYPEMAP, SIZE) {
if (result == 0) {
PyObject *o;
// alloc_size7 now carries actual size
o = PyString_FromStringAndSize($1,alloc_size7);
$result = t_output_helper($result,o);
} else {
$result = t_output_helper($result,Py_None);
}
free($1);
}
%typemap(default) int flags {
$1 = 0;
}
%enddef
%max_output_withsize(char *output_buf, unsigned int *output_size, unsigned int max_output);
int xd3_encode_memory (const uint8_t *input,
usize_t input_size,
const uint8_t *source,
usize_t source_size,
uint8_t *output_buffer,
usize_t *output_size,
usize_t avail_output,
int flags);
int xd3_decode_memory (const uint8_t *input,
usize_t input_size,
const uint8_t *source,
usize_t source_size,
uint8_t *output_buf,
usize_t *output_size,
usize_t avail_output,
int flags);
int xd3_main_cmdline (int ARGC, char **ARGV);
/* Is this the right way? */
enum {
/*XD3_JUST_HDR,*/
/*XD3_SKIP_WINDOW,*/
/*XD3_SKIP_EMIT,*/
/*XD3_FLUSH,*/
XD3_SEC_DJW,
XD3_SEC_FGK,
/*XD3_SEC_TYPE,*/
XD3_SEC_NODATA,
XD3_SEC_NOINST,
XD3_SEC_NOADDR,
/*XD3_SEC_OTHER,*/
XD3_ADLER32,
XD3_ADLER32_NOVER,
XD3_NOCOMPRESS,
XD3_BEGREEDY,
XD3_COMPLEVEL_SHIFT,
XD3_COMPLEVEL_MASK,
XD3_COMPLEVEL_1,
XD3_COMPLEVEL_3,
XD3_COMPLEVEL_6,
XD3_COMPLEVEL_9,
};

344
lib/xdelta3/xdelta3.vcxproj Normal file
View file

@ -0,0 +1,344 @@
<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" ToolsVersion="12.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup Label="ProjectConfigurations">
<ProjectConfiguration Include="Debug|Itanium">
<Configuration>Debug</Configuration>
<Platform>Itanium</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Debug|Win32">
<Configuration>Debug</Configuration>
<Platform>Win32</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Debug|x64">
<Configuration>Debug</Configuration>
<Platform>x64</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Release|Itanium">
<Configuration>Release</Configuration>
<Platform>Itanium</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Release|Win32">
<Configuration>Release</Configuration>
<Platform>Win32</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Release|x64">
<Configuration>Release</Configuration>
<Platform>x64</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="xdelta3-64|Itanium">
<Configuration>xdelta3-64</Configuration>
<Platform>Itanium</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="xdelta3-64|Win32">
<Configuration>xdelta3-64</Configuration>
<Platform>Win32</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="xdelta3-64|x64">
<Configuration>xdelta3-64</Configuration>
<Platform>x64</Platform>
</ProjectConfiguration>
</ItemGroup>
<ItemGroup>
<ClCompile Include="xdelta3.c">
<PreprocessorDefinitions Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">%(PreprocessorDefinitions)</PreprocessorDefinitions>
<PreprocessorDefinitions Condition="'$(Configuration)|$(Platform)'=='Debug|Itanium'">%(PreprocessorDefinitions)</PreprocessorDefinitions>
<PreprocessorDefinitions Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">%(PreprocessorDefinitions)</PreprocessorDefinitions>
<PreprocessorDefinitions Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">/DXD3_DEBUG=0 /DXD3_USE_LARGEFILE64=1 /DREGRESSION_TEST=1 /DSECONDARY_DJW=1 /DSECONDARY_FGK=1 /DXD3_MAIN=1 /DXD3_WIN32=1 /DEXTERNAL_COMPRESSION=0 /DXD3_STDIO=0 /DXD3_POSIX=0;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<PreprocessorDefinitions Condition="'$(Configuration)|$(Platform)'=='Release|Itanium'">/DXD3_DEBUG=0 /DXD3_USE_LARGEFILE64=1 /DREGRESSION_TEST=1 /DSECONDARY_DJW=1 /DSECONDARY_FGK=1 /DXD3_MAIN=1 /DXD3_WIN32=1 /DEXTERNAL_COMPRESSION=0 /DXD3_STDIO=0 /DXD3_POSIX=0;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<PreprocessorDefinitions Condition="'$(Configuration)|$(Platform)'=='Release|x64'">/DXD3_DEBUG=0 /DXD3_USE_LARGEFILE64=1 /DREGRESSION_TEST=1 /DSECONDARY_DJW=1 /DSECONDARY_FGK=1 /DXD3_MAIN=1 /DXD3_WIN32=1 /DEXTERNAL_COMPRESSION=0 /DXD3_STDIO=0 /DXD3_POSIX=0;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<PreprocessorDefinitions Condition="'$(Configuration)|$(Platform)'=='xdelta3-64|Win32'">/DXD3_DEBUG=0 /DXD3_USE_LARGEFILE64=1 /DREGRESSION_TEST=1 /DSECONDARY_DJW=1 /DSECONDARY_FGK=1 /DXD3_MAIN=1 /DXD3_WIN32=1 /DEXTERNAL_COMPRESSION=0 /DXD3_STDIO=0 /DXD3_POSIX=0;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<PreprocessorDefinitions Condition="'$(Configuration)|$(Platform)'=='xdelta3-64|Itanium'">/DXD3_DEBUG=0 /DXD3_USE_LARGEFILE64=1 /DREGRESSION_TEST=1 /DSECONDARY_DJW=1 /DSECONDARY_FGK=1 /DXD3_MAIN=1 /DXD3_WIN32=1 /DEXTERNAL_COMPRESSION=0 /DXD3_STDIO=0 /DXD3_POSIX=0;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<PreprocessorDefinitions Condition="'$(Configuration)|$(Platform)'=='xdelta3-64|x64'">/DXD3_DEBUG=0 /DXD3_USE_LARGEFILE64=1 /DREGRESSION_TEST=1 /DSECONDARY_DJW=1 /DSECONDARY_FGK=1 /DXD3_MAIN=1 /DXD3_WIN32=1 /DEXTERNAL_COMPRESSION=0 /DXD3_STDIO=0 /DXD3_POSIX=0;%(PreprocessorDefinitions)</PreprocessorDefinitions>
</ClCompile>
</ItemGroup>
<ItemGroup>
<ClInclude Include="xdelta3-blkcache.h" />
<ClInclude Include="xdelta3-cfgs.h" />
<ClInclude Include="xdelta3-decode.h" />
<ClInclude Include="xdelta3-djw.h" />
<ClInclude Include="xdelta3-fgk.h" />
<ClInclude Include="xdelta3-hash.h" />
<ClInclude Include="xdelta3-internal.h" />
<ClInclude Include="xdelta3-list.h" />
<ClInclude Include="xdelta3-lzma.h" />
<ClInclude Include="xdelta3-main.h" />
<ClInclude Include="xdelta3-merge.h" />
<ClInclude Include="xdelta3-second.h" />
<ClInclude Include="xdelta3-test.h" />
<ClInclude Include="xdelta3.h" />
</ItemGroup>
<PropertyGroup Label="Globals">
<ProjectGuid>{8F9D37B5-B78E-4816-BE61-AEF679DBF3BC}</ProjectGuid>
<Keyword>Win32Proj</Keyword>
<RootNamespace>xdelta3</RootNamespace>
</PropertyGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>true</UseDebugLibraries>
<CharacterSet>MultiByte</CharacterSet>
<PlatformToolset>v120</PlatformToolset>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Itanium'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>true</UseDebugLibraries>
<CharacterSet>MultiByte</CharacterSet>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>true</UseDebugLibraries>
<CharacterSet>MultiByte</CharacterSet>
<PlatformToolset>v120</PlatformToolset>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>MultiByte</CharacterSet>
<PlatformToolset>v120</PlatformToolset>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Itanium'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>MultiByte</CharacterSet>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>MultiByte</CharacterSet>
<PlatformToolset>v120</PlatformToolset>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='xdelta3-64|Win32'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>MultiByte</CharacterSet>
<PlatformToolset>v120</PlatformToolset>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='xdelta3-64|Itanium'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>MultiByte</CharacterSet>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='xdelta3-64|x64'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>MultiByte</CharacterSet>
<PlatformToolset>v120</PlatformToolset>
</PropertyGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
<ImportGroup Label="ExtensionSettings">
</ImportGroup>
<ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Itanium'" Label="PropertySheets">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" Label="PropertySheets">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Condition="'$(Configuration)|$(Platform)'=='Release|Itanium'" Label="PropertySheets">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="PropertySheets">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Condition="'$(Configuration)|$(Platform)'=='xdelta3-64|Win32'" Label="PropertySheets">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Condition="'$(Configuration)|$(Platform)'=='xdelta3-64|Itanium'" Label="PropertySheets">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Condition="'$(Configuration)|$(Platform)'=='xdelta3-64|x64'" Label="PropertySheets">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<PropertyGroup Label="UserMacros" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<LinkIncremental>true</LinkIncremental>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Itanium'">
<LinkIncremental>true</LinkIncremental>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
<LinkIncremental>true</LinkIncremental>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<LinkIncremental>false</LinkIncremental>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Itanium'">
<LinkIncremental>false</LinkIncremental>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
<LinkIncremental>false</LinkIncremental>
<IncludePath>$(WindowsSdkDir)\include;$(VCInstallDir)include;..\xz\include</IncludePath>
<LibraryPath>$(LibraryPath);$(VSInstallDir);$(VSInstallDir)lib\amd64;..\xz\bin_x86-64</LibraryPath>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='xdelta3-64|Win32'">
<LinkIncremental>false</LinkIncremental>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='xdelta3-64|Itanium'">
<LinkIncremental>false</LinkIncremental>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='xdelta3-64|x64'">
<LinkIncremental>false</LinkIncremental>
</PropertyGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<ClCompile>
<PrecompiledHeader>NotUsing</PrecompiledHeader>
<WarningLevel>Level3</WarningLevel>
<Optimization>Disabled</Optimization>
<PreprocessorDefinitions>WIN32;XD3_MAIN=1;XD3_DEBUG=0;XD3_USE_LARGEFILE64=1;REGRESSION_TEST=1;SECONDARY_DJW=1;SECONDARY_FGK=1;XD3_WIN32=1;EXTERNAL_COMPRESSION=0;SHELL_TESTS=0;_DEBUG;_CONSOLE;SECONDARY_LZMA=0;LZMA_API_STATIC;SIZEOF_SIZE_T=4;SIZEOF_UNSIGNED_LONG_LONG=8;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<RuntimeLibrary>MultiThreaded</RuntimeLibrary>
<AdditionalIncludeDirectories>../xz/include</AdditionalIncludeDirectories>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
<AdditionalDependencies>kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies)</AdditionalDependencies>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Itanium'">
<ClCompile>
<PrecompiledHeader>NotUsing</PrecompiledHeader>
<WarningLevel>Level3</WarningLevel>
<Optimization>Disabled</Optimization>
<PreprocessorDefinitions>WIN32;XD3_MAIN=1;XD3_DEBUG=0;XD3_USE_LARGEFILE64=1;REGRESSION_TEST=1;SECONDARY_DJW=1;SECONDARY_FGK=1;XD3_WIN32=1;EXTERNAL_COMPRESSION=0;SHELL_TESTS=0;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<RuntimeLibrary>MultiThreaded</RuntimeLibrary>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
<ClCompile>
<PrecompiledHeader>NotUsing</PrecompiledHeader>
<WarningLevel>Level3</WarningLevel>
<Optimization>Disabled</Optimization>
<PreprocessorDefinitions>WIN32;XD3_MAIN=1;XD3_DEBUG=0;XD3_USE_LARGEFILE64=1;REGRESSION_TEST=1;SECONDARY_DJW=1;SECONDARY_FGK=1;XD3_WIN32=1;EXTERNAL_COMPRESSION=0;SHELL_TESTS=0;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<RuntimeLibrary>MultiThreaded</RuntimeLibrary>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
<AdditionalDependencies>kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;..\..\..\..\src\xz\bin_x86-64\liblzma_static.lib;%(AdditionalDependencies)</AdditionalDependencies>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<ClCompile>
<WarningLevel>Level3</WarningLevel>
<PrecompiledHeader>NotUsing</PrecompiledHeader>
<Optimization>MaxSpeed</Optimization>
<FunctionLevelLinking>true</FunctionLevelLinking>
<IntrinsicFunctions>true</IntrinsicFunctions>
<PreprocessorDefinitions>WIN32;XD3_MAIN=1;XD3_DEBUG=0;XD3_USE_LARGEFILE64=1;REGRESSION_TEST=1;SECONDARY_DJW=1;SECONDARY_FGK=1;SECONDARY_LZMA=1;XD3_WIN32=1;EXTERNAL_COMPRESSION=0;SHELL_TESTS=0;_DEBUG;_CONSOLE;LZMA_API_STATIC;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<RuntimeLibrary>MultiThreaded</RuntimeLibrary>
<AdditionalIncludeDirectories>..\..\..\..\src\xz\include</AdditionalIncludeDirectories>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
<EnableCOMDATFolding>true</EnableCOMDATFolding>
<OptimizeReferences>true</OptimizeReferences>
<AdditionalDependencies>kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;..\..\..\..\src\xz\bin_i486\liblzma_static.lib;%(AdditionalDependencies)</AdditionalDependencies>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|Itanium'">
<ClCompile>
<WarningLevel>Level3</WarningLevel>
<PrecompiledHeader>NotUsing</PrecompiledHeader>
<Optimization>MaxSpeed</Optimization>
<FunctionLevelLinking>true</FunctionLevelLinking>
<IntrinsicFunctions>true</IntrinsicFunctions>
<PreprocessorDefinitions>WIN32;XD3_MAIN=1;XD3_DEBUG=0;XD3_USE_LARGEFILE64=1;REGRESSION_TEST=1;SECONDARY_DJW=1;SECONDARY_FGK=1;XD3_WIN32=1;EXTERNAL_COMPRESSION=0;SHELL_TESTS=0;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<RuntimeLibrary>MultiThreaded</RuntimeLibrary>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
<EnableCOMDATFolding>true</EnableCOMDATFolding>
<OptimizeReferences>true</OptimizeReferences>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
<ClCompile>
<WarningLevel>Level3</WarningLevel>
<PrecompiledHeader>NotUsing</PrecompiledHeader>
<Optimization>MaxSpeed</Optimization>
<FunctionLevelLinking>true</FunctionLevelLinking>
<IntrinsicFunctions>true</IntrinsicFunctions>
<PreprocessorDefinitions>WIN32;XD3_MAIN=1;XD3_DEBUG=0;XD3_USE_LARGEFILE64=1;REGRESSION_TEST=1;SECONDARY_DJW=1;SECONDARY_FGK=1;SECONDARY_LZMA=1;XD3_WIN32=1;EXTERNAL_COMPRESSION=0;SHELL_TESTS=0;_DEBUG;_CONSOLE;LZMA_API_STATIC;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<RuntimeLibrary>MultiThreaded</RuntimeLibrary>
<AdditionalIncludeDirectories>..\..\..\..\src\xz\include</AdditionalIncludeDirectories>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
<EnableCOMDATFolding>true</EnableCOMDATFolding>
<OptimizeReferences>true</OptimizeReferences>
<AdditionalDependencies>kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies);..\..\..\..\src\xz\bin_x86-64\liblzma_static.lib</AdditionalDependencies>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='xdelta3-64|Win32'">
<ClCompile>
<WarningLevel>Level3</WarningLevel>
<PrecompiledHeader>NotUsing</PrecompiledHeader>
<Optimization>MaxSpeed</Optimization>
<FunctionLevelLinking>true</FunctionLevelLinking>
<IntrinsicFunctions>true</IntrinsicFunctions>
<PreprocessorDefinitions>WIN32;XD3_MAIN=1;XD3_DEBUG=0;XD3_USE_LARGEFILE64=1;REGRESSION_TEST=1;SECONDARY_DJW=1;SECONDARY_FGK=1;XD3_WIN32=1;EXTERNAL_COMPRESSION=0;SHELL_TESTS=0;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<RuntimeLibrary>MultiThreaded</RuntimeLibrary>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
<EnableCOMDATFolding>true</EnableCOMDATFolding>
<OptimizeReferences>true</OptimizeReferences>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='xdelta3-64|Itanium'">
<ClCompile>
<WarningLevel>Level3</WarningLevel>
<PrecompiledHeader>NotUsing</PrecompiledHeader>
<Optimization>MaxSpeed</Optimization>
<FunctionLevelLinking>true</FunctionLevelLinking>
<IntrinsicFunctions>true</IntrinsicFunctions>
<PreprocessorDefinitions>WIN32;XD3_MAIN=1;XD3_DEBUG=0;XD3_USE_LARGEFILE64=1;REGRESSION_TEST=1;SECONDARY_DJW=1;SECONDARY_FGK=1;XD3_WIN32=1;EXTERNAL_COMPRESSION=0;SHELL_TESTS=0;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<RuntimeLibrary>MultiThreaded</RuntimeLibrary>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
<EnableCOMDATFolding>true</EnableCOMDATFolding>
<OptimizeReferences>true</OptimizeReferences>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='xdelta3-64|x64'">
<ClCompile>
<WarningLevel>Level3</WarningLevel>
<PrecompiledHeader>NotUsing</PrecompiledHeader>
<Optimization>MaxSpeed</Optimization>
<FunctionLevelLinking>true</FunctionLevelLinking>
<IntrinsicFunctions>true</IntrinsicFunctions>
<PreprocessorDefinitions>WIN32;XD3_MAIN=1;XD3_DEBUG=0;XD3_USE_LARGEFILE64=1;REGRESSION_TEST=1;SECONDARY_DJW=1;SECONDARY_FGK=1;XD3_WIN32=1;EXTERNAL_COMPRESSION=0;SHELL_TESTS=0;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<RuntimeLibrary>MultiThreaded</RuntimeLibrary>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
<EnableCOMDATFolding>true</EnableCOMDATFolding>
<OptimizeReferences>true</OptimizeReferences>
</Link>
</ItemDefinitionGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
<ImportGroup Label="ExtensionTargets">
</ImportGroup>
</Project>

7
lib/xdelta3/xdelta3.wxi Normal file
View file

@ -0,0 +1,7 @@
<Include>
<?define PRODUCT_ID=60131be5-be4d-4975-9108-dd0be735890d ?>
<?define PACKAGE_ID=82bf21ca-ee08-4701-ab78-37210dac82ce ?>
<?define COMPONENT_ID=85bc3206-05f8-41f8-b500-6ea32e5d6a8f ?>
<?define MANUAL_ID=07f387bc-a0c5-4af9-88db-1a84443f1fc5 ?>
<?define SOURCE_ID=4e1503a9-3ed1-4e06-b0c0-890462b1a4fd ?>
</Include>

131
lib/xdelta3/xdelta3.wxs Normal file
View file

@ -0,0 +1,131 @@
<?xml version='1.0'?>
<?include $(sys.SOURCEFILEDIR)\xdelta3.wxi ?>
<Wix xmlns='http://schemas.microsoft.com/wix/2003/01/wi'>
<Product Id='$(var.PRODUCT_ID)'
Name='Xdelta 3.0u'
Language='1033'
Codepage='1252'
Version='3.0.1.1'
Manufacturer='Josh.MacDonald@Gmail.Com'>
<Package Id='$(var.PACKAGE_ID)'
Keywords='Installer'
Description='Xdelta 3.0u'
Comments='http://xdelta.org'
Manufacturer='Josh.MacDonald@Gmail.Com'
InstallerVersion='300'
Languages='1033'
Compressed='yes' />
<Media Id='1'
Cabinet='xdelta30t.cab'
EmbedCab='yes' />
<Directory Id='TARGETDIR' Name='SourceDir'>
<Directory Id='ProgramFilesFolder' Name='PFiles'>
<Directory Id='Xdelta'
Name='Xdelta'>
<Component Id='Main'
Guid='$(var.COMPONENT_ID)'>
<File Id='XdeltaEXE'
Name='xdelt30t'
LongName='xdelta30t.exe'
DiskId='1'
Source='G:\jmacd\svn\xdelta3\Release\xdelta3.exe'
Vital='yes'>
</File>
</Component>
<Component Id='Readme'
Guid='$(var.MANUAL_ID)'>
<File Id='Readme'
Name='readme.txt'
LongName='readme.txt'
DiskId='1'
Source='G:\jmacd\svn\xdelta3\readme.txt'
Vital='yes'>
<Shortcut Id="startupmenuReadme"
Directory="ProgramMenuDir"
Name="readme.txt"
LongName="Xdelta3 readme.txt"
/>
</File>
</Component>
<Component Id='Copyright'
Guid='$(var.MANUAL_ID)'>
<File Id='Copyright'
Name='COPYING'
LongName='COPYING'
DiskId='1'
Source='G:\jmacd\svn\xdelta3\COPYING'
Vital='yes'>
<Shortcut Id="startupmenuCopyright"
Directory="ProgramMenuDir"
Name="COPYING"
LongName="GNU Public License"
/>
</File>
</Component>
<Component Id='Source'
Guid='$(var.SOURCE_ID)'>
<File Id='Source'
Name='xdelt30t.zip'
LongName='xdelta3.0u.zip'
DiskId='1'
Source='G:\jmacd\svn\xdelta3\xdelta3.0u.zip'
Vital='yes'>
<Shortcut Id="startupmenuSource"
Directory="ProgramMenuDir"
Name="xdelt30t.zip"
LongName="xdelta3.0u.zip"
/>
</File>
</Component>
</Directory>
</Directory>
<Directory Id="ProgramMenuFolder" Name="PMenu" LongName="Programs">
<Directory Id="ProgramMenuDir"
Name="xdelt30t"
LongName="Xdelta 3.0u">
</Directory>
</Directory>
<!-- <Merge Id='CRT' -->
<!-- Language='0' -->
<!-- DiskId='1' -->
<!-- src='C:\Program Files\Common Files\Merge Modules\microsoft_vc80_crt_x86.msm' -->
<!-- /> -->
<!-- <Merge Id='CRT Policy' -->
<!-- Language='0' -->
<!-- DiskId='1' -->
<!-- src='C:\Program Files\Common Files\Merge Modules\policy_8_0_Microsoft_VC80_CRT_x86.msm' -->
<!-- /> -->
</Directory>
<Feature Id='Complete'
Level='1'>
<ComponentRef Id='Main' />
<ComponentRef Id='Readme' />
<ComponentRef Id='Copyright' />
<ComponentRef Id='Source' />
</Feature>
<!-- <Feature Id='CRT_WinSXS' Title='CRT WinSXS' Level='1'> -->
<!-- <MergeRef Id='CRT' /> -->
<!-- <MergeRef Id='CRT Policy' /> -->
<!-- </Feature> -->
<InstallExecuteSequence>
<RemoveRegistryValues/>
<RemoveFiles/>
<InstallFiles/>
<WriteRegistryValues/>
</InstallExecuteSequence>
</Product>
</Wix>

2
src/errorAndExit.h
View file

@ -1 +1,3 @@
#pragma once
void errorAndExit(); void errorAndExit();

2
src/parseArguments.h
View file

@ -1,3 +1,5 @@
#pragma once
#include <stdbool.h> #include <stdbool.h>
struct arguments { struct arguments {

2
src/utils/min.h
View file

@ -1,3 +1,5 @@
#pragma once
//minimum of two integers //minimum of two integers
#define min(a,b) \ #define min(a,b) \
({ __typeof__ (a) _a = (a); \ ({ __typeof__ (a) _a = (a); \

8
src/xdelta3.c Normal file
View file

@ -0,0 +1,8 @@
//To fix compile errors with xdelta3
#define SIZEOF_SIZE_T 4
#define static_assert(e,m) /* do nothing */
#define XD3_ENCODER 0
typedef unsigned int usize_t;
typedef unsigned long long xoff_t;
#include "../lib/xdelta3/xdelta3.h"