                              DB_File

                           Version 1.859

                           21 August 2023

 Copyright (c) 1995-2023 Paul Marquess. All rights reserved. This
  program is free software; you can redistribute it and/or modify
              it under the same terms as Perl itself.

IMPORTANT NOTICE

If are using the locking technique described in older versions of DB_File, please read the section called "Locking: The Trouble with fd" in DB_File.pm immediately. The locking method has been found to be unsafe. You risk corrupting your data if you continue to use it.

DESCRIPTION

DB_File is a module which allows Perl programs to make use of the facilities provided by Berkeley DB version 1. (DB_File can be built version 2, or greater, of Berkeley DB, but it will only support the 1.x features).

If you want to make use of the new features available in Berkeley DB 2.x, or greater, use the Perl module BerkeleyDB instead.

Berkeley DB is a C library which provides a consistent interface to a number of database formats. DB_File provides an interface to all three of the database types (hash, btree and recno) currently supported by Berkeley DB.

For further details see the documentation included at the end of the file DB_File.pm.

PREREQUISITES

Before you can build DB_File you must have the following installed on your system:

* Perl 5.8.3 or greater.

* Berkeley DB.

  The official web site for Berkeley DB is

    http://www.oracle.com/technology/products/berkeley-db/db/index.html

  The latest version of Berkeley DB is always available there. It
  is recommended that you use the most recent version available.

  The one exception to this advice is where you want to use DB_File
  to access database files created by a third-party application, like
  Sendmail or Netscape. In these cases you must build DB_File with a
  compatible version of Berkeley DB.

  If you want to use Berkeley DB 2.x, you must have version 2.3.4
  or greater.  If you want to use Berkeley DB 3.x or 4.x, any version
  will do. For Berkeley DB 1.x, use either version 1.85 or 1.86.

BUILDING THE MODULE

Assuming you have met all the prerequisites, building the module should be relatively straightforward.

Step 1 : If you are running either Solaris 2.5 or HP-UX 10 and want to use Berkeley DB version 2, 3 or 4, read either the Solaris Notes or HP-UX Notes sections below. If you are running Linux please read the Linux Notes section before proceeding.

Step 2 : Edit the file config.in to suit you local installation. Instructions are given in the file.

Step 3 : Build and test the module using this sequence of commands:

         perl Makefile.PL
         make
         make test

NOTE: If you have a very old version of Berkeley DB (i.e. pre 1.85), three of the tests in the recno test harness may fail (tests 51, 53 and 55). You can safely ignore the errors if you're never going to use the broken functionality (recno databases with a modified bval). Otherwise you'll have to upgrade your DB library.

INSTALLATION

make install

UPDATES

The most recent version of DB_File is always available at

http://www.cpan.org/modules/by-module/DB_File/

TROUBLESHOOTING

Here are some of the common problems people encounter when building DB_File.

Missing db.h or libdb.a

If you get an error like this:

cc -c -I/usr/local/include -Dbool=char -DHAS_BOOL -O2 -DVERSION="1.64" -DXS_VERSION="1.64" -fpic -I/usr/local/lib/perl5/i586-linux/5.00404/CORE -DmDB_Prefix_t=size_t -DmDB_Hash_t=u_int32_t DB_File.c DB_File.xs:101: db.h: No such file or directory

or this:

LD_RUN_PATH="/lib" cc -o blib/arch/auto/DB_File/DB_File.so -shared -L/usr/local/lib DB_File.o -L/usr/local/lib -ldb ld: cannot open -ldb: No such file or directory

This symptom can imply:

You don't have Berkeley DB installed on your system at all. Solution: get & install Berkeley DB.
You do have Berkeley DB installed, but it isn't in a standard place. Solution: Edit config.in and set the LIB and INCLUDE variables to point to the directories where libdb.a and db.h are installed.

Undefined symbol db_version

DB_File seems to have built correctly, but you get an error like this when you run the test harness:

$ make test PERL_DL_NONLAZY=1 /usr/bin/perl5.00404 -I./blib/arch -I./blib/lib -I/usr/local/lib/perl5/i586-linux/5.00404 -I/usr/local/lib/perl5 -e 'use Test::Harness qw(&runtests $verbose); $verbose=0; runtests @ARGV;' t/*.t t/db-btree..........Can't load './blib/arch/auto/DB_File/DB_File.so' for module DB_File: ./blib/arch/auto/DB_File/DB_File.so: undefined symbol: db_version at /usr/local/lib/perl5/i586-linux/5.00404/DynaLoader.pm line 166.

at t/db-btree.t line 21 BEGIN failed--compilation aborted at t/db-btree.t line 21. dubious Test returned status 2 (wstat 512, 0x200)

This error usually happens when you have two version of Berkeley DB installed on your system -- specifically, if you have both version 1 and a newer version (i.e. version 2 or better) of Berkeley DB installed. If DB_File is built using the db.h for the newer Berkeley DB and the version 1 Berkeley DB library you will trigger this error. Unfortunately the two versions aren't compatible with each other. The undefined symbol error is caused because Berkeley DB version 1 doesn't have the symbol db_version.

Solution: Setting the LIB & INCLUDE variables in config.in to point to the correct directories can sometimes be enough to fix this problem. If that doesn't work the easiest way to fix the problem is to either delete or temporarily rename the copies of db.h and libdb.a that you don't want DB_File to use.

Undefined symbol dbopen

DB_File seems to have built correctly, but you get an error like this when you run the test harness:

... t/db-btree..........Can't load 'blib/arch/auto/DB_File/DB_File.so' for module DB_File: blib/arch/auto/DB_File/DB_File.so: undefined symbol: dbopen at /usr/local/lib/perl5/5.6.1/i586-linux/DynaLoader.pm line 206. at t/db-btree.t line 23 Compilation failed in require at t/db-btree.t line 23. ...

This error usually happens when you have both version 1 and a more recent version of Berkeley DB installed on your system and DB_File attempts to build using the db.h for Berkeley DB version 1 and the newer version library. Unfortunately the two versions aren't compatible with each other. The undefined symbol error is actually caused because versions of Berkeley DB newer than version 1 doesn't have the symbol dbopen.

Incompatible versions of db.h and libdb

DB_File seems to have built correctly, but you get an error like this when you run the test harness:

$ make test PERL_DL_NONLAZY=1 /home/paul/perl/install/bin/perl5.00560 -Iblib/arch -Iblib/lib -I/home/paul/perl/install/5.005_60/lib/5.00560/i586-linux -I/home/paul/perl/install/5.005_60/lib/5.00560 -e 'use Test::Harness qw(&runtests $verbose); $verbose=0; runtests @ARGV;' t/*.t t/db-btree.......... DB_File was build with libdb version 2.3.7 but you are attempting to run it with libdb version 2.7.5 BEGIN failed--compilation aborted at t/db-btree.t line 21. ...

Another variation on the theme of having two versions of Berkeley DB on your system.

Keep getting "At least one secondary cursor must be specified to DB->join"

When you either run the DB_File test harness, or attempt to run a script that uses DB_File you get the error message below.

BDB0588 At least one secondary cursor must be specified to DB->join.

To date thus issue has only been reported on Windows. If you encounter this issue on another platform, please report the issue. See the FEEDBACK section for details.

This issue boils down to the size of the C type time_t.

The typical reason for getting this error is when running a 32-bit Perl (which will use a 32-bit time_t) along with a 32-bit Berkeley DB library. The key point is how Berkeley DB has been built. If it has been built with a newish version of Visual C++, time_t will default to 64-bit, even when the rest of the library has been built 32-bit. This means that Perl thinks time_t is 32-bit, but Berkeley DB thinks it is 64-bit.

More details of how the size of time_t in Windows is shown below (taken from http://msdn.microsoft.com/en-us/library/w4ddyt9h.aspx)

In versions of Visual C++ and Microsoft C/C++ before Visual C++ 2005,
time_t was a long int (32 bits) and hence could not be used for dates
past 3:14:07 January 19, 2038, UTC. time_t is now equivalent to
__time64_t by default, but defining _USE_32BIT_TIME_T changes time_t
to __time32_t and forces many time functions to call versions that take
the 32-bit time_t.

When DB_File is built, it uses a Berkeley DB header file, called db.h. This file contains the definition of a number of key data structures used by Berkeley DB. Unfortunately one of those data structures includes a time_t. This is the root case for this issue. When you build DB_File, it assumes time_t is 32-bit, but the Berkeley DB library is expecting it to be 64-bit.

Solution:

There are a few options available.

Use a 64-bit Perl along with a 64-bit Berkleley DB.
Use a 32-bit Perl along with a 32-bit Berkeley DB where _USE_32BIT_TIME_T has been defined.
If you do need to interoperate with a Berkeley DB library that uses a time_t that is different from Perl you need to edit the file DB_File.xs and find these lines
```
 /* #define time_t __time64_t */
 /* #define time_t __time32_t */
```
If your Berkeley DB library uses a 64-bit time_t, uncomment the first line. If your Berkeley DB library uses a 32-bit time_t, uncomment the second line.

MacOS Notes

You are running MacOS 10.13 (or possibly later), and the test DB_File test harness file db-hash.t fails like this

t/db-btree.t .. ok
t/db-hash.t ... Use of uninitialized value $value in string eq at t/db-hash.t line 224.
Use of uninitialized value $values[0] in string eq at t/db-hash.t line 224.
Use of uninitialized value $value in lc at t/db-hash.t line 224.
Use of uninitialized value $h{""} in string eq at t/db-hash.t line 243.
Use of uninitialized value in numeric eq (==) at t/db-hash.t line 252.
Use of uninitialized value in numeric eq (==) at t/db-hash.t line 252.
Use of uninitialized value in numeric eq (==) at t/db-hash.t line 252.
Use of uninitialized value in numeric eq (==) at t/db-hash.t line 252.
Use of uninitialized value in numeric eq (==) at t/db-hash.t line 252.
Use of uninitialized value in numeric eq (==) at t/db-hash.t line 252.
Use of uninitialized value in numeric eq (==) at t/db-hash.t line 252.
Use of uninitialized value in numeric eq (==) at t/db-hash.t line 252.
Use of uninitialized value $foo[18] in join or string at t/db-hash.t line 261.
Use of uninitialized value $foo[36] in join or string at t/db-hash.t line 261.
Use of uninitialized value $foo[48] in join or string at t/db-hash.t line 261.
Use of uninitialized value $foo[58] in join or string at t/db-hash.t line 261.
Use of uninitialized value $foo[59] in join or string at t/db-hash.t line 261.
Use of uninitialized value $foo[60] in join or string at t/db-hash.t line 261.
Use of uninitialized value $foo[62] in join or string at t/db-hash.t line 261.
Use of uninitialized value $foo[63] in join or string at t/db-hash.t line 261.
Use of uninitialized value $foo[92] in join or string at t/db-hash.t line 261.
Use of uninitialized value $foo[114] in join or string at t/db-hash.t line 261.
Use of uninitialized value $foo[140] in join or string at t/db-hash.t line 261.
Use of uninitialized value $foo[187] in join or string at t/db-hash.t line 261.
Use of uninitialized value $foo[188] in join or string at t/db-hash.t line 261.
Use of uninitialized value $foo[189] in join or string at t/db-hash.t line 261.
t/db-hash.t ... 1/166 Use of uninitialized value $h{"Fred"} in string eq at t/db-hash.t line 572.
Use of uninitialized value $v in concatenation (.) or string at t/db-hash.t line 748.
t/db-hash.t ... Dubious, test returned 25 (wstat 6400, 0x1900)
Failed 76/166 subtests
t/db-recno.t .. ok
t/pod.t ....... ok

It appears that MacOS 10.13 has a copy of BerkeleyDB version 1 included in /usr/lib/system/libsystem_c.dylib. It is not clear why this isn't working with this module. The following ticket has been reported to Apple https://discussions.apple.com/thread/8125401

Solution

It has been reported that this issue can be resolved by installing a separate version of the Berkeley DB library and configuring DB_File to use it.

One option is to use a prebuild version of Berkeley DB. One is available at MacPorts (https://www.macports.org/). Once you have MacPorts installed, run this

sudo port install db48

To get DB_File to use this library edit config.in and change the values for the LIB and include variables as follows

LIB        = /opt/local/lib/db48
INCLUDE    = /opt/local/include/db48

Alternatively, set the two envornment variables DB_FILE_LIB and DB_FILE_INCLUDE as follows

export DB_FILE_LIB=/opt/local/lib/db48
export DB_FILE_INCLUDE=/opt/local/include/db48

Now rebuild DB_File from scratch.

Solaris build fails with "language optional software package not installed"

If you are trying to build this module under Solaris and you get an error message like this

/usr/ucb/cc: language optional software package not installed

it means that Perl cannot find the C compiler on your system. The cryptic message is just Sun's way of telling you that you haven't bought their C compiler.

When you build a Perl module that needs a C compiler, the Perl build system tries to use the same C compiler that was used to build perl itself. In this case your Perl binary was built with a C compiler that lived in /usr/ucb.

To continue with building this module, you need to get a C compiler, or tell Perl where your C compiler is, if you already have one.

Assuming you have now got a C compiler, what you do next will be dependant on what C compiler you have installed. If you have just installed Sun's C compiler, you shouldn't have to do anything. Just try rebuilding this module.

If you have installed another C compiler, say gcc, you have to tell perl how to use it instead of /usr/ucb/cc.

This set of options seems to work if you want to use gcc. Your mileage may vary.

perl Makefile.PL CC=gcc CCCDLFLAGS=-fPIC OPTIMIZE=" "
make test

If that doesn't work for you, it's time to make changes to the Makefile by hand. Good luck!

Solaris build fails with "gcc: unrecognized option `-KPIC'"

You are running Solaris and you get an error like this when you try to build this Perl module

gcc: unrecognized option `-KPIC'

This symptom usually means that you are using a Perl binary that has been built with the Sun C compiler, but you are using gcc to build this module.

When Perl builds modules that need a C compiler, it will attempt to use the same C compiler and command line options that was used to build perl itself. In this case "-KPIC" is a valid option for the Sun C compiler, but not for gcc. The equivalent option for gcc is "-fPIC".

The solution is either:

1. Build both Perl and this module with the same C compiler, either
   by using the Sun C compiler for both or gcc for both.

2. Try generating the Makefile for this module like this perl

       perl Makefile.PL CC=gcc CCCDLFLAGS=-fPIC OPTIMIZE=" " LD=gcc
       make test

   This second option seems to work when mixing a Perl binary built
   with the Sun C compiler and this module built with gcc. Your
   mileage may vary.

Linux Notes

Some older versions of Linux (e.g. RedHat 6, SuSe 6) ship with a C library that has version 2.x of Berkeley DB linked into it. This makes it difficult to build this module with anything other than the version of Berkeley DB that shipped with your Linux release. If you do try to use a different version of Berkeley DB you will most likely get the error described in the "Incompatible versions of db.h and libdb" section of this file.

To make matters worse, prior to Perl 5.6.1, the perl binary itself always included the Berkeley DB library.

If you want to use a newer version of Berkeley DB with this module, the easiest solution is to use Perl 5.6.1 (or better) and Berkeley DB 3.x (or better).

There are two approaches you can use to get older versions of Perl to work with specific versions of Berkeley DB. Both have their advantages and disadvantages.

The first approach will only work when you want to build a version of Perl older than 5.6.1 along with Berkeley DB 3.x. If you want to use Berkeley DB 2.x, you must use the next approach. This approach involves rebuilding your existing version of Perl after applying an unofficial patch. The "patches" directory in the this module's source distribution contains a number of patch files. There is one patch file for every stable version of Perl since 5.004. Apply the appropriate patch to your Perl source tree before re-building and installing Perl from scratch. For example, assuming you are in the top-level source directory for Perl 5.6.0, the command below will apply the necessary patch. Remember to replace the path shown below with one that points to this module's patches directory.

patch -p1 -N </path/to/DB_File/patches/5.6.0

Now rebuild & install perl. You should now have a perl binary that can be used to build this module. Follow the instructions in "BUILDING THE MODULE", remembering to set the INCLUDE and LIB variables in config.in.

The second approach will work with both Berkeley DB 2.x and 3.x. Start by building Berkeley DB as a shared library. This is from the Berkeley DB build instructions:

Building Shared Libraries for the GNU GCC compiler

If you're using gcc and there's no better shared library example for
your architecture, the following shared library build procedure will
probably work.

Add the -fpic option to the CFLAGS value in the Makefile.

Rebuild all of your .o files. This will create a Berkeley DB library
that contains .o files with PIC code. To build the shared library,
then take the following steps in the library build directory:

% mkdir tmp
% cd tmp
% ar xv ../libdb.a
% gcc -shared -o libdb.so *.o
% mv libdb.so ..
% cd ..
% rm -rf tmp

Note, you may have to change the gcc line depending on the
requirements of your system.

The file libdb.so is your shared library

Once you have built libdb.so, you will need to store it somewhere safe.

cp libdb.so /usr/local/BerkeleyDB/lib

If you now set the LD_PRELOAD environment variable to point to this shared library, Perl will use it instead of the version of Berkeley DB that shipped with your Linux distribution.

export LD_PRELOAD=/usr/local/BerkeleyDB/lib/libdb.so

Finally follow the instructions in "BUILDING THE MODULE" to build, test and install this module. Don't forget to set the INCLUDE and LIB variables in config.in.

Remember, you will need to have the LD_PRELOAD variable set anytime you want to use Perl with Berkeley DB. Also note that if you have LD_PRELOAD permanently set it will affect ALL commands you execute. This may be a problem if you run any commands that access a database created by the version of Berkeley DB that shipped with your Linux distribution.

Solaris Notes

If you are running Solaris 2.5, and you get this error when you run the DB_File test harness:

libc internal error: _rmutex_unlock: rmutex not held.

you probably need to install a Sun patch. It has been reported that Sun patch 103187-25 (or later revisions) fixes this problem.

To find out if you have the patch installed, the command "showrev -p" will display the patches that are currently installed on your system.

HP-UX 10 Notes

Some people running HP-UX 10 have reported getting an error like this when building DB_File with the native HP-UX compiler.

ld: (Warning) At least one PA 2.0 object file (DB_File.o) was detected.
The linked output may not run on a PA 1.x system.
ld: Invalid loader fixup for symbol "$000000A5".

If this is the case for you, Berkeley DB needs to be recompiled with the +z or +Z option and the resulting library placed in a .sl file. The following steps should do the trick:

1: Configure the Berkeley DB distribution with the +z or +Z C compiler flag:

    env "CFLAGS=+z" ../dist/configure ...

2: Edit the Berkeley DB Makefile and change:

    "libdb= libdb.a" to "libdb= libdb.sl".

3: Build and install the Berkeley DB distribution as usual.

HP-UX 11 Notes

Some people running the combination of HP-UX 11 and Berkeley DB 2.7.7 have reported getting this error when the run the test harness for DB_File

...
lib/db-btree.........Can't call method "DELETE" on an undefined value at lib/db-btree.t line 216.
FAILED at test 26
lib/db-hash..........Can't call method "DELETE" on an undefined value at lib/db-hash.t line 183.
FAILED at test 22
...

The fix for this is to rebuild and install Berkeley DB with the bigfile option disabled.

AIX NOTES

I've had reports of a build failure like this on AIX 5.2 using the xlC compiler.

    rm -f blib/arch/auto/DB_File/DB_File.so
    LD_RUN_PATH="" ld  -bhalt:4 -bM:SRE -bI:/usr/local/5.8.1/lib/perl5/5.8.1/aix/CORE/perl.exp -bE:DB_File.exp -bnoentry -lc
-L/usr/local/lib version.o DB_File.o  -o blib/arch/auto/DB_File/DB_File.so
-L/usr/local/BerkeleyDB/lib -ldb -lpthread
ld: 0711-317 ERROR: Undefined symbol: .mutex_lock
ld: 0711-317 ERROR: Undefined symbol: .cond_signal
ld: 0711-317 ERROR: Undefined symbol: .mutex_unlock
ld: 0711-317 ERROR: Undefined symbol: .mutex_trylock
ld: 0711-317 ERROR: Undefined symbol: .cond_wait
ld: 0711-317 ERROR: Undefined symbol: .mutex_init
ld: 0711-317 ERROR: Undefined symbol: .cond_init
ld: 0711-317 ERROR: Undefined symbol: .mutex_destroy
ld: 0711-345 Use the -bloadmap or -bnoquiet option to obtain more information.
make: 1254-004 The error code from the last command is 8.

Editing Makefile.PL, and changing the line

$LIBS .= " -lpthread" if $^O eq 'aix' ;

to this

$LIBS .= " -lthread" if $^O eq 'aix' ;

fixed the problem.

FEEDBACK

General feedback/questions/bug reports can be sent to https://github.com/pmqs/DB_File/issues (preferred) or https://rt.cpan.org/Public/Dist/Display.html?Name=DB_File

Alternatively, email me at [email protected].

How to report a problem with DB_File.

When reporting any problem, I need the information requested below.

The complete output from running this
```
perl -V
```
Do not edit the output in any way. Note, I want you to run "perl -V" and NOT "perl -v".

If your perl does not understand the "-V" option it is too old. DB_File needs Perl version 5.00405 or better.
The version of DB_File you have. If you have successfully installed DB_File, this one-liner will tell you:

perl -e 'use DB_File; print qq{DB_File ver $DB_File::VERSION\n}'

If you are running windows use this

perl -e "use DB_File; print qq{DB_File ver $DB_File::VERSION\n}"

If you haven't installed DB_File then search DB_File.pm for a line like this:

$VERSION = "1.20" ;
The version of Berkeley DB used to build DB_File and the version that is used at runtime. (These are usually the same)

If you are using a version older than 1.85, think about upgrading. One point to note if you are considering upgrading Berkeley DB - the file formats for 1.85, 1.86, 2.0, 3.0 & 3.1 are all different.

If you have successfully installed DB_File, these commands will display the versions I need

perl -MDB_File -e 'print qq{Built with Berkeley DB ver $DB_File::db_ver\n}' perl -MDB_File -e 'print qq{Running with Berkeley DB ver $DB_File::db_version\n}'

If you are running windows use this

perl -e "use DB_File; print qq{Built with Berkeley DB ver $DB_File::db_ver\n}" perl -e "use DB_File; print qq{Running Berkeley DB ver $DB_File::db_version\n}"
A copy the file config.in from the DB_File main source directory.
A listing of directories where Berkeley DB is installed. For example, if Berkeley DB is installed in /usr/BerkeleDB/lib and /usr/BerkeleyDB/include, I need the output from running this
```
ls -l /usr/BerkeleyDB/lib
ls -l /usr/BerkeleyDB/include
```
If you are having problems building DB_File, send me a complete log of what happened. Start by unpacking the DB_File module into a fresh directory and keep a log of all the steps
```
[edit config.in, if necessary]
perl Makefile.PL
make
make test TEST_VERBOSE=1
```
Now the difficult one. If you think you have found a bug in DB_File and you want me to fix it, you will greatly enhance the chances of me being able to track it down by sending me a small self-contained Perl script that illustrates the problem you are encountering. Include a summary of what you think the problem is and a log of what happens when you run the script, in case I can't reproduce your problem on my system. If possible, don't have the script dependent on an existing 20Meg database. If the script you send me can create the database itself then that is preferred.

I realise that in some cases this is easier said than done, so if you can only reproduce the problem in your existing script, then you can post me that if you want. Just don't expect me to find your problem in a hurry, or at all. :-)