Django database backend for SQLite on Amazon EFS
MIT License
django_sqlite_efs is a custom database backend for Django, designed to work with SQLite on AWS Lambda using Amazon EFS (Elastic File System) and Amazon DynamoDB. This backend provides a solution to protect SQLite databases from corruption caused by concurrent writes in network-attached storage environments that lack proper file locking mechanisms for SQLite.
A basic Django polls application:
To install django-sqlite-efs
, simply use pip:
pip install django-sqlite-efs
Create a DynamoDB table to support distributed locking. The table should have the following schema:
pk
(Type: String)expires_at
field for automatic cleanup of expired locks.Attribute Name | Type |
---|---|
pk | String |
lock_id | String |
expires_at | Number |
Configuring expires_at
with a TTL (Time-to-Live) policy is recommended for automatic removal of expired locks.
In your Django project, update the settings.py
file to use the custom database backend provided by django-sqlite-efs
:
DATABASES = {
'default': {
'ENGINE': 'django_sqlite_efs',
'NAME': 'path_to_your_sqlite_db_file',
"OPTIONS": {
# `timeout` - number of seconds to wait for lock acquisition.
# It must be at least several seconds less than the timeout of
# your Lambda function. Default and minimum value is 3.
"timeout": timeout
# Setting `init_commands` is not recommended because it overrides
# default commands, which may lead to unexpected behavior.
}
}
}
Additionally, configure the following settings in settings.py
or as environment variables:
Ensure that your AWS credentials are correctly configured via environment variables or IAM roles. The package uses boto3
to interact with DynamoDB. The Lambda function must have PutItem
and DeleteItem
permissions on the DynamoDB table.
SQLite uses file-based locking to prevent concurrent writes, but this is unreliable on Amazon EFS because EFS employs advisory locks. Advisory locks do not prevent processes from writing to a locked file if they have adequate permissions.
The django_sqlite_efs backend mitigates this by using Amazon DynamoDB to manage database locks:
INSERT
, UPDATE
, DELETE
), the backend attempts to acquire a lock in DynamoDB.SELECT
) do not acquire a lock, allowing for concurrent read access without blocking.Concurrent Writes: This backend does not support concurrent write operations. During any write operation, the database is locked, blocking all reads and writes until the operation completes.
High Latency:
This solution is designed for environments where write operations are infrequent.
This project is licensed under the MIT License. See the LICENSE file for more details.