sqlite3的程序安全訪問測試
前言
想寫個demo, 由C程式寫sqlite3資料庫中的表A, 由java程式去讀sqlite3資料庫中的表A.
這樣,就涉及到sqlite3的執行緒安全和程序安全訪問。
寫好測試程式後,跑起來,很失望。
sqlite3同一個表無法由2個程序同時讀寫表記錄。
總有一個程序會出現“database is locked”的報錯,去查了資料。大概意思是:因為是檔案型資料庫,無法使2個程序同時向同一個檔案寫資料。我的應用是一個讀(讀完一條記錄刪一條),一個寫,應該會觸發sqlite的檔案讀寫鎖吧。
嘗試sql執行失敗後重試,倒是可以繼續跑。但是每一個SQL都有可能失敗,都要重試。這執行效率就下來了。
換其他資料庫了,失望。
其實,sqlite作者是可以解決這個問題的。只需要封裝sqlite3_xx的介面實現不去直接操作檔案,而是由同一個資料庫代理程式(第一次開啟資料庫時建立資料庫代理程式的例項)訪問資料庫檔案,這樣,就可以達到多個程序共享一個數據庫檔案,資料庫的開啟和關閉,帶上引用計數就好。改動的程式碼也不大。誰知道sqlite作者咋想的呢,哪個資料庫的使用者沒有多程序訪問同一個資料庫的需求呢?
如果使用者非要使用sqlite3, 來支援多程序安全訪問。只能自己再封裝一組介面,由自己的程式來控制資料庫檔案的訪問(同一個資料庫檔案只有一個訪問者,對使用者隱藏了資料庫檔案的操作). sqlite3自己可以保證執行緒安全性。這樣也能程序安全,其實這活應該由sqlite3資料庫來完成。如果使用者要想那麼多,誰還來用這麼難用的資料庫。畢竟大部分資料庫都是可以程序安全的。
說來說去,其實sqlite3需要加一個服務程式,那些sqlite3_xx介面,都是通過本地socket來向服務程式讀寫資料,這樣就正常了。這樣改動也不大。
也許sqlite的需求場景,就是給一個程式用吧。
demo下載點
src_test_sqlite_process_safe.7z
demo預覽
# @file README
# 下載 sqlite-autoconf-3250200.tar.gz
# tar -xzvf sqlite-autoconf-3250200.tar.gz
# cd sqlite-autoconf-3250200
# ./configure ./configure --prefix /home/sqlite3_was_install
# make
# make install
# 將 /home/sqlite3_was_install 拷貝到工程的doc目錄
// @file main.cpp // @brief 測試sqlite3的程序安全性訪問 // @note // 實驗環境: // debian8.8 // 實驗結論: // sqlite3做的渣, 不支援多程序訪問同一個資料庫的同一個表中的記錄 // 當多程序同時高頻度讀寫同一個表時, 任何一句sql都可能執行失敗, 失望 // 當一句正常的sql執行失敗時,總不能去無限的重試吧,那還叫資料庫麼? // #include <stdlib.h> #include <stdio.h> #include <stdint.h> #include <string.h> #include <unistd.h> #include <signal.h> #include <string> #include <errno.h> #include "const_define.h" #include "sqlite3.h" void init(const char* psz_log_owner_name); void uninit(); void proc_sig_term(int num); int fn_test(int argc, char** argv); // @ref https://www.tutorialspoint.com/sqlite/sqlite_c_cpp.htm int fn_sqlite_db_init(); int fn_test_sqlite_read(); int fn_test_sqlite_write(); typedef int (*PFN_cb_sqlite_proc)(void* cb_info, int argc, char **argv, char **azColName); int cb_sqlite_proc_select(void* cb_info, int argc, char **argv, char **azColName); int cb_sqlite_proc_insert(void* cb_info, int argc, char **argv, char **azColName); bool fn_open_db(sqlite3*& db); void fn_close_db(sqlite3*& db); void fn_free_sqlite_msg(char*& psz_msg); bool fn_open_or_create_tbl(sqlite3* db, const char* psz_tbl_name, const char* psz_field_declare); // sqlite sql example // CREATE TABLE TBL_NAME(ID INT PRIMARY KEY NOT NULL,NAME TEXT NOT NULL,AGE INT NOT NULL,ADDRESS CHAR(50),SALARY REAL); // INSERT INTO TBL_NAME(ID,NAME,AGE,ADDRESS,SALARY) VALUES (1, 'Paul', 32, 'California', 20000.00 ); // DELETE from TBL_NAME where ID=2; // UPDATE TBL_NAME set SALARY = 25000.00 where ID=1;; // SELECT * from TBL_NAME; bool fn_sql_exec( sqlite3* db, const char* psz_sql, std::string& str_err_msg, TAG_SQLITE3_CALLBACK_INFO* p_cb_info, PFN_cb_sqlite_proc pfn_cb); bool is_file_exist(const char* psz_file_path_name, int& i_err_code, std::string& str_err_msg); int main(int argc, char** argv) { char sz_buf[MAX_MSG_LENGTH] = {'\0'}; #ifdef MAKE_FILE_MACRO__BIN_NAME sprintf(sz_buf, "%s", MAKE_FILE_MACRO__BIN_NAME); init(sz_buf); MYLOG_D("MAKE_FILE_MACRO__BIN_NAME = [%s]\n", MAKE_FILE_MACRO__BIN_NAME); #else init(NULL); #endif // #ifdef MAKE_FILE_MACRO__BIN_NAME fn_test(argc, argv); uninit(); MYLOG_D("THE END\n"); return EXIT_SUCCESS; } void uninit() { } void proc_sig_term(int num) { MYLOG_D("SIGTERM = %d, num = %d\n", SIGTERM, num); MYLOG_D("maybe can do some clean task before quit\n"); exit(1); } void init(const char* psz_log_owner_name) { int i = 0; // daemon(0, 0); // clear screen (print 25 empty line) for (i = 0; i < 25; i++) { MYLOG_D("\n"); } signal(SIGTERM, proc_sig_term); } int fn_test(int argc, char** argv) { bool b_rc = false; MYLOG_D(">> fn_test()\n"); do { if (2 != argc) { break; } fn_sqlite_db_init(); b_rc = true; switch (argv[1][0]) { case 'r': fn_test_sqlite_read(); break; case 'w': fn_test_sqlite_write(); break; default: b_rc = false; break; } if (!b_rc) { break; } b_rc = true; } while (0); if (!b_rc) { MYLOG_D("usage : this_program r(ead)/w(rite)\n"); MYLOG_D("e.g. this_program r\n"); MYLOG_D("e.g. this_program w\n"); } return 0; } int cb_sqlite_proc_insert(void* cb_info, int argc, char **argv, char **azColName) { int i = 0; TAG_SQLITE3_CALLBACK_INFO* p_cb_info = (TAG_SQLITE3_CALLBACK_INFO*)cb_info; TAG_COL_NAME_AND_VALUE name_value; if (NULL != p_cb_info) { p_cb_info->clear(); for(i = 0; i < argc; i++) { name_value.set(i, azColName[i], argv[i]); p_cb_info->vec_result.push_back(name_value); } } return 0; // return 0, to continue sql exec } int cb_sqlite_proc_select(void* cb_info, int argc, char **argv, char **azColName) { int i = 0; TAG_SQLITE3_CALLBACK_INFO* p_cb_info = (TAG_SQLITE3_CALLBACK_INFO*)cb_info; TAG_COL_NAME_AND_VALUE name_value; if (NULL != p_cb_info) { p_cb_info->clear(); for(i = 0; i < argc; i++) { name_value.set(i, azColName[i], argv[i]); p_cb_info->vec_result.push_back(name_value); } } // i want to read only one row set return 1; // return not 0, to stop sql exec } bool fn_open_db(sqlite3*& db) { int i_rc = 0; // Open database // sqlite3_open and sqlite3_open_v2 : db name is utf-8 i_rc = sqlite3_open_v2( SQLITE_OBJ_DATABASE_PATH_NAME, // Database filename (UTF-8) &db, // OUT: SQLite db handle SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE | SQLITE_OPEN_SHAREDCACHE, // Flags NULL // Name of VFS module to use ); // @note // 一個程式讀,一個程式寫時,出現瞭如下錯誤 // fn_sql_exec error = [database is locked] // 讀的程式和寫的程式如果在讀失敗和寫失敗後,繼續嘗試執行各自讀和寫的sql, 會有成功的一天 // 但是這樣就有問題了, 我寫失敗,難道我還要重試? 誰知道要嘗試多少次才會成功? // 每執行一次SQL都有可能失敗,這誰受的了:( // 由於多程序的讀寫,即使加上SQL執行失敗的重試, 也會使sqlite資料庫的訪問變成序列的了. 渣... return (SQLITE_OK == i_rc); } void fn_close_db(sqlite3*& db) { if (NULL != db) { sqlite3_close(db); db = NULL; } } bool fn_open_or_create_tbl(sqlite3* db, const char* psz_tbl_name, const char* psz_field_declare) { int i_rc = SQLITE_ERROR; char sz_buf[MAX_SQL_LENGTH] = {'\0'}; std::string str_err_msg = ""; TAG_SQLITE3_CALLBACK_INFO cb_info; TAG_COL_NAME_AND_VALUE name_value; do { if ((NULL == db) || (NULL == psz_tbl_name) || (NULL == psz_field_declare)) { break; } // check tbl is exist sprintf(sz_buf, "select count(type) from sqlite_master where type='table' and name='%s'", psz_tbl_name); if (!fn_sql_exec(db, sz_buf, str_err_msg, &cb_info, cb_sqlite_proc_insert)) { MYLOG_D("1 fn_sql_exec error = [%s]\n", str_err_msg.c_str()); break; } if (cb_info.vec_result.size() <= 0) { break; } name_value = cb_info.pop_first(); if (name_value.str_value == "1") { // tbl is exist i_rc = SQLITE_OK; break; } // table not exist, create it sprintf(sz_buf, "CREATE TABLE %s(%s);", psz_tbl_name, psz_field_declare); if (!fn_sql_exec(db, sz_buf, str_err_msg, &cb_info, cb_sqlite_proc_insert)) { MYLOG_D("2 fn_sql_exec error = [%s]\n", str_err_msg.c_str()); break; } i_rc = SQLITE_OK; MYLOG_D("table created successfully\n"); } while (0); return (SQLITE_OK == i_rc); } void fn_free_sqlite_msg(char*& psz_msg) { if (NULL != psz_msg) { sqlite3_free(psz_msg); psz_msg = NULL; } } int fn_test_sqlite_read() { sqlite3* db = NULL; char sz_buf[MAX_SQL_LENGTH] = {'\0'}; int i_err_code = 0; std::string str_err_msg; TAG_SQLITE3_CALLBACK_INFO cb_info; TAG_COL_NAME_AND_VALUE name_value; do { if (!fn_open_db(db)) { MYLOG_D("can't open database: %s\n", sqlite3_errmsg(db)); break; } MYLOG_D("opened database successfully\n"); sprintf(sz_buf, "%s", "ID INT PRIMARY KEY NOT NULL," \ "NAME TEXT NOT NULL," \ "AGE INT NOT NULL," \ "ADDRESS CHAR(50)," \ "SALARY REAL"); if (!fn_open_or_create_tbl(db, SQLITE_OBJ_TABLE_NAME, sz_buf)) { break; } do { if (is_file_exist(FLAG_FILE_TO_STOP_READ, i_err_code, str_err_msg)) { MYLOG_D("find stop flag file[%s], stop read now\n", FLAG_FILE_TO_STOP_READ); break; } sprintf(sz_buf, "SELECT * from %s;", SQLITE_OBJ_TABLE_NAME); if (!fn_sql_exec(db, sz_buf, str_err_msg, &cb_info, cb_sqlite_proc_select)) { MYLOG_D("3 fn_sql_exec error : %s\n", str_err_msg.c_str()); } else { if (cb_info.vec_result.size() > 0) { cb_info.show_result(); // process row set was read back // ... // delete it name_value = cb_info.pop_first(); if (name_value.str_name == "ID") { sprintf(sz_buf, "DELETE from %s where ID=%s;", SQLITE_OBJ_TABLE_NAME, name_value.str_value.c_str()); if (!fn_sql_exec(db, sz_buf, str_err_msg, &cb_info, cb_sqlite_proc_select)) { MYLOG_D("delete failed : %s\n", str_err_msg.c_str()); } else { MYLOG_D("delete one row ok\n"); } } else { MYLOG_D("can't find first col name is \"ID\"\n"); } }else { MYLOG_D("can't read any row set\n"); } } } while (1); } while (0); fn_close_db(db); return 0; } int fn_test_sqlite_write() { sqlite3* db = NULL; char sz_buf[MAX_SQL_LENGTH] = {'\0'}; int i_err_code = 0; std::string str_err_msg; int i_id = 0; TAG_SQLITE3_CALLBACK_INFO cb_info; do { if (!fn_open_db(db)) { MYLOG_D("can't open database: %s\n", sqlite3_errmsg(db)); break; } MYLOG_D("opened database successfully\n"); sprintf(sz_buf, "%s", "ID INT PRIMARY KEY NOT NULL," \ "NAME TEXT NOT NULL," \ "AGE INT NOT NULL," \ "ADDRESS CHAR(50)," \ "SALARY REAL"); if (!fn_open_or_create_tbl(db, SQLITE_OBJ_TABLE_NAME, sz_buf)) { break; } do { if (is_file_exist(FLAG_FILE_TO_STOP_WRITE, i_err_code, str_err_msg)) { MYLOG_D("find stop flag file[%s], stop write now\n", FLAG_FILE_TO_STOP_WRITE); break; } sprintf(sz_buf, "INSERT INTO %s (ID,NAME,AGE,ADDRESS,SALARY) VALUES (%d, 'Paul', 32, 'California', 20000.00 );", SQLITE_OBJ_TABLE_NAME, i_id++); if (!fn_sql_exec(db, sz_buf, str_err_msg, &cb_info, cb_sqlite_proc_insert)) { MYLOG_D("4 fn_sql_exec error : %s\n", str_err_msg.c_str()); } MYLOG_D("insert ok, id = %d\n", i_id - 1); cb_info.show_result(); } while (1); } while (0); fn_close_db(db); return 0; } bool fn_sql_exec( sqlite3* db, const char* psz_sql, std::string& str_err_msg, TAG_SQLITE3_CALLBACK_INFO* p_cb_info, PFN_cb_sqlite_proc pfn_cb) { int i_rc = SQLITE_ERROR; char sz_buf[MAX_SQL_LENGTH] = {'\0'}; char* psz_err_msg = NULL; do { str_err_msg.clear(); if ((NULL == db) || (NULL == psz_sql)) { break; } strcpy(sz_buf, psz_sql); i_rc = sqlite3_exec(db, sz_buf, pfn_cb, p_cb_info, &psz_err_msg); if ((SQLITE_OK != i_rc) && (SQLITE_ABORT != i_rc)) { str_err_msg = psz_err_msg; fn_free_sqlite_msg(psz_err_msg); break; } } while (0); return ((SQLITE_OK == i_rc) || (SQLITE_ABORT == i_rc)); } int fn_sqlite_db_init() { int i_rc = 0; MYLOG_D("sqlite3_libversion() = [%s]\n", sqlite3_libversion()); MYLOG_D("sqlite3_libversion_number() = [%d]\n", sqlite3_libversion_number()); MYLOG_D("sqlite3_sourceid() = [%s]\n", sqlite3_sourceid()); sqlite3_shutdown(); sqlite3_config(SQLITE_CONFIG_MULTITHREAD); i_rc = sqlite3_threadsafe(); // SQLITE_THREADSAFE = 1 or 2, but not 0(0 is not thread safe) // 0 is 單執行緒模式 // 1 is 序列模式 // 2 is 多執行緒模式 MYLOG_D("sqlite3_threadsafe() = %d\n", i_rc); if (i_rc > 0) { i_rc = sqlite3_config(SQLITE_CONFIG_SERIALIZED); MYLOG_D("sqlite3_config() = %d\n", i_rc); if (SQLITE_OK == i_rc) { MYLOG_D("can now use sqlite on multiple threads, using the same connection\n"); } else { MYLOG_D("!! warning : setting sqlite thread safe mode to serialized failed\n"); } } else { MYLOG_D("!! warning : current sqlite database is not compiled to be threadsafe\n"); } sqlite3_initialize(); // 當多程序一起訪問同一個資料庫檔案時 // 都在此點開始執行, 也會有如下錯誤 // database is locked MYLOG_D("press enter key to continue\n"); getchar(); return 0; } bool is_file_exist(const char* psz_file_path_name, int& i_err_code, std::string& str_err_msg) { bool b_rc = false; int i_rc = -1; do { if ((NULL == psz_file_path_name) || (0 == strlen(psz_file_path_name))) { i_err_code = -1; str_err_msg = "param 1 error"; break; } i_rc = access(psz_file_path_name, F_OK); if (0 != i_rc) { i_err_code = errno; str_err_msg = strerror(i_err_code); } else { b_rc = true; } } while (0); return b_rc; }
// @file const_define.h
#if not defined(__CONST_DEFINE_H__)
#define __CONST_DEFINE_H__
#include <string.h>
#include <string>
#include <list>
#ifndef SAFE_DELETE
#define SAFE_DELETE(p) \
if (NULL != (p)) { \
delete (p); \
(p) = NULL; \
}
#endif // #ifndef SAFE_DELETE
#ifndef SAFE_DELETE_ARY
#define SAFE_DELETE_ARY(p) \
if (NULL != (p)) { \
delete[] (p); \
(p) = NULL; \
}
#endif // #ifndef SAFE_DELETE
#define TITLE_LINE80 "================================================================================"
#define LINE80 "--------------------------------------------------------------------------------"
#if not defined(MYLOG_D)
#define MYLOG_D printf
#endif // #if not defined(MYLOG_D)
#define SQLITE_OBJ_DATABASE_PATH_NAME "/var/log/my_sqlite_test_2.db"
#define SQLITE_OBJ_TABLE_NAME "tbl_test_3"
#define FLAG_FILE_TO_STOP_READ "/var/log/flag_stop_read"
#define FLAG_FILE_TO_STOP_WRITE "/var/log/flag_stop_write"
#define MAX_SQL_LENGTH (1024 * 8)
#define MAX_MSG_LENGTH (1024 * 4)
typedef struct _tag_col_name_and_value {
int i_index;
std::string str_name;
std::string str_value;
_tag_col_name_and_value()
{
clear();
}
void clear()
{
i_index = 0;
{
std::string str;
str_name.swap(str);
}
{
std::string str;
str_value.swap(str);
}
}
void set(int i_index, const char* name, const char* value)
{
clear();
this->i_index = i_index;
str_name = ((NULL != name) ? name : "");
str_value = ((NULL != value) ? value : "");
}
}TAG_COL_NAME_AND_VALUE;
typedef struct _tag_sqlite3_callback_info {
// 出參
std::list<TAG_COL_NAME_AND_VALUE> vec_result;
std::list<TAG_COL_NAME_AND_VALUE>::iterator it;
_tag_sqlite3_callback_info()
{
clear();
}
void clear()
{
vec_result.clear();
}
TAG_COL_NAME_AND_VALUE pop_first()
{
TAG_COL_NAME_AND_VALUE rc;
for (it = vec_result.begin(); it != vec_result.end(); it++) {
rc = *it;
vec_result.pop_front();
break;
}
return rc;
}
void show_result() {
TAG_COL_NAME_AND_VALUE rc;
MYLOG_D("%s\n", TITLE_LINE80);
MYLOG_D("%s\n", "show_result");
MYLOG_D("%s\n", LINE80);
for (it = vec_result.begin(); it != vec_result.end(); it++) {
rc = *it;
MYLOG_D("[%d] name = [%s], value = [%s]\n",
rc.i_index,
rc.str_name.c_str(),
rc.str_value.c_str());
}
}
}TAG_SQLITE3_CALLBACK_INFO;
#endif // #if not defined(__CONST_DEFINE_H__)
# ==============================================================================
# @file makefile
# ==============================================================================
# @note
# howto build project
# make BIN_NAME="bin_name_by_you_want" rebuild
# makefile code need tab key not sapce key
MY_MAKE_FILE_PATH_NAME = $(MAKEFILE_LIST)
# macro from Makefile command line
# BIN_NAME
# macro to C project
MAKE_FILE_MACRO__BIN_NAME="make_file_macro__bin_name"
# var define on Makefile
BIN = output_not_give_bin_name
IS_BUILD_TYPE_VALID = 0
ifdef BIN_NAME
IS_BUILD_TYPE_VALID = 1
BIN = $(BIN_NAME)
MAKE_FILE_MACRO__BIN_NAME=$(BIN_NAME)
else
IS_BUILD_TYPE_VALID = 0
endif
LINE80 = --------------------------------------------------------------------------------
# CC = g++ -std=c++98
CC = g++
# -Werror is "warning as error"
CFLAGS = -Wall -Werror -g
INC = -I. -I../../doc/sqlite3_was_install/include/
LIBPATH = -L/usr/lib/ -L/usr/local/lib/ -L../../doc/sqlite3_was_install/lib/
ifeq (1, $(IS_BUILD_TYPE_VALID))
LIBS = -lstdc++ -pthread -lsqlite3
else
LIBS =
endif
DEPEND_CODE_DIR = ../common/ \
DEPEND_CODE_SRC = $(shell find $(DEPEND_CODE_DIR) -name '*.cpp')
DEPEND_CODE_OBJ = $(DEPEND_CODE_SRC:.cpp=.o)
ROOT_CODE_SRC = $(shell find ./ -name '*.cpp')
ROOT_CODE_OBJ = $(ROOT_CODE_SRC:.cpp=.o)
SUB_CODE_DIR = ./empty_dir
SUB_CODE_SRC = $(shell find $(SUB_CODE_DIR) -name '*.cpp')
SUB_CODE_OBJ = $(SUB_CODE_SRC:.cpp=.o)
.PHONY: help
help:
clear
@echo "usage:"
@echo
@echo "build project by given bin name"
@echo "make BIN_NAME=\"bin_name_by_you_want\" rebuild"
@echo
.PHONY: clean
clean:
clear
@echo
@echo
@echo
@echo
@echo
@echo
@echo
@echo
@echo
@echo
@echo
@echo
@echo
@echo
@echo
@echo
@echo
@echo
@echo
@echo
@echo
@echo
@echo
@echo
@echo
@echo "make clean begin"
@echo $(LINE80)
@echo "@file $(MY_MAKE_FILE_PATH_NAME)"
@echo "IS_BUILD_TYPE_VALID = $(IS_BUILD_TYPE_VALID)"
@echo "BIN = $(BIN)"
@echo $(LINE80)
rm -f $(ROOT_CODE_OBJ) $(DEPEND_CODE_OBJ) $(SUB_CODE_OBJ)
ifeq (1, $(IS_BUILD_TYPE_VALID))
rm -f ./$(BIN)
endif
@echo "make clean over"
.PHONY: all
all:$(BIN)
@echo $(LINE80)
@echo make all
chmod 777 ./$(BIN)
find . -name "$(BIN)"
$(BIN) : $(ROOT_CODE_OBJ) $(DEPEND_CODE_OBJ) $(SUB_CODE_OBJ)
$(CC) $(CFLAGS) -o [email protected] $^ $(SHLIBS) $(INC) $(LIBPATH) $(LIBS)
.cpp.o:
$(CC) -c $(CFLAGS) -DMAKE_FILE_MACRO__BIN_NAME="\"$(MAKE_FILE_MACRO__BIN_NAME)\"" $^ -o [email protected] $(INC) $(LIBPATH) $(LIBS)
.PHONY: rebuild
rebuild:
make -f $(MY_MAKE_FILE_PATH_NAME) clean
ifeq (1, $(IS_BUILD_TYPE_VALID))
@echo $(LINE80)
make -f $(MY_MAKE_FILE_PATH_NAME) all
chmod 775 ./$(BIN)
ldd ./$(BIN)
else
@echo $(LINE80)
@echo "error : Makefile command line input error, please see help"
@echo "please run => make help"
@echo $(LINE80)
endif
#!/bin/bash
# ==============================================================================
# @file build_all_project.sh
# ==============================================================================
make BIN_NAME="test_sqlite_thread_safe" rebuild