1. 程式人生 > >FTS資料庫優化(Android)原理與應用詳解(1)

FTS資料庫優化(Android)原理與應用詳解(1)

在Android的官方開發文件上,有建議在使用文字類的資料庫全文搜尋(full-text search)時,使用FTS優化查詢速度。有關FTS的介紹文章不多,本文調研整理一下有關知識,供在Android上使用FTS之前參考。
1.什麼是FTS?
FTS,即full text searches的縮寫。是SQLite提供的一個針對文字類模糊查詢的優化工具。不出所料,其優化方式也是在索引上做文章,這部分在4中介紹,暫時不展開。FTS並非標準SQL語言支援的功能。Android的資料庫底層基於SQLite,所以也支援FTS。
2.如何在Android上使用FTS?——Android官方demo解析

SQLite提供了一種內嵌於SQL語句中的使用FTS的方法,簡單地說,需要做兩件事:建立FTS的virtual table、在原始資料庫發生增刪改的時候trigger FTS virtual table同步。這樣,對應的查詢就可以在FTS virtual table上進行了。至於建立以及使用索引的事情,是SQLite在背後偷偷做的,使用者無需關心。Android官方給出了一個doc和一個project來演示如何使用FTS,本文先從這裡入手,分析一下,然後再做補充。
Android原始碼中也提供了這個demo:development/samples/SearchableDictionary
doc內容有限,僅僅是demo project的講解。
這個demo同時也是Android搜尋框架的demo,搜尋框架相關內容可以參考另外兩篇文章:
如何將自己的App作為外部資料來源提供給Android系統搜尋?

Android框架/系統服務是怎樣管理第三方Search資料來源的?

這是一個Eclipse project,如果使用Android Studio,可以使用匯入功能:File - New - Import Project,選擇工程根目錄即可,Android Studo會自動建立一個gradle工程並且將原始Eclipse工程匯入。

這個demo的資料庫是一個字典資料,有單詞和解釋兩個欄位,在程式碼中是raw res檔案res/raw/definitions.txt,資料樣例:
abbey - n. a monastery ruled by an abbot
abide - v. dwell; inhabit or live in
abound - v. be abundant or plentiful; exist in large quantities
absence - n. the state of being absent
absorb - v. assimilate or take in
abstinence - n. practice of refraining from indulging an appetite especially alcohol
absurd - j. inconsistent with reason or logic or common sense

資料庫工具類DictionaryDatabase.java中實現了SQLiteOpenHelper:
    //The columns we'll include in the dictionary table
    public static final String KEY_WORD = SearchManager.SUGGEST_COLUMN_TEXT_1;
    public static final String KEY_DEFINITION = SearchManager.SUGGEST_COLUMN_TEXT_2;

    private static final String DATABASE_NAME = "dictionary";
    private static final String FTS_VIRTUAL_TABLE = "FTSdictionary";
    private static final int DATABASE_VERSION = 2;

    /**
     * This creates/opens the database.
     */
    private static class DictionaryOpenHelper extends SQLiteOpenHelper {

        ......

        /* Note that FTS3 does not support column constraints and thus, you cannot
         * declare a primary key. However, "rowid" is automatically used as a unique
         * identifier, so when making requests, we will use "_id" as an alias for "rowid"
         */
        private static final String FTS_TABLE_CREATE =
                    "CREATE VIRTUAL TABLE " + FTS_VIRTUAL_TABLE +
                    " USING fts3 (" +
                    KEY_WORD + ", " +
                    KEY_DEFINITION + ");";

        ......

        @Override
        public void onCreate(SQLiteDatabase db) {
            mDatabase = db;
            mDatabase.execSQL(FTS_TABLE_CREATE);
            loadDictionary();
        }

        /**
         * Starts a thread to load the database table with words
         */
        private void loadDictionary() {
            new Thread(new Runnable() {
                public void run() {
                    try {
                        loadWords();
                    } catch (IOException e) {
                        throw new RuntimeException(e);
                    }
                }
            }).start();
        }

        private void loadWords() throws IOException {
            Log.d(TAG, "Loading words...");
            final Resources resources = mHelperContext.getResources();
            InputStream inputStream = resources.openRawResource(R.raw.definitions);
            BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));

            try {
                String line;
                while ((line = reader.readLine()) != null) {
                    String[] strings = TextUtils.split(line, "-");
                    if (strings.length < 2) continue;
                    long id = addWord(strings[0].trim(), strings[1].trim());
                    if (id < 0) {
                        Log.e(TAG, "unable to add word: " + strings[0].trim());
                    }
                }
            } finally {
                reader.close();
            }
            Log.d(TAG, "DONE loading words.");
        }

        /**
         * Add a word to the dictionary.
         * @return rowId or -1 if failed
         */
        public long addWord(String word, String definition) {
            ContentValues initialValues = new ContentValues();
            initialValues.put(KEY_WORD, word);
            initialValues.put(KEY_DEFINITION, definition);

            return mDatabase.insert(FTS_VIRTUAL_TABLE, null, initialValues);
        }

        @Override
        public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
            Log.w(TAG, "Upgrading database from version " + oldVersion + " to "
                    + newVersion + ", which will destroy all old data");
            db.execSQL("DROP TABLE IF EXISTS " + FTS_VIRTUAL_TABLE);
            onCreate(db);
        }
    }

可以看到,onCreate()(處理建立資料庫)邏輯中,建立FTS virtual table,並且解析字典資料,並且插入到FTS virtual table中。關鍵是建立FTS virtual table的部分,其SQL為
CREATE VIRTUAL TABLE FTSdictionary USING fts3 (suggest_text_1, suggest_text_2);
這實際上是SQLite為FTS提供的一個語法糖,使得建立FTS virtual table可以和使用標準SQL建立一般的table一樣簡單,無需破壞程式設計風格和可讀性。
建立完了virtual table,看看增刪改,從loadDictionary()方法看到其插入操作與一般的table無異。
再看看查詢,在DictionaryDatabase.java中
    /**
     * Returns a Cursor over all words that match the given query
     *
     * @param query The string to search for
     * @param columns The columns to include, if null then all are included
     * @return Cursor over all words that match, or null if none found.
     */
    public Cursor getWordMatches(String query, String[] columns) {
        String selection = KEY_WORD + " MATCH ?";
        String[] selectionArgs = new String[] {query+"*"};

        return query(selection, selectionArgs, columns);

        /* This builds a query that looks like:
         *     SELECT <columns> FROM <table> WHERE <KEY_WORD> MATCH 'query*'
         * which is an FTS3 search for the query text (plus a wildcard) inside the word column.
         *
         * - "rowid" is the unique id for all rows but we need this value for the "_id" column in
         *    order for the Adapters to work, so the columns need to make "_id" an alias for "rowid"
         * - "rowid" also needs to be used by the SUGGEST_COLUMN_INTENT_DATA alias in order
         *   for suggestions to carry the proper intent data.
         *   These aliases are defined in the DictionaryProvider when queries are made.
         * - This can be revised to also search the definition text with FTS3 by changing
         *   the selection clause to use FTS_VIRTUAL_TABLE instead of KEY_WORD (to search across
         *   the entire table, but sorting the relevance could be difficult.
         */
    }

    /**
     * Performs a database query.
     * @param selection The selection clause
     * @param selectionArgs Selection arguments for "?" components in the selection
     * @param columns The columns to return
     * @return A Cursor over all rows matching the query
     */
    private Cursor query(String selection, String[] selectionArgs, String[] columns) {
        /* The SQLiteBuilder provides a map for all possible columns requested to
         * actual columns in the database, creating a simple column alias mechanism
         * by which the ContentProvider does not need to know the real column names
         */
        SQLiteQueryBuilder builder = new SQLiteQueryBuilder();
        builder.setTables(FTS_VIRTUAL_TABLE);
        builder.setProjectionMap(mColumnMap);

        Cursor cursor = builder.query(mDatabaseOpenHelper.getReadableDatabase(),
                columns, selection, selectionArgs, null, null, null);

        if (cursor == null) {
            return null;
        } else if (!cursor.moveToFirst()) {
            cursor.close();
            return null;
        }
        return cursor;
    }
看到使用了一個關鍵字“MATCH”,其SQL語句如下,不同於標準SQL中的LIKE
SELECT <columns> FROM <table> WHERE <KEY_WORD> MATCH 'query*'

在這個demo中,資料庫比較簡單,只有FTS virtual table本身。看一下資料庫中的表的情況,經過上述操作,一共有四個表:
FTSdictionary
FTSdictionary_content
FTSdictionary_segdir
FTSdictionary_segments
可見,雖然在語法上有“virtual table”,但實際上仍然是在資料庫中建立了四個表。
如果本身的資料庫已經很複雜了,那麼需要在對應的資料庫表增刪改的時候,同步trigger FTS virtual table。