Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When locating the database, we need to associate the datatype #240

Open
FishermanZzhang opened this issue Mar 11, 2021 · 1 comment
Open

Comments

@FishermanZzhang
Copy link

Versions

eg: 0.7.1

Describe the bug

describe as the title.
For example:
mysql have database test and clickhouse also have database, which also named test.

extract metadata mysql fistly and then clickhouse

but clickhouse failed. the ck tables which in database test, also import to mysql.

To Reproduce

Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior

add databaseType.databasename to locate the database just like the DB which locate the table by databasename.tablename

Actual behavior

If applicable, add screenshots to help explain your problem.
But do not paste log screenshots here.

Full Output Logs

Please paste your log here.

Additional context

Add any other context about the problem here.

@FishermanZzhang
Copy link
Author

FishermanZzhang commented Mar 11, 2021

关于extract metadata 部分,临时修改了一下代码, 导入成功。

(1)
如果已经存在DBS, 直接修改DBS中唯一键即可。
我使用的mysql

ALTER TABLE DBS drop  UNIQUE_DATABASE;
ALTER TABLE DBS add UNIQUE  UNIQUE_DATABASE(NAME, DB_TYPE);

如果是在init阶段则需改
https://github.com/Qihoo360/Quicksql/blob/master/metastore/scripts/qsql-metadata-mysql-init.sql#L8

`UNIQUE KEY `UNIQUE_DATABASE` (`NAME`, `DB_TYPE`)`

(2) https://github.com/Qihoo360/Quicksql/blob/master/meta/src/main/java/com/qihoo/qsql/metadata/collect/MetadataCollector.java#L102

DatabaseValue origin = client.getBasicDatabaseInfo(dbValue.getName(), dbValue.getDbType());

(3) https://github.com/Qihoo360/Quicksql/blob/master/meta/src/main/java/com/qihoo/qsql/metadata/MetadataClient.java#L103

    /**
     * select by dbName.
     *
     * @param databaseName database name
     * @return database value
     */
    public DatabaseValue getBasicDatabaseInfo(String databaseName) {
        return getBasicDatabaseInfo(databaseName, "");
    }


    /**
     * select by dbName.
     *
     * @param databaseName database name
     * @param databaseType database type
     * @return database value
     */
    public DatabaseValue getBasicDatabaseInfo(String databaseName, String databaseType) {
        DatabaseValue databaseValue = null;
        String sql = String.format("select DB_ID, `DESC`, NAME, DB_TYPE from DBS where name ='%s'", databaseName);
        if (!databaseType.isEmpty()){
            sql += String.format("and DB_TYPE='%s'", databaseType);
        }
        LOGGER.debug("getBasicDatabaseInfo sql is {}", sql);
        try (PreparedStatement preparedStatement = connection.prepareStatement(sql)) {
            try (ResultSet resultSet = preparedStatement.executeQuery()) {
                if (resultSet != null && resultSet.next()) {
                    databaseValue = new DatabaseValue();
                    databaseValue.setDbId(resultSet.getLong("DB_ID"));
                    databaseValue.setName(resultSet.getString("NAME"));
                    databaseValue.setDbType(resultSet.getString("DB_TYPE"));
                    databaseValue.setDesc(resultSet.getString("DESC"));
                }
            }
            return databaseValue;
        } catch (SQLException ex) {
            throw new RuntimeException(ex);
        }
    }

但查询依然不会成功,原因还是没有加入DB_TYPE
看了一下代码,需要修改fetcher, 如下。

    private static class MetadataFetcher {

        private String dbType;
        private String dbName;
        private String tableName;

未完待续。。。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant