WISDOM-DX: An Automatic DX Evaluation System Using a QA System Based on Web Information

The promotion of digital transformation (DX) is an urgent issue for Japanese society. To promote companies’ DX initiatives, various surveys on DX have been manually conducted by private research companies, industry associations, local governments, and government agencies. However, 95% of companies are either not working on DX at all or are only in the beginning stage of working on it. They have difficulty understanding the purpose and methods of DX that are appropriate for them. Although the surveys introduce the general DX trends and the DX initiatives of top-ranked companies, it is difficult for most of the companies to recognize their own positions and to find referable good practices from the surveys. Although it is necessary and effective for the companies to make objective evaluations for benchmarking such as scoring their DX initiatives and rankings among other companies, it is not easy for them to conduct the benchmark surveys themselves, which require designing the evaluation items, conducting the evaluation, and benchmarking for DX promotion, because the survey cost in time and expense is not small. Instead of this kind of manual survey, Web information could be helpful in conjunction with a sophisticated search technique because companies that are active in DX disseminate a lot of information on the Web through public relations, investor relations, and other promotional activities. However, it has not been clarified what kind of queries are effective for benchmarking DX initiatives. There is no reported method for obtaining the appropriate Web information of companies’ DX and evaluating the companies using the information. To make it possible for companies to make objective evaluations, this paper proposes WISDOM-DX, a system that leverages a question answering (QA) system based on Web information that automatically evaluates companies' DX initiatives. By modeling evaluation items in the form of 5W1H (when, who, where, what, why, how) questions, WISDOM-DX evaluates DX initiatives by scoring an answer set generated by the QA system. WISDOM-DX thus makes it possible to obtain consistent benchmark results in a timely, efficient manner. To examine the feasibility of using Web data, WISDOM-DX and a baseline method that used Google Custom Search were evaluated by ranking 464 companies that responded to the DX Stocks 2021 survey from which DX experts selected 48 companies for distinction as DX Stocks 2021 or Noteworthy DX Companies 2021. Regarding the top 48 companies ranked by WISDOM-DX, 27 of them were included among the 48 selected companies and 17 of them had received DX-related awards or certifications, indicating that 91.7% had a certain level of achievement for their DX initiatives. In contrast, 11 of the top 48 companies ranked by the baseline method were included among the 48 selected companies and 20 of them had received DX-related awards or certifications, indicating that 64.6% had a certain level of achievement for their DX initiatives. When WISDOM-DX and the baseline method were evaluated for searching for the 48 selected companies, the area under the precision-recall curve (AUPR) values obtained by WISDOM-DX and the baseline method were 0.541 and 0.181, respectively. In addition, the respective precision values were 56.3% and 22.9%. The survey of WISDOM-DX with the questionnaire to the evaluated companies showed that 60.7% offered positive responses and 32.1% neutral responses regarding the agreeability of their rankings, and that 46.4% offered positive responses and 39.3% neutral responses regarding the usefulness of the system. These results show that WISDOM-DX had more promising performance than the baseline method, and that it offers the prospect of automating large-scale analysis and evaluation of DX initiatives as a first step in using Web data for benchmarking companies. We will provide support functions to improve WISDOM-DX for practical use by companies and research organizations.

The promotion of digital transformation (DX) is an urgent issue for Japanese society. To promote companies’ DX initiatives, various surveys on DX have been manually conducted by private research companies, industry associations, local governments, and government agencies. However, 95% of companies are either not working on DX at all or are only in the beginning stage of working on it. They have difficulty understanding the purpose and methods of DX that are appropriate for them. Although the surveys introduce the general DX trends and the DX initiatives of top-ranked companies, it is difficult for most of the companies to recognize their own positions and to find referable good practices from the surveys. Although it is necessary and effective for the companies to make objective evaluations for benchmarking such as scoring their DX initiatives and rankings among other companies, it is not easy for them to conduct the benchmark surveys themselves, which require designing the evaluation items, conducting the evaluation, and benchmarking for DX promotion, because the survey cost in time and expense is not small. Instead of this kind of manual survey, Web information could be helpful in conjunction with a sophisticated search technique because companies that are active in DX disseminate a lot of information on the Web through public relations, investor relations, and other promotional activities. However, it has not been clarified what kind of queries are effective for benchmarking DX initiatives. There is no reported method for obtaining the appropriate Web information of companies’ DX and evaluating the companies using the information. To make it possible for companies to make objective evaluations, this paper proposes WISDOM-DX, a system that leverages a question answering (QA) system based on Web information that automatically evaluates companies' DX initiatives. By modeling evaluation items in the form of 5W1H (when, who, where, what, why, how) questions, WISDOM-DX evaluates DX initiatives by scoring an answer set generated by the QA system. WISDOM-DX thus makes it possible to obtain consistent benchmark results in a timely, efficient manner. To examine the feasibility of using Web data, WISDOM-DX and a baseline method that used Google Custom Search were evaluated by ranking 464 companies that responded to the DX Stocks 2021 survey from which DX experts selected 48 companies for distinction as DX Stocks 2021 or Noteworthy DX Companies 2021. Regarding the top 48 companies ranked by WISDOM-DX, 27 of them were included among the 48 selected companies and 17 of them had received DX-related awards or certifications, indicating that 91.7% had a certain level of achievement for their DX initiatives. In contrast, 11 of the top 48 companies ranked by the baseline method were included among the 48 selected companies and 20 of them had received DX-related awards or certifications, indicating that 64.6% had a certain level of achievement for their DX initiatives. When WISDOM-DX and the baseline method were evaluated for searching for the 48 selected companies, the area under the precision-recall curve (AUPR) values obtained by WISDOM-DX and the baseline method were 0.541 and 0.181, respectively. In addition, the respective precision values were 56.3% and 22.9%. The survey of WISDOM-DX with the questionnaire to the evaluated companies showed that 60.7% offered positive responses and 32.1% neutral responses regarding the agreeability of their rankings, and that 46.4% offered positive responses and 39.3% neutral responses regarding the usefulness of the system. These results show that WISDOM-DX had more promising performance than the baseline method, and that it offers the prospect of automating large-scale analysis and evaluation of DX initiatives as a first step in using Web data for benchmarking companies. We will provide support functions to improve WISDOM-DX for practical use by companies and research organizations.