A Technology Forecasting Model Integrating Spatiotemporal Features under the “Structure-Content” Framework
-
摘要: 科学技术发展是一种动态非线性的复杂演进过程.为了提升技术发展的精准预测, 本文基于大语言模型(Large Language Model, LLM)、图卷积神经网络(Graph Convolutional Networks, GCN)、双向长短期记忆神经网络(Bi-directional Long Short-Term Memory, BiLSTM)以及鲁棒随机配置网络(Robust Stochastic Configuration Networks, RSCN), 提出了一种全新的"结构−内容"时空技术预测模型(Spatiotemporal Technological Forecasting Model with LLM as Representation, STTeFL 模型).首先, 通过结合图卷积神经网络和双向长短期记忆神经网络, 分别捕捉技术网络中的空间依赖关系和时间演化规律, 从而突破了传统预测模型在动态性和结构表征上的局限性, 克服了传统技术预测模型的“伪动态”和“静态”限制; 其次, 引入大语言模型对技术网络中的节点特征和边特征进行双重语义表征, 将预测框架从单一的结构维度扩展至“结构−内容”双维度分析, 显著增强了模型对技术发展信息的理解能力和表征深度. 最后, 通过集成RSCN, 模型能够有效应对极端不均衡数据分布的挑战, 进一步提升了预测的鲁棒性和准确性.本文提出的预测框架在多个指标上均优于当前多种技术预测方法, 为推动技术预测建模和评估未来技术发展轨迹提供了有力的支持.
-
关键词:
- 科学技术发展 /
- 图卷积神经网络 /
- 大语言模型 /
- 双向长短期记忆神经网络 /
- 鲁棒随机配置网络
Abstract: The development of science and technology is a dynamic, nonlinear, and complex evolutionary process. To enhance the precision of technological development predictions, this paper proposes a novel “Structure-Content” spatiotemporal technological forecasting model (Spatiotemporal Technological Forecasting Model with LLM as Representation, STTeFL model) based on Large Language Models (LLM), Graph Convolutional Networks (GCN), Bidirectional Long Short-Term Memory Networks (BiLSTM), and Robust Stochastic Configuration Networks (RSCN). Firstly, by integrating Graph Convolutional Networks and Bidirectional Long Short-Term Memory Networks, the model captures the spatial dependencies and temporal evolution patterns within technological networks, thereby overcoming the limitations of traditional forecasting models in terms of dynamic behavior and structural representation, and addressing the “pseudo-dynamic” and “static” constraints of conventional technological forecasting models. Secondly, the introduction of Large Language Models enables dual semantic representation of node and edge features within the technological network, expanding the predictive framework from a single structural dimension to a dual-dimensional “Structure-Content” analysis. This significantly enhances the model's ability to understand and represent information related to technological development. Finally, by incorporating RSCN, the model effectively addresses the challenges posed by extremely imbalanced data distributions, further improving the robustness and accuracy of predictions. The proposed forecasting framework outperforms various current technological forecasting methods across multiple metrics, providing strong support for advancing technological forecasting modeling and evaluating future technological development trajectories. -
图 3 查准率和查全率及其不均衡处理性能的消融实验比较
Fig. 3 Ablation comparison of precision, recall ratio and non-equilibrium treatment performance
图 4 基于预测结果构建的深度学习技术网络
Fig. 4 Deep learning technology network based on prediction results
图 5 网络社群检测分析后的规模结果
Fig. 5 Scale of each community after network＇ s community detection
表 1 模型性能指标的计算方法
Table 1 Calculation method of model performance measure inde
测度指标 计算公式 Recall $\dfrac{TP}{TP + FN}$ Precision $\dfrac{TP}{TP + FP}$ AUC $\sum\limits_{i=1}^{n-1} \dfrac{(FPR_{i+1} - FPR_i) \times (TPR_{i+1} + TPR_i)}{2}$ AUPR $\sum\limits_{i=1}^{n-1} \dfrac{(R_{i+1} - R_i) \times (P_{i+1} + P_i)}{2}$ F1 Score $2 \times \dfrac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$ 
下载: 导出CSV
表 2 所提模型与基线模型性能比较
Table 2 Performance comparison between the proposed model and the baseline models
模型 特征 AUC Precision Recall F1 AUPR STTeFL 内容LLM表征 0.8837 ±0.0024 0.8361 ±0.0166 0.8066 ±0.0134 0.8206 ±0.0084 0.9054 ±0.0026 LogR 链路相似性 0.6279 ±0.0141 0.7383 ±0.0370 0.2565 ±0.0282 0.3807 ±0.0313 0.5002 ±0.0265 SVM 链路相似性 0.5113 ±0.0137 0.7778 ±0.1070 0.0227 ±0.0321 0.0442 ±0.0371 0.4039 ±0.0332 RF 链路相似性 0.6915 ±0.0115 0.6650 ±0.0236 0.3844 ±0.0231 0.4867 ±0.0649 0.5270 ±0.0368 XGBoost 链路相似性 0.8004 ±0.0149 0.4773 ±0.0170 0.6058 ±0.0301 0.5335 ±0.0301 0.5431 ±0.0135 AdaBoost 链路相似性 0.7085 ±0.0267 0.6324 ±0.1254 0.4188 ±0.0406 0.5039 ±0.0351 0.5278 ±0.0483
下载: 导出CSV
表 3 对比模型的实验结果
Table 3 Experimental results under different comparative models
对比方式 模型 AUC Precision Recall F1 AUPR 大模型 STTeFL_gpt 0.8781 ±0.0042 0.7031 ±0.0199 0.8855 ±0.0098 0.7835 ±0.0089 0.9000 ±0.0062 STTeFL_bert 0.8803 ±0.0052 0.8163 ±0.0126 0.8061 ±0.0242 0.8135 ±0.0085 0.8959 ±0.0083 特征表达 STTeFL_doc 0.7872 ±0.0606 0.6306 ±0.1107 0.8477 ±0.1253 0.7052 ±0.0400 0.8026 ±0.0630 STTeFL_sim 0.8677 ±0.0076 0.6037 ±0.0221 0.9333 ±0.0093 0.7328 ±0.0135 0.8842 ±0.0105 维数选择 STTeFL_32N 0.8700 ±0.0001 0.5553 ±0.0201 0.9595 ±0.0006 0.7017 ±0.0088 0.8870 ±0.0024 STTeFL_128L 0.8708 ±0.0000 0.6289 ±0.0014 0.9255 ±0.0002 0.7480 ±0.0005 0.8882 ±0.0000
下载: 导出CSV
表 4 深度学习领域未来潜在技术举例
Table 4 Examples of potential future technologies in the deep learning area
序号 核心节点 潜在可能技术 涉及领域 1 A43D-001; A47B-091; H01Q-001; H04S-003 天线设计与优化、立体声与多声道音频处理、塑料成型与模具 5G通信、虚拟现实、汽车制造 2 G01J-005;B65D-025;G07C-015;G05G-005 光学测量与传感器、航空设备与飞行控制、时间记录与考勤系统 环境监测、航空航天、企业管理 3 B21D-037;G08B-007;E04H-012;F28F-013 计算机硬件与软件、机械测试与测量、金属成型与加工 设备监测、零部件制造、金融科技
下载: 导出CSV
A1 LLM 模型的参数设置
A1 LLM model parameter settings
模型 参数设置 activation _ function (激活函数): gelu _ new n _ embd (嵌入维度): 1600 n_layer (Transformer 层的数量): 48 n_head (每个 Transformer 层中的自注意力头数): 25 vocab_size (词汇表大小): 50257 n_ctx (上下文窗口大小): 1024 n_positions (最大位置编码数): 1024 gpt2-xl initializer_range (初始化范围): 0.02 layer_norm_epsilon (层归一化的 epsilon 值): 1e-5 attn_pdrop (注意力层的丢弃率): 0.1 embd_pdrop (嵌入层的丢弃率): 0.1 resid_pdrop (残差连接的丢弃率): 0.1 bos_token_id (开始标记 ID): 50256 eos_token_id (结束标记 ID): 50256 max_length (生成文本的最大长度): 16 hidden_size (隐藏层大小): 2048 num_attention_heads(每个 Transformer 层中的自注意力头数):16 num_hidden_layers(Transformer 层的数量): 24 vocab_size (词汇表大小): 32000 max_position_embeddings (最大位置编码数): 4096 intermediate_size (前馈神经网络的隐藏层大小): 5504 Sheared-LLaMA-1.3B hidden_act (隐藏层激活函数): silu bos_token_id (开始标记 ID): 1 eos_token_id (结束标记 ID): 2 pad_token_id (填充标记 ID): 0 pretraining_tp (预训练 TP): 1 rms_norm_eps (RMS 层归一化的 epsilon 值): le-5 rope_theta (位置编码的 theta 值): 10000.0 max_length (生成文本的最大长度): 16 activation_function (激活函数): gelu n_embd (嵌入维度): 768 n_layer (Transformer 层的数量): 12 n_head (每个 Transformer 层中的自注意力头数): 12 vocab_size (词汇表大小): 30522 n_ctx (上下文窗口大小): 512 bert-base-uncased n_positions (最大位置编码数): 512 layer_norm_epsilon (层归一化的 epsilon 值): 1e-12 attn_pdrop (注意力层的丢弃率): 0.1 hidden_dropout_prob (隐藏层丢弃率): 0.1 bos_token_id (开始标记 ID): 101 eos_token_id (结束标记 ID): 102 pad_token_id (填充标记 ID): 0 position_embedding_type (位置编码类型): absolute Linear Layer: 输入维度: 768 输出维度: 16
下载: 导出CSV
-
[1] Zhao J Y, Dong, Z J, Yao X L, et al. Optimizing collaboration decisions in technological innovation through machine learning: identify trend and partners in collaboration-knowledge interdependent networks. Annals of Operations Research, 2024, 1-42 [2] 连芷萱, 王芳, 康佳, 等. 基于图神经网络和粒子群算法的技术预测模型. 情报学报, 2023, 42(4): 420−435Lian Z X, Wang F, Kang J et al. Graph Neural Network-based and Particle Swarm Optimization Technological Prediction Model (in Chinese). Journal of the China Society for Scientific and Technical Information, 2023, 42(4): 420−435 [3] Xi X, Ren F F, Yu L A, Yang J. Detecting the technology's evolutionary pathway using HiDS-trait-driven tech mining strategy. Technological Forecasting & Social Change, 2023, 195: 122777 [4] 潘文雯, 赵洲, 俞俊, 吴飞. 基于文本引导的注意力图像转发预测排序网络. 自动化学报, 2021, 47(11): 2547−2556Pan W W, Zhao Z, Yu J, Wu F. Textually Guided Ranking Network for Attentional Image Retweet Modeling. Automatica Sinica, 2021, 47(11): 2547−2556 [5] Daud N N, Ab Hamid S H, Saadoon M et al. Applications of link prediction in social networks: A review. Journal of Network and Computer Applications, 2020, 166: 102716 doi: 10.1016/j.jnca.2020.102716 [6] Lee W S, Han E J, Sohn S Y. Predicting the pattern of technology convergence using big-data technology on large-scale triadic patents. Technological Forecasting and Social change, 2015, 22(7): 317−329 [7] 张斌, 李亚婷. 学科合作网络链路预测结果的排序鲁棒性. 信息资源管理学报, 2018, 8(04): 89−97Zhang B, Li Y T. Ranking robustness of link prediction results in disciplinary collaboration network (in Chinese). Journal of Information Resources Management, 2018, 8(04): 89−97 [8] Xi X, Zhao J Y, Yu L A et al. Exploring the Potentials of Artificial Intelligence Towards Carbon Neutrality: Technological Convergence Forecasting Through Link Prediction and Community Detection. Computers & Industrial Engineering, 2024, 190: 110015 [9] Sasaki H, Sakata I. Identifying potential technological spin-offs using hierarchical information in international patent classification. Technovation, 2021, 100: 102192 doi: 10.1016/j.technovation.2020.102192 [10] 贾承丰, 韩华, 吕亚楠, 张路. 基于 Word2vec 和粒子群的链路预测算法. 自动化学报, 2020, 46(8): 1703−1713Jia C F, Han H, Lv Y N, Zhang L. Link Prediction Algorithm Based on Word2vec and Particle Swarm. Acta Automatica Sinica, 2020, 46(8): 1703−1713 [11] 王守辉, 于洪涛, 黄瑞阳, 马青青. 基于模体演化的时序链路预测方法. 自动化学报, 2016, 42(5): 735−745Wang S H, Yu H T, Huang R Y, Ma Q Q. A Temporal Link Prediction Method Based on Motif Evolution. Automatica Sinica, 2016, 42(5): 735−745 [12] Kwon O, An Y, Kim M, Lee et al. Anticipating technology-driven industry convergence: evidence from large-scale patent analysis. Technology Analysis & Strategic Management, 2020, 32(4): 363−378 [13] Kipf T N, Welling M. Semi-Supervised Classification with Graph Convolutional Networks, In: Proceedings of the 5th International Conference on Learning Representations, 2017 [14] Zhou L X, Schellaert W, Martinez-Plumed F et al. Larger and more instructable language models become less reliable. Nature, 2024, 634: 61−68 doi: 10.1038/s41586-024-07930-y [15] 宁传峰. 随机配置网络不平衡数据分类算法研究. 中国矿业大学[D], 2023. [16] Lee C, Hong S, Kim J. Anticipating multi-technology convergence: a machine learning approach using patent information. Scientometrics, 2021, 126: 1867−1896 doi: 10.1007/s11192-020-03842-6 [17] Wang J, Lee J J. Predicting and Analyzing Technology Convergence for Exploring Technological Opportunities in the Smart Health Industry. Computers & Industrial Engineering, 2023109352 [18] Hong S C, Lee C Y. Effective Indexes and Classification Algorithms for Supervised Link Prediction Approach to Anticipating Technology Convergence: A Comparative Study. IEEE transactions on engineering, 2023, 70(4): 1430 doi: 10.1109/TEM.2021.3098602 [19] Grover A, Leskovec J. Node2vec: scalable feature learning for networks. KDD '16: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016855−864 [20] Dastile, X., Celik, T., & Potsane, M. Statistical and machine learning models in credit scoring: A systematic literature survey. Applied Soft Computing, 2000, 91: 106263 [21] Kim T S, Sohn S Y. Machine-learning-based deep semantic analysis approach for forecasting new technology convergence. Technological Forecasting & Social Change, 2020, 157: 120095 [22] Zhao H, Zhao C, Zhang X et al. An ensemble learning approach with gradient resampling for class-imbalance problems. INFORMS Journal on Computing, 2023, 35(4): 747−763 doi: 10.1287/ijoc.2023.1274 [23] Kumar A, Singh S S, Singh K et al. Link prediction techniques, applications, and performance: A survey. Physica A: Statistical Mechanics and its Applications, 2020, 553: 124289 doi: 10.1016/j.physa.2020.124289 [24] Kim K S, Cho N W. Predicting the patterns of technology convergence in defense technologies. In 2022 IEEE International Conference on Big Data and Smart Computing (BigComp). 2022: 72-75 -
下载:
计量
- 文章访问数: 80
- HTML全文浏览量: 56
- 被引次数: 0
