wangyu8460958 最近的时间轴更新 wangyu8460958 最近的时间轴更新 |
wangyu8460958V2EX 第 227278 号会员，加入于 2017-04-22 10:01:56 +08:00 |
| 使用 A 机器上的 Python 脚本将 B 机器(DB Server)上的查询信息通过邮件发送出去 Python • wangyu8460958 • 2017-05-09 21:29:39 PM |
| Python 队列脚本执行时报错 Warning: Truncated incorrect DOUBLE value: '[2627290]' r = r + self.execute(query, a) Python • wangyu8460958 • 2017-05-05 18:23:54 PM |
| 定义 1 个数组 a，它的长度为 1000,不设置初始值，应该如何来写？ 编程 • wangyu8460958 • 2017-04-27 15:37:13 PM |
| 请问下在 Python 中如何将一个数组的值赋给另外一个数组 问与答 • wangyu8460958 • 2017-04-25 16:11:41 PM • 最后回复来自 konia | 1 |
| Python 如何分批次导入 100W 条 mysql 数据 Python • wangyu8460958 • 2017-04-27 15:53:42 PM • 最后回复来自 wangyu8460958 | 4 |
wangyu8460958 最近回复了
2017-04-27 15:53:42 +08:00 回复了 wangyu8460958 创建的主题 › Python › Python 如何分批次导入 100W 条 mysql 数据 |
下面是我自己写的脚本，下面的脚本还能够优化吗？
#coding=utf-8
import MySQLdb
import numpy as np
conn = MySQLdb.connect(host='localhost',port = 3306,user='root',passwd='123456',db='google')
cur = conn.cursor()
conn.autocommit(1)
query_sql = "select id from google.Video where created_at < date_sub(now(), interval 1 year);"
insert_sql = "insert ignore into tmp.Video_2017bak (select * from google.Video where id = %s);"
delete_sql = "delete from google.Video where id = %s;"
cur.execute(query_sql)
dataList = cur.fetchall()
aaa = np.array(dataList)
ids = []
for i in range(len(aaa)):
ids.append(aaa[i])
if (i+1)%100==0 :
cur.executemany(insert_sql,ids)
ids = []
cur.executemany(insert_sql,ids)
ids = []
for i in range(len(aaa)):
ids.append(aaa[i])
if (i+1)%100==0 :
cur.executemany(delete_sql,ids)
ids = []
cur.executemany(delete_sql,ids)
ids = []
cur.close()
conn.close()
#coding=utf-8
import MySQLdb
import numpy as np
conn = MySQLdb.connect(host='localhost',port = 3306,user='root',passwd='123456',db='google')
cur = conn.cursor()
conn.autocommit(1)
query_sql = "select id from google.Video where created_at < date_sub(now(), interval 1 year);"
insert_sql = "insert ignore into tmp.Video_2017bak (select * from google.Video where id = %s);"
delete_sql = "delete from google.Video where id = %s;"
cur.execute(query_sql)
dataList = cur.fetchall()
aaa = np.array(dataList)
ids = []
for i in range(len(aaa)):
ids.append(aaa[i])
if (i+1)%100==0 :
cur.executemany(insert_sql,ids)
ids = []
cur.executemany(insert_sql,ids)
ids = []
for i in range(len(aaa)):
ids.append(aaa[i])
if (i+1)%100==0 :
cur.executemany(delete_sql,ids)
ids = []
cur.executemany(delete_sql,ids)
ids = []
cur.close()
conn.close()
2017-04-22 20:49:13 +08:00 回复了 wangyu8460958 创建的主题 › Python › Python 如何分批次导入 100W 条 mysql 数据 |
@skydiver 我想使用脚本让它每周执行一次，所以想使用 python 脚本来做。
Select Language